The Conductor Interaction Method (Rachovides et al)

Interesting concept combining gaze input with hand gestures by Dorothy Rachovides at the Digital World Research Centre together with James Walkerdine and Peter Phillips at the Computing Department Lancaster University.

"This article proposes an alternative interaction method, the conductor interaction method (CIM), which aims to provide a more natural and easier-to-learn interaction technique. This novel interaction method extends existing HCI methods by drawing upon techniques found in human-human interaction. It is argued that the use of a two-phased multimodal interaction mechanism, using gaze for selection and gesture for manipulation, incorporated within a metaphor-based environment, can provide a viable alternative for interacting with a computer (especially for novice users). Both the model and an implementation of the CIM within a system are presented in this article. This system formed the basis of a number of user studies that have been performed to assess the effectiveness of the CIM, the findings of which are discussed in this work.


More specifically the CIM aims to provide the following.

—A More Natural Interface. The CIM will have an interface that utilizes gaze and gestures, but is nevertheless capable of supporting sophisticated activities. The CIM provides an interaction technique that is as natural as possible and is close to the human-human interaction methods with which users are already familiar. The combination of gaze and gestures allows the user to perform not only simple interactions with a computer, but also more complex interacones such as the selecting, editing, and placing of media objects.



—A Metaphor Supported Interface. In order to help the user understand and exploit the gaze and gesture interface, two metaphors have been developed. An orchestra metaphor is used to provide the environment in which the user interacts. A conductor metaphor is used for interacting within this environment. These two metaphors are discussed next.

—A Two-Phased Interaction Method. The CIM uses an interaction process where each modality is specific and has a particular function. The interaction between user and interface can be seen as a dialog that is comprised of two phases. In the first phase, the user selects the on-screen object by gazing at it. In the second phase, with the gesture interface the user is able to manipulate the selected object. These distinct functions of gaze and gesture aim to increase system usability, as they are based on human-human interaction techniques, and also help to overcome issues such as the Midas Touch problem that often experienced by look-and-dwell systems. As the dialog combines two modalities in sequence, the gaze interface can be disabled after the first phase. This minimizes the possibility of accidentally selecting objects through the gaze interface. The Midas Touch problem can also be further addressed by ensuring that there is ample dead space between media objects.

—Significantly Reduced Learning Overhead. The CIM aims to reduce the overhead of learning to use the system by encouraging the use of gestures that users can easily associate with activities they perform in their everyday life. This transfer of experience can lead to a smaller learning overhead [Borchers 1997], allowing users to make the most of the system’s features in a shorter time.

• Rachovides, D., Walkerdine, J., and Phillips, P. 2007. The conductor interaction method. ACM Trans. Multimedia Comput. Commun. Appl. 3, 4 (Dec. 2007), 1-23. DOI= http://doi.acm.org/10.1145/1314303.1314312

Gaze and Voice Based Game Interaction (Wilcox et al., 2008)

"We present a 3rd person adventure puzzle game using a novel combination of non intrusive eyetracking technology and voice recognition for game communication. Figure 1 shows the game, and its first person sub games that make use of eye tracker functionality in contrasting ways: a catapult challenge (a) and a staring competition(b)."

"There are two different modes of control in the main game. The user can select objects by looking at them and perform ’look’, ’pickup’, ’walk’, ’speak’, ’use’ and other commands by vocalizing there respective words. Alternatively, they can perform each command by blinking and winking at objects. To play the catapult game for example, the user must look at the target and blink, wink or drag to fire a projectile towards the object under the crosshair. "

Their work was presented at the ACM SIGGRAPH 2008 with the associated poster:

• Wilcox, T., Evans, M., Pearce, C., Pollard, N., and Sundstedt, V. 2008. Gaze and voice based game interaction: the revenge of the killer penguins. In ACM SIGGRAPH 2008 Posters (Los Angeles, California, August 11 - 15, 2008).

Low cost open source eye tracking from Argentina

By using low cost webcams such as the Lifecam VX-100 or similar this person from Argentina have produced an eye tracker capable of running the Gaze Talk interface. The total cost for the eye tracker hardware is US$ 40-50. The software runs on a typical desktop or laptop computer using the OpenCV based image processing algorithms.

"My goal is to develop an open source system that enables people with severe motor disabilities to interact with the computer using their eye movements."

The project is running for another three weeks and the outcome will be very interesting. Check out the development blog at http://www.eyegazetracking.com/

The Inspection of Very Large Images by Eye-gaze Control

Nicholas Adams, Mark Witkowski and Robert Spence from the Department of Electrical and Electronic Engineering at the Imperial College London got the HCI 08 Award for International Excellence for work related to gaze interaction.

"The researchers presented novel methods for navigating and inspecting extremely large images solely or primarily using eye gaze control. The need to inspect large images occurs in, for example, mapping, medicine, astronomy and surveillance, and this project considered the inspection of very large aerial images, held in Google Earth. Comparative search and navigation tasks suggest that, while gaze methods are effective for image navigation, they lag behind more conventional methods, so interaction designers might consider combining these techniques for greatest effect." (BCS Interaction)

Abstract

The increasing availability and accuracy of eye gaze detection equipment has encouraged its use for both investigation and control. In this paper we present novel methods for navigating and inspecting extremely large images solely or primarily using eye gaze control. We investigate the relative advantages and comparative properties of four related methods: Stare-to-Zoom (STZ), in which control of the image position and resolution level is determined solely by the user's gaze position on the screen; Head-to-Zoom (HTZ) and Dual-to-Zoom (DTZ), in which gaze control is augmented by head or mouse actions; and Mouse-to-Zoom (MTZ), using conventional mouse input as an experimental control.

The need to inspect large images occurs in many disciplines, such as mapping, medicine, astronomy and surveillance. Here we consider the inspection of very large aerial images, of which Google Earth is both an example and the one employed in our study. We perform comparative search and navigation tasks with each of the methods described, and record user opinions using the Swedish User-Viewer Presence Questionnaire. We conclude that, while gaze methods are effective for image navigation, they, as yet, lag behind more conventional methods and interaction designers may well consider combining these techniques for greatest effect.

• Download paper as PDF
  

This paper is the short version of Nicolas Adams Masters thesis which I stumbled upon before creating this blog. A early version appeared as a short paper for COGAIN06.

Apple develops gaze assisted interaction?

Apple recently registered a patent for merging several modalities including gaze vectors for novel interaction methods. The direction of gaze is to be used in combination with finger gestures (or other input devices) to modify the object that the user is currently looking at. Will be interesting to see what types of devices they are aiming for. May not be high precision eye tracking since stability and high accuracy is hard to obtain for a 100% population in all environments.

From the patent document:
"There are many possible applications that would benefit from the temporal fusion of gaze vectors with multi-touch movement data. For the purpose of example, one simple application will be discussed here: Consider a typical computer screen, which has several windows displayed. Assume that the user wishes to bring forward the window in the lower left corner, which is currently underneath two other windows. Without gaze vector fusion there are two means to do this, and both involve movement of the hand to another position. The first means is to move the mouse pointer over the window of interest and click the mouse button. The second means is to use a hot-key combination to cycle through the screen windows until the one of interest is brought forward. Voice input could also be used but it would be less efficient than the other means. With gaze vector fusion, the task is greatly simplified. For example, the user directs his gaze to the window of interest and then taps a specific chord on the multi-touch surface. The operation requires no translation of the hands and is very fast to perform."

"For another example, assume the user wishes to resize and reposition an iTunes window positioned in the upper left of a display screen. This can be accomplished using a multi-touch system by moving the mouse pointer into the iTunes window and executing a resize and reposition gesture. While this means is already an improvement over using just a mouse its efficiency can be further improved by the temporal fusion of gaze vector data. "

• Download patent documents.

TeleGaze (Hemin, 2008)

"This research investigates the use of eye-gaze tracking in controlling the navigation of mobile robots remotely through a purpose built interface that is called TeleGaze. Controlling mobile robots from a remote location requires the user to continuously monitor the status of the robot through some sort of feedback system. Assuming that a vision-based feedback system is used such as video cameras mounted onboard the robot; this requires the eyes of the user to be engaged in the monitoring process throughout the whole duration of the operation. Meanwhile, the hands of the user need to be engaged, either partially or fully, in the driving task using any input devices. Therefore, the aim of this research is to build a vision based interface that enables the user to monitor as well as control the navigation of the robot using only his/her eyes as inputs to the system since the eyes are engaged in performing some tasks anyway. This will free the hands of the user for other tasks while controlling the navigation is done through the TeleGaze interface. "

TeleGaze experimental platform consists of a mobile robot, an eye gaze tracking equipment and a teleoperation station that the user interacts with. The TeleGaze interface runs on the teleoperation station PC and interprets inputs from the eyes into controlling commands. Meanwhile, presenting the user with the images that come back from the vision system mounted on the robotic platform.

More information at Hemin Sh. Omers website.

Associated publications:

• Hemin Omer Latif, Nasser Sherkat and Ahmad Lotfi, "TeleGaze: Teleoperation through Eye Gaze", 7th IEEE International Conference on Cybernetic Intelligent Systems 2008, London, UK. Conference website: www.cybernetic.org.uk/cis2008

• Hemin Omaer Latif, Nasser Sherkat and Ahmad Lotfi, "Remote Control of Mobile Robots through Human Eye Gaze: The Design and Evaluation of an Interface", SPIE Europe Security and Defence 2008, Cardiff, UK. Conference website: http://spie.org/security-defence-europe.xml

COGAIN 2008 Proceedings now online

Download the proceedings as PDF (3.6Mb)

Contents

Overcoming Technical Challenges in Mobile and Other Systems

• Off-the-Shelf Mobile Gaze Interaction
  J. San Agustin and J. P. Hansen, IT University of Copenhagen, Denmark
  

• Fast and Easy Calibration for a Head-Mounted Eye Tracker
  C. Cudel, S Bernet, and M Basset, University of Haute Alsace, France

• Magic Environment
  L. Figueiredo, T. Nunes, F. Caetano, and A. Gomes, ESTG/IPG, Portugal

• AI Support for a Gaze-Controlled Wheelchair
  P. Novák, T. Krajník, L. Přeučil, M. Fejtová, and O. Štěpánková. Czech Technical University, Czech Republic)

• A Comparison of Pupil Centre Estimation Algorithms
  D. Droege, C Schmidt, and D. Paulus University of Koblenz-Landau, Germany
  
  

Broadening Gaze-Based Interaction Techniques

• User Performance of Gaze-Based Interaction with On-line Virtual Communities
  H. Istance, De Montfort University, UK, A. Hyrskykari, University of Tampere, Finland, S. Vickers, De Montfort University, UK and N. Ali, University of Tampere, Finland
  
  
• Multimodal Gaze Interaction in 3D Virtual Environments
  E. Castellina and F. Corno, Politecnico di Torino, Italy
  

• How Can Tiny Buttons Be Hit Using Gaze Only?
  H. Skovsgaard, J. P. Hansen, IT University of Copenhagen, Denmark. J. Mateo, Wright State University, Ohio, US
  

• Gesturing with Gaze
  H. Heikkilä, University of Tampere, Finland
  

• NeoVisus: Gaze Driven Interface Components
  M. Tall, Sweden
  
  

Focusing on the User: Evaluating Needs and Solutions

• Evaluations of Interactive Guideboard with Gaze-Communicative Stuffed-Toy Robot
  T. Yonezawa, H. Yamazoe, A. Utsumi, and S. Abe, ATR Intelligent Robotics and Communications Laboratories, Japan
  

• Gaze-Contingent Passwords at the ATM
  P. Dunphy, A. Fitch, and P. Oliver, Newcastle University, UK
  

• Scrollable Keyboards for Eye Typing
  O Špakov and P. Majaranta, University of Tampere, Finland
  

• The Use of Eye-Gaze Data in the Evaluation of Assistive Technology Software for Older People.
  S. Judge, Barnsley District Hospital Foundation, UK and S. Blackburn, Sheffield University, UK
  

• A Case Study Describing Development of an Eye Gaze Setup for a Patient with 'Locked-in Syndrome' to Facilitate Communication, Environmental Control and Computer Access.
  Z. Robertson and M. Friday, Barnsley General Hospital, UK

COGAIN 2008 Video

Some highlights from the visit to COGAIN 2008 last week in Prague which was a great event. It demonstrates the mobile solution integrating a head mounted display and an eye tracker by Javier San Agustín. A sneak peak of the NeoVisus iTube interface running on the SMI IViewX RED. A demonstration of the Neural Impulse Actuator from OCZ Technolgies by Henrik Skovsgaard. Demo of the gaze controlled wheelchair developed by Falck Igel and Alea Technologies. Thanks to John Paulin Hansen for creating the video.

Mixed reality systems for technical maintenance and gaze-controlled interaction (Gustafsson et al)

To follow up on the wearable display with an integrated eye tracker one possible application is in the domain of mixed reality. This allows for interfaces to be projected on top of a video stream (ie. the "world view") Thus blending the physical and virtual world. The paper below investigates how this could be used to assist technical maintenance of advanced systems such as fighter jets. It´s an early prototype but the field is very promising especially when an eye tracker is involved.

Abstract:

"The purpose of this project is to build up knowledge about how future Mixed Reality (MR) systems should be designed concerning technical solutions, aspects of Human-Machine-Interaction (HMI) and logistics. The report describes the work performed in phase2. Regarding hardware a hand-held MR-unit, a wearable MR-system and a gaze-controlled MR-unit have been developed. The work regarding software has continued with the same software architecture and MR-tool as in the former phase 1. A number of improvements, extensions and minor changes have been conducted as well as a general update. The work also includes experiments with two test case applications, "Turn-Round af Gripen (JAS) and "Starting Up Diathermy Apparatus" Comprehensive literature searches and surveys of knowledge of HMI aspects have been conducted, especially regarding gaze-controlled interaction. The report also includes a brief overview of ohter projects withing the area of Mixed Reality."

• Gustafsson, T., Carleberg, P., Svensson, P., Nilsson, S., Le Duc, M., Sivertun, Å., Mixed Reality Systems for Technical Maintenance and Gaze-Controlled Interaction. Progress Report Phase 2 to FMV., 2005. Download paper as PDF

Nokia Research: Near Eye Display with integrated eye tracker

During my week in Tampere I had the opportunity to visit Nokia Research to get a hands on with a prototype that integrates a head mounted display with an eye tracker. Due to a NDA I am unable to reveal the contents of the discussion but it does work and it was a very neat experience with great potential. Would love to see a commercial application down the road. For more information there is a paper available:

• T. Järvenpää, V. Aaltonen (2008) Compact near-to-eye display with integrated gaze tracker. (SPIE Proceedings paper)
  
  

Hands-On with the Nokia NED w/ integrated eye tracker

Paper abstract:
"Near-to-Eye Display (NED) offers a big screen experience to the user anywhere, anytime. It provides a way to perceive a larger image than the physical device itself is. Commercially available NEDs tend to be quite bulky and uncomfortable to wear. However, by using very thin plastic light guides with diffractive structures on the surfaces, many of the known deficiencies can be notably reduced. These Exit Pupil Expander (EPE) light guides enable a thin, light, user friendly and high performing see-through NED, which we have demonstrated. To be able to interact with the displayed UI efficiently, we have also integrated a video-based gaze tracker into the NED. The narrow light beam of an infrared light source is divided and expanded inside the same EPEs to produce wide collimated beams out from the EPE towards the eyes. Miniature video camera images the cornea and eye gaze direction is accurately calculated by locating the pupil and the glints of the infrared beams. After a simple and robust per-user calibration, the data from the highly integrated gaze tracker reflects the user focus point in the displayed image which can be used as an input device for the NED system. Realizable applications go from eye typing to playing games, and far beyond."