Eye Gaze Interactive Air Traffic Controllers workstation (P.Esser & T.J.J Bos, 2007)

P.Esser and T.J.J Bos at the Maastricht University have developed a prototype for reducing the repetitive strain injuries Air Traffic Controllers sustain while operating their systems. The research was conducted at the National Aerospace Laboratory in the Netherlands. The results indicate a clear advantage compared to the traditional roller/track ball, especially for large distances. This is expected since Fitt's law does not apply in the same manner for eye movement as physical limb/hand movement. Sure eye movement over longer distances takes more time to perform than short ones but it does not compare to moving you arm one inch vs. one meter. Certainly there are more applications that could benifit from gaze assisted interaction, medical imaging in the field of radiology is one (such as CT, MRI, these produce very high resolution images with resolutions up to 4096x4096 pixels)

Summary of the thesis "Eye Gaze Interactive ACT workstation"
"Ongoing research is devoted to finding ways to improve performance and reduce workload of Air Traffic Controllers (ATCos) because their task is critical to the safe and efficient flow of air traffic. A new intuitive input method, known as eye gaze interaction, was expected to reduce the work- and task load imposed on the controllers by facilitating the interaction between the human and the ATC workstation. In turn, this may improve performance because the freed mental resources can be devoted to more critical aspects of the job, such as strategic planning. The objective of this Master thesis research was to explore how human computer interaction (HCI) in the ATC task can be improved using eye gaze input techniques and whether this will reduce workload for ATCos.


In conclusion, the results of eye gaze interaction are very promising for selection of aircraft on a radar screen. For entering instructions it was less advantageous. This is explained by the fact that in the first task the interaction is more intuitive while the latter is more a conscious selection task. For application in work environments with large displays or multiple displays eye gaze interaction is considered very promising. "



Download paper as pdf

GazeTalk 5

The GazeTalk system is one of the most comprehensive open solutions for gaze interaction today. It has been developed with the disabled users in mind and supports a wide range of everyday tasks. It dramatically increases the quality of life for the disabled suffering from ALS or similar conditions. The following information is quoted from the COGAIN website.

Information about Gazetalk 5 eye communication system

GazeTalk is a predictive text entry system that has a restricted on-screen keyboard with ambiguous layout for severely disabled people. The main reason for using such a keyboard layout is that it enables the use of an eye tracker with a low spatial resolution (e.g., a web-camera based eye tracker).

The goal of the GazeTalk project is to develop an eye-tracking based AAC system that supports several languages, facilitates fast text entry, and is both sufficiently feature-complete to be deployed as the primary AAC tool for users, yet sufficiently flexible and technically advanced to be used for research purposes. The system is designed for several target languages, initially Danish, English, Italian, German and Japanese.

Main features

• type-to-talk
• writing
• email
• web – browser
• Multimedia – player
• PDF – reader
• letter and word prediction, and word completion
• speech output
• can be operated by gaze, headtracking, mouse, joystick, or any other pointing device
• supports step-scanning (new!)
• supports users with low precision in their movements, or trackers with low accuracy
• allows the user to use Dasher inside GazeTalk and to transfer the text written in Dasher back to GazeTalk

GazeTalk 5.0 has been designed and developed by the Eye Gaze Interaction Group at the IT University of Copenhagen and the IT-Lab at the Royal School of Library and Information Science, Copenhagen.

Read more About Gazetalk or view GazeTalk manual

Short manual on data recording in GazeTalk

GazeTalk Videos

• Introduction to GazeTalk and its features
  Click here to play the video
  
• Watch the video "Introduction to GazeTalk and its features" (same as above!) in Windows Media File (wmv) format (7.5 MB).
• Watch a YouTube video on Gaze Interaction for People with ALS/MND .
• Watch another YouTube video on a Talk Given with the Eyes Only by Arne Lykke Larsen
• Watch a YouTube video on ALS-Communication and GazeTalk , where the developer, Dr. John Paulin Hansen explains how GazeTalk works (in Danish, subtitles in English). The video includes short clips of using GazeTalk and a brief interview of John Paulin Hansen. The video was produced by www.synvision.dk, intiative: Birger Bergmann Jeppesen (bigerbj (at) webspeed (dot) dk), for more information, see www.als-communication.dk)
  

Download Gazetalk

Eye tracking using a webcamera

From the French ESIEA school of engineering comes a custom developed eye tracker using a simple web camera. It has head tracking capabilities and works in low light situations.

eye tracking from Hubert Wassner on Vimeo.


eye-tracking V2 from Hubert Wassner on Vimeo.


eye command from Hubert Wassner on Vimeo.

More information:
Hubert Wassner Blog (prof. of Comp.Sci) French / English (automatic translation)

Eye tracking in space

The Eye Tracking Device (ETD) is used to determine the influence of prolonged microgravity and the accompanying vestibular (inner ear) adaptation on the orientation of Listings Plane (a coordinate framework, which is used to define the movement of the eyes in the head).

"The working hypothesis is that in microgravity the orientation of Listings Plane is altered, probably to a small and individually variable degree. Further, with the loss of the otolith-mediated gravitational reference, it is expected that changes in the orientation of the coordinate framework of the vestibular system occur, and thus a divergence between Listing?s Plane and the vestibular coordinate frame should be observed. While earlier ground-based experiments indicate that Listing?s Plane itself is to a small degree dependent on the pitch orientation to gravity, there is more compelling evidence of an alteration of the orientation of the vestibulo-ocular reflex (VOR), reflex eye movement that stabilizes images on the retina during head movement by producing an eye movement in the direction opposite to head movement, thus preserving the image on the center of the visual field, in microgravity.

Furthermore, changes in bodily function with relation to eye movement and spatial orientation that occur during prolonged disturbance of the vestibular system most likely play a major role in the problems with balance that astronauts experience following re-entry from space.

In view of the much larger living and working space in the ISS, and the extended program of spacewalks (EVAs) being planned, particular care must be given to assessing the reliability of functions related to eye movement and spatial orientation.

The performance of the experiments in space are therefore of interest for their expected contribution to basic research knowledge and to the improvement and assurance of human performance under weightless conditions."


NASA Image: ISS011E13710 - Cosmonaut Sergei K. Krikalev, Expedition 11 Commander representing Russia's Federal Space Agency, uses the Eye Tracking Device (ETD), a European Space Agency (ESA) payload in the Zvezda Service Module of the International Space Station. The ETD measures eye and head movements in space with great accuracy and precision.

"The ETD consists of a headset that includes two digital camera modules for binocular recording of horizontal, vertical and rotational eye movements and sensors to measure head movement. The second ETD component is a laptop PC, which permits digital storage of all image sequences and data for subsequent laboratory analysis. Listing's Plane can be examined fairly simply, provided accurate three-dimensional eye-in-head measurements can be made. Identical experimental protocols will be performed during the pre-flight, in-flight and post-flight periods of the mission. Accurate three-dimensional eye-in-head measurements are essential to the success of this experiment. The required measurement specifications (less than 0.1 degrees spatial resolution, 200 Hz sampling frequency) are fulfilled by the Eye Tracking Device (ETD)."

More information:
http://www.nasa.gov/mission_pages/station/science/experiments/ETD.html
http://www.spaceflight.esa.int/delta/documents/factsheet-delta-hp-etd.pdf
http://www.energia.ru/eng/iss/researches/medic-65.html

Realtime computer interaction via eye tracking (Dubey, 2004)

Premnath Dubey conducted research on eye tracking and gaze interaction for his masters thesis in 2004 at the Department of Computing at Curtin University in Australia.

Abstract
"This thesis presents a computer vision-based eye tracking system for human computer interaction. The eye tracking system allows the user to indicate a region of interest in a large data space and to magnify that area, without using traditional pointer devices. Presented is an iris tracking algorithm adapted from Camshift; an algorithm originally designed for face or hand tracking. Although the iris is much smaller and highly dynamic. the modified Camshift algorithm efficiently tracks the iris in real-time. Also presented is a method to map the iris centroid, in video coordinates to screen coordinates; and two novel calibration techniques, four point and one-point calibration. Results presented show that the accuracy for the proposed one-point calibration technique exceeds the accuracy obtained from calibrating with four points. The innovation behind the one-point calibration comes from using observed eye scanning behaviour to constrain the calibration process. Lastly, the thesis proposes a non-linear visualisation as an eye-tracking application, along with an implementation."

Download paper as PDF.

Low cost eye tracking

Marcelo from Argentina have developed a low cost solution using the Logitech Quickcam Express webcamera. The video it produces has a resolution of 352 x 288 pixels. It is mounted close to the eye and with extra illumination from two lamps. Marcelos crude eye tracker relies on an elliptic fitting of the pupil in the visible light spectrum which differes from most commercial alternatives (which uses infrared light to create reflections on the eye ball, this is typically the second step to increase the accuracy)

Considering the low resolution of the camera and the simplicity of the setup the results are noteworthy. Hope to see more development of this!

Hot Zone prototype (Wakaruru)

The following video demonstrates a prototype of the Hot Zone system for controling windows applications. Call out the Hot Zone are made by pressing a single Hotkey and blink. The menu can be closed by looking outside the zone and blinking. Submitted to YouTube by "Wakaruru"

Submitted to YouTube by "Wakaruru"

The second video demonstrates how Hot Zone could be used to work with real world applications (PowerPoint). The center of the zone is located at the current gaze position when it's call out (like a context menu). the commands in the five zones depend on current selected object (based on what you are looking at now). No mouse needed (the cursor was designated as the gaze position using API). Pure blink without hotkey pressed was designated as single mouse click. Thus blink on the text area start text editing. Keyboard is only used for typing and hotkey to call out the zone. The eye tracking device used is ASL EH6000.

Submitted to YouTube by "Wakaruru"

Lund Eye Tracking Acadamy

From the Lund Eye Tracking Academy (LETA) comes an excellent text on the more practical aspects of eye tracking in research settings.

"This text is about how to record good quality eye-tracking data from commercially available video-oculographic eye-tracking system, and how to derive and use the measures that an eye-tracker can give. The ambition is to cover as many as possible of the measures used in science and applied research. The need for a guide on how to use eye-tracking data has grown during the past years, as an effect of increasing interest in eye-tracking research. Due to the versatility of new measurement techniques and the important role of human vision in all walks of life, eye-tracking is now used in reading research, neurology, advertisement studies, psycholinguistics, human factors, usability studies, scene perception, and many other fields "

Download the document (PDF, 93 pages, 19Mb)

Open Source Eye Tracking

Zafer Savas is an electronics engineer living in Ankara in Turkey who has released an eye tracker implemented in C++ using the OpenCV library. The project is well documented and enables crude DIY eye tracking within minutes, hence it is a good starting point. It contains a sample database of eye images used to train the algorithm, the program has a function to capture images so that users can build their own database. Great!

Download the executable as well as the source code. See the extensive documentation.

You will also need to download the OpenCV library.

"The purpose of the project is to implement a real-time eye-feature tracker with the following capabilities:

• RealTime face tracking with scale and rotation invariance
• Tracking the eye areas individually
• Tracking eye features
• Eye gaze direction finding
• Remote controlling using eye movements

The implemented project is on three components:

1. Face detection: Performs scale invariant face detection
2. Eye detection: Both eyes are detected as a result of this step
3. Eye feature extraction: Features of eyes are extracted at the end of this step

Two different methods for face detection were implemented in the project:

1. Continuously Adaptive Means-Shift Algorithm
2. Haar Face Detection method

Two different methods of eye tracking were implemented in the project:

1. Template-Matching
2. Adaptive EigenEye Method

schematics for the procedure/algorithms

Response to demonstration video

The response for my video demonstration has been overwhelming with several articles and over 7000 viewings in a during the first three days (google neovisus). There has been an interest from the gaming community, Human–Computer Interaction researchers, users and caretakers in the assistive technology sector, medical and science applications such as radiology.

I wish give thanks to everyone for their feedback. At times during the long days and nights in the lab I've been wondering if this would be just another paper in the pile. The response has proven the interest and gives motivation for continuous work. Right now I feel that this is just the beginning and determined to take it towards the next level.

Please do not hesitate to contact me with ideas and suggestions.

Thank you.

Video demonstration online

The associated video demonstration for my masters thesis on gaze interaction is now online!

Click on the image to view or download the video.

Thesis release

My masters thesis on gaze interaction, the NeoVisus prototype, is now out! Download it by clicking on the image below.

Download PDF (3.3 MB)

MedioVis at University of Konstanz

German student Simon Fäh uses Speech Recognition and Eye-Tracking to get a more fluid control of the visual information seeking system MedioVis. (video in German)

Eye typing at the Bauhaus University of Weimar

The Psychophysiology and Perception group, part of the faculty of Media at the Bauhaus University of Weimar are conducting research on gaze based text entry. Their past research projects include the Qwerty on-screen dwell based keyboard, IWrite, pEYEWrite and StarWrite. Thanks to Mario Urbina for notification.

QWRTY

"Qwerty is based on dwell time selection. Here the user has to stare for 500 ms a determinate character to select it. QWERTY served us, as comparison base line for the new eye typing systems. It was implemented in C++ using QT libraries."

IWrite
"A simple way to perform a selection based on saccadic movement is to select an item by looking at it and confirm its selection by gazing towards a defined place or item. Iwrite is based on screen buttons. We implemented an outer frame as screen button. That is to say, characters are selected by gazing towards the outer frame of the application. This lets the text window in the middle of the screen for comfortable and safe text review. The order of the characters, parallel to the display borders, should reduce errors like the unintentional selection of items situated in the path as one moves across to the screen button.The strength of this interface lies on its simplicity of use. Additionally, it takes full advantage of the velocity of short saccade selection. Number and symbol entry mode was implemented for this editor in the lower frame. Iwrite was implemented in C++ using QT libraries."

• See Iwrite movie
• Download Iwrite movie (1.37Mb)

PEYEWrite
"Pie menus have already been shown to be powerful menus for mouse or stylus control. They are two-dimensional, circular menus, containing menu items displayed as pie-formed slices. Finding a trade-off between user interfaces for novice and expert users is one of the main challenges in the design of an interface, especially in gaze control, as it is less conventional and utilized than input controlled by hand. One of the main advantages of pie menus is that interaction is very easy to learn. A pie menu presents items always in the same position, so users can match predetermined gestures with their corresponding actions. We therefore decided to transfer pie menus to gaze control and try it out for an eye typing approach. We designed the Pie menu for six items and two depth layers. With this configuration we can present (6 x 6) 36 items. The first layer contains groups of five letters ordered in pie slices.."

• See pEYEwrite movie
• Download pEYEwrite movie (1.68Mb)

StarWrite
In StarWrite, selection is also based on saccadic movements to avoid dwell times. The idea of StarWrite is to combine eye typing movements with feedback. Users, mostly novices, tend to look to the text field after each selection to check what has been written. Here letters are typed by dragging them into the text field. This provides instantaneous visual feedback and should spare checking saccades towards text field. When a character is fixated, both it and its neighbors are highlighted and enlarged in order to facilitate the character selection. In order to use x- and y-coordinates for target selection, letters were arranged alphabetically on a half-circle in the upper part of the monitor. The text window appeared in the lower field. StarWrite provides a lower case as well, upper case, and numerical entry modes, that can be switched by fixating for 500 milliseconds the corresponding buttons, situated on the lower part of the application. There are also placed the space, delete and enter keys, which are driven by a 500 ms dwell time too. StarWrite was implemented in C++ using OpenGL libraries for the visualization."

• See StarWrite movie
• Download StarWrite movie (1.42Mb)

Associated publications

• Huckauf, A. and Urbina, M. H. 2008. Gazing with pEYEs: towards a universal input for various applications. In Proceedings of the 2008 Symposium on Eye Tracking Research & Applications (Savannah, Georgia, March 26 - 28, 2008). ETRA '08. ACM, New York, NY, 51-54. [URL] [PDF] [BIB]

• Urbina, M. H. and Huckauf, A. 2007. Dwell time free eye typing approaches. In Proceedings of the 3rd Conference on Communication by Gaze Interaction - COGAIN 2007, September 2007, Leicester, UK, 65--70. Available online at http://www.cogain.org/cogain2007/COGAIN2007Proceedings.pdf [PDF] [BIB]

• Huckauf, A. and Urbina, M. 2007. Gazing with pEYE: new concepts in eye typing. In Proceedings of the 4th Symposium on Applied Perception in Graphics and Visualization (Tubingen, Germany, July 25 - 27, 2007). APGV '07, vol. 253. ACM, New York, NY, 141-141. [URL] [PDF] [BIB]

• Urbina, M. H. and Huckauf, A. 2007. pEYEdit: Gaze-based text entry via pie menus. In Conference Abstracts. 14th European Conference on Eye Movements ECEM2007. Kliegl, R. & Brenstein, R. (Eds.) (2007), 165-165.

Project finalization and thesis defence

Last Thursday I held the defense for my masters thesis at the Cognitive Science department at Lund University. I wish to thank Dr. Björn Samuelsson for his excellent, and well beyond expected, review of the thesis. Björn comes from a background in Theoretical Physics / Complex Systems with an international recognition in Random Boolean Networks. I kindly appreciate his great detail of attention which raises the overall outcome of my project, hopefully to the level of international scientific publications.

I´ve put together a video demonstration that together with the thesis will be released this week. It demonstrates the U.I components and the prototype interface and will be posted to the major video sharing networks (YouTube etc.)

GaCIT 2008 - Summer School on Gaze, Communication, and Interaction Technology

August 18-22, 2008, Tampere, Finland

"Vision and visual communication are central in HCI. Researchers need to understand visual information processing and be equipped with appropriate research tools and methods for understanding users' visual processes.

The GaCIT summer school offers an intensive one-week camp where doctoral students and researchers can learn and refresh vision related skills and knowledge under the tutelage of leading experts in the area. There will be an emphasis on the use of eye tracking technology as a research tool. The program will include theoretical lectures and hands-on exercises, an opportunity for participants to present their own work, and a social program enabling participants to exchange their experiences in a relaxing and inspiring atmosphere."

The following themes and speakers will be included:

• Active Vision and Visual Cognition (Boris Velichkovsky)
  Topics will include issues in visual perception with relevant aspects of attention, memory, and communication.

• Hands-on Eye Tracking: Working with Images and Video (Andrew Duchowski)
  Topics will include: Eye tracking methodology and experimental design review; setting up an image study: feedforward visual inspection training; visualization and analysis; exercise: between-subjects static fixation analysis; and analysis of eye tracked video.

• Designing Eye-Gaze Interaction: Supporting Tasks and Interaction Techniques (Howell Istance)
  This part of the summer school will examine how user needs and tasks can be mapped on particular ways of designing gaze-based interaction. This will cover traditional desktop applications as well as 3D virtual communities and on-line games.

• Eye tracking in Web Search Studies (Edward Cutrell)
  Topics will include: Eye tracking research on web search and more general web-based analysis with a brief hands-on session to try out the techniques.

• Participant Presentations
  Time has been reserved for those participants that are doing or planning to do work related to the theme of the summer school to present and discuss their work or plans with the other participants.

Visit the website for more information.

SR Labs and I-MED Medical Console

From SR Labs in Italy comes the I-MED Medical Console. The interface contains some really nice interaction methods. The system is to be used in medical field by surgeons or medical staff alike whom can receive a benificial advantage of hands free interaction (viewing x-rays during surgery etc)

SR Labs have also developed the iABLE software suit which enables web browsing and sending emails. Unfortunally, there isn't much information is available in English. Seems to be the Italian alternative for MyTobii.

Lund Eye-Tracking Academy (LETA) June11-13, 2008

SMI and Lund University Humanities Lab in Lund, Sweden, will realize the 1st Lund Eye Tracking Academy (LETA). Hosted by Kenneth Holmqvist and his team, the course will take place on June 11-13, 2008.

"We have decided to start our "eye-tracking academy" to help students, researchers and labs who start with eye-tracking to get up and running, to give them a flying start. Although eye-tracking research can be both fascinating and extremely useful, doing good eye-tracking research requires a certain minimum knowledge. Without having a basic understanding of what can be done with an eye-tracker, how to design an experiment and how to analyse the data, the whole study runs of risk of just producing a lot data which cannot really answer any questions.

Currently, to our knowledge, there exists no recurring and open course in eye-tracking in Europe outside Lund and everybody is trained on a person-to-person level in the existing larger research labs. In Lund, we already hold a 10 week, 7.5 ECTS Master-level course in eye-tracking. Now we offer another course, an intensive 3 day course, open to all interested in eye-tracking, who want to get a flying start and acquire some basic understanding of how to run a scientifically sound eye-tracking experiment and get high quality data that they know how to analyze.

This training course is open for all researchers and investigators just before or in the early phases of using eye-tracking, and for users wanting to refresh their knowledge of their system. It is open for attendees from universities as well as from industry. As part of the course, attendees will work hands-on with sample experiments, from experimental design to data analysis. Participants will train on state-of-the-art SMI eye-tracking systems, but the course is largely hardware independent and open to users of other systems."

Scheduled dates
11-13 June, 2008. Starting at 09:00 in the morning and ending the last day at around 16:00.

Course contents
• Pro and cons of headmounted, remote and contact eye-trackers.
• High sampling speed and detailed precision – who needs it?
• Gaze-overlaid videos vs datafiles – what can you do with them?
• How to set up and calibrate on a variety of subjects on different eye-trackers?
• Glasses, lenses, mascara, and drooping eye-lids – what to do?
• How to work with stimulus programs, and synchronize them with eye-tracking recording?
• How to deal with the consent forms and ethical issues?
• Short introduction to experimental design: Potentials and pitfalls.
• Visualisation of data vs number crunching.
• Fast data analysis of multi-user experiments.
• Fixation durations, saccadic amplitudes, transition diagrams, group similarity measures, and all the other measures – what do they tell us? What are the pitfalls?

Teaching methods
Lectures on selected topics (8h)
Hands-on work in our lab on prespecified experiments: Receiving and recording on a subject (9h). Handson initial data analysis (3h).

Eye-tracking systems available for this training
2*SMI HED 50 Hz with Polhemus Head-tracking
3*SMI HiSpeed 240/1250 Hz
SMI RED-X remote 50 Hz
2*SMI HED-mobile 50/200 Hz

Attendance fee
€750 incl course material, diploma and lunches if you register before June 5th.
We will only run the course if we get enough participants. Register online.

The course content is equivalent to 1 ECTS credit at Master's level or above, although we cannot currently provide official registration at Lund University for this credit.

SWAET 2008 videos online

The recorded videos from the presentations at the SWAET2008 conference in Lund a couple of weeks ago is now online.

• Javier San Agustin Lopez, Julio C. Mateo and John Paulin Hansen
  Hands-free selection by Combining Gaze and EMG

• Henrik H. T. Skovsgaard, and John Paulin Hansen
  Zoom selection by gaze

• Yvonne Kammerer, Katharina Scheiter, Wolfgang Beinhauer
  How to improve gaze-based menu selection by menu design and multimodal input

Check out the complete list of presentations at the Lund University Humanities lab website.

Modify you webcam to capture infrared light

Found this a while back, might be useful if building a home-brew webcam eye tracker..

• Check out this excellent HowTo guide.

The most common method of eye tracking today is to use IR light to create corneal reflections. To do this you need a pair of IR LED´s (perhaps a couple of these) and a camera that is capable of receiving that spectrum of light. More or less all cameras can do this but manufacturers usually place a filter in front of the sensor so that it only picks up the perceivable spectrum of light. By removing this and replacing it with a filter that only passes IR you get a camera suitable for corneal reflection based eye tracking.

Next step would probably be to locate the head and the eyes in the video sequence. A good starting point for this is the OpenCV library. Martin Wimmer has created the EyeFinder system which locates head and eyes in real-time. Upon locating the eyes an area is defined as a region of interest which is passed on to image processing algorithms. These will typically locates the pupil by ellipse fitting and the glints/reflections that the IR lights create and somewhere along those lines it becomes "easier said than done" =) (I have yet to try building my own =)

IUI Gaze interaction tutorial (2005)

Back in 2005 Kari-Jouko Räihä, Aulikki Hyrskykari, Päivi Majaranta at the Unit for Human-Computer Interaction of the University of Tampere held a tutorial session on gaze based interaction at the Intelligent User Interfaces (IUI) conference. Luckily they posted the info online with a nice reference list. A good starting point for gaze interaction, especially in combination with the COGAIN bibliography. I will follow this move and publish a list of all the papers I´ve been reading while writing my thesis (which is soon to be finalized btw)

Inspiration: Gaze-Based Interaction for Semi-Automatic Photo Cropping

From the Visualization Lab, a part of the Computer Science Division at UC Berkley, comes an application that records gaze position while viewing images and then uses the data to perform cropping in a more intelligent way.

Abstract
"We present an interactive method for cropping photographs given minimal information about the location of important content, provided by eye tracking. Cropping is formulated in a general optimization framework that facilitates adding new composition rules, as well as adapting the system to particular applications. Our system uses fixation data to identify important content and compute the best crop for any given aspect ratio or size, enabling applications such as automatic snapshot recomposition, adaptive documents, and thumbnailing. We validate our approach with studies in which users compare our crops to ones produced by hand and by a completely automatic approach. Experiments show that viewers prefer our gaze-based crops to uncropped images and fully automatic crops."

Original well-composed images (left), adapted to two different aspect ratios using our gaze-based approach. An ADL document (right) using our crops. If eye movements are collected passively during document construction, our approach allows adaptation of images to arbitrary aspect ratios with no explicit user effort.

Associated paper:

• Gaze-Based Interaction for Semi-Automatic Photo Cropping. Anthony Santella, Maneesh Agrawala, Doug DeCarlo, David Salesin, Michael Cohen. ACM Human Factors in Computing Systems (CHI), 2006. pp. 771-780. PDF (2.2M)

Also check out

• A. Santella, and D. DeCarlo, "Visual Interest and NPR: an Evaluation and Manifesto". In Proceedings of the Third International Symposium on Non-Photorealistic Animation and Rendering (NPAR) 2004, pp 71-78, 2004.

• A. Santella, and D. DeCarlo, "Robust Clustering of Eye Movement Recordings for Quantification of Visual Interest". In Proceedings of the Third Eye Tracking Research and Applications Symposium (ETRA) 2004, pp 27-34, 2004.

SR Research: EyeLink Remote

"The EyeLink Remote system is designed for areas of eye tracking research where a head rest or head mount is not desirable but high accuracy and resolution are still important.

Head distance is accurately measured at 500 Hz using a small target sticker placed on the participants’ forehead. The use of a distance measure that is independent of the eye allows for head position to be tracked even during blinks, providing an extremely fast 2 msec blink recovery delay. The fast head position sampling rate also allows for head tracking during very quick head movements. (website)

Click image for a video demonstration

Rather impressive 500 Hz sampling rate which enables saccade detection algorithms and fast head movements. The recovery time from blinks is specified at only 2 milliseconds. However, the target sticker on the forehead is a workaround from actual face tracking. OK in the lab but not feasible in everyday, real world gaze interaction scenarios. (besides that my guess is that this tracker does not come cheap due to the high speed cameras used)

Gaze beats mouse: a case study.

Michael Dorr (publications) at the Institue of Neuro and Bioinformatics of University of Lübeck modified the open source version of LBreakout to be driven by gaze input using an SMI eye tracker. This was then used in a multiplayer setup where gaze/eye tracking took on the mouse. Source code and more information can be found here.

Click on images for video demonstration.

Related paper:

• Michael Dorr, Martin Böhme, Thomas Martinetz, and Erhardt Barth. Gaze beats mouse: a case study. In The 3rd Conference on Communication by Gaze Interaction - COGAIN 2007, Leicester, UK, pages 16-19, 2007. [ bib .pdf ]

Inspiration: Gaze controlled web browser

Craig Hennesy a Ph.D candidate at the Electrical and Computer Engineering dept. at University of British Columbia have done some work on gaze interaction. This includes the use of gaze position to scroll documents as the reader approaches the bottom of the window. This is used in other applications as well (really useful feature).

I´m currently working on my own implementation which will provide this functionality for a wide range of documents (web, pdf, word etc.) in combination with some new approaches on navigating the web using gaze alone.

"This video illustrates the use of eye-gaze tracking integrated with web browsing. The goal of this application is to reduce the use of the mouse when reading by removing the need to scroll up or down with the mouse. The simple scrolling application allows you to scroll down by looking below the article, scroll up by looking above the article, and go Back and Forward in the browser by looking to the left and right respectively.

In this demo the eye is tied to the mouse cursor so you can see where the user is looking, in the real application the motion of the eye stays behind the scenes and the mouse functions as a normal computer mouse."

Gaze interaction hits mainstream news

The New Scientist technology section posted an article on the Stephen Vickers work at De Montford University for the eye controlled version of World of Warcraft which I wrote about two months ago (see post)

Update: The New Scientist post caused rather extensive discussions on Slashdot, with more than
140 entries.

Great to see mainstream interest of gaze driven interaction. Gaming is truly one area where there is a huge potential, but it also depends on more accessible eye trackers. There is a movement for open source based eye tracking but the robustness for everyday usage is still remains at large. The system Stephen Vickers have developed is using the Tobii X120 eye tracker which is clearly out of range for all but the small group of users whom are granted financial support for their much needed assistive technology.

Have faith
In general, all new technology initially comes at a high cost due to intensive research and development but over time becomes accessible for the larger population. As an example, not many could imagine that satellite GPS navigation would be commonplace and really cheap a decade or two ago. Today mass-collaboration on the net is really happening making the rate of technology development exponential. Make sure to watch Google Techtalk Don Tapscott on Wikinomics.

Interface evaluation procedure

The first evaluation of the prototype is now completed. The procedure was designed to test the individual components as well as the whole prototype (for playing music etc.). Raw data on selection times, error rates etc. was collected by custom development on the Microsoft .NET platform. Additional forms were displayed on-screen in between the steps of the procedure combined with standardized form based questionnaires to gather a rich set of data.

A more subjective approach was taken during the usage of the prototype as a whole to capture the aspects which could not be confined by automatic data collection or forms. Simply by observing participants and asking simple questions in normal spoken language. While perhaps being less scientifically valid this type of information is very valuable for understanding how the users think and react. Information that is crucial for improving the design for the next iteration. There is sometimes a difference the results gained by verbal utterance, questionnaires and measurable performance. For example, interfaces can be very efficient and fast but at the same time extremely demanding and stressful. Just measuring one performance factor would not tell the whole story.

This is why I chosen to use several methods to combine raw performance data, form based questionnaires and unconstrained verbal interviewing. Hoping that it can provide multiple aspects with potential for gathering a rich source of data.

For the evaluation I used a basic on-screen demographic form gathering age, sex, computer experience, conditions of vision etc. In-between the evaluation of the individual components I used the NASA Task Load Index as a quick reflection form and at the end of the session I handed out both a IBM Computer Usability Satisfaction Questionnaire and a Q.U.I.S Generic User Interface Questionnaire. The only modification I performed was to remove a few questions that would not apply to my prototype (why ask about help pages when the prototype contains none)

I´ve found the 230 Tips and Tricks for Better Usability Testing guide to be really useful and should be read by anyone conduction HCI evaluation.

Questionnaires used in my evaluation, in PDF format:

• Basic demographics (implemented on-screen, multiple pages)
• NASA Task Load Index (on-screen using sliders)
• IBM Computer Usability Satisfaction Questionnaire
• QUIS User Interface Questionnaire

SWAET Conference

The first day of the SWAET 2008 conference at Lund University was filled with interesting presentations and inspiring conversations. Finally I had a chance to meet some of the researchers that I´ve know mainly by their publications and last names.

From the schedule of day one three talks stand out in the field of gaze interaction. These are recorded on video with the intention of future online distribution. For now enjoy these papers.

• Javier San Agustin Lopez, Julio C. Mateo and John Paulin Hanse. Hands-free selection by Combining Gaze and EMG

• Henrik H. T. Skovsgaard, and John Paulin Hansen. Zoom selection by gaze

• Yvonne Kammerer, Katharina Scheiter, Wolfgang Beinhauer. The impact of menu design and multimodal input on gaze-based menu selection

The conference was attended by three manufacturers, SMI, SmartEye and Tobii which all had systems on display.

The SMI system demonstrated was brought up from the dark dungeons of the lab to host the prototype I have been working on for the last couple of months. In general, it was well received and served its purpose of a eye catching demonstration of my "next-gen" gaze interface. To sum up it was great to get out there to gather feedback confirming that I´m on the right track. (doubts sometimes rise when working solo) The day ended with a a lovely dinner at the university´s finest dining hall.

Big thank you goes out to Kenneth Holmqvist, Jana Holsanova, Philip Diderichsen, Nils Holmberg, Richard Andersson and Janna Spanne for hosting this event.

Open invitation to participate in the evaluation

The invitation to participate in the evaluation of the prototype is out. If you have the possibility to participate I would be most thankful for your time. The entire test takes roughly 30 minutes.

The invitation can be downloaded as pdf.

If you wish to participate send me a message. Thank you.

More information on the procedure and structure of the evaluation as well as gained experience will be posted once the testing is completed. The final results will be published in my master thesis by the end of May. Stay tuned.

Alea Technologies IntelliGaze demo

A demonstration of the IG30Pro remote eye tracker used to play a first person shooter game. Notice the algorithms used to stabilize the pointer and filtering out the jitter usually associated with eye trackers.

Alea Technologies

The German firm Alea Technologies offers a solution called IntelliGaze consisting of a remote based eye tracking system as well as a software suite. The system is designed to be a flexible solution in terms of both hardware and software. The eye tracker is stand-alone and can be used to create a customizable setup when it is combined with a various displays. Additionally, they provide API´s for application developments.

A clear usage for their technology is users with disabilities such as ALS. The software contains a "desktop" system which acts as a launcher for other applications (Windows natives, Grid, Cogain). In general, they seem to target Tobii Technologies who have been very successful with their MyTobii application running on the P10 eye tracker. The game is on.

Quote:

"The ease of setup and intuitive operation bring a completely new degree of freedom to patients who had to rely on older technologies like manual scanning or cumbersome pointing devices. By using the latest camera technology and very sophisticated image processing and calibration methods, the IG-30 system is far superior to alternative gaze input systems at a similar price point. The modular structure of the system allows the patients with a degenerative disease the continued use of their touch-screen or scanning software package. Using existing computers and monitors, the IG-30 system can also be easily integrated into an existing PC setup. The economic pricing of the IntelliGaze^TM systems opens this high-tech solution to a much wider range of users."

Tracking technology	Hybrid infrared video eye- & head-tracking Binocular & monocular tracking
Working Volume centered at 600 mm distance	300 x 200 x 200 mm3 [WxHxD]
Accuracy , static	0.5°, typical
Accuracy , over full working volume	1°, typical
Sampling Rate	50 Hz
Max. head-movement velocity	15 cm/s
Recovery-time after tracking loss (head was too fast or moved out of range)	40 ms
System Dimensions	ca. 300 x 45 x 80 mm3 [WxHxD]
Mounting Options	on monitor via VESA-adapter on Tablet-PC via customized interfaces
System Weight	ca. 1,2 kg.

Gaze Interaction Demo (Powerwall@Konstanz Uni.)

During the last few years quite a few wall sized displays have been used for novel interaction methods. Not seldomly these have been used with multi-touch, such as the Jeff Han´s FTIR technology. This is the first demonstration I have seen where eye tracking is used for a similar purpose. A German Ph.D candidate, Jo Bieg, is working on this out of the HCI department at the University of Konstanz. The Powerwall is 5.20 x 2.15M and has a resolution of 4640 x 1920.

The demonstration can be view at a better quality (10Mb)

Also make sure to check out the 360 deg. Globorama display demonstration. It does not use eye tracking for interaction but a laser pointer. Nevertheless, really cool immersive experience, especially the Google Earth zoom in to 360 panoramas.

Inspiration: EyeMusic & EyeFollower by Interactive Minds

From the German based lab Interactive Minds provides the EyeMusic interface which lets you play songs. Not much information is available except the screenshot below.

http://www.interactive-minds.de/en/software/eye_music.htm

However, their eye tracker "EyeFollower" seems more impressive. 120Hz sampling and 0.45 deg. accuracy. Furthermore, it allows for larger and faster head movements than most other remote based systems. This is really important for making the tracking flexible, which is what you want with a remote based system in the first place. People move around, we all change posture over time. Imagine sitting at a desk then turning to the side taking notes, talking on the phone. When you return to the computer it should instantaneously continue to track your gaze without any noticeable delay (or having to re-calibrate) These are not easy tasks to solve but are necessary for the advancement of eye tracking/gaze interaction. Interactive Minds provides a demonstration video of their EyeFollower. Looks great.

Inspiration: Looking my Way through the Menu: The Impact of Menu Design and Multimodal Input on Gaze-based Menu Selection

As discussed earlier in my blog the differences between gaze vs mouse based interaction calls for interfaces that are especially designed for the purpose. A group of German researchers present a novel approach based on a radial menu layout. The work has been carried out by Yvonne Kammerer and Katharina Scheiter both at the Knowledge Media Research Center, University of Tuebingen in conjunction with Wolfgang Beinhauer at the Fraunhofer Institute for Industrial Engineering, Stuttgart.

My own prototype contains U.I elements that are based on the same on the style of interaction. However, this work goes further towards a multi-level menu system while my component is aiming more for a quick one-level saccade selection. The advantages of the radial layout in a gaze based menu component is discussed in this paper. Interesting concept, looking forward to the presentation at the SWAET2008 conference.

Abstract
"In this paper a study is reported, which investigates the effectiveness of two approaches to improving gaze-based interaction for realistic and complex menu selection tasks. The first approach focuses on identifying menu designs for hierarchical menus that are particularly suitable for gaze-based interaction, whereas the second approach is based on the idea of combining gaze-based interaction with speech as a second input modality. In an experiment with 40 participants the impact of menu design, input device, and navigation complexity on accuracy and completion time in a menu selection task as well as on user satisfaction were investigated. The results concerning both objective task performance and subjective ratings confirmed our expectations in that a semi-circle menu was better suited for gaze-based menu selection than either a linear or a full-circle menu. Contrary to our expectations, an input device solely based on eye gazes turned out to be superior to the combined gaze- and speech-based device. Moreover, the drawbacks of a less suitable menu design (i.e., of a linear menu or a full-circle menu) as well as of the multimodal input device particularly obstructed performance in the case of more complex navigational tasks." Download paper as pdf.

Eye Tracking, Algorithms and Mathematical Modelling (Leimberg & Vester-Christensen, 2005)

Extensive and technical publication on eye tracking algorithms by D. Leimberg and M. Vester-Christensen at the Department of Informatics and Mathematical Modeling at the Technical University of Denmark. The thesis contains a range of approaches and discussion of the implementation of these. Rich with illustrations and examples of the result of various methods.

Abstract
This thesis presents a complete system for eye tracking avoiding restrictions on head movements. A learning-based deformable model - Active Appearance Model (AAM) - is utilized for detection and tracking of the face. Several methods are proposed, described and tested for eye tracking, leading to determination of gaze. The {AAM} is used for a segmentation of the eye region, as well as providing an estimate of the pose of the head.

Among several, we propose a deformable template based eye tracker, combining high speed and accuracy, independently of the resolution. We compare with a state of the art active contour approach, showing that our method is more accurate. We conclude, that eye tracking using standard consumer cameras is feasible providing an accuracy within the measurable range." Download paper as pdf


Following up are two papers (also found at the end of the thesis)

• M. Vester-Christensen, D. Leimberg, B. K. Ersbøll, L. K. Hansen, Deformable Models for Eye Tracking, Den 14. Danske Konference i Mønstergenkendelse og Billedanalyse, 2005 [full] [bibtex] [pdf]
  
  
• D. Leimberg, M. Vester-Christensen, B. K. Ersbøll, L. K. Hansen, Heuristics for speeding up gaze estimation, Proc. Svenska Symposium i Bildanalys, SSBA 2005, Malmø, Sweden, SSBA, 2005 [full] [bibtex] [pdf]

Novel Eye Gaze Tracking Techniques Under Natural Head Movement (Zhiwei&Qiang, 2007)

Intelligent Systems Lab at Rensselear Polytechnic Institute, US. Using a stereo setup Zhu Zhiwei and Ji Qiang are able to produce a head movement tolerant eye tracker that calibrates in less than five seconds and provides around 1.6 deg. accuracy with 25 frames per second.

Accuracy illustration. Targets in blue. Tracked fixations in red.

Zhiwei Zhu and Qiang Ji, Novel Eye Gaze Tracking Techniques Under Natural Head Movement, IEEE Transactions on Biomedical Engineering, 54(12), p2246-60,2007. download the paper.

A neural network based real-time gaze tracker

Continuing on the neural network based eye trackers from the earlier post, this work by Nischal Piratla back in 2001 while his time at the Colorado State University. It uses a low resolution CCD camera in combination with a 3 layer back propagation neural network. Although low frame rates, 5 per second, it´s not too bad for running on a Pentium II 266MHz. However, the horizontal bar on the forehead would not do today =)

Abstract
"A real-time gaze-tracking system that estimates the user's eye gaze and computes the window of focused view on a computer monitor has been developed. This artificial neural network based system can be trained and customized for an individual. Unlike existing systems in which skin color features and/or other mountable equipment are needed, this system is based on a simple non-intrusive camera mounted on the monitor. Gaze point is accurately estimated within a 1 in. on a 19-in. monitor with a CCD camera having a 640 × 480 image resolution. The system performance is independent of user's forward and backward as well as upward and downward movements. The gaze-tracking system implementation and the factors affecting its performance are discussed and analyzed in detail. The features and implementation methods that make this system real-time are also explained."

Download paper as pdf

Neural Network Gaze Tracking using Web Camera

Using a simple low resolution web camera without IR illumination David Bäck uses a neural network approach for eye tracking. Low accuracy (2-4 deg.) mainly due to the poor image quality from the web camera and no IR illumination. However, a nice introduction to the basics of technical implementation.

Abstract
"Gaze tracking means to detect and follow the direction in which a person looks. This can be used in for instance human-computer interaction. Most existing systems illuminate the eye with IR-light, possibly damaging the eye. The motivation of this thesis is to develop a truly non-intrusive gaze tracking system, using only a digital camera, e.g. a web camera.

The approach is to detect and track different facial features, using varying image analysis techniques. These features will serve as inputs to a neural net, which will be trained with a set of predetermined gaze tracking series. The output is coordinates on the screen.

The evaluation is done with a measure of accuracy and the result is an average angular deviation of two to four degrees, depending on the quality of the image sequence. To get better and more robust results, a higher image quality from the digital camera is needed."

Download paper as pdf

Mix08 Talk: New forms of interaction

Mainly on interfaces based on multitouch and speech, however many issues are relevant for gaze interaction as well.

Title: Touch Me: Where Are Interfaces Going?
Speaker(s): Chris Bernard, Dale Herigstad, Daniel Makoski, Dave Wolfe, Doug Cook, Yoshihiro Saito
Description: The keyboard and mouse are aging input devices. For the future of computing and UX, where are interfaces going? Are these enough? Is touch-screen Surface/iPhone/iPod Touch just a gimmick? Where should Man Machine Interface (MMI) go?

WMV link: http://msstudios.vo.llnwd.net/o21/mix08/08_WMVs/PNL10.wmv
MP4 link: http://msstudios.vo.llnwd.net/o21/mix08/08_MP4s/PNL10.mp4

WPF and Windows Media Player COM control

The WMPLib offers direct control to Windows Media Player to your C# applications. This provides a rich set of events and controls that is really useful if you would like to create a custom media player.

The code I wrote scans the computer (My Music) and builds up a library of artists, albums and songs using the directory structure. Any images found are used to create folder icons. I later found out that it is possible to access the Windows Media Player media collection library to access this information so next time I´ll probably use that) However, this requires the albums to be imported/added to the Windows Media Player.

There is a few neat trick available when using the WMP. One is that it sends events when ever the player state changes (see list of events). This enabled me to initiate a progress bar that displays the current position in the song (code 3) as well as skipping to the next song when a the current playing song was completed (code 8).

To add WMP in you code

Using WMPLib;

WindowsMediaPlayer _wmp;

public MyApp()
{
_wmp = new WindowsMediaPlayer();
// The event below tells WMP to call your own MediaProcess code
// when the WMP player changes state
_wmp.PlayStateChange += new _WMPOCXEvents_PlayStateChangeEventHandler(MediaProcessChange);

}
private void MediaProcessChange(int newState)
{
//The integer "newState" contains the state of the media player, ranging from 1 to 11
// see http://msdn2.microsoft.com/en-us/library/bb249361(VS.85).aspx for mor infomation

switch(newState)
{
// case 1: // Stopped
// case 2 : // Paused

case 3 : // Song starts playing

progress.Maximum = _wmp.currentMedia.duration;
progress.SetValue(ProgressBar.ValueProperty, _wmp.controls.currentPosition);

BackgroundWorker _worker = new BackgroundWorker();
_worker.WorkerReportsProgress = false;
_worker.WorkerSupportsCancellation = true;

_worker.DoWork += delegate(object s, DoWorkEventArgs args)
{
while (_wmp.controls.currentPosition <>
{
// Dispatcher to update the U.I progressbar every two seconds
Dispatcher.BeginInvoke(DispatcherPriority.Background, (SendOrPostCallback)delegate { progress.SetValue(ProgressBar.ValueProperty, _wmp.controls.currentPosition); }, null);
Thread.Sleep(2000);
}
}; // End delegate
_worker.RunWorkerAsync();
break;

// case 4 : // ScanFordward
// case 5 : // ScanBackwards
// case 6 : // Buffering
// case 7 : // Waiting

case 8: // Song ended, play next

PlayNextSong(null,null);
break;

// case 9 : // Transitioning
// case 10 : // Ready
// case 11 : // Reconnecting
}
}

The "progress" object is a XAML added obj of the type ProgressBar. These handy U.I elements can be set to the maximum value (instead of 1 to 100) The maximum value I set to the entire lenght of the song. This way the current position scale to the bar.

There is a good tutorial for making a media player at Pauls Blog. Additionally, another custom media player by Sacha Barber. The MSDN network has extensive API info as always (but perhaps in a really boring format)