Eye Tracking, Algorithms and Mathematical Modelling (Leimberg & Vester-Christensen, 2005)

Extensive and technical publication on eye tracking algorithms by D. Leimberg and M. Vester-Christensen at the Department of Informatics and Mathematical Modeling at the Technical University of Denmark. The thesis contains a range of approaches and discussion of the implementation of these. Rich with illustrations and examples of the result of various methods.

Abstract
This thesis presents a complete system for eye tracking avoiding restrictions on head movements. A learning-based deformable model - Active Appearance Model (AAM) - is utilized for detection and tracking of the face. Several methods are proposed, described and tested for eye tracking, leading to determination of gaze. The {AAM} is used for a segmentation of the eye region, as well as providing an estimate of the pose of the head.

Among several, we propose a deformable template based eye tracker, combining high speed and accuracy, independently of the resolution. We compare with a state of the art active contour approach, showing that our method is more accurate. We conclude, that eye tracking using standard consumer cameras is feasible providing an accuracy within the measurable range." Download paper as pdf


Following up are two papers (also found at the end of the thesis)

• M. Vester-Christensen, D. Leimberg, B. K. Ersbøll, L. K. Hansen, Deformable Models for Eye Tracking, Den 14. Danske Konference i Mønstergenkendelse og Billedanalyse, 2005 [full] [bibtex] [pdf]
  
  
• D. Leimberg, M. Vester-Christensen, B. K. Ersbøll, L. K. Hansen, Heuristics for speeding up gaze estimation, Proc. Svenska Symposium i Bildanalys, SSBA 2005, Malmø, Sweden, SSBA, 2005 [full] [bibtex] [pdf]

Novel Eye Gaze Tracking Techniques Under Natural Head Movement (Zhiwei&Qiang, 2007)

Intelligent Systems Lab at Rensselear Polytechnic Institute, US. Using a stereo setup Zhu Zhiwei and Ji Qiang are able to produce a head movement tolerant eye tracker that calibrates in less than five seconds and provides around 1.6 deg. accuracy with 25 frames per second.

Accuracy illustration. Targets in blue. Tracked fixations in red.

Zhiwei Zhu and Qiang Ji, Novel Eye Gaze Tracking Techniques Under Natural Head Movement, IEEE Transactions on Biomedical Engineering, 54(12), p2246-60,2007. download the paper.

A neural network based real-time gaze tracker

Continuing on the neural network based eye trackers from the earlier post, this work by Nischal Piratla back in 2001 while his time at the Colorado State University. It uses a low resolution CCD camera in combination with a 3 layer back propagation neural network. Although low frame rates, 5 per second, it´s not too bad for running on a Pentium II 266MHz. However, the horizontal bar on the forehead would not do today =)

Abstract
"A real-time gaze-tracking system that estimates the user's eye gaze and computes the window of focused view on a computer monitor has been developed. This artificial neural network based system can be trained and customized for an individual. Unlike existing systems in which skin color features and/or other mountable equipment are needed, this system is based on a simple non-intrusive camera mounted on the monitor. Gaze point is accurately estimated within a 1 in. on a 19-in. monitor with a CCD camera having a 640 × 480 image resolution. The system performance is independent of user's forward and backward as well as upward and downward movements. The gaze-tracking system implementation and the factors affecting its performance are discussed and analyzed in detail. The features and implementation methods that make this system real-time are also explained."

Download paper as pdf

Neural Network Gaze Tracking using Web Camera

Using a simple low resolution web camera without IR illumination David Bäck uses a neural network approach for eye tracking. Low accuracy (2-4 deg.) mainly due to the poor image quality from the web camera and no IR illumination. However, a nice introduction to the basics of technical implementation.

Abstract
"Gaze tracking means to detect and follow the direction in which a person looks. This can be used in for instance human-computer interaction. Most existing systems illuminate the eye with IR-light, possibly damaging the eye. The motivation of this thesis is to develop a truly non-intrusive gaze tracking system, using only a digital camera, e.g. a web camera.

The approach is to detect and track different facial features, using varying image analysis techniques. These features will serve as inputs to a neural net, which will be trained with a set of predetermined gaze tracking series. The output is coordinates on the screen.

The evaluation is done with a measure of accuracy and the result is an average angular deviation of two to four degrees, depending on the quality of the image sequence. To get better and more robust results, a higher image quality from the digital camera is needed."

Download paper as pdf

Mix08 Talk: New forms of interaction

Mainly on interfaces based on multitouch and speech, however many issues are relevant for gaze interaction as well.

Title: Touch Me: Where Are Interfaces Going?
Speaker(s): Chris Bernard, Dale Herigstad, Daniel Makoski, Dave Wolfe, Doug Cook, Yoshihiro Saito
Description: The keyboard and mouse are aging input devices. For the future of computing and UX, where are interfaces going? Are these enough? Is touch-screen Surface/iPhone/iPod Touch just a gimmick? Where should Man Machine Interface (MMI) go?

WMV link: http://msstudios.vo.llnwd.net/o21/mix08/08_WMVs/PNL10.wmv
MP4 link: http://msstudios.vo.llnwd.net/o21/mix08/08_MP4s/PNL10.mp4

WPF and Windows Media Player COM control

The WMPLib offers direct control to Windows Media Player to your C# applications. This provides a rich set of events and controls that is really useful if you would like to create a custom media player.

The code I wrote scans the computer (My Music) and builds up a library of artists, albums and songs using the directory structure. Any images found are used to create folder icons. I later found out that it is possible to access the Windows Media Player media collection library to access this information so next time I´ll probably use that) However, this requires the albums to be imported/added to the Windows Media Player.

There is a few neat trick available when using the WMP. One is that it sends events when ever the player state changes (see list of events). This enabled me to initiate a progress bar that displays the current position in the song (code 3) as well as skipping to the next song when a the current playing song was completed (code 8).

To add WMP in you code

Using WMPLib;

WindowsMediaPlayer _wmp;

public MyApp()
{
_wmp = new WindowsMediaPlayer();
// The event below tells WMP to call your own MediaProcess code
// when the WMP player changes state
_wmp.PlayStateChange += new _WMPOCXEvents_PlayStateChangeEventHandler(MediaProcessChange);

}
private void MediaProcessChange(int newState)
{
//The integer "newState" contains the state of the media player, ranging from 1 to 11
// see http://msdn2.microsoft.com/en-us/library/bb249361(VS.85).aspx for mor infomation

switch(newState)
{
// case 1: // Stopped
// case 2 : // Paused

case 3 : // Song starts playing

progress.Maximum = _wmp.currentMedia.duration;
progress.SetValue(ProgressBar.ValueProperty, _wmp.controls.currentPosition);

BackgroundWorker _worker = new BackgroundWorker();
_worker.WorkerReportsProgress = false;
_worker.WorkerSupportsCancellation = true;

_worker.DoWork += delegate(object s, DoWorkEventArgs args)
{
while (_wmp.controls.currentPosition <>
{
// Dispatcher to update the U.I progressbar every two seconds
Dispatcher.BeginInvoke(DispatcherPriority.Background, (SendOrPostCallback)delegate { progress.SetValue(ProgressBar.ValueProperty, _wmp.controls.currentPosition); }, null);
Thread.Sleep(2000);
}
}; // End delegate
_worker.RunWorkerAsync();
break;

// case 4 : // ScanFordward
// case 5 : // ScanBackwards
// case 6 : // Buffering
// case 7 : // Waiting

case 8: // Song ended, play next

PlayNextSong(null,null);
break;

// case 9 : // Transitioning
// case 10 : // Ready
// case 11 : // Reconnecting
}
}

The "progress" object is a XAML added obj of the type ProgressBar. These handy U.I elements can be set to the maximum value (instead of 1 to 100) The maximum value I set to the entire lenght of the song. This way the current position scale to the bar.

There is a good tutorial for making a media player at Pauls Blog. Additionally, another custom media player by Sacha Barber. The MSDN network has extensive API info as always (but perhaps in a really boring format)

Gaze Media Player

The component I´ve been working on is now capable of the basics. Great feeling just looking a song titles and then skipping through the playlist by gaze =) There is room for improvements, would be nice have a component like a slider where one could go to a specific part of the song. (don´t know how many times I´ve been playing guitar to a song while learning it and going back and forth between mouse and guitar. Let´s see what a weekend could do =)

Screenshot of the music player component, layout not finalized.. Updated version has a song progression bar and volume controls.

The play-button houses another ellipse shaped menu with the regular controls for (next, previous, play, stop)

RApid GAze-Based Interaction Techniques (RAGABITS)

Stephen Vickers at the Computer Human Interaction Research Group at the De Montfort University, Uk have developed interaction techniques that allows gaze based control of several popular online virtual worlds such as World of Warcraft or Second Life. This research will be presented at ETRA 2008, US under the title RAGABITS (RApid GAze-Based Interaction Techniques) and is espcially intented for users with severe motor impairments.

Selection method seems stable. None of the usual jitter can be seen. Nice!



Quote from http://www.ioct.dmu.ac.uk/projects/eyegaze.html

"Online virtual worlds and games (MMORPG's) have much to offer users with severe motor disabilities. It gives this user group the opportunity as entirely able-bodied to others in the virtual world. if they so wish. The extent to which a user has to reveal their disability becomes a privacy issue. Many of the avatars in Second Life appear as stylized versions of the users that control them and that stylization is the choice of the user. This choice is equally appropriate for disabled users. While the appearance of the user's avatar may not reveal the disability of the person that controls it, the behavior and speed or interaction in the world may do.

Many users with severe motor impairments may not be able to operate a keyboard or hand mouse and may also struggle with speech and head movement. Eye gaze is one method of interaction that has been used successfully in enabling access to desktop environments. However, simply emulating a mouse using eye gaze is not sufficient for interaction in online virtual worlds and the users privacy can be exposed unless efficient gaze-based interaction techniques, appropriate to activities in on-line worlds and games can be provided.

This genre of gaming (MMORPG's) is constantly evolving and regardless of the aim of the game they all involve common tasks such as, avatar creation, social interaction (chatting, IM), interaction with in world objects (pick up, open, shoot etc), navigating and walking around the environment. Our research involves analyzing these common tasks so that suitable gaze based interaction techniques to support them can be used in place of a mouse and keyboard. These will have different performance/effort trade-offs, and will include extended mouse/joystick emulation, gaze gestures, toolglasses and gaze-aware in-world objects. These techniques need to be integrated into a coherent and efficient user interface suited to the needs of an individual user with a particular disability. The research aims to model tasks inherent in using these worlds so that predictions can be made about the most appropriate gaze based interaction techniques to use. When these have been identified, they can be assembled into a front end or user interface. One possible outcome could be a software device for automatic configuration of a gaze-control interface for new games, which could use knowledge of a specific user's disability and the eye tracking equipment that they have."

Last week of prototyping

Being back from a short easter holiday the final week of developing the prototype is here. The functionality of the interface is OK but there is major tweaking and bug testing to be performed. The deadline for any form of new features is Monday 31st of March. I intend to use all of April for setting up the evaluation experiments and procedures. As always there is so much more that I would like to incorporate, every day brings new ideas.

The final version will include:

• A configurable dwell button component. This enables drag and drop dwell buttons into Windows projects with individual configuration on dwell-time and icons etc.
  
  
• A novel gaze based menu system that utilized saccade selection (two step dwell) This highly configurable interface component displays itself when activated. It aims at solving the midas touch problem while utilizing the screen real estate in a better way. The two steps in the activation process can be set to specific activation-speeds (dwells) More on this later..
  
  
• A gaze-based memory game using a 36 card layout. By perfoming a dwell the cards "turn" over and shows its symbol (flag). The user the selects another card. If matching then remove them. If different, turn them back over. Got some nice graphical effects.
  
  
• A gaze based picture viewer that zooms into the photos in gaze of the user. More on this later.
  
  
• A gaze based media/music player. This component will scan the computer for artists, albums and songs. These items are then accessible by a gaze driven interface where the user can create play-lists and perform the usual functions (volume+-, pause, stop, next, previous etc.)
  
  
• Perhaps just one little surprise more..
  

Eight weeks so far. Curiosity, dedication and plain ol´ hard work, nothing else to it.

Inspiration: Real-Time Facial and Eye Gaze Tracking System (Park&Kim, 2005)

While my main concern is gaze interaction it´s important to understand the underlying technology which makes it all possible (and the limitation it poses) Much work has gone into solving issues surrounding image processing that makes the "new" remote eye tracking systems we see today possible. One solution to battle the low resolution of cameras is presented in the paper below (using dual cameras) Additionally, the issues of reflections from glasses solved using dual IR-leds. Interesting paper, technical information on Kahlman algorithms etc. used to achieve a 15 frames / second eye tracker on an old Pentium IV 2.0GHz.

Abstract
The goal of gaze detection is to locate the position (on a monitor) where a user is looking. Previous researches use one wide view camera, which can capture the user's entire face. However, the image resolution is too low with such a camera and the fine movements of user's eye cannot be exactly detected. So, we propose the new gaze detection system with dual cameras (a wide and a narrow view camera). In order to locate the user's eye position accurately, the narrow-view camera has the functionalities of auto focusing/panning/tilting based on the detected 3D eye positions from the wide view camera. In addition, we use the IR-LED illuminators for wide and narrow view camera, which can ease the detecting of facial features, pupil and iris position. To overcome the problem of specular reflection on glasses by illuminator, we use dual IR-LED illuminators for wide and narrow view camera and detect the accurate eye position, which is not hidden by the specular reflection. Experimental results show that the gaze detection error between the computed positions and the real ones is about 2.89 cm of RMS error.


The paper is written by prof. Kang Ryoung Park who is heading the Computer Graphics & Vision department at the Sangmyung University, Korea (publications)

Co-author prof. Jaihie Kim is the head of the Computer Vision lab at the Yonsei University, Korea (publications)