Redirecting the Gaze X/Y to replace Mouse X/Y

To utilize the wide range of functionality that has been developed for mouse based interaction in our gaze interaction software we need to replace the mouse X/Y coordinates with the gaze X/Y.

This requires one to dig rather deep into the Windows system and DLL files. The function Move takes two integers and you guessed right, the X and Y position of the mouse pointer.
This method is to be called every time the tracker provides us with new gaze data. Subsequently the position of the pointer should be moved to the new position.

So first we modify the EyeTrackerServer class

   // The decoded string from tracker UDP stream
datareceived = System.Text.Encoding.ASCII.GetString(received);

// Create and instance of the object that redirect the gaze to the mouse
RedirectGazeToMouse gazeMouse = new RedirectGazeToMouse();

if (datareceived.Length > 0)
{
// Extract the X & Y coordinates from the UDP stream from the tracker
extractGazePosition(datareceived);

// Move the mousepointer according to the gaze X&Y coordinates
gazeMouse.Move(gazeData.GazePositionY, gazeData.GazePositionX);
}

It is very distracting to actually see the mouse pointer moving on the screen, updated 50 times per second, because you would fixate on it and it would move slightly (remember eye trackers are not as precise) so one would end up "chasing" it around. Let's hide the mouse pointer.
In the main application window (typically Windows1.xaml.cs or similar) this is done by placing the one line in the constructor:

public Window1()
{    
InitializeComponent();

// Hide the mouse cursor since it is replace by gaze coordinates
Cursor = Cursors.None;

// Initialize and start the Eye Tracker
myServer = new EyeTrackerServer();
myServer.start();
}

And here is the RedirectGazeToMouse class. (many thanks to Pinvoke.net)

using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
using System.Windows;

namespace GazeHero
{

public class RedirectGazeToMouse
{
[DllImport("user32.dll", EntryPoint = "SendInput", SetLastError = true)]

static extern uint SendInput(uint nInputs, INPUT[] pInputs, int cbSize);

[DllImport("user32.dll", EntryPoint = "GetMessageExtraInfo", SetLastError = true)]

static extern IntPtr GetMessageExtraInfo();

private enum InputType
{
INPUT_MOUSE = 0,
INPUT_KEYBOARD = 1,
INPUT_HARDWARE = 2
}

[Flags()]
private enum MOUSEEVENTF
{
MOVE =        0x0001,  // mouse move
LEFTDOWN =    0x0002,  // left button down
LEFTUP =      0x0004,  // left button up
RIGHTDOWN =   0x0008,  // right button down
RIGHTUP =     0x0010,  // right button up
MIDDLEDOWN =  0x0020,  // middle button down
MIDDLEUP =    0x0040,  // middle button up
XDOWN =       0x0080,  // x button down
XUP =         0x0100,  // x button down
WHEEL =       0x0800,  // wheel button rolled
VIRTUALDESK = 0x4000,  // map to entire virtual desktop
ABSOLUTE =    0x8000,  // absolute move
}

[Flags()]
private enum KEYEVENTF
{
EXTENDEDKEY = 0x0001,
KEYUP =       0x0002,
UNICODE =     0x0004,
SCANCODE =    0x0008,
}

[StructLayout(LayoutKind.Sequential)]
private struct MOUSEINPUT
{
public int dx;
public int dy;
public int mouseData;
public int dwFlags;
public int time;
public IntPtr dwExtraInfo;
}

[StructLayout(LayoutKind.Sequential)]
private struct KEYBDINPUT
{
    public short wVk;
    public short wScan;
    public int dwFlags;
    public int time;
    public IntPtr dwExtraInfo;
}

[StructLayout(LayoutKind.Sequential)]
private struct HARDWAREINPUT
{
    public int uMsg;
    public short wParamL;
    public short wParamH;
}

[StructLayout(LayoutKind.Explicit)]
private struct INPUT
{
    [FieldOffset(0)]
    public int type;
    [FieldOffset(4)]
    public MOUSEINPUT mi;
    [FieldOffset(4)]
    public KEYBDINPUT ki;
    [FieldOffset(4)]
    public HARDWAREINPUT hi;
}

This function moves the cursor to a specific point at the screen. X coordinate of the position as pixel Y coordinate of the position as pixel
Returns 0 if there was an error otherwise 1


public uint Move(int x, int y)
{
 double ScreenWidth = System.Windows.SystemParameters.PrimaryScreenWidth;
 double ScreenHeight = System.Windows.SystemParameters.PrimaryScreenHeight;

 INPUT input_move = new INPUT();
 input_move.mi.dx = (int)Math.Round(x * (65535 / ScreenWidth), 0);
 input_move.mi.dy = (int)Math.Round(y * (65535 / ScreenHeight), 0);

 input_move.mi.mouseData = 0;
 input_move.mi.dwFlags = (int)(MOUSEEVENTF.MOVE | MOUSEEVENTF.ABSOLUTE);
 INPUT[] input = {input_move};

 return SendInput(1, input, Marshal.SizeOf(input_move));
}


CURRENTLY NOT USED IN THE GAZE PROJECT BUT COULD BE FURTHER ON..
This function simulates a simple mouseclick at the current cursor position
All right if it is 2. All below indicates an error.

 public static uint Click()
 {
   INPUT input_down = new INPUT();
   input_down.mi.dx = 0;
   input_down.mi.dy = 0;
   input_down.mi.mouseData = 0;
   input_down.mi.dwFlags = (int)MOUSEEVENTF.LEFTDOWN;
   INPUT input_up = input_down;
   input_up.mi.dwFlags = (int)MOUSEEVENTF.LEFTUP;
   INPUT[] input = {input_down, input_up};

   return SendInput(2, input, Marshal.SizeOf(input_down));
 }

}

}