Background

This section provides you with some background knowledge about the internal functionality of wiigee and may help you to understand it's implementation and, consequently, some of it's behaviours. To some extent, it can be regarded as a stripped-down version of the paper Gesture Recognition with a Wii Controller and is not intended to serve as a programming guide. For this purpose, consult the basic tutorial as well as the JavaDoc.


What is a gesture?

In order to be able to train and recognize arbitrary gestures, we have to define what (from a technical point of view) a gesture actually is. The term is typically associated with a characteristic pattern of incoming signal data which, in the case of the wiimote, refers to the data the built-in accelerometer is constantly sending via Bluetooth. An accelerometer is a small device detecting the acceleration in all three dimensions an external force exerts on it. With these comments in mind, gesture describes an ordered, finite series of three-dimensional acceleration vectors with an explicit start and an explicit end. In wiigee, start and end of a gesture are by default triggered by pressing one of the wiimote's buttons although this behaviour can be altered. (See Recognition.)

reference_gestures
Gesture examples. Although these are all 2D, wiigee is capable of detecting 3D-gestures just as well.



System components

wiigee is built upon a classic recognition pipeline which was specifically optimized for the wiimote's hardware. The following image shows the pipeline and thus the way the vector data takes during processing.

systemcomponents
wiigee system components


The first component is a Filter, removing vectors which are a) below a given threshold of approx. 1.2g, g being the acceleration of gravity, and b) roughly equivalent to their predecessor, thus eliminating suspected duplicates. The second component is a Quantizer clustering the incoming vector data in order to generate a discrete code identifying each gesture. Here, a standard k-mean algorithm is utilized, k being the number of clusters or letters in our code-alphabet. In the case of the wiimote, we identified k = 14 to be a sufficient size for the code-alphabet, the cluster centers carefully placed on a three-dimensional sphere with a dynamic radius. From the theory of speech recognition we picked a discrete, left-to-right hidden markov model as the third component, the Model, and further optimized it for the wiimote. It has proven to deliver reliable results for patterns with spatial as well as temporal variation which fits the needs of our accelerometer-based gestures almost ideally. After this third component, a gesture (or rather: a series of codes) is mapped to a model probability. This model probabilty is then classified by the fourth component, a traditional Bayes-classifier, which identifies the most probable gesture during a recognition task.


Workflow

Since the wiigee library aims at not merely recognizing gestures but allowing you to define your own, it distinguishes two modes: training as well as recognition.

workflow
wiigee workflow


During training you (or your users) record the gestures your application later takes as input commands. For this purpose, each intended gesture is to be performed several times. This allows wiigee to learn the gesture and internally generate a code for it (represented by SQUARE in the above illustration). During recognition wiigee tries to identify the gesture which has just occured by computing which of the trained gestures fits the performed one most probably and reports this. You, as a developer, may then trigger a desired function accordingly. The following two subsections further describe the two modes.

Training
Let's assume you design an application in which you want the user to draw a circle into the air. Since different users perform such a circle in different ways (take n users and you get n circles), you have to make sure that your system recognizes circles in a reliable as well as robust way. Thus, telling your system only once what a circle is will not be sufficient. You have to train it repeatedly to get reliable results for a great range of users.
In wiigee, training is a recording process triggered by a specific TrainButton. This TrainButton must be held down during recording. Releasing it marks the end of the recording process. Repeating this whole procedure further trains the system and, by that, makes it more likely that a gesture is correctly identified during the later phase of recognition. We found that 5 to 10 training sessions are a necessary minimum to get feasible results but recommend 10 to 15. Once you are satisfied with the number of training sessions, pressing an explicit CloseGestureButton concludes the training phase.
Note that at this moment wiigee does not provide it’s own methods to store training data into a file or to load such stored data. This is a feature which will come with a later release.

Recognition
Now that your desired gestures have been recorded, wiigee internally sets up it's parameters for the recognition process. Your are now able to perform your newly recorded gestures. By default, a specific RecognitionButton is to be pressed and held down during your or your user's gesture performance. After you released this RecognitionButton wiigee tries to identify the gesture and fires a GestureEvent containing information about the detected gesture along with the calculated probability.
Alternatively, you may bind start and end of a gesture not to a button but to the wiimote’s motion. You may do so by setting the RecognitionButton to the “virtual button” Wiimote.MOTION and then tune it’s sensibility via the method setMotionChangeTime(int milliseconds). However, this approach is still experimental and we recommend the default method.


Starting the engine

To make your wiimote(s) discoverable during the execution of your application you have to press an initial button combination on each device: press the wiimote buttons button1 and button2 at once. This way, connections to your wiimote(s) are established and you or your users may start performing a training or a recognition movement. Your application is now able to perform gesture recognition.