IV.Automatic Recognition of Player’s affective states

IV.Automatic Recognition of Player’s affective states

A.Features extraction and Modelling

Body Movement Features. Each movement feature is individually computed for each rotation axis and for each body part listed here

Since we are dealing with time-related features, a dynamic learning algorithm was better suited for building our system. A Recurrent Neural Network algorithm (RNN) [67]-[69] was therefore selected. The inputs to the network are the features listed in Table II. The number of output nodes corresponds to the number of selected affective labels. The RNN classifies any new incoming input by taking into consideration a number r of previously-classified instances. This parameter is called the depth of the RNN.

Each of the 161 game-point windows was split into a set of consecutive overlapping intervals of n frames each, where n was experimentally decided. We call these intervals n-frame windows. Each n-frame window overlaps with the subsequent n-frame window by n-1 frames. The features listed in Table II were computed for each n-frame window, rather than for each game-point window, providing a better measure of the dynamic of the movement. Each feature is computed for each body part and each rotational axis indicated in Table II. To take into account individual differences, the feature values of each participant were normalized according to the maximum and minimum values observed in each body part over the whole data set for that participant.
TABLE III

Initial Network Parameters

TABLE IV

ID	Angular Velocity	Angular frequency	Angular Acceleration	Orientation	Body Direction	Amount of movement	Body Rotation	Accuracy (%)
1	×	×	×	×	×	×	×	37
2	×							40
3		×						41
4			×					36
5				×				13
6					×			21
7						×		42
8							×	32
9	×	×						47
10		×	×					44
11			×	×				29
12				×	×			17
13					×	×		40
14						×	×	45
15		×				×		53
16		×					×	38
17		×			×			38
18	×					×		55
19			×			×		49
20	×		×					47
21	×	×	×					45
22	×	×				×		57
23	×		×			×		52
24		×	×			×		55
25		×				×	×	51

The classification of each n-frame window into one of the 4 emotion classes takes into account the classification of the previous r n-frame windows allowing the network to exhibit dynamic temporal behaviour. Then, the results obtained for the consecutive n-frame windows forming a game-point window are pooled together and using a majority rule an overall affective state is assigned to the game-point window. The reason to use a majority rule comes from the fact that the labelling of the data was performed at game-point window level. As such, we aim to test the system both at n-frame level and game-point level.

To build and test the system, the data set (i.e., 161 game-point windows) was split into training set (2/3^rds) and testing set (1/3^rd). The training set was used to tune the RNN parameters by applying a 5-fold cross validation. The remaining 1/3^rd was used to test the overall performance of the RNN (see Results section) after the tuning of all the parameters. Each parameter (Table III) was tuned individually by keeping all the other parameters constant. The range of values tested for each parameter is indicated in the table. Then, the identified values were tested again using all the parameters and slightly changing them for fine tuning. This last modification of the parameters was repeated until the system’s performance stabilized, meaning that the changes on the overall performance were less than 0.1% between consecutive runs. The optimal value for n-frame window size was 10. This means that the 161 game-point windows produced 65030 10-frame windows to be used as the training set (107 game-point windows) and 22680 10-frame windows to be used as the testing set (54 game-point windows).
TABLE V

Highest correlation values between pairs of features from Table II. Vel= Angular Velocity, Freq= Frequency. R == Right. Z = Z-rotation axis, X= X-rotation axis, Y=Y-rotation axis.

The tuning of the network also involved the identification of the best discriminative features. As the motion-captured data provided the 3D rotational information for each segment of the body, a testing of the RNN, using different combinations of these features and of each joint, was performed separately. Table IV shows performance results for some of the feature combinations. As shown in Table IV, the best RNN performance results were obtained using the features: the overall Amount of Movement, and the Angular Velocity and Angular Frequency of the right arm, forearm and hand along the three rotational axes (i.e., a total of 19 features). The Person correlation was computed for each pair of these 19 features (171 pairs) to identify possible redundancies. 93.5% of the pairs showed very low correlation (< .1). The remaining set of pairs showed higher but still quite low correlation values (<.5) as shown in Table V. The correlation between the feature Amount of Movement and any of the other 18 features was lower than 0.1.