Continuous recognition of player’s affective body expression as dynamic quality of aesthetic experience

Continuous recognition of player’s affective body expression as dynamic quality of aesthetic experience

Nikolaos Savva, Alfonsina Scarinzi, Nadia Bianchi-Berthouze

Abstract— The emergence of full-body computer games raises an interesting question: Can body movement be used to measure the aesthetic experience of players? In this paper, we aim to take a first step towards answering this question. Such a question emerges from the fact that various studies have shown the dual role of body movement, i.e., a window on people’s emotional and mental states as well as a means to affect people’s cognitive and affective processes. In this paper, firstly we investigate the possibility of automatically recognizing the emotional expressions conveyed by the player’s body movement in a Nintendo sport game. Our results showed that our automatic recognition system achieved recognition rates comparable to human observers’ benchmarks. Secondly, by taking a pragmatist definition of aesthetic experience into account, we argue that the tracked body expressions do not only express what the player may be feeling. Given their modulating role on cognition and affect, these body expressions also let the player actively construct and assign affective meanings to the unfolding of the game. We argue that the player’s variety of emotional bodily expressions constitutes the emotional rhythmic dynamic of aesthetic experience and, as such, they provide a measure of its distinctive quality.
Index Terms—Aesthetic experience, automatic recognition of emotion, embodied interaction, full-body computer game, engagement.

I.INTRODUCTION

HCI studies on players engagement in interactive full-body games (e.g., [6]-[8]) have shown that body movement can not only provide insights on how the person is experiencing the game but it can also increase the player’s level of engagement and create a more complete experience of the technologically-mediated situation. In fact, studies presented in [6]-[12] have shown that the player’s body movement can affect the player’s motivations to play the game, their feeling of being in the virtual word, their affective states and their social engagement; with the potential to increase, in turn, their level of engagement or even to modify the quality of engagement with the game (e.g., a shift from a hard-fun type to an easy-fun type [13]). According to [6], players enter an affective-loop in which the more they move the more they are affected and the more they want to move. In other words, body movement has a strong influence on the overall engagement experience and it provides the means to modulate such experience.

These studies on the role of movement and the affective- loop resonate with the role of movement and emotional engagement in a new approach to aesthetics that the field of HCI has developed: the so-called aesthetics of interaction [14], [1], following the pragmatist philosopher John Dewey. Accordingly, aesthetic experience is considered to be not pre-defined; it is not confined to being in line with the intentions of the designer of the system but emerges from the user’s active interaction with the system in context. Designing for aesthetic experience means inviting people to actively participate in creating sense and meaning [1], offering the conditions for leading users to move and engage emotionally whilst interacting with the system. Following aesthetics of interaction, bodily and emotional engagement becomes hence a vehicle of the conditions of aesthetic experience.

Moreover, an aesthetic experience, according to aesthetics of interaction that follows John Dewey's pragmatist aesthetics, is not sharply marked off from other experiences, such as ordinary experiences. Rather, when the satisfying qualities of an ordinary experience are lifted high above the threshold of perception and appreciated for their own sake, we feel an aesthetic experience [16], [17]. In other words, in pragmatist terms an aesthetic experience has a satisfying emotional quality that can be immediately felt as such [17:40] and appreciated for its own sake.

In the literature on pragmatist aesthetics in general and in the literature on aesthetics and HCI in particular, the question of how to measure the emotional satisfying level of an experience that qualifies it as aesthetic in the interaction with a technologically-mediated environment has not been tackled so far. In this paper, we aim to take a first step in this direction by arguing that the body expressions of the players provide the basis to measure their aesthetic emotional satisfaction. The reason is twofold. On the one hand, the player’s body expressions do provide information on how the player may be emotionally involved in the game. On the other hand, when these body expressions are either acted or imposed by the game (e.g., an aggressive shot in response to the cheering of the virtual audience or to the speed of the game), their qualities unconsciously bias the affective meaning the players assign to the emerging game experience according to the valence and arousal they portray. It is hence important as a first step towards building the measure for aesthetic emotional satisfaction, to investigate the possibility of automatically detecting the affective content expressed by the player’s body movement during game play. We then discuss how these tracked bodily expressions could be used to measure the player’s emotional bodily (or embodied) conditions leading to aesthetic experience.

The paper is organized as follows. First, in order to provide the rationale behind our approach, we briefly review studies on embodied cognition showing the relation between body movement and cognitive and affective processes. We then review the literature on game aesthetics and aesthetic experience and we discuss its relation with emotion and body movement. Then, we briefly summarize previous studies that have investigated the possibility of automatically detecting affective states from bodily expressions. Finally, we present our approach and results on the automatic detection of the player’s affective body expressions during game play. We conclude with a discussion on how this automatic monitoring of the player’s affective body expressions could be used to measure the player’s emotional bodily conditions underlying an aesthetic experience and leading to it. Finally, in the appendix, we provide a glossary for terms related to aesthetic experience that are used in this paper. These definitions are based on pragmatist aesthetics inspired by the works of John Dewey as well as by the works about game aesthetics we used in this papers.