Kinetic Imaginations: Exploring the Possibilities of Combining AI and Dance

1 Introduction

Can computers be creative and artistically inventive? What kind of exchanges or collaborations between artists, audience and artificial intelligences might be useful or interesting? These are some of the motivating questions behind research into so called "computational creativity". This paper applies these issues in the area of dance and choreography and investigates whether AI can be kinetically inventive and if so, what this might mean for the interaction between dancers and intelligent software.

The results and discussions presented in the paper emanate from experiments performed in the interdisciplinary project AI am bringing together a dancer, researchers and creative technologists. The project investigates how AI can unveil new territories of possible dance movements, thereby supporting the creative process of dancers and choreographers.

The paper begins by briefly discussing how kinetic creativity may be characterized in formal and abstract terms as an exploration of a conceptual space. This notion entails the possibility of simulating and implementing kinetic imagination in a computer, at least hypothetically. Formal and computational approaches to dance and choreography are then discussed, illustrating how formalization and abstraction of movement can unlock kinetic spaces of imagination in theory and practice. Some examples of how choreographers and dancers have applied software and AI in choreography are also given.

The authors' specific take on kinetically creative AI is then outlined: its assumptions and simplifications are briefly discussed as well as the statistical approach to movement analysis, based on dimensionality reduction of motion data and exploration of pose maps. The concept of a virtual dancing avatar which improvises in real time is presented, and experiences of bringing the avatar to the dance studio as a virtual dance partner are reported and discussed. This is followed by an analysis of the creative interplay between human dancer and avatar. Finally, some conclusions are presented.

2 Kinetic creativity in humans and computers

Creativity is sometimes described theoretically as an exploration of conceptual spaces [Boden, 2004]. Within this theoretical framework, the act of composing music, designing objects or making any other form of art can partly be understood as an investigation of possibilities distributed in an abstract space. Metaphorically, the artist walks along a landscape of both habitual and hitherto unexplored regions. By examining the contents of the conceptual map by various techniques, unexpected and potentially useful outcomes can be encountered.

In the context of dance and choreography, this notion of creativity raises a crucial issue: Can one define a conceptual space of possible dance movements? This is a complex and open-ended question which cannot be answered without clarifying what one means by "possible" and "dance movement". Do all mathematically conceivable configurations of limbs and joints represent "possible" poses, or should physiological or other constraints be considered? If so, which constraints, and how? Furthermore, movements consist not only of poses and spatiality, but also dynamics. Can temporal aspects of movement be represented conceptually as spaces?

This paper will briefly touch on these issues but does in no way attempt to resolve them. However, some general reflections about the very idea of "conceptual movement spaces" can be made. First, it is worth noting that many contemporary dance theorists and choreographers approach movement in formal and systematic ways, conceiving of body and physical space in abstract and geometrical terms. These systematic approaches can partly be understood as efforts to construct conceptual spaces of possible movements. Arguably, this is one of the artistic benefits of formalizing kinetics: to unveil movement possibilities that might not appear "naturally", by mere "intuition" and without deliberate intellectual effort. Second, the notion of conceptual spaces lends itself quite naturally to AI and design of computational systems. For example, exploration of "ideas" (locations in conceptual space) can be implemented computationally as simple mathematical operations. 1

3 Formal and computational choreography

The creative choreographic process typically starts with a period of movement exploration around a chosen theme ranging from a piece of music, a narrative, to simply a movement idea. At this stage, practitioners find ways to tap into their primal, intuitive sense, what can be referred to as "embodied knowledge" or "the body's wisdom". A variety of techniques can be used such as improvisation and specific tasks or games as a means to stimulate new ways of moving and making. One common approach in movement exploration is to investigate the possibilities of the body in space.

There are a number of facets through which space can be explored through improvisation. One is its geometrical manifestations and their consequent expressive possibilities. This approach to the study of movement was first coined by Rudolf Laban as Choreutics or Space Harmony. According to Laban, the body is seen in space as "living architecture", linking the relationship of joints and limbs to each other with their possible relations to imagined polygons and solids. The limbs can trace shapes, move within and between vertical and horizontal planes, and inhabit dynamic relations of the diagonals in an imagined cube. Laban proposes the relation between space and its dynamic counterpart. For example, the lower level of space is associated with heaviness and high diagonals with lightness [Laban, 1966]. A dancer may explore these concepts of space through improvisation and then build on the resulting movements to create choreography.

The division of space into the basic components provides a comprehensive structure through which all movement can be analyzed and explored. Creation can take place in a conceptual space where the body reacts and interacts with imagined objects and relations with their bodies. This work has arguably been the fundamental influence that shapes what we recognize today as contemporary dance [Preston-Dunlop, 1998, p. 270-5] and has shaped the choreographic styles of famous names such as William Forsythe, whose work is among the pioneering efforts to present movement and choreographic strategies as models for interdisciplinary study.

1 This does not imply that creativity as such is a trivial phenomenon, reducible to spatial exploration; Boden [Boden, 1998, p. 349] also emphasizes the importance of domain knowledge and expertise.

Forsythe has revolutionized the ballet tradition by applying Laban's spatial ideas of tracing and transposing movement to traditional methods of choreography. Some ideas include tracing U's and O's in relation to three-dimensional axes, also tracing everyday objects and manipulating them with different body parts. The result is an abstract way of thinking about space, applying it to one's own body and relating to others in space. Forsythe's ideas have been released as the CDROM Improvisation Technologies (1995), where the ideas are visually presented to aid the user's imagination. Forsythe later presented the interactive website Synchronous Objects (2009), a cross-disciplinary investigation into his piece One Flat Thing reproduced (2000) in collaboration with scientists and designers at Ohio State University. The work analyzes the choreographic structures to produce visual and annotative representations, thus making the case for the application of choreographic knowledge in other fields.

However, it is important to note that use of computers to visualize possibilities of space and time was pioneered before this. In the 1980s Merce Cunningham started choreographing with the aid of the computer software LifeForms, developed at the Simon Fraser University. The software worked as a "visual idea generator" through which Cunningham could view and manipulate movement poses and sequences as well as combine multiple bodies and phrases in digital space and time [Schiphorst, 1993]. In 1992, Trackers was premiered as his first work choreographed with the aid of the software.

Cunningham continued to experiment with other technologies such as real-time motion capture and AI within performance. From his collaboration with Open Ended Group (OEG), Loops (2001) is an example in which both were employed at once. The movement of Cunningham's hands was recorded and simultaneously analyzed by an AI, which autonomously determined how the resulting visuals would behave.

OEG's contribution to the use of AI in live dance performance has revolutionized the possibilities of exchange between science, technology and dance in the past fifteen years. Other choreographer's that have benefited from their expertise include Trisha Brown, Bill T. Jones and Wayne McGregor.

In the collaboration of OEG and McGregor, AI and choreography are fully merged into what they call the Choreographic Language Agent (CLA). CLA is a software that uses a language system with which the choreographer instructs his dancers to generate its own movement. The input to the system is a sentence which produces an animation in response. CLA has been used in rehearsal as well as development [deLahunta, 2009]. The abstract geometric forms generated by the CLA software have recently been replaced by a more human-like shape, used as an "eleventh dancer" in the studio [Rothwell, 2014].

Another case of computer-aided choreography is Scuddle, a tool which generates incomplete movement material based on a Laban inspired formalism [Carlson et al., 2011].

The examples above illustrate how dancers and choreographers use formal systems and computer software as choreographic tools and catalysts for enhanced creativity. The presented approach positions itself in the same tradition, but in contrast to previous work, our experiments center around an improvising, humanoid AI, acting as a virtual dance companion. [McCormick et al., 2013] describe a similar approach where a dancing AI learns movements from its human partner. However, their AI is designed for movement recognition rather than creative elaboration.

4 Pose maps

The pursuit of creating a kinetically creative AI can be addressed scientifically in different ways. Here we approach the problem statistically and assume that human dancers have a style or repertoire which can be characterized as a tendency to perform certain poses. By recording and analyzing a dancer's movements and creating a statistical model of them, we assume that the dancer's style - the "signal" in the data - can be captured, at least to some extent. This model can then serve as a basis for exploring novel variations of observed movements.

For purposes of simplification, we focus on the postural content of movements rather than their temporal dynamics. In other words, the statistical analysis only deals with how limbs and joints are configured in particular moments, rather than how poses within a movement develop over time.

By performing a statistical analysis which reduces the complexity of the training data, a "map" of possible poses is created. This pose map preserves topological relations in the training data and contains regions of observed as well as unfamiliar poses. A central notion in this approach is that it enables exploration of possible movements. In other words, the pose map can serve as a conceptual space of possibilities.

In concrete terms, the analysis consists of the following steps. First, recorded movements are stored as vectors containing orientation data for each joint. 3D orientations between joints are represented by unit quaternions (4 values). Secondly, the dimensionality of the data is reduced by nonlinear kernel principal component analysis (KPCA) [Sch¨olkopf et al., 1998]. A more detailed account of the method and some of its limitations is given in [Berman and James, 2014].

For the purposes of our experiments, a database was created by recording about 5 minutes of material. The moves were performed by the team's dancer and were derived from poses that relate to certain spatial points, and the movement that arises from moving back and forth between these poses. This approach is directly influenced by Rudolf Laban's Choreutics or Space Harmony method [Laban, 1966].

A skeleton model with 52 joints was used, out of which 21 finger joints with static orientations were excluded from the analysis, yielding 31 × 4 = 124 input dimensions. By KPCA we can reduce the input dimensionality to an arbitrary number of map dimensions. The exact choice depends on the context and can be guided experimentally, taking into account various considerations such as accuracy versus simplification.

Figure 1 shows the contents of a pose map and illustrates several of its typical characteristics. First, the distribution of poses embodies a topological structure where postural similarity is reflected in distance across the map. Second, the map accommodates not only examples that are representative for the training set, but also less typical ones. This property of PCA distinguishes it from some other dimensionality reduction techniques such as self-organizing maps [Kohonen, 2001] and is highly relevant for our purposes.

5 Map exploration

Our tool allows dancers and other users to manually explore pose maps of arbitrary dimensionality. Furthermore, a method for automatic map exploration has also been developed. Trajectories across the map are generated by an algorithm and can then be synthesized as full movements by sequencing the constitutive poses denoted by the path. This algorithm constitutes a kind of automatic "improvisation" and has been designed to satisfy several constraints. Primarily, it should be able to yield pose combinations that have not been observed in the original data. Furthermore, the algorithm should produce somewhat realistic and familiar output by favoring familiar regions of the map. On the other hand, it should also permit exposure of novel poses beyond observed territories. Additionally, the trade-off between familiarity and novelty should be guided by a parameter, enabling users to experiment with different options. Finally, the length of generated paths should be controllable, and their shape should yield a desirable result when intermediate poses are aligned in time; sharp twists or turns may result in jagged or unnatural movement.

The requirements were satisfied by an algorithm based on randomness, attraction to observed poses (familiarity) and deviation from observed poses (novelty):

1. Select a random observed pose in the map as the departure point p
2. Generate a set of destination candidates {qi}, where each candidate is a random observed pose plus a random vector of magnitude N (the "novelty" parameter)
3. Choose the destination q as the candidate among {qi} whose distance to p has the smallest difference from the preferred distance E (the "extension" parameter)
4. Choose some intermediate points between p and q, where each intermediate point lies between a point on the straight line between p and q and its nearest observation in the map (closer to the nearest observation for lower N values)
5. Smooth the resulting path using spline interpolation
6. Create the next trajectory by treating the current destination as a departure and repeating steps 2-5

The method allows movements to be generated and rendered graphically in real time in the form of a dancing avatar. 2 By adjoining paths so that the endpoint of one trajectory becomes the departure for the subsequent one, a continuous stream of movement is obtained. "Foot skating", resulting from a lack of physical constraints, is eliminated with a simplistic friction model. A video showing an example of a generated motion sequence can be found at http://tiny.cc/kinetic-imag

The example stems from a pose map of 7 dimensions, a value which was reached experimentally. As can be expected, the map dimensionality governs the avatar's degree of freedom. Lower values impose more dependencies between joints and their orientations. Consequently, a small number of dimensions constrains the postural repertoire, while higher values cause unnatural and extreme poses.

6 Assessment

The method developed within the project was assessed in the studio by the team's dancer. Her specialty lies in modern dance and improvisational techniques for movement exploration and performance. She is also a budding dance-maker, with experience in composing short works. The dancer was actively involved in the research from its inception. The assessment consisted of a number of experiments aiming at investigating the AI's usefulness as a creative catalyst, and is based on her own subjective analysis of her sensations and perception of new ways of moving and approaching movement. The dancer recorded her findings through note-taking and video.

2 In comparison to a previous version of the algorithm [Berman and James, 2014], the new version honors the extension parameter better, thereby providing more fine-grained control over the behavior.

The software was assessed as an improvisational and compositional tool. In the improvisation experiment, the avatar's movements were used as visual stimuli from which the dancer's own movements could originate. The avatar's degree of novelty, extension and speed were manipulated and the effects of the parameter variations on the generated movement observed. The dancer noted how freely her movement ideas flowed as a result of the stimulus. Video recordings were taken so that she could assess her own improvisation.

The next stage of the experiment focused on compositional strategies referred to by the dancer as choreogravatar, free association and spatial blueprint. For choreogravatar, the dancer learned the avatar's movements as she would from a choreographer. A video of the avatar's improvisation was recorded, with the novelty parameters set to minimum to ensure that all movements would be humanly achievable. The dancer attempted to learn the movements as precisely as possible. This second experiment resulted in two movement phrases. A structured improvisation was then created based on movements learned in the previous phrases, to further investigate their possible expressive qualities.

In free association, a phrase was created based on the kinaesthetic memory of learning and improvising with the avatar. This was followed by a task of selecting or "cherrypicking" specific movements observed from the avatar according to the interest or surprise they evoked, and incorporating them into the free-associated phrase.

Spatial blueprint shifted focus from the avatar's visual form to the pose map, using it as a guide for generating movement. The software contained a window showing the realtime journey of the AI through the pose map where the current path and location was highlighted. Ten trajectories were chosen and drawn as ten separate frames in the order that they appeared. The shape of the pathway and the direction of the AI's journey along it was noted. This information was interpreted by tracing the pathway with varying combinations of body parts and transposing the 2D information into 3D space intuitively. The pathways were also explored as floor plans directing the location of the body as a single unit. A fourth phrase was created.

7 Observations

The dancer was surprised by the complexity of the avatar's movements in contrast to the simplicity of the dataset, and expressed her eagerness to use it in the studio. She observed that with parameters set at low novelty, the avatar performed movements recognizable from the dataset but in a different order and interspersed with movements never seen before. Combined with a medium extension, she reported a "cautious" quality in its improvisation. When extension was increased, the dancer noted that the movement became more expansive, taking recognizable poses to unnatural extremes.

The dancer records that innovative behavior in the avatar increased with the novelty parameter. With maximum novelty and medium extension, the avatar's movement is described as more brave and reminiscent of that of a break dancer. When extension and novelty were at their maximum settings, the avatar's movement went to extremes humanly impossible to replicate. Even then, the dancer has found this to be inspiring.

It was observed that although the dataset was composed to reflect only spatial aspects of the movements, the algorithm produced interesting variations in flow and rhythm. These deviations contributed to the expressive appeal of the avatar's novel behavior and hence increased its capability to inspire creative responses.

In the improvisation experiment, the effect of the avatar's style on the dancer's was immediately felt. She began to sequence movements in unusual ways and incorporate the avatar's irregular dynamic variations. The dancer responded to the avatar's extremes in its conceptual space with her own exploration of the extremes of her physical space.

In the choreogravatar experiment, the dancer reported difficulties and frustration when trying to learn phrases from the avatar. Glitches, along with the avatar's lack of real-world physical limitations contributed to this. During this experience, the dancer realized that in her pursuit of precise replication, she had plagued herself with the "detail overload" that Schiphorst refers to in her account of Cunninghams use of LifeForms [Schiphorst, 1993]. The excessive attention to detail blocked the intuitive aspect of composition and consequently the iterative process characteristic of developing choreography.

The dancer's solution to detail overload was to use the learned movements as a basis for a structured improvisation, in which she could explore the spatial and dynamic potential of the movements in a broader sense. She investigated the expressive possibilities of repetition and how some movements can be made locomotive. During this approach, the dancer noted a sense of flow.

The free-association and spatial blueprint experiments allowed the most intuitive approach to the avatar's movements, facilitating greater creative transformation and flow. The phrases derived from these methods were the most expressive and innovative, best demonstrating the creative potential of human-algorithm collaboration.

The experiments described above resulted in four movement phrases of varying lengths. They can be used as motifs in further choreographic development. Each motif can be varied to explore countless spatial possibilities, with dynamic qualities and rhythmic variations added to produce a more nuanced and therefore expressive result. The use of pathways as floor plans could be explored as a way of connecting the motifs with each other or converting otherwise static movement into locomotive movement.

It was found that the large quantity of spatial information in the movements became boring without variation in dynamics and rhythm. This was not surprising given the essential relationship to space and dynamics as defined in Laban's Space Harmony. However, the minimal variation of dynamics is linked to the capacity of the software and presents possibility for future development.

8 Discussion

Our experiments indicate that statistically derived pose maps constitute a useful basis for exploration of possible dance movements. The dancer in the team reports various positive outcomes and potentials for artistic applications of this kinetic AI. Using the software, the dancer finds movement inspiration and is able to further evolve and extend ideas generated by the AI. She can use kinetic material from the AI as building blocks for new choreography.

The dancer's experiences from using the AI in the studio validates the relevance of the approach, identifies weaknesses in the system, but also raises several questions of more theoretical and philosophical nature. Is it meaningful to call this AI creative or artistic? And how should one characterize the interaction between dancer/choreographer and software?

In order to address these questions, we return to the notion of creativity as exploration of conceptual spaces. In more concrete terms, what is the space that the dancer and the AI explore, and how do they do it? [Wiggins, 2006] provides a formal language for talking about such matters. His "Creative Systems Framework" formalizes ideas from [Boden, 2004]. In Wiggins' framework, any creative system - biological or not - essentially consists of the following components:

The universe of the creative system is the set of all possible "concepts". In the present context, concepts are movements (or aspects of movements), and the universe therefore contains all possible movements. It is important to note that the universe is inhabited by both relevant concepts and those of no particular interest or value.

A conceptual space is a subset of the universe, and results from a selection procedure which carves out regions of the universe to be explored. In the context of dance, the conceptual space may consist of humanly realistic movements.

Like the universe, the conceptual space makes no distinction between concepts on the basis of their artistic potential. This is the purpose of the evaluation function: to attribute value to concepts by assessing their usefulness or relevance.

Finally, the exploration strategy describes how the conceptual space is searched and explored. Typically, the exploration mechanism uses the evaluation function as a "guide". But it may also use "blind" strategies which serve to increase the chance of reaching distant but desirable concepts.

This formalization enables us to ask whether the AI is creative. First of all, what is its conceptual space? The pose map presented in section 4 seems to be an obvious candidate. It is constructed by reducing the space of all possible poses (the universe), in our case by dimensionality reduction. Furthermore, an algorithm exists for exploring the contents of the pose map.

But if pose maps are conceptual spaces in Wiggins' terms, they are spaces of poses, not of movements. When the software "improvises" by traveling along a pose map, the journey can be visualized as a movement by rendering the encountered poses as an animation. However, neither the AI nor the dancer evaluate the constitutive poses within the movements per se. Hence, the pose maps do not serve as conceptual spaces in our experiments.

Another option is to conceive of pose maps as components of movement spaces. This is a much more abstract notion. Our approach treats movements as sequences of poses. More specifically, a conceivable movement is a pose sequence constituting a possible trajectory across a pose map. 3 The movement space is the set of possible movements. In contrast to a pose map, whose shape is a Euclidean space, the shape of a movement space is somewhat more difficult to conceive. But in a formal sense, the notion as such is perfectly workable.

How is this conceptual movement space explored? When the AI "improvises", it generates movements as paths across a pose map, "performs" individual movements and steps from one movement to another. In other words, the AI "walks" across the movement space. In the meantime, the dancer can observe the exploration as it unfolds and evaluate the encountered movements. This formal characterization of the process seems to capture important aspects of the activity. Importantly, it highlights the role of the dancer in the evaluation of the AI's output. Arguably, if this is a creative system, then the system contains both the dancer and the AI. This hybrid of human and software can perhaps be understood as a human-computer creative partnership [Kantosalo et al., 2014; Lubart, 2005] or a case of "distributed cognition" [Hollan et al., 2000].

8.1 Evaluation and embodiment

As reported by the dancer, movement ideas generated by the AI are evaluated not only by observing the avatar but also by trying to perform the moves physically. By embodying the ideas, the dancer interprets and implements themwith her own body. In Wiggins' model, embodiment fits into the notion of concept evaluation. But it may also be understood in terms of conceptual and material space, as part of an interaction between ideas and material [Dahlstedt, 2012] - an aspect which seems crucial for understanding the creative process in dance.

8.2 Transformational and meta creativity

So far, the analysis has focused on the observing and evaluating role of the dancer in relation to the AI. In the actual experiments, the interaction was more interactive. As described above, the dancer is able to experiment with different parameters governing novelty, extension, etc. By doing so, she effectively alters the exploration strategy of the system. In other words, we are dealing with a kind of "exploration of possible explorations" or "creativity at the meta-level" in Wiggins' terms.

During the earlier phases of the experiments, the team also experimented with different dimensionality reductions: linear versus nonlinear, and different number of PCA components. The team thereby modified the conceptual movement space itself, a process which Boden calls "transformational creativity" [Boden, 1998]. 4 Furthermore, different variants of algorithms for path generation were explored. In other words, most components of the creative system have been subject to meta exploration. To this end, one may perhaps conceive of the whole team as parts of the creative system.

3 "Possible trajectory" here means a path that may be created by the trajectory generation algorithm.

4 For Wiggins, modification of the exploration strategy is another case of transformational creativity.

The interaction between humans and machine in the current setup can be developed further in many different directions. For example, the dancer's assessment of the movements are currently not fed back to the AI. The inclusion of such a feedback mechanism could enable positive and negative reinforcements of exploration strategies, thereby further enhancing the transformational creativity of the system as a whole.

9 Conclusions

The convergence of dance and AI poses several questions: What kind of interactions and exchanges between human and virtual dancers can be conceived? And how can AI contribute to the creative process of dancers and choreographers? This paper has presented an attempt to approach these questions by developing and experimenting with a kinetically creative AI. Some conclusions from the work can be drawn. First, the performed experiments provide some empirical support for AI as a potentially useful tool for enhanced improvisation and composition in dance. The specific approach, based on automatic exploration of pose maps derived statistically from motion capture data, has the ability to reach beyond the "input" into novel territories of movement. Experiments performed by the dancer in the team show that the avatar's movement are surprising, inspiring and carry choreographic potential. In future work, it would be interesting to perform more systematic evaluations with a larger user group and a standard procedure such as Creative Support Index [Carroll and Latulipe, 2009].

Second, the creative process that has emerged between the dancer and the AI has many nontrivial qualities which cannot be easily characterized in terms of creative capacity and agency. Here we have analyzed the algorithms of the AI and the interplay between researcher, dancer and software as a joint exploration of conceptual movement spaces. In this picture, humans and machine are components of the same creative system. Naturally, other conceptualizations can be made. We see the presented results and the theoretical analysis as contributions to an ongoing discussion about how creative collaborations between artists and AI can be described, enabled and analyzed.