The Gestural Sound Toolkit is a collection of Max/MSP patches and objects for easy and fast gesture-to-sound scenario prototyping. It includes receivers from motion capture systems, signal processing modules, machine learning modules, and sound synthesis modules.

The function of the toolkit is to give participants a sketching tool to develop prototypes using interactive gestural-sound mappings. It integrates complex techniques for sound synthesis with machine learning of movement, making these techniques available to participants with no background in interactive systems, sound design, and more generally in programming and physical computing. It comprises modules for receiving movement data from sensors, analysing data through machine learning and gesture recognition, and mapping participants’ gestures to sound synthesis.

Overview patch with modules for interface communication, analysis, machine learning, sample-based synthesis and audio output.

The toolkit consists of three main categories of modules. The first one is the Receiver module. It receives motion data from devices such as the IMU sensors, Leap Motion, Myo and other input devices.

The second category of modules performs movement and gesture analysis and machine learning for gesture recognition, based on algorithms developed by Dr Baptiste Caramiaux. The system is based on a pre-recorded database populated by a user’s gestures and permits the early recognition of a live gesture as soon it starts. It also estimates variations of characteristics of speed, scale and orientation. This is particularly useful in the case of real-time interactions with sound in the prototyping phase as it facilitates procedures of gestural-sound mapping.

The Synthesis modules compose the third block of our toolkit. They enable participants to play with pre-recorded sounds and to manipulate them. In the Trigger module participants can play sound samples once, as if pressing a key on a keyboard or hitting a drum snare. The Scratch module works similarly to a vinyl player and the variation of speed changes the pitch of the sound sample. The Scrubbing module allows users to start the sound playback from a chosen playhead position and to change it in real-time. The Manipulate module controls pitch, speed and cutoff filter values of the sampler. Other sound synthesis modules are currently under development.

The modules can be assembled and linked as the users prefer. They are individual and can be copied, duplicated and rearranged.

Download

The toolkit is freely available from Github at https://github.com/12deadpixels/Gestural-Sound-Toolkit

Research Outputs

This toolkit has been designed by myself and Baptiste Caramiaux, and implemented in order to investigate research questions on sonic interaction design, through a series of workshops at IRCAM, Goldsmiths, Parsons (New York) and ZHdK (Zurich). The results of this research have been published at the ACM SIGCHI 2015 conference.

• Caramiaux, Baptiste, Altavilla, Alessandro, Pobiner, Scott and Tanaka, Atau. “Form Follows Sound: Designing Interactions from Sonic Memories”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 3943-3952. 2015 (http://dx.doi.org/10.1145/2702123.2702515)

The following is a short documentation video, displaying the toolkit being used by our participants in the context of the workshops.

Acknowledgements and Credits

GST has been developed as part of the MetaGestureMusic project, which received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP/2007-2013) / ERC Grant Agreement n. FP7-283771.

Original authors: Alessandro Altavilla, Baptiste Caramiaux