An Exploration of Machine Learning Image-Based Gestural Recognition in a Live Electroacoustic Context

This project is an interactive performance system designed for musicians interested in gesturally controlled live performance, inspired by the work of Imogen Heap and her physical controllers that expand soundscapes. It focuses on a live-performance piece, allowing the performer to explore the electroacoustic expansion of a previously solely acoustic performance. A primary motivation was doing more with less — eliminating the need for specialised hardware.

Developed through technical experimentation and artistic exploration as a classical cellist and DJ, the system maps key body landmarks to X and Y coordinates using the MediaPipe Pose Landmark Detection Kit in Python. That data is communicated to Max/MSP via OSC, where it is parsed and converted into MIDI CC messages controlling audio effects in Ableton Live. Conceptually the project draws from hyperinstruments, utilising movement itself as a MIDI controller — blurring the boundary between instrumental technique and digital manipulation.