GravField

Overview

We present "Gravitational Field" (GravField) an experimental and participatory live performance set in a co-located mixed reality environment. Participants wearing AR headsets are invited to join and collectively create real-time music through their collaborative body movements, guided by the programmed settings from live-coders. This innovative experiment harnesses intercorporeal signals derived from participants' body movements, including factors like distance, area formation, relative height differences, and head motion synchronization. Live-coders dynamically map these signals into metaphorical audiovisual patterns in augmented reality, providing participants with cues about the relationships between players. The live-coded patterns shape and influence participants' improvisation of body movements, while adjustments in live coding dynamically alter how these movements generate sound, influencing the overall improvisational experience. In addition, spectators outside the live performance area can enjoy the mixed reality performance through projection screens and TVs, enhancing audience engagement. GravField aims to explore the vast potential of intercorporeal signals, creating a communicative, playful, and co-creative space where players' interconnected bodies become the instruments of expression.

From live coding to live-configured objects

The later Nudging the Somas study develops GravField around an “object jockey” who changes the properties of virtual objects while participants move together. Ropes, springs, and magnetic fields provide recognizable metaphors; their apparent weight, elasticity, and force can change during the encounter. Visual and sonic feedback makes these configurations perceptible through movement.

The objects invite coordination without prescribing a fixed choreography. Participants can interpret a pull, negotiate distance, or discover another way of moving together as the configuration changes. The CHI 2026 paper examines this process through performance videos, interviews, soma trajectories, and field notes. It reports how ambiguity and live adjustment supported open-ended participation, and develops design principles for guiding collective movement while preserving agency. [→]

Publications

• 2026 — Nudging the Somas: Exploring How Live-Configurable Mixed Reality Objects Shape Open-Ended Intercorporeal Movements. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. Barcelona, Spain. Full Paper. [→]

Demonstrations

• 2025 — Improvising within “GravField”: A Participatory Live-coding Performance Exploring How Digital Objects Mediate Intercorporeal Movements in Collocated Mixed Reality. NIME 2025. Music Track. [→] · Proceedings and performance video
• 2024 — GravField: Live-Coding Bodies through Mixed Reality. Adjunct Proceedings of SIGGRAPH Asia 2024 (SA ’24). Tokyo, Japan. XR. [→]
• 2024 — GravField: Towards Designing an Inter-bodily Live-Coding Performance System within Collocated Mixed Reality Field. Companion Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY Companion ’24). Tampere, Finland. Work-in-Progress. [→]
• 2024 — GravField: A Participatory Performance Exploring Intercorporeality as Live-Coding Instruments within a Co-located Mixed Reality. Proceedings of the 2024 International Conference on Live Coding (ICLC ’24). Live Performance. [→]

Exhibitions

NIME 2025, Canberra, Australia — participatory live performance. Conference programme.

SIGGRAPH Asia 2024 XR. "GravField". Tokyo, Japan.

The International Conference on Live Coding 2024 Live Performance. "GravField". Shanghai, China.

@inproceedings{Hu_2024,
  series={CHI PLAY ’24},
  title={GravField: Towards Designing an Inter-bodily Live-Coding Performance System within Collocated Mixed Reality Field},
  url={http://dx.doi.org/10.1145/3665463.3678807},
  DOI={10.1145/3665463.3678807},
  booktitle={Companion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play},
  publisher={ACM},
  author={Hu, Botao Amber and Huang, Yuemin and Chai, Mingze and Hu, Xiaobo Aaron and Tao, Yilan Elan},
  year={2024},
  month=Oct,
  pages={124–130},
  collection={CHI PLAY ’24}
}