Body Oracle
2024
Concept
In our modern information age, we increasingly rely on verbal communication via text messages, emails, and webpages. Mehrabian's 7-38-55 rule suggests that spoken words communicate only 7% of meaning, while tone of voice conveys 38%, and body language 55%. Although emojis can deliver emotion via text, we still lack bodily awareness during digital communication. Our current vocabulary, both Western and Eastern, often falls short in describing complex bodily phenomena such as proprioception and social proprioception—the awareness of spatial positioning of body parts of oneself and others. This linguistic limitation potentially hampers our ability to comprehend and communicate these crucial aspects of human interaction—namely, body language.
What would an alternative world look like if our language included characters that inherently enabled us to be aware of the body language?
Inspired by Chinese Oracle-Bone Inscriptions, "Body Oracle" is a computational linguistic speculative design project, introducing a new system of AI-generated hieroglyphic characters representing corporeal and intercorporeal poses. These intuitive, cross-linguistic, cross-cultural signs—akin to emojis—aim to expand our capacity to conceptualize, understand, and communicate bodily and inter-bodily experiences. "Body Oracle" will be presented as an interactive art installation featuring a screen with a camera that recognizes participants' collective movements in real-time, translating them into these characters.
As Ludwig Wittgenstein said, "The limits of my language mean the limits of my world." Drawing on linguistic relativity—the idea that language shapes worldview—we speculate that these new hieroglyphic signs will function as a sociolinguistic mediator, influencing collective semantic cognition of bodily awareness. They may act as a true "Oracle," offering the transformative potential to conceptualize, describe, reinterpret, predict, and ultimately reshape our social reality through enhanced bodily awareness in all forms of communication.
Interactive Art Installation
The artwork will be exhibited as an interactive installation featuring a large screen and a camera system. As participants enter the installation space, their collective movements are recognized in real-time and translated into these new AI-generated characters, displayed on the screen. Participants will be prompted with suggestions like "be friendly" or "show disagreement," encouraging them to form collective movements. This process demonstrates the ambiguity in modern language while showcasing the effectiveness of the new "Body Oracle" characters in capturing the essence of body positions and reflecting intentionality.
The installation space will be designed to accommodate groups of 3-5 participants at a time. The floor will be marked with subtle positioning cues to guide participants' movements within the camera's field of view. Soft, ambient lighting will create a comfortable atmosphere for movement and interaction. The main screen, approximately 2 meters wide and 3 meters tall, will display the AI-generated characters in real-time, with animations showing the transition from human movements to symbolic representations. An additional smaller screen will provide instructions and prompts to guide participants' interactions. The installation will also include an area where visitors can view a curated selection of generated characters from previous sessions, showcasing the range and evolution of the AI-generated symbology.
Body Oracle Archive
The "Body Oracle Archive" is a system designed to document and showcase the symbols and processes generated by the "Body Oracle" interactive installation. It not only collects the generated characters from each interaction but also records participants' movement data and feedback. This archive serves to reveal the relationship between human bodily movements and the newly generated symbols, exploring how these characters convey emotions and intentions in various social contexts. By curating and displaying this archive, visitors can observe the evolution of the characters and how different participants' performances and creativity influence the outcomes. This archive not only provides valuable data for future research but also offers a space for participants and audiences to explore the evolution of bodily awareness and nonverbal communication.
How it works
The core of Body Oracle is a Generative Adversarial Network (GAN) designed to learn how Oracle Bone Inscriptions (OBI) form from hieroglyphic objects. We train the AI system with real OBI and then include a large dataset of videos featuring multiple human collective kinesthetic movements. This allows the system to abstract and create simplified strokes resembling the spatial positioning of all humans presented in the video frames.
a. Neural Network Model for Limb-to-Oracle Character Conversion
We invited volunteers to mimic existing oracle characters representing body poses. Using this data, we trained a pix2pix-based image transformation model and a neural style transfer model. These models established a mapping relationship between human limbs and Chinese oracle characters, forming our training dataset.
Next, we train a TensorFlow Lite human pose classification model using the MoveNet architecture. This model recognizes key points of the human body, which are essential for our motion capture programs.
b. Real-time motion capture interface
After training the neural network model, we implemented it in a web-based interactive interface. TensorFlow Lite recognizes human body keypoints, providing real-time feedback on posture uploads. When the difference in confidence between recognized keypoints falls below a preset threshold (which may vary in different environments)—indicating minimal body movement—the interface displays a countdown timer. After three seconds, it captures a still frame, which is then fed into the body-character conversion model. The processed output is displayed in the screen's center, giving birth to a new character.
Impact and Goals
"Body Oracle" envisions an alternative world that cultivates a deeper understanding of bodily awareness in social contexts, bridging the gap between verbal and nonverbal communication. By encouraging playful exploration of movement and interaction, the installation fosters collective awareness of our bodies in space, paving the way for innovative forms of communication in our increasingly digital world.
As Ludwig Wittgenstein said, "The limits of my language mean the limits of my world." Drawing on linguistic relativity—the idea that language shapes worldview—we speculate that these new hieroglyphic signs will function as a sociolinguistic mediator, influencing collective semantic cognition of bodily awareness. They may act as a true "Oracle," offering the transformative potential to conceptualize, describe, reinterpret, predict, and ultimately reshape our social reality through enhanced bodily awareness in all forms of communication.
Oracle Bone Inscriptions (OBI), a milestone in Chinese history, represent significant cultural heritage. However, due to their obscure nature, they have become a form of Chinese characters "lying dormant" on artifacts, with few people actively seeking to understand their charm. Through interactive and creative approaches, we aim to lower the cognitive threshold, spark individual interest and creativity, and integrate OBI into people's lives through engaging experiences.
