Design New Realities (CAA Fall 2023)

byBotao Amber Hu

Design New Realities

"Design New Realities" is an innovative and practical course offered at the China Academy of Art. This course aims to equip students with the necessary knowledge and skills to create immersive stereoscopic mixed reality experiences using the iPhone and HoloKit. HoloKit team has been invited by the China Academy of Art to deliver a course centered around the utilization of HoloKit. By utilizing HoloKit as an alternative to expensive equipment like HoloLens, students are provided with an affordable and accessible solution for rapidly prototyping their mixed reality ideas. Through a combination of theoretical instruction, hands-on exercises, and practical tactics, students will develop the ability to design captivating mixed reality concepts, explore the intricacies of the design process, optimize workflows for efficient prototyping, and proficiently document their works.


School of Design and Innovation, China Academy of Art

Room B4-140

Tuesday, Wednesday, Thursdays from 1pm-4pm

Course Objectives

  • Understand the fundamentals of mixed reality and its applications.
  • Explore the elements that make a mixed reality concept successful.
  • Learn practical tactics and best practices in designing mixed reality experiences.
  • Utilize paper prototyping and speculative design to conceptualize mixed reality ideas.
  • Develop an optimized workflow for quick prototyping using the iPhone and HoloKit.
  • Acquire skills in documenting and presenting mixed reality projects effectively.

Teaching Team


Week 1: Introduction

Week 2: Presence and Co-Presence

Week 3: Developing on HoloKit

  • Get familiar with development environments: Mac, iPhone, Xcode, Unity
  • Learn the basic of Unity, C#, ShaderGraph and Visual Effect Graph
  • Resources:

Week 4: Developing collocated multiplayer on HoloKit

Week 3 - 4: Workshop


  • HoloLive
  • HoloLife
  • HoloLove

Week 5:

Mixed reality creation workflow

Bibcam Workflow

In this course, we offer an enhanced workflow for working on Augmented Reality (AR) using Bibcam. This new approach differs from the traditional AR creation process, which typically involves developing AR content in Unity, compiling it for iPhone, and then capturing it on location. In the traditional method, if the visual outcome is unsatisfactory, the iteration process needs to be restarted, leading to significant time consumption.

In contrast, our new workflow streamlines the process. It begins by shooting the scene using Bibcam format, which encompasses the camera video stream, ARKit tracking information, depth graph, and human stencil data, all consolidated into a single frame. Subsequently, the AR work is created and the visual aspects are iterated and refined in Unity without compiling into phones, based on the composition of the video background and AR objects. This approach allows creators to envision and pre-visualize the AR content more effectively before compiling it onto the phone, resulting in a more efficient and intuitive workflow.

Co-located Multiplayer AR


Warp Drive

Cell Space