MIT.nano Immersion Lab Brings Motion‑Capture Filmmaking to Emerson College Students

MIT.nano Immersion Lab partners with Emerson College to teach virtual production

The MIT.nano Immersion Lab, a high‑precision motion‑capture facility housed in the MIT.nano building, has opened its doors to Emerson College students for a semester‑long virtual‑production course. Under the guidance of assistant professor Daniel Pillis (SM ’24) and lab technologist Talis Reks, students wear full‑body suits that stream data to a 28‑camera OptiTrack rig, instantly generating digital avatars that can dance, fight, or play guitar on a computer screen.

Technical approach: From suit to screen

The core of the lab’s capability is the OptiTrack motion‑capture system. Twenty‑eight infrared cameras are positioned around a 10 × 10 m capture volume, tracking reflective markers on the suits at up to 240 Hz. The raw marker trajectories are fed into Nexus software, which performs real‑time skeletal reconstruction and streams the resulting animation to Unity or Unreal Engine via a low‑latency UDP link. Because the pipeline is fully configurable, students can:

1. See their avatar’s pose instantly on a monitor, allowing on‑the‑fly adjustments.
2. Record high‑resolution BVH or FBX files for later editing in Maya or Blender.
3. Add facial rigs using a separate set of head‑mounted markers, a capability the lab plans to expand next semester.

The system’s accuracy—sub‑millimeter positional error and sub‑degree joint orientation—means that subtle gestures, such as a guitarist’s finger bends, are captured faithfully. This level of detail is usually only available in large‑scale studio environments that cost hundreds of thousands of dollars.

Real‑world applicability for students

Emerson’s curriculum blends theory with production. After a week of on‑site capture, students export their motion data and return to Boston to integrate it into short films, game prototypes, or interactive installations. Notable outcomes include:

• Nick Forsch’s “Enter”, a narrative short about a human transported into a digital realm, which earned an EVVY Award nomination for its professional‑grade visual effects.
• Evan Costa’s virtual Beatles performance, a recreation of the 1960s “Ed Sullivan Show” that combined motion capture with period‑accurate lighting rigs.

Both projects illustrate how the lab’s data can be repurposed across media: film, video‑game development, and even VR experiences. The hands‑on exposure also demystifies the pipeline that powers blockbuster titles like Avatar and The Mandalorian, giving students a credible portfolio piece that can accelerate entry‑level hiring.

Limitations and future work

While the Immersion Lab offers unprecedented access, there are practical constraints:

• Throughput – The capture volume can accommodate only one performer at a time, limiting group choreography unless sessions are staggered.
• Marker occlusion – Fast‑moving props or loose clothing can hide markers, requiring careful costume design.
• Post‑processing time – Although data is streamed live, cleaning up jitter and retargeting to high‑poly rigs still demands several hours of manual work.

To address these issues, Pillis and Reks are piloting markerless AI‑driven pose estimation using a custom TensorFlow model trained on the lab’s own dataset. Early tests show promise for reducing setup time and expanding capture to multiple actors simultaneously.

Extending the lab’s impact

The partnership demonstrates a model for how university‑level research facilities can serve broader educational communities. By providing a professional‑grade pipeline at a fraction of the cost, the MIT.nano Immersion Lab equips the next generation of storytellers with the technical fluency needed in a media landscape increasingly driven by AI‑generated assets and real‑time rendering.

Looking ahead, the curriculum will incorporate hand‑finger tracking, multimodal sensor fusion (combining inertial measurement units with optical data), and interactive generative capture, where AI agents respond to a performer’s movements in real time. These advances will not only raise the bar for student projects but also generate new research data for the lab’s own investigations into human‑computer interaction.

---

For more information about the MIT.nano Immersion Lab, visit the official lab page.