VR Digital Human Project

A VR Digital Human Project

We were approached by the estate of Jackie Robinson in the spring of 2019 to develop an exhibition at the Smithsonian Postal Museum in Washington in celebration of Jackie’s 100th Birthday. We created a story that couldn’t be told in any other medium with a historic player who changed baseball forever.

This was an ambitious project. We wanted to make a version of Jackie that was instantly recognizable and felt right. We wanted to use recreate a historic location that made sense for the time period. And, we wanted to do this in VR.

CREDITS:

Creative Director
Alexander Reyna
Role
UX/UI, Visual and Motion design

We worked with the Estate to develop a story set in the spring of 1946 based on Jackie’s memories in this tumultuous period of time.

we began by planning how we’d use the environment and Jackie to drive the narrative. Because we were working in VR we needed Jackie to walk towards the viewer, engage and make eye contact, but never invade personal space.

We recreated key elements from our digital version of Polo Grounds including camera location, placement of Homeplate, and various field elements to help guide the positioning for our MOCAP.

We began by amassing as much detail as possible about the time, Jackie’s mannerism, his period clothing, and how fabric moved while in motion.

Through collaboration with our partners at Smithsonian Institute, Cooperstown, and MLB Network we uncovered priceless artifacts from history including Jackie’s Jersey from the 1946 season, his bat, his signature padded cap used to protect his head from errant balls, and the actual contract which changed baseball forever.

LookDev

Because the concept was rooted in the 1940s, we experimented with a direction that hearkened back to a desaturated print aesthetic.

We pushed details farther than we would for a consumer-level product with a seductive level of visual polish where we sweated every pixel and detail. We pushed for the most exact digital human representation possible because the fans asked for it.

It's hard to represent human likeness in any medium, but more difficult in VR because of added technical and performance challenges.

We’ve experimented with numerous digital human projects over the years and our goal has always been to capture personality and likeness in a way that feels right. We’ve peered closely at the textures and colors of the sport. We’ve spent years exploring cloth and fabric, the subtle nature of how skin interacts with light, and how the human body changes as it performs feats of athletic prowess.

Realtime screen capture in Unity 2018.4

Realtime capture of custom skin shader development in Unity 2018.4

We placed Jackie in the 1940s Polo Grounds where he hit his first Major League Home Run and set him in the center of a metaphorical hurricane representing the turbulent mood of that moment. We recreated small details like stadium signage, admission tickets, grass mow patterns, scoreboards, and even Home Plate which was removed before the stadium was demolished in 1964.

Technical Solutions

With a locked VR camera we knew that we’d need to add details where it counts.

This meant that anything close to Homeplate needed extra attention and polish. We experimented with performance-intensive approaches like tessellation or real-time parallax shaders but settled on a high poly sculpted approach for dirt, grass, and other signature items. Basically lots of geo and tons of hero grass cards.

Lighting and mood were critical. We developed an in-game technical system that allowed us to dynamically change the time of time from dark and storm Hurricane to magic hour sunset complete with volumetric light shaft, clouds, and dust motes in the air, all using performance-friendly baked lighting.

High poly dirt sculpted, decimated, and stitched into low poly geo

Dynamic Time of Day lighting with volumetrics

Animation

Motion is at the heart of great storytelling.

We knew that we needed strong animation and technology to capture the essence of Jackie’s personality, facial expressions, and emotions. Our approach included a MOCAP shoot based on the physical dimensions of the virtual environments

We built a FACS system that allowed us to use a wide range of realistic expressions and emotions on digital Jackie’s face. We developed new technical solutions for realtime wrinkle maps and custom shaders for skin, cloth, and other organic materials.

We also built tech inside our game engine that allowed us to make sure that digital Jackie knew where the “user” was and would acknowledge and make eye contact when appropriate.