The Sensation of Sound

Created by: NYT VR and Lytro

Year of release: 2017

Platform: Web VR

Viewing System: Gear VR

Producer: Maureen Towey,

Narrated by Rachel Kolb, animations by James Merry

The New York Times has produced more 360 videos than possibly anyone. For longer than a year, the Times ran its Daily 360 video feature, producing more than 400 videos along the way. The videos weren’t all amazing. Often, it felt as if the camera had been plopped in some far flung location without much thought of engaging the viewer in an experience. This is a problem that plagues many journalistic ventures into 360 degree storytelling. Places are prioritized over people. Scenery supersedes story. Audio is an afterthought (or not thought of at all).

But the true value of the Times’ Daily 360 feature may not have been the videos themselves. It was the opportunity to try out different techniques. Each of the more than 400 videos wasn’t just a video, it was a prototype to see what works, and what doesn’t, in immersive stories.

That’s what I thought of anyway, when I came across “The Sensation of Sound,” a 360 video published by NYT VR last fall. The video’s producer was Maureen Towey, a former staffer at the Times who contributed a number of videos to the Daily 360 series during its run.

“The Sensation of Sound” begins with a simple shot of a woman on a stage. Her name is Rachel Kolb. She stands beside a grand piano, in front of an empty theater and looks directly at the viewer. Her first words aren’t spoken. They’re signed, while the text subtitles appear to Kolb’s right:

“Have you heard music before?” she signs.

After this, Kolb speaks, while the subtitles continue. She tells the viewer she’d been profoundly deaf her entire life until the age of 20, when she received a cochlear implant. She says one of the questions she got most often from friends was about her experience of music. And the experience for her went well beyond what she could sense with her ears.

Visually, the film begins with a relatively simple composition - just a person on a stage. But the Times partnered with Lytro, a company that specializes in “Light Field imaging” to capture a fuller, more voluminous picture in VR.

“While other 360-video and VR cameras can capture live action from a single point, Lytro’s custom camera rig can capture an entire volume of light information, allowing a viewer to actually move around inside of video,” The Times claimed in a press release.

I experienced the film first on Gear VR and later on my iPhone, so I didn’t get to walk around inside the images. But Lytro’s visuals still felt more immersive and compelling than the standard flat 360 views I’m used to seeing. And the visuals get even more compelling as the video continues.

“I had never heard music, not really,” Kolb says, about a minute into the video. “But that did not mean that I wasn’t in some way musical.”

She turns to her right, toward the piano, where a hand-drawn animation of a child suddenly appears poised at the keys.

“When I was a child, I played piano and guitar for several years,” Kolb says, as another animation appears to her left, this time of a child with a guitar.

Kolb continues to describe her experiences with music while the animations multiply, eventually taking over the full screen.

“Music is a celebration of feeling movement,” Kolb says toward the end. It’s not just something you hear. “[It] is also visual, physical, tactile.”

It’s hard to avoid comparing “The Sensation of Sound” to one of the most widely viewed virtual reality experiences to date, “Notes on Blindness.” While “Notes on Blindness” told the story of a character losing his sight, “The Sensation of Sound” tells the story of a character gaining her hearing. Both use animation and rich sound design to immerse viewers in the sensory experience of the characters. Both were first introduced to a wide audience via The New York Times. But “The Sensation of Sound” is no “Notes on Blindness.” It’s much shorter, it lacks interactivity, and, in this version at least, it’s less ambitious.

But I don’t think “Notes on Blindness” is really the most useful comparison here. Instead, I see “The Sensation of Sound” as an example for news organizations that want to pursue compelling stories in VR. It’s less ambitious than “Notes on Blindness,” but it was also almost certainly cheaper and faster to produce. That makes it more attainable for cash-strapped news outlets. And, while Lytro’s Light Field technology adds a layer of immersion, complexity and cost to the main shot, the experience doesn’t depend on it to be compelling. What matters is the focus on the character, and her experience.

Quite a few news organizations have now experimented with 360 degree storytelling, but few, if any, have put out as much content as The New York Times. And while others have created more complicated and even more powerful experiences, “The Sensation of Sound” feels - to me at least - like a certain kind of revolution. It shows that compelling VR stories can be told with a relatively simple premise, and possibly a manageable budget, by focusing on people instead of plopping the camera in some far off place.

It shouldn’t be a surprise that people are what make VR stories great. They’re what makes every story great. I hope to see more news organizations follow this approach as journalism in VR expands in the years to come.