Immersive Nocturne arts festival exhibit born from an interchange of ideas

A unique storytelling project created by faculty members at Dalhousie finds a public audience this weekend as part of Halifax's annual Nocturne: Art at Night festival.

Faculty members from the Faculty of Computer Science's Graphics and Experiential Media (GEM) Lab and the Fountain School of Performing Arts’s Cinema and Media Studies program collaborated on Interchanges: Shared Augmented Reality Stories, an immersive, in-person experience in which partners participate in one of two six-minute stories.

The first story, Standville Museum, features a mysterious quest where a detective’s son is kidnapped by a shadow and participants must follow clues to complete the quest. The second story, Spill, requires participants to gather information about a guest at a tea party by eavesdropping on a cast of colourful non-player characters.

The team hopes the public will experience something they haven’t seen before with Interchanges. Visitors have been able to experience the exhibit — hosted by the Museum of Natural History — since Nocturne began Wednesday (Oct. 13), and those who are still interested have until end of the day Saturday (Oct. 16).

Augmented reality

The concept for this story was born in winter of 2020 when Computer Science faculty members Derek Reilly and Joseph Malloch of the GEM Lab approached Shannon Brownlee, a Fountain School faculty member and associate Director of the Cinema and Media Studies program, about a potential collaboration. They knew she had an interest in virtual reality (VR) and augmented reality (AR) cinema.

Reilly explains they had an opportunity through a grant he received from Snap Inc., the company that makes Snapchat, to create a device or production that would feature the future of augmented reality storytelling.

“We teamed up under that grant and what we were looking at is how we can apply techniques from architecture and urban planning, space analysis techniques to allow us to figure out where to best place content and story events within a large space like a building or house,” explains Reilly.

Reilly adds that they needed some real examples to research that, so he, Malloch, and a team of students including Master of Computer Science students Abbey Singh and Ramanpreet Kaur and (then) undergraduate CS students Peter Haltner and Matt Peachey collaborated with Brownlee and three of the Cinema and Media Studies (CMS) students, David Choco Manco, Tory Dunlop, and Colton Morris to create the stories that would help build the settings where they could test this new technique. Computer Science student Shannon Frederick and several others later joined the team as developers.

“They created three different stories and we developed a tool that allows the authors to take their stories and realize them in augmented reality in a way that allows us to use [the spatial analysis techniques] that I described.”

Learning from each other

Due to COVID-19, most of the collaborative work on this project happened remotely. When Nocturne announced it would be happening this year, they realized they had an opportunity to deploy some of these stories and have people try them out.

“Over the summer, we’ve been working to really put those stories into production,” explains Reilly. “We’ve been working closely with Shannon and with others in the Fountain School and its costume studies, voice acting, and stage design and technical theatre programs to give us the setting that we wanted for two of the stories and so we created this setting that can be swapped for the two stories. In one instance it’s a museum and in the other it’s someone’s home where a tea party is being held.”

Brownlee explains that the stories were developed by the CMS students involved in the project, noting that it was a very experimental and iterative process, as none of them had experience with non-linear storytelling nor with Microsoft’s HoloLens – the augmented reality technology that the GEM Lab was working with – that the user wears on their head to provide a more immersive, integrated experience with your physical surroundings.

A collaborative undertaking

The collaborative nature of this project allowed the team to benefit from each other’s unique perspectives and expertise. “Honestly, I don’t think we could have done this research without this collaboration. While we’re good at creating systems and interactive techniques, we’re not necessarily very good at creating content. Especially when we’re talking about stories or different types of immersive experiences, content is at least as, if not more, important than the underlying technologies and interaction techniques. So, we needed that interdisciplinary collaboration to make this project a success,” says Reilly.

Brownlee echoes the sentiment, saying “The roles are not as clear-cut as one would expect. I have really come to appreciate that the computer scientists are excellent story editors and critical viewers. their process gives them in-depth knowledge of the stories, so they know where there are plot holes and other logical problems. They also have a great ‘feel’ for the stories — it’s not just nuts and bolts for them.”

She goes on to explain that as the three CMS students completed parts of their stories, they were then implemented in unity by the GEM Lab team. Together, the group learned more about AR and VR, discussed aesthetics and design. The CMS students worked on their screenplays and script, refining their stories and visions based on what they were learning about the technology and art form, and on their personal, artistic priorities and interests.

David Choco Manco, a FASS Honours student in Cinema and Media Studies and Sociology and Social Anthropology originally from Mexico, explains that during a CMS summer course, the students were studying VR, immerision and choice-making. They started interacting with students in the GEM Lab as they were researching VR as well, and a couple of weeks later, they were invited to create a choice-making story for this project.

“We had the creative freedom to choose what kind of story we wanted to write,” explains Choco Manco, who wrote the StandVille Museum story.  “I am glad to be part of this project and It was interesting to have a lot of meetings adjusting things, rewriting dialogues, and more to do something that we all wanted: ‘The creation of the stories into the real world’. All of us have put a lot of effort into this and I am glad that we can finally show it to the public.”

This collaborative project provided Computer Science masters student Abbey Singh with the opportunity to engage in a unique research experience.

“I had the pleasure to work closely with authors to bring their story to life. Part of the development of the author’s story was through my tool called Story CreatAR. In Story CreatAR, authors were able to specify spatial rules to define where story elements (avatars, furniture, etc.) should be placed. We are delighted to see how the public have reacted and are excited to see the impact of both stories.”