Since the introduction of immersive technologies to the mainstream market, beginning in earnest in 2016 with the release of HTC and Valve’s Vive room-scale virtual reality (VR) system, immersive technologies have offered incredible creative potential for new artistic and narrative works.

Immersive technologies now cover a range of technologies, such as room-scale, desktop, and inside-out tracking VR; phone– and headset-based augmented reality (AR); 360-degree video; and mixed reality. Additionally, within each of these technologies lies the capacity for both interactive and passive content.

Despite its cross-disciplinary applications, immersive technologies are not yet widely integrated into the practice of working artists and content producers. Barriers to adoption include the relatively high cost of production equipment, new and constantly evolving technology streams and a lack of appropriate training opportunities geared toward the arts.

Ontario-based trade association Interactive Ontario (IO) recognized this challenge. Together with its national collaborator, the Cultural Human Resources Council (CHRC), funding was secured through the Canada Council for the Arts to deliver a 5-day program for 20 participants in 2019. The program was delivered in Toronto by Dames Making Games (DMG) and Gamma Space Collaborative Studio, nonprofit artist-run centres based out of the Toronto Media Arts Centre (TMAC). It was made possible with the support of the Canadian Media Producers Association (CMPA) and Actra Fraternal Benefit Society (AFBS).

Since 2012, DMG has provided free immersive and interactive skills training for marginalized artists using a peer-led facilitative approach: Short workshops with a focus on hands-on practice in a variety of formats, direct professional and community support, and participant-determined balance of group and solo work. These intensives have seeded hundreds of video games, wearables, linear narrative works and new tools and platforms for social action.

This approach helps artists and producers across a range of disciplines learn new skills rapidly by applying their individual creative vision and existing artistic practice. This provides a practical foundation for taking advantage of the expanded opportunities for storytelling offered by immersive and interactive technology.

Two key factors in the success of a program to address the low uptake of immersive technologies by artists are: Selecting the right immersive technology and creating an accessible training framework delivered in a conducive learning environment. These issues are discussed in detail here, and provide a blueprint for future endeavours. The program relies on competent instruction and expert mentors to support participants through the learning stages from initial exposure to expertise. In light of the COVID-19 crisis, which is impacting all spheres of activity, consideration for a remote delivery of this learning program has been included.

Selecting The Right Immersive Technology

IO and CHRC’s outreach revealed strong interest from filmmakers interested in exploring immersive technologies. Eighteen out of 20 participants in the program were involved in the film sector. All of the participants had experience with digital media production; two had worked with immersive technologies in a direct capacity, and one had ongoing involvement. The group represented a diverse mix of skills, resources, networks and experiences.

The program emphasized filling gaps in participants’ skill sets while allowing them to ground their projects’ content and expressive methods in their artistic practice. While a broad survey of technologies was presented during the program, the technologies selected by the coordinators for hands-on instruction needed to support:

Rapid iteration and testing. Waiting for projects to compile and load onto a device slows the learning process and can be distracting. For example, mobile platforms such as AR on Android and iOS, require multiple time-consuming steps to test changes.

Content production with existing skill sets. VR requires working with pre-made assets or learning how to create 3D models from scratch, a technical barrier.

Collaboration. The tools selected needed to be suitable for small groups to work with a single code base and library of digital assets.

Interactive Volumetric Video: Where Video Meets Video Game

Volumetric video meets all of the above criteria. Additionally, unique expertise exists within the DMG community, including artists and technical mentors with years of experience working with the technology.

Volumetric video involves the capture of a performance or location in three-dimensional space. The data captured contains spatial information, as well as video, and can be viewed on 3D displays, VR headsets and even traditional flat screens – albeit without the conveyance of depth information.

Some advantages of volumetric video:

  • Uses video-based clips for actors, props and other elements in a virtual scene
  • Video clips can be edited with readily available software
  • Allows the exploration of interactivity in widely used, well-documented game engine software
  • Original assets can be created without 3D modelling
  • Can break out auxiliary production tasks, such as writing, video editing and sound design to less powerful computer setups

Some disadvantages:

  • Doesn’t allow for the translation of traditional cinematic production techniques, such as framing and jump cuts
  • Requires knowledge of game engine software if interactivity is desired
  • Requires specialized depth-capture equipment and physical space for props and actors

Its potential to balance DMG’s facilitation priorities and IO and CHRC’s goal of supporting artistic creativity with immersive technologies made volumetric video the ideal technology choice.

Program Delivery

Once the technology was selected, DMG customized the delivery plan and curriculum for IO, CHRC and participants.

Format

The intensive was held over five days, from November 25 to 29, 2019, in DMG’s classroom and studio space. Each workshop day lasted eight hours, including break times. Each day started with instruction and presentations in the morning, followed by hands-on work and in-studio production in the afternoons.

After the first day of introduction, participants organized into groups of four after presenting narrative and interaction design ideas to each other. This supported DMG’s approach by making space for self-directed participation and peer learning, as individuals sought complementary skill sets and experiences in their groups.

The Learning Environment

Access to education varies greatly among intersections of marginalized groups. DMG has an explicit mandate to ensure diverse participation in all programs it delivers. An effective learning environment begins with a space that explicitly prioritizes the wellbeing and safety of participants. Participants with less experience, fewer resources and who have historically faced barriers to tech spaces based on their gender identity, race, orientation or cultural identity need to be provided with equitable access to instruction, materials, equipment and feedback in order to fully engage with and benefit from a learning environment.

IO and CHRC reached out to digital media producers through its industry network. DMG centers LGBTQ+ and gender-marginalized individuals, so its outreach was focused on those identity groups.

Providing equitable access means intentionally removing barriers marginalized groups frequently face: Child care, food security and transportation.

To reduce barriers to access for people experiencing socioeconomic challenges, transportation, travel and accommodation subsidies were offered to all participants. Meals and snacks were provided to all participants, and all dietary requirements were accommodated. Applicants were invited to request any other accommodations they required. To build confidence that this environment would be upheld throughout the program, this goal was made explicit in materials and presentations, and trusted instructors who support these values in their work and interactions were selected. Additionally, participants, facilitators, mentors, and speakers were regularly reminded of DMG’s code of conduct, which was posted in the physical space and made available online.

Situating the program within a creative hub allowed participants to engage with a broader community of artists with overlapping interests and specialties. Visitors, people attending workshops with other arts organizations, and professional members working in adjacent spaces provided ample opportunities for organic community participation, conversation and testing. Unhindered access to all spaces and relevant equipment within the hub was provided to provide participants with creative freedom and a sense of ownership in the learning experience.

Accessing Resources

A major barrier to participation in technology-intensive programs is the availability of specialty equipment.

TMAC has leveraged its members’ networks to develop relationships with equipment manufacturers, hardware and platform holders, and software makers to secure free equipment and software licenses. By accessing these relationships, a significant portion of the program budget could be allocated to human resources and costs associated with supporting participants directly.

For example, DMG was able to quickly secure two volumetric video capture workstations from AMD, two additional new VR headsets from Oculus, and Xbox Kinects by OCADU’s game:play lab.

Providing VR-capable workstations and volumetric video capture stations may be challenging for smaller groups as this hardware is not readily available to rent – or affordable to buy – in most areas of Canada.

Facilitation

The program was led by two main facilitators:

  • Max Lander, a visual artist and VR developer with specialization in volumetric capture and interactivity in immersive environments, led the development and delivery of the curriculum.
  • Henry Faber, director of Gamma Space, coordinated the technical aspects of the program, recruited mentors and speakers and provided industry expertise.

Lander and Faber have extensive knowledge in key areas of the curriculum, including:

  • State of production and distribution platforms and tools
  • Knowledge of both experimental and commercial development pipelines
  • Technical expertise in VR and volumetric video
  • Knowledge of funding, business development and publishing opportunities

Mentors

DMG’s membership is a key source of experienced mentors. Through semi-formalized training and year-round programs that cultivate a culture of mutual mentorship, DMG supports a pool of several hundred highly skilled and engaged practitioners.

Six mentors supported the program in-person and via online communication tools – during the workshops as well as during studio time:

  • three Unity generalists with experience in VR
  • a narrative designer
  • an audio designer
  • a volumetric video capture specialist

Industry Speakers

IO provided industry professionals from its network, who gave talks regarding their projects, success in the industry, as well as adjacent applications for immersive technologies. They complemented the facilitators’ knowledge base and helped to provide additional industry context.

Getting Participants Out of Their Comfort Zone

Getting participants out of their comfort zones can be a challenge in any instructional program. For artists, learning how to use new technology that has been designed for industrial and commercial users and applications may seem especially difficult. However, artists are well-suited to experimentation and tinkering, and with the right framework, the possibilities of immersive technologies can lead to creative breakthroughs not possible with traditional media.

Facilitators with direct experience creating immersive and interactive work, an abundance of examples that participants can experience firsthand, and an environment that encourages exploration without fear will support the success of a program that seeks to ease participants through the learning process.

For CHRC-IO’s Arts Futures Program, that meant ensuring the following:

  • Ample examples and opportunities to try a variety of immersive technologies within the context of the program by facilitators
  • Presentations by individuals and organizations currently working on immersive projects
  • Frequent one-on-one check ins by facilitators and mentors to pre-empt and resolve roadblocks, and provide a creative sounding board
  • Group discussions about participants’ current work (e.g., films in production) and drawing connections to topics covered in the program
  • An emphasis on regular testing and peer feedback

Facilitators also demonstrated troubleshooting techniques and methodical research to empower participants to educate themselves when direct support is not available:

  • How and when to use search engines, official documentation and community tutorials to initiate projects, solve problems and find inspiration
  • Developing medium-specific literacy in order to critically explore commercial and artistic VR projects, streams and “Let’s Play” videos
  • How to find and integrate pre-built assets and resources from the Unity Asset Store and GitHub

Setup and Tools

Tools

Chat

A Slack channel was created within the existing DMG Slack team for all participants, facilitators and mentors to chat with each other and share resources while working. Participants were invited to the channel prior to the first day of the program to introduce themselves.

A separate private channel for mentors and facilitators was used for unobtrusive coordination during the program, timely administrative conversations and quick sharing of links and resources behind the scenes.

Project management

Basecamp was used by the facilitators to coordinate equipment prep, scheduling and other day-to-day requirements.

Presentations

Instructors shared Google Slides presentations to avoid the need to swap computers connected to the projector, and reduce turnover and downtime. All presenters emailed or used Slack to send their materials to the coordinator a day before the program started to avoid distracting delays and issues with coordinating slides.

Example projects

Games and immersive experiences were downloaded from Steam and the Oculus Store ahead of the first day of the program.

Classroom Setup

Desks were arranged in five groups of four during studio and production activities. During presentations, they were moved into rows or a U-shape to ensure all participants could see the presenter and screen.

Because the presentations included the display of code and a development environment, a bright projector was necessary for legibility. Working code was provided via Slack and by using a collaborative environment (e.g., Glitch or VS Code Live Share) to further support the visibility of content that may have been hard to follow on a distant screen.

Equipment

  • Four entry-level VR capable computers (AMD RX480 Graphics Card minimum) for participants with 23” monitors (minimum size) and WiFi connectivity
  • Four Oculus Rift headsets, four pairs of Oculus Touch controllers

Software

Computers were pre-installed with four free software applications:

Unity Hub Unity Hub was installed on each machine with the latest version Unity. Hub allows older versions of Unity to be installed concurrently, which can be useful for accessing older or beta versions required by certain tutorials or cutting-edge features. While our workshop focused on the use of the Oculus Rift headset, it is recommended to install the latest Android tools when installing Unity for compatibility with Oculus Quest headsets.

Oculus SDK for Unity The Oculus Integration Package for Unity is downloadable from the Oculus site, and is also available as a plugin in the Unity Asset Store. Having this pre-installed for participants allows a quick look at the results of their work in VR in minutes.

Depthkit DepthKit is used for manipulating depth levels and video masking post-capture. Though primarily used in the studio setups, additional settings can be tweaked prior to import into Unity for fine-tuned results.

Custom Unity Project Starter Template An example scene was prepared using the Depthkit plugin, sample volumetric video and custom scripts to provide participants with an accessible starting point for making their projects. Participants were able to drop their newly captured media into this scene and instantly have it interactive in VR as well as be provided some simple introductions to scripting. The project can be downloaded from the GitHub link above and opened in Unity as-is.

Optional

Though it wasn’t necessary for participants to bring their own laptops, many did, which allowed them to write, edit audio and video, use Adobe Creative Cloud and Blender for 3D models if they were interested.

Instruction on the use of these tools was beyond the scope of the intensive, but many of the participants were comfortable in various applications that aided in the creation of their projects.

Encouraging participants to bring in the tools they were comfortable with helped them focus on the unfamiliar aspects of production.

Studio Setup

Two different studios were created with three capture setups: Gamma Space’s windowless Gamma Gallery and TMAC’s Expansion Space multipurpose studio, with floor-to-ceiling windows on the east wall for natural light.

Three setups allowed instruction and creation to happen simultaneously as needed, while two different lighting options provided opportunities for both natural and studio lighting as required by the projects.

Equipment

  • Three computers running Windows that meet the Depthkit minimum recommended specifications found at https://docs.depthkit.tv/docs/equipment
  • Three screens for monitoring recording
  • Two sets of LED panel lights for lighting action, e.g., Dedo Felloni Bicolour panels (PDF). Any lights of similar size, softness and output are suitable.
  • Three Xbox Kinect 2 sensors. These are commonly available secondhand, but require the appropriate PC connector. Complete list of Depthkit-compatible cameras.
  • Wireless Internet connection to download updates and transfer smaller files.
  • Three SSD USB-C Toshiba Hard Drives. This allows the quick transfer of captured video to the VR machines without the network being a bottleneck.

Software

  • DepthKit software was used to capture volumetric video. It is free to use for short clips, while the Pro subscription offers longer recording time and more sophisticated built-in matting options for green screen work.
  • Dropbox for quickly transferring facilitator settings if required.

Program Activities

Day 1

Morning: XR Technology Overview including examples from industry members

In the DMG classroom, facilitators kicked off the program by presenting an overview of current technologies, including VR, AR, and mixed reality. This included associated costs with production, audience size for each platform and accessibility considerations. Commercial and experimental examples of XR work were presented by:

  • Faber and Lander, as independent producers and artists
  • Alex Bethke and Mark Cautillo from Sinking Ship Entertainment
  • Wayne Shipley from Stitch Media

Max Lander + Pyramid Attack - Night of the Living Dead VR

Sinking Ship - Dino Dana

Afternoon: Introduction to Volumetric Video

Participants were encouraged to try various demos provided by industry speakers, as well as games and experiences selected by the facilitators. While most of these demos are not available on a public platform, examples of volumetric video made with Depthkit can be found online:

During the afternoon session in the Gamma Gallery, Dr. Cindy Poremba from OCADU gave a hands-on demonstration of the principles of volumetric video capture and showcased works in VR that used the technology.

The afternoon closed with a demonstration of how volumetric video clips can easily be imported into Unity with the Depthkit plugin. Depthkit provides comprehensive documentation for setting up and capturing volumetric video, as well as importing that video into Unity using the Depthkit Unity Plugin: What is Volumetric Filmmaking?

Day 2

Morning: Narrative design workshop

In the DMG classroom, participants were provided examples and learned techniques for developing branching narratives and spatial storytelling by artist Kylie Caraway. Key points that were covered included:

  • Plot – linear or nonlinear story
  • Setting – time and place, the story world and its rules
  • Characters
  • P.O.V. – subjectivity and cinematography
  • Genre, mood, tone and themes
  • Style –  colors, visual aesthetics and audio
  • Experience – linear or nonlinear structure
  • Presence – embodiment, point of view (POV) and subjectivity
  • Agency – influence on the story and/or outcomes, mobility
  • Immersivity – points of interest, spatialized audio and responsivity

Participants were given an exercise to design a narrative and to plot that narrative out in three-dimensional interactive space called Storyboarding in Space. Storyboarding in Space is a tool for planning VR narratives, designed by Caraway, where artists take their main narrative elements and map them out in a 360 degree space relative to a central camera using post it notes. Each narrative element (sounds, objects of interest, characters) is given their own sticky notes (colour coordinated to other elements of the same type) and placed around the central camera. This map then provides the foundations for laying out a 3 dimensional scene in Unity.

Storyboarding in Space familiarizes participants with the idea that stories in VR happen spatially more than linearly. It was a particularly helpful framing as the participants were already familiar with many other storyboarding techniques. Participants then shared their ideas and formed groups around the ideas that resonated most with them. Four groups of five were formed.

Afternoon: Capture sessions

After lunch, the groups developed their ideas and developed a shot list. Those shot lists were then used as a guide in the studios for volumetric video capture for the rest of the afternoon.

Day 3

Morning: Spatial audio design and implementation

Robby Duguay and Matt Miller from Level Curve, a games and VR audio production studio, provided details about the workflow of putting spatial audio into Unity, along with an overview of binaural audio recording and its properties.

Afternoon: Introduction to Unity

In the afternoon, Lander gave an overview of Unity that included basic concepts (e.g., transform, colliders, Rigidbodies) and navigating the user interface. Next, participants were shown how the Oculus VR plugin makes interacting with basic Unity concepts greatly simplified in VR through the use of the included elements such as grab and gaze.

Setting up Oculus In Unity

  1. Navigate to the Asset Store then download and import the Oculus Integration Package.
  2. In the Project Settings > XR Settings - check VR supported and confirm Oculus is on the list.
  3. Delete the camera in the scene and navigate into the Oculus folder to find OVRCameraRig and drag into the scene.
  4. After pressing play your scene can now be seen through a connected Oculus Headset. In order for the Oculus Avatar system to work correctly (required to see the VR hands included in the integration package), a registered Oculus Avatar ID must be entered in Unity.

Steps for correctly setting up the Avatar SDK: Avatar Developer Guide - Rift For the purposes of this program, the instructor’s developer ID was used so that participants could access those features.

Finally, participants were introduced to scripting and showed how the scripts he prepared could be used as building blocks for more sophisticated interactions. These three pieces–basic Unity concepts, Oculus Integration for VR and intro to scripting–provided the participants with enough foundational knowledge to be able to navigate and manipulate the sample Unity project provided at the beginning of the program.

Sample scripts

Located in iO-Arts-Futures/ioSampleProject/Assets/

  • pointerTrigger.cs - When attached to the right hand, this script uses the trigger button to detect a playable Depthkit clip and plays it.
  • cycleClips.cs - When attached to a Depthkit clip gameobject, this script allows us to load in multiple different video clips that can be cycled through by pressing A on the Oculus Touch controller.
  • gazeToPlay.cs - When attached to the center-eye camera game object, this script functions similar to the pointerTrigger in that it uses raycasting to detect a playable Depthkit Clip and plays it when the player looks at it.
  • rotateToPlayer.cs - This script must be placed on the uppermost parent game object of a Depthkit clip and will keep the clip facing the player as they move through the environment.
  • pauseAtStart.cs - This script must be placed on each Depthkit clip in the scene that will be played with any of the above interaction scripts. It loads the correct clip thumbnail so the object displays correctly in the scene and then pauses it, in preparation for being played via the pointer or gaze triggers.

Day 4

All day: Production time

Day 4 was entirely dedicated to production. Mentors were on hand to answer questions and provide technical expertise.

Facilitators helped participants develop a process for framing their narrative against a single point of polished interaction, helping them to develop their themes and intent through the completion of an action by the user.

Day 5

Morning: Distribution

The morning of the final day was an opportunity to explore distribution of immersive technologies. Arthur Yeung from Albedo Informatics discussed AR content distribution. Faber and David Plant from Trinity Square Video discussed VR distribution, as well as corporate and arts funding opportunities.

Afternoon: Final production time, wrap-up and presentations

During the afternoon, participants were given the opportunity to wrap up their projects with mentor support.

At the end of the afternoon, participants presented their work to each group and members of the public visiting the building. Guest artist Karen Vanderborght demonstrated their Grey Matter AR project.

Conclusion

In order to increase adoption of immersive technology by artists, educational initiatives – especially informal and community-based programs – must address two key challenges:

Narrowing the scope of curriculum to the most appropriate tools and platforms for the participant cohort. When selecting technology, it’s critical to consider the resources available to the program early on, because teaching immersive technology is inherently experiential.

  • What expertise do potential instructors, mentors and other professionals providing support have? Ensuring these individuals have relevant, hands-on experience with the specific technology being taught, rather than only generalized knowledge of VR, AR and mixed reality.
  • Considering participants’ interests, skills and experience, what new technology will allow rapid creation? 360-degree video is a better choice for filmmakers and artists with linear storytelling experience, and Head-Mounted Display (HMD) based 3D VR is a natural step for experienced game developers.
  • What equipment, space and partnerships do are immediately accessible? What relationships can be leveraged to expand those resources? Partnering with a university lab outfitted with production-ready computers and Oculus Quest headsets makes delivering a VR program easier than an AR intensive.

Providing an equitable training framework and learning environment that addresses barriers to access. Explicitly acknowledging that the spaces and equipment that make experimenting with, creating for and pushing the boundaries of immersive tech are not readily available to all artists is necessary if we hope for more creative applications and approaches to VR, AR, and mixed reality. Consider:

  • What do typical program participants look like? If it is a homogenous group, why might that be?
  • If you are trying to include people at multiple intersections of marginalization, you need to address a spectrum of barriers including financial, practical and circumstantial factors. Common issues include: Unaffordable participation fees, distance to the workshop space and lack of mobility, lack of access to on-site childcare, access to free or affordable meals and inflexible format.

With a considered and focused approach to designing curricula, seeking practical support and taking steps to ensure barriers to participation are reduced as much as possible, community-based informal learning initiatives can significantly contribute to the number of Canadian artists and content producers who utilize immersive technology in their work.

Responding to COVID-19

At the time of this document’s writing, the COVID-19 crisis has restricted in-person learning. Resource sharing is also challenging, but advancements in technology have allowed for peer-based programs like this to continue, though there are additional logistical and material considerations.

Remote group learning facilitation. Rapid and widespread adoption of virtual meeting and screen sharing tools – such as Zoom, Google Hangouts and Slack – has reduced barriers to access to expert technical, business and creative support in group and one-on-one format. The ubiquitousness of these tools has increased literacy around remote mentorship and informal learning techniques.

Broader access to industry speakers. Because the “new normal” is remote working, industry experts may be more likely to agree to participate from different locations around the world. This has the added benefit of reducing program travel budgets.

Streamlined VR equipment. The Oculus Quest inside-out, all-in-one headset is better suited to rapid development now as it can connect via highspeed USB cable to a VR capable computer. This allows for rapid iteration when connected to a game engine without the need to make a build and deploy it to the headset after every change.

VR Kits. A VR-capable laptop, Oculus Quest, Kinect (or other DepthKit compatible camera) can be assembled into an easily shippable hardcase that could be distributed to participants anywhere for under $2,000 CAD.

While it is unknown how long the pandemic will affect learning environments, the infrastructure and resources for delivering this program can be adapted to a fully remote format. If accessibility and inclusion of a variety of cultural experiences, geographies and socioeconomic situations is desired, it is worth considering offering hybrid in-person/remote programs even once it is possible to reconvene in person.

However, it’s important to recognize other barriers may emerge – such as access to high-speed internet and unlimited bandwidth, and physical constraints on at-home work environments.

Credits

Editor

Jennie Robinson Faber

Translator

Marilyne Coleslaw

Instructors

  • Wayne Shipley, Stitch Media
  • Karen Chapman, Filmmaker
  • Alex Bethke, Sinking Ship Entertainment
  • Mark Cautillo, Sinking Ship Entertainment
  • Arthur Yeung, Albedo Informatics
  • David Plant, Trinity Square Video
  • Karen Vanderborght, Artist

Speakers

  • Henry Faber, Gamma Space
  • Max Lander, VR Developer/Photographer
  • Cindy Poremba, OCADU
  • Kylie Caraway, Artist
  • Robby Duguay, Level Curve
  • Matt Miller, Level Curve

Mentors

  • Richa Thomas
  • Kae Bagg

Special Thanks

  • OCADU game:play lab
  • Charles Street Video
  • Izzie Colpitts-Campbell
  • Lucie Lalumière
  • Grégoire Gagnon

Photographer

Lindsay Duncan

First published: 20 July 2020 · Last updated: 22 July 2020