VR Sketchnoting: Bringing Conversations to (Virtual) Life

Update: I finally got around to uploading Dan Rubin's closing keynote. Follow the links at the bottom of the page to browse all four keynotes from the conference on Google Poly.

As many readers will know, one of my roles here at Sketch Group is as a graphic recorder. I’m one of a team of illustrators whose job it is to take visual notes at events—conferences, workshops, panel discussions, and so on. It’s stimulating, rewarding, creative work, and we're regularly humbled by just how fundamentally this work can change the dynamic of an event.

Attendees are fascinated by watching a large canvas come to life, and event organisers love having something different to engage delegates at their event—not to mention the value of having a large, colourful, visual summary of the content to photograph and share after the event is over. 

Usually our canvas is a large sheet of paper, positioned to the side of the main stage, so that attendees can watch it come to life throughout the event. Graphic recorders are required to listen, synthesise, and capture the content that they hear.
 
Last week, the organisers of Web Directions Summit 2017 engaged me to graphically record the keynote sessions at their conference. However, this time I wasn’t drawing on paper—in fact, I didn’t even have a canvas to draw on, and there were no markers in my tool belt.
 
On this occasion, I was drawing in 3D space, using an HTC Vive virtual reality headset. In what I believe might just be a world-first (I don’t know of anyone else using the tool in this way before, so I’m claiming it!) I sketched the presentations in virtual space, in real-time, using a tool called Google Tilt Brush.

​The idea

The idea of taking VR Sketchnotes (that’s what I’m calling them, unless a better name comes along; sculpt-noting, anyone?) came to me earlier this year, when I attended the International Forum of Visual Practitioners conference, in Decatur, Georgia. I was attending as a delegate (the only one in attendance from Australia) and saw a demonstration in the Exhibitor’s Hall of a 3D mind map that had been created by the wonderful folks at Lizard Brain Solutions. I was curious to learn more about how it had been created, and exchanged a series of emails with the team after returning to Australia. When I learned that the mind map had in fact not been created on the fly (it comprised a series of text nodes that had been imported from Adobe Illustrator, and then positioned afterwards) I realised there was potential to explore how sketching in VR in a real-time environment could add a new element to the experience of attending an event, much like graphic recording does already.
 
When I saw the amazing promotional video for Tilt Brush, the dots were connected; I was determined to ensure that the device I purchased would be one that ran this app. A week later, with a sizeable hole in my pocket, I'd purchased my headset and set it up at home. My foray into the world of VR had begun.

​​The challenges

​As you can imagine, there were more than a few hurdles to overcome in order to sketch a conference presentation in 3D. Here are a few that come to mind:

• The cost: VR headsets continue to come down in price, but a decent setup is still not cheap. The headset and controllers will set you back over $1,000—plus you’ll need a decent Windows PC to power it—one with plenty of RAM, CPU grunt, and a high-end graphics card (Apple is playing catchup in this space, but no doubt they'll have a similar offering available soon).
• The logistics: The conference I sketched was in Sydney, so I had to travel by plane with all of my gear. While I regularly travel with a case full of markers and a plan carrier containing a big roll of paper, on this occasion I had to carry the headset and its various bits and pieces, the computer, a couple of tripods to hold the sensors, and make sure I had enough cables and adapters to ensure things would run smoothly at the venue. And I had to make sure I packed it all up afterwards and didn’t leave anything behind.
• No visual: Because I was donning the headset, I couldn’t see anything outside of my virtual canvas. That means I had no visual on the speaker, the slide deck they were talking to, or the audience. I had to rely solely on listening. This meant that several visual gags went completely over my head. But overall, listening is the main channel through which I visualise content, so it didn’t feel like a huge handicap.
• Attendee expectations: I’ve graphically recorded a bunch of conferences for the Web Directions team in the past; dare I say, some of the regular attendees have come to expect my large, colourful sketches as part of the conference experience. So it wasn’t a huge surprise that I had a few skeptics in the audience ask “What’s wrong with good old markers?” and comment “You look silly in that thing.” I ignored them and chose to celebrate the fact that this was a freaking world-first here people. There will always be nay-sayers when you push the boundaries.

• Sharing: The biggest frustration is that the sketches I created exist only in the digital realm—and the monitor showing what I was creating could only display one sketch at a time. That said, I did my best to download, convert, record, and publish still images, videos, animated GIFs, and 3D files that could be browsed and shared online (in a web browser or with a VR headset) immediately after I’d completed my sketches. This blog post is part of that process—while there weren’t any big graphic murals stuck to the wall like in past conferences, the long tail of exploring this content online is larger, due to the multiple formats through which this content can be repurposed.
• Lack of conventions: This was the most challenging aspect of the first sketch I created. While I’d practiced at home on a few TED talks, I hadn’t really landed on any conventions for how to lay out a 45-minute sketch. I feel like the opening keynote that I sketched suffered as a result of the fact that I was still figuring this out—the text is less legible because there is other text visible through it, and the lighting makes for a less inspiring mood. The second, third, and fourth sketches that I did were markedly improved, from a layout and a lighting perspective, and I’m quite proud of them. I’ll elaborate on some tips on this stuff in a future post.

The result

Due to my travel commitments at the moment, I haven’t had a chance to process all of the sketches from the conference properly yet. However, I have uploaded some stills, videos, and 3D sketches that you can browse on your computer, your phone, or with your own VR headset, if you have one.
 
I’ll update this post and fill in the blanks next week after I’ve had a chance to process all of the files from the sketches. In the meantime, though here are a few samples for you to explore.

​The VRdict ...

​VR Sketchnoting is a new and innovative way to visualise content in real-time. Capturing content in virtual space has the potential to add to an attendee's experience of an event in new and unique ways. In addition to the increased engagement of watching a VR sketchnote be created in the moment, having 3D digital sketches of content can empower attendees to explore and reflect on the content after an event in a number of ways, via a range of digital channels. This digital collateral can also be repurposed as promotional content by event organisers and speakers after the event.

Creating a VR sketchnote at an event comes with its challenges; there are some logistics involved that require one to be organised, and the concept may be met with resistance by some folks, who may consider it too different or new. However, the upsides that I've listed above suggest that this is an area worthy of further exploration, even if it's not yet clear exactly what role it will play and what form it will take. Meanwhile, the goal posts continue to move as technology evolves. We'll keep you updated as we continue to explore this emerging field. 

Browse all four of the Web Directions Summit 2017 sketches on Google Poly:

• Chris Messina: Lessons from the Death of the Personal Computer
• Amélie Lamont: Don't Kill Them Softly: Building a Culture of Fearless Feedback
• Genevieve Bell: Artificial Intelligence: Making a Human Connection
• Dan Rubin: A Life on the Web