We're in a major transition, but we just don't notice: Part 2

Google Glass, while a failed piece of consumer technology, was a very important step toward envisioning what a realy head-mounted display could look like.

Google Glass, while a failed piece of consumer technology, was a very important step toward envisioning what a realy head-mounted display could look like.

After playing video games for an hour, a notification pops up on your screen saying that it’s 6pm and time to make dinner. You also see a notification from you wife which says “what are we eating?” She's on her way back from work. Reluctantly, you put down your game controller, say goodbye to your online friends, and grab your glasses from the table and put them on.

You scurry into the kitchen. The notification from your wife is present on the screen of your glasses. You tap twice on the rim of your glasses which brings up your digital assistant. You say “show me yellow curry recipes.” A list of your favourite recipes from Epicurious appears before you. Your scroll through the list with your eyes. Once you’ve found one you like, you look at the bare wall of the kitchen and pin the recipe to it. You reach your hand out in front of you make a pinching gesture. The curry recipe gets larger and fills the whole wall. Following the recipe meticulously, you carefully prepare the dish.

You’re about to add the final ingredients to the pot when your wife gestures to the wall and annotates the recipe. Using her finger, she handwrites two additional ingredients below which you reluctantly add. You don’t like cilantro.

After a very satisfying dinner, you double tap your glasses again and you instruct your digital assistant “set the oven to self-clean mode and put Netflix on the living room TV. Start Stranger Things to where we left off last Wednesday evening.” The TV show starts playing where you left off…


In Part 1 of this article, I discussed many of the technical transitions taking place in the mobile (and desktop) industry - including the move to wireless connectivity, the consolidation of ports to USB-C, and broader usage of the ARM architecture. However, I deliberately left out one important trend because it deserves more attention. The rise of mixed reality or, as it's formerly known, virtual reality and augmented reality.

While I don't want to understate the evolutionary improvements of the graphic user interface (GUI), the mouse/keyboard and folder/icon system hasn’t changed much in thirty-years. While mobile devices introduced the multitouch UI, there hasn't been a huge jump in touch interaction for ten years. Both the desktop and mobile interfaces are two-dimensional. No matter how responsive the interaction, our documents are separated by glass and live in a realm we cannot enter. Mixed reality takes our interfaces into the real world and that technology is almost almost here.

Mixed reality is a loaded term. There used to be a distinct separation between virtual reality (VR - which presents us with an immersive, separate reality) and augmented reality (AR - which augments and enhances our current reality). Historically, VR and AR have only been possible using different head mounted displays. But it's likely future head mounted displays will be capable of both experiences.

So why is mixed reality a big deal? Think of your home computer or office workstation. If it's anything like mine, you probably have a desktop or laptop hooked up to a display with a mouse and keyboard. You're likely running Windows or macOS. Now, imagine you have your phone and a pair of mixed reality glasses. When you sit down at your desk, you dock your phone, turn on your glasses, and holographic monitor screens appear. It's likely you'll still have an input device (probably a keyboard) but other interactions could be done with eye movement or hand gestures.

The movie Minority Report has been one of the more prominent science fiction examples of what a mixed reality computer interface could look like.

The movie Minority Report has been one of the more prominent science fiction examples of what a mixed reality computer interface could look like.

But mixed reality extends beyond holograms. Voice interfaces will also be prevalent. Wireless headphones are the first step and they are a technology that truly feels like the future. Combine these with the smarter digital assistants that will inevitably come. More advanced dictation and voice commands will lead to a more hands-free experience.

Mixed reality isn't quite here yet. The current iteration of the mixed reality glasses currently takes the form of goggles like Microsoft’s HoloLens.

Microsoft Hololens

Microsoft Hololens

Voice interfaces are much further along and it's likely we'll be interacting with our computers like in the movie Her before we have Minority Report style displays.

Mixed reality lies on a spectrum, however. This capability could be integrated into our phone cameras - to adjust depth-of-field, insert objects, and make other photo edits which would have been unthinkable only ten-years ago. Physical feedback (vibration or haptic) also augments our reality. Apple's 3D Touch already allows iOS to respond to pressure. Vibration motors under the display give users physical feedback. One could even argue Nintendo pioneered physical feedback with the Nintendo 64 Rumble Pack.

Not only is mixed reality complex, it's been a long time coming. In fact, it's so incremental that we probably won't even notice it when it's here