Thursday, April 25, 2019

Teaching the World About Cider Through Augmented Reality

Ben Johnson, SVP, Digital Products
Design / Technology / Mobile

For most people, augmented reality Is still a buzz word and a gimmick (much in the same way that talking to a voice assistant was a bit of a gimmick just a few years ago). For some early adopters it's become an amazing tool to connect the physical and digital. Part of my role at Rightpoint is to ensure that we are on the bleeding edge of technology and to help partner with organizations that also want to build experiences at the forefront of what's possible. Augmented reality is a huge opportunity for both marketers and IT across all industries but especially CPG, Retail, Education, Healthcare, and Automotive We're seeing a massive shift in the adoption of this technology allowing companies to better connect the physical and digital worlds.

Last month, in partnership with Angry Orchard, we helped launch the Angry Orchard Cider + Food app. Yes, it is chock-full of augmented reality, but what's different about this augmented reality app is that it isn't a gimmick, it is teaching people about the amazing ways that you can pair Cider with food (much in the same way that wine pairs with food) all while transporting you to the orchards where Angry Orchard began.

So, why Augmented Reality and why now? Today, more than 850 million devices can experience augmented reality. This is primarily due to recent investments in hardware and software technology by Apple (namely with the launch of iOS 12 last year). If you are one of the hundreds of millions of people who purchased an iPhone in the last four years, you can experience the Cider and Food App.

A Behind the Scenes Look at How the App Works


When you first open the app it prompts you to scan a bottle. 60 times per second the Cider + Food app is scanning the environment looking for specific queues within the Angry Orchard labels (specifically we target a portion of the label that is common across all labels). But each style of cider is unique, so we need to know exactly which Angry Orchard label the app is looking at so that it can display the correct food pairings.


We like to think of our phones as smart phones, but out of the box they aren't always that smart, so we have to teach the app to correctly identify each label. We do this using a technique known as machine learning. As far as a computer is concerned, these labels are very similar and out in the real world different lighting conditions, camera motion, and other factors make differentiating between labels a challenge. So we used a combination of more than 1200 images to teach the app to differentiate between Cider labels on your device using a technology called CoreML (Core Machine Learning). Why is this important? Because in a low or no network scenario (e.g. a bar..), the app will work perfectly and doesn't require an internet connection.


In addition to identifying the label, we're also working behind the scenes to better understand the environment around the phone; how far away we are from the bottle, from the floor and from the walls around us. We do this using Apple's ARKit (augmented reality kit) to establish a coordinate system in three dimensional space. Because we know the size of an Angry Orchard bottle or can we can actually place Ryan Burk (head cidermaker of Angry Orchard) directly in front of the label. We also created some novel interaction patterns to connect the two dimensional and three dimensional worlds (swipe left and right on the pairing cards and you'll see the video of Ryan rotate around the bottle).


With all of this amazing technology as a foundation, we can now showcase the amazing recipes that Ryan and his team created to pair with each of their ciders. More recipes can be unlocked when you scan new bottles and everything is shareable. On top of that, we wanted to give drinkers an opportunity to transport themselves to the orchard in Walden NY where the magic really happens. We can do this because we established a coordinate system so when you step back from the bottle or turn your phone, you're able to experience an immersive 360 degree portal into the sights and sounds of the Walden orchard.

I'm really proud of the work our team put into this app and our partnership with Angry Orchard. It's exciting when companies are willing to push the boundaries of technology and we're thrilled to see the app out in the wild, teaching the world about how to pair food with cider.