Good Design is Why You’re Not Wearing AR Glasses

November 6, 2018

“Every first iteration of technology has a very high likelihood of being incredibly socially awkward.” Khoi Vinh

A few years from now, augmented reality may render most screens obsolete, as computing becomes integrated into the world around us. This presents a host of challenges for designers: What do we want an augmented reality world to look like?


KHOI: Welcome to Wireframe, from Adobe. A podcast about good design. And specifically about user experience design — how we shape technology to fit into our lives.

I’m Khoi Vinh, principal designer at Adobe.

And each week on this show, I, along with one of our producers, will explore one aspect of good design. This week, I’ve got Amy Standen here in the studio. Hi Amy…

AMY: Hi Khoi. So I want to introduce you to someone.

TAPE: Oh now I messed it all up. I’m an idiot.

This is Thad Starner and he is NOT an idiot. He’s a professor in the college of computing at the Georgia Institute of Technology.

AMY: Did the comforters all just fall down on you?

THAD: Yeah basically.

In order to have a quiet place for our interview, Thad built himself a fort out of pillows and a comforter — a fort that fell down as soon as we started talking. BUT! It’s precisely this flair for engineering that made me want to talk to Thad in the first place.

TAPE: Ok. I’m in good shape now.

For the last 15 years, Thad’s been running one of the longest experiments of its kind — on himself: an experiment in augmented reality.

Since 1993, Thad’s been wearing a face-mounted, wearable computer – mounted on a pair of glasses, and connected to a one handed keyboard – called a Twiddler – that he wears hanging down by his hip.

THAD: What’s great about the twiddler is it’s a one handed device. So I just put my hand at my side when I’m talking with the student and it doesn’t get in the way of the conversation.

Whenever Thad has a conversation, he’s taking notes the whole time on his twiddler. And then – next time he runs into you, he pulls up the notes about your last conversation – or about pretty much anything else, and reads those notes –– and this is the really crazy part — while he’s talking to you, on a tiny screen, at eye level – a head-up display –  affixed to his glasses.

THAD: When you talked to me, the display would be sitting there above my left eye. And you wouldn’t know if I was referring to it or not. So what’s great about having a head up display and a one handed keyboard – is that it could actually provide me information just in time, just when I needed it. And as a prof., that makes me seem much more intelligent than I am.

KHOI: So how does he access that information so quickly with this twiddler with this one handed input device?

AMY: Well he actually uses a voice recorder that’s built into this wearable computer is only recording his voice. So while he’s talking, the computer can hear the terms he’s using and automatically search through this massive text document – basically a record of everything he’s said or taken notes on. So just by saying something – and a couple of key strokes on his Twiddler, he can call things up. Then he can scan them on this very small screen he’s seeing just above his left eye and make sure he’s finding the thing he wants.  

And this might sound, to some people – maybe to you Khoi – kind of weird, or creepy, or just awkward I’m guessing. But to Thad, the calculation is pretty simple. Basically using these tools makes him better.

THAD: I’m a better professor, I’m a better student. I’m a better neighbor. I’m more social when I actually have that ability to pull up the context of what we last talked about… makes me appear a whole lot more socially graceful than I actually am.

AMY: Now Khoi, I know this might not be YOUR definition of socially graceful – using a computer to navigate social interactions. But Thad finds that by augmenting his reality this way — and he would know, he’s actually the guy who coined the term AUGMENTED REALITY — he can do a better job at being a person in the world.

Thad is responsible for advising 25 students each with a different research project and set of questions and concerns. So having this tiny screen that allows him to suddenly pull up information and look at his notes that’s a gesture of interest, a way of showing that shows he finds their work important.

KHOI: Ok I mean if it works for him. I do think even in an academic environment there are times when you just want to be human to human rather than having this intermediary or augmentation.

AMY: I asked Thad about this. Does anyone ever seem bothered by it?

THAD: Not really. Most people who are concerned about this stuff are more concerned about audio recording or video recording surreptitiously .. keyboard. You can incorporate the use of the machine so well, you become a cyborg. And so everyone just ascribes the interaction to you being better than you are.

Thad told me he’s gotten so good at using his wearable computer that when he talks, he actually has changed the way he structures a sentence, so he can start talking about something – look up his notes on that thing, while he’s in the beginning of the sentence and then finish his sentence with the piece of information he just looked up.

KHOI: That’s actually pretty interesting b/c in one way it’s the exact opposite of what we want to do as designers? Like we don’t want to force people to alter their behavior to accommodate machines and tech. And that’s kind of what he’s doing: he’s changing his natural form of making sentences so that he can give the tech time to provide the information he’s looking for. On the other hand – people do change their behaviors all the time in order to make better use of tech, let technology augment their abilities. So I think that’s really fascinating.

AMY: He says he pretty much has this down. Except there was this one time.. When he was taking a course from a fellow professor and she asked a question:

THAD: At the beginning of the course, we talked about deixis…

AMY: I didn’t know what that was either. Deixis is a term in linguistics for words like “he” or “it” that require context to understand.

THAD: What did we say was important? And me being a smart alec raised my hand, because I knew I could get to my notes on any course very quickly. So I raised my hand and said:  “well, at the beginning of the course, we said Deixis was important because …

AMY: The whole class waited… to hear Thad’s brilliant answer.

And then… nothing.

Thad’s reality augmenter, his social graces machine, had failed him.  

THAD: Ah. Ah.. I’ll have to get back to you on that.” I’d gone to the wrong mode on my text editor!

AMY: Ba dum bum. Wa waaaaah.

KHOI: Ha ha

AMY: So It’s easy, I know, to dismiss this – as one researcher’s wacky tech experiment.

But Thad’s work has been incredibly influential. In 1998, Thad was demonstrating an early version of this system at a conference when he was approached by two young technies by the names of Sergey Brinn and Larry Page – of course the founders of Google. Eventually, Thad joined Google to develop Google Glass, which was released to much fanfare in in 2012.  

AMY: I’m sure you remember that, Khoi.

KHOI: Of course — there was that 12 page Vogue spread with all the models wearing GG?

AMY: Exactly. And it was pretty much downhill from there.

KHOI: It was heavily hyped but it was almost dead on arrival. People found it almost ludicrous how conspicuous it was to wear google glass and it wasn’t even that bulky a piece of hardware it was just awkward enough and that created a social stigma right away and so it didn’t get very far.

So google glass was a flop. Google shuttered production on Glass in 2015. The website went dark. But that is not where this story ends.

Since GG came out – pretty much every major tech company has been working on some version of an augmented reality headset. Microsoft has the HoloLens; SnapChat had Spectacles. Bose has a version – Apple is apparently working up to its own release of an AR headset.

KHOI: Hey Amy, maybe we can take a moment here to tell the difference between AR and VR for listeners who may not be totally clear.

AMY: Yeah that’s a good idea. Go for it.

KHOI: Well they’re spelled differently.

AMY: Thank you.

KHOI: Ha ha. Yeah so virtual reality takes you out of the world around you and takes you somewhere else well AR is a kind of layer on top of what you see already. This is why google glass was such a big shift because you’re wearing them out in the real world and only you have this additional information..

AMY: Just like only Thad is seeing this little square out of the left field of his peripheral vision.

KHOI: Exactly.

AMY: So — we can look at Thad’s set up, and the Google Glass that came after — as some of the earliest iterations of augmented reality. And augmented reality, as a technology, to me feels pretty, I dunno kind of niche, or geeky — but a lot of designers and other people in tech consider totally inevitable. That in a decade or so, augmented reality is going to be the primary way that regular people interact with the internet, or the digital world generally.

Khoi,  you’re nodding… sounds like you agree with them?

KHOI: I think AR is inevitable because we as a society we are consistently hungrier and hungrier to access the information available to us on screens and with augmented reality you don’t have to pull out your screen anymore you don’t have to turn to your screen on your desk anymore it’s just sort of just the way you see the world.

KHOI: Generations that grew up with smartphones in their cribs they’re going to see this as antiquated and old fashioned before too long. it’s way too much clicking, way too wide of a gap between what they need and the technology that’s going to give it to them.

Yup, and one of your colleagues at Adobe is thinking a lot about this, Khoi.

SILKA: What we’ve lived with so much for the last few decades is screens and keyboards and mouses that do really intermediate the experience.

This is Silka Miesniecks, head of emerging design at Adobe.

SILKA: It puts one layer between you and what you’re trying to create. AR can bring it back to being more natural – where it takes away the screen and allows you to interact with your hands as you would, normally.

AMY: I don’t know about you, Khoi, but to me this talk about AR gets pretty abstract and academic pretty quickly. So I asked…So I asked Silka to give me an example of just how AR could be useful in just one context. And the example she gave me is of a classroom.

SILKA: You can have the whole solar system in your room, and walk from one planet to another and understand the actual distance things are from each other. You could go to things in the past – you know, Napoleon time and watch events in real scale, or scaled down so you could see them over a large piece of land.

AMY: Listening to Silka, made me wonder why GG didn’t take off. I mean, i know that the technology was – from an AR standpoint – pretty primitive. But if Augmented reality is so inevitable — why didn’t more people get excited about Google Glass?

Khoi, did you ever think about buying a pair?

KHOI: No, not for a minute.

AMY: Yeah I heard that from several designers. One designer I spoke to is a woman named Gillian Hayes — she was a graduate student of Thad Starner back in the 1990s, and has been researching wearables for over a decade.

I thought if anyone’s going to be wearing GG other than Thad Starner, it will be Gillian. And yet…

GILLIAN: I’ve not done a whole lot of glass wearing out in public — I’m not sure I would, to be honest. From a modern design standpoint, they were a mess. But this incredible tool and platform upon which other designers could come in and create new things. It felt very much to me at the time that that’s what Google was going for. Let’s create a very open platform and see what people will do with it.

And from what I understand that did happen. After Glass flopped, designers and technologists continued to play with the technology. And there are places now where EVERYONE is wearing Glass. These aren’t early adopters or gadget fanatics, they’re just people doing their jobs.  

For instance there’s a company called AGCO in Atlanta where factory workers wear Glass to show them diagrams for how parts fit together, or to scan a serial number. Glass is just part of the workflow.

But what we didn’t see is Glass becoming part of mainstream culture. Why? Because Google never came up with something that all of us would want to DO with Glass.

KHOI: Yeah, they hadn’t found a use case for the real world. What is this technology going to actually be used for? How are real people actually going to get some value of it in everyday life – whether in business, or out in their personal lives and the world at large.

AMY: I’m thinking of examples where a new piece of tech was introduced. It was really dorky looking, everybody made fun of it. But it was so useful that it kind of pushed through and now we don’t think of it anymore. And the one that comes to my mind is the bluetooth headset. Remember the Jawbone?

KHOI: Yeah.

AMY: Pretty dorky looking right?

KHOI: Yeah, actually this is not just the jawbone, or audio headsets. The first laptop weighed 15 pounds. It was like carrying a suitcase. Every first iteration of technology has a very high likelihood of being incredibly socially awkward.

AMY: Until it finds as good use case, right? Well after the break we’ll meet designers who are working toward exactly that goal: the use case that will finally bring AR from awkward… to indispensable.

KHOI: We’ll be right back


KHOI: Welcome back to Wireframe. I’m Khoi Vihn and I’m here in the studio with producer Amy Standen.

AMY: Hi Khoi.  So a few weeks ago, I met with a woman named Estella Tse (SEE).

ESTELLA: My name is Estella Tse and I’m a virtual reality and augmented reality artist.

As an artist, Estella’s canvas is the world around her. She makes artwork that — with the help of an iPad or a pair of AR goggles — can be superimposed onto whatever space you’re in.

ESTELLA: I like the idea of blending realities. How do I make something that half exists in the real world and half in the digital world.

Basically, Estella’s job is:

ESTELLA: Like OK, this thing exists, now what?

Estella is often brought in by companies — including your shop, Khoi, Adobe — to push at the boundaries of a new piece of technology — in this case, augmented reality.

ESTELLA: As artists we’re trained to express and push the boundaries… so I get brought on to experiment and play and try to make a new form of expression…

Estella and I took a short walk in Oakland. And as we walked, you could hear the gears turning inside her head…

ESTELLA: Yeah, imagine if I had to paint a mural on that building there —

Estella is pointing at a warehouse across the street. She told me that, traditionally, if you were an artist or a graphic designer planning, say, a mural, you might take a photo of the building, bring it into Photoshop, and essentially design onto that 2D photo.

But what if, instead of looking at a tiny version of the building on your laptop screen, you were…

ESTELLA: Coming here to this physical space, and then like using an AR tool to view it and feel it? And how big that feels? When you see something in a 2D screen, you lack that feeling aspect of it. But when you work in immersive art, when you’re here and you’re standing here and you see where you are in relation to this building. You can feel the air, hear the cars going around us, smell the things around us. It’s a different experience.

And to a certain extent, this is already happening. Architects are starting to use AR to see what a building would look and feel like in space – how it fits into the block, what it’s like to walk past it, even into it.

But what really gets Estella going is thinking about what artists could do with this new medium. She  points to a nearby industrial building, covered in graffiti.

ESTELLA: I want to see what street art is gonna do with AR.

A graffiti artist working in augmented reality? Could cover an entire block, a whole neighborhood.

ESTELLA: I imagine they’ve tagged up the entire oakland street-scape where there’s different types of stuff we don’t even know is there? But if we had the right lens to view it then you know, we might see some sort of expression of some sort of statement about the gentrification happening here, or whatnot.

AMY: Secret messages.

ESTELLA: Yeah, secret messages!

Estella’s experimenting with all of these ideas – using a tool called Aero. Khoi, tell us about Aero.

KHOI: It’s a new tool, just released in beta, coming out for wide distribution in a year. Aero lets artists and designers make AR experiences using a suite of tools they’re used to. So, a designer could, for example, design something in Photoshop, or [another Adobe tool?]  and then bring it into Aero, to create a new, augmented reality experience.

And what’s big about this [maybe also mention ARKit?]  is that until now, if you wanted to make something in AR, you basically had to be a coder, or have the ability to design in complicated video game software. And that’s a problem because most designers aren’t coders. Which means a lot of us haven’t had the ability to play with AR, to see what it can do.

AMY: Khoi, One thing I heard over and over again is that designers are reallly critical, especially at this early stage of AR. Because they’re going to have to figure out not just how to make experiences that are beautiful, or easy to use. They’re also going to be the ones who determine what an AR world looks like.

I spoke with a designer named Paul Reynolds. He co-founded a company called Torch, which makes an app that regular people can use to make augmented reality experiences.

And Paul is very gung ho about AR, as you’d expect. But he also has some concerns…

PAUL: Especially if that’s driven by a brand or it’s an advertising related thing what’s going to prevent all of these things from screaming at us on the street corner, to tap on them and look at them.

AMY: Paul’s worst nightmare — and I think mine too, Khoi — is a scenario in which advertisers come to dominate augmented reality — so that, say, every car is an ad for Geico, or every streetlamp is telling you to change your cell phone plan. AR could be a sea of billboards.

PAUL: And to me that is the the potential route that this could go. If we carry over how we do mobile applications and Web applications today — they’re freemium based or advertising driven…

AMY: Because you’d be walking down the street and literally everything you look at is like look at me press here! buy this!

PAUL: We are we have called it the Tokyo or Times Square effect. We don’t necessarily want that all the time.

KHOI: He’s totally spot on with advertising. There’s a long history of new technologies being propelled to mass acceptance by advertising. If that’s gonna happen in AR where advertising could potentially dominate everything you see? That’s really uncharted territory. There’s a pandora’s box of questions and issues here that we’re gonna need to root through over the next several years.

AMY: And to Paul and other people I talked to – we’re at a crucial moment where we — by which he means designers — get to decide what the AR future looks like.

PAUL: The nice thing is these platforms are so new, if we decide as a society or as enthusiasts of these new platforms that we don’t want that, now it our opportunity to establish that patten before it’s too late.

I ran this by Silka Miesnicks, at Adobe — this question of this dystopian AR future and how worried we should be. And she had what I think is a more sort of philosophical take on it – that had to do what how we think about technology in general.

Silka says, this is what humans do. We’re nostalgic for the past, and we fear what’s in front ofbus.

SILKA: I guess I do too have the fear of the unknown. I don’t think that’s… unusual. And I think that’s actually healthy.

AMY: That fear – ideally – could make designers slow down a bit and be thoughtful about the new world they’re helping to build.

KHOI: That’s right. Design is becoming less and less of a veneer that’s put onto certain parts of the world – and more and more integrated, or enmeshed with everything else in the world. The decisions that designers make, the things we create have very real world consequences that are potentially at odds, at least in the short term with what a company wants to achieve this fiscal quarter.

AMY: So what’s your advice for a designer who’s confronting not just AR .. but other new technologies that have the potential to really resculpt how we operate in the world?

KHOI: I think the number one thing you do is you incorporate into your design process a very thoughtful and expansive view of how your design is going to impact the world. So it’s not just about making sure an ad shows up in your AR glasses. It’s really about how is this going to affect this person’s long term well being, this person’s perspective on the world. And try to anticipate the otherwise unanticipated consequences of the designs that you’re creating.

AMY: Silka agrees with that.  She told me this VERY DISCRETE AND CONCRETE ADVICE.

SILKA: That’s why I’m working in this industry, to make sure that we build a better future for everybody. It’s our responsibility to make that future better. And we can do that. And I see a lot of great things happening already.

AMY: Khoi, you told me earlier that augmented reality is inevitable. As a designer, what does that mean for you?

KHOI: This technology is inevitable.The best way to make sure it produces good outcomes, is for designers to be involved, mindful, thoughtful of the impact of their work, and to be vigilant about always looking out for the users.