The Moonshot Tech Challenge of Our Era is Context

Why It Matters and How to Study It

The world’s leading technology companies are increasingly aware that grappling with context is an urgent, necessary task––a personal computing problem that the next generation may look back upon as the social-technological moonshot of our age. ReD recently published a paper on how to study context together with our friends at Facebook Reality Labs, based on work we have done together. It led us to some reflections on why context matters, and how a concept that’s in many ways so abstract can be studied in everyday life…

In 2008, researchers at the University of Utah made an interesting discovery: they learned that drivers conducting a cell phone call were far more distracted than those conversing with a passenger in their car. Because the researchers controlled for other variables––by using hand-free headsets, for instance––they determined that something inherent to conversations with someone who is remote caused the distraction.

The researchers realized: if you’re having a conversation over a cell phone, the person on the other end knows nothing of your surroundings (whether a motorcycle has just zoomed by, for instance, or a heavy rain has just begun to fall). But a friend in the passenger seat has all that crucial context––and moderates the conversation accordingly. “When you’re in the same physical environment, you tend to adjust your discussions to the difficulty of driving,” noted the study’s lead author, David Strayer. “If driving becomes difficult, they stop talking or they point out hazards.’’

As our technologies become more and more “smart,” the goal is increasingly to build a device that behaves like that context-aware passenger in the car, rather than the context-oblivious cell phone caller. It’s a problem technologists have been grappling with for years, and well over a decade after that car conversation experiment, progress has been halting, even for the most innovative tech companies.

Recently, a ReD researcher whispered a query to her Amazon Alexa device. A context-sensitive human would typically respond with a whisper (perhaps even a whispered, “Why are we whispering?”). Instead, Alexa boomed: “IT SOUNDS LIKE YOU’RE WHISPERING! IF YOU’D LIKE ME TO WHISPER, SAY ‘WHISPER MODE’!” It is impressive that Alexa can whisper, without a doubt. Wouldn’t it be even more impressive if Alexa were attuned enough to context to know exactly when to whisper?

Like the fish who asks, “What’s water?,” we are both surrounded by context and oblivious to it. We use it and navigate it automatically, without thinking. By “context” we mean –– well, what do we mean, exactly? It turns out that if we are to become conscious of context and to make it helpful to product designers and engineers, we need to unpack the word.

One reason we choose to focus on the word context here, and in our work, is because of the richness and capacity of a word with connotations at once physical (context in the sense of environment), social (context in the sense of milieu), and intangible (context in the sense of atmosphere or vibe). Context can also be psychological, historical, political, or temporal, all at once. 

The richness of context is what makes it exciting to study. It’s also what makes it daunting.

Nevertheless, the world’s leading technology companies are increasingly aware that grappling with context is an urgent, necessary task––a personal computing problem that the next generation may look back upon as the social-technological moonshot of our age.

Why is studying context an imperative for technologists? Because the history of computing is in a sense a history of miniaturization. The first computers were the sizes of buildings: their context was cumbersome, but uniform. Decades later came personal computing, which put smaller devices in new, more varied contexts: cubicles, offices, homes. Then came smartphones, which meant that computing could fit in purses and pockets, and could happened anywhere. Now comes a revolution of smart devices––smart cameras, smart glasses, smart speakers––which means that computing happens everywhere and in every context you find yourself in.

Let’s take up the example of smart glasses. Many recall the Google Glass experiment that failed among the broader public; marketed to consumers in 2013, the product was swiftly discontinued in 2015 (it’s still available for specialized business uses). And in a sense, it’s no wonder. For a designer who wants to make a product that is sensitive to context, what could be a bigger challenge than a device you wear, that goes with you everywhere, and that rests right in your field of vision?

Though Google Glass fell short of its ambitions, many technologists and analysts agree that it was simply an idea ahead of its time, and that a new generation of what we might call smarter glasses may plausibly replace our smart phones within the next decade (though they might in fact turn out to be rings, earpieces, or other wearable devices). These glasses (or whatever) will be smarter because they are likely to contain artificial intelligence that is closely attuned to and learning from––you guessed it––context.

ReD Associates recently teamed up with Facebook Reality Labs, which is charged with developing the next generation of personal computing technologies. Our team asked: in order for such a device to moderate itself sensitively––to offer only helpful, desired content, at just the right moments––what would such a device truly need to know?

The answer, we knew, was context. But how, in practice, should researchers go about studying context, to inform how a device’s machine learning should “make sense of” (or parse) context and design for helpful interventions?

As we learned on our joint project, it boils down to a few strategies (which we recently detailed in our thoroughly titled academic paper, “Hybrid Methodology: Combining Ethnography, Cognitive Science, and Machine Learning to Inform the Development of Context-Aware Personal Computing and Assistive Technology”––presented at a conference on ethnography in industry).

First, we learned, you need to study context with an interdisciplinary team. A designer or architect may have much to say about the lived experience of the physical environment; a cognitive scientist will have a take on mental effort; an anthropologist will have strong opinions on social dynamics. And when researching context to inform technological product development, it is essential, too, to have computer scientists at hand to help ensure that the data that is collected is not only open-ended enough to be rich, but structured enough to feed into, for instance, a machine learning workflow. Each discipline brings its own perspective, and all are needed to ultimately design a product that touches on the physical, social, cognitive, and more.

Second, you need to break down context to study its component parts systematically and in interaction – for example, looking at environmental context, social context, temporal context, psychological context, and how each interacts with the other. We chose a standard and clearly defined everyday activity to closely examine these different components of context: cooking in the home. Thus, exploring environmental context might include questions like: Did the individual recently move into this apartment, or has she lived there awhile? How familiar is she with the kitchen? Points of observation about social context might include: Is the individual’s dog present, and whose turn is it to feed her if she whines? Studying temporal context might mean asking: What did the individual do previously today? This year? This lifetime? And how does that affect how she is approaching cooking at this very moment?  And understanding psychological context might mean observing: Is this individual in a good mood or bad? And what is her affinity for cooking to begin with? Ultimately, the researcher’s job is to interpret how all of these aspects of context relate to one another and help shape what might be a relevant intervention a device might make in the moment.  

Third, you need a rich and wide-ranging variety of data types that helps capture the many dimensions of context that we normally take for granted. Our team gathered data from kitchen floorplans, mental effort monitoring, video recordings from head-mounted cameras, and even the “thick description” of anthropologists’  fieldnotes of observed behaviors. The richness and variety of the resulting analyses left our team poised to develop original work products that informed early user experience design, further research agendas for cognitive science teams, and the early software infrastructure for assistive technologies.

Finally, in order to thoroughly study context in a way that is translatable to a machine, it was essential to create abstractions that grouped important elements of context together. The idea of frameworks or abstractions is of course already familiar to any social scientist. But when working with a technology company on the early development of next-gen smart devices, it was essential to find the right level of abstraction that could feed into machine learning processes. Without abstractions, machine learning models would have to cope with an infinite number of categories, with one data point each. Clustering data into categories eliminates that absurdity; broadly, for both machines and the humans working with them, it could potentially increase efficiency and utility. In our setting, the most useful level of abstraction would allow a machine learning model to reduce the inherent complexity of context and to hone in on what is most relevant for the human in the given moment.

The puzzle of solving for context is one for technologists of all stripes. Whether your device or service is ubiquitous and wearable, or stationary and used infrequently, consumers in the near future will be making choices around which devices and services are most sensitively attuned to context.

People are increasingly aware that companies are gathering data in order to better sell to them, but they’d much prefer for that data to be used to better serve them. For example, in a recent ReD Associates study for a financial services company, we encountered people who were deeply frustrated that, after taking on a mortgage, salespeople from the company were rapidly following up to offer more loans. This was a marketer’s context-sensitivity––the purse strings had been loosened, after all. But it was hardly the kind of context-sensitivity that increased people’s long-term affinity for a financial institution that was supposed to have their best interests at heart. We had people tell us: Shouldn’t my bank know that after I’ve taken out a giant loan, I don’t want to take out another?

Parsing context and responding appropriately is the technological equivalent of a tightrope act. But it’s also becoming necessary as people seem increasingly likely to choose the brands, devices, and services that get context right: that behave like a trusted friend in the passenger seat.

By David Zax & Maria Cury