Social Agents in Homes

How should we expect intelligent social agents (e.g. Alexa, Google Home) to behave in multi-user, social environments like the home?

Given current advancements in technology, AI will inevitably have significant potential to augment advanced social interactions with humans. Growing popularity in home agents (40+ million devices in homes in 2019) warrants numerous questions these interactions in the home and what agents should and shouldn't be able to do.

• Should a social agent respond to a family member who wants to know another household member's location?
• If the agent notices a household member cheated in a board game, or if someone is drinking underage in the house, should it tell?
• Should an agent capable of sensing emotion notify family members of odd behaviors?
• Who has permission to ask the agent to do various tasks?

These types of questions are becoming increasingly important as social agents become ubiquitous in the home.

To begin our research, each team member individually brainstormed potential scenarios—futures—of social interactions in the home. We decided most scenarios should focus on multi-user scenarios involving multiple family members to capture these interactions. We used this to better understand themes before making formal scenarios and concretely defining our users.

After identifying themes, we worked in small groups to identify potential scenarios. We generated a list of ideas (shown above) by theme and labeled them based on the context of their interactions (i.e. 2 parents, 1 child in a room), which we deemed important in ensuring good social multi-user interactions.

We split them up amongst ourselves to create more detailed scenarios between 3-5 sentences that would eventually develop into storyboards for interviews. Over 1-2 weeks, we iteratively refined our scenarios and eliminated unnecessary scenarios—ones that either captured a similar interaction or did not provide enough value to the set. Two members of our team developed our personas.

Narrowing down our ideas, we selected close to 30 scenarios to sketch out into storyboards. In our scenarios, it is important to capture the people involved, where it takes place, the activity, trigger, response, and resolution. These had to be reflected in our boards to help illustrate a story of an interaction not yet possible with today’s technology, but that which will be in the future.

We critiqued each other's storyboards, eliminated and revised, and then did the same through pilot testing on friends and family in mock interviews over several iterations.

We noticed a major need to showcase more positive storyboards that got in the interactions. When we presented randomized sets in mock interviews that weighed the negative interviews more, subjects would focus on the “how” or the result of the interaction instead of the interaction itself. Eventually, we narrowed our storyboards down to 23.

Speed Dating in research consists of rapid comparisons between design opportunities and speculative futures by presenting low-fidelity concepts in a relatively structured and fast approach, similar to romantic speed dating. Providing participants with exposure to numerous concepts allows for better explanation into what they like and dislike across a wide set of futures and help us better understand user needs in an ambiguous space as a result.

Each interview lasted 1.5 hours, primarily at families’ homes in Pittsburgh, while some were on-campus. In each scenario, 2-3 researchers would conduct interviews to help guide participants through their thoughts and feelings. Scenarios were presented randomly using a Latin Square to help control for bias. Over the course of the interviews, we removed four and added two scenarios to reflect our feedback on whether scenarios provided value or if we were missing a theme.

I participated in running six out of 18 interviews.

We transcribed each interview and marked important quotes from participants. We cut out these quotes to include in another affinity diagramming to identify themes from the qualitative data. From here, we referenced and related themes to derive groupings and narratives shared between multiple families.

While the project is still ongoing until submission to CHI in Fall 2019, we’ve learned a lot about what families want from social agents in the home, and have more questions to follow. Deriving these insights required looking past scenario-specific cases and pulling out abstract ideas and themes from the quotes. Some of our findings are listed below.

1. Families desire social agents to understand house rules
   Families tend to think the agent should be calibrated to individual households. Most families think parents should have primary control of the agent, but this hierarchy was often debated.
2. Many families wouldn't trust the agent to be proactive in making decisions for them
   Most families want the agent to confirm actions before doing them. People don't trust the agent would be designed with enough transparency enough to be entirely trustworthy.
3. Many families believe an agent could interfere with their children growing up
   Parents think an agent's surveillance would inhibit children from learning valuable lessons on their own. Many parents expressed the desire for their children to have room to experiment and make mistakes, and are worried children may additionally become reliant on agents in the home.
4. Many families think the advanced social agent would remove some autonomy from household members
   Families tend to think the agent should be calibrated to individual households. Most families think parents should have primary control of the agent, but this hierarchy was often debated.
5. Many families don't believe social agents will understand the intimate relationships between family members
   The level of surveillance granted to agents will make more things "provable" when family members can be verified or monitored. This is concerning to multiple families. Many see it as an "extra set of eyes."