Skip Navigation or Skip to Content
My signatureAn animated signature of me signing my first name

Social Agents in Homes

Spring 2019
Alexa Echo device

Understanding the future role(s) and desired behavior of social agents in multi-user homes.


Research Assistant


UX Research


Ph.D. Candidate, 3 masters students, 3 undergraduate students; advised by John Zimmerman and Jodi Forlizzi


Speed Dating, Affinity Diagramming, Storyboarding, Interviews, Adobe XD


How should we expect intelligent social agents (e.g. Alexa, Google Home) to behave in multi-user, social environments like the home?

Given current advancements in technology, AI will inevitably have significant potential to augment advanced social interactions with humans. Growing popularity in home agents (40+ million devices in homes in 2019) warrants numerous questions these interactions in the home and what agents should and shouldn't be able to do.

  • Should a social agent respond to a family member who wants to know another household member's location?
  • If the agent notices a household member cheated in a board game, or if someone is drinking underage in the house, should it tell?
  • Should an agent capable of sensing emotion notify family members of odd behaviors?
  • Who has permission to ask the agent to do various tasks?

These types of questions are becoming increasingly important as social agents become ubiquitous in the home.


We focused on users consisting of families with two adults and at least one child 12 or older in group interviews to understand:

  1. the future role(s) of social agents in multi-user homes.
  2. how to design for social agents in the home.

Generating Themes

To begin our research, each team member individually brainstormed potential scenarios—futures—of social interactions in the home. We decided most scenarios should focus on multi-user scenarios involving multiple family members to capture these interactions. We used this to better understand themes before making formal scenarios and concretely defining our users.

google doc screenshot of brainstorm
Individually brainstorming scenarios

Affinity Diagramming

We used Affinity Diagramming to group scenarios into themes. Five umbrella themes emerged from this exercise:

  1. Proactivity
    An agent performs actions without being asked by a user
  2. Permissions
    An agent controls whether an action can be done or access given
  3. Deception
    An agent deceives a user upon request
  4. Social Dull, Dirty, and Dangerous work (Social DDD)
    An agent takes a larger role in the social interactions
  5. Computer-like Behavior
    An agent performs algorithmically or computationally, as asked by today's devices
Affinity diagram post-its
Affinity diagramming scenarios
Scenario considerations
Scenario considerations
Formal scenario brainstorming
Scenario sorting and discussion
Affinity mapping
Affinity mapping

Users & Scenarios

After identifying themes, we worked in small groups to identify potential scenarios. We generated a list of ideas (shown above) by theme and labeled them based on the context of their interactions (i.e. 2 parents, 1 child in a room), which we deemed important in ensuring good social multi-user interactions.

We split them up amongst ourselves to create more detailed scenarios between 3-5 sentences that would eventually develop into storyboards for interviews. Over 1-2 weeks, we iteratively refined our scenarios and eliminated unnecessary scenarios—ones that either captured a similar interaction or did not provide enough value to the set. Two members of our team developed our personas.

Scenario considerations
Computer-like behavior
Formal scenario brainstorming
Affinity mapping
Social Dirty, Dull, & Dangerous

Generating Storyboards

Narrowing down our ideas, we selected close to 30 scenarios to sketch out into storyboards. In our scenarios, it is important to capture the people involved, where it takes place, the activity, trigger, response, and resolution. These had to be reflected in our boards to help illustrate a story of an interaction not yet possible with today’s technology, but that which will be in the future.

Love notes storyboard

We critiqued each other's storyboards, eliminated and revised, and then did the same through pilot testing on friends and family in mock interviews over several iterations.

Movie night storyboard
Example #1
House chores storyboard
Example #2
Family support storyboard
Example #3
Birthday cake storyboard
Example #4
Conflict resolution storyboard
Example #5
The gift storyboard
Example #6

We noticed a major need to showcase more positive storyboards that got in the interactions. When we presented randomized sets in mock interviews that weighed the negative interviews more, subjects would focus on the “how” or the result of the interaction instead of the interaction itself. Eventually, we narrowed our storyboards down to 23.

Interviews & Speed Dating

Speed Dating in research consists of rapid comparisons between design opportunities and speculative futures by presenting low-fidelity concepts in a relatively structured and fast approach, similar to romantic speed dating. Providing participants with exposure to numerous concepts allows for better explanation into what they like and dislike across a wide set of futures and help us better understand user needs in an ambiguous space as a result.

Each interview lasted 1.5 hours, primarily at families’ homes in Pittsburgh, while some were on-campus. In each scenario, 2-3 researchers would conduct interviews to help guide participants through their thoughts and feelings. Scenarios were presented randomly using a Latin Square to help control for bias. Over the course of the interviews, we removed four and added two scenarios to reflect our feedback on whether scenarios provided value or if we were missing a theme.

I participated in running six out of 18 interviews.


We transcribed each interview and marked important quotes from participants. We cut out these quotes to include in another affinity diagramming to identify themes from the qualitative data. From here, we referenced and related themes to derive groupings and narratives shared between multiple families.

Affinity diagramming transcribed quotes
Affinity diagramming transcribed quotes


While the project is still ongoing until submission to CHI in Fall 2019, we’ve learned a lot about what families want from social agents in the home, and have more questions to follow. Deriving these insights required looking past scenario-specific cases and pulling out abstract ideas and themes from the quotes. Some of our findings are listed below.

  1. Families desire social agents to understand house rules
    Families tend to think the agent should be calibrated to individual households. Most families think parents should have primary control of the agent, but this hierarchy was often debated.
  2. Many families wouldn't trust the agent to be proactive in making decisions for them
    Most families want the agent to confirm actions before doing them. People don't trust the agent would be designed with enough transparency enough to be entirely trustworthy.
  3. Many families believe an agent could interfere with their children growing up
    Parents think an agent's surveillance would inhibit children from learning valuable lessons on their own. Many parents expressed the desire for their children to have room to experiment and make mistakes, and are worried children may additionally become reliant on agents in the home.
  4. Many families think the advanced social agent would remove some autonomy from household members
    Families tend to think the agent should be calibrated to individual households. Most families think parents should have primary control of the agent, but this hierarchy was often debated.
  5. Many families don't believe social agents will understand the intimate relationships between family members
    The level of surveillance granted to agents will make more things "provable" when family members can be verified or monitored. This is concerning to multiple families. Many see it as an "extra set of eyes."


The paper for this research was published in CHI 2020. You can access the paper via the link below.

Paper Download