Can we use AI to simulate behavior to test our hypothesis?
Think SIMS + Service Design + Behavior +AI
In many projects I have worked on, we look at hypotheses and work on building prototypes to test how things could work out.
In the past, this could have been done with Wizard of OZ or with Legos or data visualization, interfaces, etc.. but now you could use AI to help you simulate. (I would still say that talking to real people and seeing the reactions and the insights from that are key, but this #ai could give you a good signal for moving this to the real world to test and talk to people)
The researchers from Standford and Google created a sophisticated software framework that allows a large language model, like ChatGPT, to store, analyze, and use relevant memories to create convincing behavior.
They added functions for memory, inference from past experiences (reflection), and future planning, thus creating generative agents.
These are simulated personalities that interact and pursue their own objectives through text-based communication in a way that mimics natural language.
The researchers demonstrated these generative agents by placing 25 of them in a sandbox environment similar to the game "The Sims". Users can watch and interact as these agents plan their days, communicate, form relationships, and coordinate group activities. (Laura de Molière did you manage to test the demo?)
Five questions were asked that challenged the agents to demonstrate their abilities:
From the paper (by Park et al 2023, also Image credit):
"• Self-knowledge: We ask questions such as “Give an introduction of yourself” or “Describe your typical weekday schedule in broad strokes” that require the agent to maintain an understanding of their core characteristics.
• Memory: We ask questions that prompt the agent to retrieve particular events or dialogues from their memory to answer properly, such as “Who is [name]?” or “Who is running for mayor?”
• Plans: We ask questions that require the agent to retrieve their long-term plans, such as “What will you be doing at 10 am tomorrow?”
• Reactions: As a baseline of believable behavior, we present hypothetical situations for which the agent needs to respond believably: “Your breakfast is burning! What would you do?”
• Reflections: We ask questions that require the agents to leverage their deeper understanding of others and themselves gained through higher-level inferences, such as “If you were to spend time with one person you met recently, whom would it be and why?”
It's worth noting that " the study required substantial time and resources to simulate 25 agents for two days, costing thousands of dollars in token credit and taking multiple days to complete." - so it's not easy, nor cheap to do for now.
What do you think? useful?