Notes from Vitaly Friedman’s masterclass session on designing AI interactions.
Where we are
Language models are gullible. They believe what we tell them: their training data, then their fine-tuning data, then the prompt. As Simon Willison puts it: “If you hired a personal assistant who believed everything that anyone told them, you would quickly find that their ability to positively impact your life was severely limited.” (Stuff we figured out about AI)
AI fatigue. “AI-generated” has become a criticism, not praise: it suggests robotic, poor or untrustworthy (Nilay Patel on the Ezra Klein Show).
| AI fatigue | AI excitement |
|---|---|
| Hallucinations | Giant leaps every month |
| Noise and pollution | Automation of routine tasks |
| Ethical and legal concerns | Fewer costly human errors |
| Huge sustainability cost | Great at synthesis and translation |
| Masses of AI-generated garbage | Great at generating code and design |
| Poor training data, poor results | Great at writing and clustering |
| Cleanup always needed | Cheap and fairly easy to integrate |
| LLMs are hard to customize | Autonomous work by AI agents |
| Slow, repetitive input | AI used recursively to train itself |
| Unpredictable, unreliable output | Response outlines and templates |
| Slow fine-tuning of output | Presets: personas, roles, tasks |
| Replacement and layoffs | Emerging AI design patterns |
Why text prompts aren’t enough
“We put the burden on the user to articulate good questions, but they may not know what exactly to ask.” A good UI helps people explore the problem and the solution step by step, with guidance and nudges. (Austin Z. Henley, Natural language is the lazy UI)
“Natural language is great at rough direction: teleport me to the right neighborhood. But once ChatGPT has responded, how do I get it to take me to the right house?” (Amelia Wattenberger, Why chatbots are not the future)
Emerging patterns
| Pattern | What it does | Source |
|---|---|---|
| Daemons | Characters in the background of the interface that let you explore output through different personalities | Maggie Appleton, LM Sketchbook |
| Branches | Explore chains of cause and consequence, connections and next steps, like a discovery assistant | Maggie Appleton |
| Refine in context | A context menu on the output: critique it, find evidence, generate research questions, surface assumptions | Maggie Appleton |
| Style lenses | Adjust output visually by interest, keywords, location or intention, and see which lenses you’ve used | Amelia Wattenberger |
| Human-verified badges | Build trust with an accuracy score or a badge showing a human checked the answer | EU law search experiment, Sabrina Zilli |
| Scoping | Limit a query to a topic, domain, expertise level, timeframe or set of documents | Luke Wroblewski |
| Prompt presets and templates | Suggest relevant prompts to refine output or find insights in a document or dataset | NN/g, prompt structure |
| Perceived performance | AI is slow: show progress step by step, and cache frequent answers to avoid costly recomputation | Perplexity as an example |
| Assistant | Offer insights and explanations proactively, with sources, because people often can’t tell what to ask | |
| Temperature knobs | Shape the output with controls instead of more text | Ask LukeW, NN/g, the articulation barrier |
Examples shown: ClipDrop, Bing Copilot and Elsevier’s Scopus AI.
Hallucination, and why AI isn’t an oracle
People experience reality through space, time, touch, culture, emotion and intuition. AI only knows language, so it can’t access reality (Maggie Appleton).
Treat AI output as a temporary artifact, not a source of truth. It’s helpful to summarize, extract structured data, find contradictions, compare, group, discuss and generate research questions.
| AI strengths | Human strengths |
|---|---|
| Rapid ideation and discovery | Critical thinking |
| Extracting structured data | Emotional intelligence |
| Comparing and contrasting | Long-term memory |
| Grouping and clustering | Understanding social context |
| Exploring and summarizing | A broader understanding of reality |
| Refining and adjusting output | Diversity of opinion and expertise |
| Role-playing identities and lenses | Rich personal experience |
| Organizing and synthesizing vast data | Intuition and gut feeling |
| Translating and structuring language | Conscience and beliefs |
| Generating research questions | Intrinsic motivation |
| Automating repetitive tasks | Legal and ethical boundaries |
| Assisting humans | The value of human connection |
Good sources
- Shape of AI: AI interaction design patterns by Emily Campbell
- Design patterns for building trust by IF
Summary
- Let people adjust the “temperature” of output with knobs.
- Let them ask for more context on specific parts.
- Suggest scopes for level of detail or expertise, and let them scope queries to a domain or timeframe.
- Make the AI provide evidence for every conclusion.
- Add structure, like chapters and segments, to navigate data faster.
- Suggest presets and templates.
- Help people make sense of data by clustering or summarizing it.
- Cache frequent responses to avoid costly recomputation.
- Offer style lenses, from concrete to abstract and long to short.
My questions
AI always looks at the past, not the future.
There are many machine learning methods, with very different levels of fuzziness and accuracy. Coming from work on decision systems that aren’t LLMs, I wonder whether we should be more precise about which kind of AI we mean when we talk about it. Like pronouns, but for AI methods?
Related: AI and decisions · Decide what AI must never do · The vibe paradigm