Three Lessons I Learned About Applying Design Thinking to Machine Learning Research
I’m a PhD researcher in the Stanford AI Lab, working on building systems for machine learning and making it easier for people to use and deploy machine learning models. For the past three months I’ve had the wonderful opportunity to learn about the design process through Creativity in Research Scholars (CIRS) program at Stanford’s d.school.
Here are three lessons (and concrete tools!) I’ve learned about applying the design thinking process to machine learning research.
If you found this blog post useful, you can follow me and my research journey on Twitter (@realDanFu).
Lesson 1: Always Be Curious — Conduct Empathetic Interviews
For me, curiosity is the lifeblood of research — it’s what drives me when I wake up every morning and what drives me to build new things and to explore new ideas. One of the great things about being at Stanford is that you are always surrounded by people who feed and grow your curiosity. I’ve been feeling the absence of those connections especially hard this past year as we’ve all been separated for COVID — especially the spontaneous hallway conversations.
Empathetic interviews have helped me recreate some of that old magic — and I’ll be taking the lessons with me long after COVID ends.
What are empathetic interviews? They’re both a concrete methodology (how to learn as much as you can when you talk to someone), and a philosophy (do them whenever you can!).
In an empathetic interview, it’s all about them — not about you.
When you conduct an empathetic interview with someone, your goal is to try to put yourself in their shoes, and get a picture of the world from their perspective. My goal going into every interview was to absorb as much as their life and process as I could.
For the first three weeks of the CIRS program, we practiced empathetic interviews on each other, and then we reached out to individual stakeholders in our research. I used this opportunity to reconnect with some previous collaborators, but I also sent a lot of cold emails to people using machine learning in the wild (if you received an email out of the blue from me recently, this program is probably why!).
Assume a beginner’s mindset.
Cold emails can be scary, but it’s part of the process, and part of what you learn to do. One of the principles that helped make it more bearable was assuming a beginner’s mindset. Doing so removes the cost of failure — and makes you a better and more curious learner as well.
I now try to conduct an empathetic interview with someone new at least once a week. It’s especially invaluable during the pandemic, but I get so much out of each interview that I bet I’ll still be doing them long after life is back to normal.
Takeaways:
- In empathetic interviews, it’s all about them — not about you.
- Assume a beginner’s mindset to be a better learner.
- Cold emails — they’re not as scary as you think!
Lesson 2: Make the Implicit Explicit with Visual Thinking
In research, our thinking is often full of implicit assumptions and associations. And most of the time, we’re not even aware of them. Crazy things can happen when we connect the dots — research breakthroughs, new ideas, new solutions to old problems or vice versa. Visual thinking is a toolset for making the implicit explicit.
The basic idea behind visual thinking is simple — use charts/diagrams to visualize and brainstorm ideas. There are a bunch of different tools for visual thinking (check out the blog post linked above for some examples). Here’s a version of a quad chart for my sub-area of ML research (ML systems), for example. On one access you have challenges/boosters for using ML, on the other you have ideas and solutions running from theory to practice.
As you can see, visual thinking doesn’t have to be fancy, pretty, or well-designed. The goal is to use the tools to make it easier to think through complex ideas.
Three concrete benefits that I’ve noticed about visual thinking include:
- Increasing your working memory/RAM. When you’re thinking through complex topics, there’s a lot of stuff to keep in mind — getting some of it down on paper lets you think about more stuff at once. In the quad chart, I was only actively thinking about one quadrant at a time, but I could easily go back and reference previous thoughts easily.
- Making implicit assumptions explicit. When you put something in a visual chart, you have to make an active decision about where to put it. This can reveal gaps in your thinking that you didn’t know before. For example, are there challenges in my quad chart that don’t have corresponding solutions in the theory portion? What about in practice?
- Easier communication. I think this is all something we’re familiar with in research — we can all point to papers with great figures that can explain all the main ideas in half a page. You can do this with visual thinking too: even ugly sketches can help!
This blog post is actually a great example of the second point. It wasn’t until I put pen to paper that I realized that lessons 1 and 2 are actually examples of lesson 3 (coming up soon) — check out the dotted lines below!
Takeaways:
- Visual thinking is a powerful tool for increasing working memory, making implicit connections explicit, and communicating ideas.
- Visual thinking doesn’t have to be pretty; it just has to work!
Lesson 3: Fail Early, Fail Often with Rapid Prototyping
The last lesson is one that I was aware of before, but that bears repeating (and constant relearning) — fail early, fail often with rapid prototypes.
This is something that you hear a lot in product development, but it applies to research too. The canonical example is running experiments — what is the minimum experiment to confirm (or deny) a hypothesis? How quickly can you get that running?
But sometimes experiments are expensive to run (days+). A strategy of rapid prototyping can still help you clarify and organize your thinking. Here are a few examples of quick prototypes that I’ve used recently — none of which required writing a single line of code.
Lo-Fi Interactive Prototype
When I do ML research, I want it to ultimately be useful to someone. Sometimes I find it useful to sketch out the interfaces that consumers of my research will use to interact with it. Here’s an example of a quick sketch I did for an interface to explore how image models behave under various image transformations:
And a low-fi example of how one might use the pen tool to see if vision models are fooled by cat whiskers:
Obviously this is far from a fully-functional demo — but it gets the core idea across, and lets me start thinking about what’s missing in terms of functionality. In this case, it became clear after drawing a few more of these that I would also want to look at model activations for each class (e.g., grad-CAM) to see why a model’s prediction might change.
PS This would make a great class project — for any enterprising young undergrads out there!
Paper Outline
When you’re in the thick of experiments, sometimes it can be easy to lose track of the big picture. When I feel this way, I try to put together a quick introduction outline for the next paper I want to write (I like using Jennifer Widom’s intro bullets when I do).
These outlines help clarify my thinking and point out holes in my logic, or other experiments I want to run. Sometimes, they make it clear that it’s time to pivot — the last time I did this a few weeks ago, it became clear to me and my collaborators that we were actually bouncing around two sets of ideas at once, and it would be better to tackle each idea on its own.
Experiment outlines are a great candidate for this too. I like using this one.
Empathetic Interviews and Visual Thinking are All About Prototyping Too
One of the things I realized when I was outlining this blog post was that lessons 1 and 2 can actually function as examples of rapid prototyping. You can prototype ideas when you talk to people and brainstorm with people, and you can use visual thinking tools to rapidly prototype ideas.
The idea for an interactive model behavior explorer actually came from an empathetic interview I was conducting. The sketch is just the next step in prototyping, the first step was talking and listening.
And visual thinking tools can act as prototypes as well — I used a spider chart to plan out this blog post, and in doing so came up with the connections between lessons 1, 2, and 3 that you’re reading about right now.
Takeaways
- There’s a lot of research ideas that you can make lo-fi prototypes for — without writing a single line of code.
- Paper outlines are prototypes — and you can use them no matter where you are in the research process.
- Both of the other two lessons are really about rapid prototyping too!
Conclusion
It’s been a joy participating in CIRS these past three months — I’ll be taking these lessons with me as I continue forward on my research journey. I hope reading about my experiences with design thinking has been useful — if you found it interesting, you can follow along with my research on Twitter, or by checking out my research website!
Acknowledgments
A huge thank you to the fearless leaders who have been teaching CIRS remotely these past three months during the pandemic — Jon Feiber, Saara Khan, and Lars Thorben Neustock. And a shoutout to my fellow CIRS scholars — you made the whole experience fun and super worthwhile!