Compared to other problems I might have…
Several weeks ago there was a flurry of activity on Twitter around a concept known as “synthetic users”. This was borne out of a company aptly called “SyntheticUsers” with the claims that one could “Test [your] idea or product with AI participants and take decisions with confidence” by harnessing the power of AI. As someone who’s spent years doing hundreds of user research interviews with humans, I was intrigued. I wanted to see what the fuss was all about (because there was serious fuss.)
In the past, I worked at a couple of start-ups where my team and I needed to validate ideas using traditional user research and testing methods. At times, we would have to lean away from a rigid academic approach to research and get scrappy, particularly when time was really tight, and as such, I get the marketing premise behind SyntheticUsers. I also truly wanted to see what the experience could be like, replacing my researcher hat with a product-focused one. And what I found was both surprising and not…
My prompt — I created an account and logged into the discord server that the current app is hosted on. I specified “Minorities who need to sign up for health insurance but have not had great success in the past.” as my target audience and I listed problems such as “They feel they are generally not prioritized by caregivers.” and “They are looking for maternal care options for themselves and family.” I asked the program to generate 3 user interviews for me and I watched what came back.
Each user came back with “transcripts” that comprised multiple areas including name, age, background, problem description from their POV, challenges, etc. All the things I would ask as a researcher into this space. And while the content initially seemed promising, I immediately saw some problems that surface the issues with the current state of tools such as SyntheticUsers that go beyond ethical concerns (which, I do not want to address here because that’s a post for another day.)
Length and readability — All 3 transcripts were between 1069 and 1876 words. This isn’t unusual for shorter interview sessions but what I noticed was that the content returned had been largely summarized. There was no storytelling, no personal anecdotes that could surface further insights. Just high-level summaries that all felt very similar. Upon running the transcripts through an open-source readability tool, they all came back at a very similar reading level (11th-13th grade, or early college) with near identical Gunning-Fog scores (14.5–17). These 3 “different users” sure did share their information using very similar speech patterns.
Identities — All 3 were minorities, which was what I wanted. They included:
- Faraaz Saleem, who is 28 years old and from Karachi, Pakistan
- Serafina Del Rosario, who is 34 years old, and from Mexico City
- Saffron O’Leary, who is 29 years old, and from Dublin, Ireland
Clearly the tool was able to pick varying countries of residence and ethnicities and it even included a married lesbian couple, allowing the responses to be centered on the unique constraints faced by same-sex couples. I can’t fault the tool for the lack of age range for my participants since I specified “maternal care” but I would’ve expected both younger and older. 6 years is pretty limited.
Sentiment — Here’s where it all sort of fell apart. Recall I said earlier that the content all read at similar levels, and while this might be pure chance (hey, I’m taking a stats course right now so I get that samples can be biased) the actual content itself is painfully identical. Rather than list out all the similarities, I want to show you a screen grab of one section from each of the 3 users. I have highlighted, using different colors to really show where the content is almost word-for-word duplication:
User #1
User #2
User #3
A particularly egregious line lies in “Compared to other problems I might have” which is quoted by all 3 users, an almost statistical impossibility in a research session with 3 users, supposedly all from different countries and backgrounds. The remainder of the transcripts highlight the same duplicative content issues and this is where I struggle to see a solid use case. As researchers, part of our value is being able to take the complexities of the human experience and not only surface these complexities as part of our interviews, but also connect across these differences to highlight where a particular product or process direction could go, all the while still navigating the unique needs of our target audience. If every user responds in the same way, with the same problems, then we as researchers either performed our research across a truly homogeneous set of interviewees (aka, we picked a heavily biased sample set) or we didn’t do a deep enough analysis of the problem space that our interviewees experience (aka, we didn’t ask the right questions or we didn’t ask the questions in the right way, we used leading questions to get specific answers, etc.) Regardless, we don’t have a true depth of understanding of our users and as a result, our research isn’t useful.
Conclusion — Is this the end of companies like SyntheticUsers? Absolutely not. My guess is that the language models will get better and at some point in the future, we may have the ability to lean on tools such as these to make our jobs as researchers easier. Some areas where AI may help research include…
- Clean-up of transcripts with human participants. Note: I realize there are ethical considerations with feeding notes from individuals into these tools. I would highly recommend that anyone doing this does a super deep clean of their transcripts prior to putting anything into an online ML / AI tool, particularly by those who are sharing their lived experiences and especially those in marginalized groups who are being both left out of, and exploited by, some AI/ML tools. Also, keep in mind I said “clean-up” and not analysis. While automation may save time, it can miss important information that researchers really focus deeply on finding. To me, this is still hard to pull out of tools like SyntheticUsers largely because this process can take multiple reviews by different sets of eyes. It goes beyond pulling together similar thoughts and putting them on a whiteboard.
- Learning how to “do” UX. I would’ve loved to have had something that simulated participant feedback when I was learning how to do research. Granted, the best way to learn is to do, there will always be areas that can be difficult. I can recall trying to ask the right question in the right way early in my research career and frustrating my interviewee with my poorly structured and executed session. This can be devastating for someone sharing a very personal experience and having to endure questions from a bad interviewer. Interview subjects can’t be our learning guinea pigs.
- Helping emulate the research session prior to doing the research. To piggyback on the prior bullet, having some way to ensure my questions weren’t written in a leading way, that my questions were well thought out and that I had potential branching questions that worked together would be interesting. This one’s probably harder to explain but the idea of “simulating” an interview prior to doing real human interviews may be really useful.
In short, I’m not down on SyntheticUsers as much as I’m waiting to see where they go with their offering. The reality is AI/ML is here (and it’s not going anywhere.) No amount of hand wringing and frustration is going to make it go away. As professional researchers, we must continue to advocate for our users, our work and our methods but we also need to find ways to co-exist with these types of tools in this new world. We aren’t there yet but with some time, we may hopefully get closer. Compared with all other problems we as researchers might have, being replaced by tools like SyntheticUsers is probably not one of them.
