I recently wrote an article in which I discussed the misconceptions about AI replacing software developers. Some of these misconceptions apply much more widely. In particular, there seems to be a knee-jerk reaction to think that, for better or worse, any new technology might be able to replace existing jobs, technologies, business models and so on. But in the age of AI, once that knee-jerk reaction passes, the mind should go not to replacement but to augmentation, by which I mean simply making people, processes or technologies better.

Even having just written about this challenge for software developers, I fell victim to this bias myself last week when I was trying to formulate a user survey. My hope is that by sharing that experience, I can help others bypass the bias for AI-as-replacement and embrace AI-as-augmentation instead.

The Trouble With User Surveys

It all started with a discussion with a client about a user survey. Surveys are valuable tools for marketers but, frankly, they are kind of a pain to do. They can be expensive and time consuming, and results are often less precise than marketers hope. So, when I mentioned that maybe, somehow, we could use AI instead of a traditional survey, I got a positive response from the team.

I started to play around with some AI tools and did a bit of research to see how far I could get with using them to formulate a replacement for the user survey. Honestly, it was not very far. So I reached out to some colleagues and friends to see if any of my connections had thoughts about how to proceed. I got two categories of responses.

1. Wow! It would be great if we could do something like that.
2. You know, I tried this too, and there are some problems with this idea.

Given the interest from group 1, I dug in further, armed with the feedback of group 2. I found three significant issues with the idea of replacing surveys with generative AI.

• A key part of the survey is respondent segmentation — My good friend and great product leader Rich Sharples pointed out that an important part of surveys is understanding at least some user demographics. For example, respondent sentiment may follow different trends based on where people work or live. Unfortunately, it’s difficult to develop a prompt such that you can segment it with a GenAI model. This is especially true if you are sourcing public internet data, which is highly unstructured.
• The data source must be well-curated — The segmentation issue explained above highlights the challenge of using unstructured public data, which was where both Rich and I started our experiments. Another great PM friend of mine, Sundar Thirugnanam, pointed out that using public data can also be unreliable. Trusting the data can be an issue especially on technology-related topics. In addition to the problem of bots spewing out content to influence metrics, people will often comment on a forum as a snap reaction or as they’re seeking answers to an unrelated question. So, if you want to use AI to replace a survey, you need a well-curated (and likely private) dataset. But if you have a well-curated data set, why would you do a survey in the first place?
• Generative AI can help, but more is required — An alternative that sits between public and well-curated private data is poorly curated private data. This is a very likely scenario inside a company. For example, it would be possible to conduct an internal employee survey that leverages internal unstructured sources like e-mail, collaboration spaces (Slack, Teams etc.), and even web conferences combined with structured data from an HR system. However, just pointing a LLM at that data and doing some data science basics may not be the best approach. However, a combination of LLMs and machine learning tools could help. But even in that case, one still must consider whether a well-thought-out survey response would be more valuable than an offhand comment or sentence sourced from a discussion thread somewhere. Also, in this case your people might get a Big Brother vibe from HR, which is probably a bad thing.

Maybe needless to say, my conclusion was that replacing surveys with GenAI is not a great idea. However, in the process I learned a few important things about AI and the replacement bias notion that could generalize to other cases. As I walk through the learnings specific to surveys, I encourage you to think about the kinds of augmentation-not-replacement lessons they might suggest for other domains.

AI May Not Replace Surveys, But It Can Make Surveys A Lot Better

At a high level, a survey has two processes. There is the data-collection process and the response-analysis process. GenAI is not necessarily a replacement for the data-collection process. However, GenAI does open up some new doors in response analysis.

First, GenAI enables more forgiving and flexible questions. Surveys have been dominated by multiple-choice questions because they are easier to analyze and they focus responses very narrowly on what the survey creator wants to know. But the capabilities of GenAI allow survey writers to ask more open-ended questions. Instead of “Which of these four shampoos do you use?,” a survey can ask “What is your favorite shampoo and why?” or “What shampoo have you tried before that you stopped using—and why did you stop?” You may not even need words. Questions can now be a snippet of music or a picture.

GenAI also enables bigger scale and reduced margin for error. Survey results have to be analyzed, and sometimes that puts a cap on how many people can be surveyed. But again, given the speed of these new AI tools, a lot more people can be engaged by a survey, because the extra time required to analyze more data is only marginal. The broader the survey, the better the results thanks to a decreasing margin of error.

This flexibility and scale means that surveys can now approach the effectiveness of a focus group. Surveys are generally a good balance of cost and scale to gather data, but the gold standard has historically been the focus group. However, focus groups are very expensive, and the in-person nature of them can both limit scale and bias the outcomes. But by combining the scalability of a survey with the opportunity for more flexible questions, surveys can now capture some of the benefits of focus groups.

Ultimately, this approach could foster more nuanced analysis. If people are able to more fully engage with a survey, they can share more nuanced information and sentiments. GenAI tools are great at summarizing voluminous and otherwise ambiguous information. A richer set of data may even allow for asking follow-up questions of the dataset without sending a new survey. In theory, this base of information could be reused or appended to new surveys as time goes by. Respondent XYZ loved our product two years ago, but now they find it merely adequate: Did their rationale change, or was it something we did?

Market leader SurveyMonkey has a new product called SurveyMonkey Genius, and there are others out there such as Alchemer, Knit and QuestionPro. Many of these vendors are initially focused on using AI to help with the data-collection process by helping people craft better survey questions. But some vendors are also starting to move into response analysis. So, again, while marketers and others will still need surveys, AI is opening doors to better surveys and better insights from them, which is definitely a good thing.

Using AI To Augment Business Processes, Customer Experience And More

While my survey experiment here is just one example of overcoming replacement bias, you can easily extend the thought of AI augmentation into other areas. For example, I do a lot of traveling for work, so I often think of ways to improve air travel. How about, instead of using AI-powered facial scanning to replace a security guard at an airport, use the technology to smooth out the check-in experience or provide premium services? For example, someone who looks tired waiting for a connection could be offered time in a premium lounge. Or an airline could give assistance to travelers who need help due to a physical limitation or based upon their airline status (Mr. Andersen, please proceed to the front of the line). So instead of replacing a person, you come away with elevated customer loyalty and better NPS scores.

Of course there are more examples than I—or any person—can possibly think of, so here’s a simple rubric to shift one’s bias. Instead of walking around looking to do more with fewer resources, you can follow this three-phase process:

1. Be on the lookout for processes that don’t scale well by adding more resources, are somehow constrained by human involvement, or could be improved if the human actor had more information.
2. Once you identify the process ask yourself either “How can I improve this process?” or “What do people hate most about this process?”
3. With that information, start to triage the type of AI that could help you. If you need more scale or more specific data, start with machine learning or vision apps. If you need more summarized data or connected information, start with generative AI.

This is a good way to build a dialogue about the problem and its impact before trying to decide the solution. And it helps keep the focus on better rather than cheaper.