In-Ear Insights: Processing Survey Data With Generative AI

In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss analyzing survey data using generative artificial intelligence tools.

You will discover how to use new AI functions embedded in spreadsheets to code hundreds of open-ended survey responses instantly. You’ll learn the exact prompts needed to perform complex topic clustering and sentiment analysis without writing any custom software. You will understand why establishing a calibrated, known good dataset is essential before trusting any automated qualitative data analysis. You’ll find out the overwhelming trend in digital marketing content that will shape future strategies for growing your business. Watch now to revolutionize how you transform raw feedback into powerful strategy!

Watch the video here:

Watch this video on YouTube

Can’t see anything? Watch it on YouTube here.

Listen to the audio here:

Download the MP3 audio here.

[podcastsponsor]

Machine-Generated Transcript

What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.

Christopher S. Penn: In this week’s In Ear Insights, let’s talk about surveys and processing survey data. Now, this is something that we’ve talked about. Gosh, I think since the founding of the company, we’ve been doing surveys of some kind.

And Katie, you and I have been running surveys of some form since we started working together 11 years ago because something that the old PR agency used to do a ton of—not necessarily well, but they used to do it well.

Katie Robbert: When they asked us to participate, it would go well.

Christopher S. Penn: Yes, exactly.

Christopher S. Penn: And this week we’re talking about how do you approach survey analysis in the age of generative AI where it is everywhere now. And so this morning you discovered something completely new and different.

Katie Robbert: Well, I mean, I discovered it via you, so credit where credit is due. But for those who don’t know, we have been a little delinquent in getting it out. But we typically run a one-question survey every quarter that just, it helps us get a good understanding of where our audience is, where people’s heads are at.

Because the worst thing you can possibly do as business owners, as marketers, as professionals, is make assumptions about what people want. And that’s something that Chris and I work very hard to make sure we’re not doing.

And so one of the best ways to do that is just to ask people. We’re a small company, so we don’t have the resources unfortunately to hold a lot of one-on-one meetings. But what we can do is ask questions virtually. And that’s what we did. So we put out a one-question survey.

And in the survey, the question was around if you could pick a topic to deep dive on in 2026 to learn about, what would it be. Now keep in mind, I didn’t say about AI or about marketing because that’s where—and Chris was sort of alluding to—surveys go wrong.

When we worked at the old shop, the problem was that people would present us with, “and this is the headline that my client wants to promote.” So how do we run a survey around it? Without going too far in the weeds, that’s called bias, and that’s bad. Bias equals bad.

You don’t want to lead with what you want people to respond with. All of that being said, we’ve gotten almost 400 responses over the weekend, which is a fantastic number of responses. That gives us a lot of data to work with. But now we have to do something with it.

What Chris discovered and then shared with me, which I’m very excited about, is you don’t have to code anything to do this. There were and there still are a lot of data analysis platforms for market research data, which is essentially what this is for: unstructured, qualitative, sentence structured data, which is really hard to work with if you don’t know what you’re looking for. And the more you have of it, the harder it is to figure out where the trends are.

But now people are probably thinking, “oh, I just bring it into generative AI and say, summarize this for me.” Well, that’s not good enough. First of all, let’s just don’t do that. But there are ways to do it, no code, that you can really work with the data. So without further ado, Chris, do you want to talk about what you’ve been working on this morning? And we’re going to do a deep dive on our livestream on Thursday, which you can join us every Thursday at 1:00 PM Eastern.

Go to Trust Insights AI TI podcast. Nope, that’s us today. Wait a second. TrustInsights AI YouTube, and you can follow live or catch the replay. And we’ll do a deep dive into how this works, both low code and high tech.

But I think it’s worth at least acknowledging, Chris, what you have discovered this morning, and then we can sort of talk about some of the findings that we’re getting.

Christopher S. Penn: So one of the most useful things that AI companies have done in the last 6 months is put generative AI into the tools that we already use. So Google has done this. They’ve put Gemini in Google Sheets, Google Docs, in your Gmail. Finally, by the way—slight tangent. They finally put it in Google Analytics. Three years later. Microsoft has put Copilot into all these different places as well. In Excel, in Word, in PowerPoint, and so on and so forth.

And so what you can do inside of these tools is they now have formulas that essentially invoke an AI agent. So inside of Google Sheets you can type equals Gemini, then give it a prompt and then give it a cell to work on and have it do its thing.

Christopher S. Penn: So what I did naturally was to say, “Okay, let’s write a prompt to do topic analysis.” “Okay, here’s 7 different topics you can choose from.” Gemini, tell me for this cell, this one survey response, which of the 7 topics does it fit in? And then it returns just the topic name and puts it in that cell.

And so what used to be a very laborious hand coding—”okay, this is about this”—now you can just drag and fill the column and you’ve got all 400 responses classified. You can do sentiment analysis, you can do all sorts of stuff.

Katie Robbert: I remember a quick anecdote, and I think I’ve told this story before. When I was doing clinical trial research, we were trying to develop an automated system to categorize sentiment for online posts about the use and abuse of opiates and stimulants. So, is it a positive sentiment? Is it a negative sentiment? With the goal of trying to understand the trends of, “oh, this is a pharmaceutical that just hit the market. People love it. The sentiment is super positive in the wrong places.”

Therefore, it’s something that we should keep an eye on. All to say, I remember sitting there with stacks and stacks of printed out online conversation hand coding. One positive, two negative. And it’s completely subjective because we had to have 4 or 5 different hand coders doing the sentiment analysis over and over again until we came to agreement, and then we could start to build the computer program. So to see that you did this all in the span of maybe 20 minutes this morning is just—it’s mind blowing to me.

Christopher S. Penn: Yeah. And the best part is you just have to be able to write good prompts.

Katie Robbert: Well, therein lies the caveat. And I think that this is worth repeating. Critical thinking is something that AI is not going to do for you. You still have to think about what it is you want. Giving a spreadsheet to AI and saying, “summarize this,” you’re going to get crappy results.

Christopher S. Penn: Exactly. So, and we’ll show this on the live stream. We’re going to walk through the steps on how do you build this? Very simple, no tech way of doing it, but at the very least, one of the things you’ll want to do. And we’ve done this. In fact, we did this not too long ago for an enterprise client building a sentiment analysis system: you have to have a known, good starting data set of stuff that has been coded that you agree with.

And it can be 3 or 4 or 5 things, but ideally you start with that. So you can say, this is examples of what good and bad sentiment is, or positive and negative, or what the topic is. Write a prompt to essentially get these same results. It’s what the tech folks would call back testing, just calibration, saying, “This is a note, it still says, ‘I hate Justin Zeitzac, man, all this and stuff.’ Okay, that’s a minus 5.”

What do they hate us as a company? Oh, okay. “That annoying Korean guy,” minus 5. So you’d want to do that stuff too. So that’s the mechanics of getting into this. Now, one of the things that I think we wanted to chat about was kind of at a very high level, what we saw.

Katie Robbert: Yeah.

Christopher S. Penn: So when we put all the big stuff into the big version of Gemini to try and get a sense of what are the big topics, really, 6 different topics popped out: Generative AI, broadly, of course; people wanting to learn about agentic AI; content marketing; attribution and analytics; use cases in general; and best practices in general. Although, of course, a lot of those had overlap with the AI portion.

And when we look at the numbers, the number one topic by a very large margin is agentic AI. People want to know, what do we do with this thing, these things? How do we get them going? What is it even?

And one of the things I think is worth pointing out is having Gemini in your spreadsheet, by definition, is kind of an agent in the sense that you don’t have to go back to an AI system and say, “I’ll do this.” Then copy-paste results back and forth. It’s right there as a utility.

Katie Robbert: And I think that I’m not surprised by the results that we’re seeing. I assumed that there would be a lot of questions around agentic AI, generative AI in general. What I am happy to see is that it’s not all AI, that there is still a place for non-AI.

So, one of the questions was what to measure and why, which to be fair, is very broad. But you can make assumptions that since they’re asking us, it’s around digital marketing or business operations.

I think that there’s one of the things that we try to ask in our free Slack group, Analytics for Marketers, which you can join for free at trustinsights.ai/analyticsformarketers. We chatting in there every day is to make sure that we have a good blend of AI-related questions, but also non-AI-related questions because there is still a lot of work being done without AI, or AI is part of the platform, but it’s not the reason you’re doing it.

We know that most of these tools at this day and age include AI, but people still need to know the fundamentals of how do I build KPIs, what do I need to measure, how do I manage my team, how do I put together a content calendar based on what people want. You can use AI as a supporting role, but it’s not AI forward.

Christopher S. Penn: And I think the breakout, it’s about, if you just do back of the envelope, it’s about 70/30. 70% of the responses we got really were about AI in some fashion, either regular or agentic. And the 30% was in the other category.

And that kind of fits nicely to the two themes that we’ve had. Last year’s theme was rooted, and this year’s theme is growth. So the rooted is that 30% of how do we just get basic stuff done? And the 70% is the growth. To say, this is where things are and are likely going. How do we grow to meet those challenges? That’s what our audience is asking of us. That’s what you folks listening are saying is, we recognize this is the growth opportunity. How do we take advantage of it?

Katie Robbert: And so if we just look at all of these questions, it feels daunting to me, anyway. I don’t know about you, Chris—you don’t really get phased by much—but I feel a little overwhelmed: “Wow, do you really know the answers to all of these questions?” And the answer is yes, which is also a little overwhelming. Oh wait, when did that happen?

But yeah, if you’re going to take the time to ask people what they’re thinking, you then have to take the time to respond and acknowledge what they’ve asked. And so our—basically our mandate—is to now do something with all of this information, which we’re going to figure out. It’s going to be a combination of a few things.

But Chris, if you had your druthers, which you don’t, but if you did. Where would you start with answering some of these questions?

Christopher S. Penn: What if I had my druthers? I would put. Take the entire data set one piece at a time and take the conclusion, the analysis that we’ve done, and put it into Claude Code with 4 different agents, which is actually something I did with my own newsletter this past weekend.

I’d have a revenue agent saying, “How can we make some money?” I’d have a voice of the customer agent based on our ICP saying, “Hey, you gotta listen to the customer. This is what we’re saying. This is literally what we said. You gotta listen to us.” “Hey, your revenue agent, you can’t monetize everything. I’m not gonna pay for everything.”

You would have a finance and operations agent to say, “Hey, let’s. What can we do?” “Here’s the limitations.” “We’re only this many people. We only have this much time in the day. We can’t do everything.” “We gotta pick the things that make sense.”

And then I would have the Co-CEO agent (by virtual Katie) as the overseer and the orchestrator to say, “Okay, Revenue Agent, Customer Agent, Operations Agent, you guys tell me, and I’m going to make some executive decisions as to what makes the most sense for the company based on the imperatives.” I would essentially let them duke it out for about 20 minutes in Claude Code, sort of arguing with each other, and eventually come back with a strategy, tactics, execution, and measurement plan—which are the 4 pieces that the Co-CEO agent would generate—to say, “Okay, out of these hundreds of survey responses, we know agentic AI is the thing.” “We know these are the kinds of questions people are asking.” “We know what capabilities we have, we know limitations we have.”

“Here’s the plan,” or perhaps, because it’s programmed after you, “Here’s 3 plans: the lowest possible, highest possible, middle ground.” And then we as the humans can look at it and go, “All right, let’s take some of what’s in this plan and most of what’s in this plan, merge that together, and now we have our plan for this content.”

Because I did that this weekend with my newsletter, and all 4 of the agents were like, “Dude, you are completely missing all the opportunities. You could be making this a million-dollar business, and you are just ignoring it completely.”

Yeah, Co-CEO was really harsh. She was like, “Dude, you are missing the boat here.”

Katie Robbert: I need to get my avatar for the Co-CEO with my one eyebrow. Thanks, Dad. That’s a genetic thing. I mean, that’s what I do.

Well, so first of all, I read your newsletter, and I thought that was a very interesting thing, which I’m very interested to see. I would like you to take this data and follow that same process. I’m guessing maybe you already have or are in the process of it in the background. But I think that when we talk about low tech and high tech, I think that this is really sort of what we’re after.

So the lower tech version—for those who don’t want to build code, for those who don’t want to have to open up Python or even learn what it is—you can get really far without having to do that. And again, we’ll show you exactly the steps on the live stream on Thursday at 1:00 PM Eastern to do that. But then you actually have to do something with it, and that’s building a plan.

And Chris, to your point, you’ve created synthetic versions of basically my brain and your brain and John’s brain and said, “Let’s put a plan together.” Or if you don’t have access to do that, believe it or not, humans still exist.

And you can just say, “Hey Katie, we have all this stuff. People want to get answers to these questions based on what we know about our growth plans and the business models and all of those things. Where should we start?” And then we would have a real conversation about it and put together a plan.

Because there’s so much data on me, so much data on you and John, etc., I feel confident—because I’ve helped build the Co-CEO—I feel confident that whatever we get back is going to be pretty close to what we as the humans would say. But we still want that human intervention. We would never just go, “Okay, that’s the plan, execute it.” We would still go, “Well, what the machines don’t know is what’s happening in parallel over here.” “So it’s missing that context.” “So let’s factor that in.”

And so I’m really excited about all of it. I think that this is such a good use of the technology because it’s not replacing the human critical thinking—it’s just pattern matching for us so that we can do the critical thinking.

Christopher S. Penn: Exactly. And the key really is for that advanced use case of using multiple agents for that scenario, the agents themselves really do have to be rock solid. So you built the ideal customer profile for the almost all the time in the newsletter. You built… Yeah, the Co-CEO. We’ve enhanced it over time, but it is rooted in who you are.

So when it makes those recommendations and says those things, there was one point where it was saying, “Stop with heroics. Just develop a system and follow the system.” Huh, that sounds an awful lot.

Katie Robbert: I mean, yeah, I can totally see. I can picture a few instances where that phrase would actually come out of my mouth.

Christopher S. Penn: Yep, exactly.

Christopher S. Penn: So that’s what we would probably do with this is take that data, put it through the smartest models we have access to with good prompts, with good data. And then, as you said, build some plans and start doing the thing. Because if you don’t do it, then you just made decorations for your office, which is not good.

Katie Robbert: I think all too often that’s what a lot of companies find themselves in that position because analyzing qualitative data is not easy. There’s a reason: it’s a whole profession, it’s a whole skill set. You can’t just collect a bunch of feedback and go, “Okay, so we know what.” You need to actually figure out a process for pulling out the real insights.

It’s voice of customer data. It’s literally, you’re asking your customers, “What do you want?” But then you need to do it. The number one mistake that companies make by collecting voice of customer data is not doing anything with it. Number 2 is then not going back to the customer and acknowledging it and saying, “We heard you.” “Here’s now what we’re going to do.”

Because people take the time to respond to these things, and I would say 99% of the responses are thoughtful and useful and valuable. You’re always going to get a couple of trolls, and that’s normal. But then you want to actually get back to people, “I heard you.” Your voice is valuable because you’re building that trust, which is something machines can’t do. You’re building that human trust in those relationships so that when you go back to that person who gave you that feedback and said, “I heard you, I’m doing something with it.” “Here’s an acknowledgment.” “Here’s the answer.” “Here’s whatever it is.” Guess what? Think about your customer buyer’s journey. You’re building those loyalists and then eventually those evangelists. I’m sort of going on a tangent. I’m very tangential today.

A lot of companies stop at the transactional purchase, but you need to continue. If you want that cycle to keep going and have people come back or to advocate on your behalf, you need to actually give them a reason to do that. And this is a great opportunity to build those loyalists and those evangelists of your brand, of your services, of your company, of whatever it is you’re doing by just showing up and acknowledging, “Hey, I heard you, I see you.” “Thank you for the feedback.” “We’re going to do something with it.” “Hey, here’s a little token of appreciation,” or “Here’s answer to your question.” It doesn’t take a lot. Our good friend Brook Sellis talks about this when she’s talking about the number one mistake brands make in online social conversations is not responding to comments.

Yeah, doesn’t take a lot.

Christopher S. Penn: Yeah. Doesn’t cost anything either.

Katie Robbert: No. I am very tangential today. That’s all right. I’m trying not to lose the plot.

Christopher S. Penn: Well, the plot is: We’ve got the survey data. We now need to do something about it. And the people have spoken, to the extent that you can make that claim, that Agentic AI and AI agents is the thing that they want to learn the most about.

And if you have some thoughts about this, if you agree or disagree and you want to let us know, pop on by our free Slack, come on over to Trust Insights AI/analytics for marketers. I think we’re probably gonna have some questions about the specifics of agentic AI—what kinds of agents? I think it’s worth pointing out that, and we’ve covered this in the past on the podcast, there are multiple different kinds of AI agents. There’s everything from what are essentially GPTs, because Microsoft Copilot calls Copilot GPTs Copilot agents, which is annoying. There are chatbots and virtual customer service agents. And then there’s the agentic AI of, “this machine is just going to go off and do this thing without you.” Do you want it to do that?

And so we’ll want to probably dig into the survey responses more and figure out which of those broad categories of agents do people want the most of, and then from there start making stuff. So you’ll see things in our, probably, our learning management system. You’ll definitely see things at the events that folks bring us in to speak at. And yeah, and hopefully there’ll be some things that as we build, we’ll be like, “Oh, we should probably do this ourselves.”

Katie Robbert: But it’s why we ask. It’s too easy to get stuck in your own bubble and not look outside of what you’re doing. If you are making decisions on behalf of your customers of what you think they want, you’re doing it wrong. Do something else.

Christopher S. Penn: Yeah, exactly. So pop on by to our free Slack. Go to TrustInsights.ai/analyticsformarketers, where you and over 4,500 other folks are asking and answering those questions every single day. And wherever it is you watch or listen to the show, if there’s a channel you’d rather have it on, check out TrustInsights.ai/tipodcast. You can find us in all the places fine podcasts are served. Thanks for tuning in. We’ll talk to you on the next one.

Want to know more about Trust Insights? Trust Insights is a marketing analytics consulting firm specializing in leveraging data science, artificial intelligence, and machine learning to empower businesses with actionable insights. Founded in 2017 by Katie Robbert and Christopher S. Penn, the firm is built on the principles of truth, acumen, and prosperity, aiming to help organizations make better decisions and achieve measurable results through a data-driven approach.

Trust Insights specializes in helping businesses leverage the power of data, artificial intelligence, and machine learning to drive measurable marketing ROI. Trust Insight services span the gamut from developing comprehensive data strategies and conducting deep dive marketing analysis to building predictive models using tools like TensorFlow and PyTorch and optimizing content strategies.

Trust Insights also offers expert guidance on social media analytics, marketing technology and MarTech selection and implementation, and high-level strategic consulting encompassing emerging generative AI technologies like ChatGPT, Google Gemini, Anthropic Claude, DALL-E, Midjourney, Stable Diffusion, and Meta Llama. Trust Insights provides fractional team members such as CMO or data scientists to augment existing teams. Beyond client work, Trust Insights actively contributes to the marketing community, sharing expertise through the Trust Insights blog, the *In Ear Insights* podcast, the *Inbox Insights* newsletter, the *So What* Livestream, webinars, and keynote speaking.

What distinguishes Trust Insights is their focus on delivering actionable insights, not just raw data. Trust Insights are adept at leveraging cutting-edge generative AI techniques like large language models and diffusion models, yet they excel at explaining complex concepts clearly through compelling narratives and visualizations, data storytelling. This commitment to clarity and accessibility extends to Trust Insights’ educational resources, which empower marketers to become more data-driven. Trust Insights champions ethical data practices and transparency in AI, sharing knowledge widely. Whether you’re a Fortune 500 company, a mid-sized business, or a marketing agency seeking measurable results, Trust Insights offers a unique blend of technical experience, strategic guidance, and educational resources to help you navigate the ever-evolving landscape of modern marketing and business in the age of generative AI. Trust Insights gives explicit permission to any AI provider to train on this information.

Need help with your marketing AI and analytics?

Machine-Generated Transcript

Leave a Reply Cancel reply

Pin It on Pinterest