So What? Using Generative AI for Survey Analysis

So What? Marketing Analytics and Insights Live

airs every Thursday at 1 pm EST.

You can watch on YouTube Live. Be sure to subscribe and follow so you never miss an episode!

In this week’s episode of So What? we focus on using generative AI for survey analysis. We walk through how to set up your survey data for analysis, create prompts for your survey analysis and what you can get from generative AI for your survey analysis. Catch the replay here:

 

In this episode you’ll learn: 

• how to set up your survey data for analysis
• creating prompts for your survey analysis
• what you can get from generative AI for your survey analysis

Upcoming Episodes:

• TBD

Have a question or topic you’d like to see us cover? Reach out here: https://www.trustinsights.ai/resources/so-what-the-marketing-analytics-and-insights-show/

AI-Generated Transcript:

Katie Robbert 0:15
Well, hey there everyone. Happy Thursday. Welcome to So What? their marketing analytics and insights live show I am Katie, joined by my trusty chief statistician, John, here are the numbers sorry, we got.

Chris is once again traveling this week, we’re keeping him busy, we’re keeping him on the road, we’re mostly keeping him out of trouble. So, John, it got cold this morning, like really cold in New England. Daylight

John Wall 0:42
Savings Time and weather both on the suck list. I’m sorry, I just can’t deny that it’s, I agree with all of it.

Katie Robbert 0:49
I wholeheartedly agree. So what doesn’t suck is what we’re talking about today, we are talking about using generative AI for survey analysis. So we’ll sort of go over the basics of how you can use tools like ChatGPT, to do some qualitative data analysis. This came up because every quarter we ask a one question survey. And so the one question survey, we asked people, we give them a text box, they can add in as much or as little as they want in terms of responses. But then we have to do something with that data. And we use it to inform our educational tools, our resources, our content, any upcoming webinars or services. And so it’s really valuable data. But getting to the summary of what’s in the data is something that we often, you know, don’t build in time for or we forget, or we just sort of keep pushing off. And so I actually want to do this exercise with all of our past surveys as well, to see what comes up because I think it would be really interesting exercise. And so, John, we recently asked, if you had money leftover in your marketing budget this year, what would you spend it on? What kind of responses were you hoping that we were going to get?

John Wall 2:12
You know, for us, the big thing with this quarter is we’ve seen a lot of bad signs, you know, and interestingly enough, Chris’s seen some stats as far as things possibly turning around within the last month or two, like things not being as bad as they have been. But a big part of it was just to, you know, validate that data, whether that’s really the case or not, like, do people actually have budget? And if they did, what would they spend it on? And then yeah, it versus obviously, you know, it’s a blind promotion stunt for us. I mean, we’re just like, hey, what will people throw money too, because we have different services we can provide, we have training, we have live events, you know, there’s all kinds of different things we can do. So as we’re looking at year end, you know, burn your budget, before it’s gone offers, we want to be able to offer, you know, what our customers are most likely to jump on. So it helps us with make some product decisions. And then yeah, give us really a lot more feel for just how the market is doing and what’s going on.

Katie Robbert 3:08
And historically, we’ve asked questions around more specific topics like Google Analytics, or, you know, what kind of marketing challenges or education. And so this time, we want to leave it a little bit broad, because John, to your point, we really wanted to make sure that if we had offers or services or content or resources, that they were the right ones, we weren’t just sort of throwing everything at the wall and hoping that it would stick or resonate with people. So let’s go ahead and get into it. So the first step was to actually get the data. So I wasn’t going to have everybody watched me download the data. But so I just I downloaded our data from our website, we use gravity forms, and I downloaded all the responses into a single CSV file. And so what I’m going to do next is I’m actually going to bring it that file into ChatGPT. So I’ve already so you guys don’t have to watch me stumble through typing, because I don’t I don’t know if this happens to you. But I can type fine until somebody’s watching. Oh, yeah,

John Wall 4:10
that’s classic. performance anxiety, that’s, you’re supposed to be a great typer. And when the stakes aren’t there, it’s fine. But yeah, it becomes sweaty quickly, when the whole world’s watching. It

Katie Robbert 4:20
really does. I make most of my mistakes when people are watching, which is I know there’s a whole psychology behind that. But we that’s probably a different show. So in ChatGPT, four, you have to be using chat to be four, and you have to be using the paid version, so that you can have the advanced settings of being able to upload your data. So if you have that, you’ll see this little plus sign right here, that’s where you can add your attachment. So let me go ahead and do that. So first, first of all, you have to have data, you have to have it in some sort of a readable form, I would probably recommend a CSV file for ease of use. And so let me go ahead and add that There we go. Great. Alright, so you can see it just added my spreadsheet. And then we have to build the prop, we actually have to tell the system what we wanted to do. So when I did this a couple of weeks ago, I built a very straightforward prop. But John, what I want to do is get from you what you think I missed, so that we can make this a better, more valuable prompt. And so I started with their persona. So you are a market research analyst. You think that that’s accurate?

John Wall 5:35
Yeah, that’s good enough for this group. Sure. For our first one. Okay.

Katie Robbert 5:39
And then I said, and you asked your audience, if you had money leftover in your marketing budget this year, what would you spend it on? Which is the exact question that we asked in the survey? I then said you were analyzing the data to be easily understood. I realized in reading through this, I didn’t add in a purpose. Like what here’s what we’re planning to do with the data to give a little bit more context. So what would you say we, you want to be able to use this data to tailor content promotion services?

John Wall 6:11
Yeah, well, how about just run down after this is what you need to answer? What are you asking it down there? What’s the rest of the I

Katie Robbert 6:16
basically, say I say, show your work and provide a download link? And I’ll get into what these two pieces of the prompt mean in just a second. Okay, cool. So so we can go through the questions, but I want to make sure I sort of get the full context. Yeah.

John Wall 6:31
And so you’re asking for a chart. Okay, so that’s good. So as far as I don’t know, for our audience, you could say that it’s primarily focused on business to business. You could also say, Yeah, we’re going to use this data to determine what products and services to offer in the fourth quarter. Also identify any trends that would reflect positive or negative economic growth?

Yeah, it’s too bad. Chris is on the road. He’s the master of this stuff. I know, written 10 million props. So this is good. We’re gonna see how we, how we stack up and if we can crash the bus? Well,

Katie Robbert 7:21
you know, and I feel like this is a good exercise, to not have Chris included on because Chris is an expert in artificial intelligence and generative AI and prompt engineering. Whereas I like to think that I’ve represent everybody else who isn’t an expert, but I know enough to get the system running. So I want to show what it looks like from a non expert, a non data scientist perspective, where a lot of us are just sort of like, Huh, what can the system do? What do I need to be able to make it do? So you said identify any trends that indicate economic Kate?

John Wall 8:06
Are you even say indicate the state of the economy

Katie Robbert 8:08
perfect. And again, when people are watching is when I also forget how to spell. So so far, we have you are a market research analyst and you asked your audience, if you had money leftover in your marketing budget this year, what would you spend it on, you are analyzing the data to be easily understood, you will use this data to determine what products and services to offer in the fourth quarter your audience is mostly B2B attached other responses. I’d like to add that in just in case, like, I like to cut down on any ambiguity between the prompt and the system. And so I don’t want this stuff to come back and be like is this is the data that’s attached the data, like the data that you want to use? To me, this just sort of cuts down a step of saying, like, attach to other responses that you’ll be analyzing? You could also add in into the sentence like, are you able to read the data? So that’s also a good QA check. If you’re unsure of how your data is structured? You could say, I’m attaching some data, are you able to read it? And the system will say either yes or no. If they say, if the system says no, you can say what do I need to do in order for you to be able to read it? Or can you help me? And then you can jump into your prompt? You know, so that’s just another way to approach it. I happen to know, my data is literally one column of plain text with a bunch of responses. So I’m not too worried about the data not being readable. So we’ve given it the data, we’ve given the prompt and we said, this is what you need to answer. What are the themes? Can you put these responses into some kind of chart? What doesn’t make sense in the responses? What will people primarily spend their budgets on? Identify any trends that indicate the state of the economy anything Oh, also you would add to that show on anything I mean, so we can keep asking your questions. But is there anything off the jump that you would want to know?

John Wall 10:05
Now I’m all about iterative, you know, it’s let’s pull the lever and see what it spits out. Okay?

Katie Robbert 10:11
So I always like to include show your work. When you include this, when you’re doing some kind of data analysis, what it’s going to give you back is Python code. And so if you’re someone like Chris, who likes to build their own code sets to do data analysis, this is something that would help you start to piece together that code so that you could do it in a repeatable way offline from using a generative AI system. So I don’t actually need this, I just like to include it just because I’m curious. And then something that I learned from watching Andy Crestodina, was adding in provide a download link. And it’s such a simple thing. But it was so genius to me of like, oh, that’s actually really smart. because how else are you going to get the information out of this system? And it’s like provided download link in the sessions like, okay, here you go. So let’s see what happens. So I’m going to do this, it should move pretty quickly, because it’s a very short dataset, a very short prompt. So the first thing that it does is load the CSV file and examine the contents and structure, once we understand the data, we can proceed to analyze it. So again, this is what you want to make sure is that it’s a readable data set. And because I said, show your work, the system is actually telling me step by step, what it’s going to do, which I personally find to be very helpful. You know, you can absolutely skip this step. But for me, it’s a good quality assurance check of is it doing what I expected? Was it vague? Do I need to go back and rethink how I’m approaching the prompt?

John Wall 11:53
So that’s granted, the other show work that’s done is that actually, will that give you a download for the Python? Or what is it? So that is the Python. That’s interesting. So you can just grab that code. And

Katie Robbert 12:03
yep, and you have a copy code down here. And so if you wanted to start to build your own code set, this is how you would do it by asking the system to show its work. Okay, so now it’s going through to analyze responses, I will take this six step process based on the information we gave. And so again, we said, show your work. So you can see all of the different pieces of code that are starting to assemble, which is really interesting. So again, you could use that to start to assemble your own code set. And so now we’re starting to get some information. So it says the categorization of the responses into themes has revealed the following 28 responses that did not fit into predefined categories were too unique to categorize. So we didn’t give it the categories ahead of time. And I purposely didn’t want to do that. Because I feel like if we said it has to fit into these categories, it would be too biased, or it would sort of be too narrow. But if you’re a company that has like a set of products or a set of services, you could say which responses fit into these following categories.

John Wall 13:20
So that’s interesting, it’s really granular, because we’ve gotten that 28 is far and away the largest bucket of stuff.

Katie Robbert 13:29
18 responses related to data analytics, machine learning and AI, seven responses to content creation, five, to SEO and web three, to training and development, three to technology tools, and so on. So for that and it keeps going down the list. So what we’re starting to get is topics, categories of topics, we have data analytics, content creation, SEO, and web training, development, these are the things that our audience have told us are the most important to them. If they had extra money, this is where it would go. So that’s incredibly valuable. So let’s see. Okay, next, I will create a chart to visualize these counts, which will help in understanding what products and service to offer in the fourth quarter. So you can see again, you start to get the code that you can start to piece these things together. So this is a chart. So if you had to put this into a presentation, a PowerPoint deck, the visualization is already done for you. And says here’s the chart showing the frequency of each theme. Based on survey responses, you can see that data analytics is the most common theme followed by content creation, SEO and web. And this is the key you can download the chart image from the following link. And it gave me a link to go ahead and download it. So I’m going to get a JPEG right into my downloads folder that I can just drop into a presentation or an email and say this is what the survey found. And it says next I will take a closer look at the other category to identify any additional themes. And so we We know that this was the largest bucket. But we also know that there’s probably going to be some hidden gems in here. And again, you know, because I asked for it to show the work, you see how the code is starting to come together?

I like this one, some responses are quite unique or off topics such as MDMA shoes, puppies and unicorns, which don’t seem to make sense in the context of the question. It could be jokes or errors. So that’s actually really interesting. So we can start to say, All right, so if we were going to do a more in depth, we could ask the system to like pull out anything with references to the following.

Other responses project deck, suggest activities not covered it by the initial themes, such as sponsorships, social media marketing assistant, and sending small tangible presents to would be clients, which could which could suggest new categories related to engagement or direct marketing. So that’s really interesting. And so we could go back and look at who responded with like sponsorships. And you know, John, that could be an opportunity to say, if you find X number of dollars, here’s some things if you want it to sponsor.

John Wall 16:17
Yeah, that’s, uh, you know, stuff to jump on there. That’s interesting. Yeah. It’s funny, I hadn’t thought about how, you know, over in the Slack community, there’s a lot of frank discussion, a lot of joking. And of course, that translated right over to something.

Katie Robbert 16:33
So to your question about the economic indicators, there are responses that are indicative of caution due to economic concerns, like, I think the economy is going is going to shit. And it’s going to hide, and it’s time to hide away nuts, which may align with savings and investments. You know, so we have to take that with a grain of salt, of course. And again, asking it to show its work. And so now we’re seeing after refining the themes, and recategorize and other responses here, the updated counts, and of course, we got a network error. So that’s unhelpful. So we would have to, you know, start this all over again. But so far, John, do you think that this is a valuable exercise?

John Wall 17:16
Yeah, it is, you know, not surprisingly, data analytics is huge. I think the another step would be have it output a CSV file, you know, by category. So then we could go back and read the specific ones and see get a better feel for what’s going on there. But and yeah, I don’t know. I mean, what’s your thought my gut was, you know, there was a whole long tail of ones that are three or less, I would just cut that off unless there was something else going on?

Katie Robbert 17:42
Well, and I think that those are questions like, if you’re purely using generative AI for this analysis, I think that it’s good to add that into your prompt. And so you know, if you sort of give an initial look to the CSV file and be like, it looks like there might be some junk in there, or it looks like there might be some really short answers, you could probably include in the prompt, only look at responses that are five words or longer or disregard any responses that are one word answers, or disregard any responses that include the following terms such as puppies and unicorns, which I’m pretty sure it was my response when I was testing the survey.

John Wall 18:24
I mean, it’s like any import export of data, like every time you do it, you end up with a grocery list of like, okay, next time we run this, these are the 12 things we got to add.

Katie Robbert 18:34
So now, what’s interesting is, so because we got the numeric error, we are regenerating it. And the way that it’s putting together its responses is different from what we just got five minutes ago. So I’m going to let it see if it will finish. And then we can go through what the responses are. But I find that to be really interesting that it’s presenting the data a different way this time.

John Wall 19:00
Yeah, that’s really interesting for you know, long term testing, and then we just have I know, some models have a seed value that you can set so that you could get, you know, closer aligned. But yeah, it is interesting how it’s really a randomizer in some ways, right? I mean, you’re you’re getting a different set of probable answers. So yeah, that’s a little bit shaky. For folks pursuing the truth. You know, it’s a little bit creepy that it comes up with a different take every time.

Katie Robbert 19:30
Well, you know, and it’s funny when, earlier this year, when Chris and I were going over a lot of these systems, one of the questions that got came up a lot was if I ask your prompt, and you ask a prompt, and it’s the exact same prompt, will we get the exact same answer and the answer is no. We’re two different people on two different computers. And even if it’s the exact same product, I think we actually did this exercise on one of Live streams, you’re going to get different responses, even if you hit go at the exact same time, because the amount of data being collected by these systems at any given second is constantly changing the back end database. And so there’s no possible way for you to get identical answers every single time. Okay, so it did finish this time without an error. So that means that we’ll be able to ask it, new questions. So let me go back up to the top, you see the chart again. So let’s see what it did this time. Alright, so again, it sort of went through, let’s start by loading and examining the contents provided in the data. So that’s great. And you can see the code is there. And then once it did that, it said, here are the first five responses from the dataset. So it’s validating that it can read the data. So we could look at our data set, go like Yep, those are the first five responses got it. To analyze this data, I will do the following things, which is just repeating what we asked it to do, which is great, because then it means that we’ve structured the prompt in such a way that the system understood what we were asking. So it says, First, I will proceed with a qualitative analysis to identify the themes, then we’ll quantify these themes and visualize the results, which is entering it didn’t do this last time. And so if you’re wondering why I couldn’t tell you why we got a different, you know, set of responses this time than we did a few minutes ago. But that may be something to definitely look into when you’re testing this on your own is, if you’re not happy with the first set of results, try it again, see what happens. So we see the most common words found in the responses include terms like marketing, AI, tools, training and money. Make sense. However, the frequency analysis includes many common English words that are not indicative of specific themes. To get a clearer picture, I’ll need to read through the responses and categorize them based on the context context in which the words are used. So that it says I will now do this. So the thematic categorization revealed the following distribution, AI and machine learning marketing tools and technology, financial caution, training, and education and other. So this again, we start to get these categories which can turn into topics where we could create content, or take a look at our services and see what aligns with AI and machine learning, or marketing, or tools and technology, tech stacks, those kinds of things. And again, it says the other category did not clearly fit. So that’s the same as last time, but it’s approaching it differently this time. The other category, responses to other included the following thing. So this time, it gave us a better list of other things. And this, this list actually helps us think through like, what else are we missing. So we have creating promo videos, start a community sending small tangible presents conference registration, I would throw it back into the company, which makes sense. And then based on the responses seems that could be additional themes, such as client relations, research, visual advertising, corporate savings and professional development. So get more the refined categories. So it took all of the data that had above and refined to the categories and said, these are the categories that I want to give you that I have found in your data. This is my answer to you. And then it created a visual representation. And again, you can see it’s a very clear chart of the frequency. So if you had to bring this into a presentation, you’d be able to say like based on the responses, this is the category that we need to be focusing on. And you get the download link for the chart, which is great. And it says based on this analysis, it seems that people would primarily spend their budgets on AI and machine learning enhancements, followed by various marketing efforts. tools and technology are also significant, but to a lesser extent. So this is something we can do something with John.

John Wall 24:07
Yeah, it’s interesting, though, I don’t like the categories as much as the first round. You know, I think that would be one thing that, you know, we could take the categories from the first prompt and force those to get a take on it. But yeah, it’s because you know, the interesting, it’s interesting to have just marketing as a general topic and tools. I mean, obviously, those show up usually and everything. But then having AI and machine learning, it’s its own category. That’s interesting, because that talks to a lot of the stuff that we’ve been seeing, as far as, you know, public speaking for AI requests have been through the ceiling. And you know, there’s a lot of stuff going on on that front. So that does match that that does kind of validate that the stuff we’ve been seeing over there is the way the markets gonna go at least for the next quarter or two, probably.

Katie Robbert 24:51
I would agree with that. And so you had asked about the economic trends, so this time it said the trends indicating the state of the economy presence of categories like financial caution suggests that there is a sense of economic uncertainty among the respondents. Now, I want to give the caveat that we have less than 100 responses in this dataset. So I would not say that it is statistically significant in terms of accurately representing our audience to say this is exactly what they want. For us, this is a guide. If we wanted a more statistically significant representative sample, we would obviously need more responses in order to say yes, we feel confident saying that these are the things that people want. And at the end, it says, Would you like to proceed with any further analysis or have any questions on this data? So John, one of the things that you had asked was, can it provide a CSV file of the counts, right of the topics?

John Wall 25:51
Yeah, or a list of all the responses and the categories that they’re in?

Katie Robbert 26:02
I’ll have the sponsors and the glories. See the whole people watching me type thing that they are in, please provide a download link to this file. John, when you write prompts, do you say please, and thank you?

John Wall 26:28
Yes, I want the robot overlords to pass the House over when they come, Kelly.

Katie Robbert 26:35
This guy was nice to me, he wasn’t a jerk. So it says I’ve saved the categorize responses to a CSIS a CSV file, you can download it from the following link. So I’m going to download it. And let me pull it up. So okay, that’s interesting. Oh, you have to share that one. If you want to. Yeah, I’m gonna see if I can share this one instead, share screen window, categorised responses. Alright, so I got exactly what I asked for, I got a very basic spreadsheet with here, the responses, and here are the categories. And so I can just go ahead and sort these and see everything that falls into what ChatGPT thought was AI and machine learning, Client Relations, and so on, so forth. So that’s actually really helpful too, because you can then go back to your original file, so we have emails and contacts in our files. But knowing that we’re using a public system, like ChatGPT, it’s not firewalled in our own server, I didn’t want to provide any contact information, any personally identifiable, identifiable information does not belong in a system like ChatGPT. So I purposely cut all of the emails Oh, so I would not, I would caution you do not use systems like this, in order to do that kind of analysis. With that data. Our data set is small enough, that we can then go back and re pull it again and say, who responded to what?

John Wall 28:16
Yeah, and I mean, I really would love to just for a second, like, shine a light on this, because this is amazing. Like, because we’re dealing with, you know, 40 Odd responses here be like, Alright, whatever. But if you had 5000 responses, just to be able to do this, this is fantastic, you would be able to chew this up. And then yeah, privacy is totally concerned. But the way you can always deal with that is, you know, whether in your Salesforce or Hubspot, or wherever you’re in, your records do have a unique key. So when you’re doing your export, you take the key, and then that way, you can run this report, and you can match it back and upload it back in. And so now you can run full reports and do all kinds of call lists or whatever, based on these categories. So that’s a really powerful way to get through a project that could be a massive pain, you know, like a job like this could really suck if you have 10,000 responses, you know, or if you were hitting Chris’s newsletter, and we’ve got 25,000 responses, you know, this could really be a game changer for you to identify what the top 10 things are that need to have action taken on them.

Katie Robbert 29:19
And no, and I think that that’s huge, because it really is the goal of artificial intelligence, especially generative AI is to make things more efficient. And so I feel like this is a really good example of, instead of me fumbling through Excel and trying to figure out, you know, pivot tables and creating columns and copying and pasting, you know, we’ve spent, I mean, less than 30 minutes twice now going through ChatGPT, having it do the work and I now have a usable spreadsheet. I have a bar chart that I can present I have a rough summary analysis, all of which I would say go had and you know, make sure you do your QA and edit it yourself. But I feel like it is really helpful to get you, you know, 90% of the way with this kind of analysis. Yeah,

John Wall 30:14
and it’s fantastic for exception analysis to you know, as you’re looking at those categories, if you agree with those, you can spot check and match them up. And now you only have to deal with the other ones, you know, and so this would be a great pass, you could take and go through the others come up with the next batch of categories, or see if stuff needs to be, you know, properly aligned, or whatever.

Katie Robbert 30:32
Well, and you know, so one thing that you could do, you know, with that as you’re going through it is, if you say, you know, this doesn’t really fit under saw a direct mail campaign that doesn’t really fit under AI, AI and machine learning, you could recategorize it recategorize, all these even change the names of the categories, and then re upload the file into ChatGPT. And do the exercise again, but it’s still not going to take you as long as if you were trying to do this manually. And ChatGPT gave you the code so that you could start to build it yourself.

John Wall 31:05
Yeah, right. You know, worst case, you’d be saving an updated prompt. But yeah, if you really want to get crazy, you could go into the Python and write that.

Katie Robbert 31:16
And you know, and for people like Chris, that’s exactly what they want to be doing. And so he uses it to build code for different analysis projects. And he’s found that it, you know, I think he stated it gets him like 85 to 90% of the way there. And then he just has to sort of put all the pieces together. So for this analysis, John, are there other questions that you would want to ask based on this data? Like, What haven’t we covered?

John Wall 31:43
Yeah, you accurately categorizing is really 99% of the work, right? Because ultimately, we do get to the point where we have to read through the analysis and come up with some hypothesis of what we want to test. Yeah, that’s always a great question. Just ask what it thinks it should do with it, see if it comes back with any wisdom or stuff that we had forgotten?

Katie Robbert 32:03
I personally have found that these responses like when you ask a question like that the responses tend to be fairly generic. But to your point, it is sort of like a good starting place of like, oh, okay, I hadn’t even thought of that, for example. So it’s saying strategic planning and marketing and sales initiatives, investment decisions, customer engagement, economic insight, content development, r&d, prioritization, feedback, loop product, building, sales training, measure ROI. Like, there’s a lot of things that ChatGPT thinks we can do with this dataset, which is really interesting. Are any of these surprising to you?

John Wall 32:42
No, that’s a, you know, it’s funny, a couple of weeks ago, I was running a similar set of queries with Jenny on crisis communications, and it was the same deal, you know, I came up with a list of like, 16 different topics. And so yeah, it’s a great way to cast a net, and make sure that you’re not missing some major category that you could have been taking advantage of, it’s a great safety net for your analysis.

Katie Robbert 33:06
The other kind of question that I think doesn’t get asked a lot to the systems is, What haven’t we done with this data? Or you could also say, what else could you do with this data? And so this will be interesting to see what the system thinks we could be doing. And then if you wanted to dig into any one of these, you could say what you could ask ChatGPT, what do I need to do to get started?

John Wall 33:43
Yeah, and, you know, for this one question thing, it’s, you know, a lot of this stuff doesn’t apply. But man, if you had some kind of huge data set, or if you just wanted to dump some anonymized data out of your CRM system and run some of these reports, that would be killer. You know, as far as sentiment analysis across the board. And longitudinal do, that’s, you know, we have the past 10 quarters of, you know, one cues. So that can definitely be done to we can see what the trends are on that front. And where that goes.

Katie Robbert 34:15
Predictive modeling is something John, you and I talked about last week. So to your point, if we had a much larger data set, we could do predictive modeling with that data to say, Okay, what does this look like? If we forecast it out, we have a customer journey mapping, content gap analysis, that one makes sense, competitive analysis, resource allocation, workshop and training, community engagement. So some of these do to your point, do feel like a little bit of a stretch, but it is, again, to your point, it cast a really wide net of, you know, What haven’t we thought about or that’s really interesting, or maybe it sparks another idea?

John Wall 34:55
Yeah, this is funny. Shout out to the ninja. They’re talking about new custom GPU He were just talking about that earlier this week. As far as price drops, huge price drops across the board. It’s actually much cheaper now, and then some vision stuff being able to upload images. And yeah, there’s a whole bunch of stuff rolling out in the new release. That’s Chris is very excited about he got one. I don’t know.

Katie Robbert 35:19
Any anything else we should know about this dataset. And here’s the thing, you can keep asking as many questions as you want to ChatGPT and it won’t get exhausted with Okay, stop asking questions like it is there specifically to do this kind of work for you. So you can exhaust it like, you can try to exhaust it with all of the different questions like any questions you have, this is a great place to ask all of them. You can anticipate the questions that might come up from your steering committee, your decision makers to C suite, ask those questions and see what ChatGPT gives back so that you can at least be formulating an answer knowing that those questions are likely to be asked. So let’s see. Anything else we should know about this dataset says response bias. Okay, the data might be subject to response if the survey was not distributed evenly. That’s a good one for us to think about. Sample size and representativeness. We did talk about how it is a smaller data set. Survey context, the way the question was phrased. Making sure that, you know, we don’t have that survey bias, data granularity, qualitative data complexity potential for misinterpretation, qualifiers and non Response Time Pro relevance, ethical and privacy considerations for validation, missing contextual data, all really good things that we should be thinking about. So you could say, Can you write a methodology for this analysis? I can use publicly. Yeah,

John Wall 36:54
that’s the heartbreak list there. All researchers hate that that’s throwing all their hypotheses into the dumpster.

Katie Robbert 37:02
And it says, certainly, here’s a sample methodology. And again, you want to make sure that you don’t just copy and paste these things into but you know, John, as you were saying that if you have a much larger dataset, this is the type of information you would also want to include. And having the system do this for you, especially since the system did the analysis. Having the system also write the methodology is a really good idea so that you can say, Okay, this is what it says it did. Does this align with the way that you would expect it to work? It’s actually going pretty well in depth, which is great.

John Wall 37:36
Yeah, and this is just classic. Like nobody wants to write this document, right? Like having this put together, the first draft for you is a huge timesaver.

Katie Robbert 37:47
All right, so we have all the things objective, the primary goal was to understand the preferences of B2B audience regarding their allocation leftover marketing budget, that’s accurate, we aim to identify key themes. That’s accurate data collection. Yep, that’s accurate data processing. That’s accurate. So again, this is sort of a good way to also check to make sure that ChatGPT analyze the data the way that you would expect it to. So this is just even if you don’t plan to use this publicly, it’s a good way to approach it. So you can say, Okay, this is what we did. Because especially if you’re presenting this data to someone else, they’re likely to ask like, Well, how did you get from A to B? Data Export analysis limitations? That’s great conclusion. Can you put this methodology into a down lodo lootable. Doc for me. Please provide download link.

What I’ve learned through doing these exercises is there’s really no limit to what you can ask the system to do to help and assist you to taking it things farther. So let’s see if this is the thing that breaks it.

John Wall 39:11
Yeah, does it actually will it do a dot docx file?

Katie Robbert 39:15
I don’t know. I know it’ll do a CSV. I know it will do images. Yeah. So I don’t know thing

John Wall 39:21
to see if it. It’s afraid of Microsoft. They shouldn’t be given the ownership status of OpenAI. But,

Katie Robbert 39:29
I mean, even if it’s a plain text file, I mean, that’s a good point, you could say, you know, please create a dot txt file that you can just open within Word or some other processing software. But yeah, it looks like this one may have been thinking a little bit more. So it’s also a good way to sort of test the limitations of this system that you’re working with. Here we go. Here we go. The methodology document has been created. You can access it through this downloaded You just pull it up, and I will. Oh, hey, look at that. Let me share my screen show everybody what we got, I think this is a pretty good place to start to wrap up this conversation, because I’m actually really impressed by this go. I got a document. Wow. All right. So yeah, there, it sorry, it’s pretty well formatted, very easy to copy and paste, I could just make my edits directly in here. And so, you know, I could go back through and say, you know, can you put all the pieces of code together into one, you know, file, so I can download that. So there’s a lot of different ways you can approach getting the information out of ChatGPT, you could ask it to put the summarization the analysis into a document and download that. So again, you don’t have to like, copy and paste everything, just tell the system what you want it to do in very clear language, and it should be able to do it for you. But in less than 45 minutes, John, we have analyzed our data, we have started to understand what our audience wants to spend their money on, if they had it. We have a methodology statement, we know that there are limitations with the data. This is all within 45 minutes. I mean, that’s pretty good.

John Wall 41:25
Yeah, being able to clean up and categorize the data in you know, five minutes is just game changing.

Katie Robbert 41:31
So what else would you add? Before we start to let to give ChatGPT a break anything else?

John Wall 41:37
No, I think that’s a good, you know, place to run with. The thing with this, too, is, you know, you see that and then over the next three days, you come up with more ideas for like, Okay, how could we use this? And where could it go. And so this is definitely it’s great to play around with it. And anything that can speed up showing the data formatting and different file types, all those headaches that slow everything down, it’s great to find ways to make that happen faster.

Katie Robbert 42:01
I would say too, and so what we saw was like limitations representativeness, you can start to ask the system, those questions of how many more responses do I need to get in order to have a representative sample? And it could start to outline those plans for you of if I have 70 responses, but my audience is 10,000 people? How many responses do I need to get in order to have it to be statistically significant?

John Wall 42:27
Yeah, 70 responses if you’re if you’re getting, you know, under 30, in the individual categories, yeah, you’re nowhere near where you need to be.

Katie Robbert 42:39
Alright, so we covered bringing your qualitative data into ChatGPT. To do a basic straightforward analysis, I have my charts, I have my summaries, I have my methodology, I think I’m good to go. So if I had to take this to a meeting, I would feel pretty confident to say I at least have talking points to say this is what we’ve learned from this data, we can certainly do a deeper dive. But now we can create a content calendar, we can, you know, think about promotional emails, highlighting certain products, we have one to one outreach that we can do. So there’s a lot of next steps that we can now take from this short amount of time that we just spent doing this exercise.

John Wall 43:23
Sounds good? Oh, yeah. You know, we had one other question asking about security risk for getting the data. And yeah, it’s absolute security risk, you don’t want to be just randomly loading, you know, stuff laying around your company. And we have clients that we’ve worked with on this stuff as far as being able to set up something in house on your own server. So then you can upload all the goodness you want. And then you have the added benefit of that’s your training set to so that you can adapt the model to your business specifically, and not share it with anyone. So yeah, good question. Thanks for that. Yeah.

Katie Robbert 43:51
So if you’re using public systems, if you’re subscribing to ChatGPT, and it’s not firewalled with on your own servers, you’re not building your own own large language models. As much as you can don’t include any of that information. John, you made a really good point earlier about systems like Hubspot and Salesforce, have unique identifiers attached to each record. So use those numbers instead, when you’re doing the analysis. So then you can re import the data into your CRM system, whatever system you’re using, and it matches back to those individual customers rather than using their names, their emails and their company’s credit card info. Yeah. Yeah. Don’t do that. All right. Any final words?

John Wall 44:34
Get back to work. There’s plenty to churn through. I got it.

Katie Robbert 44:39
All right. Well, we’re gonna be on hiatus for the next two weeks. We’ll be back here on November 30. Until then, you know what, stay classy San Diego.

Christopher Penn 44:53
Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it. For more resources and to learn more Check out the Trust Insights podcast at trust insights.ai/t AI podcast and a weekly email newsletter at trust insights.ai/newsletter. Got questions about what you saw in today’s episode? Join our free analytics for markers slack group at trust insights.ai/analytics for marketers See you next time.

---

Need help with your marketing AI and analytics? Get help with your AI strategy » Learn generative AI for marketers with our new course » Prove the ROI of your marketing with our advanced analytics services » Just got a "silly" question you want an answer to? Ask us here »	You might also enjoy: The Woefully Incomplete Book of Generative AI Empower Your Marketing With Private Social Media Communities The Trust Insights Leadership Team AI and Copyright Law: How Copyright Applies to AI-Generated Content INBOX INSIGHTS, November 8, 2023: Solving Problems with Predictive Analytics, Testing AI Models
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! Click here to subscribe now »	Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. Click here for Google Podcasts » Click here for Apple Podcasts » Link for all other podcasting software »