Mailbag Monday: Risks of ChatGPT Insights?

Katie and Chris answer your marketing, data, and AI questions every Monday.

This week, Charlie asked, “For the case of using ChatGPT because someone is not savvy enough to draw out insights, it may be “dangerous” in the sense that if we aren’t skilled enough to draw out initial insights we may not be able to understand when ChatGPT gets it wrong. What are the risks of ChatGPT insights?”

Tune in every Monday to get your question answered!

Got a question you want us to answer? Reach out to us at trustinsights.ai/contact or join our free slack group, at trustinsights.ai/analyticsformarketers

Subscribe to our weekly newsletter to catch past episodes at trustinsights.ai/newsletter

AI-generated transcript:

Katie Robbert 0:00
Welcome back to another episode of mailbag Monday where Chris and I are tackling all of the questions that we get in through our mailbag. So Chris, what is in the bag today?

Christopher Penn 0:09
Surprise, it’s another ChatGPT. Surprise, surprise, Charlie asks for the case of using ChatGPT. Because someone is not savvy enough to draw out insights. Is it dangerous in the sense that if we aren’t skilled enough to draw initial insights, we may not be able to understand when ChatGPT gets it wrong? And the answer is yes,

Katie Robbert 0:28
yes. Absolutely. Yes. Chris, this is something you and I have talked about a lot. And I covered recently on the martec coffee talk, when asked about ChatGPT, what are some of the risks? And one of the biggest risks is a lack of subject matter expertise, when using a system like this, and so, you know, we were recently talking about, you know, you know, is ChatGPT, something that only marketers are talking about? Or is it more, you know, broad? And the answer is it’s more broad, everybody’s talking about my parents who are in their 70s and retired are talking about it. You know, Chris, I wouldn’t be surprised if your parents have questions about it, because they’ve heard something about it. It’s big in the healthcare space, and ChatGPT. And systems like it have a lot of really big potential to be helpful, if you know what you’re doing with it. And with that, I mean, so an example. And we’ve brought this up a few times before, we were asking the system to write a simple blog post about SEO in 2023. And the results that gave Chris, as you mentioned, because you understand SEO really well, I’ve been doing SEO for a long time, is that the recommendations that it gave were about five years out of date, whereas when I looked at it, I was like, oh, that’s decent advice, you know, that could sort of pass muster. And it comes down to subject matter expertise. And so to Charlie’s question about, Are there risks of not understanding the insights that ChatGPT is giving you? Absolutely. That’s true of any analysis, or sort of any outcome? If you don’t know what it’s supposed to be? Then how do you know if it’s wrong?

Christopher Penn 2:13
Exactly. And this is where this is going to be true of every single domain. So here’s an example. This is something that I had asked it the other day, just out of curiosity, I said, Write me a recipe for doing the clam chowder. And he came up with this recipe, right, eight slices of bacon, two tablespoons of butter, a large IBM three celery stalks, garlic cloves, a bunch of herbs, four cups of chicken broth, diced potatoes, two cups of heavy cream. Here’s the question. If you are a professional cook, is this right? Right, the process the instructions, you know, cook the dice, bake and remove the bacon set aside, add the butter pot with bacon, fat, minced garlic, stuff like that. I think it’s slightly off on the liquid measurements, not by a ton. But I think it’s slightly off. I’ve cooked recipes similar to this in the past. I want to say I think it’s, and I think it’s slightly off also on the amount of flour that you’re supposed to use to get the right consistency. Now, obviously all of that is to preference and things. But it looks credible, until you actually go and do it. And then you find out actually there were some issues. But unless you are a professional cook, or someone who has cooked this recipe a lot, it’s not immediately clear that there might be flaws in it, even if those flaws are relatively minor.

Katie Robbert 3:34
Well, and this is where at least for this for a New England Clam Chowder, I feel like the liquid amounts are subjective. So you might like a thicker chowder as opposed to a thinner chowder. But there’s really no risk. Whereas if you’re using a system like ChatGPT, to help you write code, which Chris is something you’ve mentioned, it can do, especially if you bring it into Microsoft Visual Studio, if you don’t know what the code is supposed to stay supposed to say, rather than you don’t know where it’s breaking down. And it could be a very expensive endeavor. To have a system like ChatGPT help you write the code, if you don’t know what the code is supposed to be doing. You know, I can say I want it to write code that’s going to write me a recipe. But if I don’t know, the code itself, and the system isn’t working, then I don’t know how to fix it. And so it’s the same with you know, one of the things that Chet GPT-4 rather the system GPT-4 will be able to do and we’ve been testing some of this is to give it a chart and say what is this chart say? What are the insights from this chart? Well, if you don’t understand the data in the first place, then it’s not a shortcut to getting insights because the insights might be completely off especially if you’re not giving the system the context it needs in order to pull out the correct insights.

Christopher Penn 4:57
Backward. Let’s do another one here this time because this is a really good test for marketing stuff I said, I gave it a long long prompt to interpret the default channel grouping data from Google Analytics. And I said write me some recommendations. So in this case, organic search in the data is 57% of it in the have organic social is 57%, organic 56.9%. And it says, For recommendations organic socials, leading marketing channel accounts with more than half the website’s traffic is a positive sign. Important to diversify, to avoid over reliance on one source, no marketing channels to see more than 50% of the results. We’re diversifying other channels with paid social display advertising, affiliate marketing, influencer marketing, content marketing, which is fine. Organic video only drove nine sessions. This is just a video contest, not effectively reaching the target audience. Now, I happen to know whose data this is this is from the Save Warrior Nun campaign. Its recommendation here is incorrect. Because it’s not that the video content is not reaching the right people is that we forgot to put links in our videos, like 79 videos, there’s no links any of them, right? Oops. Yeah, I mean, this is this is what you get when you have a bunch of volunteers doing something. But if you were if you didn’t have subject matter expertise, in not only Google Analytics, but also in what the campaign was was not doing, you might say, Yes, this is an accurate recommendation and then hand off to a stakeholder, the stakeholder yells the video team like hey, you need to create better video not that’s not the problem. So, you know, to answer this, this question that Trailhead Yeah, it is a very risky system, if you don’t know what you’re doing.

Katie Robbert 6:42
Absolutely. So yeah, the bottom line, if you are going to use a system like ChatGPT, or really do any kind of analysis or anything, make sure that there’s a certain level of subject matter expertise in it before going down the road. Because you know, it could be something as simple as a clam chowder recipe, which is very low risk, or you could be making big financial decisions for your marketing team.

Christopher Penn 7:07
Exactly. If you’ve got questions from mailbag Monday that you want to ask, drop them into our free slack channel. Go to Trust Insights, AI slash analytics for marketers, where you have over 3000 other marketers asking and answering each other’s questions all the time. And if you want to catch up on previous episodes of mailbag Monday, make sure you subscribe to the Trust Insights newsletter, go to trust insights.ai/newsletter, where you can find back issues. Thanks for tuning in. We’ll talk to you next time.

Transcribed by https://otter.ai