So What? Marketing Analytics and Insights Live
airs every Thursday at 1 pm EST.
You can watch on YouTube Live. Be sure to subscribe and follow so you never miss an episode!
In this episode of So What? The Trust Insights weekly livestream, you’ll learn about prompt engineering basics and how to improve your AI interactions. You will also understand why current large language models require a different approach than older versions.
Watch the video here:
Can’t see anything? Watch it on YouTube here.
In this episode you’ll learn:
- A reusable framework for prompt engineering
- What’s changed with prompt engineering in 2025
- Pitfalls to watch out with the new models
Transcript:
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Katie Robbert – 00:34
Well, hey, everyone. Happy Thursday. Welcome to So What? The Marketing Analytics and Insights live show. I am Katie, joined by Chris and John. How’s it going, fellas?
John Wall – 00:42
Hello!
Katie Robbert – 00:43
Chris, you look concerned.
Christopher Penn – 00:45
I think something was up with my audio during the intro because the music was really distorted, but everything sounds fine now, so I’m good.
Katie Robbert – 00:52
Okay. I mean, you’ve been having camera issues, audio issues. I think really what it comes down to is you likely have a poltergeist. That’s the only logical explanation that I will accept.
Christopher Penn – 01:04
Sure.
John Wall – 01:05
Don’t get sucked into your television then. That’s the only advice I can give on that front.
Katie Robbert – 01:11
One of your kids says they’re here, then just back away. Speaking of things, it’s a terrible segue. I didn’t, I didn’t have one for that. This week we’re revisiting prompt engineering basics. And so we wanted to go back to where we started with prompt engineering because what we found, and Chris, you specifically, is a lot of people got very excited and built frameworks and started prompting two years ago when generative AI became widely available. And we have to give the system some instructions. We have to tell it what we’re after. That’s where we at Trust Insights were able to come up with some of our. Oh goodness, I can’t even find them now. Some of our basic frameworks of race. That’s where we started. So role, action, context, execution. I don’t see it in our tags, but that’s okay.
Katie Robbert – 02:11
And then, you know, we took it a step further with pair and then we blended it together and made Repel. Because why make it easy? But Chris and I would like to hear you speak to a little bit of sort of what you’re seeing. But it feels like as the large language models, the LLMs, are evolving, people’s skillset of prompting is not evolving with it. And so you brought this to us, and so that’s why we said, let’s revisit prompt engineering basics. So what is it that you’re seeing that makes you feel that people are stuck in 2023?
Christopher Penn – 02:52
A few things. First, people are still giving absurdly short prompts. There’s a great blog post about B2B marketing. Right. You know, that’s. It is 2023 still. And you see this especially when you start looking at content on places say like LinkedIn, where there is just so much slop, particularly in the comments looking at you, comment bots, that it indicates a lack of a level of sophistication. So that’s one big thing. The second big thing is that in, you know, ChatGPT and GPT4, GPT4 came out in June of 2023. It is a toddler, right? When you think about this, it is still in the terrible twos. And yet all the platforms models have evolved from face rolling idiot two years ago to above PhD level capabilities today.
Christopher Penn – 03:53
And that means that the way you prompted a face rolling idiot and spelling everything out is different now than how you would prompt a PhD. You still need to provide things like context and details and what your desired outcomes are, but you no longer have to rigidly adhere to one strict prompt framework because the models are smart enough, if you tell them to ask you questions. And this is something where I’ve seen a lot of people run into trouble because they’re doing what’s called cognitive offloading, which is where you say, you know what, I don’t want to do this task. I’m just going to have AI do it. And in doing so, you kind of give up your own agency.
Christopher Penn – 04:38
And Katie, you and I have talked about this on the podcast where people are, you know, I think we actually had an episode saying, is AI making us lazy?
Katie Robbert – 04:48
We did. And so it sounds like, if I can paraphrase for you, that despite the large language models being at PhD level,
Christopher Penn – 05:03
we’re still talking to them like they’re face rolling idiots. We are. And more importantly, because they’ve evolved so much, we are not leveraging that those capabilities. So one of the things that we talk about a lot is this between offloading and enhancing. Offloading is when you are asking AI to do things and you’re like, bye, I don’t want to do this anymore. Which is fine for some tasks like expense reports, but enhancing is when AI is making you do things and making you think more, because the more you’re thinking, the better you get as a professional.
Katie Robbert – 05:38
So I want to call up this comment. How many prompters know how to talk to a human PhD, let alone an AI version, which I feel is tongue in cheek. But at the same time, if you’re positioning these large language models as PhD version, I can actually see how that could be intimidating. Like just a real example. Chris, we run into this a lot. People in general struggle with how they feel they should be talking to you because of the level of expertise you have in AI. They themselves don’t want to come across as uneducated or unprepared. And so they just kind of don’t bother at all. And so I feel like, it’s sort of the same thing. Like, if you’re saying these are PhD level, you know, language models, use them as such.
Katie Robbert – 06:30
How do we take the intimidation factor out of it and make it approachable so that somebody doesn’t feel like they’re not smart enough to do it?
Christopher Penn – 06:41
The good news is that unlike a curmudgeonly human like me, the models are much more polite and helpful than I am. What we need to do is we need to be clear in our thinking. And I will show what I call the magic trick to the extent that there is anything magic in AI, because there really isn’t. But this is the magic trick. If you take away nothing else from this entire live stream, it is to paste this sentence at the end of your prompts, particularly your initial prompt in a conversation to say, ask me one question at a time until you have enough information to complete the task. So here’s things, just a simple example. Let’s go into stock chat, GPT, nothing special here. Let’s make it a little bit bigger here. And I’m going to give it a terrible prompt.
Christopher Penn – 07:28
I’m going to say, write me a blog post about B2B marketing and let’s see what it comes up with. And here’s a complete blog post draft for you.
Katie Robbert – 07:40
I mean, it did what you asked.
Christopher Penn – 07:41
It did what I asked. And it’s, and it’s not wrong, but it is slop. Like this is. This is just generic garbage. Now, list.
John Wall – 07:52
You have a list.
Christopher Penn – 07:54
I have a list. I have a great list. Now let’s do this. Write me a blog post about B2B marketing. Ask me one question at a time until you have enough information to complete the task. Now we hit go. Go.
Katie Robbert – 08:11
Yeah, no, go ahead. Sorry, I was going to make an observation. My observation is it. I find it interesting because I personally, I. I would get annoyed with the one question at a time strategy. Whereas, like, I just want to know what all the questions are up front. And I feel like that comes down to the type of person you are in terms of how you consume information. And so, you know, we’ve joked, slash, not joked about how you can’t ask certain people more than one question at a time because they’re only going to pay attention to the last question you asked or one part of the information. And so you get frustrated continually following up and being like, what about this other thing? And what about this other thing?
Katie Robbert – 09:01
And so I guess my question to you, Chris, my one question to you is, do you get the same result from a large language model if you use the ask me all of your questions upfront versus ask me one question at a time? Meaning, does the info. Do you think the information that it’s compiling and the questions that would ask would change given the strategy?
Christopher Penn – 09:27
Yes, you get slightly better answers when you have one question at a time. And the reason for that is every time you’re going back and forth and providing data, you’re adding new tokens to the context window, you’re adding more words to its memory. And as if you have it generate all the questions all at once, it doesn’t have a chance to refine the subsequent questions based on your responses to triangulate faster on what you want. If you do one question at a time and you give answer, it can triangulate faster and ask deeper questions faster because it can make some inferences based on your answers. Now, is it a substantial difference? Depends on the task, but usually not.
Christopher Penn – 10:06
What I will often do, for example, before I go and work out is I will have it generate a long ass list of questions and I will take that with me with my voice memos app and you know, groan and mumble as I’m on the treadmill through all the answers and upload those back. But then I typically have two or three more cycles of, okay, here’s all the answers I gave you. Now give me another set of questions and stuff. Whereas if I’m doing iteratively when I’m sitting at my desk not sweating to death, I can usually get to those second and third level questions faster.
Katie Robbert – 10:37
Okay, John, question for you. One single question. How long would you say your average prompt is when you’re using generative AI?
John Wall – 10:51
Yeah, I try to keep it as simple as possible for the first run, but I have used this trick every time of that’s kind of the big key is you realize how complicated the prompt is or not based on if you throw it a quick question and if it only gives you one or two things, or if you throw it a question and it gives you 10 or 15 questions, it starts mining deep down the rabbit hole of wherever you’ve sent it. But yeah, it’s kind of a mix. You, you want to be directed to the point so that it can be on target, but you also have to give it enough information so that it won’t just come back with slop and, you know, will at least go down getting closer to the target.
Katie Robbert – 11:35
So as we’re trying not to get frustrated with asking one question at a time. Because I mean, and this is just my own personality, like an N of one. I’m like, let’s just get to the point faster. Like, can we just move it along? But I understand what you’re saying and how like, you know, it makes the output more detailed, better, etcetera. Etcetera. So we go through well, I guess, you know, let’s go back to the beating because you said one of the things you said was you no longer need to use any kind of prompt engineering framework like Race or Pair or Repel. So are those out the window? Can you really just give it, write me a blog post about B2B marketing. Ask me one question at a time.
Christopher Penn – 12:25
So the frameworks are good for us as humans to remember what it is that we should be including. They are still very valuable because we, the human operators have issues remembering things and so they’re good. I’ve actually defaulted back to recommending Race as a general framework to people because it’s easier for them to remember. And with the advent of tools like Deep Research, Repel is kind of overkill these days. The however, with the models themselves, if you are using a reasoning model, so using ChatGPT 5 thinking, if you’re using Gemini 2.5 Pro, Claude, and its series of models in the four or 4.1 generation Deepseek R2, Alibaba Quen 3, 235 AB, all of those models have reasoning all which is fancy for it does a rough draft before it presents the answers on screen.
Christopher Penn – 13:18
If you decompose what’s going on in the reasoning of those models, you can see they’re trying to figure this out themselves. So they’ve got their own internal logic that they’re trying to reason out. Like, well, what is this user trying to do? What are. What are they thinking about? What’s the, you know, what’s going on under the hood that I should be attempting to guess at. And as a result, you don’t need this the strict level of prompt framework that you did in 2023 when no models had any reasoning. And what you see is what you.
Katie Robbert – 13:56
Does every model have reasoning built in at this point?
Christopher Penn – 14:02
For the ones that you’re going to run into as a standard user, the answer is yes, with a lot of asterisks.
Katie Robbert – 14:09
Okay, let’s. Well, and because what I want to do in this episode is make sure people are clear. Like if you take our advice, here’s what’s going to happen and it sounds like there’s some conditions in which that might not work. And so when you say the standard models people are going to run into, you’re talking about sort of like the big names like Claude, Gemini, ChatGPT, for example. I’m sure there’s others that I’m not thinking of at the moment, but those seem to be like the big names like most people I talk to. They’re like, oh, I’m using ChatGPT or I’m using Gemini. Gemini now built into Google Workspaces, which a lot of people use. Copilot, for example, is one of the major names that come up.
Katie Robbert – 14:53
And so it sounds like, if nothing else, if they’re not quite sure where to start, they can do the ask me one question at a time until you have. Until you have all of the information you’re looking for. Is that sort of. Is that a good assumption or am I missing something?
Christopher Penn – 15:11
That is generally a good assumption. The two asterisks one is within Microsoft Copilot. Whether a query gets routed to a reasoning model or not is dependent on the prompt and how Microsoft is feeling that day. And so it is an uneven experience inside of Copilot. Hence this lovely slide from. From our talk that. That I get frequently in ChatGPT is the asterisk is it depends on how you prompt it. And there’s two ways to invoke reasoning if it’s not doing reasoning. The first is to make sure it’s on thinking. So because that is a. A model that forces reasoning. And the second is in your prompt to tell it think hard, which. Which tells the model router where to send to the prompt to. Because ChatGPT and GPT5 is not one model.
Christopher Penn – 16:06
GPT5 is actually four different models with an invisible delegator up front that reads your prompt and says which one of the four models here could do this task the fastest and the cheapest?
Katie Robbert – 16:21
Okay, so I’m going to keep picking this apart because I want to make sure that we’re being clear. It’s not enough to just say ask me one question at a time until you have enough information to complete the task. Because you may not be using a reasoning model. So do you have to add on to that, you know, magic. I don’t remember what you called it. Your magic, whatever. Say think hard. Like is that. Are those two things together the key to success? So if I sign up with the fast, you know, GPT and not the thinking or GPT40 and I say ask me one question at a time until you have enough information to complete the task, think hard. If I’m understanding what you’re saying, it would switch to a thinking model and.
Christopher Penn – 17:12
Say that is correct.
Katie Robbert – 17:13
This is not the model you want.
Christopher Penn – 17:16
That is correct. So by telling it to think hard, you can now see the little reasoning bubble pop up, which is the dumbest possible interface. But what it has done behind the scenes is basically flipped from auto mode to thinking mode for this particular chat, even though the interface itself doesn’t change.
Katie Robbert – 17:39
Gotcha. Jim has a question. Are they going to undo the invisible router and let people choose their model?
Christopher Penn – 17:48
No. No.
Katie Robbert – 17:50
Well, there you go, Jim.
Christopher Penn – 17:51
No, with an asterisk. Enterprise customers supposedly will retain full access to all the model choosing stuff, but everybody else supposedly will have 40 for paid customers, which is available, which is a non reasoning model. But people really liked how sycophantic it was and then the current GPT5 family. So as usual, OpenAI making a hot mess of product market fit.
Katie Robbert – 18:24
All right, so you said, you know, there was. Okay, so my question to you was, if you say, you know, if you use this sort of magic question and then you add think hard, then it’s going to do the thinking. But you said there was asterisks to every system having a reasoning model. So obviously with ChatGPT, we covered Copilot and it’s sort of like iffy. Is there anything we need to be aware of with the other major models like Anthropic’s Claude or Google’s Gemini?
Christopher Penn – 19:00
So Claude has a toggle called extended thinking, which it does thinking all the time, but it does more thinking if you have that turned on. And the general best practices just leave it turned on, you’ll do much. It’s just a better practice for that one. Gemini, you don’t get a choice. Gemini does is reasoning all the time, which again is the sensible thing. And then there’s this new crop. You don’t. You won’t see it in the major three in the west, but there’s a new crop of models like Mistral from France and Alibaba’s Quinn and Deep Seek that are called hybrid thinking models where you tell them to think or not to think, depending on how you address them. Kind of like the whole think hard thing. And that hybrid reasoning model decides for itself whether it needs to do extended reasoning or not.
Katie Robbert – 19:54
And since we’re getting back to basics, can you, without getting too overly technical, define when you say thinking, what does that mean? Like to a layman, if you said this model is thinking, this model is not thinking, you know, to someone who’s new to this might be like, well, aren’t they all think? Like what does that mean?
Christopher Penn – 20:14
So thinking and reasoning are synonymous in this case, what that means all the time is that the model is doing a rough draft behind the scenes of its answer first before it puts put something out on the screen.
Katie Robbert – 20:29
So it wasn’t doing that before when it wasn’t a reasoning model, it was just like here you go.
Christopher Penn – 20:35
Correct. Okay, that’s right. All right, let’s continue on this very basic example. Who is our target audience for this B2B marketing blog? Let’s see. Our target audience would be marketing managers and midsize enterprises who are confused about the basics of B2B marketing. That seems like a reasonable target audience.
Katie Robbert – 21:03
What are the options? Somebody would respond B2B marketers pretty high. Okay, just wondering.
Christopher Penn – 21:10
Yep. But you can see already by the words we’re using in each answer, it’s going to dial in faster to what it needs to know next. Because some of some people like me are very verbose in their answers and every word helps it figure out faster what it is that we’re talking about. What’s the goal for the blog post? Educate, inspire or persuade? I want to create an forboding sense of doom.
Katie Robbert – 21:40
This is why you’re not our content.
Christopher Penn – 21:42
Manager or dark cautionary tale. Do you want the doom to feel realistic or dramatic and exaggerated? Let’s go for dramatic and exaggerated. A B2B business horror story.
Katie Robbert – 21:53
I knew you were going for that.
Christopher Penn – 21:56
Huh? Let’s see. Do I have a poster narrative format or a warning manual? Like a survival guide with office commentary? Let’s go with the narrative format following a fictional marketing manager who spirals into disaster.
Katie Robbert – 22:12
I’m actually shocked by that. I thought you were going to go with the warning guide.
Christopher Penn – 22:16
No, no. What industry? Let’s go with professional services like a law firm and a mid-sized law firm in the B2B space. How should the ending feel? Complete catastrophe or a last-minute bittersweet survival?
John Wall – 22:35
It’s like zombie movie.
Christopher Penn – 22:36
Let’s go with a pyrrhic victim and see it has now had to flip into thinking mode. You can see that we’ve done something in this prompt chain to make it go, oh, my routing is having issues. Let’s see. The law firm wins the end but such terrible cost. It feels like a tragedy should be straight-faced or stylized. Let’s do a stylized gothic tale of whoa. Well, let’s see. Should it be hard? Should the story teach real B2B marketing fundamentals? Yes, it should teach some basic fundamentals, like remember to not rely on social media and things that you don’t own. All right, here we are, the shadow of Ashford and goal a gothic B2B marketing tragedy. This is qualitatively and quantitatively different than best practices for B2B marketing in 2025, which is where we started. Right.
Christopher Penn – 23:44
So just in this series of questions, you can see how quickly this went off the rails in a good way to create something that’s very different.
Katie Robbert – 23:56
What’s interesting, and this is probably something that I would add to the initial prompt, is, you know, it didn’t tell you when it was done asking questions. It just was like it just decided in the background it was done and created the thing. But what if you had more context to provide? Like, obviously you can give it feedback here to be like, oh, I don’t like this, or whatever. But, you know, that to me is a little jarring where it’s like, okay, I’m done. Here’s the thing. Like, there’s no warning that it’s done asking questions.
Christopher Penn – 24:29
That’s right. And so you could give that feedback, or you could include in the prompt and say, like, show me a progress indicator of the questions as you’re asking them. But because it is kind of almost a choose-your-own-adventure now with the ask me one question at a time, you might not know where this. I, I had no idea were going to end up with a gothic marketing strategy out of this. I, you know, we started with just a terrible prompt just to illustrate the prompt strategy, but we ended up in this very strange place.
Katie Robbert – 24:58
So, John, when you do this tactic of ask me one question at a time, you know, do you find that the questions are helpful? Does it keep things on the rails or does it kind of like go this route of the seduction of glittering platforms?
Christopher Penn – 25:13
Right.
John Wall – 25:14
Right now there’s a whole mental shift that it took me a while to. To get to of this idea that it’s all about probability. Right. There’s not absolute answers. It’s the models take a look at everything out there and determine what they think the most probable answer is. So if you give it a huge prompt upfront, that’s kind of just very general. It’s going to grab from the whole Internet. It’s going to be all over the place, but with a lot of small individual focus questions, you know, you’re keeping the focus very tight as it goes through the whole process. So, you know, in theory, it’s going to be closer to what you think you know, the quote unquote, truth is or what’s correct, you know, what’s accurate.
John Wall – 25:58
But yeah, it can go way off the rails too, depending on how you answer the questions.
Christopher Penn – 26:03
Right.
John Wall – 26:03
I mean, that’s really the. At the heart of it is if you start steering it in a direction where nobody else has ever gone before. Yeah, there’s kind of no telling what you’re going to pull out. But it has helped me as far as, you know, being able to land closer to where I want to go. You know, doing it in one shot is a real roll of the dice, but going multiple questions, it seems like it can at least dial in closer to where it’s supposed to be.
Christopher Penn – 26:30
Yep. So next we’re going to move this into the canvas. So for folks who are unfamiliar, the canvas in OpenAI is essentially kind of like a Google Doc. We’re going to have the post be here. Let’s see. In the canvas, you will note that there’s our general post. There’s an edit panel here now. So I can say make this longer, change the reading level and add final polish, add emoji, if you wanted to do such a thing, suggest edits. So I’m going to say let’s take this post and make it much longer and hit go. And now it should revise our content within the canvas to make this much longer.
Christopher Penn – 27:13
So in terms of prompting basics, since we’re talking about that today, a lot of the tools today have built-in utilities to do things like changing style, changing length, changing tone, changing readability that you may not have known that those tools were there. Right. So I can, I could adjust the length even longer. I could change the grade level, you know, to college, to graduate school. I could change it down to kindergarten, which is a gothic marketing tale for kindergartners.
Katie Robbert – 27:43
Well, I was, so I was in Google Gemini yesterday working on some stuff internally and I was using its version of Canvas and so as it was giving me the output, I could go into specific parts of the content and when I highlighted it gave me a little pop up that says Ask Gemini. So basically exactly that. And I could say I want to tweak this specific to, you know, whatever without having to redo the entire thing, which, you know, if you’ve ever had to do that sometimes can really kind of go off the rails because you’re like, no, no, I really liked two thirds of it. I just wanted to focus on this one thing.
Katie Robbert – 28:23
And now they give you the opportunity to do that, which I personally have found incredibly helpful because it’s usually like minor tweaks to things without having to do redo the entire thing and be like, well, let’s, you know, revert back. Like it’s, you have a little more control over the information that you’re editing, which is very helpful.
Christopher Penn – 28:45
Exactly. So part of prompting basics is just learning the tools, learning what capabilities are available in these systems. So things for example like in OpenAI you have deep research agent mode, creating images, study and learn, web search and so on and so forth. As you mentioned Katie, if you go into Google’s Gemini you have video and deep research canvas image guided learning, which is their version of study and learn, etc. So those are all of the things that from a basic perspective, if you use the tools that are there, it will save you the time of saying, look, I need to rewrite this for this. However, you can always just specify and say, I want it to do this kind of thing. Those are a couple of the basics.
Christopher Penn – 29:31
The other basic I want to spend some time on, we talk about this in our Mastering Prompt Engineering course is knowledge blocks. If there was one other thing besides the asking one question at a time that I would advocate everybody do, it is to have pre-built knowledge blocks. I call them digital Legos. And what these things are blocks of knowledge, blocks of information that you can drop into a chat that may be thousands of words long so that you don’t have to retype that information, but it instantly makes everything better because you know exactly what the model, knows exactly what you were. It’s, it’s supposed to be know about you. So for example, I have a Trust Insights knowledge block here that says, you know, here’s who we are, here’s what we do and so on and so forth.
Christopher Penn – 30:24
We have our sales playbook, the 135-page document that says here’s how we sell and market, and for us to be able to just drop that whole document in is a huge time saver. So the knowledge blocks I recommend people have, especially for the folks who are in marketing, are make sure you have an about us, right? That is a detailed explanation of who you are, have a how you do your marketing, any kind of reports that you have, ideal customer profiles which if you don’t have them and you don’t know how to build them, contact us. We’re more than happy to walk you through our process of building them or build them for you. Any kind of strategic planning documents that you have, any kind of frameworks.
Christopher Penn – 31:10
So for example, I have knowledge blocks for my versions of a SWOT and a Porter’s Five Forces and a BCG growth matrix because I have very strong personal opinions about how they should be done. And so when I do them, I have them written out. It’s like, this is how I want you to do it.
Katie Robbert – 31:27
I don’t care what the industry does. This is how I do it.
Christopher Penn – 31:30
Exactly. Exactly. Well, for example, we used to work at a PR firm and those folks did SWOT analysis and I thought they did it horribly because they kept mixing up things like industry trends and stuff like that. That is not what a SWOT is for. That’s what a Porter’s Five Forces is for. Putting industry trends into SWOT analysis is stupid. You should do it anyway that go.
Katie Robbert – 31:53
But that’s. Well, yeah, I digress. That’s not your personal opinion of a SWOT, that’s just proper use of a SWOT. But anyway, yeah, I digress. Okay. No, and I think it’s worth noting though is. Well, I guess let me say rather than worth noting, the question, the one question that I have for you is obviously we can put about us and everything that’s like public on our website, but do we need to go deeper and put in things that aren’t on our website? Is that a risk? Is that a bad idea? How detailed do we need to be?
Christopher Penn – 32:29
It depends. It depends on a. The privacy of the system you’re using. So for example, ChatGPT right now is under court order to retain everything that’s put into it. So if you have stuff that’s super private that you don’t want end up in court deposition somewhere, maybe don’t put it in there. Google’s Gemini there though Gemini workspace version says they don’t train on your data and a human will review it only if you do something dumb like ask it how to build a very bad thing. Anthropic Claude has very strong privacy protections. If you use the Chinese hosted version of Deep Seek, there are zero privacy protections on it. Everything that you put in there is going to be seen by multiple people in the People’s Republic of China.
Christopher Penn – 33:15
If you put anything into Elon Musk’s XAI and Grok, you have no privacy there. Everything’s going to be consumed and Elon may retweet it. The key lesson here is you need to check the privacy policies and the terms of service on every platform that you use. And what I would suggest you do is have a Gemini or GPT or something that you’ve written that evaluates privacy policies and says, on a scale of zero to 100, here’s how private your data is in this platform. And then you can make the decision like, yeah, this is not good.
Katie Robbert – 33:52
This is usually about the time that I bring out my tinfoil hat and assume that everything is dangerous and don’t use anything. We all. It’s well documented that I am the least likely to take any kind of risks on the team. I am like the most conservative when it comes to business. So when I see people are putting like their strategic plans into these large language models, I’m like, oh, don’t do that. You know. But you know, Chris, to your point, if you evaluate the privacy policy, you know, and do your due diligence, then you can make those judgment calls of how much risk are you willing to take.
Katie Robbert – 34:36
Like if your strategic plan is, you know, make more money by posting more things on social media, probably not an industry secret, probably okay to share, but like, if you get deeper into the weeds like those, you know, maybe there’s the high-level AI friendly version and then the more deeper detailed version of things, you know, just. That’s my little tinfoil hat talk. And so just, you know, be careful.
Christopher Penn – 35:01
One of the things that we show in our presentations about this is sort of the levels of privacy that you have, right? Local AI models, the ones that you run on your computer and you can unplug the Internet, they are the only ones that you are guaranteed privacy because as we say, often there is no cloud, it’s just somebody else’s computer. The tools that you see in green, those are generally safe, mostly private. That means that they’re not going to train any data and a human will only look at them if you do something stupid like violate the terms of service. The ones in yellow can be made private but are not private out of the box. And the ones in red are not private at all, cannot be made private. And in some cases, like with XAI’s Grok, you probably shouldn’t use them.
Katie Robbert – 35:54
If you want a copy of this for yourself to hand over to whoever’s trying to force you to use tools that you’re like, I don’t know about this. You can go to Trust Insights Aldata Privacy. We try to keep this updated about once a quarter as things change. But I would say as of today, this is a fairly good representation of how secure your data is if you put it into a large language model.
Christopher Penn – 36:20
Exactly. And companies like OpenAI have released open models. OpenAI made a big splash a couple weeks ago. Not GPT5 but their first open models since 2020. The last model they released openly was GPT2 and then they released the poorly named GPT OSS20B and the GPT OSS120B, thereby demonstrating that there was no product marketing manager in the room.
John Wall – 36:55
Whatever it takes.
Christopher Penn – 36:57
It is widely suspected that these two open source models are actually GPT5 Nano and GPT5 Mini. They are very good models. They are extremely fast. You can run them locally on your computer and you have the same control that you do in their development studio. However, they are, they hallucinate approximately 90% of the time because they are trained on fluency and skill, not knowledge. I could go in and say, write a blog post about B2B marketing. Ask me one question at a time until you have enough information to complete the task. I’m using this inside LM Studio. LM Studio is private hosting. You can see there’s it’s thinking and it’s now starting to do the exact same process as its cloud-based big brother. The difference is this is private.
Katie Robbert – 37:52
When I hear 90% hallucination, my first thought is like, oh man, you really are choosing to take an adventure today. Like you don’t know what you’re going to get and maybe that’s what you want. Maybe you want something totally wacky out of the box. Maybe it’s a good brainstorming partner because it’s going to think of things that are like totally from a different universe. Maybe that’s a good thing. But yeah, that’s 90% hallucination to me. Sounds problematic.
Christopher Penn – 38:20
It is in the sense if you’re, if you are relying on generative AI to present factually correct information, that is definitely a problem. However, you shouldn’t be doing that in the first place. Generally speaking. What you should be doing is you should be bringing the data and then having AI manipulate the language. That’s the best practice. In general, today’s models that you use in like a ChatGPT or a Claude typically have a web search tool. And so they will just go out and search the web to get information if they’re not confident that they’re they’ve come up with a good answer. But the hallucination rates even in the big tools still exist. And it’s one of the reasons why we tell people, anytime you have AI output in finance, law or health, please have a human subject matter inspected.
Katie Robbert – 39:12
I would go so far as to say any AI output, but again, most conservative person on the team.
John Wall – 39:19
How about, can you go into it? So you had mentioned that 90% hallucination rate is based because it takes fluency first over accuracy. What exactly does that mean? You know, by prioritizing on fluency.
Christopher Penn – 39:33
So fluency means that it’s really good at manipulating language, so it can do things like tool calling. So if you’re using it to write code, it. It can say, oh, you have a tool that does linting, you have a tool that does static typing, you have a tool that does this or that. And I know it exists and I know how to use it. So in the past we’ve talked on episodes of the podcast and stuff about MCP servers, model context protocol. And what that means is the model knows, oh, this tool exists and I can use it.
Katie Robbert – 40:04
Gotcha.
Christopher Penn – 40:05
Okay, so it’s been trained on tool use, but they sacrificed knowledge to get those capabilities in it.
Katie Robbert – 40:15
See, putting the platform first.
John Wall – 40:17
That’s right.
Katie Robbert – 40:18
Absolute wrong thing to do. That’s a different episode.
Christopher Penn – 40:21
That is a different episode. So knowledge blocks the. Ask me one question at a time. And knowing what platforms you can use safely with your data are kind of the big three basics for prompting. You know, the basics of prompt engineering. Now a lot of people are circulating, you know, OpenAI’s GPT5 prompt optimizer, which is good, it’s. It’s good for GPT5. And we recommend, if you have custom GPTs that you put them through the prompt optimizer so that your instructions are updated. But in general, you don’t necessarily need to use a special tool to do that. You can ask a model, say, here is my prompt. Using your knowledge of prompt engineering after January 1, 2024, how would you revise this prompt to be most effective? What is my intent of the prompt? What are the downstream effects of this prompt?
Christopher Penn – 41:21
Rewrite the prompt with this information after you’ve thought this through. And what it will do is, will essentially optimize your prompt for that model. And as a reminder, optimizing a prompt for one model does not optimize it for every other model. So if you do that on a Gemini instance, you can’t. You’re not going to get as good a performance. If you just copy and paste a Gemini prompt into ChatGPT, you’ll see it’s still language, it still understands what you’re trying to do, but it will look different than if you said, here’s this prompt, optimize it for ChatGPT.
Katie Robbert – 41:56
So it sounds like, to paraphrase what you’re saying, is you don’t necessarily need a strict framework anymore, but what you do need is critical thinking, which has been sort of theme for the past couple of weeks that we’ve been talking about. And so critical thinking in the sense of what could go wrong? Did I provide enough information? Do I have the expertise in order to determine whether or not the information is correct that I’m getting back out? And so really thinking through what am I actually asking this machine to do? Like am I being thorough enough? And you know, does it have to be structured in a framework? It can be if that’s how you personally organize information. Otherwise just making sure you’re kind of just covering all your bases.
Christopher Penn – 42:47
Exactly. What I generally say for people who are working with AI is critical thinking, creative thinking. Right. The tools can do just about anything. You have to have the brain space to go, oh, I, I, I should ask it to try, see if you can do this right. And then contextual thinking, which is do you know where your data is and how are you using it with AI, do you understand the context of the request you’re making? Could you provide it with more information when you were and Katie, you’ve been talking about this a ton lately with soft, AKA durable skills, transferable skills. These are, these are the ones specific just for working with technology and then you got many more for working with humans.
Katie Robbert – 43:36
Yeah, not everyone can work with humans and I feel like those of us who for the most part can are going to be in demand because it’s the narrative is flipping back to, okay, if AI can do the hard skills, which is the technology, then what’s left? That’s the soft skills, which aren’t really that soft. They’re actually really difficult. I would say they’re harder than the tech because I don’t know about you, but John’s pretty unpredictable and I don’t know what I’m gonna get day to day with him.
John Wall – 44:13
So grab bag, that’s pretty, I’m sure.
Christopher Penn – 44:14
It’s always a grab bag. It’s always a grab bag. But yeah, in terms of how you should be prompting in mid-2025, rely more on the knowledge in these tools. I mean, the fact of the matter is that on most benchmarks now, like Diamond GPQA for example, these tools all operate at or above human PhD range. You can see here, GPT4 started out as a face rolling idiot. Right. And then as of 2025 like Gemini 2.5 up, where it’s above PhD range in terms of its ability to pass this particular very difficult academic test. That means you should be using these tools with the understanding they have many more capabilities than they did two years.
Katie Robbert – 45:03
Ago, perhaps offline or more appropriately, in our free Slack group, Analytics for Marketers. You can explain to me what exactly a face rolling idiot is.
Christopher Penn – 45:17
Save it for the so many answers.
Katie Robbert – 45:20
To that question, and I look forward to hearing all of them.
Christopher Penn – 45:26
But yes. So that’s the thing is with revisiting prompt engineering, people are prompting like it’s summer 2023, when really it’s it. We’re well past that. And the skills that separate you from the rest of the herd now all revolve around critical thinking, creative thinking, and contextual thinking.
Katie Robbert – 45:47
All right, use your brain, people. Bottom line, use your brain.
Christopher Penn – 45:54
Any final thoughts?
John Wall – 45:57
That’s it. Go do some good stuff. Get amazing prompts running.
Christopher Penn – 46:02
All right, thanks, folks. We will talk to you all on the next one. Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it. For more resources and to learn more, check out the Trust Insights podcast at TrustInsights AI TI podcast and a weekly email newsletter at TrustInsights AI. Got questions about what you saw in today’s episode? Join our free analytics for Marketers Slack Group at TrustInsights AI analytics for marketers. See you next time.
|
Need help with your marketing AI and analytics? |
You might also enjoy: |
|
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! |
Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. |
Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.
Newsletter
Podcast
