In this week’s In-Ear Insights, Katie and Chris review the new Trust Insights PARE Framework and 4 power questions to ask generative AI language models when you’re working with them. These four questions will dramatically improve the performance of generative AI. Get the free download, no strings attached, at:
https://www.trustinsights.ai/powerquestions
Watch the video here:
Can’t see anything? Watch it on YouTube here.
Listen to the audio here:
- Need help with your company’s data and analytics? Let us know!
- Join our free Slack group for marketers interested in analytics!
[podcastsponsor]
Machine-Generated Transcript
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Christopher Penn 0:00
This week’s In-Ear Insights, I have something new Katie is something that I built because one of the things that we were talking about recently was making content more accessible to people, particularly useful tips and things.
Because as much as I like to geek out on the intricacies of like, the latent space and the language model, most people don’t care.
Most people like just just tell me what to do, within reason.
Is that a fair statement?
Katie Robbert 0:24
That is, I would say, that is completely a fair statement.
And that sort of, you know, that’s the challenge that I know that, you know, you run into, and I run into is that, you know, we want to make sure people understand the why.
And they want to know the why until we actually tell them, and then they’re like, great, I don’t care.
So they think they want to know, but they really just want to get to doing the thing.
And it does, it is a challenge because you and I are so like, it is so much in our DNA to really understand how the thing works.
Before we do the thing that we feel like we also then have to get everybody else on the same page as us before they can do the thing.
And they’re just like, great, what button do I push? Exactly.
Christopher Penn 1:05
So one of the most popular things that we’ve produced along those lines is our race strategy for prompting for generative AI, right, so roll action, context, execution.
And if you follow this framework for using gender of AI, it’s a good starting point.
It’s not by any means the be all end all the founding, but it’s a good place to start, especially if you’re new to language models.
Well, one of the things that we’ve done a lot of in our keynotes and workshops, and our webinars and stuff, is talk about ways to improve the performance of a model, once you start using it once you start prompting it.
And so last night, I was born as and as one does, I created the foot the for generative AI power questions, or language models, I figured today, we would talk through these a little bit and help folks understand what they are when to use them.
They’re all pretty straightforward.
So the four questions are, what do you know about the topic? What questions do you have? What did I forget to ask? And did you fulfill the conditions of the prompt? And the way you would use these questions during a session with ChatGPT or Google Gemini or anthropic squad? It depends on what you’re trying to do.
But these four questions will help you get much better results of the model.
So before I go on, what questions do you have gaming? First
Katie Robbert 2:32
of all, you know how much I love surprises.
So I’m processing as quickly as my poor brain can possibly process this early in the morning.
Second question.
That was the first one was not a question actually.
So first, here we are.
First question.
Oh, my goodness, let me take a step back for a second for those listening.
So Chris was covering, we developed or we came up with the race framework for people who are trying to figure out how to get started with generative AI.
And so as Chris mentioned, that is role action, context, execution.
And so basically, the role is who you need generative AI to be who you’re talking to.
So if you if you’re going to be asking a question about B2B marketing, the role that generative AI needs to play is a B2B marketer.
So today, you need to be B2B marketer, who’s an expert in this, the action, what do you need it to do? I need you to help me with my email marketing strategy, context.
Here’s my limitations.
Here’s what I know, here’s what I have, here’s my audience, so and so forth, and then execute with all of this information, I need you and it’s it’s a reiteration of the action I need you to build helped me build a strategy.
So that’s the race framework.
So I just wanted to sort of do a quick review for anyone who’s not familiar with it.
So now to this new for generative AI power questions, we can get into it after we get into the framework.
But you know, why it’s called Power questions is one of my question versus just questions.
Christopher Penn 4:12
So it sounds cool, subjective.
Alright,
Katie Robbert 4:15
so we have expand, augment, complete, validate.
As you were talking, my poor marketer brain was like, but that doesn’t spell anything.
Because, as marketers, we’re so ingrained for the acronyms.
So, you know, I’ll be overthinking that later.
Okay, so now that I’ve bought myself some time to process the information that I’ve been presented by reviewing the information that I’ve currently had, alright, so it’s interesting, because basically, like, in a very simplistic way, if you’re working with a generative AI model, you’re asking it to, like check itself, you’re saying, Are you lying to me? Or do you actually know what you’re talking about? Without asking that specific question because that’s the the wrong question to ask, but you’re saying, Do you even know about this thing? am I even talking to the right person? You know, find me somebody who does know about this thing.
If it’s not you, then let’s not have this conversation because it’s a waste of my time.
And then it’s like, what questions do you have, which is, you know, that’s, that’s helpful.
Because we as humans, even if we use something like the race framework, we may not be giving enough information.
So we may have some holes.
So you’re giving the model a chance to say, well, either your information was contradictory, or you didn’t give me enough of it? And then what is it complete? What I did, I
Christopher Penn 5:34
forgot to ask,
Katie Robbert 5:35
What did I forget to ask? Again, it’s sort of it goes with? What questions do you have? And then what did I forget to ask? You know, I would be interested to know how these models come up with the responses.
And then validate, do you have full confidence of the prompt? Or, you know, full awareness? Like, do you have enough information to do this thing? Which is also a really good question.
And I can see where you’ve borrowed from straight management, because these are questions that you would want to ask somebody that you’re working with somebody that you delegate to, you know, you know, so Chris, if you say to me, Hey, Katie, I need you to write a SQL query.
Your first question should be, do you know how to write a SQL query? So that’s the expand? And then what questions do you have? And I would say, great, what SQL? wouldn’t actually that wouldn’t actually be my question.
But like, as an example, you know, and so on.
So I think it’s, I think it’s a really good framework for people who are trying to get more out of it.
For those folks that are saying, Well, I tried to use generative AI, and it’s not working for me.
So they’ve just dismissed it.
Christopher Penn 6:42
Right? So there’s some, there’s some technical things going on behind the scenes here that are not exposed by these power questions, but are really important.
So the first one, what do you know about this topic does two things one, to your point, it validates like, does this model know this topic? Well? Is it is it lying to me? So if you were to put in something simple, like, what do you know about B2B marketing strategy, it’s gonna come up with a decent starting point, if you were to put in what do you know about junction 13 613 1644? On open reading frame, one, a and the SARS cov.
Two virus, the model is probably gonna go and, and make a bunch of things up, because that’s not something that there’s been a lot of training data about.
And so when you start with what do you know about this topic, or wouldn’t know about the best practices for it? You’re validating Yes, you know what you’re talking about, or no, you don’t know what you’re talking about, I need to provide extra context, I need to provide the data for that.
The second thing it does.
And this is part of the C in the race framework context, the way language model works, is kind of like a Choose Your Own Adventure book.
If you remember those books, one of your kids like, pirate story, it says at the end of this page, choose this one or this one.
As you’re reading that book, you are making the choices and you keep going, you’re choosing your own adventure, and you vet but you don’t cover the entire book, in one setting, you cover only the choices that you made to tell that story.
That’s how a language model works, except it doesn’t by word.
And the key part is that, as it’s generating everything that it makes, and everything you prompt, it is part of the story, if you will, and that helps the model make the next choice.
So when you ask the model, what do you know about this topic, and it spits out eight paragraphs of hey, here’s what I know about B2B marketing and all the stuff that becomes part of the next prompt.
So you are by by asking what it knows you are essentially pre loading a lot of knowledge into this the session that you’re working with, to get the model up to speed faster, that’s less information you have to provide in the prompt, would
Katie Robbert 8:47
you advise people who are using this tactic to actually know the topic themselves? Because, you know, how, what is the likelihood that a generative AI model is still hallucinating the information like how do we know that? So if you say, you know, what do you know about beekeeping? Now, if I know nothing about beekeeping, and I asked the model, how do I know that the information that’s giving me is correct? If you
Christopher Penn 9:17
know nothing about it, you don’t know.
In the same way that if you asked an intern, a human intern and they made something up, you would not know either.
The rule of thumb is this the more specific the information, the more likely hallucination is because beekeeping is a very broad general topic, right? So as B2B marketing, so it’s like that something like what does the genomic sequence on you know section 13,644 of open reading frame one a on the SARS cov.
Two viruses is such a specific piece of knowledge that is so niche and so, so there can be so little training data about it.
The probability of hallucination is very high, because the model probably hasn’t trained on an information same is true for very new information if you are if you just read a new paper that came out last week on a given topic.
And you were to query a model about that topic without providing the paper, the model is going to fall back on what it statistically knows.
And it’s probably gonna hallucinate so broad, you’re probably safe, narrow, specific niche, danger of hallucination increases brand new knowledge, the danger of hallucination increase.
And as you go further down that rabbit hole, your the need for subject matter expertise increases,
Katie Robbert 10:29
okay.
So if you don’t know a lot about the topic, and you’re asking the model, then you have no way of validating that whether or not it’s giving you correct information.
So, So rule number one, if you’re asking the model, what it knows, you also need to know the information you can’t ask for something that you don’t know about and be able to feel competent, that it also knows, correct.
Christopher Penn 10:53
And this goes back to no surprise, the five Ps, right.
So if you’re writing this just as a social media post on LinkedIn, you’re the purpose is pretty clear.
And the risk is pretty low.
Right? Yeah, you might get made fun of if you say something silly.
But if you are publishing a legal document on your website, that you will be held legally accountable for the risk is very high.
And therefore, you absolutely should not be doing that without someone who has subject matter expertise, reviewing by keeping the human in the loop human review of that content.
I mean, in general, you should be having humans review AI output.
But if you don’t have that subject matter expertise, and it’s something that is potentially risky, don’t do it, don’t do it, because there’s a very good chance that something will be off.
Katie Robbert 11:38
No.
And that I mean, it’s a good best practice in general, whether or not you’re using generative AI or, you know, humans, someone who has subject matter expertise should be present.
Christopher Penn 11:50
Exactly.
The second question, what questions do you have is essentially, it does two things.
One, these models are very knowledgeable, particularly the newest vintages, Claud, three, Gemini, 1.5, etc.
They’re very, very knowledgeable.
One of the flaws that we have as humans is, we don’t think to ask what they you know, what questions they might have, when I’m writing code? This is the most important question to ask, because, as I’m explaining to the model, what I want to do, I ask it, what questions you have.
And very often it asks things like, Hey, do you want some error checking? Because you didn’t? You didn’t tell me about it? Like, oh, yeah, I guess? Do you want this? Do you want that? Is this is this important to you? And and we forget about those things? And that goes with Question three, which is what did I forget to ask? Again, these mountains have a lot of completeness of information in them.
So as you are prompting them, when you get that first set of results back.
And you clarify, when you have a conversation with a model, say what I forget to ask about this, what am I what am I overlooking what are my blind spots? This is a question that helps you with professional development.
Because as you prompt, you want to you want to use these tools to help give you more things to research more things to learn about.
Katie Robbert 13:03
It’s really good.
It’s a really good framework, both for the machines, but also just for interacting with people, you know, and only to pick on you a little bit because you know, we know that you prefer machines over people.
What strikes me is interesting is so I’ve worked with you for almost 10 years now.
And I’ve never heard you ask these questions to a person.
And so it’s I would encourage anyone who’s listening, who’s like, I don’t know where I’m struggling with delegation, because that’s the number one question that I always get is, How do I delegate better? This is a really good framework for delegation.
For someone that you’re like, well, and this was always, you know, a challenge, Chris, that we had at the agency was, well, I can’t delegate this to anyone, because they won’t do it the same way that I do.
But these questions would be helpful in that conversation.
So I can, so as the as the person to person in the conversation, I’ve asked these questions, and I know how they work with people.
And so it’s fascinating to me, then to see it translated from person to machine, where you can ask the same questions and get similar results as if you’re talking to a person, you know, it’s no, have you ever run into an instance.
So I think about like job interviews, for example, because you can always sort of tell, you know, how the interview is going, when you say to the person you’re interviewing, do you have any questions for me? And they’re like, No, I’m good.
You’re like, okay, that means you’re not very well prepared.
Have you ever run into that instance, with a large language model where you said, What questions did you Did I forget to ask, what questions do you have for me and the mat and the model is like, No, I’m good.
I have another meeting to get to.
Christopher Penn 14:47
That it won’t say it like that.
But I have run into cases where it says you’re the information provided is complete and comprehensive.
I don’t have any questions.
So let’s begin the task.
And do you believe it? I do because As I’ve gotten in the habit, the machines have trained me, believe it or not to build better requirements, because I will keep asking you what questions you have what questions you have.
And I keep copying and pasting responses back and forth until the requirements for a piece of code are so comprehensive that when I say, Okay, let’s generate the code, what questions you have? Since we’ve answered them all, you’re all set
Katie Robbert 15:21
to stay.
And as an example, go ahead.
Christopher Penn 15:24
I have here that was I was working on some a piece of code for for WordPress last night.
And the machine kept walking me through what about this? What about this? What about this? Do you want error logs? Do you want to show progress of the task? Do you want to do this? And so it basically was training me to write better requirements? Because this then becomes the prompt, for a part of the prompt for generating the code at
Katie Robbert 15:48
any point as a human? Does it get tiresome? Like, I guess? Like, how much patience Do you feel like you have to have? Or is it because there’s no emotion involved with the machine? Like it’s not a problem? It’s
Christopher Penn 16:03
not a problem.
It’s a machine.
I know, it’s done.
I know, it has no existing context.
We call these things in the keynotes, we deliver we call them the world’s smartest, most forgetful entrance, right? Every time we start talking to you guys, like it remembers nothing.
But it’s very smart.
So I feel like when I’m working in a session with one of these things, I know it’s, it has no memory, I couldn’t tell me what had for lunch.
And our expectations of it, the my expectations of it are much lower than my expectations for human beings.
Right, which is why I’m more patient with it, because I know it has no memory whatsoever.
And so I understand every time I address the machine, that I have to provide all this information, it’s it’s not going to know it.
That’s interesting.
Katie Robbert 16:48
All right.
And so the last, the last question here is validate.
And so to fulfill the conditions of the prompt, and so then that again, goes back to you, the human making sure you’re writing a clear enough prompt, so that the machine can go Yeah, I did exactly what you asked me because one of the things I’ve learned from you is that it’s not going to infer what you could have meant, it is going to take things literally, because there is no like, Well, maybe he meant this, but he really said this, like that’s not how these large language models work.
And so you have to be very clear and specific with the prompt that you’re giving it.
So that when you get to that point of Did you fulfill the conditions of the prompt completely? Like it’s going to say yes.
And you’re gonna say, well, that’s not what I needed.
That’s on you, the human because you did not specify exactly what it was that you needed.
Christopher Penn 17:44
This question I use the most when I see that it has done something incorrect.
So let’s say we’re doing a thing with B2B Marketing Strategy and spits out a whole bunch of stuff.
And I say, I want this in a bullet point outline format.
And it gives me you know, prose from an article I say, check your work.
Did you fulfill the conditions of the prompt completely? And we’ll go back and go, Oh, I’m sorry.
You said a bullet point outline format here.
It isn’t a bullet point out.
Thanks.
So anytime that I spot a mistake, in the in the output, I will ask this question to get to clarify.
And the reason you would want to do that is instead of starting over is because you want to take advantage of all the knowledge things, the story that you’ve read through so far, and the choose your own adventure that is working with the AI, you would want to to keep all that knowledge and just say, hey, just check your work, and fix the mistakes you’ve made.
So do you
Katie Robbert 18:34
feel like these questions for language models are really sort of the next phase of people getting better with using generative AI? Because again, sort of, you know, like I mentioned earlier, in this episode, I’ve seen a lot of people complaining, well, I asked her motivated to do something and it couldn’t do it.
So they just kind of gave up.
But I feel like maybe they’re not asking these questions.
And so one of the things you talk about Chris is priming the model.
And so I feel like expand this first, what do you know about this is that exact example of priming the model? Because so I’m, you know, this is me still processing, you know, these questions that you showed me this morning, you know, so the first thing you want to do is what is the model know about this topic.
And so that way that sort of gives you the sense of this is a good model to use, or I should find a different model that maybe has more information, or I have to provide the information to the model so that it knows what I’m looking for.
What questions do you have? You know, again, I feel like that’s just a really good question to be asking, in general, whether it’s to large language models or people after you finish talking like what questions do you have, you know, it’s something that we do as speakers.
We give a 45 minute presentation, and then we literally leave the next 10 minutes for questions like What questions do you have? I just gave you a lot of information.
And so naturally It lends itself to, okay, people are processing all of this information.
You know, they probably need some more context, they need some time they need to like ask about it.
And then the third one was, I’ve now lost.
Christopher Penn 20:15
Well, I actually like the way you’re describing this.
So I’m going through now and I am changing the section headings to spell something like what you’re talking about the very beginning.
This acronym doesn’t make any sense.
But
Katie Robbert 20:32
it just makes it easier for people to remember it.
So like, you know, it’s the same way that I came up with my own goofy acronym for artificial intelligence.
Because now I mean, now that I’ve spoken about it enough, I can remember regression, classification, and generation.
But it’s still easier for me to remember find organized classify, which spells out fog.
So now we have pair prime, augment, refresh, evaluate.
Interesting.
So we have race and pair.
Christopher Penn 21:07
Which is awkward, because the Brazilian version of the race family is actually paste because I was playing with some translation over the weekend, a couple of our friends in the analytics from our community are speak Brazilian one has their native language.
And then someone on LinkedIn actually made a translation of the race framework in Spanish.
They particularly like the translation as well, because it was it so I had Gemini go through it and and work with me to clean it up.
So So yes, you have race and pear, but it’s paced in Brasilia,
Katie Robbert 21:36
I think that’s okay.
Yeah,
Christopher Penn 21:38
that’s totally fine.
Katie Robbert 21:38
So, do these frameworks work with any generative AI model? Or other does it really only work with like ChatGPT, and Gemini,
Christopher Penn 21:50
these work with any large model? So large models would be ChatGPT, any version of it, Google Gemini, any version of Anthro pics, Claude any version of Microsoft thing, they will work with mixed drawl, for example, they will not work as well with very small models.
So a very small model would be like, any of like the original llama family models, but in general, you would for the what most marketers are using AI for you wouldn’t be using those models anyway, you’re using one of the large public models.
Katie Robbert 22:19
Okay.
What about you know, and I, I would imagine that this is going to be a separate topic.
But what about image generation? So there’s content generation, and there’s image generation, I feel like we’re still very, we’re in a different world with how we approach image generation.
So to be clear, we’re talking about content generation, text generation word generation, not using this with like, a Photoshop, for example,
Christopher Penn 22:45
or that’s why we say it’s like power questions for language models, which
Katie Robbert 22:49
I can see for someone not as familiar, they would not necessarily know the difference.
And you know, it’s when I say things like that, I mean, that I’ve made those mistakes before.
Christopher Penn 23:00
Ya know, and that’s the thing about, I’m hesitant to do that with image models.
Because image models are rapidly becoming language models, we already see this with a ChatGPT, where it’s built to use its language to essentially write queries for Dolly.
And then you don’t need to do that weird janky image prompting language Geminis the same way.
So as these tools become more and more multimodal, the skill for image generation is essentially going away.
And it’s being incorporated into the way a language model works, because it’s more natural language anyway, like, I want to draw a picture of this, this this is kind of look like this, instead of arcane practice, like 16 by nine 8k high def, instead of like, no, no one speaks like that.
But people are used to telling a language about, hey, here’s what I want.
And so it makes total sense that that particular sub style of prompting will go away.
For most people except the technical users.
If there are technical users who want to finely control the output of an image model.
They will be the ones to install comfy UI, they will be the ones to have a specific model from civet AI on their desktop that they use.
They have like 40 different models for individual purposes.
The average marketer, the average person is not going to go through all that, that labor.
So we now have our power questions for power questions.
If you would like a copy of this, by the time this episode airs, you can go to trust insights.ai/power questions and get a copy, no downloads, I mean, no form to fill out just it’s like our race framework, just grab it and go, of course, there is an ad for our services, the bottom, I would be remiss if I did not do that.
Katie Robbert 24:36
I would I would be a little upset with you.
Any
Christopher Penn 24:39
other thoughts, Katie, on this framework?
Katie Robbert 24:42
No, I think it’s interesting.
You know, I think it’s going to be helpful for those of us who have struggled to get what we feel is like the right output from generative AI and we’re just like, well, it must not be for me.
I can’t get it to work like Chris gets to get it to work into like two seconds.
But I can’t.
So like, let me just leave it to him.
So I feel like the more we can give people like myself these tools and tips to say, don’t give up just yet, you’re just not you have, there’s more things that you can ask, there’s more things that you can do.
Before you give up, I think that that’s going to be incredibly beneficial and helpful.
And it will start to break down that barrier to adoption.
Because I mean, I give generative AI a task, and I still feel like, well, I could probably still still do it better myself, because I’m not asking those questions.
Christopher Penn 25:36
Yep.
And I remind folks to, you know, again, this is something we teach in our course, which you can find at TrustInsights.ai AI, slash AI course, keep a prompt library, keep a prompt notebook of stuff.
And then as you learn things that the race framework and and the pair framework and stuff, you’ll update those prompts, keep them up to date, they, they are software, right, a prompt is software, it’s just not written in a programming language, but it’s still software that you’re writing code for, which means that you should be iterating you should be improving it, it should have a part of a lifecycle which involves, you know, maybe not formal Scrum, and, and, and, you know, daily stand up stuff, but at the same time, if it’s a problem that’s important to your business, maybe it does, because maybe the software development lifecycle apply, it should always apply to things that are important to your business.
Katie Robbert 26:25
Some kind of lifecycle, you know, some sort of regular check in and maintenance definitely should apply.
It’s I rarely ever see a task as one and done and then you just move on from it.
To your point, especially if it’s important to the business.
It’s something that you should revisit frequently.
So
Christopher Penn 26:42
it feels like you could write up something on the AI prompts development lifecycle, Katie,
Katie Robbert 26:48
maybe that’s a big ask for a Monday morning.
Well, no, it’s it’s definitely an it’s interesting, because you know what? Generative AI really is, at the end of the day, it’s just another piece of software, it’s just another tack to put into your stack.
And you need to approach it as if you as like you would any other piece of software.
So that’s definitely a topic for another day.
But we can start exploring what that looks like.
Christopher Penn 27:14
Exactly.
If you’ve got some thoughts about the pair framework, the race framework, or just how you are getting results out of generative AI pop onto a free slack go to trust insights.ai/analytics for marketers, where you have over 3000 other marketers are asking and answering each other’s questions every single day.
And wherever it is you’re watching or listening to the show.
If there’s a challenge you’d rather have it on instead, go to trust insights.ai/ti podcast, you can find us on the channel of your choice.
And while you’re there, please leave us a rating and a review.
It does help share the show.
Thanks for tuning in.
I will talk to you on the next one.
|
Need help with your marketing AI and analytics? |
You might also enjoy: |
|
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! |
Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. |
Newsletter
Podcast
