In this week’s In-Ear Insights, Katie and Chris talk through how to improve your prompt engineering for large language models like ChatGPT, GPT-4, and other services through the use of the software development lifecycle. Learn how to apply the SDLC to your individual work with AI tools, and why it’s so important.
[podcastsponsor]
Watch the video here:
Can’t see anything? Watch it on YouTube here.
Listen to the audio here:
- Need help with your company’s data and analytics? Let us know!
- Join our free Slack group for marketers interested in analytics!
Machine-Generated Transcript
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Christopher Penn 0:00
In this week’s In-Ear Insights, we’re talking about prompt engineering, which is the art and science of writing prompts to talk to large language model AI systems, and how prompt engineering is really actually, software development.
This past weekend, I was doing a bit of ruminating on it, and realizing that with things like Microsoft copilot, and Google palm and GPT, four, and all the galaxy of acronyms out there, when we roll out the ability for a, an office worker to talk to PowerPoint, or Excel with a prompt, and access to GPT, for the large language model and have it do something like create a PowerPoint presentation from this Excel spreadsheet, we are really talking about how a human being talks to a computer, right how human being gives instructions to a computer, which is software development.
So the art of prompt engineering is how we write those programs to talk to computers.
And now that this is coming to Microsoft Office and Google Docs and stuff.
Everyone, every employee who works in office productivity software, is going to become a software developer, right? Because when they write prompts, they’re writing software.
So Katie, I want to ask you about the software development lifecycle and how when you think about prompt engineering, how it’s going to evolve.
So as a refresher, for folks who don’t remember, the software development cycle looks like this.
Okay.
Do you want to step through this real quick?
Katie Robbert 1:32
Yeah, absolutely.
So the software development lifecycle is exactly that.
It’s a lifecycle.
And it’s meant to be repeated over and over and over again.
So you start with your purpose? What’s the question we’re trying to answer? What’s the problem we’re trying to solve? Then you do the requirements gathering? What do I need? And you can use the five P’s for this? So what is the purpose people process platform performance? To understand, you know, what are all the bits and pieces that I need to answer the question to solve the problem, then you go into design.
And so design is an interesting one, because it doesn’t always mean, you know, putting together, you know, images and pictures design in this instance, could be, you know, based on your purpose and your requirements, it could be just a simple user interface, it could be something it could, you know, it doesn’t have to include like, really intricate graphics.
And so this is something that, you know, the requirements will dictate, then you develop, then you actually do the thing.
And so based on all this other information, you actually create the thing, and then you test it.
So you need to make sure that you have someone who isn’t you QA in this thing.
And then once you figure out what’s going on, what’s working, what’s not, then you, you know, refine it, you improve it, and then you deploy it to the public, and then you maintain it.
And then you start all over again, because once it’s out there, and people are using the thing, they’re going to have feedback, so you got to start over again, with what’s the problem we’re trying to solve.
And you just keep going around and around and around.
Until, you know, there’s no more feedback until you can’t be improve it.
And then you move on to a different version of the thing.
Christopher Penn 3:19
Okay, so now, with that, how, how do you see it applying to prompt engineering?
Katie Robbert 3:29
Well, I think you start at the top with the purpose.
So what is the problem that you’re trying to solve with prompt engineering? And so my understanding of prompt engineering is that it is the set of instructions that you are giving to something like a chat GPT or to the machine, so that you can get something back.
And so the problem that you’re trying to solve with prompt engineering is I have a question that I need answered, or I have, you know, a piece of content that I need a first draft for, or I have a set of notes that I need a summary for.
Christopher Penn 4:08
Okay, so then how do we apply requirements gathering to that?
Katie Robbert 4:14
So then, with the requirements gathering, if you think about the five P’s, so, you know, you already covered the purpose, you already know what question you’re trying to answer.
So then who are the people? So are is it just solely you? Who needs to get their outcome of this thing? Is there you know, is Chris asking me to do this because he needs this thing? So what are his needs? What are the questions is he’s trying to answer and so on and so forth.
And then what is the process? How am I going to go ahead and write this? Do I already know how to write this? Do I have to do some research? Do I have to ask others who have done prompt engineering before? Are there best practices? Can I borrow the platform? It could be chat GPT there’s a lot of other AI pools out there now and then performance, how do I know that I got the thing that I wanted.
And so those, that’s how you gather requirements for something like prompt engineering that feels very simple.
But you want to make sure that you’re not wasting your time.
Christopher Penn 5:13
Right.
And the reason this is so important is that as companies start to use large language models and prompt engineering within them, when everyone from the janitor to the CEO, will be using these in some capacity, you’re going to want to have a system for storing and deploying prompts at scale.
So it’s not just, you know, you sitting at your desk writing a blog post, this has the potential to be how you build certain types of enterprise software at the very least enterprise software processes.
So let’s walk through this let’s let’s talk about how we can apply this to actual real life situation.
So let’s start with we’ll start writing a blog post, because a lot of folks have been asking how do I write better props for for all these large language models? And the answer is the software development lifecycle.
So our purpose is we’re going to write a blog post for what SEO purposes.
Katie Robbert 6:10
Sir, what is SEO look like in 2023?
Christopher Penn 6:14
I like that.
Okay.
So we our requirements, what are our requirements for for this post?
Katie Robbert 6:21
So the requirements, I would say, you know, if I were doing this cold, I would say the requirements are, the problem I’m trying to solve is, I need a first draft.
The purpose is, I, as a marketer, need a first draft of a blog post about 20, about SEO and 2023, so that I can expedite the process of writing the blog post, because I needed yesterday and I’m procrastinating.
Christopher Penn 6:52
Okay.
So one of the things that prompt engineering needs because of the way these models work is there’s there are three parts there’s, there’s role, user and assistant, and each of these people, right, the five Ps has a distinct role to play.
When we write prompts, we have to implicitly declare those things.
So the the, the first part, which is the role part is where we tell the system what it who it is, or what it will kind of person it is.
So gonna start off where you will act as a blogger, you have expertise in blogging, content creation, long form content, content marketing content for SEO.
The reason we do this like this, is because we are giving it guardrails, we’re saying this is what the knowledge what words and associations we want to draw from.
Now, the next part is we want to also declare some knowledge about it.
So you have specialization in search, engine, optimization, engine marketing, SEO, and SEM, and optimizing content for Okay, so that tells it, you’re more barbells like these, these, this is the topic where you have expertise.
Katie Robbert 8:15
So I do want to point out, Chris, so as we were walking through the software development lifecycle, there was that sticky design phase.
And so typically, we think of that as graphic design.
In this case, what you’re doing is designing the prompt.
And so design, again, is sort of one of those it’s dependent on what it is that you’re doing design is going to mean different things.
And so you’re actually designing the prompt to then bring into development.
And so you have to do those two things in parallel, you could make an argument that, in this instance, design and development are the same thing.
And I think you’d be right.
Christopher Penn 8:56
That was actually I was going to ask is, is this feels like I’m doing a bit of both at the same time.
Katie Robbert 9:01
You are? And so it, you know, it depends on the kind of development that you’re doing, like, is there a front end? No, so then design is not going to mean that you’re going to bring in, you know, UX designers and IA and, you know, your typical creative directors, you don’t need that for this design, in this instance, is literally designing the prompt, but also at the same time.
You’re starting to get into that development phase, because you’re getting into those specifics.
And so the design is sort of that template that you’ll reuse over and over again.
And then the development, are there specific pieces of the prompt that are going to change each time you’re talking to the machine.
Christopher Penn 9:45
Okay.
So do you feel like we’re ready to move on to the next section, which is test?
Katie Robbert 9:52
So I do I mean, so.
And this is the thing with software development is at some point You have to say it’s good enough to start testing.
One of the downfalls of software development is that we get ourselves into this mindset that it has to be perfect and have zero issues the first time out.
And that’s just not true.
And that’s why assists.
That’s why a process like Agile is so great because it kind of it time boxes you into, here’s everything we can get done within two weeks, and then we’ll test it and then we’ll continue to iterate it versus more Waterfall development, which can feel like it just go on forever, and never even get to the testing phase.
So go ahead.
Christopher Penn 10:40
For the individual user, who’s just wants to improve their props, which is the better methodology for them? Is it more of an agile methodology or more of a waterfall methodology for the first single person who’s just trying to make better prompts for their blog posts,
Katie Robbert 10:55
I would argue that Agile is still a better process.
Because again, it’s the there is no such thing as perfect development, you know, you have to start testing, defining your requirements and design the prompts is going to save you a lot of, you know, computing time probably resource time and cost to the system itself.
But at some point, you have to say, Okay, I’ve been writing this for three straight days, and I still haven’t tested it, I should probably just get something in there and see what happens.
And so timeboxing it and saying, I’m going to do three iterations of this, or four iterations of this is going to help you start to refine it faster.
Whereas with waterfall, if you’ve if you’ve never seen a waterfall process, it literally looks like a set of stairs, where you have to complete one thing before you move on to the next.
And so with waterfall, one of the challenges with software development is you have to fully complete the software development before you test.
And it doesn’t allow for ease of going back to previous stages, like refining your requirements, like refining your design, if you find way down the line in the testing phase that you didn’t get something right.
Whereas agile lets you iterate faster, almost like this little circles.
Christopher Penn 12:16
Gotcha.
Okay, so while GPT-2, four cranks away here, we’re at the testing phase of this where we’ve just started having it do its thing.
And it is it is spitting out, you know, some I think, okay, stuff we have said, you know, the the world of SEO is evolving.
Core web vitals user experiences King, I would say that’s probably not the future of SEO, that’s the present day of SEO, ad machine learning Voice Search and natural language processing, zero click searches, which, again, that’s like from 2018, they could tell that it’s drawing on its heritage of information.
It’s already known semantic SEO and topic clusters.
Again, that’s that’s relatively old news.
But it is it is still relevant.
And so I think the the purpose here of satisfying writing a first draft of a blog post, you know, is is pretty okay, it’s, it’s, it’s done a pretty decent job.
But I feel like it’s missing some stuff.
Katie Robbert 13:22
So this is where in your requirements, you define the user acceptance testing criteria.
So if your performance measure if your success measure is just it wrote a blog post, well, then you’re never going to know is the blog post? Good enough? Does it have the following five facts are the facts from 2018 are the facts from 2023.
And so that’s where spending more time upfront and this up, I have never met a developer who enjoys doing requirements is like, you know what, let’s spend more time doing requirements, because it’s going to save me some time on the back end.
I’ve never ever in my life, like find me that person.
And I will give them $1.
But you have to spend that time up front, defining these things.
Because otherwise, this is where you start to waste time is, you know, I might look at this and know, I won’t know a lot about SEO just in this example and be like, Yeah, well, it wrote a post, okay, I’m done.
And you’re saying this as a subject matter expert, you’re saying half of this information is out of date.
And so how do we know that it has satisfied the requirements without defining those things?
Christopher Penn 14:39
So if I’m, again, if I’m a person, you know, sitting at my desk writing a prompt, this is the prompt that we came up with.
And as we saw it came up with okay stuff, right.
It certainly was lucid and coherent.
It made sense.
It wasn’t anything new because by definition can’t necessarily write anything new by drawing on its knowledge base, but there’s Some stuff that I feel like we probably should have had.
So is this now the improved cycle.
Katie Robbert 15:05
This is and so this is where you start refining.
So you have your baseline of what you started with.
And then based on the output that it gave you what in the output didn’t work.
So that’s where you start to refine.
And so typically, you would have some sort of a tracker system, or spreadsheet, or even just a document that says, you know, these are the things that work, don’t touch those things.
These are the things that didn’t work, let’s refine those.
And then you generate the content again, and see, did I fix the things that weren’t working? Or did I make them worse? And so it’s, you know, very simplistic bug tracking of, you know, did I close the bug? Is it you know, critical? Can I move forward with these things? Or do I have to start over again, because it’s blocking my ability to get this out to a, you know, public, you know, production site.
And, with testing, with QA testing, again, you know, the thing about Agile is it kind of time boxes it.
So you could be testing forever, you could be refining forever.
Or you could say, because you did your requirements.
This is the acceptance criteria, this is what I know, it’s good enough to get out there, it has to have the following five things when it has these five things, we can go out there, and then we can move it into maintenance mode.
Christopher Penn 16:32
Like with SEO, if we’re writing a post for SEO, we typically want to have some kind of focus some kind of topic or focus.
So I’ve added that in because it’s important.
I think the integration of large language models within search engines is important and obviously won’t know that.
So I think we need to provide that fact, I told you to give it some length, a specific length.
And they will told it to, to write in a specific type of tone.
So let’s see.
If it comes up with anything better with in our our improve cycle.
For convenience sake, I’m switching to GPT-3 point five instead of GPT.
Four for speed, because GPT four is substantially slower.
In reality, that would probably be part of the requirements as well as deciding which which of the engines of the models you’re going to use.
Well, yeah, that’s
Katie Robbert 17:17
the platform part of the piece.
So as we’re just watching this spin through, so this is, this is all part of the testing phase.
And so we found that version one was okay, but didn’t fully work.
So now we’re in the test and improve.
And the thing that actually, as I’m looking at this graphic, if you’re listening to this, we have the software development lifecycle up on our screen, if you want to switch over to YouTube and see this, I would actually almost because like, yeah, and a loop around tested improve, because that’s where you tend to spend the most time.
But again, sort of you have that acceptance criteria defined from your requirements gathering in order to know when you’re done.
And you can move on from that test and improve to deploy.
Christopher Penn 18:13
Yep.
Okay.
So we’ve got our post, this is better.
It matches more than tone, we’ve got future of SEO, we’ve got a key phrase I repeated four or five times.
So it’s actually did a great job there.
We checked the box on large language model integration.
So this post is in much better shape now than it was even just a couple of minutes ago.
So the question now becomes, how do we take this prompt? What is deploy mean?
Katie Robbert 18:46
So I would say that this then, in this specific context, deploy means that this then becomes the template and other people in your organization can start to reuse it, the set of instructions that you would need to give with this is basically calling out the words and phrases that need to be replaced, to switch the context.
And so, you know, if you’re always forever writing a blog, you can start as the you will act as a blogger, you have expertise in blogging, content creation, so and so forth, and then start to call out.
You know, you have a specialization in search engine optimization.
Well, my post is about PPC ads.
And so that’s the phrase that you start to switch out.
So you have a specialization in, you know, paid media or PPC ads, or whatever the topic is.
And so then you keep write the first draft of this blog answering the following questions.
Keep that part, you know, what is the future of SEO look like in 2023? What those PPC, what do PPC ads look like in 2023, or whatever the topic is, and so you start to create that template that in this instance, because you’re not If you don’t have a product to deploy, but you have this template that everyone can start to use, and it starts to become the standard.
And so that is good enough to give people a starting place.
Christopher Penn 20:15
I would agree with that.
And add to it that what you said is exactly right, this is the template.
And if we were to take this apart, I would maybe turn this into like a variable field.
So we’ll just put placeholders here for now.
Katie Robbert 20:37
Right, and so you can put those variables in there, I would also provide people with the version that you started with just so they can see.
But so you have both versions, you have the templated version, and then you have the example versions, then go oh, that’s what you mean by a key phrase.
Okay.
That’s what you mean by details.
Christopher Penn 20:57
Exactly.
And now, here’s where the system becomes really powerful.
This, which we’ve just worked on, can now go into actual software.
So I could now integrate, take that prompt that we were just working on, and putting in the different variables and provide it with, say, my entire SEO keyword list, right? And the associated questions that go with each of them.
And now instead of having a prompt, that I paste into chat, GPT-2 manually, now I have a system that I could generate 900 blog posts, that we know are going to be good quality, because we’ve proven the prompt, feed in and have it do its thing.
And then guess what, there’s my content marketing for the year, done, because we took the time to do this prop.
So want to open people’s eyes to the reality that when you’re doing product engineering, you’re doing software development.
And if you want it to be scalable, and be part of what it’s going to make your company some money, you’ve got to use the software development lifecycle.
Katie Robbert 22:06
And I agree with that.
And then when you get past the deploy phase, so you know, in this example, you said you put it into your software, do you create generate 900 blog posts for the year? Well, the platform’s themselves are rapidly changing.
And that’s where you get into this last phase of maintain.
And so maintenance includes things like user feedback, hey, I tried this prompt.
And I’m finding that it’s throwing an error at this at this stage, because the system has changed or the way in which it’s writing has changed, or whatever the thing is.
And so that’s where you start to continually refine it.
So I’ve never known a piece of software to ever just be static, it’s never just been deployed, and then sat there.
Anyone who does that doesn’t have good software.
And so this software is constantly being maintained and updated and bug fixes.
Because with software, there’s so many variables that are out of your control.
The way in which the language that you code in is going to have updates, new libraries, there’s going to be new methodologies, the platforms that you develop in are going to be constantly evolving.
And if you’re using third party software, like a chat TPCC that you didn’t develop yourself, then you need to make sure that your prompts are reflective of the changes in those systems.
And so if you don’t actively seek out user feedback, then you should make some sort of reminder of like, at least once a month, maybe once a quarter, depending on how often you’re using it, to check in to make sure that the piece of software, in this case, the prompt that you deployed, is still working, as expected, is still getting the same result, that user acceptance testing that you got from the original set of requirements.
And if not, you need to start you know, that’s where the process starts all over again, what is the purpose? What is the problem we’re trying to solve? And then you don’t have to write requirements from scratch, but just sort of note what’s changed.
And so then you can go through the other stages more quickly.
Christopher Penn 24:10
Exactly.
And even if you don’t do the software development part, you know, scaling this thing, you still want to have a system, some kind of governance around how you manage your library of these things, including, you may not want to give them away, right? Because you’re still developing software companies generally don’t give their source code away for free.
So that’s something to keep in mind to remember about the governance around the software do you develop?
Katie Robbert 24:36
Well, and then there’s also you know, how you check software in and out version history, version control, all that good stuff.
And so that gets sort of deeper into the weeds of how software development actually operates.
But it’s good enough for companies that are just sort of at the starting line of using prompt engineering, and systems like chat GPT to have MAE Add a collaborative Google Doc to say, you know, on this date on, you know, march 20 2023, this was the prompt that we all decided we were going to use, because we went through all of those stages.
And then we find out that, you know, chat GPT has a major update on, you know, April 15.
And so then you start to document on April 16.
This is the prompt we’re starting to use now.
And so you can see that version history as you go through it.
And you can see how it’s evolved.
So that when you go back and say, Hey, we want to do this thing, you already have all that documentation.
So you know, what’s working, what’s not working?
Christopher Penn 25:39
So one of the things that I know, You’ve asked me that I haven’t gotten around to yet is putting together a library of prompts for Trust Insights.
Right? What would that So is that what that looks like to you is just a bunch of Google Docs, how would you tell me hey, with as little overhead as possible, because nobody loves excess overhead? Right? How would How should I think about making this library of prompts so that all of us within Trust Insights can use them?
Katie Robbert 26:05
So I think it can be as simple as a Google Doc or a set of Google Docs to have something, you know, like a use case? Like, what is this prompt us for the date that it was last revised, the prompt with the variables, and then the example.
And then, you know, once a quarter, we go through the ones that we use most often and say, here’s what we’ve learned from this prompt, or anytime the prompt has to change, because something external has changed.
We just note it in that talk and say, last revised on, you know, June 2, or last week, or, you know, and if we look at the prompts and say, Oh, we never got around to using this particular prompt, then, you know, that tells us maybe that was the wrong use case.
But I think it could be a simple Google doc to say, if you’re looking for, you know, a really strong prompt for writing a blog post, it’s 500 words.
Here’s the prompt to start with.
Or if you’re looking to summarize a set of call notes.
Here’s the prompt to use for that.
Christopher Penn 27:06
Would you would you state it as user story? Or would you make it a prompt story?
Katie Robbert 27:12
What’s What’s the prompt story?
Christopher Penn 27:14
I don’t know, I just made that up.
Okay.
Then I don’t know,
Katie Robbert 27:17
I would state it as a user story, because you as the person are trying to accomplish a task by using these prompts with this software.
So it still kind of starts with, you know, as a time crunched marketer, I want to use a blog generating prompt so that I can have a first draft of a good blog post, or as a CEO, I want to use the notes summarizing prompt, so that I can understand what happened in the six other meetings that I missed because I was off, you know, goofing off and sailing around on my yacht.
Christopher Penn 28:00
How can you never invite me on your yacht?
Katie Robbert 28:02
It’s not See Worthy.
So yeah, user stories.
Christopher Penn 28:09
Okay.
So today, what we covered was essentially answering the question from us a more systematic perspective of how to write better prompts.
We talked about the guardrails, using the different keywords to write guardrails, we’ve talked about writing prompts in a templated format, so that they can scale up by with machines or with other employees, and a system for managing them.
If you’ve got some suggestions about how you’re managing your software development, because that’s what prompt engineering is.
And you want to share those stories, or maybe even your prompts.
You can go to our free slack or go to trust insights.ai Where slash analytics for marketers where you have over 3000 other marketers are asking and answering each other’s questions every single day.
And wherever it is, you watch or listen to the show.
If there’s a platform you’d rather have it on instead, go to trust insights.ai/t AI podcast, you can find the show on pretty much every place that’s available within reason.
And while you’re there, wherever it is you choose to consume the show, please leave us a rating and a review.
It does help to share the show.
Thanks for tuning in.
We’ll talk to you next time.
|
Need help with your marketing AI and analytics? |
You might also enjoy: |
|
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! |
Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. |
Newsletter
Podcast
