In-Ear Insights: How to Apply Agile Prompt Engineering

In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss using Agile principles for prompt engineering – agile prompt engineering. You will learn the importance of process development and project management when working with large language models (LLMs) like ChatGPT. Discover how to use the 5 P’s framework and the RACE framework to create effective prompts and streamline your workflow. Finally, understand why documentation and planning are essential for scaling your use of prompts across your organization.

Watch the video here:

https://youtu.be/kNADHfJ6g7U

Can’t see anything? Watch it on YouTube here.

Listen to the audio here:

Download the MP3 audio here.

[podcastsponsor]

Machine-Generated Transcript

What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.

Christopher Penn 0:00

In this week’s In-Ear Insights, we’re in part three of our agile prompt engineering/agile AI series.

In episode one, we covered what Agile is; in part two, we covered the basics of prompt engineering, the RACE framework, and where the opportunities are to use generative AI.

Today, let’s talk about agile and prompt engineering—taking the peanut butter and jelly and putting them together—or I guess the peanut butter and chocolate, depending on how old you are.

Katie, how do we put these two great things together?

Katie Robbert 0:37

Personally, I could go for peanut butter and fluff.

I do like peanut butter and chocolate, though.

But I think the point is, how can we mix together Agile methodology and prompt engineering? The answer will shock you: requirements gathering and the five Ps.

You cannot escape it; you cannot get away from doing documentation.

I’m not even sorry.

Prompt engineering, at its core, is a form of development, and development—software development—lends itself to Agile methodology.

That’s why Agile methodology came about.

So the question is, what is the prompt engineering lifecycle? Similar to the development lifecycle? And where does Agile fit into it? First things first, what the heck are you doing? What are you trying to accomplish? And that’s where the five P framework comes in.

The five Ps are purpose, people, process, platform, and performance.

Purpose: What is the question you’re trying to answer? People: Who’s involved? Who’s going to be impacted by this? Process: How are you doing the thing? Platform: What tools do you need? And performance: What is your outcome? Chris, I don’t know if you have it handy, but I’ve started to put together what we’re calling the prompt engineering lifecycle.

We can walk through where all of these things fit in.

Prompt engineering, similar to software development, can go through a basic core set of steps.

Software engineering would do planning, development, testing, deployment, and maintenance.

Prompt engineering can go through those same steps.

What I’ve included here is how you can utilize different frameworks at each of the phases to make your prompt engineering process more streamlined.

With planning, you start with the five Ps.

You figure out your purpose statement: Why am I doing this? What problem am I trying to solve? Who needs to give feedback on this thing? Who’s going to benefit from the outcome? What process do I need—is this like extracting data, doing market research, whatever it is? What platforms—this is your generative AI system, in addition to where the outcome is going to live, if there’s other platforms like your CMS or your marketing automation.

And then your outcome, after I do this thing, this big, amorphous thing, what is my end result? How do I measure success? That’s your planning phase.

Then you have your development and testing phases—this is where you can get agile, be more iterative.

The development phase is where you’re using frameworks like RACE.

We’ve put together a very straightforward framework that you can use as you’re starting to build out your prompts based on your five Ps.

RACE stands for Role, Action, Context, and Execute.

Role: Who are you? This is the information that you would borrow from the people part of your requirements.

Action: This is part of your outcome, what do you need it to do? Context: This is your process, your purpose, all of the information you’ve gathered.

And then Execute: Again, this comes from your outcome.

So, if you’re saying, “You need to be a B2B marketer, I need you to help me write a series of sales emails,” here’s all of the context: Here’s my audience, here’s the systems, here’s the data, here’s the offers, here’s the goals.

Now execute.

You get all of that information from the five Ps.

You put in your prompt, hit go, and then you look at it and say, “That’s not quite right.” What can I do? This is when you move on to the testing phase of the prompt engineering lifecycle.

This is where you can utilize the PAIR framework.

You’ve started with RACE, which gives you your core set of requirements, your basics.

Then, in the testing phase, this is where you get truly iterative, you start to move on to the power questions.

PAIR stands for Prime, Augment, Refresh, and Evaluate.

With Prime, you’ve given it all of this information, and then you say, “Well, what do you know about this thing? Am I even asking the right questions? What am I forgetting?” And so you start to walk through each of these sets of questions.

This is where you’re iterating, finding out more information, refining, tuning, getting closer to that final outcome that you stated in your five P planning phase.

Once you’re satisfied, you can say, “Okay, let’s clean this up and make sure that we have a very clean prompt that we can use again and again and again,” because chances are, you’re going to be asked to do this thing again.

That’s part of your deployment.

That’s what goes into your prompt library.

You say, “This is my big long, you know, 10-page prompt, but it’s repeatable, I can use it over and over again.” So that’s your deployment.

And then the maintenance part of the prompt engineering lifecycle is revisiting these prompts regularly, to give it new information.

Maybe you have a new set of customers, maybe you’ve entered a new vertical, maybe you have new offers, maybe you’ve changed the marketing automation system that you’re using to send out emails.

So all together, this is the prompt engineering lifecycle: planning, where you do the five Ps; development, where you use the RACE framework; testing, where you use the PAIR framework; deployment, where you put things into your prompt library; and maintenance, where you make sure that your prompts are being kept up to date.

Christopher Penn 6:49

I want to point out that the RACE framework is software.

This is an example of a Python script, right? This is, in this case, to make a bot for Discord.

And you’ll note up here are the imports.

This is the equivalent of the Role telling the script, “Hey, here’s the things you’re going to need.” There are a bunch of actions, like here are a couple of functions that you have.

There is a bunch of context here, all the variables and connections and things that you’re going to need to make.

And then at the very bottom is the main class where the script executes.

So Role, Action, Context, Execute—literally, you can see it in real code, in Python code.

So when we’re talking about prompt engineering and treating it like software, it’s not just because it sounds cool, but this is actually how software functions.

If you want thorough and complete responses from generative AI, incorporate that.

Just as with the PAIR framework, part of refining code is human review, code review.

Part of it is googling on Stack Exchange and all these other sites for code snippets, like, “How do I do this? I don’t remember how to do this thing.” Part of it is pressing the Run button and seeing if the code even runs.

So even though we have these frameworks, they’re not made up out of thin air.

These are made from software development practices.

I guess in this section of the show, Katie, what Agile practices, if any, would you incorporate, particularly into development and testing?

Katie Robbert 8:42

So I think this is where you would want to have your backlog.

You would start with the universe of things.

I did my five Ps; I know my purpose.

So if my purpose is—I think in the last episode, we were talking about getting scopes of work done—I would want to list out the universe of things that it does and does not include.

This is anything from: it always has to include a strategy, tactics, prerequisites, deliverables, timeline.

Those are non-negotiables.

And so you start to prioritize those as the highest pieces on your backlog.

Those are always going to be right at the top.

Other things that might be background information, might be mission statements, might—depending on who the potential client is—that may or may not be something that they need to see.

They may need to have language in there about our approach to creating proprietary software or what it looks like to work with an agency that utilizes generative AI for client work.

Those become just a list of features on your backlog.

So that when you start to build your RACE framework, when you start to get into development, you can pull from your backlog and say, “Is this a part of it already, or is this not a priority right now?” As you’re going through and building your prompts, you’re going to want to include or take out some of those backlog items.

Let’s say you have a client that you need to build a statement of work for, you have all of the background information in the prompt that you’ve built previously.

So you pull that out of your prompt library, you bring it over to your development environment, which is the generative AI interface, and you say, “What stays and what goes based on the five Ps?” What information do I need, and what is excessive in this particular instance? That’s where you start to do your iterating.

So you say, “Okay, if I take this out, what does that look like? Okay, that’s not working.

If I include this, what does that look like?” And so you’re, in some ways, doing a very concentrated set of sprints.

Software development sprints usually take about two weeks, but in this instance, it’s not going to take—it shouldn’t take two weeks, it should probably take a matter of minutes, or even a few hours, depending on how complex the ask is.

But you can iterate over and over again by pulling things out and putting things back in.

And that becomes its own set of requirements, its own sprint, its own testing, to see how it fits into the larger picture.

So it’s not just, “Okay, I’ve put everything into the prompt, I’ve hit go, and I’m done.”

Christopher Penn 11:35

Right? Especially when we’re talking about deployment.

This is where agile, the agile framework, and all these frameworks really matter because once you go from consumer use of generative AI to enterprise use, where you are taking a prompt and scaling its use, you’ve got to get development and testing right.

Because once that goes into a piece of software, then you have limited opportunities to make changes.

I will show you an example here.

This is from a piece of code that we use for Google Analytics.

We extract our data from Google Analytics and we actually take it and put it into a prompt.

So we take tabular data from our GA4, put it into a prompt, and then that gets fed to a language model.

You can see there’s a very extensive prompt here of how this thing should work.

Once it’s done, there’s limited opportunities to make changes to it because it goes into a production server and starts running.

So if we just do the RACE framework and we don’t do that testing, the PAIR framework, then we may have a prompt that underperforms, we may have a prompt that returns unreliable, non-standard results, we may have a prompt that forgets things, we may have a prompt that just doesn’t do what we wanted from the five Ps.

So that’s why this stuff matters.

That’s why agile can matter, because this is now a literal piece—this is literal enterprise code that is going to go into production, that’s going to be used, and has to be reliable.

Katie Robbert 13:24

The biggest takeaway from software development that I learned very quickly was that it’s really hard to back out your code and make changes once it’s already been committed.

What I mean by committed is you’re looking at a whole entire page of code; all of the code is dependent on itself.

By removing one individual line, you could basically be screwing up the entire whatever it is you built because it’s all—it’s a lot of “if this, then that” statements.

And those statements become dependent on each other, so they all become really tightly nested.

This was a frustration when I was working with my developers because somebody would come in and say, “Oh, you know what? I’ve changed my mind, I think I want to do this thing over here instead.” And they were like, “What, can’t you just change that line of code?” And the answer is, “Yeah, I can, if you want everything to break,” which is why QA was always so important.

We always had to test things from top to bottom, not just that one thing they changed, because there are so many nested dependencies when you’re writing code.

Agile allows you to almost keep things like siloed so that you’re developing them almost like in a vacuum and saying, “I just want to focus on this one piece here before I integrate it into the larger codebase.” Whereas with waterfall, you’re saying, “I’m going to do everything, test it, and then have to go back and fix things from top to bottom again.” Whereas with agile, you can say, “This one tiny little piece right here that I’m keeping separate from everything else I’m working on—and then I’m going to integrate it into the bigger picture.” And that becomes its own set of requirements, its own sprint, its own testing, to see how it fits into the larger picture.

When you’re talking about prompt engineering, especially if you’re using generative AI to build code, or whatever it is you’re trying to do, it’s really good to keep all of these pieces in this sort of Agile methodology so that you can focus on one piece at a time and not break everything as a whole.

The other side of that is, if you’re not doing your requirements, and something breaks, it’s really hard to figure out what it was that, you know, changed everything, because you have nothing to point back to and say, “What was I trying to do?” This is why whenever someone tells me, “Well, I just, I just threw it in there, I just winged it,” and I’ve done this, too, I am totally guilty of this—you can’t go back and say, “Well, what did I do? What changed?” Because you have nothing, no documentation to point back to.

Christopher Penn 16:11

I can speak to this; this is a literal thing that happened.

There’s one section here in my code where I have to get tabular data out of GA4 and put it into the prompt.

And I had to pull this piece out and actually work on it separately because I could not get it to look right, it kept going like all sorts of sideways and stuff like that.

Having that Agile methodology, having that set of Agile processes, allows me to carve this thing up and say, “Okay, these other pieces are baked, they’re already set, this middle piece here that, for whatever reason, will not come out as a proper looking table, we can just work on that and try to fix that.

So that by the time we’re ready for the next sprint, I’ve either solved that part or I have not solved that part.” Again, for consumer use of generative AI where you’re just talking to a tool like Gemini directly, you pull up the prompt and have at it, it is not as important.

But once you start talking about organizational use of prompts, enterprise use of prompts, deployment within code, boy, does this stuff matter.

Katie Robbert 17:24

It’s true.

I can even—and I’ll admit, I think I totally missed the boat on building a repeatable process.

About a month ago, we built our ideal customer profile, and we put a process together for that.

And I said, I took the responsibility to say, “I’m going to take this ideal customer profile, and I’m going to break it down into the individual, you know, profiles that we talked about.” And I’ll be honest, I had an idea to do that one morning, and I winged it.

And I built what I thought was a really good second version.

And I keep saying I need to go back to that.

But it’s been long enough that I didn’t document my process, I didn’t—I winged it, I’ll be honest, I winged it, I screwed up.

Because now when I want to go back and do it again, I have to try to reverse engineer, and that’s going to take a heck of a lot longer, instead of just copying and pasting something from my prompt library and saying, “Okay, now I want this outcome instead.” And that’s on me, like, shame on me for not following my own advice, and we all do it.

But then we learn from it, we see how painful it is—figuratively and literally how painful it is—to not have done that planning process at the beginning, because then we can’t be agile, we can’t do things in a more efficient way.

So now I have to start all over again.

And hopefully, I’m going to learn from my mistakes and not, you know, wing it next time.

Christopher Penn 18:57

This is again where we can take software development practices and implement them even just within our consumer prompts.

This is an example of a translation prompt that I wrote.

What’s different about it is it obviously has a bit of Role, it has a bunch of Action steps and Context.

But then there’s a section for variables where I could just say, “This is the source language I’m working with, this is the target language, here’s how I want to do it, here’s the formality and ambiguity and document context”—this is all context information here that, like code, I would change based on its usage.

But then if I do that, I retain all the other parts of this prompt that are really good, that are solid and baked.

And all I have to do is adapt that portion.

So this isn’t even really—this isn’t even in the testing phase.

This is back at the RACE framework of Role, Action, Context, Execute.

This context section should be templated, this context section should be standard variables, so that if I give this prompt to you, Katie, you can take this and use it immediately, like, “Okay, here’s what I’m gonna be translating, here’s the target language I’m translating into,” etc., etc.

And you’ll get the benefit of it.

If you build something like this for an ideal customer profile, you could hand the whole thing to me and say, “Okay, Chris, now I don’t want you to do an ideal customer profile for a marketer in the healthcare vertical.” And I just change, “target industry: healthcare,” and “target company size: this,” and the rest of the prompt gets preserved.

So this whole process, in some ways, it’s—we’re less talking about prompt engineering of individual prompts, and we’re more talking about the process of using prompts to build prompts, in a way.

Katie Robbert 20:38

It’s process development.

And project management, when you talk about waterfall, we talk about Agile—it’s so much about the process.

And it’s the part that people hate, they just want to do the thing, they just want to put fingers to keyboard, press buttons and do the thing and have a thing.

But in order to have a thing, you have to step back and do those requirements gathering, you have to do that process development, and then, believe it or not, then you can just go ahead and do the thing.

And the thing comes so much faster because you’ve done all of that upfront work.

And that’s the part that is—I feel like mind blowing to people is like, “Well, it’s gonna take me so long to do those requirements.” Possibly, but you do them once.

You don’t have to do them over and over and over.

Do them once.

And that’s where we start to talk about agile.

That’s why agile is constant planning.

But you get into that habit, you build that muscle memory, and it becomes second nature, where you’re just like, “Let me do my user stories, let me write out the five Ps, boom, I can do the thing.” I already have all of my resources, I have those prompts in my library, I’ve already done that research, I’ve already done that work, I can just change the context and bang out 10x the amount of work that I would have done previously.

Christopher Penn 21:58

Exactly.

Going back to our first episode in the series, the difference between a prefix menu versus a short order cook.

The short order cook has to get a lot of prep work done, but then you can adapt as new things come in.

If you’ve done all that prep work, and you have the prompt library and you’ve built your prompts to be modular and customizable and to be used by not just you, but other people in your organization, you are ready to handle whatever comes your way.

So to wrap up, agile absolutely does apply to prompt engineering because prompt engineering is software development.

And of course, that’s literally where agile came from, with software development.

So try incorporating these processes, starting with the five Ps, and then moving on into the RACE framework, the PAIR framework, how you’re going to deploy this stuff, and then how you’re going to maintain it.

If you’ve got some thoughts about how you are using Agile with prompt engineering and generative AI, and you want to share them, hop on over to our free Slack group, go to TrustInsights.ai/analyticsformarketers, where you have over 3000 other marketers asking and answering each other’s questions every single day.

And wherever it is you’ve consumed the show, if there’s a channel you’d rather have it on instead, go to TrustInsights.ai/tipodcast.

We can be found where most podcasts are served.

And while you’re on your channel of choice, please leave us a rating and a review.

It does help to share the show.

Thanks for tuning in.

I will talk to you in the next one.

Need help with your marketing AI and analytics?