So What? Generative AI in 2024

So What? Marketing Analytics and Insights Live

airs every Thursday at 1 pm EST.

You can watch on YouTube Live. Be sure to subscribe and follow so you never miss an episode!

 

In this week’s episode of So What? we focus on Generative AI in 2024, questions we’ve received, and some resources to guide you in the right direction!

 

In this episode you’ll learn: 

• Answers to your most common questions about Generative AI
• What to expect for Generative AI in 2024
• Where to get trusted resources for staying up-to-date

Upcoming Episodes:

• All things Generative AI

 

Have a question or topic you’d like to see us cover? Reach out here: https://www.trustinsights.ai/resources/so-what-the-marketing-analytics-and-insights-show/

AI-Generated Transcript:

Katie Robbert 0:31
Well, hey, how are you everyone? Happy New Year. Welcome to so at the marketing analytics and insights live show. I’m Katie joined as always by Chris and John who I can, I can lift up today we’re never quite in the same configuration. So when you guys are like the two top boxes, it’s a little easier than me trying to figure out left from right. Happy New Year guys, welcome back.

Christopher Penn 0:55
We haven’t done this in a while. So it’s

Katie Robbert 1:02
our chief statistician, John is also our chief Grumpy Pants

John Wall 1:05
Chief, Chief Grinch.

Katie Robbert 1:10
So we’re gonna get right back into it. As expected, we’re talking about generative AI. Specifically, we’re talking about generative AI and 2024. And we’re doing today’s episode, Ask Me Anything style. So if you have questions, feel free to drop them in the comments. And we’ll try to get to as many as we can. We do have a bunch of questions already that we’re gonna try to get to. Because that’s that’s the topic du jour generative AI, what do I need to do? How do I use it? What does it mean for me? So we’re going to try to answer some of those questions today. And this, I suspect, guys will become something we do quite often on this show. So let’s just start at the top. Is it possible Chris, to sort of summarize what’s new, in generative AI, let’s just say like in the past 10 days, 10 days.

Christopher Penn 2:07
Up bunch of things have happened. And let’s scope it to just before the holidays as well. The big one, from a regulatory perspective, was the EU Parliament passing the EU AI Act, which is, we’ve talked about this in the past, the EU tends to be the leader in the world of regulation on major technology and stuff, especially when it comes to individual rights. It for those who may or may not remember, back in the last decade, the EU was that one of the planetary leaders on data privacy, culminating in 2018, with an act called GDPR, the general data protection regulation, which became sort of the gold standard planet wide for how to protect data, and many other legislative acts, California CCPA and CPRA, followed in the footprints prints left behind by GDPR. So many of the other privacy legislation that exists in the world is derivative of that in some fashion. If you comply with GDPR, you’re pretty much compliant with all the other, you know, follow on acts, if you will. Well, the EU AI act appears to be going the same general route. And that means that because it’s setting the tone, about the ways AI should be used, it’s likely to be the sort of the gold standard for it. And then other places and legislation will will probably follow it its footsteps. There’s an entire module in the Trust Insights course, the agenda of AI for marketers course, did you get a Trust Insights to s course I believe it’s chapter six. That goes through all the five major provision the six major provisions of the EU AI Act, the one that is really important, well, this, this we call it the five D process, right. Which is number was determined don’t use inhibited cases of AI. So there’s a whole bunch of things that EU AI analysts like yeah, don’t do this with AI. It’s all the things you’d expect. There is defend defend the rights of individual users, there is document which is document the use of AI, there’s disclosing the use of AI. I can’t remember the fifth one is why it’s in the course. The big one, the big one that affects most marketers is document and disclose. The EU AI Act makes disclosure of the use of AI mandatory. We’ve been saying last two years, you really should be disclosing the use of AI helps reinforce copyright and so on and so forth. And now, with this new legislative act, it’s not optional anymore. So when we if we if we were to put up you know, the new theme music for the show, we would have to disclose generated by AI if we want to be compliant with EU AI act. But if you put up a blog AUC post image and your blog and it’s a, you know, a dog on a skateboard wearing a tutu, you have some disclosure saying you’re made by AI so that you’re compliant with the law. That’s one of the really big developments. The other two things that have come out that I think was really interesting one is, in December, we saw Mr. Shah, the French company, release a new model called MC straw, which is a mixture of experts model. And if you think about a tool like ChatGPT, and the GPT-4 model, that’s like having an expert chef in your kitchen, right, so having Gordon Ramsay living in your kitchen, with all the swearing, big straw is kind of like having maybe a slightly less a chef and like seven sous chefs all in the kitchen at the same time. They’re not as good as a single Gordon Ramsay talent wise, but they’re a lot more productive because you can have that many more people simultaneously workout. Thanks. So mixture experts is this new model architecture. It’s it’s not newest from 1991. But the implementation is new. And that is, it is proving to be best in class for open models that you can download and run on your laptop. That’s number two. And then the third big thing is that we’re starting to see really interesting multimodal use cases, things that cross a lot of really weird boundaries just today. Speaking of news, just today, met up released an audio to image audio to video generation model that is just mind boggling. You give it sampled audio, like the conversation we’re having right now. And it renders 3d avatars that are photorealistic of synthetic people having that conversation. So Katie, if you and I were recording just an audio podcast, or if John and I wanted to resurrect like old episodes of marketing over coffee, this would create 3d realistic synthetic people having that conversation. So we could get some sampled video of us sitting in a coffee shop and recreate the first 100 episodes of marketing over coffee. So that’s but that’s multi modality in ways that is absolutely crazy. You take audio, turning into text, and then use that text to feed three different video models to create the synthetic stuff. So it’s just the last 10 days, that’s what’s been happening. Well, that’s

Katie Robbert 7:23
a lot. And I think that, you know, what we what we’ve seen so far is that the technology is changing so quickly. I totally think that you should absolutely take that number three point and do that with old episodes of marketing over coffee to you know, well, you know, and I mean, in all seriousness, because one of the things we know is that content is king. Let’s just say like people love that phrase content is king. Content is what people consume. That’s how they learn about you. It’s how they engage with you. And I think that marketing over coffee would be a really good opportunity, because you have so much content, and you have so much high quality content to really experiment with some of these tools and demonstrate the capabilities while also repurposing your content, maybe to a whole new audience. So you know, thank you for that summary, Chris. One of the questions that we have is what are some use cases for operations using AI.

Christopher Penn 8:27
So before we get into that, I want to highlight one of the use cases come up actually, in our Slack group. If you go to trust insights.ai/analytics remarketing you can join our Slack group there. It’s free to join. Was someone saying recently, hey, I love marketing over coffee, I would love to share it with our team, but they only speak Portuguese. What’s the likelihood that we can get marketing over coffee in Portuguese? And the answer used to be probably not soon. So again, this is an open source library released by medical it’s called seamless communications a seamless M 40. As the name of the model, I will give you the the upfront caveat, getting this thing to work is a pain in the ass. It requires extensive technical knowledge, and a willingness to roll up your sleeves and compile code, install libraries, all that fun stuff. But if you do it, and you’re willing to endure the pain of doing it or setting up the environment for it, what it does is pretty cool. You give it audio in one language, and it translates the audio into another language. So you could have marketing over coffee in Portuguese or In-Ear Insights in Ukrainian or any of the I think it’s like 73 Different supported languages. When we talk about marketing and even marketing operations, just having your content available to other markets is pretty cool, especially if it does not require additional work from you as the human right. So translation and translation technologies like this are pretty interesting. Now, I will also say seamless M 40. As buggy as hell. I’ve been playing with it extensively. And it’s got a little ways to go. But there have been some other new tools. Unity AI is another package that exists that can do voice cloning, again, running on your desktop. So I think from a marketing operations perspective, the being able to take stuff that you’ve already built, and putting it into many more different formats is huge.

Katie Robbert 10:32
I would agree with that. I think, you know, we often talk about the video transmedia framework where you’re basically starting with one piece of content, maybe it’s recorded video, and then you are pulling it apart to repurpose it. So you doing the work once and then getting a lot out of it. This almost feels like the opposite process where you’re starting with something small, like maybe an audio clip, maybe even a blog post. And using generative AI to create a video, video file, create an audio file, translate it into multiple languages, because, you know, Lord knows, we can’t wait for John to finish learning Portuguese.

John Wall 11:14
Lavazza blind spot on my part. Testing has proven that that’s a futile exercise.

Katie Robbert 11:24
What do you think, John? About? You know, so I mean, we’re using marketing over coffee as the example. Is this the kind of thing that you would be interested in using generative AI for?

John Wall 11:32
Yeah, the translation thing is really interesting, and for marketing over coffee, but I just think in general, I think we’re going to, there’s going to be some major societal shifts, when people can just speak every language, we’re very close to just real time translation on a phone, you know, when you go around, so I, yeah, that’s amazing. That’s fantastic. The meta thing that Chris was talking about where you know, you just give it an audio file, and it creates, you know, fresh content that’s really interesting, as far as in content production for movies and TV and stories and things like that. I mean, imagine, they basically get a star to train the model once and then that’s it, you just feed it, the scripts and the shows come out the other end. I mean, that’s kind of insane to think about, if I didn’t notice they had David Schwimmer as one of the audio files in there just to drive that point home for anybody that didn’t, you know, didn’t see that come and still. Yeah, the only other thing that has hit me recently, though, is, you know, content is king has always been the mantra. But I’ve heard a lot of people with the dawn of AI going with well, content, unique content is king, you have to have something that’s at least better than the average stuff out there, or you’re gonna be one of a million of the same thing out there.

Katie Robbert 12:45
I feel like that sentiment has always been true. It’s just never been out, outwardly stated. You know, you have to have good, high quality, unique content, in order for it to stand out. But now with generative AI, and we’ll get into some of the SEO and search questions. You know, it’s even more important to have really good high quality content. Otherwise, I think, what’s the term, Chris, the sameness of it all starts to happen, because if you’re using generative AI to create your content, it’s going to look the same as everyone else, because everyone’s using the same training models. And

Christopher Penn 13:21
that’s a really important point because part of what will make you successful using generative AI for content creation in particular, which is one of marketing’s big use cases, is leveraging your own data, something that came up in this week’s inbox insights Trust Insights newsletter, which you can get to TrustInsights.ai slash newsletter we’ve talked about data is one of the differentiators for how your your gender efforts will differ from others. So a couple really simple examples. One, if you are willing to invest the time and spend a whole lot lot of time digging around in the models themselves, you can get them to approximate writing style really well. Now writing style is one of those things that is very challenging because of imprecision in language. So the reason why AI doesn’t work as well as we would expect to just because our language is shockingly imprecise, even though we use it to communicate every single day, if I say he was wearing a red dress, there’s so little information in there about specifics to visualize that, that requires additional content to create a context with generative AI style is one of those things you can’t describe because it is so ambiguous because and it’s so difficult to take and there’s so many components of it. There’s word selection, vocabulary, sentence structure, pacing, grammar, etc, etc. What you can do is feed either examples to machine to have replicate a certain type of answer or do what’s called priming where Have you have you take a we do this with our synthetic version of Katie Katie GPT. I’ve taken probably close to 100,000 words of stuff that Katie has written, and then asked the language model on each of the was a 22 aspects of writing style, describe Katie’s writing style for this, and then taking that feedback, passed it back through a model and said, now summarize each of these things into the specific keywords to create a priming representation. That allows me to have a very, very complex system prompt that’s behind the scenes in Katy GPT. So when you talk to it, it is it is going to sound more like Katy than just you is something you can do with prompts alone. That has to come from your data, you have to have the data and it has to be in good condition. And you have to have the the techniques to do that to be able to create those primary representations of those few shot learning examples to generate really good output. But when you do, it’s a lot more unique. It is a lot it is not the same as lame, as our friend JB Aleksei same as lame. Yeah, if you were just saying, hey, write me a blog post about B2B marketing influences. Yeah, you’re gonna get the same as lame stuff. If you spend the time to build to codify your writing style. And then you add in unique content that you have maybe about B2B marketing influencers, then, when you ask the machine to create stuff drawing on your data, you’re going to get stuff that is unique to you, that has never been done before.

Katie Robbert 16:36
Does John GPT exist yet?

Christopher Penn 16:38
Not yet.

John Wall 16:41
If we want to podcasting flavored version of me, I guess, yeah, that would be the way to go.

Katie Robbert 16:48
I think Go ahead, Chris.

Christopher Penn 16:50
That’s a really, really important point that John just made. Things like podcasts are super, super valuable sources of information. One of the things that I did with Katie GPT, as part of that writing style development was to take, I think it was 80 episodes of In-Ear Insights, take the transcripts, I use some Python code that ChatGPT work to split the transcripts by speaker. So I took only Katie stuff, the only things KF said put that in the knowledge base, with 800 episodes of marketing over coffee with John being on every single one of them, we can take all 100 transcript, split them up and say this is the language of John GPT.

Katie Robbert 17:28
Well, and I think that, you know, we’ve we’ve talked about this before that and this is my stance is that new tech doesn’t solve old problems. And so your point about needing to have really good data and data quality, is, it’s always been important. But now where generative AI is so much in the forefront of what everyone’s trying to do, having really good data quality data governance, is vitally important in order for these systems to work correctly. You know, if you have a CRM system, depending on the size of your company, you can kind of go through and like fix things. As you see the errors. With a system like ChatGPT, or any sort of generative AI system, it’s harder to do that, because the models unless you’re someone like Chris, are a bit of a black box to you, you can’t really go in and tweak it, you have to know that the data that you’re putting in is really good quality. And that’s something you know, just to put a little bit of a plug that we can help with, we really have a lot of deep knowledge and expertise on data quality and data governance. To help you get to that point where the data that you have is really good, and you can feel confident putting it into your generative AI model. Okay, so another question. This is sort of a big question. But what is the best way to integrate different software systems with AI? And I feel like there’s going to be a number one, it depends number two, you know, what does that mean for you? And there’s probably a number three that I’m not thinking of. So yeah, let’s just let’s just start with maybe even helping people think about how to answer that question for themselves. Maybe that’s the way to approach it. I’ll

Christopher Penn 19:20
say what are you integrating? Here’s the thing, a language model. If we’re talking language models, actually, we can bring in language models and image models and video models. So the three big and audio models, okay, so for four big things, texts, audio, video, image, they all have all these things, have API’s. All these things have, you know, inputs and outputs. And so the question you have to ask is, how does this appliance fit into the overall workflow, right? It’s like having a blender. Where does the blender fit into the to the workflow if you’re making margaritas? It’s pretty important. If you’re making steak you’re not going to use the blender. Not if you’re not if you’re saying And, and so the first part is to figure out is it this is unsurprising to mirror that the Trust Insights five p framework, what are you trying to do? What’s the purpose? You know who’s involved what’s what are the existing processes? And then when you’re talking about the integration of generative AI, it is what platforms will you be connecting? And how will you be connecting them? Again, these things just have API’s. So like any other piece of software, so if you’ve used tools like Zapier, or If This Then That, or whatever, you know, kind of how API’s work is like connecting Lego blocks together, the language model is going to be used for language purposes in this flow. And then ultimately, you have some kind of our company measured by our performance. The one big warning for everyone to remember is that language models are good at language. Language models are not good at things that are not language. And so people try to use ChatGPT, for everything or dolly for everything, it’s just going to give you a bad result, because that’s not what it’s good at. It’s good at languages not good at things that are not language. So if you’re trying to have a do regression analysis for attribution modeling, as a marketing example, it’s not going to do well with it. It can write the code in Python, that will do the attribution modeling, we can do the math. And in fact, if you’ve used ChatGPT is advanced data analysis, that’s what it’s doing behind the scenes is it’s just writing code. And then the code is executing, and it’s looking at the outputs. So my caution there is when you’re talking about integrating different systems with AI, what are the language tasks that you’re trying to do if you’re using language models? And then are you connecting those things together? If you’re doing image tasks, you know, where does that image fit into your workflow, you’ve got to have your people and processes and your purpose documented up before you do the integrations.

Katie Robbert 21:52
You talked about how the advanced models are doing it, they’re writing the code, this is a shout out to Andy Crestodina. One of the things that he showed when I saw him speaking at Content Marketing World was you can ask the systems like, like generative AI to show your work or show me the code that you’ve written. And it’s like, Okay, here’s the code I’ve written, even if that’s not part of what you’re trying to accomplish, so that, you know, again, if you have someone like a Chris Penn on your team, you can say, here’s the code for all of this, is there a way that we can then now, put this together for our own scripts so that we can run this without having to bring it to generative AI, I just thought that that was such a really interesting pro tip is that you can ask it to show its work, and it will. And then if you understand how to read code, you can say, Oh, that’s not what I wanted it to do at all. Let me fix that. And that takes a bit of that mystery out of what is generative AI doing what I’m asking it to do something?

Christopher Penn 22:51
In fact, not can you shove chose work, you can have it give you the actual code, right. So this is something I was working on earlier today. Katie, one of the things Katie asked for help with this year was getting more data out of Hubspot. We have Hubspot. We have data in it. We know what we’ve done to set this thing up. But we also know that Hubspot integrations that they have a robust Python package to connect to their API. That’s not the language I coded very well, but ChatGPT Sure does. So I know what to ask it for us. Okay, let’s write the code to get all the data out of Hubspot, so that we can do the work that we need to do to build an attribution model. This is language, this is a task the language was really good at. So when we’re talking about integrating generative AI, into different systems, questions, we have to ask, do the existing systems have API’s? If they do great, guess what, now you have a way for general AI to talk to it, does the downstream output have some kind of API? In this case, one of the things that we use a lot is we use Google’s BigQuery. Because we want to be able to look at stuff in Data Studio. And so once this is done, it’s going to write its data to BigQuery. So that Katie can look at all the things she wants to look at, in Data Studio without needing to code. So that’s an example of his integration. But the first thing she had to do was she actually had to sit down, we have to sit down together and write out user stories for what this thing was supposed to be doing.

Katie Robbert 24:21
I think we wrote probably a couple of dozen user stories, and we found the similarities between them once we wrote them, but we really need to spend some time doing those requirements up front because, you know, Chris, you’re you’re reviewing all of these things, sort of at a at a very quick pace and at a high level. But the amount of time that actually goes into writing this code, you know, working with these API’s doing this integration, it’s a multi day, multi week, multi month process, and 10 times out of 10. There’s going to be errors the first time around so you have to go back after trouble. It’s Software Development essentially, and that there’s nothing speedy, about good. Qualify Good, good software development.

Christopher Penn 25:10
That’s true. By the way, protip. If you’re doing development with ChatGPT, one of the easiest things you can do to keep your keep it and you on the rails is to have your requirements written out, preferably in like a bullet point checklist. And that gets pasted to the end of your code. You can see here, my requirements for this code are in the code itself as a big comment. When ChatGPT processes the code, each time you are refreshing its memory as to what the requirements are. And it generally develops much more coherent outputs. If you do that.

Katie Robbert 25:43
When Chris Penn starts championing requirements and requirements, documentation, I know that I’ve done my job correctly.

John Wall 25:52
So you need to have a sword, you can like tap him as circuit.

Katie Robbert 25:57
That’s right, the champion of business requirements.

Christopher Penn 26:02
Well, it’s a language model it and requirements of language. And so if you want it to generate good language, you need to provide a good language. And there’s no better language for keeping an AI on the rails and requirements.

Katie Robbert 26:16
It’s funny how the same could be said about humans.

Christopher Penn 26:20
Now, because humans could ignore stuff machines, or you can compel us to pay much more close attention.

Katie Robbert 26:26
Well, we’re not gonna get into this debate today. But I feel like that is something that we’ll cover in the podcast at some point. All right. So here’s an interesting question. So there’s been a lot of chatter about SEO and search engines and how is it how is generative AI changing? So one of the questions that we got was, how do search engines deal with AI hallucinations, now and in the future? So before we get into that, Chris, can you define what an AI hallucination is?

Christopher Penn 26:58
This is a very complicated question to answer. Because it depends on the model. So

Katie Robbert 27:05
let’s say, let’s say it’s ChatGPT, which is the one that most people are using.

Christopher Penn 27:09
So this is a very, very complicated question. I thought

Katie Robbert 27:13
it was simplifying it but perhaps, you know,

Christopher Penn 27:18
when you bill when a company like OpenAI, or Google, or whoever builds one of these models, what they are doing, to put it in the simplest possible terms, is that’s mathematically incorrect. So if you are if you know, vectors, and embeddings, and stuff like that, this is completely wrong. But this conceptually Correct. They’re basically just building really big word clouds. And every word that they scan has a word cloud around it. And that is and what a model is, is a giant series of, of word clouds. That’s conceptually what’s going on. When you prompt a foundation model, which is the basic raw model compiled from your trillions of pages of text, it is essentially looking at how we’re different word clouds intersect and overlap. And what it produces is a statistically most relevant representation of the words that belong together, right words and phrases and stuff. The Transformers architecture is just looking at contextually which words belong next to each other. So can assemble these it’s actually assembling tokens, but that’s another topic for another time. A fat a pure foundation model generates what are called hallucinations, which are statistically relevant, but factually incorrect. Associations, foundation models generate hallucinations 100% of the time, when you look at a raw model, what’s coming out of it, it is coming out with just complete hallucinations all the time because it’s all mathematics. It’s all you know, this word belongs next to Discord have just seen frequently. When you get to instruct model, which is like what the first version of ChatGPT was before it was open to the public, all destructive GPT This is a model that has been tuned. That’s called supervised fine tuning where a company like OpenAI gives it hundreds of 1000s or maybe millions of examples. What is the color of the sky blue? Who is the president united states in 1996, Bill Clinton, it’s honestly just gazillions of these things. And it essentially tells the model, I want you to change the associations between all these words and phrases sentences to match the inputs and the outputs called supervised fine tuning. At that point, you’re starting to reduce hallucination rates, because now you’re saying, hey, the Libsyn, like, who’s president united states in 2001, George W. Bush, right that it’s not George HW Bush, it’s George W. Bush and so on and so forth. So it’s it’s conditioning the word associations to fall Have these pads that’s at the point where hallucination rates start to go down. Then the third step when you make these models is called reinforcement learning with human feedback. So as people use a tool like ChatGPT, it gets better over time. Because more and more it takes the inputs it gets from our usage of it. So if I’m writing Python code, and I say, hey, check your work. I don’t think it did this, right, it knows that its previous response wasn’t as good. And so it gets tuned and shaped. Along those lines. When we’re talking about hallucinations, it depends on the level of model that we’re using, most people are going to use a service like ChatGPT, or Google Bard, and the hallucination rate is reasonably low, it’s, it is almost never going to be zero. Because what comes out of the foundation model self is 100%, hallucination, like these things are hallucination machines by design. And so it’s this fine tuning that shapes the results. And awkwardly, the hallucinations are decreasing, but so is the utility because the more rules you put on these models, the worse their outputs get in terms of quality, the more sameness, less creativity, the more rules you put on. So for example, if you talk to ChatGPT, about a sensitive topic, right, or a pullet, political opinion, whatever it will, oftentimes you say, Nope, I can’t do that. Or if it does respond, it’s very, very bland and boring. And again, that’s that reinforcement learning human feedback has been tuning in to say, hey, you wish you’d respond to these certain ways, and it’s not as creative.

John Wall 31:40
You get that, John, remember that kid in junior high school, that would just lie all the time. But that’s what’s happening here, is coming up with an answer that considers most probable and throws it out there. And it doesn’t really matter if it’s true or not. So

Katie Robbert 31:57
Well, it’s funny, you know, to sort of continue along that thought process, John, I often say to my husband like he is, he’s like, he never gives you a straight answer the first time. But he says it was such authority that you can never tell if he’s kidding or not. And so he has his whole team, believing certain things about him and his life, because he says it was such authority that they don’t think to question that it’s not correct, including, you know, his favorite bands, and you know, who he was in a past life, and you know, what his hobbies are? They’re all lies, they’re all complete fabrications. But he says it in such a way that you’re like, Oh, it must be true. And I feel like that sort of along the lines, you know, in a very, very simplistic way of what you’re describing, Chris, very simplistic.

Christopher Penn 32:43
And the challenge with language is language itself is extremely imprecise. So a part of that imprecision is why we have we struggle sometimes with getting language models to do what we want. So I’ll give you an example. Let’s take the sent. A friend of mine sent this to me the other day I left hilariously, let’s take the sentence, I never said we should kill him. Right? Based on that sentence. That’s just text. But where you put the emphasis as a human changes the meaning if I say, I never said we should kill him, I’m saying it’s not my fault, right? If I said, I never said we should kill him as you killed the wrong person. I say, I never said we could kill him, I thought we were gonna hire a hitman. That same string of text changes its meaning based on our our spoken emphasis, which is not reflected in just the words themselves. And so you have this ambiguity. And so we talk about hallucination. A good part of hallucination is because we’re dealing with ambiguity in language language is inherently imprecise. That’s one of the reasons why a tool like ChatGPT does so well with coding because Python and other languages are precise language, it’s either runs or it doesn’t run. And there’s not a lot of ambiguity in the actual word assembly of the word the tokens. Whereas writing a blog post, yeah, there’s a million different ways to talk about B2B marketing influencers. And if we don’t provide a lot of specificity in our prompting, we are going to get ambiguous and or sometimes hallucinatory outputs.

John Wall 34:23
That’s a killer example. I love that. Because it is it plays differently, totally differently with every reading.

Christopher Penn 34:29
It does I mean, that’s, that’s actually an exercise. I’ve seen people do it like public speaking, things will take a sentence and say, Now change the emphasis on each time you read the sentence on a different word and see how it changes the meanings go. Even though, like English is not a tonal language, like Chinese, for example. That emphasis changes meaning and it’s not reflected in text. So going back to your question, Katie, about how search engines deal with AI hallucinations, but number one way is they don’t use models for knowledge, right? If you look at Microsoft, Bing and its use of ChatGPT, the the GPT-4 molecule perplexity AI. If you look at other tools like this, what they are doing is they’re acting as a language interpreter. And if you didn’t watch this, when you use Microsoft Bing with copilot, you will say, hey, you know, what is a good recipe for guacamole, you will see it, translate your text into Bing search queries, make those queries get results from Bing search, archive, and then aggregate and synthesize an answer. So very smartly, what Microsoft is doing is saying this language model is good at language. But we know it’s bad at facts. So we’re going to instead offload the fact part to our search engine, Bing comes back with some results, it synthesizes that those results into language, again, that is conversational in nature. And you’re like, oh, that’s helpful. Compare that to Google Bard, which does try to use in previous in the previous verses, it does try to use the knowledge of the model. And it comes up with hallucinations, in fact, so much so that they’ve actually put in a button that says, you know, check Google’s responses, and then it goes back. And Google’s the things that wrote that oh, yeah, actually, I lied, I completely made up. And so that that structure of going to a different source of truth is how search engines deal with that. That is contingent, obviously, on the search engine itself and how good its search engine results are right. So if you ask a search engine for something that is scientifically questionable, it will give it to you, right, you ask Google Hey, you know, me videos are so many texts that proves the earth is flat, like Google will find that for you. It’s wrong, but it will still it is factually wrong. But there’s a lot of content on the internet about that, and Google will surface that. And so part of this is one of the big challenges with AI. And, and with search itself is just because a machine can find it doesn’t mean it’s true.

Katie Robbert 37:05
I think that one of my favorite descriptions of that is you have the entire solar system when every other planet is a sphere. And then the earth is just a flat like thing. It’s like how do you think this? This is a thing? But okay, let’s go with it. So I, you know, I don’t want to I think misinformation is it’s a whole topic that we can cover. You know, but I think that that’s a really important disclaimer is that there’s a lot of misinformation that’s going to be spread even more quickly than we were, you know, being able to sort of get our arms around it before. One of the questions just because we’re sort of, we’ve been talking about this for quite a bit. One of the questions that came up was why image generation prompts don’t work as well as text generation prompts. And before you get into the weeds in the technical details, one of the things that you said to me the other day just totally blew my mind. We were recording the podcast, and you said something along the lines of generative AI can’t see what it’s showing you. And it was like it was such a factual simple statement. But it just like, blew my mind, because it’s a machine. It’s numbers, its code. It doesn’t have eyeballs, or weight to actually see. So if you say, this is a picture of a chicken, it’s taking your word for it if saying, Okay, this is a picture of a chicken, it can’t look at that and be like, John, you’re lying, that’s a donkey. And that just like, totally blew my mind. So image generation prompts, why don’t they work as well as text generation prompts,

Christopher Penn 38:55
because they’re different. They’re a they’re the two things is different mechanisms, right. So image generation is done with a technique called diffusion. Whereas text generation is done with this process called transformers. They’re similar mathematically under the hood, but the not the same thing. And because of the way image generation models are trained, the prompting structure is going to be different. The general process for training an image diffusion model is to take a gazillions number of images, and they’re associated captions, or text or alt text or whatever, and, and create associations. So you would have a picture of Jimmy Carter, a picture of Jimmy Carter at Camp David and so on and so forth. Those captions from news sources from Image catalogs, Amazon alt text for for visually impaired, all that becomes this this matrix of here’s a word and here’s what’s associated with this type of type of image. And then when you prompt an image generation model, you Give it words. And then what it does is it takes sort of a mathematical average, all the images it knows contain all these different words and try to meld them together and chips away and all the pixels until it gets something that reaches a mathematical average of the words that you use. So if you had dog wearing a pink tutu, right driving or riding a skateboard, it’s going to have sort of a mathematical understanding of those different concepts from all the things is trained on. Which is why in in pure image generation systems, you have these really strangely worded prompts like a picture of a an envelope, ejecting through the cockpit of an F 18 Super Hornet photorealistic highly realistic photo, 24 millimeter, lens DSLR, right, that’s an image generation prompt. makes very little sense to us a spoken word, but it’s because we know we’re taking advantage of captions and highlights and descriptions from museum walls that that these image models were trained on. Because of that, if you don’t know the syntax of how an image generation model works, you’re going to get subpar results. Combine that with the fact that what is typically put into image generation models are very short snippets of text, again, their captions or alt text, they’re not paragraphs of description, John, as an example, describe this.

John Wall 41:22
That would be purple chicken with Santa hat and scarf.

Christopher Penn 41:28
Right. But you could spend a lot of time talking about this, right? Like this is clearly holiday chicken, it’s probably a plastic college chicken, it seems to light up, it’s got a scarf, it’s got to have this little bit of mistletoe on the hats, it’s a painted piece of plastic. The chicken doesn’t look super thrilled to be here. There’s so many different things you could say about this. What’s the old adage from, from the classics, right? a picture’s worth 10,000 words, you could easily you shouldn’t, but you could easily write 10,000 words about this thing. And yet, image models been trained on maybe 20. Right. And so when you don’t get good performance out of image models, because the words you’re using don’t occur in captions, image captures essentially what it boils down to. Now, there are services like Stable Diffusion, which are trying to help build libraries, where machines can auto tag and create more enhanced captions on images they already have control over. But that’s a long, complicated and expensive process. And so the reason why image generation prompts don’t work as well, if you’re just trying to use natural language is because the models were not trained that way. And the best models that that are, you know, Trump’s easy, simple forms, typically have very large datasets. But they still require a little bit of tweaking, when you use the dolly function inside ChatGPT. What it’s doing is very much what Bing is doing. It’s taking your words, translating it to a dolly compatible prompt and then pushing that as to the system. And that’s where that our conversation point came from earlier, Katie, which is it’s passing a prompt to Dolly, it can’t see what dolly is doing. It has no idea what dolly is doing. So it just gets the result back from the API says, Here’s your thing exists. I like it. And you know, I had this problem with a client, I was doing a picture of four people in a car on a road trip, and I kept putting five people in the car. And I keep saying no, no, try again. You had four people in the car was five people. That’s correct. Again, that’s because of the caption data that was trained on I what I ended up doing is, say, make a picture of three people on a road trip and a surprise, I got a cup of coffee for people, like what is wrong with with this training dataset. But as clearly clearly got some problems counting. I

Katie Robbert 43:52
feel like creative directors around the world are rejoicing saying, finally, people understand our pain when our clients say, I don’t know, just make something pretty, or I don’t make it blue. Like what does that mean? What shade of blue? What kind of blue? What is it about it that you want to be blue? And so it’s what strikes me and you know, we’ve sort of talked about this quite a bunch as well is that all of the struggles that people are having working with these generative AI systems are the exact same struggles that people have as managers, as communicators as delegators. The difference here and you know, Chris, this is where you get excited is that the machine doesn’t talk back and say I need to take a break and go get nice coffee. The machine says Okay, what else do you want me to do? And it’s very agreeable, and it’s very compliant. But you still have to know what it is that you’re asking. And so for someone like me seeing this just sort of play out, I’m like, well, now All of the struggles that I’ve had trying to get people to document requirements understand why they’re important. Think about specificity when you’re delegating, or you know, giving instruction like, this is all coming to light. It’s all surfacing of like, and this is why. And so now we can take all of these skills that we’re learning, working with generative AI systems, and reapply that back to our teams like, Oh, now I understand. If I’m clear, in my instruction, when I asked my team members to do something, they’ll know what I’m asking. And there won’t be frustration on both sides to be like, we didn’t tell me what you wanted. You just said fancy font. And I know what fancy font means.

Christopher Penn 45:38
Yep. And I think that’s a really good example, plus, what just came in in the comments as well, with a lot of these systems. We have to think about them, the way they work under the hood. And again, there’s something actually going to be coming covering in an upcoming course, in the Trust Insights Academy on advanced prompt engineering. But the way they work is by essentially selecting probabilities right under the hood, there’s they’re selecting different probabilities. When you give it a task that is entirely self contained, like, make a one page opt in site, the number of probabilities can be invoked for that are so many that the machine will not be able to complete the job. It’s like saying, Hey, drive from New York to San Diego, like just give me just me three, three sets of three bullet points on how to drive from New York to San Diego, like well get in your car, start driving until you get there, right, it’s gonna be unhelpful. If you say, give me turn by turn instructions on how to drive from New York to San Diego, you’re gonna get a much better result, right? The same is true of language models. It’s a system called chain of thought, or the ELB has a variety of points describing but essentially, these models when they’re choosing probabilities, they need room to do so they need to choose probabilities, then consider everything a given you choose new probabilities and consider everything given it and so on and so forth. It’s recursive. It’s the nature of the Transformers architecture. And so when you say, instead of a, give me a one page, opt in site, say, let’s first talk through, how would you do this, right and have it write an outline, and then say, Great, now let’s talk about step one, right in the HTML, step two, writing the body copy step three, writing that things. And if you follow those steps, it will generate really good results. When I’m doing something like the this code that we were working on earlier. I do not say we’re going to write code to interface with Hubspot API, because I will get nothing useful back. What I’ll say is first outline the steps that you need. Now frame out just the function names and placeholders for the functionality. And it will write that and then I will say, okay, great. Now let’s write function, insert to database, write that and it does. Okay, now let’s write the safe int conversion function. Let’s write that. And then you assemble the pieces because you’re doing chain of thought. And you end up with very strong coherent outputs. But you can’t do it in one shot has to be done in multiple steps. To what you’re saying, Katie, very much like working with a really smart intern. You can’t say, hey, go make me a website. No, no, you got to tell the intern step by step and work with the intern step by step to get the output that you want.

Katie Robbert 48:25
So as we’re wrapping up, because we’re just about at time, Chris, do you feel like working with generative AI systems is now going to change the way that you interact with human team members?

Christopher Penn 48:40
Um, yes, it makes me less tolerant of working with human team members, because I have to deal with all their needs and stuff. And it’s like, I can’t do that, you know, I’m having a bad day, like, No, I’m just going to work with the machines because machines don’t have bad days. I can work with them. I can, I can give them the exact requirements. I don’t have to worry about hurting their feelings. I don’t have to worry about all the frailties of flesh and blood it is, to me they are the perfect co workers.

Katie Robbert 49:08
So let me rephrase. Do you think it has changed your approach in terms of giving instruction to a human team member?

Christopher Penn 49:18
Oh, absolutely. In fact, we had, we were working with an intern last year. And I found myself in giving directions to the intern just essentially writing prompts. Like I wrote prompts for the for the the human intern. Yeah.

Katie Robbert 49:34
John, any final thoughts?

John Wall 49:37
No, I think that’s a great place to wrap it up. They’re great for documentation. I think that’s the right. Right approach. And it is, you know, this iterative approach is better and we just have to deal with the human inconsistencies. That’s just part of the mix, but that’s okay.

Katie Robbert 49:52
I, you know, there’s a lot of questions that we didn’t get to so we’ll definitely be doing more of these. Ama style. Live streams over the next few weeks. The other thing I just want to note, for those of you who have been watching the show is that Chris is a wealth of knowledge, you know, we haven’t yet been able to stump you on the details. And so if you’re interested in bringing Chris in to your organization, or your agency, you can visit us at trust insights.ai/ai services, and we have a lot of different options for you to be able to work with, you know, the caliber of expertise, Chris, that you have on AI in general, but also more specifically on generative AI. And then more deeply technical we’ve been doing, we’re focusing this year a lot on education. So of course, we have our new generative AI course that launched at the end of last year, you can get that at trust insights.ai/ai course, if you want something that’s more self paced. But if you’re looking to bring in that expertise in house, and have someone really talked to your team about generative AI, then I think that, you know, bringing Chris in is a really good option.

Christopher Penn 51:04
I think that’s a great place to end. And if you folks have other questions that you have about this stuff, you know, by all means, let us know because we’re always trying to, we’re trying to find, convert that knowledge into useful practices, right, because the technology is really cool. And I really could spend all day talking about it. But at the end of the day, it has to convert into something useful. So that’s gonna do it for this week. We will talk to you all next time. Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it. For more resources. And to learn more, check out the Trust Insights podcast at trust insights.ai/t AI podcast, and a weekly email newsletter at trust insights.ai/newsletter Got questions about what you saw in today’s episode? Join our free analytics for markers slack group at trust insights.ai/analytics for marketers, see you next time.

Transcribed by https://otter.ai

 

---

Need help with your marketing AI and analytics? Get help with your AI strategy » Learn generative AI for marketers with our new course » Prove the ROI of your marketing with our advanced analytics services » Just got a "silly" question you want an answer to? Ask us here »	You might also enjoy: In-Ear Insights: The Trust Insights Podcast Disclosure of AI and Protection of Copyright AI and Copyright Law: How Copyright Applies to AI-Generated Content How AI is Changing Marketing Keynote How To Build Better Analytics Dashboards and Reporting
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! Click here to subscribe now »	Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. Click here for Google Podcasts » Click here for Apple Podcasts » Link for all other podcasting software »