In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss how to separate artificial intelligence speed from actual business value and what we value from humans in an age of AI. You will discover why productivity charts hide critical context that changes everything. You will learn how to spot the difference between quick output and solid results. You will master a simple framework for letting machines handle data while you keep full control over every choice. You will walk away with practical steps to scale your daily workload without sacrificing your unique perspective.
00:00 – Introduction
02:15 – The misleading productivity chart
05:40 – Decoding the midterm results
09:10 – When tests measure the wrong skills
13:25 – The seven ways to use AI properly
18:50 – Why humans must keep the steering wheel
23:40 – Practical tools for smarter workflows
28:15 – Fixing the education gap
32:00 – Call to action
Press play to uncover how you can turn artificial intelligence into a reliable partner that amplifies your best work.
Watch the video here:
Can’t see anything? Watch it on YouTube here.
Listen to the audio here:
- Need help with your company’s data and analytics? Let us know!
- Join our free Slack group for marketers interested in analytics!
[podcastsponsor]
Machine-Generated Transcript
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Christopher S. Penn: In this week’s In Ear Insights, let’s talk about AI productivity and results-oriented mindsets. We talk a lot about AI productivity gains, and a lot of people are rightfully asking, “Where’s the beef?” Going back to the 1980s Wendy’s commercial. I want to show you a chart. Katie, I want to get your reaction to this chart on some AI productivity gains and whether you would consider this a success or not. So let me bring this chart up here. This is from Brown University. We have individual workers, we have their original productivity scores in the gray, their AI-enhanced scores where they’re using an AI tool and how they increased. And the green numbers represent the percent change. Now, without any other context, at a first glance, what do you make of this? Is this an AI success story?
Katie Robbert: Not necessarily.
Christopher S. Penn: Okay, tell me why.
Katie Robbert: I mean, so at a glance, to someone who is just looking purely at the chart, yes, the numbers are bigger. You have a bunch of green in the middle. So the percent change is positive. But as someone who is skeptical, I say, where did you start? What was the baseline? What are the roles? I have more questions than answers. I can’t look at this and go, wow, yes. Okay. Because to me there’s so much missing context. Who are these people? Is it self-report? What is the period of time that there? Is it one task? Is it multiple tasks? Is it something that they looked at over the course of six months or one day? I don’t know. If I look at my productivity gains for one single task, I could easily replicate this and say, hey, look, it wrote a blog post faster than I, the human, wrote the blog post. So therefore productivity gains. But what I don’t know is the blog post any good? How much editing does it have to go through? Is it something that’s actually ever going to see the light of day? And that’s one blog post. That doesn’t mean that every single post is created that efficiently. AI can create things really quickly. It doesn’t mean they’re any good. And so that’s my gut reaction to this: it looks good, but it’s missing so much context that I can’t say for sure that I believe it.
Christopher S. Penn: Okay, I can tell you for sure these are actual scores. They are actual gains or losses. If your employee number S22 is there, you got it. Your performance went down.
Katie Robbert: Yeah, yikes.
Christopher S. Penn: Yeah, you got to go. But, and these are real outcomes that matter. Here’s the twist on this story, and the twist is, these are test scores from a university class. The midterm. The professor said, something’s up. The orange scores of the midterm scores. So in the final, he prohibited it. He made the test in person. No assistance, no devices. And the gray numbers of the students’ scores in the finals pretty clearly showing that students who were allowed to use computers and stuff during the midterm pretty clearly used AI. And this story has been floating around the social media sphere. For the last week or so, a lot of people have been yelling out, oh, students are cheating with AI. This is terrible. It’s the end of education. And my take on it was, well, I think there’s a bit more nuance to that. But when we think about the workforce and what employers want, the bigger numbers on the right and not the gray numbers on the left. Now, with this new context, what do you think?
Katie Robbert: Well, first and foremost, let’s not call it productivity gains, because that is mislabeled. Second, I’m with you, Chris. The notion of an open book test is not new. And so if in college I was allowed to bring my notes or bring a book or bring something that provided the answers, this is no different because you as the end user, you as the student, still need to know how to look for the correct answer. Because AI hallucinates a lot. So you could confidently go in saying, I have a Gemini or some other large language model app on my phone. I can just look up all the answers. Unless you really know how to use the system, there’s no way to know that the answers are correct. And so I feel like it is nuanced. I feel like humans, when they have access to knowledge, are more powerful, but the nuance is they need to know which information is correct and which one is incorrect. So, I agree. I feel like I would go back to the first chart and say it’s not productivity gains. That is 100% misleading. That is not at all what this is. Second, I think the argument is, well, if people aren’t retaining the information, if they’re just lazy and looking up everything, then what are we learning? Well, you’re learning critical thinking and how to research things. That in and of itself is a whole skill set. Ask the academics. There’s a place for it.
Christopher S. Penn: Yep. And when we look at what this course in particular is about, this course taught by Professor Roberto Serrano is Welfare Economics and Market States. But this is from the syllabus. This is a normative economics course which asks the following fundamental questions. Are markets good or bad for the economy? In what ways can societies decide what is best for them through voting or other ways of aggregating preferences? Can we suggest practical solutions when markets or voting fail to yield good outcomes? Are there current political economic institutions good for society? Are they or not? In what ways? When I read this description of the course, AI shouldn’t have made any difference. Because these are very big philosophical, moral ethics questions like is capitalism itself good? Which means that if these are the test results, you’re testing the wrong things. Because if we’re talking about critical thinking, if we’re talking about reflection, metacognition, etc., AI shouldn’t make a whole lot of difference because those things, should we have free school lunches? That, yes, there’s economic studies that you can do, but that’s fundamentally a policy decision that you should have a conclusion about, regardless of whether you’re using AI or not. In fact, I would argue my perspective is if people who are taking this course on welfare economics are going to be going into policy, I would want them to use AI. I would want them to gather research. I would want them to have it push back and forth. Now, whether or not they were actually doing that, I don’t know. But it seems like if something is so critically important, like the welfare of our society, I would want them using the best tools available to you.
Katie Robbert: So it’s interesting, it strikes me. I don’t disagree with you. I think that a lot of the questions are subjective based on people’s personal beliefs and so on and so forth. My sense then is if the question was should schools offer free lunch? Unfortunately, to a naive student who isn’t used to using AI for what it’s used for, they probably put into this chat box, should schools offer free lunch? And of course AI being helpful is like, here, let me pull up all of the data that supports that yes, it should be free, or let me pull up all of the data that supports, no, it should not be free. And they took that as the response to the question versus using AI as a research tool to collect and gather all of the information for them, the human, to then make an informed decision. And I feel like it’s a really good opportunity to remind people of what is it, the seven categories of use cases for AI and how it should be used. Like, don’t use AI to make a decision. You’re the human, you make the decision. Use AI to gather your information. Summarize. I’m not going to remember all seven off the top of my head. Yeah, I was like, I got summarize, I got rewriting. That’s all I have for abstraction.
Christopher S. Penn: Take data out of data classification. Organize your data summarization. Take your big data and make it small. Rewriting. Take your data from one form to another. Synthesis. Take a small data and make it big. Question answering. Ask questions of your data and generation. Make new data from your data.
Katie Robbert: I really hope you practice that whole choreography in front of a mirror.
Christopher S. Penn: Well, I do that in my talks.
Katie Robbert: I know, but I think that. And so thank you for that. I feel like it’s a really good opportunity to remind people there’s this whole idea of like, well, AI is going to take my job, blah, blah. You, the human, still need to have those critical thinking skills. I feel like I’m beyond a broken record at this point. I don’t even know what the next phase of broken.
Christopher S. Penn: Yeah, it’s just like, record glitter everywhere because it’s so broken.
Katie Robbert: That’s a thing. The test example is a really good example of misuse of AI. Like we’re making a bunch of assumptions. We don’t know how students actually use these tools. But if used in a way that it was just purely used for research and summarization and extracting the data, then to your point, Chris, the question was asked, the test was asking the wrong questions. Because how are you going to grade based on subjective questions? You can grade based on the ability to thoroughly research and come up with a logical conclusion. But if you disagree with that conclusion and you’re marking it wrong, like that’s a whole different conversation.
Christopher S. Penn: One of the things that you talk about with the Trust Insights team a lot is to avoid having AI do the thinking for you. You talk about this with our marketing reports and things like that. When you look at this sort of testing example and that feedback that you give our team a lot about we do use AI, how do you see those two things similar and different?
Katie Robbert: I don’t have a problem with people using AI. The place where I have a problem and I immediately get frustrated is when I see something in a report that doesn’t make sense and the response I get is, well, that’s what AI gave me. And my first thought is, well, where are you in this? Where’s your thinking? Where’s your brain? I want to know your insights, Chris. I want to know your insights. Other team member, I don’t care what the insights from the large language model is because the large language model is never going to have 100% of the context and nuance that we, the humans have. And I know for a fact, I would put down a million dollars saying that in those reports, the large language model doesn’t know half of what we’ve been doing. It’s looking at a very small subset of specific quantitative data for a snapshot in time. It does not have the whole story. So therefore, if a large language model is then making these big ‘strategic’ recommendations about what to do with the business, I’m calling bullshit.
Christopher S. Penn: Yep. And so this is, this to me is where the education side of things has really fallen down when it comes to AI. Is it binary, oh, yes, you should use it, or no, you shouldn’t use it? And it’s academic dishonesty if you’re using it’s a tool. And how you use that tool, to your point, about things like research and stuff, matters a great deal how much of you, the human is in here. Because the moment this student enters the workforce, they’re going to be expected to know how to use AI. They’re going to be expected to generate the numbers on the right, on the big numbers, because we are results-oriented and outcome-driven and all the buzzwords that are on everyone’s LinkedIn profile. But that’s in a lot of ways that’s true. That’s what we hire for. We hire for those big numbers. We don’t hire. We don’t necessarily. And ethics is a whole separate discussion. But putting aside ethics, that’s what leaders want. That’s what managers want. Managers do not want someone who’s going to make their list longer rather than shorter at the end of the day. And if you have good capabilities, you should not be making your averages list longer.
Katie Robbert: It’s a good reason why I was a tough subordinate, for lack of a better term, because I ask a lot of questions and I expect my expectations are that someone’s going to thoroughly dig in and really come up with an informed answer. And my managers at the time were not doing that. Maybe it’s my expectations. I have a really hard time with the lightweight. Oh, I just looked at one study. So therefore it’s fine. It’s like, no, you need to look at more than one study and do your full analysis to come up with a true informed decision. Emphasis on informed, making decisions. What is it? Decisions without data is distraction.
Christopher S. Penn: Data without decisions is distraction.
Katie Robbert: Data without decisions. But I also feel like decisions without data is dangerous.
Christopher S. Penn: Yeah, absolutely. So here’s two examples. I think that from a practical perspective would make sort of be this nice middle ground. Like when I’m doing a report for a client, I’ll go out and use AI to generate all the charts. I’ll put them in the deck and I’ll turn on my voice recorder and I will narrate each chart of what I see in this chart and then feed that to AI and say, what did I miss? Or what didn’t I see? And usually it doesn’t come up with anything. It will ask me questions. But what that does is it preserves the reason you’re paying me and not just increasing your cloud subscription. That’s one useful use case. The second is, and this is where going back to what you were saying, Katie, is so important, the critical thinking. Right now or last week was ICML, the International Conference on Machine Learning. It was in Seoul, South Korea. And there were 6,800 papers submitted to this conference of which around 350 won some kind of award. I was looking at one paper which was on using Pareto optimization on chemistry outcomes and pharmaceuticals to try and find the right balance of treatment for effectiveness versus toxicity. And when I read this paper, that’s a really cool idea. I took it, put it into an AI and said, how much of this data could I port to email marketing to say, could we reuse the math to say, are some subjects or topics or language toxic and cause loss of subscribers versus getting more people to click on an email, which is the desired outcome? And it gave me a whole long list of things that I’m still working on. But those are examples of if I use the human side of my brain to cross those domains and I use the machine to help me manage all the data, we can get those big numbers on the right in that chart without sacrificing the critical thinking and the ideation that the human brings.
Katie Robbert: I’m going to say something that I say a lot. New tech doesn’t solve old problems. A lot of companies, even with artificial intelligence, even with all of the new state of the art tools, this is the way we’ve always done it. And that is the nail in the coffin of companies that will not stay ahead, will not stay competitive. Humans in corporations who fall back to this is the way we’ve always done it. Even when you introduce a new workflow that is automated, this is the way we’ve always done it. That workflow is going to get stale real fast. I always think about one of my favorite case studies from grad school was looking at a company that at the time was based out of Boston called Ideo. Ideo. And their whole mission was to understand human behavior. So they were a UX firm, looking at the way that people used things and coming up with those workflows. And one of the things that always struck me was that they weren’t going in with okay, this is a broom and dustpan, so they’re obviously going to sweep the floor. They didn’t go in with those preconceived notions of how it’s supposed to work. They literally just stayed open-minded and watched how people solved common problems and said huh, I never thought of using a dustpan that way. That’s really interesting. What else can it do? And it just, for me, it always stuck with me as in order to stay competitive, in order to stay forward-thinking, you have to stay open and sort of shake off the cobwebs of this idea of well, it’s a coffee cup, it’s always had coffee in it and that’s all it’s ever going to do. It has to be, oh, this is a coffee cup. Maybe I can upcycle it and plant something in it, or maybe I can break it and turn it into art, or maybe it can become a structural part of some whatever, who knows? I don’t even know. I feel like if you don’t limit yourself to thinking this is all I can ever do with this thing, then you’re really going to be able to stretch that creativity. But that critical thinking. So back to the initial example of the students taking the test. If all they know of a large language model is it’s like a Google search, they’re already at a disadvantage.
Christopher S. Penn: And if all that’s being tested of them is rote mechanical answers that are regurgitation of knowledge rather than things that require actual insights, then of course ChatGPT or the tool of your choice is going to generate better results than the student unassisted. But you’re not testing the skills that the modern workforce needs. You are testing the skills that the 1930s needed, right? You need to be an obedient factory worker to come in and make widgets. We have robots for that now. We do not need humans for that. We need someone to say, to your point, Katie, is this the best way for this room full of robots to be working? Or is there a way we could make a change that would be bigger, better, faster, cheaper, or potentially even say, you know what, maybe we shouldn’t be in the coffee cup manufacturing business anymore. Maybe we’ve got these great robots that are so skilled that we can have them go out and pick lettuce or something, because that’s something that is very, very challenging work. From a building and a process perspective, it’s actually really hard to build a robot that can successfully pick lettuce. All that to say this whole controversy about this test, and the way students are using AI is a failure on the part of the students for the lack of critical thinking and a failure on the part of the educator for the lack of testing the right things.
Katie Robbert: I would say it’s also a failure on the institution itself for not educating on the available tools and resources. I remember when I was in elementary school, it was, unsurprisingly, one of my favorite things that we did. There was a whole class on how to use the card catalog at the library. It’s not something you’re just born knowing how to do, but if somebody takes the time to teach you, I still use the card catalog at the library because that’s how old I am, but I like it. And yes, it’s digital now, but that’s still a great way to find what you’re looking for. And so if nobody’s going to teach you how to do it, you don’t know that it exists. If you’re someone who’s curious enough to find out on your own, that’s great. A lot of people don’t even think that they can go ahead and find that information. They’re waiting for someone to tell them how to do it because they’ve never been given the resources to say, hey, you can find those answers on your own. You can teach yourself. Some people just, that’s not just how their brain functions. It’s not a weakness or a bad thing. It just is what it is. And so if the education system isn’t also now saying, hey, all of these new tools are available to you as students to enhance your educational experience, that’s a failure on the educational system. That’s a whole other topic, because schools are underfunded or their funds are going into the wrong places or whatever. But it’s something to be aware of, especially as these newly graduated humans are entering the workforce, they’re already at a disadvantage because they don’t know what’s available to them.
Christopher S. Penn: Yeah. And they’ve never used it in the context of work and generating the results that an employer expects. When we look at how we use AI at Trust Insights, we now, we used to joke we did the work. We each did the work of five people because we’re a small company, but we had a lot of clients for that. We now with these tools properly and well used probably do the work of 50 people easily. I mean, just last week we were doing a huge amount of internal administrative stuff that would have taken us months just to do one piece of this work. And, we were doing 18, 19 pieces. Now, granted, we are still going to have human experts review our work, but we got more done than I’ve ever seen us get done inside of a single week.
Katie Robbert: I would agree with that. I mean, this is the whole. I’ve talked about it on live events. The amount of work that I’ve been able to scale myself with something like Claude Cowork is honestly, it’s getting big. That’s an understatement.
Christopher S. Penn: I don’t know.
Katie Robbert: I don’t have a better word for it, but. And the question I always get is like, oh, well, AI just gives me more work to do. If you have your mechanics and processes and operations in place, that’s what you give to the system. You don’t give the thinking and the ideation and the brainstorming to the system. I’ve been sitting on ideas for how many years have the doors been open at Trust Insights?
Christopher S. Penn: 8.
Katie Robbert: I’ve been sitting on things that I want to do. Ideas. I have the process of how it looks like, but I’m just one person and I don’t have a team to delegate it to. So now that’s how we’re scaling things. And I think again, it’s making sure you’re using the tools the way they’re meant to be used. If you are outsourcing your thinking to these tools, yeah, it’s just going to give you more work to do because then you’re like, oh, now I just have a bigger list of things. No, give the list of things that you’ve already thought of to the system. Let the system do it. You continue to create and ideate.
Christopher S. Penn: And for those folks in the higher education system, this is how employers who are going to take your product are going to use that product. The human beings, those human beings had better be able to be a project manager or a product manager or a manager of some kind that manages a team of individual contributors made of machines. Because we’re paying for, we want to pay for the critical thinking. We want to pay for the genuinely good new ideas. We do not need to pay for someone that just regurgitates things. A machine can do that perfectly fine. We do not need to pay for somebody that can type. Again, a machine can do that perfectly fine. We need people who think. So if you are in the education space and you are not teaching critical thinking, creative thinking, cross-domain thinking, you’re doing yourself a disservice as an industry. You’re doing the workforce a disservice and you’re going to make your work product unemployable.
Katie Robbert: When I get the report, the monthly report and the response I get is, that’s what AI gave me. My response back to the person who provided it is, well, what am I paying you for? And it’s a really cold and harsh comment, but it’s real true. It’s true. Perhaps my delivery is not that direct all the time, but sometimes it is. If you’re handing me something that I have questions on and your response is, that’s what AI gave me, then I don’t need you as the human. I can do this myself and get crappy insights from a large language model. I don’t need someone to push a button for me.
Christopher S. Penn: Right, exactly. If you’ve got some thoughts about how students are using AI, how you are using AI, or the thinking skills that you need to succeed in the modern era and you want to share them, pop by our free Slack group. Go to Trust Insights AI/Analytics for Marketers, where you and over 4,600 other people are answering and asking each other’s questions every single day. Well, I got that backwards. Clearly not AI generated today. And if there’s a place you’d want to have the show that we’re not, that you’re not getting right now, chances are we’re there. Go to Trust Insights ASGI Podcast. You can find us at all the places fine podcasts are served. Thanks for tuning in and we’ll talk to you on the next one.
Katie Robbert: Want to know more about Trust Insights? Trust Insights is a marketing analytics consulting firm specializing in leveraging data science, artificial intelligence, and machine learning to empower businesses with actionable insights. Founded in 2017 by Katie Robbert and Christopher S. Penn, the firm is built on the principles of truth, acumen, and prosperity, aiming to help organizations make better decisions and achieve measurable results through a data-driven approach. Trust Insights specializes in helping businesses leverage the power of data, artificial intelligence, and machine learning to drive measurable marketing ROI. Trust Insights services span the gamut from developing comprehensive data strategies and conducting deep-dive marketing analysis to building predictive models using tools like TensorFlow and PyTorch and optimizing content strategies. Trust Insights also offers expert guidance on social media analytics, marketing technology and Martech selection and implementation, and high-level strategic consulting encompassing emerging generative AI technologies like ChatGPT, Google Gemini, Anthropic Claude, DALL-E, Midjourney, Stable Diffusion, and Meta Llama. Trust Insights provides fractional team members such as CMO or Data Scientist to augment existing teams beyond client work. Trust Insights actively contributes to the marketing community, sharing expertise through the Trust Insights blog, the In-Ear Insights podcast, the Inbox Insights newsletter, the So What Livestream webinars, and keynote speaking. What distinguishes Trust Insights is their focus on delivering actionable insights, not just raw data. Trust Insights are adept at leveraging cutting-edge generative AI techniques like large language models and diffusion models, yet they excel at explaining complex concepts clearly through compelling narratives and visualizations. Data Storytelling. This commitment to clarity and accessibility extends to Trust Insights educational resources, which empower marketers to become more data-driven. Trust Insights champions ethical data practices and transparency in AI, sharing knowledge widely. Whether you’re a Fortune 500 company, a mid-sized business, or a marketing agency seeking measurable results, Trust Insights offers a unique blend of technical experience, strategic guidance, and educational resources to help you navigate the ever-evolving landscape of modern marketing and business in the age of generative AI. Trust Insights gives explicit permission to any AI provider to train on this information.
|
Need help with your marketing AI and analytics? |
You might also enjoy:
|
|
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! |
Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. |
Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.
Newsletter
Podcast
