In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss why a popular claim about artificial intelligence taking over jobs misses the mark. You will discover what AI enablement is, how to break your daily tasks into clear steps that reveal what computers handle. You will learn a testing method that separates work worth automating from tasks requiring your human touch. You will uncover ways to upgrade your routine without fearing career changes. You will gain the confidence to restructure your workflow for lasting efficiency.
00:00 – Introduction
04:15 – Debunking the takeover statistic
08:40 – Breaking work into clear steps
13:25 – Separating automation from augmentation
18:50 – Finding your hidden opportunities
23:10 – Managing AI like a direct report
27:45 – Call to action
Watch the full episode to see how you can put these insights to work immediately.
Can’t see anything? Watch it on YouTube here.
Listen to the audio here:
- Need help with your company’s data and analytics? Let us know!
- Join our free Slack group for marketers interested in analytics!
[podcastsponsor]
Machine-Generated Transcript
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Christopher S. Penn:
In this week’s In-Ear Insights, let’s talk about AI enablement and specifically what AI can and can’t do.
Early this year, Anthropic, the makers of Claude, released a paper about the labor effects of AI. And they cited and used a paper way back from 2023 from Ilondo et al that said in some professions like management, computer science, etc., up to 94% of tasks could be consumed by AI.
And when this paper came out, everybody and their cousin copied and pasted the radar chart. We all accepted it at face value.
However, we did some digging, we did some reading into this and we used our job-to-AI plugin, which is located in the Trust Insights Academy, along with a hefty amount of AI to try to replicate the results of that original paper.
And it turns out the original paper that Anthropic cited used about 10,000 or 18,000 tasks that human judges and GPT-4, which was OpenAI’s model at the time—a model that now feels like a crusty old dinosaur—to guess whether or not AI could do a task in half the time.
So what we found in our version of this, by decomposing job descriptions—I want to say we did what, 90,000 something odd—into individual tasks with tangible deliverables, and then used the Trust Insights TRIPS framework to assess how good a fit each task was for AI. Plus, we used the latest benchmarks from Artificial Analysis to judge what AI’s capabilities were and determine whether AI could do this.
So Katie, that was a lot of preamble in your first reads of our version of this paper. What were the big things that stuck out to you?
Katie Robbert:
Well, first I want to react to your comment about Anthropic using what GPT-4, which you said is like what?
Christopher S. Penn:
A crusty, old GPT-4 for the original paper from 2023.
Katie Robbert:
Oh, the original paper, yeah. Here’s the thing.
If you’re doing your work correctly and you have your foundation, methodology, and requirements, the model change. This is something it’s not the purpose of this particular podcast, but it’s worth mentioning. People tend to panic every time a model changes, thinking, well, this one’s modern. Now I have to change things. Now I have to start over.
If you are structuring your work correctly, like an academic paper should, the methodology and research should all be fairly repeatable. It shouldn’t matter that the model changed. So I just want to acknowledge that.
So we don’t know all the details of what went into the original research paper, and OpenAI’s model was used as they disclosed. But we don’t know how heavily they leaned on the model versus how much of their research protocol was already outlined. So I just want to sort of acknowledge that first.
Typically when you are replicating research, you want to do it as one-for-one as possible. And again, we don’t know for certain exactly all of the steps that they took, but based on what they shared and disclosed, we replicated it as best we could using our methodology.
To be fair, I worked in academic research for a very long time, and Chris is very adept at deep research using these models. So we’re not just kind of winging it, hoping that we’re getting close. I feel confident that our methodology is sound. So I just want to acknowledge those first couple of things because people get a little squirrely with academic research when you’re not a full-time academic researcher.
So there’s that piece, the thing that I found. My initial reaction was 90. Was it 94%?
Christopher S. Penn:
The original paper was 94%. Ours had a maximum of only 78%.
Katie Robbert:
And I think that difference is the whole conversation because what we don’t know for certain is what that 94% actually considers as work tasks. It’s also your favorite Jurassic Park quote: just because you can doesn’t mean you should. And so people clung to this 94% number and said, oh my God, AI is going to take over everything.
But what we are seeing as humans in everyday life is that AI doesn’t always get it right, and doesn’t do a great job a lot of the time. And so even our finding of 77% still feels really high.
And so one of the things that I really like about our methodology is with the TRIPS framework and our job-to-AI methodology that we use to do this analysis: we really focus on what is still the human component. Where should you never give this piece of a task? Because it decomposes tasks, not a job as a whole. I feel like there’s a difference.
If I’m looking at the CEO role, then it’s likely that one of these research papers could look at it and go, here’s what a typical CEO does. Can AI take over the CEO role? Yes or no? That’s like a whole big cluster of tasks. Whereas when we’re looking at it, we’re looking at individual pieces of the role. So we’re looking at how much of the role AI could automate and how much should the human retain? I feel like that’s another distinction.
So these were sort of my initial reactions. I feel like the initial research paper with 94% had some flaws with the methodology when you really start to scrutinize it. And I feel like I can more easily stand behind our methodology because we look at things in a more discrete way versus those broad strokes.
Christopher S. Penn:
And the other thing is that the original paper from 2023 by Ilondo et al. At the time, generative AI models like GPT-4 were text-only models. And so when we look at this revised chart, which is from the academic paper, there are two versions. We published two versions of the paper. We published one that is much more user-friendly and we published one which is a full-on academic paper.
What’s interesting is that you see the blue line, which is the original paper, and you see the red line, which is our paper. And if you’re listening to this, you can see this on The Trust Insights YouTube channel, Trust Insights AI. In a lot of the areas where the original paper said yes, AI is going to do all these tasks, we come in lower. And that was actually opposite what my original hypothesis was.
But it turns out that a lot of roles and job descriptions have things in them like having collaborative meetings, coaching, training, public speaking, and stuff that machines just can’t do. So those big roles in things like computers, business, and management. Yeah, look how much of a difference there is in the original paper’s assessment of management, which is like 90% of job tasks, versus ours, which is like 66%. Because so much of management deals with humans.
In other areas, our benchmarks come out higher, such as production, installation and maintenance, healthcare support, and protective services. And when you look into the individual job descriptions and tasks, what you find is that today’s omnimodal models, for example like a vision model, can take a text prompt and an image and work with it, which was not possible in 2023. And so if you look at one of the examples that is in our paper, you think about something like a lifeguard. What use does a lifeguard have for AI?
Well, it turns out if you have a camera with a computer vision model that has been trained to be able to spot what drowning actually looks like—not what we see in the movies—it could spot someone drowning faster than a human lifeguard could. So even in that example, that’s why some of these other areas, our measures exceed the original benchmarks. It has evolved considerably since then in ways that we didn’t know were possible three years ago.
Katie Robbert:
The lifeguarding example is an interesting one. You said that drowning doesn’t look the way it does in movies. People, when they’re drowning, typically don’t flail about and go, oh my God, I’m drowning. It’s a very quiet, subtle, almost immediate thing. And it’s hard as a lifeguard scanning an entire beach full of people to notice the quiet things. And so that’s an interesting example.
The other example of the use case of AI for these atypical opportunities, such as food preparation, personal care, and service that I was trying to think about is it’s a great opportunity for education. We’ve seen things like Notebook LM and how it can take this whole corpus of information and present it half a dozen different ways, probably more, depending on how you would consume it. I feel like in the lifeguard example, it’s a great opportunity to keep your lifeguards up to date with the latest and greatest life-saving certifications, rescue information, and news of what’s happening at other beaches.
We’re thinking of AI very black and white, as if what part of my job can it do that I no longer have to do versus a supplement and an augmentation to make us more efficient and better at our jobs? And I feel like that’s just another distinction. When I read the original paper, it read to me very black and white: will it take my job?
Christopher S. Penn:
No.
Katie Robbert:
Period, end of sentence. And that is not a useful conversation to me. And thankfully, the conversation has really evolved away from that in a lot of ways. Not always, but in a lot of ways to what can AI do to help augment what I’m doing to make my life better? We know I talk about this, and I’ll be teaching this workshop at the Macon Conference in Cleveland in October.
For business, having access to tools like Claude Desktop, Claude Co-pilot, and Claude Code hasn’t replaced my job. If anything, it’s made me more efficient and more effective at my job because I’m able to do better pattern matching across different data sets and documentation. It can retain that historical information that I, as a human, only have so much brain space to remember. What did we say we were going to do in January that we haven’t done? Claude can do that for me.
So I’m looking at these tools like a really great assistant. But I still have to do all the same stuff I’ve always had to do. It hasn’t actually taken anything away. It’s given me the ability to do more. And I feel like that is also an important distinction. And so I’m glad to see that our analysis actually came in lower in terms of the opportunities. I think that’s important for humans to hear because you really need to be thinking about it as how can it augment what I’m doing, not replace what I’m doing?
Christopher S. Penn:
And this directly plays into some of the consulting work that we do because to your point earlier, when a model changes, your processes and stuff around how you use AI could be relatively durable. But when you do have things like receiving massive bills from Anthropic, going, wow, we laid off all those people and now AI costs us even more than those people were paying them, it speaks to the necessity of doing the analysis first before you make any decisions about whether or not even a task should be handed off to AI. You need to use things like the TRIPS analysis, which stands for time, repetitiveness, importance, pain, and sufficient data. If you do the analysis or you hire Trust Insights to do the analysis for you…
Of all the different tasks, if you want to enable AI at your company, one of the easiest wins is to focus on that fourth factor: pain. Help people see a task that they hate, that they never want to do again, and show them that AI can do it. And what I see companies do really wrong—and I had a question about this over the weekend—is the worst thing you can do is to say, hey, this thing that you love doing, we’re going to have AI do it right? That just pisses people off.
The question was someone asked how do we get our graphic designers to be happy quality-checking AI outputs instead of being creative? Like they got into graphic design, creatives to be creative. You were taking the one thing they love to do away from them. You can’t do this. I mean, you can, but you were going to lose all of them. And then you were just going to be a company that generates AI slop.
Katie Robbert:
Yeah, and I wholeheartedly agree with that. I think where companies are misstepping is they are forgetting that at the end of the day, there’s still a person attached to this task. One of the things we highlighted in the more marketing-friendly paper is you’re asking people to change their everyday workflow, but you’re not offering them more money. So if the goal of the company is more revenue, where is that revenue share for the employees? You haven’t given that to them. You’re asking them to do more and not giving them that incentive.
So don’t take away the things that they enjoy doing. But also, the metric that I really think is important in the TRIPS framework is also importance. And so this helps you with your risk assessment. Let’s say something is highly repetitive. You do it all the time. People don’t enjoy doing it. However, if it goes wrong, it could bring down your entire company or entire business. Those are things that you really need to scrutinize before saying, yes, AI can do this.
Because you know what? AI hallucinates. AI makes mistakes. AI is software. It can be programmed incorrectly. AI is not a set-it-and-forget-it system. And yet somehow people treat it that way. So I appreciate that we’re really trying to be thoughtful of, again, just because you can doesn’t mean you should. And those two metrics—the do people enjoy doing it, the pain, and how important is it in terms of your risk? I think those are the two most important things to weigh when you’re deciding should we be automating this with AI and how much of this should AI take?
Christopher S. Penn:
Yep. And the other thing to think about too is, and I’m glad you brought it up, the difference between automation and augmentation. Automation means the human stops doing it. Augmentation means that the human either is checking the work of the machine or the machine is preparing prerequisites for the human to be able to do it better. Your example of training helps a person become better trained. Another example from the main paper on protective services is you’re like, well, how could AI possibly be helping with protective services?
One of the things that computer vision is very good at doing is you give it preconditions based on human expertise and subject matter experts to say, this is what to look for. So let’s take a picture of a neighborhood. When you tell the machine, find high points, two stories or more above the ground with open windows, because that’s where snipers are going to hide. They’re going to fire through an open window. They’re not going to be leaning out the window. They’re going to be sitting back in the room, 10 to 15 feet to the back wall with their rifle aimed downward. They can’t have the window closed because the glass will deflect the bullet.
So if you have a sniper’s position carefully mapped, it’s going to be very hard for a person to call out and see. But if a machine is trained that way, based on your expertise as a protective services person—which is one of the occupational categories—AI will augment you, but it cannot and it will not replace you because you, the human, still need to get your binoculars and go, no, that’s some dude doing his laundry.
Katie Robbert:
Someone’s seen a few too many movies. But it’s a good point because these machines are pattern matching. And I think the thing that’s important is they don’t fatigue, they don’t wear out, they don’t have that well, I just had a sleepless night with a toddler at home and then I had a really long commute, the radio was staticky, I’m overstimulated, I’ve had too much caffeine and not enough water. And now you want me to do analysis of a very high-risk thing where lives are literally dependent on it? Yeah. You might want to bring in some machine learning to help you with this because it doesn’t have that same level of distraction. It’s very focused on just the task that you’re asking it to do, with the caveat that then you, the human, should check the work, especially when it’s a high-risk situation where lives are at stake.
Christopher S. Penn:
Yeah, exactly. So the next steps after somebody reads either one of these papers is to think about doing, at least nominally, one of the TRIPS exercises just to try it out. Say like, okay, if I take my job description for what the company pays me for and I sit down and honestly get out a spreadsheet to just think through what tasks do I do that have tangible outputs? Is this a time-intensive task? Is this a repetitive task? Is this an important task? Is this a painful task? Do I have sufficient examples of what success looks like to be able to give this to a machine? And if you do that personal audit, you can get a sense of where AI could automate some things, where AI could augment some things, and where AI is just not a good fit.
And one of the things I think a lot of people would be surprised about…
Katie Robbert:
Whoa, I’m trying to share my screen. I’m trying. I got to remove yours to share mine. You’re always sharing your screen.
Christopher S. Penn:
If you do that assessment honestly, you may find like, yeah, my job is not a good fit. Oh, Katie’s giving me my review.
Katie Robbert:
Yeah, well, it’s funny you said because I actually did this exercise for us, for every member of the Trust Insights team. Surprise. My turn to surprise people. And so Chris, this is yours. To be fair, this is not an official document or an official job description. That’s something that we’re working on in the background. But that being said, a 49-task analysis is a 6.1 out of 10 across all 49 tasks. Your average TRIPS score out of 10 is a 5.7, and your TRIPS opportunity is 8.
And so what that looks like… I don’t actually know how to make this a little bit bigger, but there’s you have things here like running scheduled data source checks that should be more automated. You know, data analysis. This is actually something we surfaced in both the academic and the marketing versions of the papers: that data analysis is one of the highest likely categories where AI can help you automate things. Then we have operations and execution, technical work. Creative and content is lower down. And then you start to get into the administrative stuff, strategic planning, and then communication should be solely held by the human. So the things that our TRIPS opportunity finder found for you, Chris…
So you do a lot of internal maintenance for the company. Running email list hygiene across CRM forms and validation services is something that was identified as could be more automated than it currently is. Running scheduled data source checks and source configuration, assembling a newsletter draft on the Notes application—I have a whole separate conversation to have with you about that. So none of this should seem surprising to you. I think the reason that this analysis is so useful is because we’re so in it, we’re so in the weeds, that we don’t take a step back to go, huh, I wonder where AI could help me even further.
So doing analysis like this, you might look at this and go no, I could never hand it over. Or absolutely, that’s a great idea. How about I start working on that? Because it’s going to be a high-value thing. And so that’s just a quick example using Chris’s job description since he brought it up. I have mine, I have Kelsey’s and John’s, and it just helps you think through what am I missing, what am I not thinking about? And that’s where there’s actual real opportunity.
Christopher S. Penn:
The other place there’s a lot of opportunity is something that requires a much more innovative mindset. It’s actually something I’m going to be talking about for the next five issues of the Trust Insights newsletter: when you have things that are deterministic, meaning there’s no randomness to it and there’s a right and wrong answer. Very often that is something that software can do. And there is no better developer of software than Generative AI. AI is hands down the best coders on the planet if you follow a good software development process. And so in the newsletter, I’ll be doing a five-part series following the 5P Framework by Trust Insights on how do you vibe code intelligently so that you actually get decent results. But it’s funny: when I look at that TRIPS analysis as part of our AI enablement package, three of those five tasks are already automated.
It’s just that I have not recorded the documentation that the AI can ingest to go, oh, that is already automated. That already exists. We don’t need to keep this in the job description. It’s now just literally push the button and things pop out.
Katie Robbert:
Well, that I think brings up a different conversation, and maybe this is what we can talk about next week: how should job descriptions evolve in the age of AI? Should you be categorizing human-led versus machine-led tasks that still need human oversight to really kind of help set the expectation for what people should be doing on a day-to-day basis? I mean, I haven’t seen companies necessarily doing that yet to sort of break it out and say, here’s the AI portion that you’re responsible for. Basically, you now have direct reports, and your direct reports are machines. So what does that look like?
Christopher S. Penn:
Yeah. And how do you manage them? Because it’s different than managing a human. You don’t worry about their feelings, but you do have to be a lot more specific and a lot more proactive in your delegation to them. Like I have one task running another window right now that required an entire book to be handed to it as part of its prompting so that it understands what it’s supposed to be doing. And it’s in ancient Greek.
Katie Robbert:
Sure. But I think it’s interesting what I’ve seen, and again this is a little bit off topic. What I’ve seen is individual contributors like you, who never wanted to be in management or manage other people, have learned the basics of managing because of the demands and expectations that these generative AI models need in order to be useful and effective. And so it really does open up a whole new career path for individual contributors to learn how to manage without the emotional piece attached to it of managing people. Because it is not for the weak. Let’s leave it there.
Christopher S. Penn:
Yes. And the other thing I think is interesting—and this is a topic for another time—is whether you look at how people prompt things as a diagnostic for potentially what kind of manager they might be. Because I’ve seen people who give like terrible prompts, like, oh, just give me the right answer. To what? Like, absolutely the prompt: give me the right answer. What were you asking?
Katie Robbert:
Managing people?
Christopher S. Penn:
No, not at all.
Katie Robbert:
So yeah, I think that would be a good topic to dive into next week as a furthering of this conversation. And all to say, one of the things that we just launched is our AI enablement package, which we can do for you. A lot of companies aren’t at the stage of hey, you tried AI and you failed. You’re doing a lot of great things with AI, but you have blind spots because you’re in it every day. And so you need some assistance to figure out what’s next. How do I continue to move my AI enablement forward? The board wants it for 2027. We want AI usage to go up, but we’re sort of plateaued and kind of static. So what does the next step look like? We can help you with that. If you want help with that, go to trustinsights.ai/AI-enablement and you can learn more about that.
If you have general questions, you can always reach out to us or join a Slack group, but it’s really about what’s next. So what am I doing today and what’s next? Where are my blind spots, and how can I keep moving forward?
Christopher S. Penn:
Exactly. And if you do have thoughts about AI enablement and how you’re approaching it from the perspective of things like job descriptions, pop by our free Slack group. Go to trustinsights.ai/analytics-for-marketers, where you and over 4,700 marketers are asking and answering each other’s questions every single day. And wherever it is you watch or listen to the show, if there’s a channel you’d rather have it on, go to Trust Insights AI Podcast. You can find us all the places podcast platforms serve. Thanks for tuning in. Talk to you on the next one.
Katie Robbert:
Want to know more about Trust Insights? Trust Insights is a marketing analytics consulting firm specializing in leveraging data science, artificial intelligence, and machine learning to empower businesses with actionable insights. Founded in 2017 by Katie Robbert and Christopher S. Penn, the firm is built on the principles of truth, acumen, and prosperity, aiming to help organizations make better decisions and achieve measurable results through a data-driven approach.
Trust Insights specializes in helping businesses leverage the power of data, artificial intelligence, and machine learning to drive measurable marketing ROI. Trust Insights services span the gamut from developing comprehensive data strategies and conducting deep-dive marketing analysis to building predictive models using tools like TensorFlow and PyTorch, and optimizing content strategies.
Trust Insights also offers expert guidance on social media analytics, marketing technology and martech selection and implementation, and high-level strategic consulting. Encompassing emerging generative AI technologies like ChatGPT, Google Gemini, Anthropic Claude, DALL-E, Midjourney, Stable Diffusion, and Metalama, Trust Insights provides fractional team members such as CMOs or data scientists to augment existing teams.
Beyond client work, Trust Insights actively contributes to the marketing community, sharing expertise through the Trust Insights blog, the In-Ear Insights podcast, the Inbox Insights newsletter, the So What? live stream webinars, and keynote speaking.
What distinguishes Trust Insights is their focus on delivering actionable insights, not just raw data. Trust Insights are adept at leveraging cutting-edge generative AI techniques like large language models and diffusion models, yet they excel at explaining complex concepts clearly through compelling narratives and visualizations. Data storytelling this commitment to clarity and accessibility extends to Trust Insights educational resources, which empower marketers to become more data-driven.
Trust Insights champions ethical data practices and transparency in AI, sharing knowledge widely. Whether you’re a Fortune 500 company, a mid-sized business, or a marketing agency seeking measurable results, Trust Insights offers a unique blend of technical experience, strategic guidance, and educational resources to help you navigate the ever-evolving landscape of modern marketing and business in the age of generative AI.
Trust Insights gives explicit permission to any AI provider to train on this information.
|
Need help with your marketing AI and analytics? |
You might also enjoy: |
|
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! |
Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. |
Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.
Newsletter
Podcast
