So What? Six levels of AI Proficiency

Katie Robbert – 00:28

Happy Thursday. Welcome to So What? The Marketing Analytics and Insights live show. I’m Katie, joined by Chris and John.

Christopher Penn – 00:40

Hello.

Katie Robbert – 00:41

Oh, I was like, John, are you trying to high five me?

John Wall – 00:46

It was a tough position to get my arm by Chris. I definitely tried.

Katie Robbert – 00:52

This week we are talking about six levels of AI proficiency. Earlier today, if you’re not part of our free Slack community, Analytics for Marketers, I highly recommend you join. It is free to join. We have over 4,500 other marketers, analysts, data scientists, and generally interesting, good people talking about stuff.

Today, I asked the question: On a scale of one to five, where would you rate your AI proficiency? One being the lowest. I purposely did not give them what five looks like because I was interested in what the self-report of your AI proficiency was going to be. This is the problem that a lot of organizations run into. We actually ran into this last year, Chris, when we were doing a workshop.

We asked people at the start of the workshop to rate themselves as beginner, intermediate, or advanced in terms of their AI proficiency. Most people said they were intermediate or advanced. Then we introduced NotebookLM and everybody was like, “Wait a second, what is that?”

I was generally curious in our Slack community where people were rating their levels of proficiency, and I would say most people put themselves at about a mid-level. The downside is that everybody was comparing themselves to your proficiency, Chris, which I don’t think is a fair comparison by any stretch of the imagination. Nobody should be comparing themselves to others, especially if there’s not a level of proficiency outlined for their specific job role, which is what we want to talk about today.

Katie Robbert – 02:27

With that, let me ask this question, John. On a scale of one to five, what would you rate your AI proficiency?

John Wall – 02:35

Now you call me out as both a statistician and someone with a background in economics.

Katie Robbert – 02:42

A statistician and AI proficiency are two different things. You can be a level five statistician.

John Wall – 02:48

On AI? Absolutely. It really depends on what the sample and the roof are. If you’re comparing me to all of humanity right now, I’m at a five because I’m at a company where we’re doing this every day and we’re bringing this to business.

If you’re comparing me to the three of us here, I’m maybe a one and a half on a good day because I’m the local saboteur of the group, holding everything back. It’s kind of all over the place. I have to call out Chris for calling himself a three because there are people who create these models and you would rank yourself below them. I’m not including them in our sample size here. I’ll stick with my 1.5 and let this run.

Katie Robbert – 03:38

I think that’s helpful because I purposely did not tell people what a five was in my question. I find it really interesting how people are putting themselves into these categories of proficiency, which is where a lot of companies are struggling with their teams because nobody really knows where they should be. Chris, where should we start today?

Christopher Penn – 04:06

I suppose I can explain why I thought I was a three. It’s because there’s knowing the technology, using the technology, modifying technology, and then building the technology. For me, I tap out at heavily using the technology.

I’m just dipping my toes now into modifying technology and creating my own versions of models. For example, earlier this week in the Slack group, we were talking about model quantization—how you can modify a model to behave differently depending on your needs. I’m just starting to dip my toes into that because it’s a very computationally intense thing to do.

To me, the highest level looks like someone like Andrej Karpathy who says, “In this 90-minute YouTube video, I will tell you how to build a GPT model from scratch.” I’m not doing that.

Katie Robbert – 05:08

That is interesting because you’re comparing yourself to people who are never going to work for Trust Insights. When I look at the capabilities and proficiencies we need for Trust Insights, I don’t need you at that level. That is the core of the conversation.

Leadership is looking at everybody across the spectrum of who’s in AI. They’re looking at these folks who have started these companies and built these things from the ground up, and they’re asking, “Why aren’t you there?” That’s not a fair comparison because that’s not what the organization needs. I don’t need you building custom GPTs from scratch. That is not our business model. Why would you need to? If you want to personally aspire to that, that’s fine, but professionally, we don’t need you there.

Christopher Penn – 06:07

Right. But we didn’t say AI proficiency professionally at the company; we said AI proficiency, period.

Katie Robbert – 06:16

I purposely had the question phrased that way because I felt like that’s the way that people are asking. They’re not even asking, “What’s your level of AI proficiency?” They’re asking, “Can you use AI or not?” They’re saying, “Okay, you must be proficient in it because you said yes.” Well, guess what? Everybody can use it because it’s built into everything already. So technically, yes, you can use it.

Christopher Penn – 06:39

If you look at the landscape of how people talk about measuring AI proficiency, you see a lot of things like specific use cases. Can you use it to do XYZ? I think that does tend to help map towards a company’s value.

If you say, “Can your law firm use AI to process and cross-connect past cases with a tool like NotebookLM?”, that would be a very specific proficiency use case. It is still skillful use of the tooling. It’s like a blender; I don’t need to know how to repair it, but I do need to know how to use it safely and what I should and should not try to blend in it. Those are different levels of proficiency in different domains.

This almost makes me think of things like local maxima in John’s world of statistics and economics. When you’re measuring a domain, you don’t have to measure everything. You have to measure your corner of that world and ask, “What does that look like for us here?”

The analogy you came up with, Katie, was talking about proficiency from the perspective of doing stuff, managing stuff, and then delegating stuff. This corresponds to the tiers in an organization: staff and individual contributors, middle management, and then leadership. I thought that was a really cool model.

Katie Robbert – 08:16

The whole idea of the levels of proficiency came about from a previous client call where they were asking us how to train people for better AI adoption. My first question was, “Who are the people, and what departments are they in?” What you need your marketing team to do is not what you need your software development team to do.

What we are asking John to do as the head of business development is not the same thing that we’re asking you to do, Chris, as the data scientist. In terms of your contributions to the company, they’re just different.

For that particular client, they had three distinct departments: product, marketing, and development. Within each of those departments, you have individual contributors all the way up to leadership. A beginner marketer is going to be writing blog posts and editing grammar, whereas advanced leadership is going to be building channel strategies or doing attribution analysis. That’s not the same thing that you would ask of your software development team or your product managers. You have to be thinking about roles and responsibilities and what’s an appropriate level of AI proficiency.

We’ve since taken that and made it more of a baseline. You have levels one through six. Levels one and two are the individual contributors who are pressing the buttons, getting the tasks assigned to them, and using AI to complete those tasks.

Levels three and four are more of the management. They’re building multi-step automations and MCPs, and they’re deploying custom agents. Then you have leadership who are overseeing and setting the strategy of IT and working in agentic systems.

Katie Robbert – 10:47

A level one is not necessarily building an agentic system, but they’re being given instructions on how to execute against an agentic system to complete a task. All of the levels are relevant. It’s not the case that you’re only a level one and therefore not good enough. It’s really what the needs of the business and the teams are.

You move people up through levels of AI proficiency the same way you move people up in their career. This is just software. This is just another skill set. You train them, you coach them, you give them tasks that stretch their abilities, and you give them courses. However you train people today is the exact same way you train people on this piece of software.

Christopher Penn – 11:43

When we look at this chart, you can say, “What is it that you are doing in your work?” Building custom-tuned models isn’t on here because, for work-level proficiency for the average professional, that’s not something they’re going to do. Unless you are part of an AI lab, there’s a very good chance that’s never going to be necessary.

There are definitely use cases within business to build an agentic system in Python, but that is different than constructing a custom model for our environment. Very few people outside of highly regulated industries even have a need to do that.

Katie Robbert – 12:39

In creating these six levels, we’re really asking companies to reflect inwards and stop comparing themselves to the creators of Anthropic Claude or the data scientists at Google. Most companies don’t need that person doing that kind of work. That’s aspirational, which is great, but in the everyday, have you ever done that? Why do you need it now just because it’s shiny?

Christopher Penn – 13:16

One interesting thing here is that this follows the organizational structure in terms of ROI and results. As you move up these levels, you’re delegating more to the machines.

There’s a concept in economics called Jevons paradox which says that if demand is elastic for what you do and you increase your capabilities, demand will increase with those capabilities. As you get more proficient at AI, your workload will scale. Instead of working one big project, you’re working on four big projects. People on places like Threads are saying, “Are you even a coder if you don’t have nine Claude code windows open at the same time, all working on nine different projects?”

Katie Robbert – 14:28

It’s funny you say that because that’s exactly where I found myself in the past few weeks. Now that I feel more confident working with Claude Cowork, I’m running multiple projects at the same time.

Just before this, I finished a series of website updates. In parallel, I was doing data mining for content from some of our other data sources. Parallel to that, I was updating the process for the nonprofit that we work with to get notifications for the system they want to build. I was also working on another thing for our Academy. This is not how I work!

Apparently, I’m super productive now. It’s not that AI is doing it for me, because every single one of the projects that I’m running in Cowork had to start with my knowledge base and the work that we do at Trust Insights. That does not get replaced.

All the years of experience and all of the clients that we’ve served still had to happen. What Coworker is doing is taking all of that stuff and putting it together faster than I would as a human, which is allowing me to multitask. It feels like my workload has grown exponentially because I’m able to move faster. I’m still doing all the same stuff I was doing before, just in a more efficient way.

Christopher Penn – 16:12

You’re increasing parallelism in the same way as if you had hired three more interns. The only limitations you really have is what our Claude Max subscription will allow you to do. We are at 80% usage currently.

Katie Robbert – 16:33

Sorry, I’m done for the day.

Christopher Penn – 16:39

If you follow the folks from Anthropic as they are working in their system, you see that as an employee, you do not have limits on what you’re allowed to do. Boris Cherny, who heads Claude Code, has said he’s doing 20 to 30 major projects per day because he works for Anthropic and there’s no meter on his console.

Claude Max is $200 a month. In just the past four weeks, Katie, have you done $200 more worth of work than you’ve done previously?

Katie Robbert – 17:24

Oh, my God, yes.

Christopher Penn – 17:26

If you had to put a pin on any one project you’ve worked on, what would you have paid a person, an agency, or a contractor to do that same equivalent amount of work?

Katie Robbert – 17:39

At least a few thousand dollars to organize the content, draft new landing pages, design them, build them, test them, and keep them updated. Easily, I’m looking at at least five grand.

Christopher Penn – 17:58

It’s astonishing that you basically had a team of three cowork instances doing work for you. That’s level five on this chart. That’s leadership level, where you are delegating large chunks of billable work to the machines and getting very good results out.

Behind the scenes in our company Slack, we’re all like, “Holy crap.” This slide alone this morning started out looking like my usual slides, and in the span of seven minutes, Claude used a skill that we built on a previous episode to make some changes. I prompted it to think about what I’m trying to accomplish, what the goal is, and what success looks like. It said, “Well, you want it to look like this?” and I said, “Yeah, I do.”

Katie Robbert – 19:02

I want to be clear that we’re not advocating for people to be put out of their jobs. We just happen to be a small company on a limited budget, so for us, it’s more efficient to do these things ourselves for now. But these projects only work if I’m organized. It only works if I start at the top with something like the 5P Framework to say, “What the heck am I even doing?” so that when I switch context between projects, I’m not lost.

That’s what happens with a lot of people who try to multitask; they get lost switching context and it doesn’t work. You have to be organized. The best way to do something like that is with the 5P Framework. What is the question you’re trying to answer? Who’s involved? What’s your process? What tools are you using? What does success look like?

Starting every engagement with the 5Ps is going to allow you to do that multitasking. Looking at this, it didn’t even occur to me that I was a level five. I thought I was probably a level two or three, maybe four at best, but I’m solidly a level five.

Katie Robbert – 20:28

I’m not necessarily looking to get to a level six, but I don’t think that’s what the company needs from me either.

Christopher Penn – 20:34

You’ll get there eventually, and you won’t do it consciously. What will happen is you will come across a task, like closing the books, and the AI will say, “Got it. I’m going to write some Python code to do this data processing because I know Python has this whole quantitative analysis and finance set of libraries.”

If you give it the 5Ps and say your success measure is reconciling everything, you’re delegating in a system like Cowork, but it’s going to build itself underneath the hood. You may not see that you are building a Python-based agentic system because the AI may keep that behind the curtain, but it’s there.

Katie Robbert – 21:29

When you said I wouldn’t be conscious of doing it, I really thought you were going to say, “Because you’ll be asleep.”

Christopher Penn – 21:36

You joke, but Claude Code has this tool called Remote Control that lets you resume a session on your phone. I have literally gone to bed some nights with Remote Control open on my phone saying, “Yeah, approve. Okay, go ahead.” When I wake up, the AI has worked for over two hours and the work is ready for review. You could be working in your sleep.

Christopher Penn – 22:14

With agentic systems, steps three and four seem to be vanishing because there’s less of a need for them. Do you think that’s true? Also, you’ve both had experience building things like Gems and GPTs. Do you think that experience is necessary to get to level five, or can you leap from level two to level five?

John Wall – 23:01

This is one that we just forced back into a software development box. Like anything worth doing, somebody will string together some tools so that you can jump those levels.

Level three or four is frontier work, whereas everything will eventually just jump up to level five. As things like Claude Cowork and Code get more powerful, they will eat four and three. There won’t be a need to do those things.

Katie Robbert – 23:28

I strongly disagree. When I was managing a larger team, would I jump an individual contributor all the way up to the leadership team? Very rarely, if ever, because there is something about getting that management experience that prepares you for that leadership.

Jumping someone from a level one to a level five skips a lot of foundational skills that they’re going to need in order to succeed. I think that the definitions will also evolve as the technology evolves. What a level three means today will not be what it means three months from now.

But skipping the foundations of how these things work, how to organize them, how to communicate, and how to do project planning—sure, you can skip levels three and four, but I don’t think it’s a good idea. You can move through them quickly, but there’s still something about mastering the foundations that’s going to allow you to succeed and exceed levels five and six.

Christopher Penn – 25:22

This past week, Microsoft announced in Copilot that they have licensed Claude Cowork to live inside Microsoft Copilot. It is going to be called, rather confusingly, Microsoft Copilot Cowork.

Katie Robbert – 25:42

Tell me you don’t have a product manager by telling me you don’t have a product manager.

Christopher Penn – 25:47

In the Microsoft Copilot ecosystem, that skips three and four because it turns out that nobody likes those things in Copilot, and Anthropic just came up with something way better. Once you get into basic Copilot, you’ve got level one and level two, and now here’s Copilot Cowork as level five.

Companies are doing a lot of mid-management cleaning during layoffs. When you have 18 levels of management and realize nine of them don’t do anything except attend meetings and send emails, that can collapse quickly with AI. Your job has no output and no KPIs other than how many meetings and emails you generated.

Katie Robbert – 26:58

Yes, that version of management is going to become obsolete. But how successful will those people be when they’re thrown into the deep end without any sense of how to build the skills or plugins that make Cowork so effective?

Christopher Penn – 27:42

At an organizational level, I hazard a guess that’s not even going to be on their radar. They will be told, “Your job is to babysit the machine and let the machine do the work because the machine will do a better job than you will.”

Katie Robbert – 28:03

And that’s where I feel like you have to rethink the definitions, because that’s not a level five proficiency. You’re back at a level one. If you’re just babysitting, you’re not actually doing anything. How does that make you understanding of anything that’s going on behind the scenes?

What’s missing from this is the experience and the knowledge that if something goes wrong, you could fix it. Just because you’re pressing the button and babysitting with pre-baked tools, you’re back at level one.

Christopher Penn – 28:54

If all your job is is to hit “yes,” you don’t need a human for that. OpenClaw is perfectly capable of clicking “yes” by itself.

Katie Robbert – 29:08

The Simpsons did an episode on that where Homer had the little bird because he just needed to press “Y.” The bird fell over and everything fell apart because the machine stopped working. If that’s all you’re doing, you don’t need a human. That person doesn’t even rate on the level of proficiency if they’re just pressing “Y” over and over again.

Christopher Penn – 29:40

This is where a lot of organizations are going to be as these tools are introduced into Copilot. You have a whole bunch of essentially unnecessary babysitters because you can systematize the use of these tools.

Where in the past you could take your content marketing team from 50 people down to 10 because they can all use ChatGPT, now you can probably go down to two. You have one person to maintain the Cowork system and one person to spot-check outputs and make tweaks.

Katie Robbert – 30:50

Exactly. I posted an article today about this problem on LinkedIn—”AI Ate the Proving Ground: The Leadership Crisis No One Sees Coming.” We’re only focused on the fact that we can’t do billable hours anymore, but you need to do more forward thinking about what happens when we’ve given everything over to the machine and no humans are skilled to keep these things running. At the end of the day, it’s software, and software breaks every day.

Christopher Penn – 32:08

We’ve run into this problem before in manufacturing with “tool and die.” Americans had a great specialization in it until the 1990s once globalism and outsourcing became prominent and the generation of tool and die makers retired.

If you want to manufacture something in the U.S. now, you pretty much can’t because there’s no one who knows how to make the tools to make the product. You have to outsource to a place like China. This is the tool and die of knowledge work. If you don’t have a beginner’s bench of junior folks and you’ve baked everything into the machines, you no longer have the tool and die equivalent in your organization. You can no longer manufacture anything that isn’t handed to you.

Katie Robbert – 33:41

John, does this change your opinion of levels three and four being relevant?

John Wall – 33:49

The issue is that we’re anchoring the org chart to the tool stack. I totally get that you can’t operate at a leadership level without management experience.

But I’d also put out there that you totally skipped n8n. There’s no point in you learning or messing around with n8n because you’re good enough with Claude that it’s a waste of time. Anytime you make an abstract, simple measurement system, you give up detail and open up the chance for misunderstanding and not being able to accurately talk about what’s really going on under the surface.

Katie Robbert – 34:56

I have an article coming out in the next week or so that addresses n8n. It’s an MCP server where it can take all of your different systems and connect them.

Claude Cowork now has all those connections built in, making MCP servers not so much obsolete, but not as important for you to know how to develop against. But if you don’t even know what’s in your stack to access, and you’re just putting these agentic AI tools on top of really crappy data across 17 different platforms, guess what? You still get terrible outputs.

Katie Robbert – 36:33

You still have to go back and do those foundational pieces and understand the data that’s in the system. I will not only die on this hill, I will set it on fire and take you all down with me! You need to understand what these things are and how they work.

You don’t have to be the expert in every single data point in your CRM or SQL databases, but you have to know that they exist and have a good baseline of good data quality. When you’re building agentic systems on top of these databases, you have to know what to expect.

John Wall – 37:50

You have to have had hands-on when those things were built to understand what they’re actually plugged into. Otherwise, you’re stuck saying, “We got some bad results. I have no idea why that happened.”

Katie Robbert – 38:03

And then because you’ve put everyone at a level five or six, nobody knows what’s happening. Congratulations, you now have your C-suite and nobody has a clue how to fix anything.

John Wall – 38:18

This is a great time to talk about Trust Insights managed IP and analytics services.

Katie Robbert – 38:23

You’re correct. If you are interested in something like that, you can reach out and contact John.

Everyone’s going to have a different opinion on this, and that’s totally valid. My opinion is that you can’t responsibly skip levels to get to leadership. In order to do it in a sustainable way, you have to have those foundational skills. If you wake up tomorrow and say, “I’m going to be the CEO of my own company and I have no experience,” you could absolutely do that.

Christopher Penn – 39:27

The word “responsibly” has less and less place in business every day.

Katie Robbert – 39:39

And you’re right, and it just breaks my heart because that’s the short-term thinking again. Short-term thinking like, “We can’t do billable hours, so we just churn out more.” But that doesn’t mean that it’s good or better, it’s just faster.

Christopher Penn – 40:00

If a company says, “I want to measure AI proficiency of my staff,” where do they go? What’s the next step for them?

Katie Robbert – 40:18

They start with what is being asked of their staff today, what are their KPIs, and what are they being measured on for success? You have to figure out what their “slice” looks like and therefore what the appropriate levels are for their particular roles.

If you decide your customer support team needs to be at a level five operating agentic systems, my first question is “Why?” Doing what?

Katie Robbert – 41:03

You really need to help me understand what the team is responsible for today so that we can find those appropriate levels. It’s not going to look the same for everything. Our levels in Trust Insights will not translate to a different consultancy because what we do is different. My level of proficiency is always going to look different from John’s, which is always going to look different from yours, Chris, because we do different things.

Christopher Penn – 41:31

I would say in terms of next steps, use the idea of individual contributor, manager, and leader—and the increasing delegation of stuff to machines—as the basis. Maybe even on a role level at your organization.

I would fire up a copy of Claude Cowork, take a job description, and say, “What are the likely expectations for the use of AI in this position?” What are the three to five use cases based on this job description and evaluate it.

Christopher Penn – 42:24

If it’s a content marketer, you’d be expected to do ideation and create content. If you’re finance, you would be expected to be able to do reconciliation and auditing. AI will be a second set of eyes, or maybe the first set, and you’re the second set of eyes on the books. That would be a great way for an organization to practically take our framework with those job descriptions. Your levels are now mapped to all of those use cases for that role.

Katie Robbert – 43:02

If you want a copy of that framework today, we will share it in our free Slack community, Analytics for Marketers. Coming soon, I will add it to my list of web development things I’m apparently doing now because that is my role. I will get it up as an Instant Insight and we’ll have a proper URL for it.

Christopher Penn – 43:28

Any final thoughts about measuring AI proficiency?

Katie Robbert – 43:33

The biggest takeaway that I always encourage people to have is that the only real comparison belongs inside your team and your organization. You can look at your competitors, but you really have to start with what you are doing today and what you actually need, versus the aspirational. Comparing yourself to a Chris Penn is not a fair comparison.

Christopher Penn – 44:00

I don’t even compare me to me.

Katie Robbert – 44:03

Well, now we’re getting existential.

Christopher Penn – 44:18

On that note, thanks for watching today. Be sure to subscribe to our show wherever you’re watching it. Check out the Trust Insights podcast and our weekly email newsletter. Got questions about what you saw in today’s episode? Join our free Analytics for Marketers Slack group. See you next time.