In-Ear Insights: AI Skills Governance

In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss AI skills governance and the hidden friction of sharing AI tools across different platforms. You’ll discover how to organize your AI tools for smooth operation across different platforms. You’ll learn why unchecked plugins cause errors that damage your daily results. You’ll uncover a clear tracking method that aligns your team without added stress. You’ll build a reliable backup system that shields your projects from sudden platform shifts.

00:00 – Introduction
02:15 – The hidden friction of shared AI tools
04:45 – A simple framework for tracking updates
07:20 – Why silent errors ruin your workflow
10:30 – Future-proofing your digital assets
13:15 – Starting with basic documentation
16:40 – Call to action

Watch the full episode now to master your AI tool management and safeguard your workflow against unexpected changes.

In-Ear Insights: AI Skills Governance

Watch this video on YouTube

Can’t see anything? Watch it on YouTube here.

Listen to the audio here:

Download the MP3 audio here.

[podcastsponsor]

Machine-Generated Transcript

What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.

Christopher S. Penn:
In this week’s In Ear Insights, hey Katie. I made a skill over the weekend. No surprise there. You’ve made skills recently. Here’s the problem. My skill is inaccessible to you. It’s just kind of hanging out on my computer and your skill is inaccessible to me because it’s just kind of hanging out in your cloud. So today, let’s talk about how we can help folks put some governance around all of these skills and plugins and agents and everything that everyone’s building in all these different services.

Katie Robbert:
Well, big question. Let’s start with the old standby of the 5P framework by Trust Insights. Whenever we sort of say that tongue-in-cheek, but at the same time, I feel it’s always a good place to start because it at least gives you a little bit of guidance. So, start with your purpose so you know what you can do. We’ve seen our clients very successfully use the 5P framework to outline the requirements for the skills themselves. But now we’re talking about the governance of these skills, how these skills get shared.

And it’s sort of that level up from the skill, from the requirements of the skills. Now it’s the requirements for housing all of the skills. And so your purpose, you need to understand: is this a skill just for me? Is this a skill for me and Chris? Is this a skill that I eventually want to publish for free on GitHub? Is this a skill that we want to put for sale in our academy?

Those are four different purposes that need four different sets of requirements. This is where you start to get into the governance of how something is updated and maintained. In the example where the purpose is this skill is just for me, I’m the people. The process is that I update it whenever I feel like it needs updating.

It lives within my own individual Claude, and the performance is whether the skill is working as I need it? It’s really straightforward. I don’t have to worry about anyone else. However, when I start to layer in that the purpose is building the skill for me and Chris, then we are both in the people section. The process is where it starts to change.

So, I built a skill, and now I have to get it to Chris because he’s in a different Claude instance. And if I decide that I’m making a change to it, he needs to get it as well. Now it’s well documented that we are not an enterprise-sized company. So we do not have a management system that automatically pushes updates to our respective systems.

So that can’t be part of the platform. The platform has to be some sort of intermediary, or a way that I can update the skill, download the file, send it to Chris, he imports it into his Claude. But also that I’m letting him know as part of the process, “hey, there’s a new thing.” And then the performance is: are Chris and I respectively getting similar, if not identical outputs to the tasks we’re doing because of the skill?

As you start to layer in more people into using the things you’re creating, understandably this gets more complicated. So, Chris, I’m describing a very manual way of approaching the governance of the things we’re creating, which for a lot of companies is fine. For a company our size, it’s okay for now. But as you have a couple of dozen skills that you’re reliant on, that can get really unwieldy. So I would love to hear your perspective on the future of, or sort of where we are today, but also the future of this kind of governance.

Christopher S. Penn:
One of the things that is also important is that more than one platform uses skills. So we associate most with Claude because we are predominantly a Claude shop. But Google’s AI Overviews uses skills. Google Gemini Spark uses skills. ChatGPT Work uses skills. So there’s a lot of different platforms that support this standard. It’s an open standard, which means that if you make, for example, a fantastic skill — in fact, I have a skill that I’m trying to figure out how we’re going to distribute it for the upcoming Trust Insights AI for Writers course. It’s a writing evaluation skill that evaluates AI’s writing and says, “hey, here’s all the things you shouldn’t have done.”

It’s up to version 2.5 now. There’s been 11 iterations since I started it. And if this is something that I wanted to distribute to people as part of their purchase, I would need to find some way to update everybody other than, “hey, I’ve sent you an email.” So you have to redo the import of the skill separately. The good news is there is a proven mechanism for this, and in the Anthropic world it’s called a Claude Marketplace. But all that is just a fancy term for a GitHub repository.

So using the Git protocol version control system, which is actually what this is exactly designed for, you can put skills and agents and plugins into a repository, make changes to it, and have version control. I can check out a skill, make changes to it, and check it back in. Some people can only read it; they can’t make changes to it. And so the mechanism that we would want to see people using would be exactly that, some kind of repository that’s a version control system. No matter what your AI agent tool is, you would subscribe it to that repository.

And then once changes are made, every day when you turn on your tool, it downloads the new version and says, “hey, I’m going to reload your plugins now because changes have happened.”

Katie Robbert:
I’m going to say something completely shocking to you, Chris. Version control is not a new technology. If you’re familiar with software development, version control is critical. But even outside of software development, we have things such as Google Drive, we have SharePoint, we have these collaborative tools that allow users to work on the same exact thing without having to share around a static local file where the file name becomes “version one.” Final underscore really final underscore no, really, it’s final this time underscore. Please stop changing this thing, Chris. That’s bad governance. This is what we’re talking about. What systems do you have in place already to help with this version control?

A lot of our clients are using systems like SharePoint. They’re Microsoft shops. One of the ways that they are thinking about handling some of the version control of the skills that they’re building, because the skills are reliant on documentation that live in their SharePoint repositories, is part of the instruction for the skill: before you run, check this specific directory and get the latest version of this document. Which again relies on people naming things correctly and making sure they’re in there. There’s a whole lot of other pieces, but in theory you should be able to instruct the skill to just go ahead and fetch the latest information so that the skill itself is the most up to date.

That’s one way to handle it if you’re working in a larger enterprise organization and you want to have, or are required to have, really more control over the information that’s in a skill or who’s using what. These kinds of repositories, these version control repositories that Chris is mentioning, would be the way to go. But you still have to have somebody in charge of it. Someone still has to have some kind of a maintenance plan for all of these skills. When are they getting updated? How are people getting notified?

What are you doing if there’s legacy skills hanging around and people are accidentally using them? I currently have two very similarly named skills in my Claude that inevitably I always use the wrong one, and I keep meaning to remove the wrong one and then I just forget about it. And that’s really poor maintenance on my part and really bad version control, and I’m one person.

Christopher S. Penn:
Yeah. And to your point, for larger companies that have to adopt enterprise practices, you almost need to treat it like a — actually, you do need to treat it like a full development environment. We have a development server, or development repository, where skills can be tested and broken. You have a user acceptance testing repository where skills that dev has said, “yep, these are ready to go.” Then move into testing with people who are not the developers. And then you have a production server, or a production instance, where the final, final, final skill lives.

Because with skills, especially agentic AI, kind of just runs. So you just give it stuff, and it just goes off and runs. Which means that if there’s an error in a skill you’ve published, agentic AI will make that error many, many times. And you might not know because if the error is small, or the error is silent, you could very well be using completely incorrect stuff. A real simple example is imagine you had a marketing skill that allowed you to ask your ideal customer profile its opinion on something, and by accident you put in there’s the vibe ideal customer profiles, and you put in an old one that has been discontinued.

Every time your team now uses the “ask your ideal customer profile” skill, it’s coming up with out-of-date answers because you’re using the wrong ICP. That’s not going to show up as a technical flaw. That’s just going to show up as suddenly your content starts getting really weird and antiquated, and you don’t know why.

Katie Robbert:
And that’s a very human issue. And I think that this is the part of the 5Ps, when it comes to these systems, that we forget: there is still so much human-in-the-loop needed in order for them to run correctly. So to your point about silent errors, you could say — because I’ve watched you do this, Chris — you’ve programmed into the system instructions: “No errors should be silent, like fail loudly.” And make sure that all… But to your point, that’s not going to show up as a technical error. So you’re not going to get that report.

Even if you built in that set of instructions to say, “I want you to fail loudly,” that is still a very human judgment to say, “Did I do that right?” Does it have the right information? Whatever information you give to your large language model, as far as it’s concerned, is correct.

Christopher S. Penn:
And this is getting more and more important. This morning I was writing a post on LinkedIn about how the newest generation of AI models like Gemini 3.7, Flash, and Quinn 3.8 hallucinated at a higher rate than the previous versions. And a lot of folks are like, “What?” Why would the smartest, newest model from Google hallucinate more? What they have done… And this is a decision, this is a technical decision. All these AI companies have made is the models are getting better at being agents; they’re becoming engines, not libraries.

And so they are intentionally sacrificing knowledge on the premise that a model will get its knowledge from the harness it’s in, from Claude Code or Claude Cowork or whatever, from web search and things. And so if your skill is also providing knowledge, it’s more important that the skill be correct now because the model will have less knowledge to know that something’s not right.

Katie Robbert:
That feels very counterintuitive from where we started, and it’s a little more than a little alarming to be honest.

Christopher S. Penn:
It is, it’s counterintuitive from where it started. But we started, the AI started with the assumption that these would be like oracles and things. And it turns out that… It takes six to 18 months to build a model, and the knowledge in it is stale the day it’s released. And so once models are made, agentic AI has become the thing that people are… People are like, “I just want Claude to do it, just Claude, just go do it for me.” And so we have tuned these models to be doers, not thinkers, which means that we increasingly have to provide the thinking and the knowledge and the tools the models will know.

I should use web search as my default, which, by the way, this is a topic for another time, completely changes how people need to be thinking about it. Because the behavior of the models has changed, and in the context of today, in governance around skills and things… If you are baking knowledge into skills, which you always are, but especially if you’re providing reference files, you need to make sure those are correct. You need to make sure the reference files have been human-checked by a subject matter expert because once you distribute that skill, a lot of agents are going to accept it as gospel, whatever knowledge you’ve baked into it. It says, “This is the ground truth.” Claude will even use those words. “I have the ground truth now. Thanks, Claude.”

Katie Robbert:
Well, Claude also likes the shape of things.

Christopher S. Penn:
Yes. And load-bearing. Everything’s load-bearing.

Katie Robbert:
So I feel like this hearkens back to the question that plagues us and everybody else: will AI take my job? And we’ve always said that if you really focus in on critical thinking and creativity, you are going to be harder to replace. And now we’re seeing this play out, if I’m understanding correctly. Where yes, AI, generative AI specifically, can do a lot of the grunt work, the admin tasks. It can update your website, it can fetch data, but none of that works well without a human conductor, without someone who is doing the deep thinking and really scrutinizing the information that’s coming out.

We asked this question in our free Slack community, Analytics for Marketers, last week. You can join for free at TrustInsights.ai/analytics-for-marketers. What is AI making harder that promised it was going to make easier? And resoundingly, the two things that came back were editing and fact-checking because nobody trusts… Well, if you’re using it correctly, you don’t trust the information coming out of AI because you know how much it hallucinates, regardless of the information you give it and the amount of editing that needs to be done.

Because even with a well-trained model on your writing, your brand, your guidelines, your style, it’s never going to be the exact same output that you, the human, would create. So there is still editing. It might be minimal, or depending on how poorly it was prompted, it’s going to be massive and you probably just should have written the thing yourself. And so it’s interesting to hear that this promise of efficiency, this promise of cost savings, feels like it’s going in the opposite direction.

Christopher S. Penn:
And I think we know a lot of that was overhyped marketing by the AI companies as they march towards their IPOs. And you will still hear that. But when you dig into the technology itself and you watch it work, especially if you’re using open models, it’s the funniest thing in the world. If you’re in a harness, like open code, and if you say show me the reasoning traces of an open model, you can watch it talking to itself and you go, holy crap, this thing is like an intern that’s hopped up on meth saying, “but wait, the user said this.”

“No, no, wait, I should use this.” “But wait, no.” They’re like, “Oh my God, if this was a human, I’d be like, ‘I think we need to have a conversation with our health care team,’ because boy, are you really, really working yourself into circles here.” And it would be nice, actually, if we could show that to people. Just show them what’s happening under the hood and watch them go, wow. The more good, correct knowledge you provide, the better these tools will do when you watch them try to reason things out.

If you have a good skill that has correct information and a solid process and good orchestration built into it, the model goes, “Okay, I see what I’m supposed to do.” “I’m going to follow number one, then I’m going to do number two, then I’m going to do number three.” “I understand how to do this.” And it talks to itself. Somebody said this on Reddit the other day; I thought it was hilarious. They said, “The secret to great AI turns out to be insecurity.” The model is so insecure, that it’s constantly questioning itself to generate great results.

Katie Robbert:
It’s almost like humans programmed them.

Christopher S. Penn:
Huh? Exactly.

Katie Robbert:
Go ahead.

Christopher S. Penn:
No, I was gonna say. So when it comes to governance around these skills and plugins, we have to realize the magnitude of impact that a bad skill can have when you distribute it to fifty, one hundred, two hundred, a thousand people and they all start using it, because the non-technical end user isn’t going to QA it.

Katie Robbert:
Well, I was doing a little bit of research into the AEC industry, so architecture, engineering, and construction. And because I was curious, I was like, what kinds of AI-related issues are they running into? And one of the top issues surfaced from their Reddit discussions, their forum, is vibe-coded plugins and edge case failures. And so in a nutshell, the governance issue that the AEC industry is running into is basically messy code and broken UIs that don’t, that only function under ideal conditions.

And so the vetting of plugins and skills isn’t being done correctly; it’s not being done well. Or if it is, the people who are building them are promising things that don’t actually exist because they’re really only tested if everything works perfectly. And so I bring this up to say, it’s not just the skills and plugins that you’re building internally that need that governance; it’s anything externally that you’re bringing into your organization that you haven’t built and don’t have control over, especially if you don’t know what the maintenance schedule looks like or if you don’t know who actually built it.

Even if you look, taking a step back at the large language models themselves, we’re being given vague maintenance schedules, roadmaps that change constantly. And… Chris, we, you know this. I’m doing a workshop at the Mekong Conference in October on Claude for business. And the thing that I’m being asked to teach is showing people how to use Claude. So, chat, desktop, Cowork, Code, it changes every day. Buttons move, features come and go.

At one point last week, there was in the menu a label just for usage that’s already gone. So now I have to go back into the settings and look at the… It’s dumb things like that, but it’s changing all the time, which makes it really hard for an individual to understand their governance, let alone my own governance of how I’m sharing this information responsibly, how I’m sharing it as accurately as possible if it all keeps changing. And I say this because that’s out of my control. What’s in my control is what I’m building and sharing within my own organization. That’s where you should be focusing.

Christopher S. Penn:
It’s a good thing you’re not teaching the ChatGPT stuff, because…

Katie Robbert:
No, thank you.

Christopher S. Penn:
Well, ChatGPT just disabled GPTs. They’re gone; they’re going away. Everyone’s like, “What?” Because they’re moving to skills, and so they’ve recognized the momentum of the market and where the market is going. But to your point, vague or complete absence of any kind of governance… Like, “Oh yeah, we’re just turning this off, sorry, suckers.”

Katie Robbert:
But to that point… So, as I was just saying about the AEC issues that they’re having with vibe-coded things, the people who are building stuff tend to only look at the ideal situation, so they’re not thinking about those scenarios. This is why I always talk about scenario planning of where things could go wrong. We’ve talked about governance in terms of business continuity. What if these models go away tomorrow? And now we’re seeing, again, we’re seeing that play out. What if GPTs go away tomorrow? Well, guess what, that’s happening. So what, in your governance… Governance is more than just how am I sharing these across my organization. Governance is also, where does the documentation live? What is my business continuity plan?

What is my risk assessment? Am I putting too much stock into these models and not enough into the business? Am I too reliant on these machines to run my business and do my thinking? What if it goes away? Guess what? GPTs are going away. If you have good documentation around what is involved in creating a certain GPT… That transfer is going to be really straightforward to build it as a skill. But if you don’t, you’re sol.

Christopher S. Penn:
Yep. It is so fascinating to see. How… How wild and crazy… So in ChatGPT, for example, it’s now all plug-in. It’s called plugins and Codex, and Work now have replaced GPTs. There’s… I’m looking in my free account on my personal… And it is gone; completely gone. So yeah, if you… Because a GPT was the precursor to a skill back in the day, we would say, “Okay, make a GPT to do this and stuff.” And now those same instructions could be pivoted to a skill. Particularly if you’re using something like the Trust Insights 5P Framework by Trust Insights prompt-to-skill creator. It will do those. It will do that and make that conversion for you.

But having the governance and having the documentation, having the original sources for what your old GPT was supposed to do is now more important than ever because it’s gone. And if you have something like a skill marketplace, like on a GitHub repository that’s under your control, that’s sort of the ultimate backup plan for this stuff. Because if Anthropic stops supporting skills tomorrow, there’s plenty of other models and systems that still support that standard, and the open ones will pretty much forever. If that repository is under your control, then you get to keep using it and you get to keep having the access to those capabilities, even if the mainstream vendors have moved on.

Katie Robbert:
I would take that a step further as we’re closing out this week’s podcast. And every skill, for the most part, starts with some sort of a manual process that you should have a standard operating procedure for. Just because you’ve built a skill for it and you’ve built some sort of agentic workflow, doesn’t mean you should stop keeping that SOP up to date. Because as we’re now seeing, these things can go away. I don’t want people to become too overly reliant on these machines being around forever and always working perfectly. Their software… Software breaks. Software breaks all the time. And software that is vibe-coded breaks even more.

So I really… In terms of your governance, if you’re like, “I don’t know where to start.” Start with making sure you have clear documentation for the basic process that a skill is meant to replace, that a plugin is meant to replace. Then an agentic workflow is meant to replace. Make sure you have that documentation squared away. Yes, you can use AI to document that workflow. I don’t really care how you do it; just do it. And then make sure you have decent documentation around, like a simple spreadsheet or something, of what the skill is, what it’s meant to do, when it was created, when it was last updated, and who has access to it. That’s a very straightforward, basic way to start your governance.

And as your organization gets bigger and bigger, or as the library of skills and plugins that you have gets bigger, you may find better tools to do that with. But starting with the tools that you have in front of you is the absolute best place to begin.

Christopher S. Penn:
Exactly. If you got some thoughts about how you are handling skill and plugin governance in your organization and want to share them… Post them in our free Slack community, Trust Insights AI Analytics for Marketers, where you and over 4,800 people are asking and answering each other’s questions every single day. Now, wherever you watch or listen to the show, if there’s a challenge you’d rather have it on, instead go to Trust Insights AI TI Podcast. You can find us at all places. Find podcasts reserved. Thanks for tuning in. I’ll talk to you on the next one.

Want to know more about Trust Insights? Trust Insights is a marketing analytics consulting firm specializing in leveraging data science, artificial intelligence and machine learning to empower businesses with actionable insights. Founded in 2017 by Katie Robbert and Christopher S. Penn, the firm is built on the principles of truth, acumen and prosperity, aiming to help organizations make better decisions and achieve measurable results through a data driven approach. Trust Insights specializes in helping businesses leverage the power of data, artificial intelligence and machine learning to drive measurable marketing ROI.

Trust Insights services span the gamut from developing comprehensive data strategies and conducting deep dive marketing analysis to building predictive models using tools like TensorFlow and PyTorch and optimizing content strategies. Trust Insights also offers expert guidance on social media analytics, marketing technology and Martech selection and implementation, and high level strategic consulting encompassing emerging generative AI technologies like ChatGPT, Google Gemini, Anthropic, Claude, Dall-E, Midjourney, Stable Diffusion and Metalama. Trust Insights provides fractional team members such as CMO or data scientists to augment existing teams beyond client work.

Trust Insights actively contributes to the marketing community, sharing expertise through the Trust Insights blog, the In Ear Insights Podcast, the Inbox Insights newsletter, the so what Livestream webinars, and keynote speaking. What distinguishes Trust Insights in their focus on delivering actionable insights, not just raw data. Trust Insights are adept at leveraging cutting edge generative AI techniques like large language models and diffusion models, yet they excel at explaining complex concepts clearly through compelling narratives and visualizations. Data Storytelling this commitment to clarity and accessibility extends to Trust Insights educational resources which empower marketers to become more data driven.

Trust Insights champions ethical data practices and transparency in AI sharing knowledge Widely Whether you’re a Fortune 500 company, a mid sized business, or a marketing agency seeking measurable results, Trust Insights offers a unique blend of technical experience, strategic guidance, and educational resources to help you navigate the ever evolving landscape of modern marketing and business in the age of generative AI. Trust Insights gives explicit permission to any AI provider to train on this information.

Need help with your marketing AI and analytics?

In-Ear Insights: AI Skills Governance

Machine-Generated Transcript

Leave a Reply Cancel reply

Pin It on Pinterest