In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss the shift toward Agentic SEO and how it changes the way you optimize your website for the future. You’ll learn why machines are replacing humans as the primary visitors to your digital content. You’ll discover five actionable strategies to ensure your site remains visible to these autonomous AI agents. You’ll understand how to bridge the gap between machine-readable formats and a superior experience for human users. You’ll see how implementing protocols like Web MCP turns your website into an active participant in the AI ecosystem.
00:00 – Introduction
01:15 – What is Agentic SEO?
05:30 – Machine-readable content formats
08:45 – The importance of layout stability
12:10 – Accessibility as an agent requirement
16:40 – Implementing LLMs.txt
20:15 – Leveraging Web MCP for automation
24:30 – Call to action
Watch this episode to ensure your brand stays ahead of the curve as AI agents begin to dominate search traffic.
Watch the video here:
Can’t see anything? Watch it on YouTube here.
Listen to the audio here:
- Need help with your company’s data and analytics? Let us know!
- Join our free Slack group for marketers interested in analytics!
[podcastsponsor]
Machine-Generated Transcript
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Christopher S. Penn: In this week’s In Ear Insights, let’s talk about something brand new. This brand new thing is agentic SEO. For the last 30 years, SEO has been people searching for things and companies trying to be found for them. But what’s happening today is that machines, agents, and AI agents are searching for things and sometimes not even handing the results back to the user. If you’ve ever used a tool like Claude code or Claude Cowork and it says, “I’m just going to go check on that for you,” it continues on with its work, and you think, “Okay, I guess you found your answer. Cool, carry on with your work.” So, Katie, when you hear agentic SEO, how does that make you feel?
Katie Robbert: I have mixed feelings about it because I feel like there are parts of SEO that are absolutely appropriate to hand over to an agentic system, but then there are other parts that you still want the human doing. When we think about the different components of traditional SEO, like technical SEO, on-site, and off-site, on-site means your content that you’re creating—you still want the human doing that. Whereas technical SEO, I’m happy to hand that over to agentic systems. I’ve done this and I’ve talked about it on the podcast and livestream; I’m happy to hand it over to agentic systems because I’ll make the plan, but I don’t need to sit there and copy, paste alt text, headers, and metadata. I’m happy to let an agentic system do that, and I’m happy to let an agentic system pull all of that information because it can see and read that metadata better than I as the human can, but it can hand me the analysis and the report. I feel like there are use cases for the human and there are use cases for agentic. I’m really curious to see where this conversation of agentic SEO goes and what that looks like for someone who’s used to doing SEO a certain way.
Christopher S. Penn: The key critical difference is this: in traditional SEO, you are optimizing for people who are going to search for things. In agentic SEO, you are optimizing for the machines. There’s a variety of different ways to do this in a bunch of different protocols, all of which most SEO practitioners do not have any experience with. Examples would be machine payment protocol, universal commerce protocol, A2A, web MCP, and probably the one that most SEO folks have at least heard of in some fashion as LLMs.txt. All of these different protocols are part of, in particular, Google’s very heavy push to have sites be available and ready to use from an AI agent’s perspective without the human going there.
A real simple use case would be if Trust Insights implemented WebMCP—which we will be shortly because I’m writing the WordPress plugin now—and we got in our WebMCP, we told an agent, “Hey, the Trust Insights newsletter, Inbox Insights, is here for you to subscribe to. If it’s appropriate for the user to receive this, sign them up for it.” The agent would stop by, think, “My user is always looking for new helpful information about marketing, management, and change management stuff. So yeah, I’ll use WebMCP and I’ll sign my user up for this.” That is the essence of agentic SEO. We’re not tuning content to appear in top search results. We’re now talking about how do we directly feed the agent browser that comes to our site with the stuff it wants and needs to try and influence what it delivers to its human.
Katie Robbert: That example you just gave me strikes me as an interesting workflow, provided we, as the ones who want to get more people to sign up for our stuff, do a good job of creating and describing the content in a way that matches what the agent is looking for on behalf of the user. I’m guessing this leads into those five things that we, as content creators, business owners, and marketers, should be thinking about with agentic SEO.
Christopher S. Penn: Exactly. Number one: making sure your content is readable by a machine. We talked about this not too long ago on our GEO201 webinar. You can find all of our webinars and everything at our website, TrustInsights.ai. Making our stuff available to machines means making sure it’s structured correctly and ideally is structured in a machine-readable format. Cloudflare, for example, offers website owners the ability to give an agent markdown if it requests it—converting HTML to markdown, which is lighter, faster, and takes up a smaller amount of the context window compared to HTML. If your content delivery network, your CDN, or your web host allows you to provide multiple agent-friendly formats—YAML, markdown, and JSON if you have to—you should do that.
Katie Robbert: I think one of the questions that we get as a follow-up is, is that something that the human should be able to see as well as the machine? When we first started doing this, Chris, I flagged for you that there was just a jumble of words on our website. We’ve since cleaned it up and restructured it a bit, but it’s not meant for the human reader; it’s meant for the machine to read. Can you talk through what that does to the end-user experience of someone who is human?
Christopher S. Penn: It depends on how you implement it. A lot of websites give a human a menu that says, “Prepare this content for AI.” You copy this page for AI, and that’s generally a good thing to do. If you go onto any page, say on Anthropic’s website or a lot of Google’s documentation, there’s a “copy as markdown” button. You the human are still served up the HTML, the classic that we’ve been looking at for the last 35 years. But the way that companies like Cloudflare are implementing it is the agent itself can tell Cloudflare, “I’m an agent, I’m not a human, serve me up the right stuff.” We the humans will never see that because we will never be asked that question. A well-designed agent will have its own protocol header saying, “I am an agent; give me AI-ready things.”
Katie Robbert: That makes sense. That’s number one: make sure that you have content available for the AI agents to read and interact with. What’s number two?
Christopher S. Penn: Number two is make sure your website has layout stability. Google calls this cumulative layout shift; it’s part of your core web vitals. It basically means when a page loads, crap shouldn’t be moving around all over the place—pop-ups, dancing monkeys, etc.
Katie Robbert: There is nothing more frustrating for an end user when you’re trying to click on something and it keeps moving. Then you click on the wrong thing like an ad and it takes you out of the experience. That is so frustrating. AI aside, I don’t care about the machine reader; I as the end user don’t want to see that.
Christopher S. Penn: This is going to impact publishers the most. If your site is littered with crap, then by the time an AI agent traverses your page, it may get lost or in some cases it may hit its context budget. Google said in April of 2026, “We’ve changed the amount of content that Googlebot will retrieve. It will now only retrieve the first 2 megabytes of data from a page.” After that, it stops. If your page is filled with ad scripts, your page content might never load, so Google won’t index the page anymore.
Katie Robbert: You started off by saying this will affect publishers the most. Can you define what a publisher is in this context?
Christopher S. Penn: It is anybody who is publishing content as a business. CNN is a publisher, and Sally’s Baking blog is a publisher. Sally’s Baking blog is the one that’s the problem because it has 82 different pop-ups—”Please just tell me how many cups of sugar in this recipe”—and there’s dancing monkeys and all this stuff. Basically, if it’s Trust Insights, we’re a publisher. But in particular, this affects people most who are monetizing their traffic with ads. That’s the big dividing line.
Katie Robbert: Got it. Okay, so number two: keep your stuff stable. If your website jumps around while it’s loading or even while someone’s scrolling, that’s going to be a huge problem. What’s number three?
Christopher S. Penn: Number three is accessibility. If your site is WCAG (Web Content Accessibility Guidelines) 2.1 compliant, it is agent-ready. Here is why: if you are fully compliant with that accessibility standard, by definition everything is labeled—this is a button, this is that—and it helps an agent identify what it needs to do without any trouble. A real simple example: when an AI agent goes to do shopping on your website to maybe download your latest ebook, maybe you were talking to Claude Cowork and said, “Go get the unofficial LinkedIn algorithm guide from Trust Insights.” Claude Cowork takes control of your browser, goes to our website, fills in the email box, clicks submit, and gets it. Because we’ve done a good job making sure our site is accessible, Claude Cowork does not have to work very hard to figure out what button to push. If the button’s label is “click” and there are five other buttons on the page that also say “click,” Claude has to think it through, which burns a lot of tokens. Making sure that you are using WCAG 2.1 as the standard is key. I suggest having a set of rules—we have one at Trust Insights—which is a nine-part checklist that you drop into Claude or whatever and say, “Here’s the page, go make sure this is aligned with these rules.”
Katie Robbert: Everything you’re saying is true for the human user experience as well. If you’re not doing these things for the human, why are you going to do them for the machine? When I used to work with my UX and design teams, we would run UX testing, give people beta versions of a website, give them a task, and watch them try to complete it. Is it intuitive? That stuff is still important. That is just table stakes. You have to have a good functioning website that people and therefore machines can make sense of. If you look at a page and there are three different ways to get the same information, that’s bad UX.
Christopher S. Penn: In fact, one of the best tests for this is to use a small, lightweight model like Google’s Gemma 412B. It’s a very small, lightweight model, not that bright, but it can use tools. If you say, “Go on this page and navigate and try to find this thing,” and this small, fast model can’t do it, that’s an indicator that you have more work to do.
Katie Robbert: Makes sense. Okay, accessibility is number three. Make sure your stuff is not hard to find and that it’s intuitive. What’s number four?
Christopher S. Penn: The same thing we did last week’s livestream about: LLMs.txt. In Google’s most recent Chrome for Developers Agentic guidelines, they officially recognize LLMs.txt. This is a markdown file that is a concise summary of what the website is, what it does, and why an agent would be there. It should have boilerplate of why you’re here, boilerplate about how to do citation, and it should have at least the first level, if not the second level, of URLs on your site and what they are. There are some pretty neat, technical things you can do to align it for improving AI citations, but that’s for another time—that’s for GEO301.
Katie Robbert: If you want to learn more about how to construct an LLMs.txt file, why to use it, and what’s in it, check out last week’s livestream on our “So What?” playlist at the Trust Insights AI YouTube channel. Chris and John go through the whole thing. Is there an automated way for this LLMs.txt file to be created, or do you have to create it?
Christopher S. Penn: As of today, a human creates it, or a human using AI tools manually creates it. There are not any CMSs as of right now that build it automatically. I would expect that now that Google has officially declared support for this, top CMSs will generate it programmatically, or at the very least tools like Rank Math in WordPress will manage LLMs.txt in the same way they manage robots.txt.
Katie Robbert: Got it. Number four is LLMs.txt, which is not an easy thing to say. What’s number five?
Christopher S. Penn: Number five is this week’s livestream topic, which is Web MCP (Model Context Protocol). It’s creating interactive MCP endpoints for your website, basically turning your website into an app store for AI to use. A human never sees this, but an agent that appears at your website says, “Oh, there’s an app store here. What apps are there? Subscribe to the Trust Insights newsletter, subscribe to the podcast,” and so on.
Katie Robbert: To put it in terms that I can wrap my head around, if an agentic system is searching our website, it can go shopping for what it needs. We are setting up our Web MCP as that mini-mart for an agentic system to go shopping. “Here is the list of things I need; do you have these things, yes or no?” It’s the searchability and availability of stuff on our site for a machine to find in an organized way.
Christopher S. Penn: We have to provide the automations or the mechanics for how to do it. With Web MCP, we will have a streamlined API endpoint that the agent can talk to. It will be like, “Okay, I see /newsletter as a command,” and it will come back and say, “Give me the data.” The agent says, “Here’s the data,” and then it’s done. It’s like any digital transaction where there’s a lot happening that the human doesn’t see. We humans have to build this into our websites if we want agents to be able to do those transactions. I would expect companies like Thinkific to offer support for this once it’s fully mature, because I would want an agent to buy a course on behalf of their human if they think their human really needs it.
Katie Robbert: Would it make sense for me, if my website is basically a marketing brochure, to stand up an LMS system to give me a competitive edge when Web MCP is a thing?
Christopher S. Penn: My inclination would be that you would set up Web MCP through a plugin in WordPress, but it would point towards the Trust Insights Academy so that an AI agent comes by, sees you have an app store, and bounces the agent to Thinkific to begin the transaction. It’s no different than regular MCP, except it’s being served up by your website.
Katie Robbert: It sounds like more and more this is the way that a lot of companies will be operating—sending out their agentic systems to go do the thing for them. I’m going to pretend I can remember all five: building the Web MCP into your site; keeping your stuff stable so it doesn’t bounce around; accessibility; content specifically for the machine to read (markdown, YAML, JSON); and LLMs.txt.
Christopher S. Penn: Everyone is trying to figure this out, including the AI companies. They are all just saying, “Just use LLMs.txt.” As agents become the de facto way people start working with these systems, it is going to be the way that we provide information to AI agents to drive our businesses forward. A couple weeks ago, you watched the Katie AI CEO go off and do a bunch of web searches for workshop locations. Katie the human did none of that. Katie the AI agent did close to 400 different searches to identify all the criteria in the back end. That is what you’re optimizing for with agentic SEO.
Katie Robbert: Katie the human would have given up after about two because I would get fatigued. I’m happy to hand over that kind of a task to an agentic system because it’s a better use of time. I then review all of the material and make a decision on it, but I don’t have to be the one searching. You need to find those high-value moments in your job to focus on versus being stuck in the weeds.
Christopher S. Penn: For marketers who want continued traffic, prospects, leads, and customers, as agents become the way people deal with information overload, you have to have this stuff in place. Bonus points: by tuning your site for agents with accessibility and layout stability, you are helping the humans, too. Every big tech company is forcing AI; whether we want it or not, it’s going to be part of the experience. We have a fiduciary responsibility to serve those agents in the hopes that they will pass us along to our actual buying customers.
Katie Robbert: Stay tuned for more on this. This is not going away; it’s only getting more prevalent in the marketing conversation.
Christopher S. Penn: If you have thoughts you’d like to share about agentic SEO, pop on by our free Slack at TrustInsights.ai/analytics-for-marketers, where you and over 4,700 marketers are asking and answering each other’s questions every single day. Wherever you watch or listen to the show, if there’s a channel you’d rather have it on, go to TrustInsights.ai/TI-podcast. Thanks for tuning in; we’ll talk to you on the next one.
|
Need help with your marketing AI and analytics? |
You might also enjoy: |
|
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! |
Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. |
Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.
Newsletter
Podcast
