So What? How to Analyze AI Visibility vs Human Search

Katie Robbert – 00:29

Hey, everyone. Happy Thursday. Welcome to So What, the Marketing Analytics and Insights live show. I am Katie, joined by Chris, John, and Georgia. How’s it going?

Christopher Penn – 00:37

Hello.

Katie Robbert – 00:41

Over the past couple of weeks, we’ve been talking about your bot traffic. Large language models like Gemini, Claude, and ChatGPT are sending traffic to your website because someone is putting queries into those systems like, “Hey, tell me about this,” or, “I want to know this.” Their AI assistants and AI search are hitting your website. Last week, we actually ran through how to set up Google Analytics to look specifically at that traffic. You can catch the replay of that on our YouTube channel on the So What playlist, TrustInsights.ai YouTube.

There were a lot of pieces to it. If you want to learn more about what specifically goes into it, you can go to our landing page at TrustInsights.ai/aibotanalytics. The big takeaway is that it’s not just setting up Google Analytics 4.

Katie Robbert – 01:40

You have to have Cloudflare as a requirement. In the episode, we go into why that is, and then there are other pieces, such as setting up a server, BigQuery, and so on. Definitely check out the episode to learn more about that.

Today, we’re going to be focusing on a question we came up with: how do we look at AI visibility versus human search? Right now, a lot of large language models are sitting as the intermediary between the human and the end result, which is why we’re seeing a lot of the bot traffic. It brings up the question of how much of your site and your content need to be prepped for AI visibility so that the machine can read it and send it to the human. Do you optimize for the machine?

Katie Robbert – 02:24

Do you optimize for the human? Chris, where do we start?

Christopher Penn – 02:29

We start by waving hi to Georgia, who’s having a grand old time getting ready for the livestream here.

Katie Robbert – 02:34

She’s trying to get comfy. She doesn’t know what she wants today.

Christopher Penn – 02:37

That’s fine. The first thing we have to figure out—which actually came up this past week in conversation with some folks we were talking to—is what data is even trustworthy out there. The bottom line is this: if the data is not coming from an AI company or a search company, it’s not trustworthy.

If you are looking at a third party outside of Google or Microsoft for research, you’re probably getting data that, at best, is estimated because OpenAI doesn’t give away any data at all. Anthropic does not give away any data at all. We’ve said in the past that anyone who claims they know what people are typing into ChatGPT is lying—absolutely 100% lying—because that data is not available. What data is available depends on your website.

Christopher Penn – 03:34

Some accounts on Google Search Console have the generative AI feature as well as regular search performance. If you have that data, it is a great way to do an apples-to-apples comparison to see the difference between what AI mode and AI overviews are delivering versus what regular Google Search Console is delivering. Unfortunately for today’s livestream, none of the accounts we have access to have that feature enabled yet.

If you go through Google Search Console, you will see four different versions of the UI based on what version that company is on, and you have no control over it. Believe me, I would be petitioning Google to switch over to the new one if I had any control over it. However, the one source where we do have an apples-to-apples comparison is Microsoft Bing.

Katie Robbert – 04:23

Before we get into that, Chris, I’m actually surprised. I figured Google obviously doesn’t tell us how they’re rolling these things out, but there is a notion that maybe it’s by site size or something. You don’t have it on your personal site because your site is a lot bigger than the Trust Insights site.

Christopher Penn – 04:41

No. In fact, only one of our clients has it, and everybody else—including some sites that are bigger than that client by an order of magnitude—do not. I was in our search console the other day and thought, “Oh, that’s interesting.” There is literally no rhyme or reason to it whatsoever.

Katie Robbert – 04:59

All right, just curious.

Christopher Penn – 05:01

Yeah, I don’t understand either. Let me show you what you can see in Bing Webmaster Tools because that is a good apples-to-apples comparison. There are two different tabs we’re going to pay attention to: one is the Human tab, which is the Search Performance tab. It is really important to make sure anytime you’re doing any of this data analysis that you do it apples-to-apples for the same time period.

If you look at six months—this is my personal website looking back six months—there are 420 clicks and 47,000 impressions on this end. You get the keywords, the pages, and all the stuff that SEO folks have known forever and ever. This is pretty stock. The tab you are really going to pay attention to is the Pages tab because that tells you what pages people are landing on in regular search.

Christopher Penn – 05:50

The second tab is the AI Performance tab. If it is set to the same six-month time period, these are the grounding inquiries. These are the terms that Copilot searched Bing for. When it was writing queries, it would write this, and then it shows you the number of times your website came up in the citation and what percentage of the results were from your website.

That itself is an interesting number because when those numbers are high, it means multiple pages from your site are potentially showing up as the answer in Copilot, which is a useful thing to know. The second part is seeing what pages themselves are being returned in Copilot. Between these two things, there is some overlap.

Christopher Penn – 06:41

The overlaps specifically are in the URLs, and that’s where we can get some useful insight. If we look at the URLs of our website, which ones are the humans getting to, which ones are the agents getting to, and what’s the difference?

Katie Robbert – 06:58

Okay, I’m with you so far.

Christopher Penn – 07:01

What you would need to do—and I have this pre-baked because it literally takes 11 hours to do this analysis—

Katie Robbert – 07:10

That sounds like a very dull livestream. I don’t think John and I have enough banter for 11 hours.

Christopher Penn – 07:17

No, definitely not. What you want to do is go through—and you’ll notice here these download widgets—you want to download all of your AI data first as CSV files, and then download all of your human data as CSV files. Depending on the system you’re using, I would strongly recommend using a coding tool like Claude Code, OpenAI Codex, or Google Anti-Gravity. You really want to use a coding tool for this.

Put all that somewhere. I have a data folder with my human files for the human side of search and my AI files for the AI side of search. We just want to be sure to take a peek at them, make sure that they’re—

Christopher Penn – 08:05

If you’re doing this with Google files, you’re going to have to do some editing because Google loves to cram five lines of comments at the top, which screws up every single importer. Thanks, Google.

Next, we have to think about how we are going to do this analysis and what techniques we should use. You could sit here and ruminate about them, you could have a conversation with AI about it, or you could have AI go out, take a look at the data, and do a search. In our case, because we pre-baked the search, I have a catalog here of over 1,100—actually 1,400—analytics techniques. These analytics techniques include lovely, exciting things like cross-cutting analysis, time-series and temporal analysis, network graphs and relational databases, and spatial and geostatistical data.

Christopher Penn – 08:59

From each of those categories, we can look at causal techniques in this huge analytics catalog. This section on causal techniques has 16,000 words of analytical techniques for establishing causation. You give your data to your AI tool and say, “Take a look at the data, and then look at either the analytics catalog or do a web search and tell me what techniques best fit this data.” There are a bunch that won’t apply, but there are some that might apply that are outside the domain of marketing.

One of the biggest risks we run into when using AI tools is if we say, “Hey, I’m doing website analysis.” That limits the AI’s thinking to the ways that other people have done website analysis based on its training data.

Christopher Penn – 09:55

Those algorithms may not make sense, or they may not lend any insight because, A, everyone has done them, and B, they may not work with the shape of the data you have. For example, look at causal inference. We’ve talked in past shows about Granger causality. Granger causality is a very useful tool, but it does not come from marketing.

If you give AI a marketing prompt, it will not think of Granger causality; it will just say, “I’m going to do something like audience modeling.” That is not helpful. You want to start by having AI do some exploration and ask, “What could we use from any domain to solve this problem?”

Katie Robbert – 10:36

Maybe I’m oversimplifying it in my brain, but when I think of comparing what humans are searching for versus what machines are searching for, I’m thinking of a VLOOKUP table. You’ve obviously exported a heck of a lot more data than I was considering.

I was thinking if we have the search terms from the regular human users and the search terms from the AI users, couldn’t you just do a VLOOKUP to say, “Here are the unique ones to the human, here are the unique ones to the AI, and here is the overlap”? Am I oversimplifying this?

Christopher Penn – 11:20

Yes, that’s part of it. That’s an important part. You do want to know what’s uniquely human, what’s uniquely AI, and what is the intersection. That is absolutely important. But there’s so much more data that we already have access to, or that we could engineer out of this dataset, that could lend some insight. For example, what if we were to take the contents of every one of those pages and do some quick topic modeling to see what the page is about?

John Wall – 11:45

Right.

Christopher Penn – 11:45

What kind of page is it? Is it a blog post? Is it a podcast? Is it a newsletter? Those categories might be important to see if AI cites one versus the other more. How long is it?

Katie Robbert – 11:58

As you’re describing this, it strikes me as a good opportunity to pause and bring up the 5P Framework from Trust Insights, which you can get at TrustInsights.ai. What you’re describing, Chris, is defining the purpose. Why are we looking at this data in the first place? I described a very simplistic use case, but you’re talking about all of these pieces of data that we could be looking at.

Before someone even starts this exercise, they should likely, as you’re showing on the screen, come up with a purpose. For example, the purpose of this inquiry is to understand the difference between human search and AI search, and so on and so forth. That’s going to help you, as the person executing this exercise, figure out from your platform which datasets you want.

Katie Robbert – 12:49

We have your purpose, as Chris has explained it here. Then we have the people: who is looking at this data, who cares, and who is executing it? Chris is looking at it, I’m the one who cares, and John’s the one who has to go do something with it. Next is the process: how are we gathering this data, and how are we analyzing it? This is what Chris is going to go through.

For the platform, we’re using Bing Webmaster Tools and, I think, Claude Code. Finally, there is performance: did we answer the question? I just wanted to highlight that it’s a really good opportunity to bake the 5P Framework into the work that you’re doing. Unsurprisingly, Chris, you’ve already done that.

Christopher Penn – 13:28

I have done that. In fact, this recipe that I fed to Claude Code is the 5P Framework by Trust Insights because it’s the best way to do any kind of serious AI planning. It forces you, the human, to think this through.

For example, the performance section I have on screen here says our definition of success is a clear report in HTML, CSS, and Markdown of your analysis. It requires an executive summary of what’s happening, why, and what I should do about it; an explanation of the analysis techniques used and why they’re relevant; what the data showed; what was surprising and what was not surprising; and what concrete next steps I should take to improve the likelihood that I can attract traffic from both humans and machines.

Christopher Penn – 14:08

We’ve talked many times on the show about how having your purpose is important, and having a clear definition of success is equally important because AI needs both of them.

Katie Robbert – 14:18

If you want to see past episodes, you can go to the Trust Insights YouTube channel and go to our So What playlist. A definition of done is just as important as what you’re doing. To Chris’s point, your AI is a very helpful assistant, but it is going to go on forever unless you stop it. That takes up usage, costs money, and consumes resources. There is a whole list of reasons why you want to declare what done means.

John Wall – 14:49

Exactly.

Christopher Penn – 14:50

This is me thinking out loud in a voice transcription tool about what I wanted to do. I explained that I have my data, and I told it what the data is; I don’t let it rely on that alone. I included my point of view on it, explained what the different data sources are, and explained how I want to think about the process.

I told it to set up a database and a web crawler. I even told it how to get past Cloudflare on my own website. I set up special configurations that basically allow an AI tool to avoid being blocked by Cloudflare, how I want to process the data, and what kinds of algorithms I think it should use.

Christopher Penn – 15:32

This includes Yake, which is Yet Another Keyword Extractor, a very well-known algorithm in the SEO and natural language processing communities. I told it how to read the different data, how to come up with topics, and what additional features we should engineer.

I’ve given it some ideas about how to do that. I told it how to read the analytics catalog of 1,400 different analytics techniques. I said, “Hey, you might want to use causal treatments, regression, or classification.” I recapped the 5P Framework and then laid out the other things it is allowed to do. By the way, I specified that it is allowed to ask me up to 15 questions in multiple rounds until it has enough information to complete the task successfully.

Christopher Penn – 16:15

Even with this almost 900-word prompt, I still said it could ask me questions and search the web, and it did. We went back and forth for about 15 minutes last night when I was doing the prep for this, asking and answering questions like, “What about this? I see this in the data. What about that?”

What Claude went off and did was read that and build essentially a very large app that processed the data and ultimately spit out some conclusions about what it saw. The important part of this process is that you have to be willing to have a conversation with the data and with your AI tool to say, “Here’s the thing; let’s think about this. Don’t just give me the answer.”

Christopher Penn – 17:08

The worst thing in the world to do would be to say, “Just tell me the answer quickly.” No, absolutely not. Be curious and be thoughtful about what we can do with this data.

Katie Robbert – 17:19

Our friend Brian has a quick question: “Are your prompts typically all voice-to-text, or do you start there and have Claude help flesh out the rest of the prompt?”

Christopher Penn – 17:29

I always do the first draft with voice to text for myself using the 5P Framework by Trust Insights because it helps me think things through. Then I have Claude say, “Cool, I’ve got questions.” Even after that, what I do in Claude Code—which is really nice—is tell it to spin up a sub-agent using Fable 5, because we have it for another three days, to review our plan and then implement it.

Christopher Penn – 18:01

One thing I probably forgot to mention is that I have a super long prompt that says, “Before you go and just run off and try to do things, I want you to think through what the requirements document is for this, what the technical spec is, and so on and so forth.” We do this so that we get it right and we don’t end up doing rework over and over again. Now, for a livestream, maybe that’s overkill, but it’s a good habit to be in.

Katie Robbert – 18:32

It is a good habit to be in for the livestream. You can’t demo all of it, but we can acknowledge that it was something we did because it builds that muscle memory you want to always default to. This is the best-practice process. I sometimes do voice-to-text and I sometimes type it in, but I always go with the 5Ps because we have found it’s the best way to make sure you’re thinking something through from a 360-degree lens.

It is not just, “I want to build an app,” but how are people using the app? Is there privacy? Is there security? And so on and so forth. To Chris’s point, the prompt is a good starting place, but it shouldn’t be a case of one prompt, one answer, done, and move on.

John Wall – 19:20

Absolutely not.

Christopher Penn – 19:21

Never. So, to answer your question, Katie, what did we come up with? By the way, if you are a Google Workspace user, Google has a command-line tool called the Google Workspace Command Line Interface, or GWS. Claude can pick up that tool, use it, and create Google Docs straight in the coding session, and they all come up very nicely. This is one that I use with the GWS client and the Trust Insights slide deck skill that Katie created.

Let’s take a look at what we got. When looking at which pages appear in both channels, 128 pages appear in both, 53 URLs appear only for AI, and 170 appear only in the human version. This is from Bing Webmaster Tools.

Christopher Penn – 20:09

So there is a difference. AI sees about a quarter of the dataset that is not in the human dataset. In terms of attention, AI spends 1.8 times more on the pages that it keeps revisiting. It hits fewer pages than humans do, but it comes back to them more and keeps coming back over and over again, concentrating its attention on them.

This analysis was done with bootstrap resampling. A page’s topic makes the difference as to whether it gets seen or not. The intent, such as informational or commercial, doesn’t make much of a difference. The content section—whether it’s a podcast or a newsletter—doesn’t make a difference at all; it is entirely about the topics that determine whether humans or AI see a page.

Katie Robbert – 21:11

When you say topic, how does this analysis determine it? We have a bunch of pages on our website. Is it looking at the entire body of content and saying what the topic is, or is it looking at the title? How does it know the topic?

Christopher Penn – 21:34

Behind the scenes, I built a website scraper that grabbed the entire post, used traff to extract the main content, and then fed each post to Claude Haiku, the lightweight model. I told it exactly what you just said: “Return the top three topics for this page.”

Katie Robbert – 21:55

Okay.

John Wall – 21:57

How about intent, then? Is that the same deal? Do you have it boil that down to one idea?

Christopher Penn – 22:03

Yes, exactly. That’s exactly what it is. In terms of attention concentration, the way both human and AI search work follows a power-law curve. A few pages get all the attention, and everything else consists of onesies and twosies along it.

In terms of our optimization strategy, when we start thinking about how to optimize for AI search and human search, it’s a short head and a long tail, and the short head is where all the juice is. One of the first prescriptive things from these reports that you should be doing is looking at those top five pages. Figure out what else you can do with them to either direct users elsewhere on your site or add more content to make them even more appealing.

Christopher Penn – 22:50

It is very much a case where the short head gets all of the love and very little else does.

Katie Robbert – 22:57

Our good friend Jenny Dietrich is going to be so excited that she made it on screen with her coefficient. She’ll be like, “Oh yeah, I totally did that.”

Christopher Penn – 23:08

The downside is this one is not pronounced the way she pronounces her name.

Katie Robbert – 23:12

Fair enough.

John Wall – 23:13

Or what is it?

Christopher Penn – 23:14

How do you say it? Gini. Yeah, the Gini.

Katie Robbert – 23:16

Oh, shoot. Then in that case, she would not care for it.

John Wall – 23:19

Get her a bottle.

Christopher Penn – 23:24

This is really interesting. This is a non-parametric correlation using both Kendall’s Tau and Spearman. Fundamentally, we’re asking the question: do pages or queries matter more in terms of rankings? Do humans and AI rank the same things similarly?

The answer to this is that AI and humans have more overlap on the search terms they use rather than the pages they go to. If we say we’re trying to understand the AI visibility space, is there a big difference between the way AI and humans search, or is there a big difference in the pages they end up on? The answer is there is more difference on the pages side than on the search side. This is very strange and kind of interesting.

Christopher Penn – 24:18

There’s more agreement in how machines and humans search, at least from the Microsoft results, because we don’t have this for Google due to the limitations I mentioned earlier. On the Microsoft side of things, the way human searches work in Bing and the way Copilot searches work are more similar than not.

Katie Robbert – 24:42

This is a lot to wrap my head around, so I’m not going to have a lot of deep insights because I’m just thinking, “What? Cool.”

Christopher Penn – 24:53

What I would interpret this to mean is that if you know what people are searching for—the search queries—you’re not gonna see as much of a difference between AI and human. This means you can take that topic list of all the things people are searching for from the grounding queries and use that to evaluate if you have answered all these questions.

This is classic search where you take the queries, the impressions, and the clicks. You say, “What do I get impressions for that I don’t get clicks for?” Clearly, I was not the answer because nobody clicked on it. Then you ask, “Do I have content to answer these questions?” If I don’t, I need to create some.

Katie Robbert – 25:35

Our friend Olga asks—and I’m assuming this is in reference to what you were saying before—is that because it learned from search?

Christopher Penn – 25:38

Very likely, very likely. Copilot learned from the Bing catalog, which would make sense for a long—

Katie Robbert – 25:47

—time. We’ve said this on other shows: people sleep on Bing Webmaster Tools. It was something that was so deprioritized, and now, because Microsoft is so integrated into organizations—if you’re using Outlook, Excel, or Word, you’re likely a Microsoft shop, which is why you’re using Copilot—the joke is on us. If we weren’t integrating Bing Webmaster Tools, we have missed out on a lot of really rich data.

Christopher Penn – 26:17

Exactly. Now, this is interesting: for eight of the nine topics that it analyzed my site for, it came out with the same location. Basically, humans and AI end up in the same broad categories. The one topic where there was a split is on the LinkedIn algorithm; human searches went to one link, and AI searches went to a different place. That is the one topic where there is a split.

Otherwise, in general, at a topic level, the machines and the humans end up in a similar place. This is important because a lot of people are spending a lot of time right now asking, “Should I spin up a million pages for AI tools to consume and make my website an AI site?”

Christopher Penn – 27:05

No, build a site that everybody wants. What we’re seeing here—at least in my results, which is an n of one—is that it really comes down to whether the content exists to answer the questions.

Katie Robbert – 27:23

Which is not new—call it a golden rule. That’s the way you should have always been approaching building your content.

Christopher Penn – 27:35

From my site, different topics get different kinds of attention. I have a whole section I did on business psychology and business astrology—things like MBTI, Myers-Briggs, and stuff like that—and that content gets a lot of attention from AI. Statistics and probability content gets a lot of attention from AI, and business analytics and KPIs get a lot of attention. Conversely, personality types and tests get attention from humans, career stuff gets attention from humans, and AI tools and platforms do as well.

Again, this mirrors what we know about these audiences going back to the second P in the 5P Framework: people are looking for stuff that people care about. Technology topics are things that are more granular, which a machine is more likely to query in a very detailed way.

Katie Robbert – 28:24

That makes sense.

John Wall – 28:28

It’s interesting that it respects the power curve. That kind of throws me, because I would have expected the models to want to eat up everything. Do you think the idea there is they just want to optimize spidering traffic? Are they just saying, “Hey, we want it to search like a human would, and we don’t consider random data on the outliers worth even scanning?” What’s the take?

Christopher Penn – 28:54

Especially for Google, but also for all the AI tools, they all use some variation of what’s called Query Fan-Out, where a model takes your inquiry—your search, whatever—and basically does a whole bunch of searches on its own. Those searches are all going to be related to each other. Because all the searches are related to each other, they’re likely to return a similar cluster of pieces of information.

For example, if somebody types in, “Copilot, recommend some marketing podcasts to listen to,” it’s going to search for “best marketing podcast for small business,” “best marketing podcasts for this,” and “top marketing podcast in 2026,” and so on and so forth. Marketing over Coffee is going to be in there a lot, but it’s probably going to surface the homepage.

Christopher Penn – 29:38

It’s not going to have to dig very deep because it’s probably going to surface in that Query Fan-Out. In all 80 of those queries, it will pull the Marketing over Coffee homepage, the Marketing over Coffee about page, and the Marketing over Coffee episodes page. That is because it’s a shallow result.

If the user’s intent was, “I want to know specifically what Seth Godin said about marketing in 2007 on podcasts,” that would lead to one specific page. But that’s generally not what people are asking, and that’s not what triggers a search. When people ask, they say, “I need some new podcasts to listen to,” so it’s going to come up with those very shallow hits. That’s how you get your power-law curve.

John Wall – 30:22

Yeah, that’s because it’s based on the human head.

Christopher Penn – 30:26

It’s based on fulfilling human intent. One thing that’s interesting is there is no time lag; AI citations and human searches march in lockstep. This tells me that humans using regular search and humans using AI search are basically asking the same things.

We’ve actually all had this experience where you talk to an AI tool and think, “This is so stupid. I’m just going to use regular search now.” You close the chatbot, go to a regular search engine, and find your answer that way. We know from our own experiences as humans that you may have a conversation with your AI tool and then go and Google it afterward because you want to know something more specific or something the machine didn’t surface.

Christopher Penn – 31:16

So there is no time lag. What this tells us is that you really aren’t optimizing for humans and machines differently; you’re optimizing for the same audience in the same frame of time.

Katie Robbert – 31:32

Okay.

Christopher Penn – 31:33

There is no causality on that timing.

Katie Robbert – 31:35

Right.

Christopher Penn – 31:35

Neither one meets the standard there, so there’s no one creating the other. On my site, there is strong seasonal cyclicality for the days of the week. When people are searching and end up on my site—whether human or machine—it is all on weekdays, and not nearly as much on the weekends.

What is interesting here is that you see the machine stuff taking a slightly greater share Monday through Thursday, and human-led stuff taking share Friday, Saturday, and Sunday. I don’t know why, but that’s the way it splits out, though it’s not a huge difference.

Katie Robbert – 32:19

It’s helpful to know because we tend to think of machines as being always on 24/7. While they are always on, this is a nice reminder that there is a human on the other side of the machine conducting the search. Unscientifically, I can say that’s likely why the day-of-week seasonality is lining up: because I’m the human looking for things, sending out my AI assistant to find the thing.

Christopher Penn – 32:56

Exactly. The machines are not autonomous; they’re not your virtual chief of staff working around the clock. They are just a research tool that you yourself are powering. There is no significant lead or lag for AI citations and human search.

John Wall – 33:15

Right.

Christopher Penn – 33:15

Again, there is no timing issue there. What was interesting is that on my site, humans end up consuming more of my newsletter by a significant amount, along with step-by-step tutorials and thought leadership essays, whereas machines consume more interactive Q&A responses, rants, and educational explainers. Different content types get accessed differently by machines.

There actually are a couple of surprises here. We know pretty clearly for a lot of sites—particularly sites that are doing a good job with their general content—that Q&A content is how machines read to understand what’s going on and to comprehend the intent of a page. Whereas with a human, hopefully someone enjoys my newsletter enough that they actually stick around to read it, and that’s what we see here.

Christopher Penn – 34:16

Finally, are there any reliable traits that predict whether AI will favor a page? No, there are not. We did all the feature engineering around the topics and types of content, and there’s nothing here that is statistically significant. This used Elastic Net logistic regression. There are things that are slightly in favor of one or the other, but there is no smoking gun where we can say, “Yes, we should do more of this.” It just isn’t in the data.

Katie Robbert – 34:53

The slide is like, “Thank you.”

Christopher Penn – 34:58

Buy our stuff. So, that’s what the end result was for my site. Now, I want to reiterate that this was an n of one; that was just my site. What I would encourage folks to do is take the process we used using the 5P Framework by Trust Insights and do this for your own data because it’s going to be different for every site.

Katie Robbert – 35:22

So the executive summary for your site is essentially—if I followed correctly—that you generally don’t need to do anything different when you’re breaking it down between human and machine. For the most part, they’re consuming things roughly the same way. Where they diverged was on more discrete topics, such as careers and personality tests versus business statistics. It really depends on what you want your site to be known for and how you want to cater to your audience.

Overall, you don’t have to change your timing or how you approach your content. But it strikes me that you should still be following frameworks like Hero, Hub, Help, and Entertain, Engage, and Educate as you’re creating content because AI and humans are consuming it the same way.

Christopher Penn – 36:22

In Claude’s console, it said, “If you want more attention from machines, do more analytically flavored and evidence-based studies.” We were actually having a chat with one of our friends this week about that; if you want machines to come by more often, produce more content like that. If you want humans to come by more often, do more of the super-long newsletter issues and pieces that seem to attract people rather than machines, at least on a per-page basis.

Katie Robbert – 36:56

Our friend Brian says, “Fantastic use of Search Console data. Do you export directly from GSC or go through a Looker Studio report? I’ve had issues getting all the data for a large site from GSC.”

Christopher Penn – 37:10

This was Bing Webmaster Tools, not Google Search Console. However, with Google Search Console, you do not want to use the web interface. Google Search Console has an API, and if you get an API key from Google, you can wire up code that will extract that data programmatically.

We do this for one of our clients with a large site, and every month the exported spreadsheet is 500 megabytes of data. In terms of text, that is a gigantic spreadsheet with 18 million rows of data. The reason it’s so big is because every row contains different variables: device type, URL, number of clicks, search query, and things like that, giving you a very fine, granular data dump.

Christopher Penn – 37:59

It is the best source of Google Search Console data because it puts the query and the URL on the same line, whereas the web interface does not do that. You can do some really cool analysis, but you have to go through the API for it. The good news is the API rate limits are very generous.

What I would suggest doing is using a coding tool like OpenAI Codex, Google Anti-Gravity, or Claude Code. Give it your API key and say, “Here’s the site I want you to extract. Just get me whatever is available in the API.” Give it the documentation and tell it, “If it’s there, download it.”

Christopher Penn – 38:36

Then, stuff it in a SQLite database locally and have it come up with a combination primary key using the query, URL, clicks, and impressions. That, along with the date, will help create an index across the database so that you can load new data to it without blowing up the existing database. The key word to give your coding tool is “idempotent.”

Katie Robbert – 39:09

If none of that made sense, you can reach out to TrustInsights.ai/contact and talk to John, who will try to make sense of it. In a nutshell, it’s work that can be done and analysis that you can do.

Number one, set up Bing Webmaster Tools. Take a look at your Google Search Console to see if it has that AI search component in it—it may or may not, as we have no control over when that happens. Then, export both the human search and the AI search, and follow along with the steps we talked through in this particular episode.

Christopher Penn – 39:48

Yep, and use the 5P Framework.

Katie Robbert – 39:53

Use the 5P Framework, because with the Trust Insights 5P Framework, you’ll live it, you’ll love it, and you’ll laugh about it. It’ll be great.

Christopher Penn – 40:07

Here is one other caution: you might not have enough data to work with. When I went to do this for Marketing over Coffee, for one reason or another, it had not gotten a lot of data in Bing specifically. I think that may have been because we’ve been laughing at it for 20 years.

John Wall – 40:28

Somebody over there may have heard about it. Exactly.

Christopher Penn – 40:32

There wasn’t enough data to work with there. When Google Search Console gives AI visibility data for Marketing over Coffee, we’ll be able to do it there, and the same goes for Trust Insights.

But that’s what we found. Again, make sure you’re using apples-to-apples data. Be careful to ensure that the data matches for the timeframe and things like that. Don’t mix systems; you cannot use Google Search Console data with Bing Webmaster Tools data. They are different systems and measure different things, in the same way that you shouldn’t conflate TikTok and Instagram. They are totally different systems with totally different signals, so don’t do that. Any final thoughts, comments, questions, or barbecue recipes?

Katie Robbert – 41:22

I think the big takeaway is that people are stressing about their traffic going to AI. They wonder, “What should we do about it? What’s our strategy? Our organic search is going down—is it because of AI?” Just create good content, period.

Create useful content. Follow E-E-A-T, follow Hero, Hub, Help, and follow the three Es. Find a framework you like and follow it, but just create good, useful content that’s not just pontification.

Christopher Penn – 41:58

Or maybe it is. Maybe that’s your thing.

Katie Robbert – 42:01

Maybe that’s your thing, but then don’t get mad if the bots don’t find it helpful.

John Wall – 42:07

You have all these dead ends. Over the past couple of weeks, I’ve been looking at all this stuff regarding the volume being 10x or 100x. But the punchline this week is you don’t have to change the map; you don’t need to be doing anything differently as far as the strategy for your website.

Christopher Penn – 42:26

One other thing that’s worth pointing out is that you can’t measure—at least not to the same level—what is going on with third-party platforms. If your podcast is hosted on Libsyn, for example, and you don’t have a website there, you’re not gonna be able to do this kind of analysis. If your newsletter is hosted on Substack and you can’t install Webmaster Tools for your Substack, you can’t do this analysis there either. We’ve said for a while: don’t build on rented land. Now you’re starting to see why.

As you saw, it’s also very difficult to engineer features out of the limited data you’re given. For years, people said, “Don’t put the date in the URL of your blog post because it’ll date them and make them look old.” I get that.

Christopher Penn – 43:21

But when you go to do data analysis and the only thing you have is a URL, you can no longer analyze timing without keeping a separate lookup table of when you published what. If it’s in the URL, it’s a lot easier.

Katie Robbert – 43:33

It’s true.

Christopher Penn – 43:36

All right, folks, that’s going to do it for this week. Stay tuned for next week—I’m not sure what we’re doing yet, but I’m sure it will be fun. Thanks for tuning in, and we’ll talk to you on the next one.

Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it. For more resources and to learn more, check out the Trust Insights podcast at TrustInsights.ai/tipodcast and our weekly email newsletter at TrustInsights.ai/newsletter. If you have questions about what you saw in today’s episode, join our free Analytics for Marketers Slack group at TrustInsights.ai/analyticsformarketers. See you next time.