So What? Setting Up AI Bot Tracking in Google Analytics

Katie Robbert – 00:37

Well, hey everyone. Happy Thursday. Welcome to So What, the Marketing Analytics and Insights live show. I’m joined by Chris and John. Howdy, fellas.

John Wall – 00:44

Hello.

Katie Robbert – 00:45

There we go. This week we are covering setting up AI bot tracking in Google Analytics, which is a continuation of what we talked about on last week’s live stream. You can catch that on the TrustInsights.ai YouTube channel—go to the So What live stream playlist. We talked about how to measure it.

Chris was showing four parts of a measurement plan for looking at your AI bot tracking, and one of those pieces was specifically Google Analytics. There is a lot more complexity in what people can and can’t do with these things. If you want to learn more, you can go to trustinsights.ai/aibotanalytics. That’s AI bot with a ‘T’ analytics.

Katie Robbert – 01:37

Today, we’re actually going to walk through, at a high level, a little bit of pre-baked stuff. As Chris mentioned, this isn’t something that you can just decide to do if you have a spare hour—it’s going to take a little bit more than that. There are a lot of considerations. We are going to walk through it, and we’re going to use my personal site, which is really just a test bed for Trust Insights. So, Chris, what else do we need to know before we get started?

Christopher Penn – 02:07

Before you get started, depending on the size of your site and how much AI bots visit, this may cost money. Cloudflare Workers is the underlying mechanism that this uses, and that is fancy for saying you’re setting up a server on somebody else’s architecture. You get 100,000 hits a day for free, and then you have to pay.

That sounds like a lot, but it’s not a lot because AI bots can be 20x to 100x the number of humans that visit your site, based on how much content you have. That is our big caution here. If you are on the free plan, pay very careful attention to your notifications and expect at some point to get a bill.

Katie Robbert – 02:55

To be fully transparent, this happened to us at TrustInsights.ai. Our website is a decent-sized website, but in terms of traffic, we sort of thought the bot traffic must not be that much—100,000 hits. Can you walk through, at a high level, what we experienced this week?

Christopher Penn – 03:17

Yeah, we hit those limits in seven hours.

Katie Robbert – 03:21

And what was the result of that?

Christopher Penn – 03:24

We got a temporary notification on Cloudflare saying, “Hey, your site has been rate limited. Please insert a credit card to continue.” I’m like, “Oh, well, crap, I guess we’re turning this feature off.”

Katie Robbert – 03:38

I think it’s important to note that it might feel like 100,000 is a lot, but we hit it in seven hours, as Chris mentioned. It is something to be aware of in terms of your setup, especially if you have it set to automatically roll over to payments.

Christopher Penn – 03:56

Yeah, which again, gets expensive.

Katie Robbert – 03:59

Yeah.

Christopher Penn – 03:59

All right, so there are three steps to this process. First, we have to configure Google Analytics. Then, you have to configure the Cloudflare worker, and then you have to configure your reporting. As Katie mentioned, we will help you do this if you would like to hire us to do it.

At a high level, the first thing you have to do is create a new property because, by default, Google Analytics does not allow bots. They try to screen them out because they want to know who the humans are visiting your site. In fact, if you’re still using sessions as a metric in Google Analytics, stop, because it does allow some measurement bots to get through.

Christopher Penn – 04:44

You want to use engaged sessions for regular Google Analytics, but that’s a different show. For this, we’re going to set up a specific property that is just for bots. We can’t use the regular Google Analytics tracking code because it blocks them out. We’re going to call this “Katie’s AI Bot Friends”—that’s the account we’re going to go with.

We’re going to go through and set it up. Let’s call this a business and industrial site, a one-person site, and our goal is to understand web traffic and look at retention. This is just a stock setup for a property. Next, we have to create a data stream. This is katierobbert.com, and this is Katie’s site. We hit “Create and continue,” and now we hit “Next.”

Christopher Penn – 05:32

Now it’s going to say, “Hey, do you want to set up a Google tag?” Absolutely not. We do not want to do this because we don’t want to mix the human traffic from Katie’s site in with the bot account. It has to be just the bot account.

We’re going to take note of the measurement ID. I’m going to have a little notepad separately, and then we have to create the integration to the Measurement Protocol. I’m going to drag this off-screen because we don’t want that to be visible here. While I do this, you and John should tell some funny stories.

Katie Robbert – 06:05

John, when we were getting set up for this and talking about it, we knew that there was going to be some time that Chris was going to have to spend setting things up. There’s only so much that he could have done ahead of time. I think it’s a really good demonstration that it’s not just setting up an additional Google Analytics property—there are a lot of additional steps that go into it.

We were joking that you and I should have our shtick and have some stories to tell. In terms of what you’re seeing because you view what’s going on in the market and with AI from a different lens, what’s your perception of people struggling with bot versus real traffic?

John Wall – 06:52

The first one right out of the gate is Google pulling an Adobe here by wanting to survey Chris on how much the product is working for him while he’s in the middle of doing something really important. I could just go on for 25 minutes about how it always seems like those things pop up. Here we are in a webinar; that could have fired this morning when he was just routing around through GA, but no, it had to happen right now. So, there’s that. Nice one, Adobe. Even though it’s a Google problem, I’m throwing Adobe under the bus for that.

Katie Robbert – 07:22

Fair enough.

John Wall – 07:23

Because they’re the champs of it. Every time I fire up Acrobat or Photoshop, I get hit with that thing. As far as the bot traffic, this is a big, ugly mess. I know you and I are both on the same page. Chris is like, “Hey, I’m just turning it on. It’s easy. We’re getting this rolling,” and we’re like, “No, you’re going through 65 screens that have all changed in the past two years.” We haven’t seen any of these alleyways in GA, so it is great to be getting a new tour of the latest and greatest with this stuff.

Katie Robbert – 07:51

If I were being held accountable for website traffic, it’s a big conversation that a lot of marketers are having right now with leadership. It’s a struggle because it could look like a lot of things—it could look like your traffic is up, or it could look like your traffic is down. The thing that I keep hearing is, “Oh, it must be AI.” But that is such a vague statement and such a blanket blame. Being able to do something like this is helpful to understand what that really means.

Christopher Penn – 08:36

Yep. We also have to create the dimensions because bots are not people. They don’t have things like devices or browsers. Cloudflare provides nine different categories of data in addition to events. They tell us the bot behavior: what kind of bot it is and the category of behavior. Remember from last week’s show, is it a crawler, a trainer, a search, or an assistant?

If you are on the enterprise plan, you’ll also get a bot score, which is a confidence score; if you’re not, you don’t get that. The IP ASN is a numeric identifier. Cloudflare has a catalog of about 100,000 different bots, and each bot has its own unique identifier, which, for our reporting, we don’t care about.

Christopher Penn – 09:25

There are two fingerprints, which are encryption fingerprints that basically give you an MD5 hash of the payload without having to look at it. We don’t care about that and we don’t need it. Request path is one of the most important fields—that is what URL the bot hit. User agent tells you who the bot is—is it GPTBot, is it Claude, etc.? Then, is it a verified bot? That’s another Cloudflare control field. Those are the custom definitions.

The last thing we need to do is create a BigQuery link to link it to a BigQuery database.

Christopher Penn – 10:08

BigQuery stores data for Google Analytics, and because what’s coming in from bots is event-level, Google Analytics—depending on your settings for your property—will only store it for between two and 14 months. If you want to do a year-over-year analysis after the first year and you don’t have this set up, that data is gone. Now, Google Analytics is set up and ready to go.

Katie Robbert – 10:37

All right, a little bit of context in case you missed the live stream last week, which again, you can get on the TrustInsights.ai YouTube channel playlist. When we talk about bot traffic—and we sort of tongue-in-cheek called it “Katie’s AI Bot Friends”—we determined on last week’s episode that we’re talking about these agentic AI assistants.

When you say, “Is it ChatGPT? Is it Gemini? Is it Claude?” what’s happening is I, the human, am typing instructions to Claude or Gemini saying, “I want to find out more about Trust Insights.” It’s saying, “Okay, cool, let me send out my team. I’m going to go find that information and come back to you.” Instead of getting Katie Robbert hitting Trust Insights, we’re getting…

Katie Robbert – 11:33

We’re not getting me specifically; I’m now anonymized. We’re getting a record that someone asked Gemini to go find this information. That’s the kind of data that we’re trying to figure out.

John Wall – 11:48

That’s right.

Christopher Penn – 11:48

You are getting the intermediaries, which is an increasingly large percentage of every search, thanks to Google’s AI Overviews, etc. Believe it or not, this was the easy part. Now comes the extremely painful, hard part.

You need to have a Cloudflare account. It can be a free one, but you need to have it set up so that you have permissions to make changes to it. Katie’s got a Cloudflare account already set up; we set it up when we configured her website. That is a prerequisite, so we’re going to skip past all that and configure a server, because that’s what we’re doing. We have to build the server in Cloudflare.

Christopher Penn – 12:36

The best way to do that is to not use the Cloudflare interface at all, which seems counterintuitive, but here we are. Instead, we’re going to go to the command line and create a Cloudflare local server configuration. We’re going to call this “Katie Bot Telemetry.” This is going to go to Cloudflare’s Node package, and it’s going to ask me some questions like, “What are we doing?”

I want to do a “Hello World” example. We’re going to use a worker-only setup because we’re only making a worker server here, which is a little server on their services. What language are we going to use? We’re going to use JavaScript because that’s what Google Analytics speaks very well. It’s now going to create an entire system setup here with source code files.

Christopher Penn – 13:31

I’m going to put the index file in here. Now comes the next difficult part, which is configuring the JSON system configuration for Katie’s website. We’re going to add a route in here at the end of this file—actually, before observability. This is going to have www.katierobbert.com, and the zone name is from your Cloudflare account, katierobbert.com. Now we’ve got the configuration file for this Cloudflare worker. Do I want to add an agent file? No, I don’t want to add an agents file.

Katie Robbert – 14:15

This is a good reminder that we can do this work for you. Chris, to your point, it doesn’t stop at just setting up a Google Analytics property, which a lot of people can do. To actually get the data correctly, we need to be doing all these other things.

Last week, we asked the question, “Does it have to be Cloudflare as the CDN specifically, because there are others out there?” You said at the time, “Yes,” because of the capabilities of what Cloudflare specifically can do in this context. Can you say a little bit more about that and what some of the alternatives that people are using might be?

Christopher Penn – 15:00

Essentially, what we’re trying to do is get ahead of our Google Analytics. Like I said, if you use the regular Google Analytics tracking code, it knocks away all the bots. It says, “I don’t want you here; I only want to track real people.”

Cloudflare is a CDN, or Content Delivery Network. That means it sits in front of your website and domain name, depending on how you set it up. What it does is say, “Okay, here comes a request.” Cloudflare looks at it and asks, “Is this something that I handle?” The worker evaluates it, and if the worker says, “No, I don’t handle that,” it passes it onto the website.

Christopher Penn – 15:44

If it’s not a bot—if it’s a human—Cloudflare says, “You know what, go to the Trust Insights website. Human, you’re okay to go over here.” If the worker says, “Hey, you’re a bot,” it then passes it into its workflow and asks, “What kind of bot are you? Are you an AI crawler? Are you one of these types?”

Great. I’m going to now ring up Google Analytics through its API and say, “Hey, Google Analytics, I got a bot for you.” If it doesn’t recognize it, it just says, “Okay, get out of here.” Once it does that, it goes to Google Analytics and says, “Here is the bot information I got for you.” Google Analytics records it, throws it to BigQuery, and then we can report on it.

Christopher Penn – 16:21

That’s the workflow of what’s happening from a technological perspective here.

Katie Robbert – 16:25

When we talk about it with our clients, we really call Cloudflare or the CDN the gatekeeper. To put it in human terms, John Wall is the Cloudflare of Trust Insights. If you want to talk to anyone at Trust Insights, you have to get through the Wall.

It is a really interesting way to think about it because you have that one person who’s vetting everything coming in, saying, “This is a legit deal, we should explore it,” or, “This is spam, we’re going to ignore it.” It’s essentially the same thing as having a head of business development who vets everything coming into the CRM before it gets to anyone else. Your CDN is saying, “Is this trash? Is this useful? Is this spam?”

Katie Robbert – 17:11

It filters it and doesn’t even bother the website yet.

John Wall – 17:16

Exactly. From that first trigger where it starts to write GA data, does the bot just get access and get let through? Or does it have to go through the whole chain before the bot can actually get to whatever content it was looking for?

Christopher Penn – 17:30

It has to go through the chain and pass, because a lot of bots get bounced away.

John Wall – 17:36

Right. So, it’s got to make it to the end of that stack to actually access content.

Christopher Penn – 17:41

Exactly. If it doesn’t pass, then it gets sent packing. There are a lot of very hostile crawlers—even things that are malicious, like bots trying to inject code into your website. Cloudflare just tells them to get out of here.

Our next step after this point is to modify the configuration. I’m just going to double-check to make sure the configuration is correct. It is. The routes are installed, the compatibility flags are installed, and the bindings are set. The only other part I have to do here is register the keys. Remember, we got Katie’s API key and measurement property key for her website from Google Analytics.

Christopher Penn – 18:36

Cloudflare has to be told what those are, but you can’t put them in the configuration file. It’s a best practice to never put configuration data, particularly valuable stuff, into a text file that you upload to a server. Instead, you want to put it in a Secrets file, and Cloudflare provides a mechanism for doing this. While I do that, you guys talk.

Katie Robbert – 18:55

I actually have a question for you, John, about bot traffic while Chris is doing that. For those who are new here and don’t know, John is the host, along with Chris, of Marketing Over Coffee. How many years now has this podcast been going?

John Wall – 19:18

It’ll be 20 years in 2027.

Katie Robbert – 19:20

Oh my goodness, you guys better do something; I’m going to be mad if you don’t. In terms of podcasts, have you seen an inflation of bots skewing your listener numbers? We’re talking about a website, and I know you have marketingovercoffee.com, which people should visit to understand why they should be listening to Marketing Over Coffee. If you’re not, you should be listening to it. Have you seen this same issue inflating listener numbers or podcast download numbers?

John Wall – 19:59

No. That’s one thing that has changed a lot in the past five or six years. There are a number of competing bodies that monitor podcast stats, and all of them are militant about chopping out all bot traffic. It’s actually been the opposite; we have a lot of situations where human traffic gets discarded and not considered part of actual traffic.

Unfortunately, I wish we had the problem where the numbers look great and maybe there were some bots in there. Instead, the infrastructure and cruft around podcast analytics have really choked it off to the point where the problem we have is…

Christopher Penn – 20:42

The network.

John Wall – 20:45

…will be doing stats ticking. For example, with this one gate that we have for a bot, the podcast will actually have three or four of them stacked up. The problem is, once it makes it through the first or second bot gate, you’ll have the third or fourth gate say, ‘Oh yeah, we’re going to ignore all the traffic that came from those previous gates. We’re going to consider that non-human traffic.’

It’s insane. I can literally see a window of anything from 12,000 down to 30 downloads for an episode, based on how those things end up being stacked up.

Katie Robbert – 21:22

That must be so frustrating as someone who sounds like they have no control over those gates.

John Wall – 21:28

Exactly. We ultimately always fall back on telling sponsors, “You have to tell us how many leads you got last month. Tell us if you’re getting the traffic.” It’s no longer about us just selling by the thousand, because that isn’t doing anybody any good if we can’t prove much of anything.

Katie Robbert – 21:47

Oh, you’re talking about the people who sponsor Marketing Over Coffee. That’s a big consideration. There are a lot of websites that have a sponsorship basis, so this is a consideration in terms of the type of metrics you’re actually able to report on.

How inflated is it? Is it true traffic? Is it bots, spam, or human? The bots don’t care about the ads—they’re not going to buy anything yet. You want to know that a human is seeing the ads, so I can see where this would be a challenge for someone selling sponsorship on a website as well.

John Wall – 22:31

Yeah, and at the other end, there’s a gigantic bucket of bots coming around. Like you said, maybe they’re not buying today, but we already have situations where bots can make purchases.

There’s also GEO as part of this. Having these bots is going to get you to show up in other places, which we know is going to be the future of the web and how information is shared. Getting into agents, getting into software tools via WebMCP—this is all how the future of the web is going to look.

Katie Robbert – 23:07

Have we filled enough airspace, Chris?

Christopher Penn – 23:11

Yes. In the time while you were talking, we authenticated to your website and passed your API keys over. Effectively, we’ve built a small server here on my computer, and then that gets pushed over to Cloudflare. You can see that Cloudflare now has that particular bot installed. The server exists and it is ready to begin listening to what we have to say.

The last part we have to do now is go to katierobbert.com, go to the worker routes, and make sure that it is plugged in. I’m going to edit this to say anything from katierobbert.com should be sent to that bot. The Geobot Telemetry is there and looks good. Let’s check our workers. There’s our worker.

Christopher Penn – 24:03

Logs are enabled, the domain is there, and it is enabled. Once this server spins up and starts working, every time a bot comes in—like we were talking about in the workflow—it’s going to see those logs. It will say, “I know what to do with these logs,” and send them to Katie’s Google Analytics 4 property. Now the wiring is all done.

Katie Robbert – 24:33

Amazing. Bing, bang, boom, done.

John Wall – 24:38

This has blown me away because I didn’t realize there was a full dev and server environment as part of Cloudflare. Has this grown out of so many people using Cloudflare just for managing raw traffic when they get overwhelmed? Are those tools normally there so you can build out what happens when things start to catch fire, prioritizing traffic and things like that?

Christopher Penn – 25:04

Cloudflare has expanded substantially. There is all kinds of crazy stuff that you can do. They have their own AI platforms where you can deploy models, workers, GPUs, and everything. They’ve gone from, “Hey, let’s manage your website traffic,” to, “Hey, if you want to do anything with AI and you don’t have the hardware for it, use us.”

Let’s say I wanted to build an AI chatbot quiz on the Trust Insights website. I could stand up a language model on a Cloudflare server, connect it to our website, and it would serve up the AI. The moment you turn that on, the meter would start spinning like the electric meter during Ghostbusters, but it is possible to do that. They’ve added this whole AWS-style compute infrastructure to your CDN.

Christopher Penn – 26:06

It’s really, A, very impressive, and B, there are so many more things that you can do with it now.

Katie Robbert – 26:06

Even if you just want a gut check on whether you have things set up correctly so you’re not about to get a $10,000 bill, just go to trustinsights.ai/aibotanalytics. We can certainly work out something to take a look at your configuration. You may have the skills in all of these different systems, but you might just want a gut check to make sure you’re not about to cost yourself a heck of a lot of money.

Christopher Penn – 26:36

One of the things that you should do on your Cloudflare account is set billing alerts. For any cloud-based service, you should set up billing alerts that say, “Hey, here’s the ceiling, and notify me when it exceeds the ceiling.”

The best-case scenario—which not a lot of companies use—is that it just cuts you off, alerts you, and stops spending above that point. Google is notorious for doing the opposite. Google will send you alerts but keep the meter running.

Christopher Penn – 27:07

You’re left saying, “Crap, it took me an hour to get to my computer, and I got $8,000 worth of compute in that hour.” Definitely make sure that you have alerts set up in your systems to warn you when you’re about to spend a whole bunch of money. The last part of this setup is—and look, there we see an AI bot has already visited Katie’s website.

Katie Robbert – 27:32

Yay!

Christopher Penn – 27:34

We actually had this experience with the Trust Insights live stream this week from our WebMCP episode. I caused some damage to the Trust Insights website this week because the plugin we made for WebMCP was too secure. I made it so secure that it was intercepting Google Analytics and saying, “You’re not running from a trustinsights.ai domain, so I’m not going to let you run.” Which could have gone better.

Katie Robbert – 28:04

We definitely saw that in our data. We looked at our data set and said, “Hey, something’s wrong, we should probably take a look.” Chris, to your credit, you did, and you were able to troubleshoot what was happening. This is exactly why you want to consult with someone who knows what they’re talking about.

Yes, we missed a couple of weeks of traffic, but it could have been a lot more devastating depending on how you had put WebMCP together. You could have vibe-coded it, included a lot of privacy holes, or we could have gotten a lot of unfriendly traffic to our site. There is a lot that could have gone wrong.

Christopher Penn – 28:54

Yeah, a piece of software that was too secure is one of the less bad problems to have.

Katie Robbert – 29:07

It’s preferable to the other end of the spectrum, for sure.

Christopher Penn – 29:13

We pull up the real-time view just to make sure the telemetry is there. The last step would be creating a dashboard to look at this data, and for that, you’d probably want to use good old-fashioned Data Studio. Data Studio is the preferred, easy way to get at things. Let’s pick Katie’s website—there’s “Katie’s AI Bot Friends.”

What’s going to be different about this is that all the standard Google fields are going to be empty, so the custom dimensions and metrics we put in are the only ones you can use. I might throw in a date control here, and then I might toss in a table with some heatmap features.

Christopher Penn – 29:58

In this table, we would have things like our bot category. Instead of views, we have to use events because that’s the only measure we’re going to have. There’s going to be no data for this right now because, obviously, there is a 24-hour delay, but you’ll be able to see what the bots have been doing.

I put in the request path here, and tomorrow, Katie, we should be able to come into this dashboard and see what bots are hitting what pages on the website and how many times each. You can make charts, graphs, and all these things in Data Studio, and it will give you a sense of where the bots are spending their time.

Katie Robbert – 30:48

I feel like I keep emphasizing it, but we all saw how quickly you just put all of this together and set up a Data Studio dashboard. Looking at that, the dashboard alone would take me an hour just to remember where all the things are. I want to make sure people understand that Chris is expertly proficient in these things, so take the time it takes him to set that up with a grain of salt.

Unless you are someone who is as well-versed in all of these systems as Chris, it’s going to take some time. It’s not going to be a case of, “I have an hour, let me set it up.” It’s going to be maybe a few days.

Christopher Penn – 31:38

It’s interesting. Even just looking here now, you can see you’ve already had four bots. Your site, like you said, is not a huge site—it only has five pages on it.

Katie Robbert – 31:50

Five pages.

Christopher Penn – 31:51

There are no humans other than me on it at the moment, and yet we’ve already had four bots pop by and consume some resources. Even a very small site like Katie’s isn’t going to hit the 100,000 events in a day mark, but for any site that is anything other than a placeholder, this is going to add up real fast.

Katie Robbert – 32:13

To your point, I might not hit it in seven hours like we did for Trust Insights, but I could very well hit it by the end of the weekend.

Christopher Penn – 32:26

With AI bots in particular, it’s not a question of how many humans visit your site; it is how much content you have. If I were to turn this on for my personal website, which has about 11,000 or 12,000 pages going back to 2007, AI bots love it because there’s a lot for them to grab and train on. My Cloudflare Workers account, if I set it up, would be done in probably an hour and a half because so many bots are hitting so many pages so often.

Katie Robbert – 33:00

That’s a huge consideration for the size of your site as well. Within Cloudflare—or within this setup—can you restrict what it’s looking at? Can you say, ‘I only want to look at bots that hit the homepage of katierobbert.com’? If you have a large site, could you segment it down and say, ‘I only care about bots hitting pages where I have something for sale’?

Christopher Penn – 33:35

The answer to that is “yes-ish.” The reason I say “yes-ish” is because you’re going to need to pop open your coding tool of choice. There is an index file that we created that contains the actual code for what it tracks, and I could make a modification to this. Instead of tracking every page, we could have it just track the top-level pages or first-level pages and nothing beyond that.

This script also has a bunch of different crawlers that I custom-built for these specific things. You could also say, “I don’t want to track certain bots.” Maybe you don’t care about Cohere—you can throw it out. Yes, you could do that, but I would have to modify the script to do it.

Katie Robbert – 34:26

That’s all custom work that someone who knows what they’re doing, like Trust Insights, could do for you. I think about some of our clients who have very large websites that are broken down into sections that are literally different cost centers and business units. I would imagine setting up something like this for each of those cost centers versus the site as a whole.

On a very large enterprise-sized site, it’s going to hit the maximum bot limits in maybe 30 minutes. It’s not going to be helpful if it’s not broken down by which section of the site is getting the most bot traffic. You may find that your blog gets all the bot traffic versus your About Me page, as an example.

Christopher Penn – 35:19

Yep, you could do that because it’s just JavaScript and I have the routing set to go to the main Google Analytics Measurement Protocol account and the API key that we set up. There’s nothing stopping me from supporting five or 10 different Measurement Protocol APIs.

If you had different Google Analytics accounts or properties for different lines of business, I could modify the script to route it. If it’s this section of the website, send it to this Measurement Protocol API. If it’s your widgets section, go to the widgets API. If it is the podcasts section, go to that API, or if it’s the left-handed smoke shifters, send it to this API.

Christopher Penn – 36:03

You could modify it that way, or you could modify it to go straight to a database. Let’s say you’re at a company that has a data lake and you have something set up like Snowflake. As long as Snowflake has a publicly exposed endpoint, you could send me the API key and I could rig up the script to send data to Snowflake instead of Google Analytics.

We have it set up with Google Analytics here because that’s what most marketers are used to. But if you have a data science team and they want the raw goods themselves, we can absolutely point the script to any database.

Katie Robbert – 36:39

Okay. I think that’s also helpful context because, as much as Google Analytics is quote-unquote user-friendly, I can see where people want to go even deeper.

Christopher Penn – 36:50

Yep. Or if you want to use it with an Adobe Analytics account using Adobe’s measurement API, we can certainly rewire it for that if you’re an Adobe shop. The one non-negotiable part here is Cloudflare up front, because it has to intercept the bots before it routes them.

Katie Robbert – 37:09

Sure, that makes sense. John, questions? Comments?

John Wall – 37:16

This is one of those things where we just have to sit back and see what comes in. Over the next couple of months, we’ll be trying to figure out who’s hitting us, what they are looking for, and what’s working or not working. It’s opening up a new Pandora’s box.

Katie Robbert – 37:29

It really is.

John Wall – 37:32

Yep.

Christopher Penn – 37:32

One of the first analyses I would suggest people do if they set this up is a set of comparisons. Last week, we told you about Bing Webmaster Tools and the pages that you can get out of that. We also told you about Google Search Console and the pages you’re able to get out of that. This setup will give you the pages the bots are hitting by bot type.

You should grab all three data sets—maybe also including your regular Google Analytics data for the pages your humans hit the most—and ask your favorite coding tool, ‘Hey, what do all four of these data sets have in common?’ Which pages are popular across the different surfaces? What are the pages that are popular only in one or two domains?

Christopher Penn – 38:13

What pages are popular with the AI bots but not with the humans? That delta can then tell you, ‘Huh, I wonder why?’ When people talk to AI, they are getting back the most probable answers, so the humans who are actually on your site are going to behave differently than the machines.

This is a way for you to compare and contrast those two data sets to see if the machines behave differently on your site or with your data than humans do. If so, and if machines are 20x your human traffic, you might start optimizing your site differently for the machines.

Katie Robbert – 38:53

It makes sense. It really depends on what your goals are. We haven’t had a chance to do it yet, but it’s a nice plug for the 5P Framework by Trust Insights, which you can get at trustinsights.ai/5p-framework. What is your goal in understanding this traffic in the first place? That is a really great place to start before instituting all of these different protocols.

As Chris just showed, he went through it quickly because we have a certain amount of time for the live stream, but it’s going to take a bit to get it set up correctly. What’s your goal for doing it in the first place? What is the question you’re trying to answer with this data? Next are the people: who needs this data, and who is going to continually use it?

Katie Robbert – 39:41

Is it a one-and-done? That may not be enough of a justification or business case to go through all of these hoops to set this up. Who is benefiting from having this data?

For your process, Chris just showed what that looks like at a very high level, and it’s not an easy process. For the platform, we just showed that there is Google Analytics, Cloudflare, BigQuery, and you may have some coding or a terminal setup. There are at least four or five different platforms.

Christopher Penn – 40:15

Yeah, you need Node installed on your machine.

Katie Robbert – 40:17

…machine, you need Node. Then there is performance: did you answer the question that you were after by setting this up in the first place? Did you find out what percentage of traffic to your site is bot traffic?

The question really is, once you have that data, what are you going to do with it? If it’s just nice to know, then it’s really not worth setting up because it’s a lot of work. But if you’re going to do something about it—as Chris mentioned, if you’re going to take that delta and think about a different way to optimize either for humans or for AI—then it is worth it. That’s when I would really recommend reaching out, and we can help you do that.

Christopher Penn – 40:57

Yeah, like the literal title of the live stream: So What? What are you going to do with the data?

Katie Robbert – 41:03

Yeah, because there are a lot of really cool, shiny-object ways to get data now that people are interested in. But that’s just new duct tape on old problems. You have your zebra-print duct tape, your pink duct tape, and your blue duct tape, but at the end of the day, it’s still just duct tape if you’re not telling me what you’re doing with any of this data.

There are a bunch of new ways to collect data and look at the information, but if you don’t have a plan for what you’re doing with it, why do it in the first place?

Christopher Penn – 41:44

Katie, 27 bots have come to your website now.

Katie Robbert – 41:49

I feel popular.

Christopher Penn – 41:52

The robots are everywhere.

Katie Robbert – 41:55

But for a website that, number one, barely anyone knows about, and number two, is only five pages, that’s a lot.

Christopher Penn – 42:05

Yeah, that’s a lot in what, 12 minutes?

Katie Robbert – 42:08

Yeah.

Christopher Penn – 42:08

We set this up 12 minutes ago. We turned it on 12 minutes ago.

Katie Robbert – 42:12

Yeah, that’s a big deal.

Christopher Penn – 42:15

It raises a very interesting question, and this is a strategy question—maybe something we should debate on the podcast sometime: what happens when your largest audience isn’t human?

Katie Robbert – 42:28

To go back to—not to belabor it, but to belabor it—the 5P Framework, that’s a great question to be asking. Then you run through this sequence again: what do we do if the majority of our audience is not human? What do we do about it? I have some thoughts, but I would want to go through a few different business cases before I actually start taking action, because I don’t want to alienate the humans that I do have.

Christopher Penn – 42:58

Right, because for now, humans are still the ones buying things. It’s one of those things that, again, the 5P Framework helps solve. A lot of people are spending a lot of time trying to get good at GEO. Once you’re good at GEO, you have more bots than humans. Now what?

Katie Robbert – 43:19

If you want to learn how to get good at GEO, we have a course for that—go to trustinsights.ai/geo101. If you take that and you’re ready for the next thing, we have a GEO 201 course as well in our academy at academy.trustinsights.ai. We’ve got all the stuff for you, and we’re going to help you out.

Christopher Penn – 43:39

Exactly. All right, Katie, your bot family is continuing to grow like a Tamagotchi.

Katie Robbert – 43:48

I understand that reference.

Christopher Penn – 43:53

I tried setting this up on Marketing Over Coffee, and it has 19 years of data. It hit its limit in 90 minutes; it was just done.

Katie Robbert – 44:07

Good news, bad news, John.

John Wall – 44:09

No surprise—yet another dumpster fire over at the Marketing Over Coffee website. That’s just the price.

Christopher Penn – 44:15

No, it’s the popularity with the bots. They are our number one fans.

John Wall – 44:20

It’s bot crazy.

John Wall – 44:22

We’ve talked about this in previous casts, but the big idea I’m really coming around to is that very soon there will be a standard for what your AI website needs to look like. How much overlap there is is another thing entirely, but basically, what your AI website looks like is going to be a major topic. We’re going to be talking more and more about that in the coming months.

Katie Robbert – 44:44

For sure.

Christopher Penn – 44:45

That’s a good idea. We could build that because I keep meaning to build the Savory Oatmeal Company website—I just haven’t gotten around to it yet, but we own the domain name.

Katie Robbert – 44:57

That’s for another day.

Christopher Penn – 45:00

That’s going to do it for this episode of the live stream. Thanks for tuning in, folks, and we will see you all on the next one. Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it.

For more resources and to learn more, check out the Trust Insights podcast at trustinsights.ai/tipodcast and our weekly email newsletter at trustinsights.ai/newsletter. If you have questions about what you saw in today’s episode, join our free Analytics for Marketers Slack group at trustinsights.ai/analyticsformarketers. See you next time.