Crawl Budget vs. Your Budget

What’s your time worth? I know what mine’s worth which is why I get frustrated when I waste countless hours trawling through Google Analytics data trying to find the source of anomalies. You can waste precious time on pointless Google issues that should be spent growing your business, which is why I say, “Get stuffed Google!”

What I learned

• Google data issues can send you down the rabbit hole
• Asking Google doesn’t always give you the answer
• Why weird traffic surges could cost you money
• It’s not a crawl budget; it’s your budget
• Time to say, “Get stuffed Google!”

Transcript

Hey, welcome back, Rankers. Back in the studio, back in cold Melbourne. Portugal was just wonderful. Hi everyone, it’s rush time again.

All right. Thank you so much. I’ve got my book coming out later on this year, “Get stuffed, Google.” I know it’s been a while coming, but every day I find something else to say. Every day I find something else to say, “Ah, get stuffed Google!” Seriously? I just wanted to share, there’s been a culmination of events and I tweeted this out back in May to John Mueller who looked into it for me, and thank you very much John, but the response I got was unsatisfactory. No offence to you of course. Just the information you came back with didn’t marry up with the data that I was seeing.

So I’m back in Melbourne and it’s a Sunday afternoon and my dogs are in the studio. Hope that’s okay. And if not, what’s wrong with you? And so I looked at this data and what was happening was, we found the Googlebot and you can see here, this is Google Search Console. All of a sudden it jumped, it spiked on this site crawling 50,000 pages a day at its peak, right? And that’s not good when the site’s only got about 3,000 products, right? So something was definitely wrong.

And the other thing that happened at the same time, you can see here, the time it took to download a page in seconds plummeted, right? Really like 20 milliseconds or something, almost nothing. And at the same time, we see this massive spike. I was like, right, okay, what’s going on here?

Then when we had a look at Google Analytics, because looking at the data and the data was sort of all of a sudden starting to look a bit weird. When we dug deeper, we found that we got this huge spike in direct traffic. And when we drilled down into the source of this direct traffic, we could filter it to the network domain of Google.com or Googlebot.com or anyway Google. It was all Google traffic that was coming in directly. And when we look at the time that this source of traffic was looking at a page, it was something like three to four hundred milliseconds, right? Whereas normally it’s maybe three, four seconds.

So, a massive surge in traffic from the Google domain and a massive extra crawl from the Googlebot as displayed by Google’s own tools. John Mueller came back and said it looks like it’s the Google Ads bot. Right? And it shouldn’t be turning up in your Google Analytics data. Well it clearly did because what happened then was that we were at our wits end. We didn’t want this climb to incur huge hosting costs because of the data, of transactions, of everything else that was happening, so we just went right … let’s just throttle the bot. So that’s what we did. It’s a heavy-handed approach, but we’ve done it before. We’ve seen it used before on big brand high traffic sites when this sort of stuff goes wrong.

And so we throttled the bot. And then as you can see there, shortly after the source of the traffic dropped off. Since then we’ve had a couple of outages, and lo and behold, at the same time we’re seeing huge spikes again from the Googlebot. So Google can’t help us. Not the people I’ve spoken to, anyway. But it brings me to another point, because I was talking to another client who has millions of pages that had a similar Googlebot resource issue. Now you can call it crawl budget, but I like to think of it as your budget. What is your time worth to work out these sorts of things? Right? It’s got nothing to do with running your business. Right? So that’s why I said, “Get stuffed Google!”

I’m talking about the things that we feel that we have to go and look at. Because we’ve indicated that something is wrong. But at the end of the day, it takes us away from the real things that matter to our business. So these crawl anomalies have been bothering me ever since Google Search Console came out with the new version. And I had these metric called crawl anomalies. Because over the years, we’ve looked at it, we can’t find any consistency with URLs listed there. Sometimes they’re 404s, sometimes they’re 200, sometimes they’re 301. There’s no consistency, certainly of the service status code. And we can’t find any consistency or pattern to what’s happening with the crawl.

And I’ve consistently asked Google about this publicly. And I did so again last year at Pub Con, and towards the end of the year, I did receive an email after I emailed them directly. And thank you Gary for the response. But once again it wasn’t helpful. Well it was helpful in the sense in that I will no longer waste my time looking at crawl anomalies. And I would suggest that for most of us, you don’t either unless there’s some specific reason that you do. Now I can’t, unfortunately I’m not going to share the answer that I got. But if you want to know, I suggest that you contact Google, what the metric is about. Because we waste so many hours. I can’t fathom how many hours I’ve spent trying to find patterns in the crawl anomalies to work out how I can fix this problem for my clients. Well the problem doesn’t actually exist.

So when I say, “Get stuffed Google,” it’s about concentrating on your business and sometimes we get so caught amongst the weeds with this sort of data. But it’s something that you clearly have to fix, especially when it’s having an impact on your client’s site. So when I say, “Get stuffed Google,” it’s about building a brand, having a great customer experience, use the Google Pay products because let’s face it, that’s what they’re driving us all into. Certainly use those. Not a problem. They work well. But use them the way that is going to help grow your business. And that’s it for this week’s show. If you can shed any light on this, if you’ve had similar problems with the Googlebot at your site then please let us know, and please share, subscribe, every little bit helps.

Thanks very much. Bye.

Jim

Jim’s been here for a while, you know who he is.