Data is only good as the methods you use to collect it. With Google Analytics you track everything including search bots which isn’t helpful. Search bots crawl every page of your site and could mess up your data. This is especially true if you have low traffic. Luckily, within Google Analytics there’s a setting you can check to filter out known search bots.
How Much Traffic Do Search Bots Generate
According to the 2015 edition of Imperva Incapsula’s Bot Traffic Report:
- Humans make up 51% of all traffic online
- Good bots (bots that follow the rules) make up 19% of traffic
- Bad bots (bots that ignore rules & best practices) make up 29% of traffic
That means that 49% of all traffic is coming from bots. If you’re tracking even a small fraction of this it will destroy any meaningful data.
And of course if you use a server side (API) method of sending data to Google Analytics you might already be tracking a ton of search bots. This implementation isn’t as common but you do tend to see it more with e-commerce & membership sites so they can send sensitive information to GA without anyone snooping.
Filter Bots & Spiders
Let’s get on to actually filtering out those bots. Log into your Google Analytics account. You should see something like this:
Go ahead and click on one of your views (either Master, Test, or unfiltered, for the image above). And then click on Admin.
And then View Settings.
And we finally find the setting. Check Exclude all hits from known bots and spiders and click Save. And that’s it. You’ve filtered out the bots.
It’s a tiny setting but it can filter out a lot of misleading data.
What Exactly is a Known Search Bot?
If you’re curious like me you’ll want to know what a known search bot is. Does it just mean googlebot? Are there hundreds, thousands, or maybe even millions?
Google Analytics uses the IAB/ABC International Spiders and Bots List. This list is updated monthly by people at Microsoft, Google, Yahoo, AOL, and a whole bunch of other companies. You’d have to be a member to get the full list (for the low low price of $14,000). But even without the full list I’m pretty confident in these companies and their desire to report bots.