Could I get some feedback on this please??
I'm trying to build up the "Search Engines" static table to be able to identify Search Bots and separate them out from the crud properly?
I know we can use GA, but for the Prod Mgrs, its easier for them to see a count on the Dashboard for site visits and then just click on the number to see the spread.
I have found this list of UserAgent strings, but my REGEX is not that good.
http://www.useragentstring.com/pages/Crawlerlist/
be handy to get an updated table to find all these that have the correct 'referer' and UA regex's
from my Site visits report, here is a snapshot of the bots...
Quantity Users Agents
833 Mozilla/5.0 (compatible; MJ12bot/v1.4.5;
http://www.majestic12.co.uk/bot.php?+)
220 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
51 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
22 Dalvik/1.6.0 (Linux; U; Android 4.4.2; SM-N9005 Build/KOT49H)
19 Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
12 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
And all i am getting in the UA section at the bottom of the page is
Quantity Robots Visits
223 Google