I need not emphasize the importance of tracking the Search Engine spiders crawling record of your websites, need I? Whether you want to check the zeal and regularity of Search Engines in visiting your website, want to find out which specific pages were crawled, want to verify the obedience of crawlers towards your robots.txt file or you simply want to filter spider activity from your web metrics reports, you need to keep track of the various spiders and bots sent away to do the Search Engine’s dirty work.
Provided here is a list of the IP addresses of the majority of Search Engine crawlers, for which I can’t take credit, which solely belongs to Dan Kramer of IPLists, whose efforts have made the world of Web 2.0 a better place. If you would like to do more than thank Dan for his efforts, you could buy him a drink.
When it comes to web metrics tracking, nothing beats manually following when Search Engines spiders crawl your website through your access logs. But in order to keep tabs on Search Engine crawlers, you need to know who they are… in short, you need to know their IP’s. Following is an IP list for Google’s and Yahoo’s bots and a link to a list of other spiders. Happy bot hunting!
Google.com IP Address Ranges
Google uses the following public IP address ranges.
iPv4
64.233.160.0 – 64.233.191.255
66.102.0.0 – 66.102.15.255
66.249.64.0 – 66.249.95.255
72.14.192.0 – 72.14.255.255
74.125.0.0 – 74.125.255.255
209.85.128.0 – 209.85.255.255
216.239.32.0 – 216.239.63.255
64.18.0.0 – 64.18.15.255
108.177.8.0 – 108.177.15.255
172.217.0.0 – 172.217.31.255
173.194.0.0 – 173.194.255.255
207.126.144.0 – 207.126.159.255
216.58.192.0 – 216.58.223.255