For those that follow these types of things, you may have noticed bot traffic starting approximately November 2014, coming from an IP range belonging to Apple. We first noticed it when our security software was blocking it as a possible spam harvester. The first incarnations were not readily identifying themselves and were connecting without a proper header request.
In the past few days though the same bot has started to identify itself as:
User Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)
Following the provided url we see that Apple is indeed running their own bot and it’s for:
“Applebot is the web crawler for Apple, used by products including Siri and Spotlight Suggestions.”
IP ranges thus far 220.127.116.11 – 18.104.22.168
Now let’s see if they can connect in a manner more in line with established norms.