News:

Create A Forum Installed

Author Topic: Web Crawlers  (Read 1283 times)

0 Members and 1 Guest are viewing this topic.

Offline aIURbliS

  • Hero Member
  • *****
  • Posts: 1215
  • Karma: 45
  • Gender: Male
  • Location: U.S.
    • View Profile

Badges: (View All)
Level 6 Sixth year Anniversary Fifth year Anniversary
Re: Web Crawlers
« Reply #15 on: January 14, 2012, 02:23:06 am »
Well some people try to track what ip addresses are Google Bots for an example. They make a list and we read that list to know which ip addresses a Google Bot uses. However it ip addresses change from time to time unfortunately. Here's something I found on Google's website: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=80553
All codes I make are tested on the latest version of Google Chrome and upon request of other browsers.