Author Topic: Web Crawlers  (Read 1173 times)

0 Members and 0 Guests are viewing this topic.

Offline CB

  • Sr. Member
  • ****
  • Posts: 253
  • Karma: 2
    • View Profile
Re: Web Crawlers
« Reply #15 on: January 13, 2012, 12:57:22 am »
I know how to get to and open the pages with regards to the IP address, but what are the tell tale signs to look for once the page is opened to know if it's a legit Human looking at the site and one that is a Bot? I can't make heads or tails from all the jargon on the page.

Offline aIURbliS

  • Hero Member
  • *****
  • Posts: 1215
  • Karma: 45
  • Gender: Male
  • Location: U.S.
    • View Profile
Re: Web Crawlers
« Reply #16 on: January 14, 2012, 01:07:27 am »
We can't really differentiate between human and bot guests, only by known ip addresses that we know if it's a bot
All codes I make are tested on the latest version of Google Chrome and upon request of other browsers.

Offline CB

  • Sr. Member
  • ****
  • Posts: 253
  • Karma: 2
    • View Profile
Re: Web Crawlers
« Reply #17 on: January 14, 2012, 01:20:27 am »
Quote
...only by known ip addresses that we know if it's a bot


Really not sure what you mean here. The way it's worded is that you are able to look at an IP address and know it's a Bot. So, if that's the case, then what is it in the address that tells you it's a Bot?

Offline aIURbliS

  • Hero Member
  • *****
  • Posts: 1215
  • Karma: 45
  • Gender: Male
  • Location: U.S.
    • View Profile
Re: Web Crawlers
« Reply #18 on: January 14, 2012, 02:23:06 am »
Well some people try to track what ip addresses are Google Bots for an example. They make a list and we read that list to know which ip addresses a Google Bot uses. However it ip addresses change from time to time unfortunately. Here's something I found on Google's website: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=80553
All codes I make are tested on the latest version of Google Chrome and upon request of other browsers.