Login   Register  
PHP Classes
elePHPant
Icontem

Class: Robots_txt

Recommend this page to a friend!
Stumble It! Stumble It! Bookmark in del.icio.us Bookmark in del.icio.us
  Classes of Andy Pieters  >  Robots_txt  >  Download .tar.gz .zip  >  Support forum Support forum (1)  >  Blog Blog  >  RSS 1.0 feed RSS 2.0 feed Latest changes  
Name: Robots_txt
Support forum Support forum
Base name: robots_txt
Description: Test if a URL may be crawled looking at robots.txt
Related classes: , , , , , , ,
Version: 1.1
PHP version: 5.0
License: GNU General Public License (GPL)
All time users: 1163 users
All time rank: 2652
Week users: 1 user
Week rank: 2467
Picture of Andy Pieters
Author: Andy Pieters <e-mail contact>
Packages: 1 Browse this author's classes Browse this author's classes
Country: United Kingdom United Kingdom - PHP jobs in United Kingdom
Age: 34
All time rank: 167674 in United Kingdom United Kingdom
Week rank: 1634 Up70 in United Kingdom United Kingdom Up
Innovation award
Innovation award
Nominee: 1x


  Detailed description  
This class can be used to check whether a page may be crawled by looking at the robots.txt file of its site.

It takes the URL of a page and retrieves the robots.txt file of the same site.

The class parses the robots.txt file and looks up for the rules defined in that file to see if the site allows crawling the intended page.

The class also stores the time when a page is crawled to check whether next time another page of the same site is being crawled it is honoring the intended crawl delay and request rate limits.

 

  Groups  
Group folder image PHP 5 Classes using PHP 5 specific features View top rated classes
Group folder image Searching Search engines, crawling and indexing View top rated classes

  Rate classes User ratings   Applications   Files Files  

  Innovation Award  
PHP Programming Innovation award nominee
January 2008
Number 8
 robots.txt is a file that sites need to have in their domain Web root to tell search engine crawlers and Web robots in general which pages should not be crawled.

This class can parse a robots.txt file of a domain to determine whether a given page should be crawled or not.

It is useful to implement a friendly crawler which respects the wishes of site owners that do not want to have certain pages crawled by Web robot programs.

Manuel Lemos

  User ratings  
Ratings
Utility
Consistency
Documentation
Examples
Tests
Videos
Overall
Rank
All time:
Not sure (50.0%)
Sufficient (62.5%)
Not sure (50.0%)
-
-
-
Insufficient (36.2%)
2076
Month:
Not yet rated by the users

  Applications that use this class  
No application links were specified for this class.
Add link image If you know an application of this package, send a message to the author to add a link here.
  Files folder image Files  
File Role Description
Plain text file Robots.txt.class.php Class Core file
Accessible without login Plain text file README.txt Doc. Usage Examples

Download all files: robots_txt.tar.gz robots_txt.zip
NOTICE: if you are using a download manager program like 'GetRight', please Login before trying to download this archive.