PHP Classes

Spider website: Crawl a site and retrieve the the URL of all links

Recommend this page to a friend!
Stumble It! Stumble It! Bookmark in Bookmark in

  Author Author  
Picture of Karol Janyst
Name: Karol Janyst <contact>
Classes: 2 packages by
Country: Poland Poland
Age: 27
All time rank: 40312 in Poland Poland
Week rank: 146 Up4 in Poland Poland Up

  Detailed description   Download Download .zip .tar.gz   Install with Composer Install with Composer  
This class can be used to crawl a site and retrieve the the URL of all links.

It can retrieve a page of a site and follow all links recursively to retrieve all the site URLs.

The class can restrict the crawling to URLs with a given extension and avoids accessing pages listed in the site robots.txt file, or pages set with the no index or no follow meta tags.

  Classes of Karol Janyst  >  Spider website  >  Download Download .zip .tar.gz  >  Support forum Support forum (2)  >  Blog Blog  >  RSS 1.0 feed RSS 2.0 feed Latest changes  
Name: Spider website
Base name: spider
Description: Crawl a site and retrieve the the URL of all links
Version: 0.1
PHP version: 5.0
License: GNU General Public License (GPL)
All time users: 2610 users
All time rank: 1386
Week users: 5 users
Week rank: 256 Up
  Groups   Rate classes User ratings   Applications   Files Files  

Group folder image HTML HTML generation and processing View top rated classes
Group folder image PHP 5 Classes using PHP 5 specific features View top rated classes
Group folder image Searching Search engines, crawling and indexing View top rated classes

  User ratings  
RatingsUtility Consistency Documentation Examples Tests Videos Overall Rank
All time: Sufficient (66%) Good (87%) - Sufficient (70%) - - Not sure (49%) 1774
Month: Not yet rated by the users

  Applications that use this package  
No pages of applications that use this class were specified.

Add link image If you know an application of this package, send a message to the author to add a link here.

  Files folder image Files  
File Role Description
Plain text file spider.class.php Class Main class file
Accessible without login Plain text file example.php Example Example file

Install with Composer Install with Composer - Download Download all files: spider.tar.gz
NOTICE: if you are using a download manager program like 'GetRight', please Login before trying to download this archive.