Top rated packages with tag crawl

Recommend this page to a friend!

			All class groups	>	Search tags: crawl

	1. Blue Crawler		crawl

Supplied by: kai <e-mail contact>

Crawl and retrieve links of Web pages

This class can be used to crawl and retrieve links of Web pages

It can retrieve a given Web pages and retrieve the URL of all linked pages recursively.

The class can also crawl all linked pages until it finds given search keywords.

Ratings	Utility	Consistency	Documentation	Examples	Tests	Videos	Overall	Rank
All time:	`Good (83.3%)`	`Good (83.3%)`	`Sufficient (75.0%)`	`Good (83.3%)`	`-`	`-`	`Sufficient (65.8%)`	386
Month:	Not yet rated by the users

	2. Web Crawler using MySQL DB		crawl

Supplied by: vivek <e-mail contact>

Retrieve Web pages and store links in a database

This class can be used to retrieve Web pages and store the URLs links in a MySQL database.

It retrieves a given Web page and parses its HTML content to extract the URLs of links and frames.

The URLs that are crawled are stored in a MySQL database table if the URL was not yet stored previously.

The class can also display in a Web page the list of URLs already stored from a given domain.

Ratings	Utility	Consistency	Documentation	Examples	Tests	Videos	Overall	Rank
All time:	`Good (83.3%)`	`Sufficient (75.0%)`	`-`	`Good (83.3%)`	`-`	`-`	`Not sure (57.5%)`	989
Month:	Not yet rated by the users

	3. Crawler		crawl

Supplied by: Md. Shaiful islam <e-mail contact>

Extract links and images from remote Web pages

This class can be used to extract links and images from remote Web pages.

It can access Web pages, parse the pages HTML and extract the URLs of the links and the images.

If necessary, the class may access a login page and emulate the submission of a login form to subsequent accesses can be done on behalf of the logged user.

Ratings	Utility	Consistency	Documentation	Examples	Tests	Videos	Overall	Rank
All time:	`Sufficient (75.0%)`	`Sufficient (69.4%)`	`-`	`Sufficient (77.8%)`	`-`	`-`	`Not sure (52.2%)`	1347
Month:	Not yet rated by the users

	4. Spider Class		crawl

Supplied by: greg jackson <e-mail contact>

Crawl a site following and retrieving linked pages

This class can retrieve Web pages and parse them to extract the list of their links to continue crawling all linked pages.

The pages may be retrieved iteratively until it is reached a given limit of pages or link depth.

It is possible to set regular expressions for both link definitions and content matches, changeable at every depth.

Ratings	Utility	Consistency	Documentation	Examples	Tests	Videos	Overall	Rank
All time:	`Sufficient (65.0%)`	`Sufficient (70.0%)`	`Not sure (55.0%)`	`Not sure (50.0%)`	`-`	`-`	`Not sure (50.0%)`	1477
Month:	Not yet rated by the users

	5. Spider website		crawl

Supplied by: Karol Janyst <e-mail contact>

Crawl a site and retrieve the the URL of all links

This class can be used to crawl a site and retrieve the the URL of all links.

It can retrieve a page of a site and follow all links recursively to retrieve all the site URLs.

The class can restrict the crawling to URLs with a given extension and avoids accessing pages listed in the site robots.txt file, or pages set with the no index or no follow meta tags.

Ratings	Utility	Consistency	Documentation	Examples	Tests	Videos	Overall	Rank
All time:	`Sufficient (66.7%)`	`Good (87.5%)`	`-`	`Sufficient (70.8%)`	`-`	`-`	`Not sure (49.2%)`	1554
Month:	Not yet rated by the users

	6. Google Crawler		crawl

Supplied by: Bari Hossain <e-mail contact>

Extract results from Google search pages

This class can be used to extract results from Google search result pages.

It retrieves the Google search result pages for given search keywords.

The class parses the result pages to retrieve the list of search result URLs, titles and excerpts from the pages that were found.

The limit number of result pages to be retrieved is configurable.

Ratings	Utility	Consistency	Documentation	Examples	Tests	Videos	Overall	Rank
All time:	`Not sure (56.2%)`	`Sufficient (68.8%)`	`-`	`Sufficient (71.9%)`	`-`	`-`	`Not sure (42.2%)`	1890
Month:	Not yet rated by the users

	7. Robots_txt		crawl

Supplied by: Andy Pieters <e-mail contact>

Test if a URL may be crawled looking at robots.txt

This class can be used to check whether a page may be crawled by looking at the robots.txt file of its site.

It takes the URL of a page and retrieves the robots.txt file of the same site.

The class parses the robots.txt file and looks up for the rules defined in that file to see if the site allows crawling the intended page.

The class also stores the time when a page is crawled to check whether next time another page of the same site is being crawled it is honoring the intended crawl delay and request rate limits.

Ratings	Utility	Consistency	Documentation	Examples	Tests	Videos	Overall	Rank
All time:	`Not sure (50.0%)`	`Sufficient (62.5%)`	`Not sure (50.0%)`	`-`	`-`	`-`	`Insufficient (36.2%)`	2076
Month:	Not yet rated by the users

	8. Mail Riddler		crawl

Supplied by: Jan Paepke <e-mail contact>

Encode email addresses in pages to stop harvesting

This class is meant to encode email addresses used in mailto: links inserted in Web pages to prevent harvesting that is done usually by spammers that crawl the Web to collect e-mail addresses of their future victims.

The class uses a combination of PHP and Javascript to encode email addresses and so make it harder, if not impossible, for spammers crawling robots to collect address to put them into their databases.

This is an alternative to the common solution of encoding addresses as "email at host dot com".

The documentation is currently only available in German.

Ratings	Utility	Consistency	Documentation	Examples	Tests	Videos	Overall	Rank
All time:	`Not sure (56.2%)`	`Sufficient (68.8%)`	`-`	`-`	`-`	`-`	`Insufficient (35.0%)`	2103
Month:	Not yet rated by the users

Copyright (c) Icontem 1999-2012	PHP Classes	- PHP Class Scripts
	PHP Book Reviews	- Reviews of books and other products