PHP Classes

Ses Tokenizer: Parse and split a string into tokens

Recommend this page to a friend!
Stumble It! Stumble It! Bookmark in Bookmark in
  Info   View files View files (4)   DownloadInstall with Composer Download .zip   Reputation   Support forum   Blog    
Last Updated Ratings Unique User Downloads Download Rankings  
2013-09-25 (1 year ago) RSS 2.0 feedNot enough user ratingsTotal: 287 All time: 6,832 This week: 1,187Up
Version License PHP version Categories  
ses-tokenizer 1.0GNU General Publi...5.3PHP 5, Text processing, Parsers
Description Author  

This class can parse and split a string into tokens.

It can take a string and split it to retrieve smaller tokens one by one.

The format of the tokens is defined by regular expressions passed to the class as parameters.

Picture of Gonzalo Chumillas
Name: Gonzalo Chumillas <contact>
Classes: 8 packages by
Country: Spain Spain
Age: 40
All time rank: 132527 in Spain Spain
Week rank: 891 Up21 in Spain Spain Up
Innovation award
Innovation award
Nominee: 2x

Details provided by the author  

The Tokenizer class allows us to split an string into tokens. Unlike other classes, it is based on regular expressions. The 'match' function is the most important function of the class. It allows to split an string into tokens and accepts a regular expression as parameter. For example:

// splits an string into 'words'
$t = new Tokenizer("Lorem ipsum dolor sit amet");
while (list($token) = $t->match("\w+")) {
    echo "$token-";

Note that you DO NOT NEED to write an explicit regular expression. In the above example, instead of typing "/^\s*\w+/" we can write "\w+". In this case, the function ignores the left spaces and start searching from the current offset position. In any case, you can use an explicit regular expresion:

// uses an explicit regular expression
$t = new Tokenizer("I'm 35 years old");
if (list($years) = $t->match("/\d+/")) {
	echo "You are $years old";
  Files folder image Files  
File Role Description
Files folder imageclasses (1 directory)
Accessible without login Plain text file LICENSE Lic. License text
Accessible without login Plain text file Doc. Documentation
Accessible without login Plain text file test.php Example Examples

  Files folder image Files  /  classes  
File Role Description
Files folder imageparser (1 file)

  Files folder image Files  /  classes  /  parser  
File Role Description
  Plain text file tokenizer.php Class Tokenizer class

 Version Control Unique User Downloads Download Rankings  
 100%Total:287All time:6,832
 This week:0This week:1,187Up