PHP Classes
elePHPant
Icontem

Ses Tokenizer: Parse and split a string into tokens

Recommend this page to a friend!
  Info   View files View files (4)   DownloadInstall with Composer Download .zip   Reputation   Support forum   Blog    
Last Updated Ratings Unique User Downloads Download Rankings
2013-09-25 (3 years ago) RSS 2.0 feedNot enough user ratingsTotal: 296 All time: 7,000 This week: 916Up
Version License PHP version Categories
ses-tokenizer 1.0GNU General Publi...5.3PHP 5, Text processing, Parsers
Description Author

This class can parse and split a string into tokens.

It can take a string and split it to retrieve smaller tokens one by one.

The format of the tokens is defined by regular expressions passed to the class as parameters.

Picture of Gonzalo Chumillas
Name: Gonzalo Chumillas <contact>
Classes: 8 packages by
Country: Spain Spain
Age: 42
All time rank: 133227 in Spain Spain
Week rank: 1284 Up34 in Spain Spain Down
Innovation award
Innovation award
Nominee: 2x

Details
tokenizer
=========

The Tokenizer class allows us to split an string into tokens. Unlike other classes, it is based on regular expressions. The 'match' function is the most important function of the class. It allows to split an string into tokens and accepts a regular expression as parameter. For example:

```php
// splits an string into 'words'
$t = new Tokenizer("Lorem ipsum dolor sit amet");
while (list($token) = $t->match("\w+")) {
    echo "$token-";
}
```

Note that you DO NOT NEED to write an explicit regular expression. In the above example, instead of typing "/^\s*\w+/" we can write "\w+". In this case, the function ignores the left spaces and start searching from the current offset position. In any case, you can use an explicit regular expresion:

```php
// uses an explicit regular expression
$t = new Tokenizer("I'm 35 years old");
if (list($years) = $t->match("/\d+/")) {
	echo "You are $years old";
}
```
  Files folder image Files  
File Role Description
Files folder imageclasses (1 directory)
Accessible without login Plain text file LICENSE Lic. License text
Accessible without login Plain text file README.md Doc. Documentation
Accessible without login Plain text file test.php Example Examples

  Files folder image Files  /  classes  
File Role Description
Files folder imageparser (1 file)

  Files folder image Files  /  classes  /  parser  
File Role Description
  Accessible without login Plain text file tokenizer.php Class Tokenizer class

 Version Control Unique User Downloads Download Rankings  
 100%
Total:296
This week:0
All time:7,000
This week:916Up