File: README.md

Recommend this page to a friend!
  Classes of Gonzalo Chumillas  >  Ses Tokenizer  >  README.md  >  Download  
File: README.md
Role: Documentation
Content type: text/plain
Description: Documentation
Class: Ses Tokenizer
Parse and split a string into tokens
Author: By
Last change:
Date: 7 years ago
Size: 930 bytes
 

 

Contents

Class file image Download
tokenizer
=========

The Tokenizer class allows us to split an string into tokens. Unlike other classes, it is based on regular expressions. The 'match' function is the most important function of the class. It allows to split an string into tokens and accepts a regular expression as parameter. For example:

```php
// splits an string into 'words'
$t = new Tokenizer("Lorem ipsum dolor sit amet");
while (list($token) = $t->match("\w+")) {
    echo "$token-";
}
```

Note that you DO NOT NEED to write an explicit regular expression. In the above example, instead of typing "/^\s*\w+/" we can write "\w+". In this case, the function ignores the left spaces and start searching from the current offset position. In any case, you can use an explicit regular expresion:

```php
// uses an explicit regular expression
$t = new Tokenizer("I'm 35 years old");
if (list($years) = $t->match("/\d+/")) {
	echo "You are $years old";
}
```

For more information send a message to info at phpclasses dot org.