| File: |
tokenizer.php |
| Role: |
Class source |
| Content type: |
text/plain |
| Description: |
source code |
| Class: |
Tokenizer
Tokenizer split strings into tokens |
| Author: |
By Domenico Pontari |
| Last change: |
New algorithm for the generation of tokens: using "explode" function instead of "preg_split". So that it is easier to manage special chars. Performances improvement expected. |
| Date: |
13 years ago |
| Size: |
13,263 bytes |
|
|
|
