PHP Classes

Tokenizer: Tokenizer split strings into tokens

Recommend this page to a friend!
  Info   View files View files (2)   DownloadInstall with Composer Download .zip   Reputation   Support forum (2)   Blog    
Ratings Unique User Downloads Download Rankings
StarStarStar 47%Total: 692 This week: 1All time: 4,721 This week: 560Up
Version License PHP version Categories
tokenizer 1BSD License5PHP 5, Text processing
Description 

Author

This package obtain a list of tokens from a text string.

You can choose single space characters or sequences of more characters as delimiters.

You can also define markers characters: special characters or special character sequences that define an indivisible portion of string. That is useful to implement a quotation system for your tokens. You can define an escaping sequence for markers too.

Furthermore, it supports the definition of characters called "operators". Operators allow you to obtain pieces of strings ready for further elaborations combining tokens together.

Innovation Award
PHP Programming Innovation award nominee
February 2010
Number 7


Prize: One copy of VS.PHP
A tokenizer is a software component that can process a text and split it into tokens that are easier to process.

Tokenizers are often used to by processors that interpret a text, like for instance source code of a given language.

First they split the code in tokens before evaluating whether the code syntax is correct according to the language grammar.

This class provides a complete solution to tokenize any type of text using pure PHP code.

It also supports the implementation of basic syntax operations that combine sequences of tokens and operators into a smaller number of tokens, thus simplifying further the process of the resulting tokens.

Manuel Lemos
Picture of Domenico Pontari
Name: Domenico Pontari <contact>
Classes: 3 packages by
Country: Italy Italy
Age: ???
All time rank: 204080 in Italy Italy
Week rank: 416 Up16 in Italy Italy Up
Innovation award
Innovation award
Nominee: 1x

  Files folder image Files  
File Role Description
Accessible without login Plain text file usage.php Example different examples of the usage
Plain text file tokenizer.php Class source code

 Version Control Reuses Unique User Downloads Download Rankings  
 0%1
Total:692
This week:1
All time:4,721
This week:560Up
User Ratings User Comments (1)
 All time
Utility:66%StarStarStarStar
Consistency:66%StarStarStarStar
Documentation:-
Examples:75%StarStarStarStar
Tests:-
Videos:-
Overall:47%StarStarStar
Rank:3046