Login   Register  
PHP Classes

Tokenizer: Tokenizer split strings into tokens

Recommend this page to a friend!
Stumble It! Stumble It! Bookmark in del.icio.us Bookmark in del.icio.us

  Author Author  
Picture of Domenico Pontari
Name: Domenico Pontari <contact>
Classes: 3 packages by
Country: Italy Italy
Age: ???
All time rank: 192074 in Italy Italy
Week rank: 1067 Up41 in Italy Italy Up
Innovation award
Innovation award
Nominee: 1x

  Detailed description   Download Download .zip .tar.gz  
This package obtain a list of tokens from a text string.

You can choose single space characters or sequences of more characters as delimiters.

You can also define markers characters: special characters or special character sequences that define an indivisible portion of string. That is useful to implement a quotation system for your tokens. You can define an escaping sequence for markers too.

Furthermore, it supports the definition of characters called "operators". Operators allow you to obtain pieces of strings ready for further elaborations combining tokens together.

  Classes of Domenico Pontari  >  Tokenizer  >  Download Download .zip .tar.gz  >  Support forum Support forum (2)  >  Blog Blog  >  RSS 1.0 feed RSS 2.0 feed Latest changes  
Name: Tokenizer
Base name: tokenizer
Description: Tokenizer split strings into tokens
Version: 1
PHP version: 5
License: BSD License
All time users: 659 users
All time rank: 4417
Week users: 0 users
Week rank: 1438 Equal
  Groups   Rate classes User ratings   Dependents   Applications   Files Files  

Group folder image PHP 5 Classes using PHP 5 specific features View top rated classes
Group folder image Text processing Manipulating and validating text data View top rated classes

  Innovation Award  
PHP Programming Innovation award nominee
February 2010
Number 7

Prize: One copy of VS.PHP
A tokenizer is a software component that can process a text and split it into tokens that are easier to process.

Tokenizers are often used to by processors that interpret a text, like for instance source code of a given language.

First they split the code in tokens before evaluating whether the code syntax is correct according to the language grammar.

This class provides a complete solution to tokenize any type of text using pure PHP code.

It also supports the implementation of basic syntax operations that combine sequences of tokens and operators into a smaller number of tokens, thus simplifying further the process of the resulting tokens.

Manuel Lemos

  User ratings  
Not enough user ratings

  Other classes that need this package  
Class Why it is needed Dependency
param This package is used to check strings in $_POST or $_GET vars Required

  Applications that use this package  
No pages of applications that use this class were specified.
Add link image If you know an application of this package, send a message to the author to add a link here.
  Files folder image Files  
File Role Description
Accessible without login Plain text file usage.php Example different examples of the usage
Plain text file tokenizer.php Class source code

Download Download all files: tokenizer.tar.gz tokenizer.zip
NOTICE: if you are using a download manager program like 'GetRight', please Login before trying to download this archive.