 |
|
Innovation award
Nominee: 1x |
This package obtain a list of tokens from a text string.
You can choose single space characters or sequences of more characters as delimiters.
You can also define markers characters: special characters or special character sequences that define an indivisible portion of string. That is useful to implement a quotation system for your tokens. You can define an escaping sequence for markers too.
Furthermore, it supports the definition of characters called "operators". Operators allow you to obtain pieces of strings ready for further elaborations combining tokens together.
|
|
| Name: |
Tokenizer |
| Base name: |
tokenizer |
| Description: |
Tokenizer split strings into tokens |
| Version: |
1 |
| PHP version: |
5 |
| License: |
BSD License |
| All time users: |
632 users |
| All time rank: |
4299 |
| Week users: |
0 users |
| Week rank: |
2051  |
| |
|
February 2010
Number 7
Prize: One copy of VS.PHP |
A tokenizer is a software component that can process a text and split it into tokens that are easier to process.
Tokenizers are often used to by processors that interpret a text, like for instance source code of a given language.
First they split the code in tokens before evaluating whether the code syntax is correct according to the language grammar.
This class provides a complete solution to tokenize any type of text using pure PHP code.
It also supports the implementation of basic syntax operations that combine sequences of tokens and operators into a smaller number of tokens, thus simplifying further the process of the resulting tokens.
Manuel Lemos |
| There are not enough user ratings to display for this class. |
| |
Packages that need this class |
|
|
| Class |
Dependency |
Why it is needed |
|---|
| param |
Required |
This package is used to check strings in $_POST or $_GET vars |
| |
Applications that use this class |
|
|
No application links were specified for this class.
If you know an application of this package, send a message to the
author to add a link here.
| |
 Files |
|
|
Stumble It!
Bookmark in del.icio.us
Support forum (2)
Blog
Latest changes
User ratings
Files
