tok.t file | source file

Tokenizer - customizable tokenizer class for use with the intrinsic class 'grammar-production' parser.

This tokenizer implementation is parameterized with a set of rules (see below); a basic set of rules is provided, but users can customize the tokenizer quite extensively simply by subclassing the Tokenizer class and overriding the 'rules_' property with a new set of rules declarations.

Summary of Classes

`Tokenizer TokenizerError TokErrorNoMatch`

Summary of Global Objects

(none)

Summary of Global Functions

`main`

Global Functions

`main (args)` | `tok.t[396]`

Test Section

Adv3Lite Library Reference Manual
Generated on 13/06/2025 from adv3Lite version 2.2.1