class Syntax::Tokenizer

The base class of all tokenizers. It sets up the scanner and manages the looping until all tokens have been extracted. It also provides convenience methods to make sure adjacent tokens of identical groups are returned as a single token.