Collect scans and collect all tokens.

Scan scans given input.

TokenBoolean is the BOOLEAN token.

TokenClose is the CLOSE token.

TokenCloseBlockParams is the CLOSE_BLOCK_PARAMS token.

TokenCloseRawBlock is the CLOSE_RAW_BLOCK token.

TokenCloseSexpr is the CLOSE_SEXPR token.

TokenCloseUnescaped is the CLOSE_UNESCAPED token.

TokenComment is the COMMENT token.

TokenContent is the CONTENT token.

TokenData is the DATA token.

TokenEOF represents an End Of File.

TokenEquals is the EQUALS token.

TokenError represents an error.

TokenID is the ID token.

TokenInverse is the INVERSE token.

TokenNumber is the NUMBER token.

TokenOpen is the OPEN token.

TokenOpenBlock is the OPEN_BLOCK token.

TokenOpenBlockParams is the OPEN_BLOCK_PARAMS token.

TokenOpenEndBlock is the OPEN_ENDBLOCK token.

TokenOpenEndRawBlock is the END_RAW_BLOCK token.

TokenOpenInverse is the OPEN_INVERSE token.

TokenOpenInverseChain is the OPEN_INVERSE_CHAIN token.

TokenOpenPartial is the OPEN_PARTIAL token.

TokenOpenRawBlock is the OPEN_RAW_BLOCK token.

TokenOpenSexpr is the OPEN_SEXPR token.

TokenOpenUnescaped is the OPEN_UNESCAPED token.

TokenSep is the SEP token.

TokenString is the STRING token.

Lexer is a lexical analyzer.

Token represents a scanned token.

TokenKind represents a Token type.