BuildTokens tokenizes a value, creating strings that can be used to create index keys.

GetTokenizer returns tokenizer given unique name.

Full text tokenizer, with language support.

PluginTokenizer is implemented by external plugins loaded dynamically via *.so files.

Tokenizer defines what a tokenizer must provide.