BuildTokens tokenizes a value, creating strings that can be used to create index keys.

GetTokenizer returns tokenizer given unique name.

PluginTokenizer is implemented by external plugins loaded dynamically via *.so files.

Tokenizer defines what a tokenizer must provide.