BuildTokens tokenizes a value, creating strings that can be used to create index keys.

EncodeGeoTokens encodes the given list of tokens as geo tokens.

EncodeRegexTokens encodes the given list of strings as regex tokens.

GetFullTextTokens returns the full-text tokens for the given value.

GetTermTokens returns the term tokens for the given value.

GetTokenizer returns tokenizer given unique name.

GetTokenizerByID tries to find a tokenizer by id in the registered list.

GetTokenizerForLang returns the correct full-text tokenizer for the given language.

GetTokenizers returns a list of tokenizer given a list of unique names.

GetTokens returns the tokens for the given tokenizer ID and value.

LangBase returns the BCP47 base of a language.

LoadCustomTokenizer reads and loads a custom tokenizer from the given file.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

ASCII 31 - Unit separator.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

Tokenizer identifiers are unique and can't be reused.

BoolTokenizer returns tokens from boolean data.

CustomTokenizer generates tokens from custom logic.

DayTokenizer generates day tokens from datetime data.

ExactTokenizer returns the exact string as a token.

FloatTokenizer generates tokens from floating-point data.

FullTextTokenizer generates full-text tokens from string data.

GeoTokenizer generates tokens from geo data.

HashTokenizer returns hash tokens from string data.

HourTokenizer generates hour tokens from datetime data.

IntTokenizer generates tokens from integer data.

MonthTokenizer generates month tokens from datetime data.

Sha256Tokenizer generates tokens for the sha256 hash part from string data.

TermTokenizer generates term tokens from string data.

TrigramTokenizer returns trigram tokens from string data.

YearTokenizer generates year tokens from datetime data.

PluginTokenizer is implemented by external plugins loaded dynamically via *.so files.

Tokenizer defines what a tokenizer must provide.