# Packages
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
# Functions
BuildTokens tokenizes a value, creating strings that can be used to create index keys.
EncodeGeoTokens encodes the given list of tokens as geo tokens.
EncodeRegexTokens encodes the given list of strings as regex tokens.
No description provided by the author
GetFullTextTokens returns the full-text tokens for the given value.
No description provided by the author
GetIndexFactory returns IndexFactory given name.
No description provided by the author
No description provided by the author
GetTermTokens returns the term tokens for the given value.
GetTokenizer returns tokenizer given unique name.
GetTokenizerByID tries to find a tokenizer by id in the registered list.
GetTokenizerForLang returns the correct full-text tokenizer for the given language.
GetTokenizers returns a list of tokenizer given a list of unique names.
GetTokens returns the tokens for the given tokenizer ID and value.
LangBase returns the BCP47 base of a language.
LoadCustomTokenizer reads and loads a custom tokenizer from the given file.
# Constants
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
ASCII 31 - Unit separator.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
Tokenizer identifiers are unique and can't be reused.
# Structs
BigFloatTokenizer generates tokens from big float data.
BoolTokenizer returns tokens from boolean data.
CustomTokenizer generates tokens from custom logic.
DayTokenizer generates day tokens from datetime data.
ExactTokenizer returns the exact string as a token.
FactoryCreateSpec includes an IndexFactory and the options required to instantiate a VectorIndex of the given type.
FloatTokenizer generates tokens from floating-point data.
FullTextTokenizer generates full-text tokens from string data.
GeoTokenizer generates tokens from geo data.
HashTokenizer returns hash tokens from string data.
HourTokenizer generates hour tokens from datetime data.
IntTokenizer generates tokens from integer data.
MonthTokenizer generates month tokens from datetime data.
Sha256Tokenizer generates tokens for the sha256 hash part from string data.
TermTokenizer generates term tokens from string data.
TrigramTokenizer returns trigram tokens from string data.
YearTokenizer generates year tokens from datetime data.
# Interfaces
IndexFactory combines the notion of a Tokenizer with index.IndexFactory.
PluginTokenizer is implemented by external plugins loaded dynamically via *.so files.
Tokenizer defines what a tokenizer must provide.