Package api defines the Tokenizer API.

Package sentencepiece implements a tokenizers.Tokenizer based on SentencePiece tokenizer.

GetConfig returns the parsed "tokenizer_config.json" Config object for the repo.

New creates a new tokenizer from the given HuggingFace repo (see hub.New).

RegisterTokenizerClass used by Tokenizer implementations.

Config struct to hold HuggingFace's tokenizer_config.json contents.

SpecialToken is an enum of commonly used special tokens.

Tokenizer interface allows one convert test to "tokens" (integer ids) and back.

TokenizerConstructor is used by Tokenizer implementations to provide implementations for different tokenizer classes.