ParseConfigContent parses the given json content (of a tokenizer_config.json file) into a Config structure.

ParseConfigFile parses the given file (holding a tokenizer_config.json file) into a Config structure.

SpecialTokenString retrieves an enum value from the enum constants string name.

SpecialTokenStrings returns a slice of all String values of the enum.

SpecialTokenValues returns all values of the enum.

Config struct to hold HuggingFace's tokenizer_config.json contents.

Tokenizer interface allows one convert test to "tokens" (integer ids) and back.

SpecialToken is an enum of commonly used special tokens.