Categorygithub.com/gomlx/go-huggingfacetokenizerssentencepiece
package
0.1.1
Repository: https://github.com/gomlx/go-huggingface.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
4
Dependents
1
Files
56 SLOC

# Packages

No description provided by the author

# Functions

New creates a SentencePiece tokenizer based on the "tokenizer.model" file, which must be a SentencePiece Model proto (see protos.Model).

# Structs

Tokenizer implements tokenizers.Tokenizer interface based on SentencePiece tokenizer by Google.