package
0.2.2
Repository: https://github.com/sugarme/tokenizer.git
Documentation: pkg.go.dev

# README

Basic examples

This demonstrates how to use different models (i.e., BERT or BPE) in a tokenizer.

To run:

go run . -model=bert
go run . -model=bpe