Categorygithub.com/wbrown/gpt_bpe

modulepackage

0.0.0-20240927210347-534087680bf6

Repository: https://github.com/wbrown/gpt_bpe.git

Documentation: pkg.go.dev

# README

gpt-bpe

An implementation of GPT2 Byte Pair Encoding Encoder/Decoder in golang. Generally very fast, bottlenecked by the regex engine for whitespace separation.

# Packages

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

# Functions

CreateReplacementsRuneTree

No description provided by the author

Create a new rune tree from an array of strings to match against.

No description provided by the author

NewEncoder Returns a GPTEncoder with the tokenizer data loaded for that vocabulary id.

No description provided by the author

NewLlama2Encoder

No description provided by the author

NewLlama3Encoder

No description provided by the author

NewMistralEncoder

No description provided by the author

NewNerdstashV1Encoder

No description provided by the author

NewNerdstashV2Encoder

No description provided by the author

No description provided by the author

No description provided by the author

# Constants

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

VOCAB_ID_LLAMA_3

No description provided by the author

VOCAB_ID_MISTRAL

No description provided by the author

VOCAB_ID_NERDSTASH_V1

No description provided by the author

VOCAB_ID_NERDSTASH_V2

No description provided by the author

No description provided by the author

No description provided by the author

# Structs

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

# Type aliases

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author

No description provided by the author