Categorygithub.com/superblocksteam/agentinternalutils
package
1.18.0
Repository: https://github.com/superblocksteam/agent.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
5
Dependents
2
Files
48 SLOC

# Functions

NOTE(frank): It's important that this util is in the internal package, because it relies on metrics that are registered in the internal package.
Tokenize takes a prompt and returns the number of tokens.