Categorygithub.com/src-d/enry/v2internaltokenizer
package
2.1.0
Repository: https://github.com/src-d/enry.git
Documentation: pkg.go.dev
Overview
Versions
2
Dependencies
2
Dependents
2
Files
193 SLOC

# Functions

Tokenize returns lexical tokens from content.

# Constants

ByteLimit defines the maximum prefix of an input text that will be tokenized.