Categorygithub.com/apparentlymart/go-textsegtextseg
package
1.0.0
Repository: https://github.com/apparentlymart/go-textseg.git
Documentation: pkg.go.dev
Overview
Versions
7
Dependencies
5
Dependents
22
Files
10.9k SLOC

# Functions

AllTokens is a utility that uses a bufio.SplitFunc to produce a slice of all of the recognized tokens in the given buffer.
ScanGraphemeClusters is a split function for bufio.Scanner that splits on grapheme cluster boundaries.
ScanGraphemeClusters is a split function for bufio.Scanner that splits on UTF8 sequence boundaries.
TokenCount is a utility that uses a bufio.SplitFunc to count the number of recognized tokens in the given buffer.

# Variables

No description provided by the author