Categorygithub.com/AlperRehaYAZGAN/postgresbasetoolstokenizer
package
9.20.6+incompatible
Repository: https://github.com/alperrehayazgan/postgresbase.git
Documentation: pkg.go.dev
Overview
Versions
2
Dependencies
5
Dependents
6
Files
187 SLOC

# Functions

New creates new Tokenizer from the provided reader with DefaultSeparators.
NewFromBytes creates new Tokenizer from the provided bytes slice.
NewFromString creates new Tokenizer from the provided string.

# Variables

DefaultSeparators is a list with the default token separator characters.

# Structs

Tokenizer defines a struct that parses a reader into tokens while respecting quotes and parenthesis boundaries.