New creates new Tokenizer from the provided reader with DefaultSeparators.

NewFromBytes creates new Tokenizer from the provided bytes slice.

NewFromString creates new Tokenizer from the provided string.

DefaultSeparators is a list with the default token separator characters.

Tokenizer defines a struct that parses a reader into tokens while respecting quotes and parenthesis boundaries.