New creates a Scanner which reads from the given io.Reader.

TokenKindDirSep identifies a token equal to DirSep.

TokenKindEnd is returned from Scanner.Scan when the wrapped io.Reader has been read to completion.

TokenKindEscape identifies a 2-rune token which always starts with the Escape rune.

TokenKindKeyValSep identifies a token equal to KeyValSep.

TokenKindNewline identifies a token which only contains runes found in the runesWhitespace or runesNewline constants.

TokenKindOther identifies all other possible tokens which are not identified by the given TokenKind constants.

TokenKindReserved identifies a single-rune token which isn't currently used by the language but reserved for later use.

TokenKindStrMark identifies a token equal to StrMark.

TokenKindTupEnd identifies a token equal to TupEnd.

TokenKindTupSep identifies a token equal to TupSep.

TokenKindTupStart identifies a token equal to TupStart.

TokenKindUnassigned is used to identify a TokenKind variable which hasn't been assigned to yet.

TokenKindVarEnd identifies a token equal to VarEnd.

TokenKindVarSep identifies a token equal to VarSep.

TokenKindVarStart identifies a token equal to VarStart.

TokenKindWhitespace identifies a token which only contains runes found in the runesWhitespace constant.

Scanner splits the bytes from an io.Reader into tokens for a Parser.

TokenKind represents the kind of token read during a call to Scanner.Scan.