# Constants
Background is for syntax highlight styles based on these tokens.
Cat: Comments.
The list of tokens.
The list of tokens.
SubCat: Preprocessor "comments".
The list of tokens.
The list of tokens.
The list of tokens.
EOF is end of file.
EOL is end of line (typically implicit -- used for rule matching).
EOS is end of statement -- a key meta-token -- in C it is ;, in Go it is either ; or EOL.
Error is an input that could not be tokenized due to syntax error etc.
Cat: Keywords (actual keyword is just the string).
The list of tokens.
The list of tokens.
incl package, import.
The list of tokens.
The list of tokens.
The list of tokens.
Cat: Literals.
The list of tokens.
The list of tokens.
The list of tokens.
SubCat: Literal Numbers.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
SubCat: Literal Strings.
unicode specifiers etc.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
doc-specific strings where syntactically noted.
The list of tokens.
esc sequences within strings.
filename.
in ruby, perl.
interpolated parts of strings in #{foo} in Ruby.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
Cat: Names.
includes slice etc.
e.g., HTML attr.
e.g., true, false -- builtin values..
e.g., this, self.
The list of tokens.
The list of tokens.
includes destructor..
function-like wrappers in python.
special entities.
The list of tokens.
The list of tokens.
for LSP -- not really sure what it is..
The list of tokens.
The list of tokens.
SubCat: Function names.
e.g., __init__ in python.
The list of tokens.
e.g., goto label.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
SubCat: Scoping names.
The list of tokens.
e.g., HTML tag.
SubCat: Type names.
for generics, templates.
SubCat: Value -- data-like elements.
SubCat: NameVar -- variable names.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
None is the nil token value -- for non-terminal cases or TBD.
SubCat: Assign operators.
<-.
=.
--.
:=.
++.
SubCat: Bitwise operators.
&.
&^.
SubCat: Bitwise Assign operators.
&=.
&^=.
|=.
<<=.
>>=.
^=.
~.
|.
<<.
>>.
^.
Cat: Operators.
The list of tokens.
SubCat: List operators.
...
SubCat: Logical operators.
&&.
!.
||.
SubCat: Math operators.
+.
SubCat: Math Assign operators.
+=.
/=.
*=.
%=.
-=.
/.
*.
%.
-.
SubCat: Relational operators.
==.
>.
>=.
<.
<=.
!=.
SubCat: Grouping punctuation.
{.
[.
(.
}.
].
).
SubCat: Separator punctuation.
:.
,.
.
;.
SubCat: String punctuation.
`.
".
\.
'.
Cat: Punctuation.
Cat: Text.
The list of tokens.
The list of tokens.
SubCat: TextStyle (corresponds to Generic in chroma / pygments) todo: look in font deco for more.
strike-through.
italics.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
bold.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
The list of tokens.
# Variables
CatMap is the map into the category level for each token.
Categories.
No description provided by the author
No description provided by the author
Names are the short tag names for each token, used e.g., for syntax highlighting These are based on alecthomas/chroma / pygments.
OpPunctMap provides a lookup of operators and punctuation tokens by their usual string representation.
SubCatMap is the map into the sub-category level for each token.
Sub-Categories.
# Structs
KeyToken combines a token and an optional keyword name for Keyword token types if Tok is in Keyword category, then Key string can be used to check for same keyword.
# Type aliases
KeyTokenList is a list (slice) of KeyTokens.
Tokens is a complete set of lexical tokens that encompasses all programming and text markup languages.