Categorygithub.com/wavetermdev/htmltoken
modulepackage
0.2.0
Repository: https://github.com/wavetermdev/htmltoken.git
Documentation: pkg.go.dev

# README

This is a fork of part of the golang.org/x/net/html package.

v0.2.0

For v0.2.0 we made a more radical change to the tokenizer package.

We added a new syntax to allow attributes to be set with '{}' syntax. Any valid JSON expression is allowed within the curly brackets (this more closely matches JSX syntax).

<div data-num={5}></div>

To support proper decoding in the client, attributes now have a an IsJson bool field which is set to true if an attribute was parsed with the new {} syntax.

If you only need the case-sensitive tokenization for tags/attributes it is recommended to use v0.1.0 and not v0.2.0.

v0.1.0

It is not a complete fork as we only want to modify and change https://pkg.go.dev/golang.org/x/net/html#Tokenizer. So this is the minimal amount of code to get html.Tokenizer working.

The reason for the fork is to allow for returning of case-sensitive tag names and attribute names. The current package normalizes the tag names and attribute names by calling (the equivalent of) strings.ToLower on them before returning them to the caller. We made a very small two line change in token.go to remove those ToLower calls. Other changes involve copying enough code from other files to get all the dependencies satisfied and get it compling again.

Why did we not fork the entire package? Because the rest of the html package is a validating html parser and is quite complicated. As the HTML rules can change over time, it would need to be continually updated and synced with the upstream to keep it compliant. As the actual syntax (tokenization rules) of HTML does not change often, this part of the package is likely much more stable.

# Functions

EscapeString escapes special characters like "<" to become "&lt;".
NewTokenizer returns a new HTML Tokenizer for the given Reader.
NewTokenizerFragment returns a new HTML Tokenizer for the given Reader, for tokenizing an existing element's InnerHTML fragment.

# Constants

A CommentToken looks like <!--x-->.
A DoctypeToken looks like <!DOCTYPE x>.
An EndTagToken looks like </a>.
ErrorToken means that an error occurred during tokenization.
A SelfClosingTagToken tag looks like <br/>.
A StartTagToken looks like <a>.
TextToken means a text node.

# Variables

ErrBufferExceeded means that the buffering limit was exceeded.

# Structs

An Attribute is an attribute namespace-key-value triple.
A Token consists of a TokenType and some Data (tag name for start and end tags, content for text, comments and doctypes).
A Tokenizer returns a stream of HTML Tokens.

# Type aliases

A TokenType is the type of a Token.