JS

This package is a JS lexer (ECMAScript 2020) written in Go. It follows the specification at ECMAScript 2020 Language Specification. The lexer takes an io.Reader and converts it into tokens until the EOF.

Installation

Run the following command

go get -u github.com/tdewolff/parse/v2/js

or add the following import and run project with go get

import "github.com/tdewolff/parse/v2/js"

Lexer

Usage

The following initializes a new Lexer with io.Reader r:

l := js.NewLexer(parse.NewInput(r))

To tokenize until EOF an error, use:

for {
	tt, text := l.Next()
	switch tt {
	case js.ErrorToken:
		// error or EOF set in l.Err()
		return
	// ...
	}
}

Regular Expressions

The ECMAScript specification for PunctuatorToken (of which the / and /= symbols) and RegExpToken depend on a parser state to differentiate between the two. The lexer will always parse the first token as / or /= operator, upon which the parser can rescan that token to scan a regular expression using RegExp().

Examples

package main

import (
	"os"

	"github.com/tdewolff/parse/v2/js"
)

// Tokenize JS from stdin.
func main() {
	l := js.NewLexer(parse.NewInput(os.Stdin))
	for {
		tt, text := l.Next()
		switch tt {
		case js.ErrorToken:
			if l.Err() != io.EOF {
				fmt.Println("Error on line", l.Line(), ":", l.Err())
			}
			return
		case js.IdentifierToken:
			fmt.Println("Identifier", string(text))
		case js.NumericToken:
			fmt.Println("Numeric", string(text))
		// ...
		}
	}
}

Parser

Usage

The following parses a file and returns an abstract syntax tree (AST).

ast, err := js.NewParser(parse.NewInputString("if (state == 5) { console.log('In state five'); }"))

See ast.go for all available data structures that can represent the abstact syntax tree.

License

Released under the MIT license.