OpenLineage SDK for Go

This library contains an SDK for the OpenLineage project. Large parts are generated from the OpenLineage specification.

The root of the library contains the core OpenLineage types, facets are located in pkg/facets.

Installing

go get github.com/ThijsKoot/openlineage-go

Usage

runID := uuid.Must(uuid.NewV7())
facet := facets.NewProcessingEngine("v1.2.3").
    WithName("my-go-engine")

openlineage.NewRunEvent(openlineage.EventTypeStart, runID, "my-job").
    WithRunFacets(facet).
    Emit()

Configuring

All configuration is captured in openlineage.ClientConfig, which can be read from the following sources:

• YAML configuration file
• Environment variables
• Variables set in code

// config which prints pretty-printed events to console
cfg := openlineage.ClientConfig{
	Transport: transport.Config{
		Type: transport.TransportTypeConsole,
		Console: &transport.ConsoleConfig{
			PrettyPrint: true,
		},
	},
}

// create a new client using this configuration
client, err := openlineage.NewClient(cfg)

File

See below for how to read a configuration file and its format.

config, err := openlineage.ConfigFromFile("my/openlineage/config.yaml")

# example file
namespace: default # default
disabled: false # default

transport:
  type: console
  console:
    prettyPrint: true

  http:
    url: https://foo
    endpoint: api/v1/lineage # default
    apiKey: ""

Environment

Use openlineage.ConfigFromEnv to read configuration values from the environment. If OPENLINEAGE_CONFIG is specified, it is processed first. Values from the environment are applied afterwards.

cfg, err := openlineage.ConfigFromEnv()

The table below contains an overview of all environment variables.

Transport

The SDK supports pluggable transports via the transport.Transport interface.

type Transport interface {
	Emit(ctx context.Context, event any) error
}

The built-in transports are HTTP and Console. HTTP uses POST-requests to an endpoint, optionally secured with bearer authentication. Console prints JSON-formatted events to stdout.

Run API

The run package contains a tracing-like API modeled loosely after OpenTelemetry's. It is separate from the core functionality because of its opinionated design. The purpose of this package is to provide an ergonomic way of emitting events within code with less verbosity. It also allows for implicit passing of Runs with context.Context to avoid having to manually propagate a run context.