Categorygithub.com/grafana/thema
modulepackage
0.0.0-20240605110052-2016107581da
Repository: https://github.com/grafana/thema.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
23
Dependents
62
Files
2.3k SLOC

# README

Thema

[!WARNING] This repository is no longer actively maintained or supported. If you are interested in continuing the development of this project, feel free to fork the repository and customize it to your needs.

Thema is a system for writing schemas. Much like JSON Schema or OpenAPI, it is general-purpose and its most obvious application is as an IDL. However, those systems treat changing schemas as out of scope: a single version of a schema for some object is the atomic unit, and versioning is left to opaque strings in external systems like git or HTTP. Thema, by contrast, makes schema change a first-class system property: the atomic unit is the set of schema for some object, iteratively appended to over time as requirements evolve.

Thema's approach is novel, so an analogy to the familiar may help. "Branching by abstraction" suggests that you refactor large applications not with long-running VCS branches and big-bang merges, but by letting old and new code live side-by-side on main, and choosing between them with logical gates, like feature flags. Thema is "schema versioning by abstraction": all versions of a schema live side-by-side on main, within logical structures Thema defines.

This holistic view allows Thema to act like a typechecker, but for change-safety between schema versions: either schema versions must be backwards compatible, or there must exist logic to translate a valid instance of schema from one schema version to the next. CUE, the language in which Thema schemas are written, allows Thema to mechanically verify these properties.

These capabilities make Thema a general framework for decoupling the evolution of communicating systems. This can be outward-facing: Thema's guardrails allow anyone to create APIs with Stripe's renowned backwards compatibility guarantees. Or it can be inward-facing: or to change the messages passed in a mesh of microservices without intricately orchestrating deployment.

Learn more in our docs, or in this overview video! (Some things have been renamed since that video, but the logic is unchanged.)

Usage

Thema defines the way schemas are written, organizing each object's history into a "lineage." Once authored, Thema also provides tools for working with lineages via a few basic operations. There are a few different usage patterns, all largely equivalent in capability:

  • CLI: a CLI command that provides access to Thema's basic operations, one lineage per invocation. Use it for fast exploration and testing of schemas, or as a tool in CI.
  • Server: An HTTP server that provides access to Thema's basic operations for a configurable set of lineages. Run it as a stateless sidecar in your infrastructure or microservice mesh.
  • Library: a library, importable in your application code, that provides a convenient interface to Thema's basic operations, as well as helpers for common usage patterns. Naturally the most flexible, and the recommended approach for creating new helpers, such as code generators, API generators, or a whole Kubernetes operator framework. (Currently only for Go1)

The CLI and server modes are bundled together in the thema command. To install:

go install github.com/grafana/thema/cmd/thema@latest

Maturity

Thema is a young project. The goals are large, but bounded: we will know when the core system is complete. And it mostly is, now - though some breaking changes to how schemas are written are planned before reaching stability.

It is not yet recommended to replace established, stable systems with Thema, but experimenting with doing so is reasonable (and appreciated!). For newer projects, Thema may be a good choice today; the decision is likely to come down to whether the long-term benefit of a simpler architecture for authoring, composing and evolving schema will offset the short-term cost of some incomplete functionality and breaking changes.

Prior/Related Art

A number of systems partially overlap with Thema - for some data, rolling together a set of schema with the relations between those schema.

  • Project Cambria - Thema's closest analogue. Limited in verifiability by (intentionally) being without a notion of linear schema ordering and versioning, and because schema and translations are written in a Turing complete language (Typescript).
  • Kubernetes resources and webhook conversions - Similar goals: multiple versions of resources (schema) and convertibility between them. Limited in verifiability by relying on convention for grouping schemas, and by expressing translation in a Turing complete language (Go).
  • Stripe's HTTP API - exhibits the backwards compatibility properties an API can have that arise from a schema system with translatability.

Footnotes

  1. Using Thema as a library in a language depends on a CUE evaluator for that language. Currently, the only CUE evaluator is written in Go. ↩

# Packages

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
Package vmux provides utilities that make it easy to implement "version multiplexing" with your Thema lineages.

# Functions

AssignableTo indicates whether all valid instances of the provided Thema schema can be assigned to the provided Go type.
BindInstanceType produces a TypedInstance, given an Instance and a TypedSchema derived from its Instance.Schema().
BindLineage takes a raw [cue.Value], checks that it correctly follows Thema's invariants, such as translatability and backwards compatibility version numbering.
BindType produces a [TypedSchema], given a [Schema] that is [AssignableTo] the [Assignee] type parameter T.
ImperativeLenses takes a slice of [ImperativeLens].
IsAppendOnly returns nil if the new lineage only contains new schemas compared to the old one.
LatestVersion returns the version number of the newest (largest) schema version in the provided lineage.
LatestVersionInSequence returns the version number of the newest (largest) schema version in the provided sequence number.
NewRuntime parses, loads and builds a full CUE instance/value representing all of the logic in the Thema CUE package (github.com/grafana/thema), and returns a Runtime instance ready for use.
ParseSyntacticVersion parses a canonical representation of a [SyntacticVersion] (e.g.
SchemaP returns the schema identified by the provided version.
SkipBuggyChecks indicates that [BindLineage] should skip validation checks which have known bugs (e.g.
SV creates a [SyntacticVersion].

# Variables

CueFS contains the raw .cue files that comprise the core thema system, making them available directly in Go.
CueJointFS contains the raw thema .cue files, as well as the cue.mod directory.
ErrPointerDepth indicates that a Go type having pointer indirection depth greater than 1, such as **struct{ V: string }) was provided to a Thema func that checks assignability, such as [BindType].

# Structs

FieldRef identifies a path/field and the value in it within a Lacuna.
ImperativeLens is a lens transformation defined as a Go function, rather than in native CUE alongside the lineage.
Instance represents data that is a valid instance of a Thema [Schema].
A Lacuna represents a semantic gap in a Lens's mapping between schemas.
Runtime holds the set of CUE constructs available in the Thema CUE package, allowing Thema's Go code to internally reuse the same native CUE functionality.
TypedInstance represents data that is a valid instance of a Thema [TypedSchema].

# Interfaces

ConvergentLineage is a lineage where exactly one of its contained schemas is associated with a Go type - a TypedSchema[Assignee], as returned from [BindType].
A CUEWrapper wraps a cue.Value, and can return that value for inspection.
A Lineage is the top-level container in Thema, holding the complete evolutionary history of a particular kind of object: every schema that has ever existed for that object, and the lenses that allow translating between those schema versions.
Schema represents a single, complete schema from a thema lineage.
TranslationLacunas defines common patterns for unary and composite lineages in the lacunas their translations emit.
TypedSchema is a Thema schema that has been bound to a particular Go type, per Thema's assignability rules.

# Type aliases

Assignee is a type constraint used by Thema generics for type parameters where there exists a particular Schema that is [AssignableTo] the type.
A BindOption defines options that may be specified only at initial construction of a [Lineage] via [BindLineage].
A ConvergentLineageFactory is the same as a LineageFactory, but for a ConvergentLineage.
LacunaType assigns numeric identifiers to different classes of Lacunas.
A LineageFactory returns a [Lineage], which is immutably bound to a single instance of #Lineage declared in CUE.
SyntacticVersion is a two-tuple of uints describing the position of a schema within a lineage.