Categorygithub.com/beneath-hq/beneathinfraenginedriver
package
1.0.0-rc.5
Repository: https://github.com/beneath-hq/beneath.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
7
Dependents
11
Files
235 SLOC

# README

engine/driver/

Drivers connect Beneath functionality to the underlying features offered by a data management system. They allow Beneath to offer a uniform user experience while benefiting from the unique advantages of different data systems.

Driver types

Fundamentally, Beneath currently distinguishes between three types of infrastructure components:

  • MessageQueue: a message bus that supports at-least-once delivery
  • LookupService: a low-latency read/write service for storing log data and indexes; must support range reads
  • WarehouseService: an OLAP service that can execute large (long-running) analytical queries efficiently

The features expected of each of these types are described in more detail in the driver interfaces in engine/driver/driver.go.

Note that one underlying data system may implement several types at once. For example, at smaller scales of data (100s GB), a Postgres database could efficiently serve as both a message queue, lookup service and warehouse service.

In the future, we may want to extend this list with fundamentally different data paradigms, such as a full-text search service or a graph data service.

Existing drivers

  • bigquery implements Google Cloud BigQuery as a highly-scalable WarehouseService driver.
  • bigtable implements Google Cloud Bigtable as a highly-scalable LookupService driver.
  • mock implements a WarehouseService driver that simply discards data and pretends to have successfully stored it.
  • pubsub implements Google Cloud Pub/Sub as a highly-scalable MessageQueue driver.
  • postgres has not yet been implemented, but is intended to implement all driver types on Postgres for use in development and small-scale (100s GB) self-hosting deployments.

Implementing a new driver

To implement a new driver, add a package named after the system to engine/driver/, and implement the relevant driver interfaces in engine/driver/driver.go. Look at existing drivers for inspiration.

Here's a list of drivers we're very interested in developing

  • Postgres as a driver for MessageQueue, LookupService and WarehouseService (as described above)
  • Apache Cassandra as a LookupService
  • Apache Kafka as a MessageQueue and possibly a LookupService (leveraging Kafka Streams and/or ksqlDB)
  • Amazon Redshift as a WarehouseService

# Packages

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author

# Functions

AddDriver registers a new driver (by passing the driver's constructor).

# Constants

DoneWarehouseJobStatus represents a warehouse job that has completed; if the job failed, the job will have a non-nil error.
PendingWarehouseJobStatus represents a warehouse job that has not yet been started.
RunningWarehouseJobStatus represents a warehouse job that is currently running.
UnspecifiedWarehouseJobStatus is the null value for WarehouseJobStatus.
Consts for UsageService.
Consts for UsageService.
Consts for UsageService.

# Variables

Drivers is a registry of drivers.

# Interfaces

LookupService encapsulates functionality necessary to lookup, replay and subscribe to data.
Project encapsulates metadata about a Beneath project.
Record todo.
RecordsIterator allows iterating over a list of records in various formats.
Service encapsulates functionality expected of components that store instance data in Beneath.
Stream encapsulates metadata about a Beneath stream.
StreamInstance encapsulates metadata about a Beneath stream instance.
UsageService stores and retrieves aggregated usage by timestamp for different resources in Beneath.
WarehouseJob represents a query job submitted to a WarehouseService.
WarehouseService encapsulates functionality to analytically query records in an instance.

# Type aliases

Driver is a function that creates an engine service (LookupService or WarehouseService) from a config object.
UsageLabel defines valid labels for UsageService.
WarehouseJobStatus represents the current status of a warehouse job.