Categorygithub.com/canonicalltd/go-dqlite
modulepackage
0.9.4
Repository: https://github.com/canonicalltd/go-dqlite.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
20
Dependents
0
Files
900 SLOC

# README

go-dqlite Build Status Coverage Status Go Report Card GoDoc

This repository provides the dqlite Go package, which can be used to replicate a SQLite database across a cluster, using the Raft algorithm.

Design higlights

  • No external processes needed: dqlite is just a Go library, you link it it to your application exactly like you would with SQLite.
  • Replication needs a SQLite patch which is not yet included upstream.
  • The Go Raft package from Hashicorp is used internally for replicating the write-ahead log frames of SQLite across all nodes.

How does it compare to rqlite?

The main differences from rqlite are:

  • Full support for transactions
  • No need for statements to be deterministic (e.g. you can use time())
  • Frame-based replication instead of statement-based replication, this means in dqlite there's more data flowing between nodes, so expect lower performance. Should not really matter for most use cases.

Status

This is beta software for now, but we'll get to rc/release soon.

Demo

To see dqlite in action, make sure you have the following dependencies installed:

  • Go (tested on 1.8)
  • gcc
  • any dependency/header that SQLite needs to build from source
  • Python 3

Then run:

go get -d github.com/CanonicalLtd/dqlite
cd $GOPATH/src/github.com/CanonicalLtd/dqlite
make dependencies
./run-demo

This should spawn three dqlite-based nodes, each of one running the code in the demo Go source.

Each node inserts data in a test table and then dies abruptly after a random timeout. Leftover transactions and failover to other nodes should be handled gracefully.

While the demo is running, to get more details about what's going on behind the scenes you can also open another terminal and run a command like:

watch ls -l /tmp/dqlite-demo-*/ /tmp/dqlite-demo-*/snapshots/

and see how the data directories of the three nodes evolve in terms SQLite databases (test.db), write-ahead log files (test.db-wal), raft logs store (raft.db), and raft snapshots.

Documentation

The documentation for this package can be found on Godoc.

FAQ

Q: How does dqlite behave during conflict situations? Does Raft select a winning WAL write and any others in flight are aborted?

A: There can't be a conflict situation. Raft's model is that only the leader can append new log entries, which translated to dqlite means that only the leader can write new WAL frames. So this means that any attempt to perform a write transaction on a non-leader node will fail with a sqlite3x.ErrNotLeader error (and in this case clients are supposed to retry against whoever is the new leader).

Q: When not enough nodes are available, are writes hung until consensus?

A: Yes, however there's a (configurable) timeout. This is a consequence of Raft sitting in the CP spectrum of the CAP theorem: in case of a network partition it chooses consistency and sacrifices availability.

# Functions

DefaultServerStore creates a new ServerStore using the given filename to open a SQLite database, with default names for the schema, table and column parameters.
Leave a cluster.
NewDriver creates a new dqlite driver, which also implements the driver.Driver interface.
NewServer creates a new Server instance.
NewServerStore creates a new ServerStore.
WithConnectionBackoffCap sets the maximum connection retry backoff value, (regardless of the backoff factor) for retrying failed connection attempts.
WithConnectionBackoffFactor sets the exponential backoff factor for retrying failed connection attempts.
WithConnectionTimeout sets the connection timeout.
WithContext sets a global cancellation context.
WithContextTimeout sets the default client context timeout when no context deadline is provided.
WithDialFunc sets a custom dial function.
WithLogFunc sets a custom logging function.
WithServerDialFunc sets a custom dial function for the server.
WithServerLogFunc sets a custom log function for the server.
WithServerWatchFunc sets a function that will be invoked whenever this server acquires leadership.

# Constants

States.
States.
States.
Available logging levels.
Available logging levels.
Available logging levels.
Available logging levels.
States.

# Variables

ErrNoAvailableLeader is returned as root cause of Open() if there's no leader available in the cluster.
ErrServerCantBootstrap is returned by Server.Bootstrap() if the server has already a raft configuration.
NewInmemServerStore creates ServerStore which stores its data in-memory.

# Structs

Conn implements the sql.Conn interface.
DatabaseServerStore persists a list addresses of dqlite servers in a SQL table.
Driver perform queries against a dqlite server.
Result is the result of a query execution.
Rows is an iterator over an executed query's results.
Server implements the dqlite network protocol.
Stmt is a prepared statement.
Tx is a transaction.

# Type aliases

DialFunc is a function that can be used to establish a network connection.
DriverError is returned in case of database errors.
DriverOption can be used to tweak driver parameters.
InmemServerStore keeps the list of target gRPC SQL servers in memory.
LogFunc is a function that can be used for logging.
LogLevel defines the logging level.
ServerInfo holds information about a single server.
ServerOption can be used to tweak server parameters.
ServerStore is used by a dqlite client to get an initial list of candidate dqlite server addresses that it can dial in order to find a leader dqlite server to use.
WatchFunc notifies about state changes.