# README
oplog
oplog is a package for writing operation log (oplog) entries for the purpose of replication and verification of the data stored in the Boundary RDBMS.
Usage
// you must init the ticket in its own transaction. You only need
// to init a ticket once in the database. It doesn't need to happen for
// every connection. Once it's persistent in the database, you can simply Get it.
initTx := db.Begin()
ticketer := &GormTicketer{Tx: initTx}
err = ticketer.InitTicket("users")
// if there's no error, then commit the initialized ticket
initTx.Commit()
userCreate := oplog_test.TestUser{
TestUser: oplog_test.TestUser{
Name: loginName,
},
}
tx := db.Begin()
// write the user to the database
tx.Create(&userCreate)
ticketer = &GormTicketer{Tx: db}
// get a ticket for writing users to the oplog
ticket, err := ticketer.GetTicket("users")
// create an entry for the oplog with the entry's metadata (likely the entry's Scope)
newLogEntry := NewEntry(
"test-users",
[]Metadata{
Metadata{
Key: "deployment",
Value: "amex",
},
Metadata{
Key: "project",
Value: "central-info-systems",
},
},
cipherer, // wrapping.Wrapper
ticketer,
)
// write an entry with N messages (variadic parameter) in the order they were sent to the database
_, err = newLogEntry.WriteEntryWith(
context.Background(),
&GormWriter{tx},
ticket,
&Message{Message: &userCreate, TypeName: "user", OpType: OpType_CREATE_OP},
)
// if there's an error writing the oplog then roll EVERYTHING back
if err != nil {
tx.Rollback()
}
// no err means you can commit all the things.
tx.Commit()
TBD/TODO
We need to discuss and decide how Boundary is going to handle the following oplog things:
- SQL migrations: you'll find the package's SQL migrations under: ./db/schema We need to decide how Boundary will manage migrations across the system and we will likely need to reference this package's migrations somehow.
oplog entry
Example Oplog Entry for the Target Aggregate
┌────────────────────────────────────────────────┐
│ FIFO with []byte buffer │
│ │
│┌─Msg 4────┐┌─Msg 3────┐┌─Msg 2────┐ │
││ ││ ││ │ │
││ Tags ││ Host ││ HostSet │ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││ ││││ ││││ ││ │
│││ Tag ││││ Host ││││HostSet ││ │
│││protobuf││││protobuf││││protobuf││ │
│││ ││││ ││││ ││ │
│││ ││││ ││││ ││ │
││└────────┘││└────────┘││└────────┘│ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││typeName││││typeName││││typeName││ │
│││ Tag ││││ Host ││││HostSet ││ │
││└────────┘││└────────┘││└────────┘│ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││ OpType ││││ OpType ││││ OpType ││ │
│││ Create ││││ Create ││││ Create ││ │
││└────────┘││└────────┘││└────────┘│ │
│└──────────┘└──────────┘└──────────┘ │
└────────────────────────────────────────────────┘
oplog tables
oplog tables:
as the diagram shows, we can split the
oplog_entries into multiple tables if
needed for performance.
┌────────────────┐
│┌───────────────┴┐
││┌───────────────┴┐ ┌────────────────┐
│││ oplog_entries │ │ oplog_ticket │
││├────────────────┤ ├────────────────┤
│││id │╲ │id │
│││aggregate_name │──┼───────┼ │aggregate_name │
└┤│data │╱ │version │
└┤ │ │ │
└────────────────┘ └────────────────┘
┼
│
│
┼
╱│╲
┌────────────────┐
│ oplog_metadata │
├────────────────┤
│id │
│entry_id │
│key │
│value │
└────────────────┘
oplog optimistic locking using tickets
Alice's Database
transaction Bob's transaction
│ │ │
│─────────BEGIN──────────────────▶│ │
│ │◀───────────────BEGIN─────────────────┤
│ ┌───────────┴───────────┐ │
│ │ ticket version:1 │ │
│ └───────────┬───────────┘ │
│ Select oplog-ticket for │ │
├──────────────"Target"──────────▶│ Select oplog-ticket for │
│ │◀──────────────"Target"───────────────┤
│ │ │
│ Write to Tables in │ │
├───────Target Aggregate─────────▶│ │
│ │ Write to Tables in │
│ │◀─────────Target Aggregate────────────┤
│ │ │
│ │ │
│ Update Ticket │ │
├────Version = 2 where───────────▶│ │
│ Version = 1 ┌─────────┴─────────┐ │
│ │ ticket version: 2 │ │
│ └─────────┬─────────┘ │
│ │ Update Ticket │
│ │◀───────────Version = 2 where─────────┤
│ │ Version = 1 ┌────┴──────────┐
│ │ │ update failed │
├────────────Commit──────────────▶│ └────┬──────────┘
│ │ │
│ │ │
│ │ │
│ │◀────────────Rollback─────────────────┤
│ │ │
│ │ │
```
# Packages
Package oplog_test provides some gorm helper funcs for testing oplog database integrations.
Package store provides storage types/behavior for the oplog.
# Functions
GetOpts - iterate the inbound Options and return a struct.
NewEntry creates a new Entry.
NewTicketer creates a new ticketer that uses dbw for storage.
NewTypeCatalog creates a catalog with the types you pass in.
TestOplogDeleteAllEntries allows you to delete all the entries for testing.
WithAggregateNames enables/disables the use of multiple aggregate names for Ticketers.
WithFieldMaskPaths represents an optional set of symbolic field paths (for example: "f.a", "f.b.d") used to specify a subset of fields that should be updated.
WithOperationOptions represents an optional set dbw.Options.
WithSetToNullPaths represents an optional set of symbolic field paths (for example: "f.a", "f.b.d") used to specify a subset of fields that should be set to null.
# Constants
No description provided by the author
OP_TYPE_CREATE defines a create operation.
OP_TYPE_CREATE_ITEMS defines a create operation for multiple items.
OP_TYPE_DELETE defines a delete operation.
OP_TYPE_DELETE_ITEMS defines a delete operation for multiple items.
OP_TYPE_UNSPECIFIED defines an unspecified operation.
OP_TYPE_UPDATE defines an update operation.
Version of oplog entries (among other things, it's used to manage upgrade compatibility when replicating)
v1: initial version v2: adds the new Message.Opts.
# Variables
No description provided by the author
Enum value maps for OpType.
Enum value maps for OpType.
# Structs
AnyOperation provides a message for anything and the type of operation it represents.
Column represents a table Column.
Columns defines a set of column properties.
ColumnValue defines a column value.
No description provided by the author
No description provided by the author
No description provided by the author
ColumnValues defines a set of column value properies.
DbwTicketer defines a ticketer that uses the dbw pkg for database operations.
Entry represents an oplog entry.
ExprValue defines an expr value that can be used as a column value.
Message wraps a proto.Message with some other bits like operation type, paths and options.
OperationOptions represent operations options which can/will affect the oplog write operation.
Queue provides a FIFO queue.
Type provides the ability to associate an interface with a Type.Name which will decouple the interface from it's reflection type string, so you can refactor the type name without breaking the catalog.
WithOnConflict defines the parameters needed for an sql "on conflict clause".
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
Writer provides a database writer for oplog operations.
# Interfaces
ReplayableMessage defines an interface for messages that can be replayed from the oplog entries.
Ticketer provides an interface to storage for Tickets, so you can easily substitute your own ticketer.
# Type aliases
Metadata provides meta information about the Entry.
Option - how Options are passed as arguments.
Options = how options are represented.
OpType provides the type of database operation the Any message represents (create, update, delete).
TypeCatalog is an abstraction for dealing with oplog data and their underlying types.