# README
storage
Storage related documentation.
Flush consistency model
Flush occurs in the following steps:
- data warm flush
- rotate commit log
- data cold flush
- rotate cold mutable index segments
- flush cold tsdb data and write most files to disk (except checkpoint files)
- flush cold index data to disk and reload
- evict rotated cold mutable index segments
- write tsdb checkpoint files (completes the tsdb cold flush lifecycle)
- data snapshot
- drops rotated commit log when we are done
- index flush
Since we rotate the commit log before we perform a data cold flush and only drop the rotate commit log after data snapshotting is done we guarantee that no writes will be lost if the node crashes. After data cold flush completes, any new cold writes will exist in the active commit log (and not be dropped) when data snapshotting finishes. This is why data snapshotting only needs to snapshot warm data blocks (that need to be flushed).
# Packages
Package block is a generated GoMock package.
Package bootstrap is a generated GoMock package.
No description provided by the author
No description provided by the author
Package index is a generated GoMock package.
Package limits contains paths to enforce read query limits.
No description provided by the author
Package series is a generated GoMock package.
# Functions
DefaultTestOptions provides a single set of test storage options we save considerable memory by doing this avoiding creating default pools several times.
NewAggregateTilesOptions creates new AggregateTilesOptions.
NewColdFlushNsOpts returns a new ColdFlushNsOpts.
NewDatabase creates a new time series database.
NewDatabaseNamespaceDataAccumulator creates a data accumulator for the namespace.
NewEntry returns a new Entry.
NewEntryMetrics builds an entry metrics.
NewLeaseVerifier creates a new LeaseVerifier.
NewMemoryTracker creates a new MemoryTracker.
NewMemoryTrackerOptions creates a new MemoryTrackerOptions.
NewMockBackgroundProcess creates a new mock instance.
NewMockColdFlushNsOpts creates a new mock instance.
NewMockdatabase creates a new mock instance.
NewMockDatabase creates a new mock instance.
NewMockdatabaseBootstrapManager creates a new mock instance.
NewMockdatabaseCleanupManager creates a new mock instance.
NewMockdatabaseColdFlushManager creates a new mock instance.
NewMockdatabaseFileSystemManager creates a new mock instance.
NewMockdatabaseFlushManager creates a new mock instance.
NewMockdatabaseMediator creates a new mock instance.
NewMockdatabaseNamespace creates a new mock instance.
NewMockdatabaseRepairer creates a new mock instance.
NewMockdatabaseShard creates a new mock instance.
NewMockdatabaseShardRepairer creates a new mock instance.
NewMockdatabaseTickManager creates a new mock instance.
NewMockFileOpsProcess creates a new mock instance.
NewMockIndexedErrorHandler creates a new mock instance.
NewMockIndexWriter creates a new mock instance.
NewMockMemoryTracker creates a new mock instance.
NewMockNamespace creates a new mock instance.
NewMockNamespaceHooks creates a new mock instance.
NewMockNamespaceIndex creates a new mock instance.
NewMocknamespaceIndexInsertQueue creates a new mock instance.
NewMockOnColdFlush creates a new mock instance.
NewMockOnColdFlushNamespace creates a new mock instance.
NewMockOptions creates a new mock instance.
NewMockShard creates a new mock instance.
NewMockShardColdFlush creates a new mock instance.
NewMockTileAggregator creates a new mock instance.
NewOptions creates a new set of storage options with defaults.
NewSeriesOptionsFromOptions creates a new set of database series options from provided options.
NewSeriesResolver creates new series ref resolver.
# Constants
AggregateTilesAPI indicates invocation via API call.
AggregateTilesBackfill indicates backfill.
AggregateTilesRegular indicates regular process.
BootstrapNotStarted indicates bootstrap has not been started yet.
Bootstrapped indicates a bootstrap process has completed.
Bootstrapping indicates bootstrap process is in progress.
# Variables
AggregateTilesProcesses is a list of available AggregateTilesProcess values.
ErrDatabaseLoadLimitHit is the error returned when the database load limit is hit or exceeded.
# Structs
AggregateTilesOptions is the options for large tile aggregation.
BootstrapEnqueueOptions is options to pass to BootstrapEnqueue when enqueuing a bootstrap.
BootstrapResult is a bootstrap result.
DatabaseBootstrapState stores a snapshot of the bootstrap state for all shards across all namespaces at a given moment in time.
DebugMemorySegmentsOptions is a set of options to debug memory segments.
Entry is the entry in the shard ident.ID -> series map.
EntryMetrics are metrics for an entry.
MemoryTrackerOptions are the options for the MemoryTracker.
MockBackgroundProcess is a mock of BackgroundProcess interface.
MockBackgroundProcessMockRecorder is the mock recorder for MockBackgroundProcess.
MockColdFlushNsOpts is a mock of ColdFlushNsOpts interface.
MockColdFlushNsOptsMockRecorder is the mock recorder for MockColdFlushNsOpts.
Mockdatabase is a mock of database interface.
MockDatabase is a mock of Database interface.
MockdatabaseBootstrapManager is a mock of databaseBootstrapManager interface.
MockdatabaseBootstrapManagerMockRecorder is the mock recorder for MockdatabaseBootstrapManager.
MockdatabaseCleanupManager is a mock of databaseCleanupManager interface.
MockdatabaseCleanupManagerMockRecorder is the mock recorder for MockdatabaseCleanupManager.
MockdatabaseColdFlushManager is a mock of databaseColdFlushManager interface.
MockdatabaseColdFlushManagerMockRecorder is the mock recorder for MockdatabaseColdFlushManager.
MockdatabaseFileSystemManager is a mock of databaseFileSystemManager interface.
MockdatabaseFileSystemManagerMockRecorder is the mock recorder for MockdatabaseFileSystemManager.
MockdatabaseFlushManager is a mock of databaseFlushManager interface.
MockdatabaseFlushManagerMockRecorder is the mock recorder for MockdatabaseFlushManager.
MockdatabaseMediator is a mock of databaseMediator interface.
MockdatabaseMediatorMockRecorder is the mock recorder for MockdatabaseMediator.
MockdatabaseMockRecorder is the mock recorder for Mockdatabase.
MockDatabaseMockRecorder is the mock recorder for MockDatabase.
MockdatabaseNamespace is a mock of databaseNamespace interface.
MockdatabaseNamespaceMockRecorder is the mock recorder for MockdatabaseNamespace.
MockdatabaseRepairer is a mock of databaseRepairer interface.
MockdatabaseRepairerMockRecorder is the mock recorder for MockdatabaseRepairer.
MockdatabaseShard is a mock of databaseShard interface.
MockdatabaseShardMockRecorder is the mock recorder for MockdatabaseShard.
MockdatabaseShardRepairer is a mock of databaseShardRepairer interface.
MockdatabaseShardRepairerMockRecorder is the mock recorder for MockdatabaseShardRepairer.
MockdatabaseTickManager is a mock of databaseTickManager interface.
MockdatabaseTickManagerMockRecorder is the mock recorder for MockdatabaseTickManager.
MockFileOpsProcess is a mock of FileOpsProcess interface.
MockFileOpsProcessMockRecorder is the mock recorder for MockFileOpsProcess.
MockIndexedErrorHandler is a mock of IndexedErrorHandler interface.
MockIndexedErrorHandlerMockRecorder is the mock recorder for MockIndexedErrorHandler.
MockIndexWriter is a mock of IndexWriter interface.
MockIndexWriterMockRecorder is the mock recorder for MockIndexWriter.
MockMemoryTracker is a mock of MemoryTracker interface.
MockMemoryTrackerMockRecorder is the mock recorder for MockMemoryTracker.
MockNamespace is a mock of Namespace interface.
MockNamespaceHooks is a mock of NamespaceHooks interface.
MockNamespaceHooksMockRecorder is the mock recorder for MockNamespaceHooks.
MockNamespaceIndex is a mock of NamespaceIndex interface.
MocknamespaceIndexInsertQueue is a mock of namespaceIndexInsertQueue interface.
MocknamespaceIndexInsertQueueMockRecorder is the mock recorder for MocknamespaceIndexInsertQueue.
MockNamespaceIndexMockRecorder is the mock recorder for MockNamespaceIndex.
MockNamespaceMockRecorder is the mock recorder for MockNamespace.
MockOnColdFlush is a mock of OnColdFlush interface.
MockOnColdFlushMockRecorder is the mock recorder for MockOnColdFlush.
MockOnColdFlushNamespace is a mock of OnColdFlushNamespace interface.
MockOnColdFlushNamespaceMockRecorder is the mock recorder for MockOnColdFlushNamespace.
MockOptions is a mock of Options interface.
MockOptionsMockRecorder is the mock recorder for MockOptions.
MockShard is a mock of Shard interface.
MockShardColdFlush is a mock of ShardColdFlush interface.
MockShardColdFlushMockRecorder is the mock recorder for MockShardColdFlush.
MockShardMockRecorder is the mock recorder for MockShard.
MockTileAggregator is a mock of TileAggregator interface.
MockTileAggregatorMockRecorder is the mock recorder for MockTileAggregator.
NamespaceRepairOptions is a set of repair options for repairing a namespace.
NewEntryOptions supplies options for a new entry.
SeriesWrite is a result of a series write.
ShardSnapshotResult is a result from a shard snapshot.
WritableSeriesOptions defines writable series options.
# Interfaces
BackgroundProcess is a background process that is run by the database.
ColdFlushNsOpts are options for OnColdFlush.ColdFlushNamespace.
Database is a time series database.
FileOpsProcess is a background process that is run by the database.
IndexedErrorHandler can handle individual errors based on their index.
IndexWriter accepts index inserts.
MemoryTracker tracks memory.
Namespace is a time series database namespace.
NamespaceHooks allows dynamic plugging into the namespace lifecycle.
NamespaceIndex indexes namespace writes.
OnColdFlush can perform work each time a series is flushed.
OnColdFlushNamespace performs work on a per namespace level.
Options represents the options for storage.
Shard is a time series database shard.
ShardColdFlush exposes a done method to finalize shard cold flush by persisting data and updating shard state/block leases.
TileAggregator is the interface for AggregateTiles.
# Type aliases
AggregateTilesProcess identifies the process used for the aggregation.
BootstrapCompleteFn is a callback for when bootstrap is complete when using BootstrapEnqueue method.
BootstrapState is an enum representing the possible bootstrap states for a shard.
FileOpsProcessFn is a file ops process function.
GetNamespaceFn will return a namespace for a given ID if present.
NamespaceBootstrapStates stores a snapshot of the bootstrap state for all shards across a number of namespaces at a given moment in time.
NamespacesByID is a sortable slice of namespaces by ID.
NewBackgroundProcessFn is a function that creates and returns a new BackgroundProcess.
NewTileAggregatorFn creates a new TileAggregator.
OnColdFlushDone is a callback that performs house keeping once cold flushing completes.
OptionTransform transforms given Options.
PageToken is an opaque paging token.
ShardBootstrapStates stores a snapshot of the bootstrap state for all shards for a given namespace.