package
2.6.9+incompatible
Repository: https://github.com/oneconcern/datamon.git
Documentation: pkg.go.dev
# Functions
BatchSize sets the batch window to fetch core objects.
BundleDescriptor sets the descriptor for this bundle.
BundleID defines the ID for a bundle.
BundleWithMetrics toggles metrics on a core Bundle object.
BundleWithRetry toggles exponential backoff retry logic on upload of core Bundle object.
BundleWithVerifyBlob toggles root key verification when uploading (enabled by default).
BundleWithVerifyHash toggles hash verification when downloading (enabled by default).
ConcurrentFileDownloads tunes the level of concurrency when downloading bundle files.
ConcurrentFilelistDownloads tunes the level of concurrency when retrieving the list of files in a bundle.
ConcurrentFileUploads tunes the level of concurrency when uploading bundle files.
ConcurrentList sets the max level of concurrency to retrieve core objects.
ConsumableStore defines the consumable storage for a bundle.
ContextStores defines the set of stores to build a context for a bundle.
CreateDiamond persists an initialized diamond with a repo descriptor.
CreateRepo persists a repository with a repo descriptor and some context's stores.
CreateSplit persists a new split for some initialized diamond for a repo.
DeleteBundle removes a single bundle from a repo.
DeleteEntriesFromRepo remove a list of file entries from all bundles in a repo.
DeleteLabel removes a single label from a repo.
DeleteRepo removes a repository from metadata.
DiamondBundleID defines the ID for the bundle produced by the diamond
NOTE: this require the package to be built with the bundle_preserve tag to enable this feature.
DiamondConcurrentFileDownloads tunes the level of concurrency when downloading diamond files.
DiamondConcurrentFilelistDownloads tunes the level of concurrency when retrieving the list of files in a diamond.
DiamondConcurrentFileUploads tunes the level of concurrency when uploading diamond files.
DiamondContextStores defines the set of stores to build a context for a diamond.
DiamondDescriptor defines the descriptor of the diamond.
DiamondExists checks if a diamond exists on a repo.
DiamondLogger injects a logging facility into diamond core operations.
DiamondMessage defines the message of the bundle descriptor created by the diamond.
DiamondRepo defines the repo a diamond belongs to.
DiamondWithMetrics toggles metrics on a core Diamond object.
Diff shows the differences between two bundles.
DownloadMetadata from the archive to main memory.
GetBundleStore extracts the metadata store for bundles from some context's stores.
GetDiamond retrieves a diamond.
GetDiamondStore selects the metadata store for diamonds from a context
In the current setup, diamond metadata are located in the vmetadata store.
GetLabelStore tells which store holds label metadata.
GetLatestBundle returns the latest bundle descriptor from a repo.
GetRepo retrieves a repository.
GetRepoDescriptorByRepoName returns the descriptor of a named repo.
GetRepoStore extracts the metadata store from some context's stores
NOTE: this is redundant with GetBundleStore.
GetSplit retrieves a split.
GetSplitStore selects the metadata store for splits from a context
In the current setup, split metadata are located in the vmetadata store.
LabelDescriptor sets the descriptor for this label.
LabelWithMetrics toggles metrics for this label.
LabelWithVersion indicates we want to retrieve a specific version of this label object.
ListBundles returns a list of bundle descriptors from a repo.
ListBundlesApply applies some function to the retrieved bundles, in lexicographic order of keys.
ListContexts provides the list of available contexts in a remote configuration store, sorted.
ListDiamonds yields all ongoing diamonds on a repo.
ListDiamondsApply applies some function to the retrieved diamonds, ordered by completion time.
ListLabels returns all labels from a repo.
ListLabelsApply applies some function to the retrieved labels, in lexicographic order of keys.
ListRepos returns all repos from a store.
ListReposApply applies some function to the retrieved repos, in lexicographic order of keys.
ListSplits yields all ongoing splits on a repo and a given diamond.
ListSplitsApply applies some function to the retrieved splits, ordered by completion time.
Logger injects a logging facility into bundle core operations.
NewBundle creates a new bundle.
NewDiamond builds a new diamond instance.
NewLabel builds a new label with a descriptor.
NewSplit builds a split for core operations (get, ...) and makes Bundle capabilities available to a Split.
PopulateFiles populates a ConsumableStore with the metadata for this bundle.
Publish a bundle to a consumable store.
PublishFile publish a single bundle file to a ConsumableStore.
PublishMetadata from the archive to the consumable store.
PublishSelectBundleEntries publish a selected list of entries from a bundle to a ConsumableStore, based on a predicate filter.
PurgeBuildReverseIndex creates or update a reverse-lookip index of all used blob keys.
PurgeDeleteUnused deletes blob entries that are not referenced by the reserve-lookup index.
PurgeDropReverseIndex drop all index chunks in the metadata store
TODO(fred): nice to have - should factorize index chunk iterations.
PurgeLock sets a purge job lock on the metadata store.
PurgeUnlock removes the purge job lock from the metadata store.
RenameRepo renames a repository in metadata.
Repo defines the repo a bundle belongs to.
RepoExists checks for the existence of a repository by name in storage.
No description provided by the author
SkipMissing indicates that bundle retrieval errors should be ignored.
SplitConcurrentFileUploads tunes the level of concurrency when uploading files.
SplitConsumableStore defines the consumable storage for a split.
SplitDescriptor sets the model descriptor for a Split.
SplitKeyFilter defines a filter on the keys to be uploaded.
SplitKeyIterator defines a custom key iterator function to upload files.
SplitLogger sets a logger on the split object.
SplitMustExist sets a split which must already be existing on metadata storage when created.
SplitSkipMissing indicates that file retrieval errors should be ignored.
SplitWithMetrics toggles metrics on a core Split object.
Update a destination bundle from a source bundle.
Upload an bundle to archive.
UploadSpecificKeys uploads some specified keys (files) within a bundle's consumable store.
No description provided by the author
No description provided by the author
No description provided by the author
WithDoneChan sets a signaling channel controlled by the caller to interrupt ongoing goroutines.
No description provided by the author
WithKVType selects the KV implementation (badgerDB or pebble).
WithLabelPrefix is an option for ListLabelsApply, to filter on labels with some given prefix.
WithLabelVersions makes ListLabels list all versions of labels (requires the vmedata bucket to enable versioning).
WithMemProf enables profiling and sets the memory profile directory (defaults to the current working directory).
WithMetrics toggles metrics for the core package and its dependencies (e.g.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
WithPurgeResumeIndex will resume index building by first reloading a fresh KV with all the already uploaded index files.
No description provided by the author
No description provided by the author
No description provided by the author
# Constants
DiffEntryTypeAdd indicates the bundle exhibits an extra entry.
DiffEntryTypeDel indicates the bundle exhibits a missing entry.
DiffEntryTypeDif indicates the bundle exhibits different entries.
No description provided by the author
No description provided by the author
No description provided by the author
# Structs
Bundle represents a bundle in its archived state.
BundleDiff describes all differences between two bundle.
BundleIOMetrics extends IOMetrics to capture bundle file count and metadata index files count.
Diamond is a generalization of Bundle to support diamond operations.
DiffEntry describes a single point of difference between two bundles.
Label describes a bundle label.
M describes metrics for the core package.
No description provided by the author
No description provided by the author
Settings defines various settings for core features.
Split is a generalization of a Bundle, to support diamond operations.
# Type aliases
ApplyBundleFunc is a function to be applied on a bundle.
ApplyDiamondFunc is a function to be applied on a diamond.
ApplyLabelFunc is a function to be applied on a label.
ApplyRepoFunc is a function to be applied on a repo.
ApplySplitFunc is a function to be applied on a split.
BundleOption is a functor to build a bundle with some options.
No description provided by the author
DiamondOption is a functor to build a diamond with some options.
DiffEntryType qualifies the type of difference between two bundles.
KeyFilter is a function to filter the files to upload.
KeyIterator retrieves all keys from a store.
No description provided by the author
LabelOption is a functor to build labels.
Option sets options for listing core objects.
No description provided by the author
SplitOption defines an option for a Split.