modulepackage
0.0.0-20210504165230-211227dad0ce
Repository: https://github.com/blugelabs/cbgt.git
Documentation: pkg.go.dev
# README
cbgt
NOTE: this is a fork of Couchbase's cbgt library prior to the BSL license change. It has most of the Couchbase specific references removed, and uses the blugelabs fork of blance. The unit tests pass, but it is not known to work beyond this.
The cbgt project provides a Go library that helps manage distributed partitions (or data shards) across an elastic cluster of servers.
Documentation
NOTE: This library initializes math's random seed (rand.Seed(time.Now().UTC().UnixNano())) for unique id generation.
# Packages
No description provided by the author
# Functions
AtomicCopyMetrics copies uint64 metrics from s to r (from source to result), and also applies an optional fn function to each metric.
This basic partition func first tries a direct lookup by partition string, else it tries the "" partition.
BlanceMap reconstructs a blance map from an existing plan.
BlancePartitionModel returns a blance library PartitionModel and model constraints based on an input index definition.
BlancePlanPIndexes invokes the blance library's generic PlanNextMap() algorithm to create a new pindex layout plan.
Functionally determine the delta of which feeds need creation and which should be shut down.
CalcMovingPartitionsCount attempts to compute the number of moving partitions during a rebalance, given few node count statistics of the cluster.
CalcNodesLayout computes information about the nodes based on the index definitions, node definitions, and the current plan.
Functionally determine the delta of which pindexes need creation and which should be shut down on our local node (mgrUUID).
Split logical indexes into PIndexes and assign PIndexes to nodes.
CasePlanFrozen returns true if the plan for the indexDef is frozen, in which case it also populates endPlanPIndexes with a clone of the indexDef's plans from begPlanPIndexes.
CfgGetClusterOptions returns the cluster level options.
Returns index definitions from a Cfg provider.
Retrieves node definitions from a Cfg provider.
Retrieves PlanPIndexes from a Cfg provider.
CfgGetVersion returns the Cfg version.
CfgNodeDefsKey returns the Cfg access key for a NodeDef kind.
CfgRemoveNodeDef removes a NodeDef with the given uuid from the Cfg.
CfgRemoveNodeDefForce removes a NodeDef with the given uuid from the Cfg ignoring the cas checks.
CfgSetClusterOptions sets the cluster level options.
Updates index definitions on a Cfg provider.
Updates node definitions on a Cfg provider.
Updates PlanPIndexes on a Cfg provider.
No description provided by the author
ConsistencyWaitDone() waits for either the cancelCh or doneCh to finish, and provides the partition's seq if it was the cancelCh.
ConsistencyWaitGroup waits for all the partitions from a group of pindexes to reach a required consistency level.
ConsistencyWaitPartitions waits for the given partitions to reach the required consistency level.
ConsistencyWaitPIndex waits for all the partitions in a pindex to reach the required consistency level.
CopyPlanPIndexes returns a copy of the given planPIndexes, albeit with a new UUID and given version.
DataSourceUUID is a helper function that fetches the sourceUUID for the sourceName.
DefaultMaxPartitionsPerPIndex retrieves "defaultMaxPartitionsPerPIndex" from the manager options, if available.
ErrorToString is a helper func that returns e.Error(), but also returns "" for nil error.
Calls f() in a loop, sleeping in an exponential backoff if needed.
FeedNameForPIndex functionally computes the name of a feed given a pindex.
FilesFeedPartitions returns the partitions, controlled by FilesFeedParams.NumPartitions, for a FilesFeed instance.
FilesFindMatches finds all leaf file paths in a subdirectory tree that match any in an optional array of regExps (regular expression strings).
FilesPathToPartition hashes a file path to a partition.
GetIndexDef retrieves the IndexDef and PIndexImplType for an index.
GetNodePlanParam returns a relevant NodePlanParam for a given node from a nodePlanParams, defaulting to a less-specific NodePlanParam if needed.
IndentJSON is a helper func that returns indented JSON for its interface{} x parameter.
IndexPartitionSettings returns the settings to be used for MaxPartitionsPerPIndex and IndexPartitions.
IsFeatureSupportedByCluster checks whether the given feature is supported across the cluster/given NodeDefs.
IsStablePlan checks whether the given plan is a stable or evolving plan by checking the partition-node assignments of partitions belonging to each of the indexes.
No description provided by the author
NewCfgMem returns an empty CfgMem instance.
NewCfgSimple returns a CfgSimple that reads and stores its single configuration file in the provided file path.
NewDestStats creates a new, ready-to-use DestStats.
NewFilesFeed creates a ready-to-be-started FilesFeed.
Returns an intiialized IndexDefs.
NewManager returns a new, ready-to-be-started Manager instance, with additional options.
NewMsgRing returns a MsgRing of a given ringSize.
NewNILFeed creates a ready-to-be-started NILFeed instance.
Returns an initialized NodeDefs.
Creates a pindex, including its backend implementation structures, and its files.
NewPIndexImpl creates an index partition of the given, registered index type.
No description provided by the author
Returns an initialized PlanPIndexes.
No description provided by the author
No description provided by the author
No description provided by the author
A NoopPlannerHook is a no-op planner hook that just returns its input.
No description provided by the author
No description provided by the author
OpenPIndexImpl loads an index partition of the given, registered index type from a given path.
OpenPIndexImplUsing loads an index partition of the given, registered index type from a given path with the given indexParams.
No description provided by the author
PIndexImplTypeForIndex retrieves from the Cfg provider the index type for a given index.
Returns true if both the PIndex meets the PlanPIndex, ignoring UUID.
Plan runs the planner once.
PlannerCheckVersion errors if a version string is too low.
PlannerGetIndexDefs retrives index definitions from a Cfg.
PlannerGetNodeDefs retrieves node definitions from a Cfg.
PlannerGetPlan retrieves plan related info from the Cfg.
PlannerGetPlanPIndexes retrieves the planned pindexes from a Cfg.
NOTE: PlanPIndex.Name must be unique across the cluster and ideally functionally based off of the indexDef so that the SamePlanPIndex() comparison works even if concurrent planners are racing to calculate plans.
PlanPIndexNodeCanRead returns true if PlanPIndexNode.CanRead is true; it's useful as a filter arg for Manager.CoveringPIndexes().
PlanPIndexNodeCanWrite returns true if PlanPIndexNode.CanWrite is true; it's useful as a filter arg for Manager.CoveringPIndexes().
PlanPIndexNodeOk always returns true; it's useful as a filter arg for Manager.CoveringPIndexes().
PrimaryFeedPartitions generates partition strings based on a PrimarySourceParams.NumPartitions parameter.
ReadableStackTrace tries to capture the caller stack frame for the calling function in a panic scenario.
RegisterFeedType is invoked at init/startup time to register a FeedType.
RegisterPIndexImplType registers a index type into the system.
Returns true if both PlanPIndex are the same, ignoring PlanPIndex.UUID.
Returns true if both PlanPIndexes are the same, where we ignore any differences in UUID or ImplVersion.
No description provided by the author
Split an IndexDef into 1 or more PlanPIndex'es, assigning data source partitions from the IndexDef to a PlanPIndex based on modulus of MaxPartitionsPerPIndex.
StartFilesFeed starts a FilesFeed and is the the callback function registered at init/startup time.
StringsIntersectStrings returns a brand new array that has the intersection of a and b.
StringsRemoveDuplicates removes any duplicate strings from the give slice.
StringsRemoveStrings returns a copy of stringArr, but with some strings removed, keeping the same order as stringArr.
StringsToMap connverts an array of (perhaps duplicated) strings into a map with key of those strings and values of true, and is useful for simple set-like operations.
StructChanges uses reflection to compare the fields of two structs, which must the same type, and returns info on the changes of field values.
Returns true if PlanPIndex children in a are a subset of those in b, using SamePlanPIndex() for sameness comparion.
Time invokes a func f and updates the totalDuration, totalCount and maxDuration metrics.
TimeoutCancelChan creates a channel that closes after a given timeout in milliseconds.
Timer updates a metrics.Timer.
UnregisterNodes removes the given nodes (by their UUID) from the nodes wanted & known cfg entries.
UnregisterNodesWithRetries removes the given nodes (by their UUID) from the nodes wanted & known cfg entries, and performs retries a max number of times if there were CAS conflict errors.
VerifyEffectiveClusterVersion checks the cluster version values, and if the cluster contains any node which is lower than the given myVersion, then return false.
Compares two dotted versioning strings, like "1.0.1" and "1.2.3".
WriteTimerJSON writes a metrics.Timer instance as JSON to a io.Writer.
# Constants
No description provided by the author
DEST_EXTRAS_TYPE_NIL means there are no extras as part of a Dest.DataUpdate/DataDelete invocation.
No description provided by the author
No description provided by the author
Default values for feed parameters.
FeedAllotmentOnePerPIndex specifies that there should be only a single feed per pindex.
FeedAllotmentOption is the manager option key used the specify how feeds should be alloted or assigned.
INDEX_DEFS_KEY is the key used for Cfg access.
INDEX_NAME_REGEXP is used to validate index definition names.
No description provided by the author
No description provided by the author
No description provided by the author
MANAGER_MAX_EVENTS limits the number of events tracked by a Manager for diagnosis/debugging.
NODE_DEFS_KEY is used for Cfg access.
NODE_DEFS_KNOWN is used for Cfg access.
NODE_DEFS_WANTED is used for Cfg access.
No description provided by the author
PINDEXES_RESTART suggests a reboot of the pindexes.
No description provided by the author
PLAN_PINDEXES_KEY is used for Cfg access.
QUERY_CTL_DEFAULT_TIMEOUT_MS is the default query timeout.
The cbgt.Version tracks persistence versioning (schema/format of persisted data and configuration).
No description provided by the author
No description provided by the author
# Variables
No description provided by the author
No description provided by the author
No description provided by the author
Key is sourceType.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
MsgRingMaxSmallBufSize is the max pool size for reused buf's.
MsgRingMaxSmallBufSize is the cutoff point, in bytes, in which a msg ring categorizes a buf as small versus large for reuse.
PINDEX_STORE_MAX_ERRORS is the max number of errors that a PIndexStoreStats will track.
PIndexImplTypes is a global registry of pindex type backends or implementations.
PlannerHooks allows advanced applications to register callbacks into the planning computation, in order to adjust the planning outcome.
PlanPIndexFilters represent registered PlanPIndexFilter func's, and should only be modified during process init()'ialization.
No description provided by the author
# Structs
Implements both Dest and PIndexImpl interfaces.
The error used on mismatches of CAS (compare and set/swap) values.
See the Cfg.Subscribe() method.
CfgMem is a local-only, memory-only implementation of Cfg interface that's useful for development and testing.
CfgMemEntry is a CAS-Val pairing tracked by CfgMem.
CfgSimple is a local-only, persisted (in a single file) implementation of the Cfg interface that's useful for non-clustered, single-node instances for developers.
ClusterOptions stores the configurable cluster-level manager options.
ConfigAnalyzeRequest wraps up the various configuration parameters that the PIndexImplType implementations deals with.
ConsistencyParams represent the consistency requirements of a client's request.
A ConsistencyWaitReq represents a runtime consistency wait request for a partition.
CoveringPIndexes represents a non-overlapping, disjoint set of PIndexes that cover all the partitions of an index.
CoveringPIndexesSpec represent the arguments for computing the covering pindexes for an index.
A DestForwarder implements the Dest interface by forwarding method calls to the Dest returned by a DestProvider.
DestStats holds the common stats or metrics for a Dest.
DiagHandler allows modules to provide their own additions in response to "diag" or diagnostic information requests.
Documentation is used for auto-generated documentation.
An ErrorConsistencyWait represents an error or timeout while waiting for a partition to reach some consistency requirements.
ErrorLocalPIndexHealth represents the unavailable pindexes and the corresponding error details which is discovered during the consistency checks.
A FeedType represents an immutable registration of a single feed type or data source type.
FilesFeed is a Feed interface implementation that that emits file contents from a local subdirectory tree.
FilesFeedParams represents the JSON expected as the sourceParams for a FilesFeed.
An IndexDef is a logical index definition.
An IndexDefEnveloped overrides IndexDef with Params and SourceParams fields that are enveloped JSON (JSON encoded as strings), for backwards compatibility.
An IndexDefNested overrides IndexDef with Params and SourceParams fields that are JSON nested objects instead of strings, for easier-to-use API.
An IndexDefs is zero or more index definitions.
IndexPrepParams can be used to override any of the unset index parameters.
A Manager represents a runtime node in a cluster.
ManagerStats represents the stats/metrics tracked by a Manager instance.
A MsgRing wraps an io.Writer, and remembers a ring of previous writes to the io.Writer.
A NILFeed implements the Feed interface and never feeds any data to its Dest instances.
A NodeDef is a node definition.
A NodeDefs is comprised of zero or more node definitions.
A NodePlanParam defines whether a particular node can service a particular index definition.
No description provided by the author
A PIndex represents a partition of an index, or an "index partition".
Note that these callbacks are invoked within the manager's sync mutex context, it is the responsibility of the user to ensure that they do NOT reacquire the manager mutex or any api that does within the callbacks.
PIndexImplType defines the functions that every pindex implementation type must register on startup.
PIndexStoreStats provides some common stats/metrics and error tracking that some pindex type backends can reuse.
A PlannerHookInfo is the in/out information provided to PlannerHook callbacks.
A PlanParams holds input parameters to the planner, that control how the planner should split an index definition into one or more index partitions, and how the planner should assign those index partitions to nodes.
A PlanPIndex represents the plan for a particular index partition, including on what nodes that the index partition is assigned to.
A PlanPIndexEnveloped overrides PlanPIndex with IndexParams and SourceParams fields that are enveloped JSON (JSON encoded as strings), for backwards compatibility.
A PlanPIndexes is comprised of zero or more planPIndexes.
A PlanPIndexNested overrides PlanPIndex with IndexParams and SourceParams fields that are JSON nested objects instead of strings, for easier-to-use API.
A PlanPIndexNode represents the kind of service a node has been assigned to provide for an index partition.
PlanPIndexNodeRef represents an assignment of a pindex to a node.
A PrimaryFeed implements both the Feed and Dest interfaces, for chainability; and is also useful for testing.
PrimarySourceParams represents the JSON for the sourceParams for a primary feed.
QueryCtl defines the JSON parameters that control query execution and which are independent of any specific pindex type.
QueryCtlParams defines the JSON that includes the "ctl" part of a query request.
RemotePlanPIndex associations are returned by CoveringPIndexes().
StopAfterSourceParams defines optional fields for the sourceParams that can stop the data source feed (i.e., index ingest) if the seqs per partition have been reached.
TaskRequest represent a generic task request like "compact" or "encrypt" for partitions.
No description provided by the author
A UUIDSeq associates a UUID (such as from a partition's UUID) with a seq number.
# Interfaces
Cfg is the interface that configuration providers must implement.
ConsistencyWaiter interface represents a service that can wait for consistency.
Dest interface defines the data sink or destination for data that cames from a data-source.
DestEx interface defines the data sink or destination for data that comes from a data-source for any generic implementations.
A DestProvider returns the Dest to use for different kinds of operations and is used in conjunction with a DestForwarder.
A Feed interface represents an abstract data source.
No description provided by the author
ManagerEventHandlers represents the callback interface where an application can receive important event callbacks from a Manager.
PIndexImpl represents a runtime pindex implementation instance, whose runtime type depends on the pindex's type.
TaskRequestHandler represents the interface that need to implemented by the partitions for using the task scatter gatherer.
VersionReader is an interface to be implemented by the configuration providers who supports the verification of homogeneousness of the cluster before performing certain Key/Values updates related to the cluster status.
# Type aliases
Key is partition or partition/partitionUUID.
A CwrQueue is a consistency wait request queue, implementing the heap.Interface for ConsistencyWaitReq's, and is heap ordered by sequence number.
DestExtrasType represents the encoding for the Dest.DataUpdate/DataDelete() extras parameter.
A DestPartitionFunc allows a level of indirection/abstraction for the Feed-to-Dest relationship.
Performs a lookup of a source partition given a document id.
Returns the current partitions and their seq's.
Each Feed or data-source type knows of the data partitions for a data source.
Returns the sourceUUID for a data source.
A FeedStartFunc is part of a FeedType registration as is invoked by a Manager when a new feed instance needs to be started.
Returns the current stats from a data source, if available, where the result is dependent on the data source / feed type.
PartitionErrMap tracks errors with the name of the partition where it occurred.
A PlannerFilter callback func should return true if the plans for an indexDef should be updated during CalcPlan(), and should return false if the plans for the indexDef should be remain untouched.
A PlannerHook is an optional callback func supplied by the application via PlannerHooks and is invoked during planning.
PlanPIndexFilter is used to filter out nodes being considered by CoveringPIndexes().
PlanPIndexNodeRefs represents assignments of pindexes to nodes.
ResultCode represents the return code indicative of the various operations recommended by the pindex implementations upon detecting a config change.
No description provided by the author
TaskPartitionStatusMap tracks the current state of a task across the partitions.