Package json implements encoding and decoding of JSON as defined in RFC 4627.

BucketMinSeqnos return list of {{vbno,seqno}..} for all vbuckets.

BucketSeqnos return list of {{vbno,seqno}..} for all vbuckets.

BucketTs return bucket timestamp for all vbucket.

CanMaintainShardAffinity - if shard affinity is enabled and storage mode is Plasma Should only be called from *indexer* and not from client/metadata_provider.

ChecksumAndCompress checksums and optionally compresses a byte slice, prepending an 8-byte header comprised of: header[0] - flags: bit 0 - compressed? bits 1-7 - unused bits header[1-4] - crc32 checksum written in big-endian order header[5-7] - unused bytes.

Sequence numbers for a specific collections.

Sequence numbers for a specific collection.

CommonStrings returns intersection of two set of strings.

Convert ConfigValue to map[uint64]uint64 map Pass the ConfigValue of indexer.planner.internal.min_parititions_per_shard.

Use fixed length byte array so that indexer can compute the position of the SHA value in the stored entry without any special encoding.

Use fixed length byte array so that indexer can compute the position of the SHA value in the stored entry without any special encoding.

ConnectBucket will instantiate a couchbase-bucket instance with cluster.

ConnectBucket2 will instantiate a couchbase-bucket instance with cluster.

Console writes a message to the admin console (Logs page of UI).

CopyDir compose destination path based on source and, - if dest is file, and path is reachable, it is a no-op.

Create a copy of IndexInstMap by fixing Immutable flag in each instance's IndexDefn.

Crc64Checksum computes the CRC64 checksum of a byte slice using the highly recommended ECMA polynomial.

Crc64Update incrementally updates an existing CRC64 checksum with more bytes.

For standalone tests where MetaKV is not present, call this method to attempt MetaKV retry only once.

ExcludeStrings will exclude strings in `excludes` from `strs`.

ExcludeUint32 remove items from list.

ExcludeUint64 remove items from list.

ExitOnStdinClose is exit handler to be used with ns-server.

FailsafeOp can be used by gen-server implementors to avoid infinitely blocked API calls.

FailsafeOpAsync is same as FailsafeOp that can be used for asynchronous operation, that is, caller does not wait for response.

FailsafeOpAsync2 is same as FailsafeOpAsync that can be used for asynchronous operation, that is, caller does not wait for response.

FailsafeOpNoblock is same as FailsafeOpAsync that can be used for non-blocking operation, that is, if `reqch` is full caller does not block.

FetchNewClusterInfoCache returns a pointer to a new instance, so the anonymous mutex in the returned object only needs to be locked if the caller shares it among goroutines.

FetchNewClusterInfoCache2 does not do an Implicit Fetch() like FetchNewClusterInfoCache.

-------------------------- Utility functions --------------------------.

This method fetch the bucket UUID.

This method will fetch the collectionID.

GetColocatedHost find the server addr for localhost and return the same.

Note that the input prefix passed to this function should be exactly the same as return value of GetStatsPrefix().

GetIndexStats gets the index and/or indexer stats selected by the given filter from all indexer nodes in parallel (generalized from planner/proxy.go getIndexStats, which is integrated with Planner data structures) and returns them in a map from nodeUUID to stats.

GetInternalClusterVersion gets the internal version for the entire GSI cluster.

GetInternalIndexerVersion gets the internal version for all the indexer nodes.

GetKVAddrs gather the list of kvnode-address based on the latest vbmap.

------------------------------------------------------------ APIs and utility functions ------------------------------------------------------------.

GetLocalIP returns the first external IP configured for the first interface connected to this node.

IPv6 Support.

This method will fetch the scope and collection ID.

This method will fetch the scopeID.

For flattened array, returns the list of secExprs and the desc array before explosion.

Mapping of major and minor versions to indexer's internal representation of server versions.

HashVbuuid return crc64 value of list of 64-bit vbuuids.

HasString does membership check for a string.

HasUint32 does membership check for a uint32 integer.

HasUint64 does membership check for a uint32 integer.

Intersection operation on two sets of vbuckets, return a sorted list of vbuckets present in both set.

IsAllAllowed checks if the user creds have ALL of the specified permissions.

IsAllowed checks if the user creds have ANY of the specified permissions.

IsIPLocal return whether `ip` address is loopback address or compares equal with local-IP-address.

MaxVbuckets return the number of vbuckets in bucket.

MetakvBigValueGet takes the virtual path of a big value token and retrieves and reassembles all its metakv pieces (stored as values of children /0, /1, /2, ..

MetakvBigValueList lists the virtual paths of each big value token stored under dirpath.

MetakvBigValueSet stores large tokens by breaking them up into pieces of max size METAKV_SIZE_LIMIT and storing the pieces under keys path/0, path/1, ....

MetakvList returns a slice of all entries (tokens) stored in dirpath in metakv.

MonitorServiceForPortChanges monitors the service change endpoint for port changes.

NewBucketInfoProvider is factory to either get BucketInfo or ClusterInfoCache.

NewClusterInfoCache constructs a new, non-shared ClusterInfoCache object, so consumer does not need to lock it unless it is multi-threaded.

NewClusterInfoProvider is factory to get ClusterInfoClient or ClusterInfoCacheLiteClient.

NewCollectionInfoProvider is factory to either get CollectionInfo or ClusterInfoCache.

NewConfig from another Config object or from map[string]interface{} object or from []byte slice, a byte-slice of JSON string.

NewKeyPartitionContainer initializes a new KeyPartitionContainer and returns.

NewKeyVersions return a reference KeyVersions for a single mutation.

NewNodesInfoProvider is factory to either get NodesInfo or ClusterInfoCache.

NewOperationsMonitor Input: id A unique identifier, to be used in all the log messages belonging to the monitor maxOps Maximum number of operations that can be monitored at a given time interval Number of milliseconds in the monitor's interval batchSize Max number of operations to be processed in one iteration of the monitor.

Constructor .

Initialize change notifier object for a clusterUrl.

NewStatistics return a new instance of stat structure initialized with data.

NewStreamPayload returns a reference to payload, `nVb` provides the maximum number of vbuckets that can be carried by a payload.

NewTsVbuuid returns reference to new instance of TsVbuuid.

NewTsVbuuid returns reference to new instance of TsVbuuid.

NewVbKeyVersions return a reference to a single vbucket payload.

OpError suppliments FailsafeOp used by gen-servers.

When parsing an input string, indexer expects the following semantics: a.

RefreshSecurityContextOnTopology refreshes the security context on the topology updates.

RemoveString delete `item` from list `xs`.

RemoveUint16 delete `item` from list `xs`.

RemoveUint32 delete `item` from list `xs`.

ShouldMaintainShardAfffinty - if shard affinity is enabled, cluster is on or ahead of 7.6 and storage mode is Plasma.

StreamID is unique id for a vbucket across buckets.

TraceMutexLOCK assists logging how long it took to lock a Mutex.

TraceMutexUNLOCK assists logging how long a Mutex was held.

TraceRWMutexLOCK assists logging how long it took to lock an RWMutex.

TraceRWMutexUNLOCK assists logging how long an RWMutex was held.

Union set operation on two sets of vbuckets, return a sorted list of vbuckets present in atleast one set.

UpdateOrRefreshSecurityContextOnTopologyChange updates the port mappings if endpointUpdate is not nil otherwise it refreshes the security context by doing a cluster info fetch.

Vbno16to32 converts vbucket type from uint16 to uint32.

Vbno32to16 converts vbucket type from uint32 to uint16.

This method is a light weight version of serviceChangeNotifier only used to retrieve and update cluster version.

WriteFileWithSync simulates iowrap.Ioutil_WriteFile but also syncs (forces bytes to disk) before closing.

a dummy; must be last (for loops).

AnyConsistency indexer would return the most current data available at the moment.

HTTP_STATUS_FORBIDDEN.

HTTP_STATUS_UNAUTHORIZED.

Used when the Shard being transferred is of a BHIVE vector index.

Logical stream types (StreamId values).

Exponential back off.

Seconds.

control command.

control command.

control command.

control command.

2.5G.

data command.

control command.

Treated same as EUCLIDEAN_SQUARED internally.

filler command for flusher(only used internally by indexer).

compressed flag.

usually application/json.

http.Request checksum field.

http.Response checksum field.

for HTTP_KEY_CONTENT_TYPE.

base (hex) of string-form ETag values in HTTP headers.

ETag value of 0 is treated as invalid or missing.

Maitenance Stream.

Catchup In Progress.

Create Index Processed.

Drop Index Processed.

Error State: not a persistent state -- but used in function return value.

Initial Build In Progress.

Nil State (used for no-op / invalid) -- not a persistent state.

Index is stream is ready.

Recovered state: used for shard rebalance.

Scheduled state: used for the indexes scheduled for creation.

Since there is a possibility of 7.7 version at the time of this patch, linear numbering scheme will not work for 8.0 version i.e.

Active(processing mutation and scan).

Initial Bootstrap.

Paused(not processing mutation/scan).

Note: Do not use this path for any setting other than writing the JSON {"indexer.default.enable_shard_affinity":<true|false>}' Otherwise, ns_server compatability would break.

Logical stream types (StreamId values).

Seconds.

Treated same as L2_SQUARED internally.

held mutex.

minimum lock wait/hold duration before logging it.

acquire lock for read.

waited for mutex.

acquire lock for write.

Logical stream types (StreamId values).

Used when moving multiple index instanes per transfer token Data of all the index instances on disk (a.k.a shard) is transferred to the destination node and index is rebuilt from the data on disk.

must be first (for loops).

control command.

control command.

Error indicates that for a follower, an error was encountered during upload.

InProgess indicates that followers are actually performing the upload work.

Posted is the initial state as generated by master.

Processed indicates that for a follower, all the upload work is completed.

types of payload.

types of payload.

Used when the Shard being transferred is of a standard index(primary or secondary) or Shard of Composite Vector index.

Projector Version.

Projector Version.

Projector Version.

Projector Version.

Projector Version.

Projector Version.

Projector Version.

QueryConsistency indexer would accept a timestamp vector, and make sure to return a stable data-set that is atleast as recent as the timestamp-vector.

0 = instance is active (normal state); rebalance is NOT manipulating this inst.

instance is a proxy that has already been merged to the real instance.

rebalance action canceled but instance not returned to active state.

rebalance instance move not yet complete.

tombstone for deleted partition.

Per bucket phase - Not used currently.

Per bucket phase - Not used currently.

Per bucket phase - Not used currently.

Per bucket phase - Not used currently.

Per bucket phase - Not used currently.

Error indicates that for a follower, an error was encountered during download.

InProgess indicates that followers are actually performing the download work.

Posted is the initial state as generated by master.

Processed indicates that for a follower, all the download work is completed.

control command.

control command.

control command.

SessionConsistency indexer would query the latest timestamp from each KV node.

SessionConsistencyStrict indexer would query the latest timestamp with vbseqbos and vbuuids from each KV node.

This token state means that index instances on source node have been successfully deleted.

Initial state of the tokens: ShardTokenCreated After planner run, rebalance master posts all transfer tokens to metaKV with "ShardTokenCreated" state.

This states indicates that the token processing is done and the token can be deleted from metaKV.

After transfer token and its sibling move to ShardTokenReady state, rebalance master will generate a new token with this state and post it to metaKV.

If any error is encountered during rebalance, the state of the token can be updated to ShardTokenError.

When empty node batching is enabled, shard rebalancer will move the transfer token state to "ShardTokenMerge".

After all the indexes in a shard are recovered successfully, then the state of the transfer token is moved to "ShardTokenReady".

On a successful download, rebalance destination node will change the transfer token state to "ShardTokenRecoverShard" which indicates that the download is successful and indexer should now start to establih streams with KV nodes for index build catchup A metaKV token in this state means that the destination node has succecssfully downloaded shard data and is yet to start establishing KV streams.

Rebalance source node, on a successful transfer will update the state of transfer token to "ShardTokenRestoreShard", along with the path from which destination node can download the shard data to local file system.

Rebalance destination node will read the transfer token in state "ShardTokenScheduledOnSource", updates its in-memory book keeping with all the potential movements during index rebalance and changes the transfer token state to "ShardTokenScheduleAck" for further processing.

Rebalance Source node will read tokens (that are in "ShardTokenCreated" state) updates in-memory book-keeping with all the potential movements during rebalance and changes the transfer token state to "ShardTokenScheduledOnSource".

Rebalance master would wait for all the shard tokens to reach the "ShardTokenScheduleAck" state.

SigarControlGroupInfo.Supported value if cgroups are supported.

Used for moving single instance per transfer token when the index instance is built using DCP on destination node Default mode pre 7.2.0 versions.

control command.

minimum duration of a stats action before logging it.

0.

control command.

control command.

control command.

no source node; idx created on dest during rebalance (replica repair).

moving idx from source to dest.

2.

6.

1.

7.

x.

3.

4.

4.5 Dest (partitioned indexes only): No-op in processTokenAsDest;.

5.

x.

Default shard type value.

control command.

data command.

data command.

List of errors leading to failure of index creation.

ErrClientCancel when query client cancels an ongoing scan request.

Index not found.

magic string;.

Index not ready.

Ingress lockdown error.

ErrorAborted.

ErrorChannelFull.

ErrorClosed.

ErrorEmptyN1QLExpression.

ErrorInvalidRequest.

ErrorNotFound.

ErrorNotMyVbucket.

ErrorUnexpectedPayload.

ErrScanTimedOut from indexer.

Byte slices for HTTP error response bodies.

0-2ms, 2-10ms, 10-100ms, 100-500ms, 500ms-1s, 1-10s, 10-30s, 30s-Inf.

= 0.

ProtobufDataPathMinorNum minor version number for mutation data path.

SystemConfig is default configuration for system and components.

Thread safe byte buffer pool A buffer pointer received by Get() method should be put back using Put() method.

cbauth admin authentication helper Uses default cbauth env variables internally to provide auth creds.

Helper object for fetching cluster information Can be used by services running on a cluster node to connect with local management service for obtaining cluster information.

Helper object that keeps an instance of ClusterInfoCache cached and updated periodically or when things change in the cluster Readers/Consumers must lock cinfo before using it.

Threadsafe config holder object.

ConfigValue for each parameter.

convergent counter After the counter is updated in one node, any client can read the latest counter value by polling the counter from every node.

DataportKeyVersions accepted by this endpoint.

IndexDefn represents the index definition as specified during CREATE INDEX.

IndexInst is an instance of an Index(aka replica).

KeyPartitionContainer implements PartitionContainer interfacefor key based partitioning.

KeyPartitionDefn defines a key based partition in terms of topologyie its Id and Indexer Endpoints hosting the partition.

KeyVersions for a single mutation from KV for a subset of index.

Payload either carries `vbmap` or `vbs`.

Helper object to execute a function with retries and exponential backoff.

-------------------------- ScanResultEntries --------------------------.

-------------------------- ScanResultKey --------------------------.

Stubs to make ClusterInfoCache replaceable with NodesInfo and CollectionInfo.

TransferToken represents a sindgle index partition movement for rebalance or move index.

TsVb is logical clock for a subset of vbuckets.

TsVbFull is logical clock for full set of vbuckets.

TsVbuuid is logical clock for full set of vbuckets along with branch value and last seen snapshot.

VbConnectionMap specifies list of vbuckets and current vbuuids for each vbucket.

VbKeyVersions carries per vbucket key-versions for one or more mutations.

Holder for VbSeqnosReaderHolder.

CollectionInfoProvider is an interface to query Collection Manifest.

Evaluator interface for projector, to be implemented by secondary-index or other entities.

IndexStatistics captures statistics for a range or a single key.

NodesInfoProvider is interface to query Nodes, NodesExt and ServerGroups information.

PartitionContainer contains all the partitions for an index instanceand provides methods to lookup topology information(i.e.

PartitionDefn is a generic interface which definesa partition.

Router definition for each instance (aka engine), where an instance might refer to an index or similar entities.

RouterEndpoint abstracts downstream for feed.

ByteSlices to implement Sort interface.

Config is a key, value map with key always being a string represents a config-parameter.

Consistency definition for index-scan queries.

Format of the data encoding, when it is being transferred over the wire from indexer to GsiClient during scan.

Endpoint provides an Indexer address(host:port)which is hosting a partition.

IndexerError - Runtime Error between indexer and other modules.

IndexInstList is a list of IndexInstances.

IndexInstMap is a map from IndexInstanceId to IndexInstance.

IndexSnapType represents the snapshot type created in indexer storage.

NOTE: This type needs to be in sync with smStrMap.

OperationTimeoutCallback takes as input: the total time elapsed since the operation was created using NewOperation().

RebalanceState (or "RState") gives the state of an *index instance*, NOT the state of rebalance, which may not even be going on.

RouterEndpointFactory will create a new endpoint instance for {topic, remote-address}.

SecondaryKey is secondary-key in the shape of - [ val1, val2, ..., valN ] where value can be any golang data-type that can be serialized into JSON.

ServerPriority - priority of the server to be elected as master when rebalance runs string should be in the format: MAJOR_VERSION.MINOR_VERSION.PATCH_VERSION-MPXX or MAJOR_VERSION.MINOR_VERSION.PATCH_VERSION-XXXX version can only range from [0,100).

Used to decide what kind of shard is being moved by the transfer tokens.

Statistics provide a type and method receivers for marshalling and un-marshalling statistics, as JSON, for components across the network.

StreamId represents the possible logical mutation streams (IDs defined below).

Stream Response Status Code.

TokenBuildSource is the type of the TransferToken.BuildSource field, which is currently unused but will be used in future when rebalance is done using file copy rather than DCP.

Map of sync.Pool per numVBuckets.

Vbuckets is temporary data type that can be used to sort list of uint16.