# README
This is the directory which holds all the common stuff required by all the different services.
The structure within this is folder is something like this:
config.go - holds all the config items util.go - holds the utlity stuff like creating tchannel, boot strapping ringpop, etc. *types.go - holds the types for the specific interface (eg: servicetypes.go - holds the object and interface definition for all common stuff used by our services) *.go - business logic implementing the interface (eg: service.go - implements the methods specific to servicetypes.go)
# Functions
AckIDFromString deserializes a string into the object.
AdminNotificationTypePtr makes a copy and returns the pointer to a MetadataNotificationType.
AreKafkaPhantomStores determines whether the given list of storehost-uuids for an extent indicates that this is a Kafka 'phantom' extent.
AwaitWaitGroup calls Wait on the given wait Returns true if the Wait() call succeeded before the timeout Returns false if the Wait() did not return before the timeout.
BoolPtr makes a copy and returns the pointer to a bool.
BootstrapRingpop tries to bootstrap the given ringpop instance using the hosts list.
CalculateRate does a simple rate calculation.
CheramiChecksumOptionPtr makes a copy and return the pointer too a CheramiChecksumOption.
CheramiConsumerGroupExtentStatusPtr makes a copy and returns the pointer to a CheramiConsumerGroupExtentStatus.
CheramiConsumerGroupStatusPtr makes a copy and returns the pointer to a CheramiConsumerGroupStatus.
CheramiDestinationStatusPtr makes a copy and returns the pointer to a CheramiDestinationStatus.
CheramiDestinationType converts from shared.DestinationType to cherami.DestinationType.
CheramiDestinationTypePtr makes a copy and returns the pointer to a CheramiDestinationType.
CheramiInputHostCommandTypePtr makes a copy and returns the pointer to a CheramiInputHostCommandType.
CheramiOutputHostCommandTypePtr makes a copy and returns the pointer to a CheramiOutputHostCommandType.
CheramiProtocolPtr makes a copy and returns the pointer to a CheramiProtocol.
CheramiStatusPtr makes a copy and returns the pointer to a CheramiStatus.
ClassifyErrorByType gives the metrics error class for any cherami or common error.
ConstructAckID is a helper routine to construct the ackID from the given args.
ConsumerGroupExtentMetricsPtr makes a copy and returns the pointer to a ConsumerGroupExtentMetrics.
ConsumerGroupMetricsPtr makes a copy and returns the pointer to a ConsumerGroupMetrics.
ContainsEmpty scans a string slice for an empty string, returning true if one is found.
ContainsString scans a string slice for a matching string, returning true if one is found.
ConvertDownstreamErrors is a helper function to convert a error from metadata client or controller client to client-cherami.thrift error that can be returned to caller.
CQLTimestampToUnixNano converts CQL timestamp to UnixNano.
CreateCassandraKeyspace creates the keyspace using this session for given replica count.
CreateHyperbahnClient returns a hyperbahn client.
CreateInputHostAdminClient creates and returns tchannel client for the input host admin API.
CreateOutputHostAdminClient creates and returns tchannel client for the output host admin API.
CreateRingpop instantiates the ringpop for the provided channel and host,.
DestinationExtentMetricsPtr makes a copy and returns the pointer to a DestinationExtentMetrics.
DestinationMetricsPtr makes a copy and returns the pointer to a DestinationMetrics.
DropCassandraKeyspace drops the given keyspace, if it exists.
DurationToSeconds converts a time.Duration to Seconds.
ExtrapolateDifference calculates the extrapolated difference in two observed value with rates, at some arbitrary time.
ExtrapolateValue extrapolates a value based on an observed value and rate at a given time.
FindNearestInt finds the integer that is closest to the given 'target'.
Float64Ptr makes a copy and returns the pointer to an int64.
FmtAckID formats a string to be used with TagAckID.
FmtAddr formats an int64 to be used with TagAddr.
FmtCGExtentStatus formats ConsumerGroupExtent to be used with TagCGExtentStatus.
FmtCnsm formats a string to be used with TagCnsm.
FmtCnsmID formats an int to be used with TagCnsmID.
FmtCnsPth formats a string to be used with TagCnsPth.
FmtCtrl formats a string to be used with TagCtrl.
FmtDLQID formats a string to be used with TagDLQID.
FmtDplName formats a string to be used with TagDplName.
FmtDst formats a string to be used with TagDst.
FmtDstPth formats a string to be used with TagDstPth.
FmtExt formats a string to be used with TagExt.
FmtExtentStatus formats ExtentStatus to be used with TagExtentStatus.
FmtFrnt formats a string to be used with TagFrnt.
FmtHostConnLimit formats an int to be used with TagHostConnLimit.
FmtHostIP formats a string to be used with TagHostIP.
FmtHostName formats a string to be used with TagHostName.
FmtHostPort formats a string to be used with TagHostPort.
FmtIn formats a string to be used with TagIn.
FmtInPubConnID formats an int to be used with TagInPubConnID.
FmtInPutAckID formats a string to be used with TagInPutAckID.
FmtInReplicaHost formats a string to be used with TagInReplicaHost.
FmtMsgID formats a string to be used with TagMsgID.
FmtOut formats a string to be used with TagOut.
FmtReconfigureID formats a string to be used with TagReconfigureID.
FmtReconfigureType formats admin.NotificationType to be used with TagNotificationType.
FmtService formats a string to be used with TagService.
FmtSlowDown formats an int to be used with TagSlowDown.
FmtStor formats a string to be used with TagStor.
FmtTbSleep formats a time.Duration to be used with TagTbSleep.
FmtTenancy formats a string to be used with TagTenancy.
FmtZoneName formats a string to be used with TagZoneName.
GetConnectionKey is used to create a key used by connections for looking up connections.
GetDateTag returns the current date used for tagging daily metric.
GetDefaultLogger is a utility routine to get the default logger.
GetDirectoryName function gives the directory name given a path used for destination or consumer groups.
GetDLQPathNameFromCGName function return the DLQ destination name based on the consumer group passed Usually pass the Consumer group name to get a DLQ path name DEVNOTE: DO NOT QUERY A DLQ DESTINATION BY THIS NAME.
GetEnvVariableFromHostPort gets the environment variable corresponding to this host port.
GetHTTPListenAddress is a utlility routine to give out the appropriate listen address for the http endpoint.
GetLocalClusterInfo gets the zone and tenancy from the given deployment.
GetOpenAppendStreamRequestHeaders converts an OpenAppendStreamRequest struct to headers to pass as tchannel headers to OpenAppendStream.
GetOpenAppendStreamRequestHTTP extracts OpenAppendStreamRequest from http headers.
GetOpenAppendStreamRequestStruct extracts OpenAppendStreamRequest from tchannel headers.
GetOpenReadStreamRequestHeaders converts an OpenReadStreamRequest struct to headers to pass as tchannel headers to OpenReadStream.
GetOpenReadStreamRequestHTTP extracts OpenReadStreamRequest from http headers.
GetOpenReadStreamRequestHTTPHeaders converts an OpenReadStreamRequest struct to http headers for OpenReadStream.
GetOpenReadStreamRequestStruct extracts OpenReadStreamRequest from tchannel headers.
GetOpenReplicationReadStreamRequestHTTP extracts OpenReplicationReadStreamRequest from http headers.
GetOpenReplicationReadStreamRequestHTTPHeaders converts an OpenReplicationReadStreamRequest struct to http headers for OpenReplicationReadStreamRequest.
GetOpenReplicationRemoteReadStreamRequestHTTP extracts OpenReplicationRemoteReadStreamRequest from http headers.
GetOpenReplicationRemoteReadStreamRequestHTTPHeaders converts an OpenReplicationRemoteReadStreamRequest struct to http headers for OpenReplicationRemoteReadStreamRequest.
GetRandInt64 is used to get a 64 bit random number between min and max.
GetTagsFromPath function return the tags name for path based on directory path name passed Usually pass the Consumer group name or a destination path name to get a tag name.
Int16Ptr makes a copy and returns the pointer to an int16.
Int32Ptr makes a copy and returns the pointer to an int32.
Int64Ptr makes a copy and returns the pointer to an int64.
InternalChecksumOptionPtr makes a copy and return the pointer too a internal shared ChecksumOption.
InternalConsumerGroupStatusPtr makes a copy and returns the pointer to a internal shared ConsumerGroupStatus.
InternalConsumerGroupTypePtr makes a copy and returns the pointer to a internal shared ConsumerGroupType.
InternalDestinationStatusPtr makes a copy and returns the pointer to a internal shared DestinationStatus.
InternalDestinationTypePtr makes a copy and returns the pointer to a internal shared DestinationType.
InternalExtentReplicaReplicationStatusTypePtr makes a copy and returns the pointer to a ExtentReplicaReplicationStatus.
IntPtr makes a copy and returns the pointer to an int.
IsDevelopmentEnvironment detects if we are running in a development environment.
IsDLQDestination checks whether a destination is dlq type.
IsDLQDestinationPath checks whether a destination path is dlq type.
IsKafkaConsumerGroupExtent determines if a consumer group extent is a Kafka consumption assignment.
IsKafkaPhantomInput determines whether the given inputhost-uuid for an extent indicates that this is a Kafka 'phantom' extent.
IsRemoteZoneExtent returns whether the extent is a remote zone extent.
IsRetryableTChanErr returns true if the given tchannel error is a retryable error.
IsValidServiceName returns true if the given input is a valid service name, false otherwise.
LoadCassandraSchema loads the schema from the given .cql file on this keyspace using cqlsh.
MaxInt returns the max of given two integers.
MaxInt64 returns the max of given two integers.
MetadataConsumerGroupExtentStatusPtr makes a copy and returns the pointer to a MetadataConsumerGroupExtentStatus.
MetadataExtentReplicaStatusPtr makes a copy and returns the pointer to a MetadataExtentReplicaStatus.
MetadataExtentStatusPtr makes a copy and returns the pointer to a MetadataExtentStatus.
MinInt returns the min of given two integers.
MinInt64 returns the min of given two integers.
NewCassandraCluster creates a cassandra cluster given comma separated list of clusterHosts.
NewClientFactory just instantiates a thriftClientImpl object.
NewCliHelper is used to create an uber specific CliHelper.
NewCounterBank returns a new instance of counterBank containing size number of counters.
NewDummyZoneFailoverManager creates a dummy zone failover manager.
NewHeartbeat creates a new Heartbeat object.
NewHostAckIDGenerator returns a HostAckIDGenerator object and starts from the given value.
NewHostHardwareInfoReader creates and returns an implementation of hardwareInfoReader that uses Cassandra as the backing store.
NewHostIDHeartbeater creates and returns a new instance of HostIDHeartbeater.
NewHTTPHandler returns a new instance of http handler.
NewLoadReporterDaemonFactory is used to create a factory for creating LoadReporterDaemon.
NewMetricReporterWithHostname create statsd/simple reporter based on config.
NewMockRingpopMonitor returns a new instance.
NewMockTimeSource returns a new instance of a controllable time source.
NewOpenAppendInWebsocketStream returns a new OpenAppendInWebsocketStream object.
NewOpenAppendOutWebsocketStream returns a new OpenAppendOutWebsocketStream object.
NewOpenConsumerInWebsocketStream returns a new OpenConsumerInWebsocketStream object.
NewOpenConsumerOutWebsocketStream returns a new OpenConsumerOutWebsocketStream object.
NewOpenPublisherInWebsocketStream returns a new OpenPublisherInWebsocketStream object.
NewOpenPublisherOutWebsocketStream returns a new OpenPublisherOutWebsocketStream object.
NewOpenReadInWebsocketStream returns a new OpenReadInWebsocketStream object.
NewOpenReadOutWebsocketStream returns a new OpenReadOutWebsocketStream object.
NewRealTimeSource returns a time source that servers real wall clock time using CLOCK_REALTIME.
NewRealTimeTickerFactory creates and instance of TickerSourceFactory used by service code.
NewRingpopMonitor returns a new instance of RingpopMonitor, rp: Ringpop instance of the local node services: list of services we need to track UUIDResolver: Resolver instance that can map uuids to addrs and vice-versa HostHardWareInfoReader: HwInfoReader instance that can get the hosts' hardware spec.
NewService instantiates a ServiceInstance TODO: have a better name for Service.
NewShardedConcurrentMap returns an instance of ShardedConcurrentMap
ShardedConcurrentMap is a thread safe map that maintains upto nShards number of maps internally to allow nShards writers to be acive at the same time.
NewTestMetricsReporter creates a test reporter that allows registration of handler functions.
NewTimer returns a Timer object to be used to perform timer functionality.
NewTimerFactory creates and returns a new factory for OneShotTimers.
NewTokenBucket creates and returns a new token bucket rate limiter that repelenishes the bucket every 100 milliseconds.
NewTokenBucketFactory creates an instance of factory used for creating TokenBucket instances.
NewUUIDResolver returns an instance of UUIDResolver that can be used to resovle host uuids to ip:port addresses and vice-versa.
NewWSConnector creates a WSConnector.
NodeMetricsPtr makes a copy and returns the pointer to a NodeMetrics.
NodeStatusPtr makes a copy and returns the pointer to a NodeStatus.
Now is the version to return UnixNanoTime.
OverrideValueByPrefix takes a list of override rules in the form 'prefix=val' and a given string, and determines the most specific rule that applies to the given string.
RandomBytes generates random bytes of given size.
RolePtr makes a copy and returns the pointer to a SKU.
RWLockReadAndConditionalWrite implements the RWLock Read+Read&Conditional-Write pattern.
ServiceLoop runs the http admin endpoints.
SetupServerConfig reads on-disk config (in config/).
SetupSignalHandler handles the passed in signal and calls the appropriate callback.
ShortenGUIDString takes a string with one or more GUIDs and elides them to make it more human readable.
SKUPtr makes a copy and returns the pointer to a SKU.
SpinWaitOnCondition busy waits for a given condition to be true until the timeout Returns true if the condition was satisfied, false on timeout.
SplitHostPort takes a x.x.x.x:yyyy string and split it into host and ports.
StartEKG starts a goroutine to check the heartbeats.
StoreExtentMetricsPtr makes a copy and returns the pointer to a StoreExtentMetrics.
StringPtr makes a copy and returns the pointer to a string.
StringSetEqual checks for set equality (i.e.
TSPtr makes a copy and returns the pointer to an Time.
Uint32Ptr makes a copy and returns the pointer to a uint32.
Uint64Ptr makes a copy and returns the pointer to a uint64.
UnixNanoToCQLTimestamp converts UnixNano to CQL timestamp.
UUIDHashCode is a hash function for hashing string uuid if the uuid is malformed, then the hash function always returns 0 as the hash value.
UUIDToUint16 uses the UUID and returns a uint16 out of it.
WaitTimeout waits for given func until timeout (return true if timeout).
WSStart register websocket handlers and spin up websocket server.
# Constants
CallerHostName is the name of thrift context header contains current host name.
CallerServiceName is the name of thrift context header contains current service name.
CallerUserName is the name of thrift context header contains current user name.
ControllerServiceName refers to the name of the cherami controller service.
DefaultUpgradeTimeout is the timeout to wait in the upgrade handler.
one day.
Flush stream thresholds; this is used by the "pumps" that wrap the websocket-stream and provide go-channel interface to read/write from the stream.
Flush stream thresholds; this is used by the "pumps" that wrap the websocket-stream and provide go-channel interface to read/write from the stream.
FrontendServiceName refers to the name of the cherami frontend service.
HostAddedEvent indicates that a new host joined the ring.
HostMaxConnPerDestination is the max connections per destination.
HostOverallConnLimit is the overall connection limit for this host.
HostPerConnMsgsLimitPerSecond is the per connection messages limit.
HostPerExtentMsgsLimitPerSecond is the per extent messages limit.
HostPerSecondConnLimit is the per second rate limit for this host.
HostRemovedEvent indicates that a host left the ring.
InputHostForRemoteExtent is a special (and fake) input host ID for remote extent.
InputServiceName refers to the name of the cherami in service.
KafkaPhantomExtentInputhost is placeholder/phantom inputhost uuid used for Kafka extents.
KafkaPhantomExtentStorehost is placeholder/phantom storehost uuid used for Kafka extents.
MaxDuration is maximum time duration.
MaxHostMaxConnPerDestination is the maximam for max connections per destination TODO: Need to figure out the suitable values.
MaxHostOverallConn is the maximam overall connection limit for this host TODO: Need to figure out the suitable values.
MaxHostPerConnMsgsLimitPerSecond is the maximam for per connection messages limit TODO: Need to figure out the suitable values.
MaxHostPerExtentMsgsLimitPerSecond is the maximam for per extent messages limit TODO: Need to figure out the suitable values.
MaxHostPerSecondConn is the maximam per second rate limit for this host TODO: Need to figure out the suitable values.
OutputServiceName refers to the name of the cherami out service.
ReplicatorServiceName refers to the name of the cherami replicator service.
SequenceBegin refers to the beginning of an extent.
SequenceEnd refers to the end of an extent.
StoreServiceName refers to the name of the cherami store service.
TagAckID is the logging tag for AckId.
TagAddr is the logging tag for address.
TagCGExtentStatus is for cg extent status.
TagCnsm is the logging tag for Consumer Group UUID.
TagCnsmID is the logging tag for the consumer ID.
TagCnsPth is the logging tag for Consumer group Path.
TagCtrl is the logging tag for Extent Controller UUID.
TagDbPath is the path to the db of the extent in manyrocks.
TagDeploymentName is the logging tag for deployment name.
TagDLQID is the logging tag for a Dead Letter Queue destination UUID.
TagDplName is the logging tag for deployment name.
TagDst is the tag for Destination UUID.
TagDstPth is the logging tag for Destination Path.
TagErr is the tag for error object message.
TagEvent is for "event" from Discovery and Failure Detection Daemon.
TagExt is the logging tag for Extent UUID.
TagExtentCacheSize is the logging tag for PathCache ExtentCache map size.
TagExtentStatus is for extent status.
TagFrnt is the logging tag for Frontend UUID.
TagHostConnLimit is the log tag for hostconnection limit.
TagHostIP is the logging tag for host IP.
TagHostName is the logging tag for host name.
TagHostPort is the log tag for hostport.
TagIn is the logging tag for Inputhost UUID.
TagInPubConnID is the logging tag for input pubconnection ID.
TagInPutAckID is the logging tag for PutMessageAck ID.
TagInReplicaHost is the logging tag for replica host on input.
TagModule is the logging tag used to identify the module within a service.
TagMsgID is the logging tag for MsgId.
TagOut is the logging tag for Outputhost UUID.
TagReconfigureID is the logging tag for reconfiguration identifiers.
TagReconfigureType is the logging tag for reconfiguration type.
TagReplicator is the logging tag for replicator host UUID.
TagRunnerName is the log tag for runner name, value is basic, timers, dlqTimedout, etc.
TagSeq is for sequence number.
TagService is the log tag for the service.
TagSlowDownSeconds is the logging tag for slow down time on consconnection.
TagState is for "state" in event handlers.
TagStor is the logging tag for StoreHost UUID.
TagTbSleep is the log tag for token bucket sleep duration.
TagTenancy is the logging tag for tenancy.
TagUnknowPth is the logging tag for Unknow Path.
TagUpdateUUID is the logging tag for reconfiguration update UUIDs.
TagZoneName is the logging tag for zone name.
TenancyProd is the tenancy of production Deployment name can be in the format of <tenancy>_<zone>.
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
# Variables
ConsumerGroupRegex regex for consumer group path.
ErrInsufficientHosts is thrown when there are not enough hosts to serve the request.
ErrListenerAlreadyExist is thrown on a duplicate AddListener call from the same listener.
ErrLoadReportThrottled is the error returned by LoadReporter when it runs out of tokens to serve the request.
ErrNoClient is returned when the host is already shutdown.
ErrUnknownService is thrown for a service that is not tracked by this instance.
ErrUUIDLookupFailed is thrown when a uuid cannot be mapped to addr.
PathDLQRegex regex for dlq destination path.
PathRegex regex for destination path.
PathRegexAllowUUID For special destinations (e.g.
ServiceToPort is service name to ports mapping This map should be syced with the port nums in config file and use by command line.
UUIDRegex regex for uuid.
# Structs
AckID designates a consumer message to ack/nack.
CounterBank represents a set of counters that all belong to the same group ex - dst or extent counters A counterBank supports methods to inc/dec/get counter values.
Heartbeat is just a timestamp and an identifier.
No description provided by the author
HostInfo is a type that contains the info about a cherami host.
HTTPHandler contains the http handlers for controller.
An Item is something we manage in a key queue.
No description provided by the author
MockLoadReporterDaemon is the mock of common.LoadReporterDaemon interface.
MockLoadReporterDaemonFactory is the mock of common.LoadReporterDaemonFactory interface.
MockRingpopMonitor is an implementation of RingpopMonitor for UTs.
MockService is the mock of common.SCommon interface.
MockTimeSource is a time source for unit tests.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
OpenReplicationReadStreamRequest is the request type for OpenReplicationReadStream API.
OpenReplicationRemoteReadStreamRequest is the request type for OpenReplicationRemoteReadStream API.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
UnboundedSemaphore operates similarly to traditional Go channel-based semaphores, but without any limitation on the number of resources that can be released
See also: https://godoc.org/github.com/dropbox/godropbox/sync2#NewUnboundedSemaphore
.
# Interfaces
No description provided by the author
CliHelper is the interface to help with the args passed to CLI.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
ZoneFailoverManager is a daemon that can be used to manage the zone failover behavior.
# Type aliases
CombinedID is the one which holds session, ackmgr and seqnum together.
ConditionFunc represents an expression that evaluates to true on when some condition is satisfied and false otherwise.
GeometricRollingAverage is the value of a geometrically diminishing rolling average.
HandleSignalFunc is the callback which gets called when a signal is trapped.
No description provided by the author
A MinHeap implements heap.Interface and holds Items.
No description provided by the author
Seconds is time as seconds, either relative or absolute since the epoch.
SequenceNumber is an int64 number represents the sequence of messages in Extent.
UnixNanoTime is Unix time as nanoseconds since Jan 1st, 1970, 00:00 GMT.