# README
Line Protocol
The line protocol is a text based format for writing points to InfluxDB. Each line defines a single point.
Multiple lines must be separated by the newline character \n. The format of the line consists of three parts:
[key] [fields] [timestamp]
Each section is separated by spaces. The minimum required point consists of a measurement name and at least one field. Points without a specified timestamp will be written using the server's local timestamp. Timestamps are assumed to be in nanoseconds unless a precision value is passed in the query string.
Key
The key is the measurement name and any optional tags separated by commas. Measurement names, tag keys, and tag values must escape any spaces or commas using a backslash (\). For example: \ and \,. All tag values are stored as strings and should not be surrounded in quotes.
Tags should be sorted by key before being sent for best performance. The sort should match that from the Go bytes.Compare function (http://golang.org/pkg/bytes/#Compare).
Examples
# measurement only
cpu
# measurement and tags
cpu,host=serverA,region=us-west
# measurement with commas
cpu\,01,host=serverA,region=us-west
# tag value with spaces
cpu,host=server\ A,region=us\ west
Fields
Fields are key-value metrics associated with the measurement. Every line must have at least one field. Multiple fields must be separated with commas and not spaces.
Field keys are always strings and follow the same syntactical rules as described above for tag keys and values. Field values can be one of four types. The first value written for a given field on a given measurement defines the type of that field for all series under that measurement.
- integer - Numeric values that do not include a decimal and are followed by a trailing i when inserted (e.g. 1i, 345i, 2015i, -10i). Note that all values must have a trailing i. If they do not they will be written as floats.
- float - Numeric values that are not followed by a trailing i. (e.g. 1, 1.0, -3.14, 6.0+e5, 10).
- boolean - A value indicating true or false. Valid boolean strings are (t, T, true, TRUE, f, F, false, and FALSE).
- string - A text value. All string values must be surrounded in double-quotes
". If the string contains a double-quote or backslashes, it must be escaped with a backslash, e.g.\",\\.
# integer value
cpu value=1i
cpu value=1.1i # will result in a parse error
# float value
cpu_load value=1
cpu_load value=1.0
cpu_load value=1.2
# boolean value
error fatal=true
# string value
event msg="logged out"
# multiple values
cpu load=10,alert=true,reason="value above maximum threshold"
Timestamp
The timestamp section is optional but should be specified if possible. The value is an integer representing nanoseconds since the epoch. If the timestamp is not provided the point will inherit the server's local timestamp.
Some write APIs allow passing a lower precision. If the API supports a lower precision, the timestamp may also be an integer epoch in microseconds, milliseconds, seconds, minutes or hours.
Full Example
A full example is shown below.
cpu,host=server01,region=uswest value=1 1434055562000000000
cpu,host=server02,region=uswest value=3 1434055562000010000
In this example the first line shows a measurement of "cpu", there are two tags "host" and "region, the value is 1.0, and the timestamp is 1434055562000000000. Following this is a second line, also a point in the measurement "cpu" but belonging to a different "host".
cpu,host=server\ 01,region=uswest value=1,msg="all systems nominal"
cpu,host=server\ 01,region=us\,west value_int=1i
In these examples, the "host" is set to server 01. The field value associated with field key msg is double-quoted, as it is a string. The second example shows a region of us,west with the comma properly escaped. In the first example value is written as a floating point number. In the second, value_int is an integer.
Distributed Queries
# Functions
AppendSeriesKey serializes name and tags to a byte slice.
CloneSeriesSegments returns a copy of a slice of segments.
CreateSeriesSegment generates an empty segment at path.
DifferenceSeriesIDIterators returns an iterator that only returns series which occur the first iterator but not the second iterator.
FilterUndeletedSeriesIDIterator returns an iterator which filters all deleted series.
FindSegment returns a segment by id.
GenerateSeriesKeys generates series keys for a list of names & tags using a single large memory block.
IntersectSeriesIDIterators returns an iterator that only returns series which occur in both iterators.
IsValidSeriesEntryFlag returns true if flag is valid.
IsValidSeriesSegmentFilename returns true if filename is a 4-character lowercase hexadecimal number.
JoinSeriesOffset returns an offset that combines the 2-byte segmentID and 4-byte pos.
MakeTagsKey converts a tag set to bytes for use as a lookup key.
MarshalTags converts a tag set to bytes for use as a lookup key.
MergeMeasurementIterators returns an iterator that merges a set of iterators.
MergeSeriesIDIterators returns an iterator that merges a set of iterators.
MergeTagKeyIterators returns an iterator that merges a set of iterators.
MergeTagValueIterators returns an iterator that merges a set of iterators.
NewConfig returns the default configuration for tsdb.
NewEngine returns an instance of an engine based on its format.
NewEngineOptions constructs an EngineOptions object with safe default values.
NewFieldKeysIterator returns an iterator that can be iterated over to retrieve field keys.
NewIndex returns an instance of an index based on its format.
NewMeasurementFields returns an initialised *MeasurementFields value.
NewMeasurementFieldSet returns a new instance of MeasurementFieldSet.
NewMeasurementSliceIterator returns an iterator over a slice of in-memory measurement names.
NewSeriesFile returns a new instance of SeriesFile.
NewSeriesIDSet returns a new instance of SeriesIDSet.
NewSeriesIDSetIterators returns a slice of SeriesIDSetIterator if all itrs can be type casted.
NewSeriesIDSliceIterator returns a SeriesIDIterator that iterates over a slice.
NewSeriesIndexHeader returns a new instance of SeriesIndexHeader.
NewSeriesIteratorAdapter returns an adapter for converting series ids to series.
NewSeriesPartition returns a new instance of SeriesPartition.
NewSeriesPartitionCompactor returns a new instance of SeriesPartitionCompactor.
newSeriesPointIterator returns a new instance of seriesPointIterator.
NewSeriesQueryAdapterIterator returns a new instance of SeriesQueryAdapterIterator.
NewSeriesSegment returns a new instance of SeriesSegment.
NewSeriesSegmentHeader returns a new instance of SeriesSegmentHeader.
NewShard returns a new initialized Shard.
NewShardError returns a new ShardError.
NewStore returns a new store with the given path and a default configuration.
NewTagKeysIterator returns a new instance of TagKeysIterator.
NewTagKeySliceIterator returns a TagKeyIterator that iterates over a slice.
NewTagValueSliceIterator returns a TagValueIterator that iterates over a slice.
ParseSeriesKey extracts the name & tags from a series key.
ParseSeriesKeyInto extracts the name and tags for data, parsing the tags into dstTags, which is then returned.
ParseSeriesSegmentFilename returns the id represented by the hexadecimal filename.
ReadAllSeriesIDIterator returns all ids from the iterator.
ReadSeriesIndexHeader returns the header from data.
ReadSeriesKey returns the series key from the beginning of the buffer.
ReadSeriesKeyFromSegments returns a series key from an offset within a set of segments.
ReadSeriesSegmentHeader returns the header from data.
RegisteredEngines returns the slice of currently registered engines.
RegisteredIndexes returns the slice of currently registered indexes.
RegisterEngine registers a storage engine initializer by name.
RegisterIndex registers a storage index initializer by name.
SeriesKeySize returns the number of bytes required to encode a series key.
SeriesKeysSize returns the number of bytes required to encode a list of name/tags.
SeriesSegmentSize returns the maximum size of the segment.
SplitSeriesOffset splits a offset into its 2-byte segmentID and 4-byte pos parts.
UnionSeriesIDIterators returns an iterator that returns series from both both iterators.
ValidateFields will return a PartialWriteError if: - the point has inconsistent fields, or - the point has fields that are too long.
# Constants
1GB.
25MB.
DefaultCacheSnapshotWriteColdDuration is the length of time at which the engine will snapshot the cache and write it to a new TSM file if the shard hasn't received writes or deletes.
DefaultCompactFullWriteColdDuration is the duration at which the engine will compact all TSM files in a shard if it hasn't received a write or delete.
DefaultCompactThroughput is the rate limit in bytes per second that we will allow TSM compactions to write to disk.
DefaultCompactThroughputBurst is the rate limit in bytes per second that we will allow TSM compactions to write to disk.
DefaultEngine is the default engine for new shards.
DefaultIndex is the default index for new shards.
DefaultMaxConcurrentCompactions is the maximum number of concurrent full and level compactions that can run at one time.
1MB.
DefaultMaxPointsPerBlock is the maximum number of points in an encoded block in a TSM file.
DefaultMaxValuesPerTag is the maximum number of values a tag can have within a measurement.
DefaultSeriesFileMaxConcurrentSnapshotCompactions is the maximum number of concurrent series partition snapshot compactions that can run at one time.
DefaultSeriesIDSetCacheSize is the default number of series ID sets to cache in the TSI index.
128K.
EOF represents a "not found" key returned by a Cursor.
Series entry constants.
flag + id.
Series entry constants.
Series entry constants.
SeriesFileDirectory is the name of the directory containing series files for a database.
SeriesFilePartitionN is the number of partitions a series file is split into.
SeriesIDSize is the size in bytes of a series key ID.
offset + id.
rhh load factor.
magic + version.
Available index types.
# Variables
ErrEngineClosed is returned when a caller attempts indirectly to access the shard's underlying engine.
ErrFieldTypeConflict is returned when a new field already exists with a different type.
ErrIncompatibleWAL is returned if incompatible WAL files are detected.
ErrIndexClosing can be returned to from an Index method if the index is currently closing.
ErrMultipleIndexTypes is returned when trying to do deletes on a database with multiple index types.
ErrShardDeletion is returned when trying to create a shard that is being deleted.
ErrShardDisabled is returned when a the shard is not available for queries or writes.
ErrShardNotFound is returned when trying to get a non existing shard.
ErrShardNotIdle is returned when an operation requiring the shard to be idle/cold is attempted on a hot shard.
ErrStoreClosed is returned when trying to use a closed Store.
ErrUnknownEngineFormat is returned when the engine format is unknown.
ErrUnknownFieldsFormat is returned when the fields index file is not identifiable by the file's magic number.
ErrUnknownFieldType is returned when the type of a field cannot be determined.
# Structs
Config holds the configuration for the tsbd package.
EngineOptions represents the options used to initialize the engine.
EngineTags holds tags for prometheus
It should not be used for behaviour other than attaching tags to prometheus metrics.
Field represents a series field.
FieldCreate holds information for a field to create on a measurement.
IndexSet represents a list of indexes, all belonging to one database.
KeyValue holds a string key and a string value.
LimitError represents an error caused by a configurable limit.
MeasurementFields holds the fields of a measurement and their codec.
MeasurementFieldSet represents a collection of fields by measurement.
PartialWriteError indicates a write request could only write a portion of the requested values.
SeriesFile represents the section of the index that holds series data.
SeriesIDElem represents a single series and optional expression.
SeriesIDSet represents a lockable bitmap of series ids.
SeriesIDSliceIterator iterates over a slice of series ids.
SeriesIndex represents an index of key-to-id & id-to-offset mappings.
SeriesIndexHeader represents the header of a series index.
SeriesPartition represents a subset of series file data.
SeriesPartitionCompactor represents an object reindexes a series partition and optionally compacts segments.
SeriesSegment represents a log of series entries.
SeriesSegmentHeader represents the header of a series segment.
Shard represents a self-contained time series database.
A ShardError implements the error interface, and contains extra context about the shard that generated the error.
Store manages shards and indexes for databases.
# Interfaces
Engine represents a swappable storage engine for the shard.
FileStoreObserver is passed notifications before the file store adds or deletes files.
MeasurementIterator represents a iterator over a list of measurements.
SeriesElem represents a generic series element.
SeriesIDIterator represents a iterator over a list of series ids.
SeriesIDSetIterator represents an iterator that can produce a SeriesIDSet.
SeriesIDSets provides access to the total set of series IDs.
SeriesIterator represents a iterator over a list of series.
SeriesKeyIterator represents an iterator over a list of SeriesKeys.
TagKeyIterator represents a iterator over a list of tag keys.
TagValueIterator represents a iterator over a list of tag values.
# Type aliases
EngineFormat represents the format for an engine.
KeyValues is a sortable slice of KeyValue.
NewEngineFunc creates a new engine.
NewIndexFunc creates a new index.
SeriesIDElems represents a list of series id elements.
Shards represents a sortable list of shards.