modulepackage
0.0.0-20250225160627-a5c4e050d9d8
Repository: https://github.com/mongodb/ftdc.git
Documentation: pkg.go.dev
# README
======================================================
ftdc -- Golang FTDC Parsing and Generating Library
Overview
FTDC, originally short for full time diagnostic data capture, is MongoDB's internal diagnostic data collection facility. It encodes data in a space-efficient format, which allows MongoDB to record diagnostic information every second, and store weeks of data with only a few hundred megabytes of storage.
This library provides a fully-featured and easy to use toolkit for
interacting data stored in this format in Go programs. The library
itself originated as a project by 2016 Summer interns at MongoDB <https://github.com/10gen/ftdc-utils>_ but has diverged substantially
since then.
Features
Current
Currently the library provides parsing of the FTDC data format and
several ways of iterating these results. Additionally, it provides the
ability to create FTDC payloads, and is the only extant (?) tool for
generating FTDC data outside of the MongoDB code base.
The library includes tools for generating FTDC payloads and document
streams as well as iterators and tools for accessing data from FTDC
files. All functionality is part of the ``ftdc`` package, and the API
is fully documented.
Upcoming
-
(pending requests and use-cases) mode to read FTDC data without flattening the document structure.
-
command line tools for parsing and generating FTDC payloads.
-
helpers for generating default collector configurations.
-
combined check
-
improved collector performance.
... and more
Documentation
All documentation is in the godoc <https://godoc.org/github.com/mongodb/ftdc>_.
Participate
File tickets in the MAKE <https://jira.mongodb.org/browse/MAKE>_
project on the MongoDB jira. The repository will shortly move back to
an offical MongoDB GitHub organization.
Pull requests are welcome.
# Packages
No description provided by the author
Recorder
The Recorder interface provides an interface for workloads and operations to to collect data about their internal state, without requiring workloads to be concerned with data retenation, storage, or compression.
Package hdrhistogram provides an implementation of Gil Tene's HDR Histogram data structure.
Package metrics includes data types used for Golang runtime and system metrics collection.
No description provided by the author
No description provided by the author
# Functions
ConvertFromCSV takes an input stream and writes ftdc compressed data to the provided output writer.
DumpCSV writes a sequence of chunks to CSV files, creating new files if the iterator detects a schema change, using only the number of fields in the chunk to detect schema changes.
FlushCollector writes the contents of a collector out to an io.Writer.
GetGennyTime determines the StartTime and EndTime of a genny workload file by passing through all of its chunks.
NewBasicCollector provides a basic FTDC data collector that mirrors the server's implementation.
NewBatchCollector constructs a collector implementation that builds data chunks with payloads of the specified number of samples.
NewBufferedCollector wraps an existing collector with a buffer to normalize throughput to an underlying collector implementation.
NewDynamicCollector constructs a Collector that records metrics from documents, creating new chunks when either the number of samples collected exceeds the specified max sample count OR the schema changes.
NewSamplingCollector wraps a different collector implementation and provides an implementation of the Add method that skips collection of results if the specified minimumInterval has not elapsed since the last collection.
NewStreamingCollector wraps the underlying collector, writing the data to the underlying writer after the underlying collector is filled.
NewStreamingDynamicCollector has the same semantics as the dynamic collector but wraps the streaming collector rather than the batch collector.
NewStreamingUncompressedCollectorJSON constructs a collector that resolves data into a stream of JSON documents.
NewStreamingUncompressedCollectorBSON constructs a collector that resolves data into a stream of BSON documents.
NewUncompressedCollectorBSON constructs a collector that resolves data into a stream of BSON documents.
NewUncompressedCollectorJSON constructs a collector that resolves data into a stream of JSON documents.
NewSynchronizedCollector wraps an existing collector in a synchronized wrapper that guards against incorrect concurrent access.
NewUncompressedCollectorBSON constructs a collector that resolves data into a stream of BSON documents.
NewUncompressedCollectorJSON constructs a collector that resolves data into a stream of JSON documents.
No description provided by the author
ReadChunks creates a ChunkIterator from an underlying FTDC data source.
ReadMatrix returns a "matrix format" for the data in a chunk.
ReadMetrics returns a standard document iterator that reads FTDC chunks.
ReadSeries is similar to the ReadMatrix format, and produces a single document per chunk, that contains the flattented keys for that chunk, mapped to arrays of all the values of the chunk.
ReadStructuredMetrics returns a standard document iterator that reads FTDC chunks.
TranslateGenny exports the contents of a stream of genny ts.ftdc chunks into cedar ftdc which is readable using t2.
WriteCSV exports the contents of a stream of chunks as CSV.
# Structs
Chunk represents a 'metric chunk' of data in the FTDC.
ChunkIterator is a simple iterator for reading off of an FTDC data source (e.g.
CollectorInfo reports on the current state of the collector and provides introspection into the current state of the collector for testing, transparency, and to support more complex collector features, including payload size controls and schema change.
GennyOutputMetadata aids in the genny output translation process.
Metric represents an item in a chunk.