Categorygithub.com/bhojpur/knative-pkgmetrics
package
0.0.3
Repository: https://github.com/bhojpur/knative-pkg.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
36
Dependents
0
Files
2k SLOC

# README

Common metrics export interfaces for Knative

See the Plan for details on where this is heading.

Current status

The code currently uses OpenCensus to support exporting metrics to multiple backends. Currently, two backends are supported: Prometheus and OpenCensus/OTel.

Metrics export is controlled by a ConfigMap called config-observability which is a key-value map with specific values supported for each of the OpenCensus and Prometheus backends. Hot-reload of the ConfigMap on a running process is supported by directly watching (via the Kubernetes API) the config-observability object. Configuration via environment is also supported for use by the queue-proxy, which runs with user permissions in the user's namespace.

Problems

There are currently 6 supported Golang exporters for OpenCensus. We do not want to build all of those backends into the core of knative.dev/pkg and all downstream dependents, and we'd like all the code shipped in knative.dev/pkg to be able to be tested without needing any special environment setup.

With the current direct-integration setup, there needs to be initial and ongoing work in pkg (which should be high-value, low-churn code) to maintain and update stats exporters which need to be statically linked into ~all Knative binaries. This setup also causes problems for vendors who may want or need to perform an out-of-tree integration (e.g. proprietary or partially-proprietary monitoring stacks).

Another problem is that each vendor's exporter requires different parameters, supplied as Golang Options methods which may require complex connections with the Knative ConfigMap. Two examples of this are secrets like API keys and the Prometheus monitoring port (which requires additional service/etc wiring).

See also this doc, where the plan was worked out.

The plan

OpenCensus (and eventually OpenTelemetry) offers an sidecar or host-level agent with speaks the OpenCensus protocol and can proxy from this protocol to multiple backends.

OpenCensus Agent configuration (From OpenCensus Documentation)

We will standardize on export to the OpenCensus export protocol, and encourage vendors to implement their own OpenCensus Agent or Collector DaemonSet, Sidecar, or other OpenCensus Protocol service which connects to their desired monitoring environment. For now, we will use the config-observability ConfigMap to provide the OpenCensus endpoint, but we will work with the OpenTelemetry group to define a kubernetes-friendly standard export path.

Additionally, once OpenTelemetry agent is stable, we will propose adding the OpenTelemetry agent running on a localhost port as part of the runtime contract.

We need to make sure that the OpenCensus library does not block, fail, or queue metrics in-process excessively in the case where the OpenCensus Agent is not present on the cluster. This will allow us to ship Knative components which attempt to reach out the Agent if present, and which simply retain local statistics for a short period of time if not.

Concerns

  • Unsure about the stability of the OpenCensus Agent (or successor). We're currently investigating this, but the OpenCensus agent seems to have been recommended by several others.
  • Running fluentd as a sidecar was very big (400MB) and had a large impact on cold start times.
    • Mitigation: run the OpenCensus agent as a DaemonSet (like we do with fluentd now).
  • Running as a DaemonSet may make it more difficult to ensure that metrics for each namespace end up in the right place.
    • We have this problem with the built-in configurations today, so this doesn't make the problem substantially worse.
    • May want/need some connection between the Agent and the Kubelet to verify sender identities eventually.
    • Only expose OpenCensus Agent on localhost, not outside the node.

Steps to reach the goal

  • Add OpenCensus Agent as one of the export options.
  • Ensure that all tests pass in a non-Google-Cloud connected environment. This is true today. Ensure this on an ongoing basis.
  • Google to implement OpenCensus Agent configuration to match what they are doing for Stackdriver now. (No public issue link because this should be in Google's vendor-specific configuration.)
  • Document how to configure OpenCensus/OpenTelemetry Agent + Prometheus to achieve the current level of application visibility, and determine a long-term course for how to maintain this as a "bare minimum" supported configuration. https://github.com/knative/docs/pull/3005
  • Stop adding exporter features outside of the OpenCensus / OpenTelemetry export as of 0.13 release (03 March 2020). Between now and 0.13, small amounts of additional features can be built in to assist with the bridging process or to support existing products. New products should build on the OpenCensus Agent approach.
  • Removal of the Stackdriver OpenCensus Exporter https://github.com/knative/pkg/issues/2173
  • Revisit Adopting OpenTelemetry SDKs instead of OpenCensus

# Packages

No description provided by the author
Package metricstest simplifies some of the common boilerplate around testing metrics exports.
No description provided by the author

# Functions

Buckets125 generates an array of buckets with approximate powers-of-two buckets that also aligns with powers of 10 on every 3rd step.
BucketsNBy10 generates an array of N buckets starting from low and multiplying by 10 n times.
ClearMetersForTest clears the internal set of metrics being exported, including cleaning up background threads, and restarts cleanup thread.
ConfigMapName gets the name of the metrics ConfigMap.
ConfigMapWatcher returns a helper func which updates the exporter configuration based on values in the supplied ConfigMap.
Domain holds the metrics domain to use for surfacing metrics.
FlushExporter waits for exported data to be uploaded.
InitForTesting initialize the necessary global variables for unit tests.
JSONToOptions converts a json string to ExporterOptions.
MaybeInsertBoolTag conditionally insert the tag when cond is true.
MaybeInsertIntTag conditionally insert the tag when cond is true.
MaybeInsertStringTag conditionally insert the tag when cond is true.
MemStatsOrDie sets up reporting on Go memory usage every 30 seconds or dies by calling log.Fatalf.
NewMemStatsAll creates a new MemStatsProvider with stats for all of the supported Go runtime.MemStat fields.
NewObservabilityConfigFromConfigMap creates a ObservabilityConfig from the supplied ConfigMap.
OptionsToJSON converts an ExporterOptions object to a JSON string.
Record stores the given Measurement from `ms` in the current metrics backend.
RecordBatch stores the given Measurements from `mss` in the current metrics backend.
RegisterResourceView is similar to view.Register(), except that it will register the view across all Resources tracked by the system, rather than simply the default view.
ResponseCodeClass converts an HTTP response code to a string representing its response code class.
UnregisterResourceView is similar to view.Unregister(), except that it will unregister the view across all Resources tracked byt he system, rather than simply the default view.
UpdateExporter updates the exporter based on the given ExporterOptions.
UpdateExporterFromConfigMap returns a helper func that can be used to update the exporter when a config map is updated.
UpdateExporterFromConfigMapWithOpts returns a helper func that can be used to update the exporter when a config map is updated.

# Constants

BackendDestinationKey points to the config map entry key for metrics backend destination.
DefaultLogURLTemplate is used to set the default log url template.
DefaultRequestLogTemplate is the default format for emitting request logs.
DomainEnv points to the metrics domain env var.
EnableProbeReqLogKey is the CM key to enable request logs for probe requests.
EnableReqLogKey is the CM key to enable request log.
ReqLogTemplateKey is the CM key for the request log template.

# Variables

TestOverrideBundleCount is a variable for testing to reduce the size (number of metrics) buffered before OpenCensus will send a bundled metric report.

# Structs

ClientProvider implements the pattern of Kubernetes MetricProvider that may be used to produce suitable metrics for use with metrics.Register().
ExporterOptions contains options for configuring the exporter.
MemStatsProvider is used to expose metrics based on Go's runtime.MemStats.
ObservabilityConfig contains the configuration defined in the observability ConfigMap.
WorkqueueProvider implements workqueue.MetricsProvider and may be used with workqueue.SetProvider to have metrics exported to the provided metrics.

# Type aliases

ResourceExporterFactory provides a hook for producing separate view.Exporters for each observed Resource.
SecretFetcher is a function (extracted from SecretNamespaceLister) for fetching a specific Secret.