Categorygithub.com/heroku/log-shuttle
modulepackage
0.22.0
Repository: https://github.com/heroku/log-shuttle.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
25
Dependents
5
Files
1.4k SLOC

# README

Log Shuttle

Travis Releases GoDoc

Log-shuttle is an open source UNIX program that delivers messages from applications and daemons to log routers and processors via HTTPs.

One of the motivations behind log-shuttle is to provide a simpler form of encrypted & authenticated log delivery. Using HTTPs & Basic Authentication is simpler than the techniques described in RFC5425. TLS transport mapping for Syslog requires that you maintain both client & server certificates for authentication. In multi-tenant environments, the maintenance of certificate management can be quite burdensome.

Log-shuttle accepts input from stdin in a newline (\n) delimited format.

When using log-shuttle with logplex it is recommended that you spawn 1 log-shuttle per logplex token. This will isolate data between tokens and ensure a good QoS.

When using log-shuttle with either Amazon's Kinesis or Amazon's Cloud Watch Logs services all the details for the service are supplied in the -logs-url (or $LOGS_URL env variable). See the Amazon Endpoints documentation for supported regions and hostnames. See the Kinesis and Cloud Watch Logs sections of this document.

To block as little as possible, log-shuttle will drop outstanding batches if it accumulates > -back-buff amount.

Kinesis

log-shuttle sends data into Kinesis using the PutRecords API call. Each Kinesis record is encoded as length prefixed rfc5424 encoded logs as per rfc6587 (this is the same format logplex accepts). One record per log line.

Log-shuttle expects the following encoding of -logs-url when using Amazon Kinesis:

```
https://<AWS_KEY>:<AWS_SECRET>@kinesis.<AMAZON_REGION>.amazonaws.com/<STREAM NAME>
```

Kinesis Caveats

Things that should be handled better/things you should know:

  1. AWS_SECRET, AWS_KEY, AMAZON_REGION & STREAM NAME need to be properly url encoded.
  2. log-shuttle assumes a 200 response means everything is good. Kinesis can return a 200, meaning the http request was good, but include per record errors in the response body.
  3. The maximum number of records in a PutRecords requests is 500, so set the batch size no higher than 498 (500 - 2 records for possible drops / lost).
  4. Logplex max line length is 10k, Kinesis max record size is 50k of base64 encoded data. A -max-line-length of somewhere less than 37500 should work for Kinesis w/o causing errors.
  5. Kinesis does not support the -gzip option as that option compresses the body of the request.
  6. Even with -kinesis-shards, no guarantees can be made about writing to unique shards.

CloudWatch Logs

log-shuttle sends logs to CloudWatch Logs using the PutLogEvents API call. Each log line is a seperate event and delivered in the same order received by log-shuttle.

log-shuttle uses the aws-sdk-go library to determine the AWS credentials it will use, starting CloudWatch Logs sequence token and to sign requests, but does not otherwise use the aws-sdk-go's clients.

log-shuttle expects the following encoding of -logs-url to use Amazon CloudWatch Logs:

```
https://logs.<AMAZON_REGION>.amazonaws.com/<log group name>/<log stream name>
```

Cloudwatch Caveats

Things that should be handled better/things you should know:

  1. log-shuttle doesn't do any special handling around service limits atm: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html
  2. log-shuttle needs more testing against CloudWatch to ensure it handles errors and limits better
  3. PutLogEvents has a hard upper limit of 5 requests per second. If log-shuttle's input / settings causes > 5 batches per second to be created this limit could be exceeded. Modulating batch size and wait duration would be needed to fix this on a case by case basis.
  4. Really long lines are not split like they are with logplex

Install

go get -u github.com/heroku/log-shuttle/...

After that $GOPATH/bin/log-shuttle should be available.

Making Debs

Requires:

  • dpkg (see also brew install dpkg)
  • go 1.6+
make debs

Docker

There is a Makefile target named docker that can be used to build a docker image.

Hacking on log-shuttle

Fork the repo, hack, submit PRs.

Testing

go test -v ./...

Submitting Code

  • Open an issue on GitHub.
  • Keep changes in a feature branch
  • Submit PR

License

Copyright (c) 2013-15 Heroku Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

# Packages

No description provided by the author

# Functions

NewBatch returns a new batch with a capacity pre-set.
NewBatcher created an empty Batcher for the provided shuttle.
NewCloudWatchLogsFormatterFunc that creates a HTTPFormatterFunc for formatting batched into Cloud Watch Logs requests tied to a specific region/host/log group/log stream.
NewConfig returns a newly created Config, filled in with defaults.
NewCounter returns a new Counter initialized to the initial value.
NewGzipFormatter builds a new GzipFormatter with the supplied delegate.
NewHTTPOutlet returns a properly constructed HTTPOutlet for the given shuttle.
NewKinesisFormatter constructs a proper HTTPFormatter for Kinesis http targets.
NewLogLineReader constructs a new reader with it's own Outbox.
NewLogplexBatchFormatter returns a new LogplexBatchFormatter wrapping the provided batch.
NewLogplexErrorFormatter returns a LogplexLineFormatter for the error data.
NewLogplexLineFormatter returns a new LogplexLineFormatter wrapping the provided LogLine.
NewMetricsReporter returns a properly constructed MetricsReporter.
NewShuttle returns a properly constructed Shuttle with a given config.

# Constants

Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Logplex max is 10000 bytes, so default to that.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
Default option values.
DepthHighWatermark is the high watermark, beyond which the outlet looses batches instead of retrying.
EOFRetrySleep is the amount of time to sleep between retries caused by an io.EOF, in ms.
length prefixed RFC5424 formatted lines.
default, just text.
newline termined RFC5424 formated lines.
LogplexBatchTimeFormat is the format of timestamps as expected by Logplex.
LogplexContentType is the content type logplex expects.
OtherRetrySleep is the time to sleep between retries for any other error, in ms.
RetryWithTypeFormat if the format string for retries that also have a type.

# Variables

Defaults that can't be constants.

# Structs

Batch holds incoming log lines and provides some helpers for dealing with their grouping.
Batcher coalesces logs coming via inLogs into batches, which are sent out via outBatches.
CloudWatchLogsFormatter formats a batch of logs for the Amazon Cloud Watch Logs service.
Config holds the various config options for a shuttle.
Counter is used to track 2 values for a given metric.
GzipFormatter is an HTTPFormatter that is built with a delegate HTTPFormatter but which compresses the request body.
HTTPOutlet handles delivery of batches to HTTP endpoints by creating formatters for each request.
KinesisFormatter formats batches destined for AWS Kinesis HTTP endpoints Kinesis has a very small payload side, so recommend setting config.BatchSize in the 1-3 range so as to not loose logs because we go over the batch size.
KinesisRecord is used to marshal LoglexLineFormatters to Kinesis Records for the PutRecords API Call.
LogLine holds the new line terminated log messages and when shuttle received them.
LogLineReader performs the reading of lines from an io.ReadCloser, encapsulating lines into a LogLine and emitting them on outbox.
LogplexBatchFormatter implements on io.Reader that returns Logplex formatted log lines.
LogplexLineFormatter formats individual loglines into length prefixed rfc5424 messages via an io.Reader interface.
MetricsReporter handles reporting of metrics to a specified source at a given duration.
Shuttle is the main entry point into the library.

# Interfaces

HTTPFormatter is the interface that http outlets use to format a HTTP request.
ResponseHandler needs to handle responses to the requests an outlet submits.
SubFormatter formats a complete batch or a subsection of a batch.

# Type aliases

NewHTTPFormatterFunc defines the function type for defining creating and returning a new Formatter.