Shim loggers for containerd

Shim loggers for containerd is a collection of containerd compatible logger implementations that send container logs to various destinations. The following destinations are currently supported:

• Amazon CloudWatch Logs
• Splunk
• Fluentd

Build

Make sure you have golang installed. Then simply run make build to build the respective binaries. You might need to execute make get-deps to install some of the dependencies.

Usage

Containerd supports shim plugins that redirect container output to a custom binary on Linux using STDIO URIs with runc v2 runtime. These loggers can be used either programmatically or with the ctr tool.

When using the NewTask API

When using the NewTask API to start a container, simply provide the path to the built binary file shim-loggers-for-containerd and required arguments. Note it's a good practice to clean up container resources with Delete API call after container exited as the container IO pipes are not closed if the shim process is still running.

Example:

NewTask(context, cio.BinaryIO("/usr/bin/shim-loggers-for-containerd", args))

When using the ctr tool

When using ctr tool to run a container, provide the URI path to the binary file shim-loggers-for-containerd and required arguments as part of the path.

Example:

ctr run \
    --runtime io.containerd.runc.v2 \
    --log-uri "binary:///usr/bin/shim-loggers-for-containerd?--log-driver=awslogs&--arg1=value1&-args2=value2" \
    docker.io/library/redis:alpine \
    redis

Arguments

Common arguments

The following list of arguments apply to all of the shim logger binaries in this repo:

Name	Required	Description
log-driver	Yes	The name of the shim logger. Can be any of awslogs, splunk or fluentd.
container-id	Yes	The container id
container-name	Yes	The name of the container
mode	No	Either blocking or non-blocking. In the non-blocking mode, log events are buffered and the application continues to execute even if these logs can't be drained or sent to the destination. Logs could also be lost when the buffer is full.
max-buffer-size	No	Only supported in non-blocking mode. Set to 1m (1MiB) by default. Example values: 200, 4k, 1m etc.
uid	No	Set a custom uid for the shim logger process. 0 is not supported.
gid	No	Set a custom gid for the shim logger process. 0 is not supported.
cleanup-time	No	Set a custom time for the shim logger process clean up itself. Set to 5s (5 seconds) by default. Note the maximum supported value is 12 seconds, since containerd shim sets shim logger cleanup timeout value as 12 seconds. See reference.
container-image-id	No	The container image id. This is part of the docker config variables that can be logged by splunk log driver.
container-image-name	No	The container image name. This is part of the docker config variables that can be logged by splunk log driver.
container-env	No	The container environment variables map in json format. This is part of the docker config variables that can be logged by splunk log driver.
container-labels	No	The container labels map in json format. This is part of the docker config variables that can be logged by splunk log driver.

Windows specific arguments

The following list of arguments apply to Windows shim logger binaries in this repo:

Name	Required	Description
log-file-dir	No	Only supported in Windows. Will be the path where shim logger's log files are written. By default it is \ProgramData\Amazon\ECS\log\shim-logger
proxy-variable	No	Only supported in Windows. The proxy variable will set the HTTP_PROXY and HTTPS_PROXY environment variables.

Additional log driver options

Amazon CloudWatch Logs

The following additional arguments are supported for the awslogs shim logger binary, which can be used to send container logs to Amazon CloudWatch Logs.

Name	Required	Description
awslogs-group	Yes	The log group in which the log stream for the container will be created.
awslogs-stream	Yes	The log stream name to stream container logs to.
awslogs-region	Yes	The region name in which the log group and log stream needs to be created in.
awslogs-credentials-endpoint	Yes	The endpoint from which credentials are retrieved from to connect to Amazon CloudWatch Logs.
awslogs-create-group	No	Set to false by default. If the provided log group name does not exist and this value is set to false, the binary will directly exit with an error
awslogs-create-stream	No	Set to true by default. The log stream will always be created unless this value specified to false explicitly. If the value is false and the log stream does not exist, logging will fail silently instead of failing the container task.
awslogs-multiline-pattern	No	Matches the behavior of the awslogs Docker log driver.
awslogs-datetime-format	No	Matches the behavior of the awslogs Docker log driver
awslogs-endpoint	No	Matches the behavior of the awslogs Docker log driver

Splunk

The following additional arguments are supported for the splunk shim logger binary, which can be used to send container logs to splunk. You can find a description of what these parameters are used for here.

Name	Required
splunk-token	Yes
splunk-url	Yes
splunk-source	No
splunk-sourcetype	No
splunk-index	No
splunk-capath	No
splunk-caname	No
splunk-insecureskipverify	No
splunk-format	No
splunk-verify-connection	No
splunk-gzip	No
splunk-gzip-level	No
splunk-tag	No
labels	No
env	No
env-regex	No

Fluentd

The following additional arguments are supported for the fluentd shim logger binary, which can be used to send container logs to Fluentd. Note that all of these are optional arguments.

Name	Required	Description
fluentd-address	No	The address of the Fluentd server to connect to. By default, the localhost:24224 address is used.
fluentd-async-connect	No	Specifies if the logger connects to Fluentd in background. Defaults to false.
fluentd-sub-second-precision	No	Generates logs in nanoseconds. Defaults to true. Note that this is in contrast to the default behaviour of fluentd log driver where it defaults to false.
fluentd-buffer-limit	No	Sets the number of events buffered in memory. The total memory limit is approximately this limit * the average log line length. Defaults to 1048576.
fluentd-tag	No	Specifies the tag used for log messages. Defaults to the first 12 characters of container ID.

License

This project is licensed under the Apache-2.0 License.