pkg.gl

go-loadgen

go-loadgen is a log load generator tool meant for testing the logging infrastructure. It is capable of producing structured or unstructured/structure logs with random alphanumeric strings and can also playback any input file at a specified rate.

Build

This program requires GO 1.19

go install github.com/intuit/go-loadgen@latest
ln -s $GOPATH/bin/go-loadgen /usr/bin/loadgen

developers

//Build command 
go build -o loadgen

Usage

Use the help command to find all available commands and flags.

HHNLWB76D9289E:go-loadgen cbhatt1$ ./loadgen --help
loadgen is a tool which generates test-data (synthetic/replay) at a controlled rate to test logging infrastructure.

Usage:
  loadgen [COMMANDS] [flags]
  loadgen [command]

Available Commands:
  help           Help about any command
  random-strings test data is generated by producing random alphanumeric strings.
  replay-file    test data is generated by replaying a file.

Flags:
      --config string                         -config /path/to/source/jsonfile (default "cfg")
      --custom-timestamp-format string        Optional custom timestamp format.The default format is 2006-01-02T15:04:05.000-07:00
  -x, --disable-timestamp                     use --disable-timestamp false will disable timestamp injection while replaying logs. When this flag is not used. Do not modify this parameter for auto-generated random alphanumberic logs.Disabling timestamps is typically suitable while replaying a log-file with pre-existing timestamps.
  -d, --duration int                          Duration of the job in seconds
      --enable-config                         use config file for configuration (default false)
  -r, --enable-log-rotation                   by default log rotation is off, use this flag to turn it on. Example --enable-log-rotation will turn log rotation on
      --enable-metrics                        true, enables prometheus metrics on http://<localhost>:<port>/metric HTTP endpoint for line emit rates and byte count. default is off.
      --file-count int                        target requests/sec. Note: log rotation does not work in multi file scenario. (default 1)
  -h, --help                                  help for loadgen
  -s, --lines-per-second int                  target lines/second (default 1000)
      --log-rotation-max-size-megabytes int   Rotate after N MegaBytes. example 100.  (default 100)
  -p, --metrics-server-port string            prometheus metrics server port. default is port 8080.  (default "8080")
  -o, --output string                         file path of output log file ex: /var/tmp/test.log
      --result-log string                     Path of the results log where metrics such as total_lines_generated will be log.
      --tags string                           Custom hardcoded tags in the form of K=V (default "t")

Example-1: To start a test with random-strings in the log file Most of the local flags to this command is mandatory.

The command generates 5 lines/second, with each line of length 1KiloBytes (1000 ASCII characters) and log entry spanning not more than 1 line (line-count) per event (no line breaks). The command produces a log file which grows at a rate of 5 KiloBytes/second and runs for 5 seconds. The overall size of the file after the test completes will be equal to 5 KB*5 seconds = 25KB

./loadgen random-strings --file-count 1 \
--duration 5 \
--enable-log-rotation \
--line-count 1 \
--lines-per-second 5 \
--multi-line-percent 0 \
--line-length  1000 \
--output /var/log/test-alphanumeric.log

Example-2: To start a test with random-strings in the log file with 100% of the log entries containing multi-line.

The flag --line-count and --multi-line-percent determines the number of line-breaks in the multi-line log entry. The resulting log file will be written at a byte rate of line-length*line-count*lines-per-second = 1000*5*5 = 25000 bytes/second. 100% of the log entries in the resulting log file contains line-break character at the end of each line, with each line spanning 1000 characters in length

./loadgen random-strings --file-count 1 \
--duration 5 \
--enable-log-rotation \
--line-count 5 \
--lines-per-second 5 \
--multi-line-percent 100 \
--line-length  1000 \
--output /var/log/test-alphanumeric.log

Example-3: To output log lines to "stdout" instead of a log file, simply declar '--output stdout' parameter. Refer to the example below. Generating logs to stdout is especially useful to simulate logging performance for application pod/containers deployed in Kubernetes.

./loadgen random-strings --file-count 1 \
--duration 5 \
--line-count 1 \
--lines-per-second 5 \
--multi-line-percent 0 \
--line-length  1000 \
--output stdout

File Replay/File Playback

To start a log replay test with a specified input file.

Note: Local flags such as duration and lines-per-second is required.

./loadgen replay-file \
--input-file-path /Users/cbhatt1/git-logging/go-loadgen/test-data/sbg-pp-data.txt \
--output test.log\
--lines-per-second 1 \
--duration 5

You can also specify the number of times a file should be replayed using the --replay-count flag.

Disable timestamp injection for replay tests

You can disable timestamp injection if your replay reference log files already have timestamp using --disable-timestamp boolean flag

If you use the --replay-count flag, then the flag --duration cannot be used, since the former flag becomes the test termination criteria, and you cannot simultaneously use two different termination criteria for the test.

In the below example, the test will complete after replaying the file "5" times.

Example:

./loadgen replay-file \
--input-file-path /Users/cbhatt1/git-logging/go-loadgen/test-data/sbg-pp-data.txt \
--output test.log\
--lines-per-second 1 \
--replay-count 5

Customized fields using tags

go-loadgen can introduce custom fields using the tags flag. Provide tags in the form of K=V and those tags will be included in the log event. This particularly useful when metadata such as testId and/or line count should be present in the logs for backend validation.

 ./loadgen replay-file \
--input-file-path /Users/cbhatt1/git-logging/go-loadgen/test-data/sbg-pp-data.txt \
--output test.log\
--enable-log-rotation \
--log-rotation-max-size-megabytes 10 \
--lines-per-second 1 \
--file-count 3 \
--duration 5 \
--tags "lineCount=1 testId=1"

Defining a test results log file

go-loadgen can generate test results log file in the specified path. --result-log is the persistent flag which enables that. Use it as shown below. It generates a file with "total_lines_generated=x" entry, which can be used as a way to assert whether the generated load is equal to the load received on the backend.

 ./loadgen replay-file \
--input-file-path /Users/cbhatt1/git-logging/go-loadgen/test-data/sbg-pp-data.txt \
--output test.log\
--enable-log-rotation \
--log-rotation-max-size-megabytes 10 \
--lines-per-second 1 \
--file-count 1 \
--duration 5 \
--result-log /var/tmp/result.log"

Log rotation and file count

For certain tests, it is critical to generate multiple log files along with log-rotation and backup file retention policies. Here is how you can achieve that using go-loadgen:

 ./loadgen replay-file \
--input-file-path /Users/cbhatt1/git-logging/go-loadgen/test-data/sbg-pp-data.txt \
--output test.log\
--enable-log-rotation \
--log-rotation-max-size-megabytes 10 \
--lines-per-second 1 \
--file-count 3 \
--duration 5

In the above CLI command :

The argument --enable-log-rotation enables log-rotation
The argument pair --rotation-max-file-size-megabytes 10 specifies how large a file can grow before the file is rotated with a new one. In this example the limit is 10MB.
The argument pair --file-count 3 determines how many log files the test has to create and update simultenously.

go-loadgen currently allows for a maximum of 3 backup files after which the oldest file will be deleted.

Log rotation default settings

Property	Default	Comments
MaxFileSize	500MB	Maximum size the file is allowed to grow
MaxAge	1 Day	After MaxAge exceeds the file is rotated automatically even if size is within the max limit.
MaxBackups	3	number of rotated files.
File Compression	Disabled

Common options

Customizing log timestamp Use 2006-01-02T15:04:05.000-07:00 with the --custom-timestamp-format flag to provide custom timestamp format. Example:

./loadgen ... --custom-timestamp-format 2006/01/02T15-04-05 ...

The above custom timestamp will override the default timestamp in both alphanumeric random logs and replay logs depending on which mode the test is executed.

Prometheus Metrics

To start a test with prometheus metrics exposed to http://localhost:<port>/metrics endpoint, use the "enable-metrics" global flag Note: the default prometheus listener port is "8080" unless specified using the --metrics-server-port <port> flag

 ./loadgen replay-file \
    --input-file-path /Users/cbhatt1/git-logging/go-loadgen/test-data/sbg-pp-data.txt \
    --output test.log\
    --lines-per-second 1 \
    --duration 5 \
    --enable-metrics \
    --metrics-server-port 8081

Example output using curl:

 cbhatt1$ curl http://localhost:8081/metrics
# HELP total_bytes_processed Total bytes processed
# TYPE total_bytes_processed counter
total_bytes_processed 2.23982e+07
# HELP total_events_processed Total log events generated by the tool.

# README