log2curl

log2curl is a tool that transforms logs on AWS Cloud Watch to cURL command easily with query ID(AWS given). Something needs to be done before running this script:

1. Retrieve query-id with start-query via AWS CLI or API
2. Define the patterns to render cURL command

How to retrieve query-id

There are several way to retrieve query-id (e.g, CLI, API, SDK). Here is an example to retrieve query-id via aws-cli.

aws logs start-query --log-group-names "/aws/apigateway/welcome" "/aws/lambda/Test01" \
--start-time 1598936400000 --end-time 1611464400000 --query-string "fields @timestamp, @message"

Installation

curl -o log2curl -L https://github.com/whyayen/log2curl/raw/main/bin/log2curl-arm64-darwin

# change permission
chmod +x ./log2curl

Getting Started

Assume we have some logs on Cloud Watch. We get query results like below format when we search on Log Insights page on AWS Console or AWS CLI:

{
    "method": "GET",
    "headers.scheme": "https",
    "host": "api.example.com",
    "path": "/users",
    "headers.Authorization": "Bearer abcdef12345",
    "parameters.user": "Lil Wayne",
    "parameters.page": "1",
    "parameters.per_page": "20"
}

And we have a query-id from executing aws logs start-query:

4ffc3f36-2979-4558-88e8-dbe256d05d20

We can run log2curl to transform logs to cURL commands now:

log2curl --scheme "headers.scheme" --method "method" --host "host" --path "path" --headers-prefix "headers" --parameters-prefix "parameters" cloud-watch -q "4ffc3f36-2979-4558-88e8-dbe256d05d20"

Example response:

Loading default config...
Start to get query results...
Get query results successfully!
Start to generate CURL command...
Finished. Save file in: log2curl.1719928478.txt

log2curl will generate a txt file in your current directory:

$ ls | grep 'log2curl'
log2curl.1669150193.txt

You can change the output by yourself:

log2curl -o ~/test.txt cloud-watch -q "4ffc3f36-2979-4558-88e8-dbe256d05d20"

Generate config

Set patterns each time is annoying. We can generate a config file, and save our settings to config.

log2curl generate --config

Generate successfully

Generate default config to $HOME/.log2curl.json successfully

cat ~/.log2curl.json

{
  "custom": {
    "host": ""
  },
  "key": {
    "headers_prefix": "headers",
    "host": "host",
    "method": "method",
    "parameters_prefix": "parameters",
    "path": "path",
    "scheme": "scheme"
  },
  "whitelist_headers": [
    "Content-Type",
    "Authorization"
  ]
}

Customize configuration

vim ~/.log2curl.json

Whitelist Headers

We could decide which fields could be used in cURL if we have multiple headers field in the log.

For example, if we have the log:

{
    "method": "GET",
    "headers.scheme": "https",
    "host": "api.example.com",
    "path": "/users",
    "headers.Authorization": "Bearer abcdef12345",
    "headers.User-Agent": "something...",
    "headers.Version": "HTTP/1.1",
    "headers.Content-Type": "application/json",
    "parameters.user": "Lil Wayne",
    "parameters.page": "1",
    "parameters.per_page": "20"
}

But our cURL request just need Authorization & Content-Type, we can set whitelist_headers in ~/.log2curl.json:

{
  "whitelist_headers": [
    "Authorization",
    "Content-Type"
  ]
}

or

log2curl --whitelist-headers "Authorization,Content-Type" cloud-watch -q "4ffc3f36-2979-4558-88e8-dbe256d05d20"

Custom Host

Sometimes we want to reproduce a Production log on Staging, we can replace host with custom.host.

{
    "custom": {
        "host": "staging.example.com"
    },
}

or

log2curl --custom-host "staging.example.com" cloud-watch -q "4ffc3f36-2979-4558-88e8-dbe256d05d20"

Failed

Sometimes, log2curl can't parse field or transform to cURL. The result will be printed invalid request in the file.

curl -X GET https://example.com/v1/users/page=1&per_page=50 \ 
 -H 'Authorization: Bearer JzCfIfzMGo'

invalid request

invalid request

curl -X GET https://example.com/v1/items \ 
 -H 'Authorization: Bearer Czx2341xa'

AWS Credentials

log2curl will use the default AWS credentials in your environment. If you want to use a specific profile, you can set the --profile option.

log2curl cloud-watch --profile "Tom" -q "4ffc3f36-2979-4558-88e8-dbe256d05d20"

Or you can set the environment variable AWS_PROFILE to specify the profile. That's supported by AWS SDK by default.

export AWS_PROFILE=Tom

Specify Region

log2curl doesn't support that region option with CLI. If you want to use a specific region, you can set the AWS_REGION environment variable. That's supported by AWS SDK by default.

export AWS_REGION=us-west-2

Specifying Credentials

If you want to specify the credentials, you can set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. That's supported by AWS SDK by default.

export AWS_ACCESS_KEY_ID=YOUR_AKID
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY

Known Issues

• All fields become string after transforming to cURL.