Analytics Log Agent

This is an analytics log agent for Matomo, running on the server you want to have log data from, to send to your Matomo instance.

There is also an integration for the Matomo plugin Agent that could be used for cases when you want to have reports for errors, like 404 logs.

This agent is in it's early stages, and at this point NOT recommended for production use.

Usage

The idea of the usages of this agent are:

If you can't use normal JavaScript tracking for you website, you instead use the logs and tail them and send the data to Matomo. (If you want to send server logs by daily batches, you instead use Log Analytics).
Track user agents that doesn't use a browser to visit your sites - like some bots and people using command lines tool to scrap your website.
Generate test data for you Matomo with logs generated from something like Dummy Log Creator.

For now, you need to build the agent manually, and then you can add in your preferred path. A config file is required, and as default it should be placed /opt/log-agent/config.toml, you can override this when starting the Matomo agent with the -config flag pointing to path to the file, like:

./log-agent --config /usr/local/bin/log-agent-config.toml

The file config.toml.example kan be used as a start for your config.tml file.

Config

Flags

Flag	Type	Default	Description
`--config`	`string`	`/opt/log-agent/config.toml`	Path to the configuration file.
`--catlog`	`bool`	`false`	Simulate `cat` command for a log file. If set to `true`, processes log file in one go.
`--rps`	`int`	`1`	Requests per second limit for `catlog` mode. Controls the rate of log file processing.
`--matomo-url`	`string`	`""`	Matomo URL. Overrides the value set in the config file.
`--token-auth`	`string`	`""`	Matomo authentication token. Overrides the value set in the config file.
`--site-id`	`string`	`""`	Matomo site ID. Overrides the value set in the config file.
`--plugin`	`bool`	`false`	If using the Matomo Agent plugin, set this flag to enable plugin functionality.
`--downloads`	`bool`	`true`	Enable or disable download tracking. Overrides the config file setting.
`--log-format`	`string`	`""`	Log format. Valid options: `nginx`, `apache`, or `csv`. Overrides the config file setting.
`--log-path`	`string`	`""`	Path to the log file. Overrides the value set in the config file.
`--user-agents`	`string`	`""`	Comma-separated list of user agents to track. Overrides the config file setting.
`--log-level`	`string`	`""`	Log level. Valid options: `debug`, `info`, `warn`, or `error`. Overrides the config file setting.
`--log-file`	`string`	`""`	Path to the agent's log file. Overrides the value set in the config file.
`--collect-title`	`bool`	`false`	Collect titles from log URLs
`--title-domain`	`string`	`""`	Override domain in log or csv with this domain for getting title (this is not implemented yet)
`--batch`	`string`	`""`	Run in batch mode, send 200 log lines per request

Each flag can be used to override corresponding values in the config.toml file, allowing you to customize the agent's behavior via command-line arguments.

File

Options for config.toml:

Config	Description	Default	Required
`matomo.url`	URL to your Matomo instances	-	Yes
`matomo.site_id`	Site id in Matomo to track to	1	Yes
`matomo.token_auth`	Token auth to your Matomo instance	-	Yes
`matomo.plugin`	If you want to use the Agent plugin in Matomo	false	No
`matomo.downloads`	If you want to track downloads	true	No
`log.log_format`	Which log format the log has	-	Yes
`log.log_path`	Path to the log to tail	-	Yes
`log.user_agents`	Array of User Agents that should be tracked	-	No
`agent.log_level`	Log level for Matomo agent	-	Yes
`agent.log_file`	File to log to	-	Yes
`title.collect_titles`	Enrich tracking with query URL in log for HTML title	false	No
`title.title_domain`	Override domain in log or csv with this domain for getting title (this is not implemented yet)	-	No
`title.cache_file`	Path to cache file	/tmp/matomo_agent-url_title_cache.txt	No

Log format

Apache and Nginx

Apache and Nginx log format supported is the combined log format, that matches this regex:

(?P<ip>\S+) - - \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<url>\S+) (?P<protocol>\S+)" (?P<status>\d+) (?P<size>\d+) "(?P<referrer>[^"]*)" "(?P<user_agent>[^"]*)"

CSV

If using CSV file, the format need to be:

timestamp, req_method, req_host, req_uri, resp_status, client_ip, req_referer, req_user_agent

Build

go build -o log-agent .

Install

Copy config.toml.example to default (or your preferred destination), /opt/log-agent/config.toml Add settings for the agent in config.toml, run log-agent.

./log-agent

config.toml could also be placed in another place, then you need to start the agent with the --config flag.

./log-agent --config /PATH/TO/YOUR/config.toml

Tail or cat

Tail

Just start like:

./log-agent [--config config.toml]

Cat

As an option you could read the logfile from start to end, if you have a log file that is not updated anymore, to do that you could run it the agent like:

./log-agent --config config.toml --catlog --rps 5

The catlog flag makes the agent run just once, and the rps flag is to set how many requests per second if needed, default is 1.

We do though recommend using Matomos official Log Analytics for this.

Todos

To configure to track bots only, this could be used: https://github.com/robicode/device-detector/tree/main
Add releases https://github.com/goreleaser/goreleaser

Linting

golangci-lint run

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/

# README