# README
tcpdp 
tcpdp is TCP dump tool with custom dumper and structured logger written in Go.
tcpdp has 3 modes:
- TCP Proxy server mode
- Probe mode ( using libpcap )
- Read pcap file mode
Usage
tcpdp proxy : TCP proxy server mode
$ tcpdp proxy -l localhost:12345 -r localhost:1234 -d hex # hex.Dump()
$ tcpdp proxy -l localhost:55432 -r db.internal.example.com:5432 -d pg # Dump query of PostgreSQL
$ tcpdp proxy -l localhost:33306 -r db.example.com:3306 -d mysql # Dump query of MySQL
With server-starter
https://github.com/lestrrat-go/server-starter
$ start_server --port 33306 -- tcpdp proxy -s -r db.example.com:3306 -d mysql
With config file
$ tcpdp proxy -c config.toml
tcpdp probe : Probe mode (like tcpdump)
$ tcpdp probe -i lo0 -t localhost:3306 -d mysql # is almost the same setting as 'tcpdump -i lo0 host 127.0.0.1 and tcp port 3306'
$ tcpdp probe -i eth0 -t 3306 -d hex # is almost the same setting as 'tcpdump -i eth0 tcp port 3306'
tcpdp read : Read pcap file mode
$ tcpdump -i eth0 host 127.0.0.1 and tcp port 3306 -w mysql.pcap
$ tcpdp read mysql.pcap -d mysql -t 3306 -f ltsv
tcpdp config Create config
$ tcpdp config > myconfig.toml
Show current config
$ tcpdp config
config format
[tcpdp]
pidfile = "/var/run/tcpdp.pid"
dumper = "mysql"
[probe]
target = "db.example.com:3306"
interface = "en0"
bufferSize = "2MB"
immediateMode = false
snapshotLength = "auto"
internalBufferLength = 10000
filter = ""
[proxy]
useServerStarter = false
listenAddr = "localhost:3306"
remoteAddr = "db.example.com:3306"
[log]
dir = "/var/log/tcpdp"
enable = true
enableInternal = true
stdout = true
format = "ltsv"
rotateEnable = true
rotationTime = "daily"
rotationCount = 7
# You can execute arbitrary commands after rotate
# $1 = prev filename
# $2 = current filename
rotationHook = "/path/to/after_rotate.sh"
fileName = "tcpdp.log"
[dumpLog]
dir = "/var/log/dump"
enable = true
stdout = false
format = "json"
rotateEnable = true
rotationTime = "hourly"
rotationCount = 24
fileName = "dump.log"
Installation
$ go get github.com/k1LoW/tcpdp
Architecture
tcpdp proxy connection diagram
client_addr
^
| tcpdp
+----------|---------------+
| v |
| proxy_listen_addr |
| + ^ |
| | | +--------+ |
| |<----+ dumper | |
| | |<--+ | |
| | | +--------+ |
| v + |
| proxy_client_addr |
| ^ |
+----------|---------------+
|
v
remote_addr
tcpdp probe connection diagram
server
+--------------------------+
| |
| +---+---+
| <--------------| eth0 |----------->
| interface +---+---+
| /target ^ |
| | |
| tcpdp | |
| +--------+ | |
| | dumper +------+ |
| +--------+ |
+--------------------------+
tcpdp read diagram
tcpdp
+--------+ STDIN +--------+ STDOUT
| *.pcap +------>+ dumper +-------->
+--------+ +--------+
tcpdp.log ( tcpdp proxy or tcpdp probe )
| key | description | mode |
|---|---|---|
| ts | timestamp | proxy / probe / read |
| level | log level | proxy / probe |
| msg | log message | proxy / probe |
| error | error info | proxy / probe |
| caller | error caller | proxy / probe |
| conn_id | TCP connection ID by tcpdp | proxy / probe |
| target | probe target | proxy / probe |
| dumper | dumper type | proxy / probe |
| use_server_starter | use server_starter | proxy |
| conn_seq_num | TCP comunication sequence number by tcpdp | proxy |
| client_addr | client address | tcpdp.log, hex, mysql, pg |
| remote_addr | remote address | proxy |
| proxy_listen_addr | listen address | proxy |
| direction | client to remote: -> / remote to client: <- | proxy |
| interface | probe target interface | probe |
| mtu | interface MTU (Maximum Transmission Unit) | probe |
| mss | TCP connection MSS (Max Segment Size) | probe |
| probe_target_addr | probe target address | probe |
| filter | BPF (Berkeley Packet Filter) | probe |
| buffer_size | libpcap buffer_size | probe |
| immediate_mode | libpcap immediate_mode | probe |
| snapshot_length | libpcap snapshot length | probe |
| internal_buffer_length | tcpdp internal packet buffer length | probe |
Dumper
mysql
MySQL query dumper
NOTICE: MySQL query dumper require --target option when tcpdp proxy tcpdp probe
| key | description | mode |
|---|---|---|
| ts | timestamp | proxy / probe / read |
| conn_id | TCP connection ID by tcpdp | proxy / probe / read |
| conn_seq_num | TCP comunication sequence number by tcpdp | proxy |
| client_addr | client address | proxy |
| proxy_listen_addr | listen address | proxy |
| proxy_client_addr | proxy client address | proxy |
| remote_addr | remote address | proxy |
| direction | client to remote: -> / remote to client: <- | proxy |
| interface | probe target interface | probe |
| src_addr | src address | probe / read |
| dst_addr | dst address | probe / read |
| probe_target_addr | probe target address | probe |
| proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read |
| proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read |
| query | SQL query | proxy / probe / read |
| stmt_id | statement id | proxy / probe / read |
| stmt_prepare_query | prepared statement query | proxy / probe / read |
| stmt_execute_values | prepared statement execute values | proxy / probe / read |
| character_set | character set | proxy / probe / read |
| username | username | proxy / probe / read |
| database | database | proxy / probe / read |
| seq_num | sequence number by MySQL | proxy / probe / read |
| command_id | command_id for MySQL | proxy / probe / read |
pg
PostgreSQL query dumper
NOTICE: PostgreSQL query dumper require --target option tcpdp proxy tcpdp probe
| key | description | mode |
|---|---|---|
| ts | timestamp | proxy / probe / read |
| conn_id | TCP connection ID by tcpdp | proxy / probe / read |
| conn_seq_num | TCP comunication sequence number by tcpdp | proxy |
| client_addr | client address | proxy |
| proxy_listen_addr | listen address | proxy |
| proxy_client_addr | proxy client address | proxy |
| remote_addr | remote address | proxy |
| direction | client to remote: -> / remote to client: <- | proxy |
| interface | probe target interface | probe |
| src_addr | src address | probe / read |
| dst_addr | dst address | probe / read |
| probe_target_addr | probe target address | probe |
| proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read |
| proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read |
| query | SQL query | proxy / probe / read |
| portal_name | portal Name | proxy / probe / read |
| stmt_name | prepared statement name | proxy / probe / read |
| parse_query | prepared statement query | proxy / probe / read |
| bind_values | prepared statement bind(execute) values | proxy / probe / read |
| username | username | proxy / probe / read |
| database | database | proxy / probe / read |
| message_type | message type for PostgreSQL | proxy / probe / read |
hex
| key | description | mode |
|---|---|---|
| ts | timestamp | proxy / probe / read |
| conn_id | TCP connection ID by tcpdp | proxy / probe / read |
| conn_seq_num | TCP comunication sequence number by tcpdp | proxy |
| client_addr | client address | proxy |
| proxy_listen_addr | listen address | proxy |
| proxy_client_addr | proxy client address | proxy |
| remote_addr | remote address | proxy |
| direction | client to remote: -> / remote to client: <- | proxy |
| interface | probe target interface | probe |
| src_addr | src address | probe / read |
| dst_addr | dst address | probe / read |
| probe_target_addr | probe target address | probe |
| proxy_protocol_src_addr | proxy protocol src address | probe / proxy /read |
| proxy_protocol_dst_addr | proxy protocol dst address | probe / proxy /read |
| bytes | bytes string by hex.Dump | proxy / probe / read |
| ascii | ascii string by hex.Dump | proxy / probe / read |