Categorygithub.com/timartiny/v4vsv6
modulepackage
0.7.1
Repository: https://github.com/timartiny/v4vsv6.git
Documentation: pkg.go.dev

# README

V4 vs V6

This repo will determine differences between A and AAAA record requests from resolvers across the world on domains the Satellite has decided to perform DNS look ups on.

Creating ZDNS files

Our goal is to perform an A and AAAA record request on lots of domains and from lots of resolvers. To facilitate this we will use ZDNS. Specifically by passing in <domain>,<resolver> lines.

Thus we first need to make a file containing those lines.

./domain_resolver_pairs.py will do that:

usage: domain_resolver_pairs.py [-h]
                                domain_json_file resolver_file
                                v4_cartesian_file v6_cartesian_file

positional arguments:
  domain_json_file   Path to the file containing JSON object on each domain
  resolver_file      Path to the file containing resolvers
  v4_cartesian_file  Path to the file to write the cartesian product, with v4
                     resolvers
  v6_cartesian_file  Path to the file to write the cartesian product, with v6
                     resolvers

optional arguments:
  -h, --help         show this help message and exit
./domain_resolver_pairs.py data/satellite-v4-and-v6-and-tls-sept22.json data/aug-30-2-single-resolvers-country-correct-sorted data/v4_cartesian_file data/v6_cartesian_file

ZDNS

With the cartesian product of domains and resolvers added to files and grouped by address type of the resolver you can run 4 different ZDNS calls:

cat data/v4_cartesian_file | ./zdns A --output-file data/v4_cartesian_A_lookups.json
cat data/v4_cartesian_file | ./zdns AAAA --output-file data/v4_cartesian_AAAA_lookups.json
cat data/v6_cartesian_file | ./zdns A --local-addr "<v6 address>" --output-file data/v6_cartesian_A_lookups.json
cat data/v6_cartesian_file | ./zdns AAAA --local-addr "<v6 address>"--output-file data/v6_cartesian_AAAA_lookups.json

The above commands took the following amounts of time, and created various amounts of data:

The following are the stats from the latest run on Nov. 12 v4_cartesian_A_lookups.json: 2.8 GB

real 2695.44
user 3170.05
sys 706.42

v4_cartesian_AAAA_lookups.json: 3.1 GB

real 2462.11
user 3255.20
sys 687.10

v6_cartesian_A_lookups.json: 2.8 GB

real 2976.76
user 3077.58
sys 711.46

v6_cartesian_AAAA_lookups.json: 3.1 GB

real 2885.43
user 3159.38
sys 705.47

The v6 lookups need to come from a v6 address. ZDNS does not automatically select the v6 address of the machine, you need to manually enter it.

Times for the v6 scans:

v6-A-Scan:
    real 3256.31
    user 1635.27
    sys 396.85

Prepare for ZGrab2

The next step will require performing a TLS certificate lookup on all of the answers provided. ZGrab2 accepts input in the form of "<ip address>, <domain>", so we want to make that list, while not adding any unnecessary duplication. To make that list we run:

cat v4_cartesian_A_lookups.json | jq '.name as $name | .data.answers[]? | select(.type=="A") | "\(.answer), \($name)"' > tmp
sort -u tmp > v4_A_ip_domain_list.dat
# Now the AAAA record version
cat v4_cartesian_AAAA_lookups.json | jq -r '.name as $name | .data.answers[]? | select(.type=="AAAA") | "\(.answer), \($name)"' > tmp
sort -u tmp > v4_AAAA_ip_domain_list.dat

The jq command took just over a minute to run and made a file with 9.8M lines, just under 300 MB.

The sort command took about 10 seconds and made a file 100K lines (480K lines for v6), 3.5 MB (24 MB for v6 addresses).

One last simplification, there will be duplicates of "ip, domain" pairs generated from v4 resolvers and v6 resolvers. So their lists should be combined and sorted into only unique pairs:

cat v4_A_ip_domain_list.dat v6_A_ip_domain_list.dat > tmp
sort -u tmp > A_ip_domain_list.dat
# v6 version
cat v4_AAAA_ip_domain_list.dat v6_AAAA_ip_domain_list.dat > tmp
sort -u tmp > AAAA_ip_domain_list.dat

These are very quick and result in two files:

A_ip_domain_list.dat: 3.6MB with 120K lines

AAAA_ip_domain_list.dat: 28MB with 560K lines.

ZGrab2

Once the A and AAAA request files are made we can run two Zgrab2 commands:

cat A_ip_domain_list.dat | zgrab2 --output-file data/A_tls_lookups.json tls
cat AAAA_ip_domain_list.dat | zgrab2 --output-file data/AAAA_tls_lookups.json tls

The following stats are for the Nov. 12 Run

A_tls_lookups.json: 6.9 GB

real 250.98
user 1547.58
sys 72.29

AAAA_tls_lookups.json: 11 GB

real 1688.55
user 7352.45
sys 331.78

Parse Scans

Full README in the directory.

Sample usage:

cmd/parseScans$ go build -o parseScans main.go
cmd/parseScans$ ./parseScans --a-tls-file ../../data/A_tls_lookups_sept_30.json --aaaa-tls-file ../../data/AAAA_tls_lookups_sept_30.json --resolver-country-code ../../data/aug-30-2-single-resolvers-country-correct-sorted --v4-a-raw ../../data/v4_cartesian_A_lookups_sept_29.json --v4-aaaa-raw ../../data/v4_cartesian_AAAA_lookups_sept_29.json --v6-a-raw ../../data/v6_cartesian_A_lookups_sept_29.json --v6-aaaa-raw ../../data/v6_cartesian_AAAA_lookups_sept_29.json --output-file ../../data/domain-resolver-results.json

This run took 20 minutes on my laptop and generated an output file of 7.6 GB formatted like:

{"domain":"bbc.co.uk","resolver_ip":"2002:4022:4e04::4022:4e04","resolver_country":"CA","requested_address_type":"AAAA","results":[{"ip":"2a04:4e42::81","address_type":"AAAA","domain":"bbc.co.uk","supports_tls":true,"timestamp":"2021-09-30T10:22:27-06:00"},{"ip":"2a04:4e42:200::81","address_type":"AAAA","domain":"bbc.co.uk","supports_tls":true,"timestamp":"2021-09-30T10:22:27-06:00"},{"ip":"2a04:4e42:400::81","address_type":"AAAA","domain":"bbc.co.uk","supports_tls":true,"timestamp":"2021-09-30T10:22:27-06:00"},{"ip":"2a04:4e42:600::81","address_type":"AAAA","domain":"bbc.co.uk","supports_tls":true,"timestamp":"2021-09-30T10:22:27-06:00"}]}
{"domain":"bbc.co.uk","resolver_ip":"2002:aa34:7e25::aa34:7e25","resolver_country":"CA","requested_address_type":"AAAA","results":[{"ip":"2a04:4e42::81","address_type":"AAAA","domain":"bbc.co.uk","supports_tls":true,"timestamp":"2021-09-30T10:22:27-06:00"},{"ip":"2a04:4e42:200::81","address_type":"AAAA","domain":"bbc.co.uk","supports_tls":true,"timestamp":"2021-09-30T10:22:27-06:00"},{"ip":"2a04:4e42:400::81","address_type":"AAAA","domain":"bbc.co.uk","supports_tls":true,"timestamp":"2021-09-30T10:22:27-06:00"},{"ip":"2a04:4e42:600::81","address_type":"AAAA","domain":"bbc.co.uk","supports_tls":true,"timestamp":"2021-09-30T10:22:27-06:00"}]}

# Packages

No description provided by the author
No description provided by the author

# Structs

AddressResult will store information about a specific IP address.
DomainResolverResult stores information on how a particular resolver responded to queries for a particular domain, including A and AAAA record requests.