az-blob-hashdeep

A simple tool for generating hashdeep compatible output for an Azure Blob Storage container. Useful for verifying migration of data to/from Azure.

The default implementation requires that the Content-MD5 is set for all the blobs in a container. It will not verify the hashes, it will only be used for the final output. Optionally --calculate can be set to download every file to memory in order to calculate the actual MD5.

How to use?

Get precompiled binaries from the releases page or use the Docker image: evenh/az-blob-hashdeep.

export AZURE_ACCOUNT_NAME=myaccount
export AZURE_ACCOUNT_KEY=secretKey # --sas-key is also supported as an alternative
export AZURE_CONTAINER=migrationcontainer

./az-blob-hashdeep generate --account-name=$AZURE_ACCOUNT_NAME \
                            --account-key=$AZURE_ACCOUNT_KEY \
                            --container=$AZURE_CONTAINER \
                            --output ~/$AZURE_ACCOUNT_NAME-$AZURE_CONTAINER.hashdeep

This will result in a output like:

%%%% HASHDEEP-1.0
%%%% size,md5,filename
## Invoked from: /Users/evenh/dev/evenh/az-blob-hashdeep
## $ ./az-blob-hashdeep generate --account-name=myaccount --account-key=secretKey --container=migrationcontainer --output /Users/evenh/myaccount-migrationcontainer.hashdeep
##
1026764,ddb5d9fb991f62be9c55383aefa8e8e3,00/00/000008af-2e78-4b21-9a0e-a44ee77d4606
97428,4fdb49a5de56a1b11c9c37264a1bb927,00/00/00006c79-1c38-45f8-a3b8-ebb299fc67a1
[more entries…]

Use prefix

There is also a potential useful flag: --prefix (-p) which will prepend a prefix to file paths. Example:

Path: foo/bar/file.txt
Prefix: old-fs-01
Outputted path: old-fs-01/foo/bar/file.txt

Generate MD5 hashes locally

If you want to generate MD5 hashes from the content of a container, pass the --calculate flag. This operation is heavily CPU bound and will eat up your cores :-)

Troubleshooting

Set ABH_DEBUG=true to see more detailed logging.

Contributing

Contributions in form of issues and pull requests are most welcome.

License

This project is licensed under the Apache 2.0 License. See LICENSE for more information.