fast-archiver

fast-archiver is a command-line tool for archiving directories, and restoring
those archives written in [Go](http://golang.org).

fast-archiver uses a few techniques to try to be more efficient than
traditional tools:

1. It reads a number of files concurrently and then serializes the output.
   Most other tools use sequential file processing, where operations like
   ``open()``, ``lstat()``, and ``close()`` can cause a lot of overhead when
   reading huge numbers of small files.  Making these operations concurrent
   means that the tool is more often reading and writing data than you would
   be otherwise.

2. It begins archiving files before it has completed reading the directory
   entries that it is archiving, allowing for a fast startup time
   compared to tools that first create an inventory of files to
   transfer.

How Fast?
---------

On a test workload of 2,089,214 files representing a total of 90 GiB of data,
fast-archiver was compared with tar and rsync for reading data files and
transfering them over a network.  The test scenario was a PostgreSQL
database, with many of the files being small, 8-24kiB in size.

Compared with tar, fast-archiver took 33% of the execution time (27m 38s vs.
1h 23m 23s) to read the test workload and output the archive to /dev/null.
The tar output had to be redirected through cat to create a comparable
scenario, because tar recognized /dev/null and shortcuts the actual data file
reading and writing.  Here's the raw timing output for some hard data::

    $ time fast-archiver -c -o /dev/null /db/data
    skipping symbolic link /db/data/pg_xlog
    1008.92user 663.00system 27:38.27elapsed 100%CPU (0avgtext+0avgdata 24352maxresident)k
    0inputs+0outputs (0major+1732minor)pagefaults 0swaps
    
    $ time tar -cf - /db/data | cat > /dev/null
    tar: Removing leading `/' from member names
    tar: /db/data/base/16408/12445.2: file changed as we read it
    tar: /db/data/base/16408/12464: file changed as we read it
    32.68user 375.19system 1:23:23elapsed 8%CPU (0avgtext+0avgdata 81744maxresident)k
    0inputs+0outputs (0major+5163minor)pagefaults 0swaps

Compared with rsync, fast-archiver piped over ssh can transfer the database
from one machine to another in 1h 30m, vs. rsync in 3h.

These huge reductions in time may not be typical, but they happen to be the
workload that fast-archiver was designed for.

Examples
--------

Creates an archive (-c) reading the directory target1, and redirects the
archive to the file named target1.fast-archive::

    fast-archiver -c target1 > target1.fast-archive
    fast-archiver -c -o target1.fast-archive target1

Extracts the archive target1.fast-archive into the current directory::

    fast-archiver -x < target1.fast-archive
    fast-archiver -x -i target1.fast-archive

Creates a fast-archive remotely, and restores it locally, piping the data
through ssh::

    ssh [email protected] "cd /db; fast-archive -c data --exclude=data/\*.pid" | fast-archiver -x


Installation
------------

The fast-archiver repository contains both a command-line tool (at the root)
and a package called ``falib`` which contains the archive reading and writing
code.  To make the build work correctly with both the library and the
command-line tool, it's necessary to setup the correct GOPATH and directory
references.

Here's a quick set of steps to setup the build:

 * Install [Go](http://golang.org).

 * Setup ``$GOPATH``, for example: ``export GOPATH=$HOME/go-projects``. Probably
   better to set it up in your ``.bash_aliases``.

 * ``go get -u github.com/replicon/fast-archiver``

_or_

 * ``go get -d github.com/replicon/fast-archiver && $GOPATH/src/github.com/replicon/fast-archiver/build.sh``


Command-line arguments
----------------------


-x
    Extract archive mode.

-c
    Create archive mode.

--multicpu
    Allows concurrent activities to run on the specified number of CPUs.  Since
    the archiving is dominated by I/O, additional CPUs tend to just add
    overhead in communicating between concurrent processes, but it could
    increase throughput in some scenarios.  Defaults to 1.


Create-mode only
================

-o
    Output path for the archive.  Defaults to stdout.

--exclude
    A colon-separated list of paths to exclude from the archive.  Can include
    wildcards and other shell matching constructs.

--block-size
    Specifies the size of blocks being read from disk, in bytes.  The larger
    the block size, the more memory fast-archiver will use, but it could result
    in higher I/O rates.  Defaults to 4096, maximum value is 65535.

--dir-readers
    The maximum number of directories that will be read concurrently.  Defaults
    to 16.

--file-readers
    The maximum number of files that will be read concurrently.  Defaults to
    16.

--queue-dir
    The maximum size of the queue for sub-directory paths to be processed.
    Defaults to 128.

--queue-read
    The maximum size of the queue for file paths to be processed.  Defaults to
    128.

--queue-write
    The maximum size of the block queue for archive output.  Increasing this
    will increase the potential memory usage, as (queue-write * block-size)
    memory could be allocated for file reads.  Defaults to 128.


Extract-mode only
=================

-i
    Input path for the archive.  Defaults to stdin.

--ignore-perms
    Do not restore permissions on files and directories.

--ignore-owners
    Do not restore uid and gid on files and directories.