Categorygithub.com/vbatts/go-mtree
modulepackage
0.5.4
Repository: https://github.com/vbatts/go-mtree.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
25
Dependents
27
Files
2.4k SLOC

# README

go-mtree

Go Go Report Card

mtree is a filesystem hierarchy validation tooling and format. This is a library and simple cli tool for mtree(8) support.

While the traditional mtree cli utility is primarily on BSDs (FreeBSD, openBSD, etc), even broader support for the mtree specification format is provided with libarchive (libarchive-formats(5)).

There is also an mtree port for Linux though it is not widely packaged for Linux distributions.

There was a Google Summer of Code project to create a portable library and parser for mtree. It is available at github.com/mratajsky/libmtree and a talk on it.

Format

The BSD mtree specification is published in mtree(5).

The format of hierarchy specification is consistent with the # mtree v2.0 format. Both the BSD mtree and libarchive ought to be interoperable with it with only one definite caveat. On Linux, extended attributes (xattr) on files are often a critical aspect of the file, holding ACLs, capabilities, etc. While FreeBSD filesystem do support extattr, this feature has not made its way into their mtree.

This implementation of mtree supports a few non-upstream "keyword"s, such as: xattr and tar_time. If you include these keywords, the FreeBSD mtree will fail, as they are unknown keywords to that implementation.

To have go-mtree produce specifications that will be strictly compatible with the BSD mtree, use the -bsd-keywords flag when creating a manifest. This will make sure that only the keywords supported by BSD mtree are used in the program.

Typical form

With the standard keywords, plus say sha256digest, the hierarchy specification looks like:

# .
/set type=file nlink=1 mode=0664 uid=1000 gid=100
. size=4096 type=dir mode=0755 nlink=6 time=1459370393.273231538
    LICENSE size=1502 mode=0644 time=1458851690.0 sha256digest=ef4e53d83096be56dc38dbf9bc8ba9e3068bec1ec37c179033d1e8f99a1c2a95
    README.md size=2820 mode=0644 time=1459370256.316148361 sha256digest=d9b955134d99f84b17c0a711ce507515cc93cd7080a9dcd50400e3d993d876ac

[...]

See the directory presently in, and the files present. Along with each path, is provided the keywords and the unique values for each path. Any common keyword and values are established in the /set command.

Extended attributes form

# .
/set type=file nlink=1 mode=0664 uid=1000 gid=1000
. size=4096 type=dir mode=0775 nlink=6 time=1459370191.11179595 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==
    LICENSE size=1502 time=1458851690.583562292 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==
    README.md size=2366 mode=0644 time=1459369604.0 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==

[...]

See the keyword prefixed with xattr. followed by the extended attribute's namespace and keyword. This setup is consistent for use with Linux extended attributes as well as FreeBSD extended attributes.

Since extended attributes are an unordered hashmap, this approach allows for checking each <namespace>.<key> individually.

The value is the base64 encoded of the value of the particular extended attribute. Since the values themselves could be raw bytes, this approach avoids issues with encoding.

Tar form

# .
/set type=file mode=0664 uid=1000 gid=1000
. type=dir mode=0775 tar_time=1468430408.000000000

# samedir
samedir type=dir mode=0775 tar_time=1468000972.000000000
    file2 size=0 tar_time=1467999782.000000000
    file1 size=0 tar_time=1467999781.000000000
    
[...]

While go-mtree serves mainly as a library for upstream mtree support, go-mtree is also compatible with tar archives (which is not an upstream feature). This means that we can now create and validate a manifest by specifying a tar file. More interestingly, this also means that we can create a manifest from an archive, and then validate this manifest against a filesystem hierarchy that's on disk, and vice versa.

Notice that for the output of creating a validation manifest from a tar file, the default behavior for evaluating a notion of time is to use the tar_time keyword. In the "filesystem hierarchy" format of mtree, time is being evaluated with nanosecond precision. However, GNU tar truncates a file's modification time to 1-second precision. That is, if a file's full modification time is 123456789.123456789, the "tar time" equivalent would be 123456789.000000000. This way, if you validate a manifest created using a tar file against an actual root directory, there will be no complaints from go-mtree so long as the 1-second precision time of a file in the root directory is the same.

Usage

To use the Go programming language library, see the docs.

To use the command line tool, first build it, then the following.

Create a manifest

This will also include the sha512 digest of the files.

gomtree validate -c -K sha512digest -p . > /tmp/root.mtree

With a tar file:

gomtree validate -c -K sha512digest -T sometarfile.tar > /tmp/tar.mtree

Validate a manifest

gomtree validate -p . -f /tmp/root.mtree

With a tar file:

gomtree validate -T sometarfile.tar -f /tmp/root.mtree

See the supported keywords

gomtree validate -list-keywords
Available keywords:
 uname
 sha1
 sha1digest
 sha256digest
 xattrs (not upstream)
 link (default)
 nlink (default)
 md5digest
 rmd160digest
 mode (default)
 cksum
 md5
 rmd160
 type (default)
 time (default)
 uid (default)
 gid (default)
 sha256
 sha384
 sha512
 xattr (not upstream)
 tar_time (not upstream)
 size (default)
 ripemd160digest
 sha384digest
 sha512digest

Building

Either:

go install github.com/vbatts/go-mtree/cmd/gomtree@latest

or

git clone git://github.com/vbatts/go-mtree.git $GOPATH/src/github.com/vbatts/go-mtree
cd $GOPATH/src/github.com/vbatts/go-mtree
go build ./cmd/gomtree

Build for many OS/Arch

make build.arches

Testing

On Linux:

cd $GOPATH/src/github.com/vbatts/go-mtree
make

On FreeBSD:

cd $GOPATH/src/github.com/vbatts/go-mtree
gmake

Related tools

# Packages

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author

# Functions

Check a root directory path against the DirectoryHierarchy, regarding only the available keywords from the list and each entry in the hierarchy.
CleanPath makes a path safe for use with filepath.Join.
Compare compares two directory hierarchy manifests, and returns the list of discrepancies between the two.
CompareSame is the same as Compare, except it also includes the entries that are the same with a Same DifferenceType.
FromKeywords makes a list of string from a list of Keyword.
Has the "keyword" present in the list of KeyVal, and returns the corresponding KeyVal, else an empty string.
HasKeyword the "keyword" present in the list of KeyVal, and returns the corresponding KeyVal, else an empty string.
InKeywordSlice checks for the presence of `a` in `list`.
KeyValToString constructs a list of string from the list of KeyVal.
KeywordSynonym returns the canonical name for keywords that have synonyms, and just returns the name provided if there is no synonym.
MergeKeyValSet does a merge of the two sets of KeyVal, and the KeyVal of entryKeyVals win when there is a duplicate Keyword.
MergeSet takes the current setKeyVals, and then applies the entryKeyVals such that the entry's values win.
NewTarStreamer streams a tar archive and creates a file hierarchy based off of the tar metadata headers.
ParseSpec reads a stream of an mtree specification, and returns the DirectoryHierarchy.
StringToKeyVals constructs a list of KeyVal from the list of strings, like "keyword=value".
ToKeywords makes a list of Keyword from a list of string.
Update attempts to set the attributes of root directory path, given the values of `keywords` in dh DirectoryHierarchy.
Walk from root directory and assemble the DirectoryHierarchy * `excludes` provided are used to skip paths * `keywords` are the set to collect from the walked paths.

# Constants

blank lines are ignored.
Lines beginning with `#` are ignored.
DefaultVisFlags is the set of Vis flags used when encoding filenames and other similar entries.
.
ErrorDifference represents an attempted update to the values of a keyword that failed.
Extra represents a discrepancy where the object is not present in the @old manifest but is present in the @new manifest.
if the first word on the line has a `/` after the first character, it interpretted as a file pathname with options.
Missing represents a discrepancy where the object is present in the @old manifest but is not present in the @new manifest.
Modified represents a discrepancy where the object is present in both the @old and @new manifests, but one or more of the keys have different values (or have not been set in one of the manifests).
if the first white-space delimited word does not have a '/' in it.
Same represents the case where two files are the same.
first line of the file, like `#mtree v2.0`.
line that has `/` prefix issue a "special" command (currently only /set and /unset).

# Variables

BsdKeywords is the set of keywords that is only in the upstream FreeBSD mtree.
DefaultKeywords has the several default keyword producers (uid, gid, mode, nlink, type, size, mtime).
DefaultTarKeywords has keywords that should be used when creating a manifest from an archive.
DefaultUpdateKeywords is the default set of keywords that can take updates to the files on disk.
ExcludeNonDirectories is an ExcludeFunc for excluding all paths that are not directories.
KeywordFuncs is the map of all keywords (and the functions to produce them).
SetKeywords is the default set of keywords calculated for a `/set` SpecialType.
UpdateKeywordFuncs is the registered list of functions to update file attributes.

# Structs

DefaultFsEval is the default implementation of FsEval (and is the default used if a nil interface is passed to any mtree function).
DirectoryHierarchy is the mapped structure for an mtree directory hierarchy spec.
Entry
Entry is each component of content in the mtree spec file.
InodeDelta Represents a discrepancy in a filesystem object between two DirectoryHierarchy manifests.
KeyDelta Represents a discrepancy in a key for a particular filesystem object between two DirectoryHierarchy manifests.

# Interfaces

FsEval is a mock-friendly method of specifying to go-mtree how to carry out filesystem operations such as opening files and the like.
Streamer creates a file hierarchy out of a tar stream.

# Type aliases

DifferenceType represents the type of a discrepancy encountered for an object.
EntryType are the formats of lines in an mtree spec file.
ExcludeFunc is the type of function called on each path walked to determine whether to be excluded from the assembled DirectoryHierarchy.
KeyVal is a "keyword=value".
Keyword is the string name of a keyword, with some convenience functions for determining whether it is a default or bsd standard keyword.
KeywordFunc is the type of a function called on each file to be included in a DirectoryHierarchy, that will produce the string output of the keyword to be included for the file entry.
UpdateKeywordFunc is the signature for a function that will restore a file's attributes.