Categorygithub.com/hack-pad/hackpadfs
modulepackage
0.2.4
Repository: https://github.com/hack-pad/hackpadfs.git
Documentation: pkg.go.dev

# README

hackpadfs Go Reference CI Coverage Status

File systems, interfaces, and helpers for Go.

Want to get started? Check out the guides below.

File systems

hackpadfs includes several implemented file systems, ready for use in a wide variety of applications:

  • os.FS - The familiar os package. Implements all of the familiar behavior from the standard library using new interface design.
  • mem.FS - In-memory file system.
  • indexeddb.FS - WebAssembly compatible file system, uses IndexedDB under the hood.
  • tar.ReaderFS - A streaming tar FS for memory and time-constrained programs.
  • mount.FS - Composable file system. Capable of mounting file systems on top of each other.
  • keyvalue.FS - Generic key-value file system. Excellent for quickly writing your own file system. mem.FS and indexeddb.FS are built upon it.

Looking for custom file system inspiration? Examples include:

Each of these file systems runs through the rigorous hackpadfs/fstest suite to ensure both correctness and compliance with the standard library's os package behavior. If you're implementing your own FS, we recommend using fstest in your own tests as well.

Interfaces

Based upon the groundwork laid in Go 1.16's io/fs package, hackpadfs defines many essential file system interfaces.

Here's a few of the interfaces defined by hackpadfs:

type FS interface {
    Open(name string) (File, error)
}

type CreateFS interface {
    FS
    Create(name string) (File, error)
}

type MkdirFS interface {
    FS
    Mkdir(name string, perm FileMode) error
}

type StatFS interface {
    FS
    Stat(name string) (FileInfo, error)
}

See the reference docs for full documentation.

Using these interfaces, you can create and compose your own file systems. The interfaces are small, enabling custom file systems to implement only the required pieces.

Getting started

There's many ways to use hackpadfs. Jump to one of these guides:

Quick start

If you've used the standard library's os package, you probably understand most of how os.FS works!

In this example, we create a new os.FS and print the contents of /tmp/hello.txt.

import (
    "fmt"

    "github.com/hack-pad/hackpadfs"
    "github.com/hack-pad/hackpadfs/os"
)

filePath := "tmp/hello.txt"
fs, _ := os.NewFS()
file, _ := fs.Open(filePath)
defer file.Close()

buffer := make([]byte, 1024)
n, _ := file.Read(buffer)
fmt.Println("Contents of hello.txt:", string(buffer[:n]))

Relative file paths

Relative paths are not allowed in Go's io/fs specification, so we must use absolute paths (without the first /). To simulate relative paths, use the SubFS interface to create a new "rooted" FS like this:

import (
    goOS "os"
    "github.com/hack-pad/hackpadfs/os"
)

fs := os.NewFS()
workingDirectory, _ := goOS.Getwd()                    // Get current working directory
workingDirectory, _ = fs.FromOSPath(workingDirectory)  // Convert to an FS path
workingDirFS, _ := fs.Sub(workingDirectory)            // Run all file system operations rooted at the current working directory

Path separators (slashes)

Following the io/fs specification:

Path names passed to open are UTF-8-encoded, unrooted, slash-separated sequences of path elements, like “x/y/z”. Path names must not contain an element that is “.” or “..” or the empty string, except for the special case that the root directory is named “.”. Paths must not start or end with a slash: “/x” and “x/” are invalid.

Note that paths are slash-separated on all systems, even Windows. Paths containing other characters such as backslash and colon are accepted as valid, but those characters must never be interpreted by an FS implementation as path element separators.

In hackpadfs, this means:

  • All path separators are "forward slashes" or /. Even on Windows, slashes are converted under the hood.
  • A path starting or ending with / is invalid
  • To reference the root file path, use .
  • All paths are unrooted (not relative paths)
  • Paths are not necessarily cleaned when containing relative-path elements (e.g. mypath/.././myotherpath). Some FS implementations resolve these, but it is not guaranteed. File systems should reject these paths via io/fs.ValidPath().

Working with interfaces

It's a good idea to use interfaces -- file systems should be no different. Swappable file systems enable powerful combinations.

However, directly handling interface values can be difficult to deal with. Luckily, we have several helpers available.

In the below example, we use hackpadfs's package-level functions to do interface checks and call the appropriate methods for us:

import (
    "github.com/hack-pad/hackpadfs"
    "github.com/hack-pad/hackpadfs/mem"
)

func main() {
    fs, _ := mem.NewFS()
    helloWorld(fs)
}

// helloWorld uses a generic hackpadfs.FS to create a file containing "world".
// Returns an error if 'fs' does not support creating or writing to files.
func helloWorld(fs hackpadfs.FS) error {
    // hackpadfs.Create(...) checks to see if 'fs' implements Create, with a few fallback interfaces as well.
    file, err := hackpadfs.Create(fs, "hello.txt")
    if err != nil {
	return err
    }
    // Same here for hackpadfs.WriteFile(...). If the file doesn't support writing, a "not implemented" error is returned.
    _, err = hackpadfs.WriteFile(file, []byte("world"))
    return err
}

Notice the package-level function calls to hackpadfs.Create(...) and hackpadfs.WriteFile(...). Since the interface we're using doesn't know about those methods, we use these helpers to detect support and run those operations in one call.

Now whenever we need to reuse helloWorld() with a completely different file system, it's ready to go!

# Packages

Package cache contains a read-only cache file system.
Package fstest runs test suites against a target FS.
Package keyvalue contains a key-value based FS for easy, custom FS implementations.
Package mem contains an in-memory FS.
Package mount contains an implementation of hackpadfs.MountFS.
Package os implements all of the familiar behavior from the standard library using hackpadfs's interfaces.
Package tar contains a tar file based file system.

# Functions

Chmod attempts to call an optimized fs.Chmod(), falls back to opening the file and running file.Chmod().
ChmodFile runs file.Chmod() is available, fails with a not implemented error otherwise.
Chown attempts to call an optimized fs.Chown(), falls back to opening the file and running file.Chown().
ChownFile runs file.Chown() is available, fails with a not implemented error otherwise.
Chtimes attempts to call an optimized fs.Chtimes(), falls back to opening the file and running file.Chtimes().
ChtimesFile runs file.Chtimes() is available, fails with a not implemented error otherwise.
Create attempts to call an optimized fs.Create() if available, falls back to OpenFile() with create flags.
Lstat stats files and does not follow symlinks.
LstatOrStat attempts to call an optimized fs.LstatOrStat(), fs.Lstat(), or fs.Stat() - whichever is supported first.
Mkdir creates a directory.
MkdirAll attempts to call an optimized fs.MkdirAll(), falls back to multiple fs.Mkdir() calls.
OpenFile attempts to call fs.Open() or fs.OpenFile() if available.
ReadAtFile runs file.ReadAt() is available, fails with a not implemented error otherwise.
ReadDir attempts to call an optimized fs.ReadDir(), falls back to io/fs.ReadDir().
ReadDirFile runs file.ReadDir() is available, fails with a not implemented error otherwise.
ReadFile attempts to call an optimized fs.ReadFile(), falls back to io/fs.ReadFile().
Remove removes a file with fs.Remove().
RemoveAll attempts to call an optimized fs.RemoveAll(), falls back to removing files and directories recursively.
Rename moves files with fs.Rename().
SeekFile runs file.Seek() is available, fails with a not implemented error otherwise.
Stat attempts to call an optimized fs.Stat(), falls back to fs.Open() and file.Stat().
Sub attempts to call an optimized fs.Sub() if available.
Symlink creates a symlink.
SyncFile runs file.Sync() is available, fails with a not implemented error otherwise.
TruncateFile runs file.Truncate() is available, fails with a not implemented error otherwise.
ValidPath returns true if 'path' is a valid FS path.
WalkDir recursively scans 'fs' starting at path 'root', calling 'fn' every time a new file or directory is visited.
WriteAtFile runs file.WriteAt() is available, fails with a not implemented error otherwise.
WriteFile runs file.Write() is available, fails with a not implemented error otherwise.
WriteFullFile attempts to call an optimized fs.WriteFile(), falls back to fs.OpenFile() with file.Write().

# Constants

Flags are bit-wise OR'd with each other in fs.OpenFile().
Flags are bit-wise OR'd with each other in fs.OpenFile().
Flags are bit-wise OR'd with each other in fs.OpenFile().
Flags are bit-wise OR'd with each other in fs.OpenFile().
Flags are bit-wise OR'd with each other in fs.OpenFile().
Flags are bit-wise OR'd with each other in fs.OpenFile().
Flags are bit-wise OR'd with each other in fs.OpenFile().
Flags are bit-wise OR'd with each other in fs.OpenFile().
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.
Mode values are bit-wise OR'd with a file's permissions to form the FileMode.

# Variables

Errors commonly returned by file systems.
Errors commonly returned by file systems.
TODO update to fs.ErrInvalid, once errors.Is supports it.
Errors commonly returned by file systems.
Errors commonly returned by file systems.
Errors commonly returned by file systems.
Errors commonly returned by file systems.
Errors commonly returned by file systems.
Errors commonly returned by file systems.
Errors commonly returned by file systems.

# Structs

LinkError records a file system rename error and the paths that caused it.

# Interfaces

ChmoderFile is a File that supports Chmod() operations.
ChmodFS is an FS that can change file or directory permissions.
ChownerFile is a File that supports Chown() operations.
ChownFS is an FS that can change file or directory ownership.
ChtimeserFile is a File that supports Chtimes() operations.
ChtimesFS is an FS that can change a file's access and modified timestamps.
CreateFS is an FS that can create files.
DirReaderFile is a File that supports ReadDir() operations.
LstatFS is an FS that can lstat files.
MkdirAllFS is an FS that can make all missing directories in a given path.
MkdirFS is an FS that can make directories.
MountFS is an FS that meshes one or more FS's together.
OpenFileFS is an FS that can open files with the given flags and can create with the given permission.
ReadDirFS is an FS that can read a directory and return its DirEntry's.
ReaderAtFile is a File that supports ReadAt() operations.
ReadFileFS is an FS that can read an entire file in one pass.
ReadWriterFile is a File that supports Write() operations.
RemoveAllFS is an FS that can remove files or directories recursively.
RemoveFS is an FS that can remove files or empty directories.
RenameFS is an FS that can move files or directories.
SeekerFile is a File that supports Seek() operations.
StatFS is an FS that can stat files or directories.
SubFS is an FS that can return a subset of the current FS.
SymlinkFS is an FS that can create symlinks.
SyncerFile is a File that supports Sync() operations.
TruncaterFile is a File that supports Truncate() operations.
WriteFileFS is an FS that can write an entire file in one pass.
WriterAtFile is a File that supports WriteAt() operations.

# Type aliases

DirEntry is an entry read from a directory.
File provides access to a file.
FileInfo describes a file and is returned by Stat().
FileMode represents a file's mode and permission bits.
FS provides access to a file system and its files.
PathError records a file system or file operation error and the path that caused it.
WalkDirFunc is the type of function called in WalkDir().