Categorygithub.com/hairyhenderson/go-fsimpl
modulepackage
0.1.7
Repository: https://github.com/hairyhenderson/go-fsimpl.git
Documentation: pkg.go.dev

# README

hairyhenderson/go-fsimpl

GoDoc Build

This module contains a collection of Go filesystem implementations that can be discovered dynamically by URL scheme.

These filesystems implement the fs.FS interface introduced in Go 1.16. This means that currently all implementations are read-only, however this may change in the future (see golang/go#45757 for progress).

Most implementations implement the fs.ReadDirFS interface, though the httpfs filesystem does not.

Some extensions are available to help add specific functionality to certain filesystems:

  • WithContextFS - injects a context into a filesystem, for propagating cancellation in filesystems that support it.
  • WithHeaderFS - sets the http.Header for all HTTP requests used by the filesystem. This can be useful for authentication, or for requesting specific content types.
  • WithHTTPClientFS - sets the *http.Client for all HTTP requests to be made with.

Many of the filesystem packages also have their own extensions.

This module also provides ContentType, an extension to the fs.FileInfo type to help identify an appropriate MIME content type for a given file. For filesystems that support it, the HTTP Content-Type header is used for this. Otherwise, the type is guessed from the file extension.

History & Project Status

This module is in active development, and the API is still subject to breaking changes.

The filesystem packages should operate correctly, based on the tests, but there may be edge cases that are not covered. Please open an issue if you find one!

Most of these filesystems are based on code from gomplate, which supports all of these as datasources. This module is intended to eventually be used within gomplate.

Supported Filesystems

Here's the list of filesystems & URL schemes supported by this module:

PackageScheme(s)Description
awsimdsfsaws+imdsAWS IMDS
awssmfsaws+smAWS Secrets Manager
awssmpfsaws+smpAWS Systems Manager Parameter Store
blobfsazblobAzure Blob Storage
blobfsgsGoogle Cloud Storage
blobfss3Amazon S3
consulfsconsul, consul+http, consul+httpsHashiCorp Consul
filefsfilelocal filesystem
gitfsgit, git+file, git+http, git+https, git+sshlocal/remote git repository
httpfshttp, httpsHTTP server
tracefsn/aa filesystem that instruments other filesystems for tracing with OpenTelemetry
vaultfsvault, vault+http, vault+httpsHashiCorp Vault

See the individual package documentation for more details.

Installation

Use go get to install the latest version of go-fsimpl:

$ go get -u github.com/hairyhenderson/go-fsimpl

Usage

If you know that you want an HTTP filesystem, for example:

import (
	"net/url"

	"github.com/hairyhenderson/go-fsimpl/httpfs"
)

func main() {
	base, _ := url.Parse("https://example.com")
	fsys, _ := httpfs.New(base)

	f, _ := fsys.Open("hello.txt")
	defer f.Close()

	// now do what you like with the file...
}

If you're not sure what filesystem type you need (for example, if you're dealing with a user-provided URL), you can use a filesystem mux:

import (
	"github.com/hairyhenderson/go-fsimpl"
	"github.com/hairyhenderson/go-fsimpl/blobfs"
	"github.com/hairyhenderson/go-fsimpl/filefs"
	"github.com/hairyhenderson/go-fsimpl/gitfs"
	"github.com/hairyhenderson/go-fsimpl/httpfs"
)

func main() {
	mux := fsimpl.NewMux()
	mux.Add(filefs.FS)
	mux.Add(httpfs.FS)
	mux.Add(blobfs.FS)
	mux.Add(gitfs.FS)

	// for example, a URL that points to a subdirectory at a specific tag in a
	// given git repo, hosted on GitHub and authenticated with SSH...
	fsys, err := mux.Lookup("git+ssh://[email protected]/foo/bar.git//baz#refs/tags/v1.0.0")
	if err != nil {
		log.Fatal(err)
	}

	f, _ := fsys.Open("hello.txt")
	defer f.Close()

	// now do what you like with the file...
}

Developing

You will require git including git daemon and consul executables on your path for running the tests.

License

The MIT License

Copyright (c) 2021-2024 Dave Henderson

# Packages

Package autofs provides the ability to look up all filesystems supported by this module.
Package awsimdsfs provides an interface to [AWS IMDS] (Instance Metadata Service), which is a service available to EC2 instances that provides information about the instance, and user data.
Package awssmfs provides an interface to AWS Secrets Manager which allows you to interact with the Secrets Manager API as a standard filesystem.
Package awssmpfs provides an interface to the AWS Systems Manager (SSM) Parameter Store which allows you to interact with the SSM Parameter Store API as a standard filesystem.
Package blobfs provides an interface to blob stores such as Google Cloud Storage, Azure Blob Storage, or AWS S3, allowing you to interact with the store as a standard filesystem.
Package consulfs provides an interface to Hashicorp Consul which allows you to interact with the Consul K/V store as a standard filesystem.
Package examples contains example programs that demonstrate usage of go-fsimpl.
Package filefs wraps os.DirFS to provide a local filesystem for file:// URLs.
Package gitfs provides a read-only filesystem backed by a git repository.
Package httpfs provides a read-only filesystem that reads from an HTTP server.
Package tracefs instruments a filesystem for distributed tracing operations.
Package vaultfs provides an interface to [Hashicorp Vault] which allows you to interact with the Vault API as a standard filesystem.

# Functions

ContentType returns the MIME content type for the given [io/fs.FileInfo].
FSProviderFunc -.
NewMux returns an FSMux ready for use.
WithContextFS injects a context into the filesystem fs, if the filesystem supports it (i.e.
WithHeaderFS injects the given HTTP header into the filesystem fs, if the filesystem supports it (i.e.
WithHTTPClientFS injects an HTTP client into the filesystem fs, if the filesystem supports it (i.e.
WrappedFSProvider is an FSProvider that returns the given fs.FS.

# Interfaces

FSProvider provides a filesystem for a set of defined schemes.

# Type aliases

FSMux allows you to dynamically look up a registered filesystem for a given URL.