Categorygithub.com/ryandeivert/memr
modulepackage
0.1.3
Repository: https://github.com/ryandeivert/memr.git
Documentation: pkg.go.dev

# README

memr: Linux memory reader

license Go Reference CI vagrant tests GitHub release (latest SemVer)

Overview

memr is inspired by AVML, with the added goal of extensibility.

  • Provides a golang io.ReadCloser interface, as memr.Reader, over the memory source
  • Callers can do whatever they please with the data that is read through the memr.Reader
    • write to a local file
    • stream to an S3 bucket
    • compress with a desired algorithm before writing, and so on

The major benefit of memr is that it supports streaming memory data, enabling writing off-host without first copying to the local disk.

Often times, particularly in cloud environments, hosts may have memory sizes that far exceed space available on disk. In these cases, it is infeasible to first write data locally.

Features

memr also adds some additional features:

  • Custom page headers can be provided using memr.PageHeaderProviderFunc
    • By default page headers will be written in the LiME format
  • Custom handling of page data using memr.PageWriterFunc
    • This is meant to replicate AVML's custom format, or (version 2 by AVML's specification), where page-level compression is performed with snappy. However, in my opinion, this should be avoided and compression should be done at the stream level, not the page level (see the compression example for more on this approach).
  • Debug logging using memr.SetLogLevel(memr.LogDebug) (or -vvv using the provided CLI)
  • Progress reporting

Examples

See here for various examples leveraging the features outlined above.

Install

With a proper Go installation, the package is installable using:

go get github.com/ryandeivert/memr

If running Linux, a sample CLI tool is included in this repo that can be installed using the below (requires go 1.16+). Note that this method should only be used on Linux systems, as it will compile a binary for your local system's GOOS and GOARCH:

go install github.com/ryandeivert/memr/cmd/memr@latest

Or download binaries directly from GitHub Releases.

CLI Usage

The memr CLI tool included in this repo supports a of couple use cases.

It supports writing to either a local file (with the --local-file flag) or an S3 bucket (with the --bucket/--key flag combination). Either method supports compression (the default), but can be disabled using --compression=false. Other basic sample CLIs are included in the examples directory.

Usage:
  memr [flags]

Examples:

Writing to local file:
memr --local-file <FILE>

Streaming directly to S3 bucket:
memr --bucket <BUCKET> --key <KEY>

Targeting a specific device:
memr /dev/mem --local-file <FILE>

Skipping compression:
memr --compress=false  --local-file <FILE>

Flags:
  -b, --bucket string       S3 bucket to which output should be sent
  -c, --compress            compress the output with snappy (default true)
  -t, --concurrency int     number of threads to use for S3 upload (default 5)
  -h, --help                help for memr
  -k, --key string          key to use for uploading to S3 bucket
  -f, --local-file string   local file to write to, instead of S3
  -p, --progress            show progress (default true)
  -r, --region string       AWS region to use with S3 client (default "us-east-1")
  -v, --verbose count       enable verbose logging
      --version             version for memr

Quick Start API Example

package main

import (
	"io"
	"log"
	"os"

	"github.com/ryandeivert/memr"
)

func main() {
	// Use memr.Probe() to enumerate all available types, attempting to find a valid reader
	// Alternatively use memr.NewReader(memr.SourceKcore) to target a specific type,
	// in this case /proc/kcore
	reader, err := memr.Probe()
	if err != nil {
		log.Fatalf("failed to load memory reader: %s", err)
	}
	defer reader.Close()

	// Open the local file for writing
	localFile := "output.mem"
	writer, err := os.Create(localFile)
	if err != nil {
		log.Fatalf("failed to open local file %q for writing %s", localFile, err)
	}
	defer writer.Close()

	// Use io.Copy to copy the memory to a file
	_, err = io.Copy(writer, reader)
	if err != nil {
		log.Fatalf("failed to copy memory to local file %q: %s", localFile, err)
	}
}

# Packages

No description provided by the author
No description provided by the author

# Functions

HeaderLime is a PageHeaderProviderFunc that returns a DefaultHeader struct.
NewReader tries to open the specified memory source for reading.
Probe enumerates all available memory sources and returns the first valid reader.
SetupLogging sets logging level on the logger using the specified LogLvl value.

# Constants

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
SourceCrash disignates /dev/crash as the memory source.
SourceKcore disignates /proc/kcore as the memory source.
SourceMem disignates /dev/mem as the memory source.

# Structs

DefaultHeader is the default header struct value.
Reader implements the io.ReadCloser interface Its values can also used as "options" for the resulting Reader returned by the Probe() and NewReader() functions.

# Type aliases

No description provided by the author
MemSource type used for memory sources.
PageHeaderProviderFunc should be used to provide a page header.
PageWriterFunc can be used to add special handling of page contents, such as page-level compression.