# README

carbites

Chunking for CAR files. Split a single CAR into multiple CARs.

Install

go get github.com/alanshaw/go-carbites

Usage

Carbites supports 2 different strategies:

Simple (default) - fast but naive, only the first CAR output has a root CID, subsequent CARs have a placeholder "empty" CID. The first CAR output has roots in the header, subsequent CARs have an empty root CID bafkqaaa as recommended.
Treewalk - walks the DAG to pack sub-graphs into each CAR file that is output. Every CAR file has the same root CID but contains a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.

package main

import (
	"io"
	"os"
	"github.com/alanshaw/go-carbites"
)

func main() {
	bigCar, _ := os.Open("big.car")
	targetSize := 1024 * 1024 // 1MiB chunks
	strategy := carbites.Simple // also carbites.Treewalk
	spltr, _ := carbites.Split(bigCar, targetSize, strategy)

	var i int
	for {
		car, err := spltr.Next()
		if err != nil {
			if err == io.EOF {
				break
			}
			panic(err)
		}
		b, _ := ioutil.ReadAll(car)
		ioutil.WriteFile(fmt.Sprintf("chunk-%d.car", i), b, 0644)
		i++
	}
}

API

pkg.go.dev Reference

Contribute

Feel free to dive in! Open an issue or submit PRs.

License

Dual-licensed under MIT + Apache 2.0

# Functions

Join

Join together multiple CAR files into a single CAR file.

JoinSimple

Join together multiple CAR files that were split using the "simple" strategy into a single CAR file.

JoinTreewalk

Join together multiple CAR files into a single CAR file using the "treewalk" strategy.

NewCarMerger

NewCarMerger creates a new CAR file (an io.Reader) that is a result of merging the passed CAR files.

NewSimpleSplitter

Create a new CAR file splitter to create multiple smaller CAR files using the "simple" strategy.

NewTreewalkSplitter

Split a CAR file and create multiple smaller CAR files using the "treewalk" strategy.

NewTreewalkSplitterFromBlockReader

Split a CAR file (passed as a root CID and a block reader populated with the blocks from the CAR) and create multiple smaller CAR files using the "treewalk" strategy.

NewTreewalkSplitterFromPath

Split a CAR file found on disk at the given path and create multiple smaller CAR files using the "treewalk" strategy.

Split

Split a CAR file and create multiple smaller CAR files.