Splitting a large file into smaller chunks.
Compress each chunk using snappy
Encrypt each chunk using Hash(content)
- Hash functions supported: blake2
- Support for symmetric encryption
Save each part in different 0-stor server
Replicate same chunk into different 0-stor servers
Save metadata about chunks and where they're in etcd server or other metadata db
Assemble smaller chunks for a file upon retrieval
- Get chunks location from metadata server
- Assemble file from chunks

Important

0stor server is JUST a simple key/value store
splitting, compression, encryption, and replication is the responsibility of client

Features

Erasure coding

Metadata

Metadata format

Format for the metadata:

type Chunk struct {
	Size   uint64    # Size of the chunk in bytes
	Key    []byte    # key used in the 0-stor
	Shards []string
}
type Meta struct {
	Epoch     int64  # creation epoch
	Key       []byte # key used in the 0-stor
	EncrKey   []byte # Encryption key used to encrypt this file
	Chunks    []*Chunk # list of chunks of the files
	Previous  []byte   # Key to the previous metadata entry
	Next      []byte   # Key to the next metadata entry
	ConfigPtr []byte   # Key to the configuration used by the lib to set the data.
}

chunks

Depending on the policy used to create the client, the data can be splitted into multiple chunks or not. Which mean the metadata can be composed of minimum one up to n chunks.

Each chunks can then have one or multiple shards.

If you use replication, each shards is the location of one of the replicate.
If you use distribution, each shards is the location of one of the data or parity block.

metadata linked list

With Previous and Next fields in the metadata, we can build metadata linked list of sequential data such as transaction log/history and block chain. The linked list can then be used to walk over the data in forward or backward fashion.

Metadata linked list will be build if user specify previous meta key when calling WriteWithMeta or WriteFWithMeta methods.

Getting started

Getting started

Now into some technical details!

Pre processing of data

Client does some preprocessing on each chunk of data before sending them to 0stor
This is achieved by configuring a policy during client creation
Supported Data Preprocessing:

walk over the metadata

Thanks to our metadata linked list, we can walk over the linked list in forward and backward mode using Walk or WalkBack API.

Those APIs returns channel of WalkResult which then can be iterated. The WalkResult consist of these fields:

key in metadata
the metadata
the data stored on 0-stor server
error if exist

It can be used for example to reconstruct the stored sequential data.

Using 0-stor client examples:

In below example, when we store the data, the data will be processed as follow: plain data -> compress -> encrypt -> distribution/erasure encoding (which send to 0-stor server and write metadata)

When we get the data from 0-stor, the reverse process will happen: distribution/erasure decoding (which read metdata & Get data from 0-stor) -> decrypt -> decompress -> plain data.

To run this example, you need to run:

0-stor server at port 12345
0-stor server at port 12346
0-stor server at port 12347
etcd server at port 2379

package main

import (
	"log"

	"github.com/zero-os/0-stor/client"
)

func main() {

	policy := client.Policy{
		Organization:           "labhijau",
		Namespace:              "thedisk",
		Protocol:               "rest",
		DataShards:             []string{"http://127.0.0.1:12345", "http://127.0.0.1:12346", "http://127.0.0.1:12347"},
		MetaShards:             []string{"http://127.0.0.1:2379"},
		IYOAppID:               "the_id",
		IYOSecret:              "the_secret",
		Compress:               true,
		Encrypt:                true,
		EncryptKey:             "ab345678901234567890123456789012",
		BlockSize:              4096,
		ReplicationNr:          0, // force to use distribution
		ReplicationMaxSize:     0, // force to use distribution
		DistributionNr:         2,
		DistributionRedundancy: 1,
	}
	c, err := client.New(policy)
	if err != nil {
		log.Fatal(err)
	}

	data := []byte("hello 0-stor")
	key := []byte("hi guys")

	// stor to 0-stor
	_, err = c.Write(key, data)
	if err != nil {
		log.Fatal(err)
	}

	// read the data
	stored, err := c.Read(key)
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("stored data=%v\n", string(stored))
}

Example using configuration file

package main

import (
	"log"
	"os"

	"github.com/zero-os/0-stor/client"
)

func main() {
	f, err := os.Open("./config.yaml")
	if err != nil {
		log.Fatal(err)
	}


	policy, err := client.NewPolicyFromReader(f)
	if err != nil {
		log.Fatal(err)
	}

	client, err := client.New(policy)
	if err != nil {
		log.Fatal(err)
	}

	data := []byte("hello 0-stor")
	key := []byte("hello")

	// stor to 0-stor
	_, err = client.Write(key, data)
	if err != nil {
		log.Fatal(err)
	}

	// read data
	stored, err := client.Read(key)
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("stored value=%v\n", string(stored))

}