modulepackage
0.6.7
Repository: https://github.com/wangjuelong/go-streams.git
Documentation: pkg.go.dev
# README
go-streams
A lightweight stream processing library for Go.
go-streams provides a simple and concise DSL to build data pipelines.
Wiki
In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.
Overview
Building blocks:
- Source - A Source is a set of stream processing steps that has one open output.
- Flow - A Flow is a set of stream processing steps that has one open input and one open output.
- Sink - A Sink is a set of stream processing steps that has one open input. Can be used as a Subscriber.
Flow capabilities (flow package):
- Map
- FlatMap
- Filter
- PassThrough
- Split
- FanOut
- Merge
- Throttler
- SlidingWindow
- TumblingWindow
Supported Connectors:
- Go channels
- File system
- Network (TCP, UDP)
- WebSocket
- Aerospike
- Apache Kafka
- Apache Pulsar
- Redis
Examples
Usage samples are available in the examples directory.
License
Licensed under the MIT License.
# Packages
Package aerospike implements the Aerospike database connector.
Package extension provides basic connector implementations.
Package flow provides streams.Flow implementations.
Package kafka implements the Apache Kafka connector.
Package nats implements the NATS Streaming connector.
Package pulsar implements the Apache Pulsar connector.
Package redis implements the Redis Pub/Sub connector.
Package util provides utility functions.
Package ws implements the WebSocket connector.
# Interfaces
A Flow is a set of stream processing steps that has one open input and one open output.
Inlet is a type that exposes one open input.
Outlet is a type that exposes one open output.
A Sink is a set of stream processing steps that has one open input.
A Source is a set of stream processing steps that has one open output.