# README
data-mover-core
The core of the Data Mover library. This has the interface definitions for Source and Destination connectors, as well as the core logic to instantiate and run a data mover.
Usage
source := SomeSourceType{} // instantiate a source that implements the Source interface
dest := SomeDestType{} // instantiate a destination that implements the Destination interface
// error handler interface, in this example this will be used for both source and destination errors
errorHandler := func(err error) bool {
fmt.Println(fmt.Sprintf("encountered error: %v", err))
return true
}
// instantiate a new data mover
mover, err := pkg.NewDataMover(
10, // source parallelism
10, // destination parallelism
source, // source connector instance
dest, // dest connector instance
errorHandler, // source error handler
errorHandler, // destination error handler
)
// call move, which will block until the move is complete, and return stats and an error
stats, err := mover.Move()
Source Interface
The source interface has two methods
Initialize
Initialize() error
This is where you can do any initialization of your connector implementation. Connect to databases, create http clients, whatever you need to do. An error should be returned, when an error is returned by Initialize
that error will be returned by NewDataMover
GetData
GetData() ([]map[string]interface{}, error)
This is the method that handles fetching data from the source. It should return an array of map[string]interface{}
and an error. This is called in a loop until it returns an empty array. When sourceParallelism
is greater than 1, this is called in parallel. Coordination of calls to GetData
is a source implementation concern, so if there is any coordination or locking that must be done to be thread safe, or to page through data, that must be done in your GetData
implementation. The mover makes no attempts to handle any of that. When an error is returned from GetData
the sourceErrorHandler
is called. If the sourceErrorHandler
returns false
then the Move()
will stop and exit.
Destination Interface
Initialize
Initialize() error
This is where you can do any initialization of your connector implementation. Connect to databases, create http clients, whatever you need to do. An error should be returned, when an error is returned by Initialize
that error will be returned by NewDataMover
Persist
Persist(data []map[string]interface{}) error
Each iteration of the source's GetData
will trigger a call to the destination's Persist
method if any data is returned from the source. Your implementation should insert the data into the destination, returning an error. When an error is returned, the destinationErrorHandler
is called. If the destinationErrorHandler
returns false
then the Move()
will stop and exit.