NewBigQueryReader returns an instance of a BigQueryExtractor ready to run a static query.

NewCSVTransformer returns a new CSVTransformer wrapping the given io.Writer object.

NewCSVWriter returns a new CSVWriter wrapping the given io.Writer object.

NewDynamicBigQueryReader returns an instance of a BigQueryExtractor ready to run a dynamic query based on the sqlGenerator function.

NewDynamicSQLExecutor returns a new SQLExecutor operating in dynamic mode.

NewDynamicSQLReader returns a new SQLReader operating in dynamic mode.

NewDynamicSQLReaderWriter returns a new SQLReaderWriter ready for dynamic querying.

NewFileReader returns a new FileReader that will read the entire contents of the given file path and send it at once.

NewFtpWriter instantiates new instance of an ftp writer.

NewHTTPRequest creates a new HTTPRequest and is essentially wrapping net/http's NewRequest function.

NewIoReader returns a new IoReader wrapping the given io.Reader object.

NewIoReaderWriter returns a new IoReaderWriter wrapping the given io.Reader object.

NewIoWriter returns a new IoWriter wrapping the given io.Writer object.

NewRegexpMatcher returns a new RegexpMatcher initialized with the given pattern to match.

NewS3ObjectReader reads a single object from the given S3 bucket.

NewS3PrefixReader reads a all objects from the given S3 bucket that match a prefix.

NewSftpWriter instantiates a new sftp writer, a connection to the remote server is delayed until data is recv'd by the writer.

NewSQLExecutor returns a new SQLExecutor.

NewSQLReader returns a new SQLReader operating in static mode.

NewSQLReaderWriter returns a new SQLReaderWriter ready for static querying.

NewSQLWriter returns a new SQLWriter.

BigQueryConfig is used when init'ing new BigQueryReader instances.

BigQueryReader is used to query data from Google's BigQuery, and it behaves similarly to SQLReader.

CSVTransformer converts data.JSON objects into a CSV string object and sends it on to the next stage.

CSVWriter is handles converting data.JSON objects into CSV format, and writing them to the given io.Writer.

FileReader opens and reads the contents of the given filename.

FtpWriter type represents an ftp writter processer.

FuncTransformer executes the given function on each data payload, sending the resuling data to the next stage.

HTTPRequest executes an HTTP request and passes along the response body.

IoReader wraps an io.Reader and reads it.

IoReaderWriter performs both the job of a IoReader and IoWriter.

IoWriter wraps any io.Writer object.

Passthrough simply passes the data on to the next stage.

RegexpMatcher checks if incoming data matches the given Regexp, and sends it on to the next stage only if it matches.

S3Reader handles retrieving objects from S3.

SCP executes the scp command, sending the given file to the given destination.

SftpWriter is an inline writer to remote sftp server.

SQLReader runs the given SQL and passes the resulting data to the next stage of processing.

SQLReaderWriter performs both the job of a SQLReader and SQLWriter.

SQLWriter handles INSERTing data.JSON into a specified SQL table.

SQLWriterData is a custom data structure you can send into a SQLWriter stage if you need to specify TableName on a per-data payload basis.