NewBigQueryReader returns an instance of a BigQueryExtractor ready to run a static query.

NewBigQueryWriter instantiates a new instance of BigQueryWriter.

NewBigQueryWriterForNewTable instantiates a new instance of BigQueryWriter and prepares to write results to a new table.

NewCSVTransformer returns a new CSVTransformer wrapping the given io.Writer object.

NewCSVWriter returns a new CSVWriter wrapping the given io.Writer object.

NewDynamicBigQueryReader returns an instance of a BigQueryExtractor ready to run a dynamic query based on the sqlGenerator function.

NewDynamicSQLExecutor returns a new SQLExecutor operating in dynamic mode.

NewDynamicSQLReader returns a new SQLReader operating in dynamic mode.

NewDynamicSQLReaderWriter returns a new SQLReaderWriter ready for dynamic querying.

NewFileReader returns a new FileReader that will read the entire contents of the given file path and send it at once.

NewFtpWriter instantiates new instance of an ftp writer.

NewFuncTransformer instantiates a new instance of func transformer.

NewHTTPRequest creates a new HTTPRequest and is essentially wrapping net/http's NewRequest function.

NewIoReader returns a new IoReader wrapping the given io.Reader object.

NewIoReaderWriter returns a new IoReaderWriter wrapping the given io.Reader object.

NewIoWriter returns a new IoWriter wrapping the given io.Writer object.

NewPassthrough instantiates a new instance of Passthrough.

NewRegexpMatcher returns a new RegexpMatcher initialized with the given pattern to match.

NewS3ObjectReader reads a single object from the given S3 bucket.

NewS3PrefixReader reads a all objects from the given S3 bucket that match a prefix.

NewS3Writer instaniates a new S3Writer.

NewSCP instantiates a new instance of SCP.

NewSftpReader instantiates a new sftp reader, a connection to the remote server is delayed until data is recv'd by the reader By default, the connection to the remote client will be closed in the Finish() func.

NewSftpReaderByClient instantiates a new sftp reader using an existing connection to the remote server.

NewSftpWriter instantiates a new sftp writer, a connection to the remote server is delayed until data is recv'd by the writer By default, the connection to the remote client will be closed in the Finish() func.

NewSftpWriterByFile allows you to manually manage the connection to the remote file object.

NewSQLExecutor returns a new SQLExecutor.

NewSQLReader returns a new SQLReader operating in static mode.

NewSQLReaderWriter returns a new SQLReaderWriter ready for static querying.

NewSQLWriter returns a new SQLWriter.

BigQueryConfig is used when init'ing new BigQueryReader instances.

BigQueryReader is used to query data from Google's BigQuery, and it behaves similarly to SQLReader.

BigQueryWriter is used to write data to Google's BigQuery.

CSVTransformer converts data.JSON objects into a CSV string object and sends it on to the next stage.

CSVWriter is handles converting data.JSON objects into CSV format, and writing them to the given io.Writer.

FileReader opens and reads the contents of the given filename.

FtpWriter type represents an ftp writter processor.

FuncTransformer executes the given function on each data payload, sending the resuling data to the next stage.

HTTPRequest executes an HTTP request and passes along the response body.

IoReader wraps an io.Reader and reads it.

IoReaderWriter performs both the job of a IoReader and IoWriter.

IoWriter wraps any io.Writer object.

Passthrough simply passes the data on to the next stage.

RegexpMatcher checks if incoming data matches the given Regexp, and sends it on to the next stage only if it matches.

S3Reader handles retrieving objects from S3.

S3Writer sends data upstream to S3.

SCP executes the scp command, sending the given file to the given destination.

SftpReader reads a single object at a given path, or walks through the directory specified by the path (SftpReader.Walk must be set to true).

SftpWriter is an inline writer to remote sftp server.

SQLExecutor runs the given SQL and swallows any returned data.

SQLReader runs the given SQL and passes the resulting data to the next stage of processing.

SQLReaderWriter performs both the job of a SQLReader and SQLWriter.

SQLWriter handles INSERTing data.JSON into a specified SQL table.

SQLWriterData is a custom data structure you can send into a SQLWriter stage if you need to specify TableName on a per-data payload basis.