Categorygithub.com/conduitio-labs/conduit-connector-http

# README

Conduit Connector for HTTP

The HTTP connector is a Conduit plugin. It provides both, a source and a destination HTTP connectors.

How to build?

Run make build to build the connector's binary.

Testing

Run make test to run all the unit tests.

Source

The HTTP source connector pulls data from the HTTP URL every pollingPeriod, the source adds the params and headers to the request, and sends it to the URL with the specified method from the Configuration. The returned data is used to create an openCDC record and return it.

Note: when using the OPTIONS method, the resulted options will be added to the record's metadata.

Configuration

namedescriptionrequireddefault valueexample
urlHTTP URL to send requests to.truehttps://example.com/api/v1
methodHTTP method to use in the request, supported methods are (GET,HEAD,OPTIONS).falseGETPOST
headersHTTP headers to use in the request, comma separated list of : separated pairs.falseAuthorization:Bearer TOKEN_VALUE,Content-Type:application/xml
params.*parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1".falseparams.query="foobar"
pollingperiodhow often the connector will get data from the url, formatted as a time.Duration.false"5m""5m"
script.parseResponse

The path to a .js file containing the code to parse the response.

The signature of the function needs to be:

function parseResponse(bytes)
        

where bytes is the original response's raw bytes (i.e. unparsed).

The function needs to return a Response object.

false/path/to/get_request_data.js

An example script can be found in test/get_request_data.js
script.getRequestData

The path to a .js file containing the code to prepare the request data.

The signature of the function needs to be:

function getRequestData(cfg, previousResponse, position)
        

where:

  • cfg (a map) is the connector configuration
  • previousResponse (a map) contains data from the previous response (if any), returned by parseResponse
  • position (a byte array) contains the starting position of the connector.

The function needs to return a Request object.

false/path/to/parse_response.js

An example script can be found in test/parse_response.js

Destination

The HTTP destination connector pushes data from upstream resources to an HTTP URL via Conduit. the destination adds the params and headers to the request, and sends it to the URL with the specified method from the Configuration.

Note: The request Body that will be sent is the value under record.Payload.After, if you want to change the format of that or manipulate the field in any way, please check our Builtin Processors Docs , or check Standalone Processors Docs if you'd like to build your own processor .

Configuration

namedescriptionrequireddefault value
urlIs a Go template expression for the URL used in the HTTP request, using Go templates. The value provided to the template is opencdc.Record, so the template has access to all its fields (e.g. .Position, .Key, .Metadata, and so on). We also inject all template functions provided by sprig to make it easier to write templates.true
methodHttp method to use in the request, supported methods are (POST,PUT,DELETE,PATCH).falsePOST
headersHttp headers to use in the request, comma separated list of : separated pairs.false
params.*parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1".false

# Packages

No description provided by the author

# Functions

No description provided by the author
NewMockrequestBuilder creates a new mock instance.
NewMockRequestBuilder creates a new mock instance.
NewMockresponseParser creates a new mock instance.
NewMockResponseParser creates a new mock instance.
No description provided by the author
Specification returns the connector's specification.

# Constants

No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author

# Variables

Connector combines all constructors for each plugin in one struct.

# Structs

No description provided by the author
No description provided by the author
No description provided by the author
MockrequestBuilder is a mock of requestBuilder interface.
MockRequestBuilder is a mock of requestBuilder interface.
MockrequestBuilderMockRecorder is the mock recorder for MockrequestBuilder.
MockRequestBuilderMockRecorder is the mock recorder for MockRequestBuilder.
MockresponseParser is a mock of responseParser interface.
MockResponseParser is a mock of responseParser interface.
MockresponseParserMockRecorder is the mock recorder for MockresponseParser.
MockResponseParserMockRecorder is the mock recorder for MockResponseParser.
No description provided by the author
No description provided by the author
No description provided by the author
No description provided by the author