# README
#User Defined Function Service
This service enables custom UDF registration with pre defined provider set.
Usage
Register UDF
endly -r=reqister
@register.yaml
pipeline:
register-udf:
action: udf:register
udfs:
- id: AddressBookToProto
provider: ProtoWriter
params:
- /Project/proto/address_book.proto
- AddressBook
- id: ProtoToAddressBook
provider: ProtoReader
params:
- /Project/proto/address_book.proto
- AddressBook
Avro Reader UDF data validation:
endly -r=test
@test.yaml
pipeline:
loadData:
action: storage:download
udf: AvroReader
source:
URL: gs://mye2ebucket/data/output/1/app_data00000.avro
credentials: $gcpSecrets
destKey: matchedData
post:
myData: $Transformed
infoMyData:
action: print
message: $AsJSON(${myData})
infoMatched:
action: print
message: $AsJSON(${matchedData})
infoAll:
action: print
message: $AsJSON(${loadData})
assert:
action: validator:assert
actual: $AsData(${matchedData})
expect: $Cat(${parent.path}/expect/mydata.json)
Predefined UDF Providers
| Provider | Arguments |
|---|---|
| ProtoReader | schemaFile, messageType, importPath |
| ProtoReader | schemaFile, messageType, importPath |
| AvroWriter | avroSchema/URL, compression |
| CsvReader | headerFields, delimiter |
Service actions
UDF Service
| Service Id | Action | Description | Request | Response |
|---|---|---|---|---|
| udf | register | register custom UDF with udf provider | RegisterRequest | RegisterResponse |
# Packages
No description provided by the author
# Functions
AsProtobufMessage generic method for converting a map, or json string into a proto message.
DateOfBirth returns formatted date of birth, it take desired age, [month], [day], [timeformat].
FromProtobufMessage generic method for converting a proto message into a map.
GZipContentCorrupter corrupt zip content modifier.
GZipper copy modifier, mofidies source using zip udf.
Hostname return host from URL.
No description provided by the author
New creates a new udf service.
NewAvroReader creates a new avro reader UDFs.
NewAvroWriter creates a new avro writer provider.
No description provided by the author
NewProtoCodec creates a new protobuf codec.
NewProtoWriter creates a new proto writer provider.
NewProtoWriter creates a new proto writer provider.
No description provided by the author
RegisterProviders register the supplied providers.
TransformWithUDF transform payload with provided UDFs name.
URLJoin joins base URL and URI path.
URLPath return path from URL.
# Constants
ServiceID represents UDF service id.
# Structs
ProtoCodec represent a proto codec.
RegisterRequest represents a register udf request.
RegisterRequest represents a register response.