pkg.gl

In order to communicate with Elastic, the following connection details need to be provided. You may specify them directly in a pipeline recipe as key-value pairs within the component's setup block, or you can create a Connection from the Integration Settings page and reference the whole setup as setup: ${connection.<my-connection-id>}.

Field	Field ID	Type	Note
Cloud ID (required)	`cloud-id`	string	Fill in the Cloud ID for the Elasticsearch instance
API Key (required)	`api-key`	string	Fill in the API key for the Elasticsearch instance (please use encoded one)

Supported Tasks

Search

Search for documents in Elasticsearch, support full text search

Input	ID	Type	Description
Task ID (required)	`task`	string	`TASK_SEARCH`
Index Name (required)	`index-name`	string	Name of the Elasticsearch index
ID	`id`	string	The ID of the document
Query	`query`	string	Full text search query for search task, query will be prioritised over filter if both are provided, if both query and filter are not provided, all documents will be selected
Filter	`filter`	object	The query dsl filter which starts with "query" field, please refer to here.
Filter SQL	`filter-sql`	string	The filter to be applied to the data with SQL syntax, which starts with WHERE clause, empty for no filter
Size	`size`	integer	Number of documents to return. If empty then all documents will be returned
Fields	`fields`	array[string]	The fields to return in the documents. If empty then all fields will be returned
Minimum Score	`min-score`	number	Minimum score to consider for search results. If empty then no minimum score will be considered

Output	ID	Type	Description
Status	`status`	string	Search operation status
Result	`result`	object	Result of the search operation

Output Objects in Search

Result

Field	Field ID	Type	Note
Data	`data`	array	The data returned from the search operation
Documents	`documents`	array	The documents returned from the search operation
IDs	`ids`	array	The ids returned from the search operation

Vector Search

Search for vector similarity search in Elasticsearch

Input	ID	Type	Description
Task ID (required)	`task`	string	`TASK_VECTOR_SEARCH`
Index Name (required)	`index-name`	string	Name of the Elasticsearch index
Field (required)	`field`	string	Field name of the vector to search for similar vectors
Query Vector	`query-vector`	array[number]	Query vector to search for similar vectors
K	`k`	integer	K of documents to do kNN vector search
Num Candidates	`num-candidates`	integer	Number of candidates to be considered for kNN vector search. Default to 2 times of k
Filter	`filter`	object	The query dsl filter which starts with "filter" field, please refer to here.
Filter SQL	`filter-sql`	string	The filter to be applied to the data with SQL syntax, which starts with WHERE clause, empty for no filter
Fields	`fields`	array[string]	The fields to return in the documents. If empty then all fields will be returned
Minimum Score	`min-score`	number	Minimum score to consider for search results. If empty then no minimum score will be considered

Output	ID	Type	Description
Status	`status`	string	Search operation status
Result	`result`	object	Result of the vector search operation

Output Objects in Vector Search

Result

Field	Field ID	Type	Note
Documents	`documents`	array	The documents returned from the vector search operation
IDs	`ids`	array	The ids returned from the vector search operation
Metadata	`metadata`	array	The metadata returned from the vector search operation
Vectors	`vectors`	array	The vectors returned from the vector search operation

Index

Index a document into Elasticsearch

Input	ID	Type	Description
Task ID (required)	`task`	string	`TASK_INDEX`
Index Name (required)	`index-name`	string	Name of the Elasticsearch index
ID	`id`	string	The ID of the document
Data (required)	`data`	object	Data to be indexed

Output	ID	Type	Description
Status	`status`	string	Index operation status

Multi Index

Index multiple documents into Elasticsearch with bulk API

Input	ID	Type	Description
Task ID (required)	`task`	string	`TASK_MULTI_INDEX`
Index Name (required)	`index-name`	string	Name of the Elasticsearch index
Array ID	`array-id`	array[string]	The array of id
Array Data (required)	`array-data`	array[object]	Array data to be indexed

Output	ID	Type	Description
Status	`status`	string	Index operation status

Update

Update a document in Elasticsearch

Input	ID	Type	Description
Task ID (required)	`task`	string	`TASK_UPDATE`
Index Name (required)	`index-name`	string	Name of the Elasticsearch index
ID	`id`	string	The ID of the document
Query	`query`	string	Full text search query for update task, query will be prioritised over filter if both are provided, if both query and filter are not provided, all documents will be selected
Filter	`filter`	object	The query dsl filter which starts with "query" field, please refer to here.
Filter SQL	`filter-sql`	string	The filter to be applied to the data with SQL syntax, which starts with WHERE clause, empty for no filter
Update (required)	`update-data`	object	Update data

Output	ID	Type	Description
Status	`status`	string	Update operation status

Delete

Delete documents from Elasticsearch

Input	ID	Type	Description
Task ID (required)	`task`	string	`TASK_DELETE`
Index Name (required)	`index-name`	string	Name of the Elasticsearch index
ID	`id`	string	The ID of the document
Query	`query`	string	Full text search query for delete task, query will be prioritised over filter if both are provided, if both query and filter are not provided, all documents will be selected
Filter	`filter`	object	The query dsl filter which starts with "query" field, please refer to here.
Filter SQL	`filter-sql`	string	The filter to be applied to the data with SQL syntax, which starts with WHERE clause, empty for no filter

Output	ID	Type	Description
Status	`status`	string	Delete operation status

Create Index

Create an index in Elasticsearch

Input	ID	Type	Description
Task ID (required)	`task`	string	`TASK_CREATE_INDEX`
Index Name (required)	`index-name`	string	Name of the Elasticsearch index
Mappings	`mappings`	object	Index mappings which starts with {"mappings":{"properties"}} field, please refer to here for vector search and here for other mappings.

Output	ID	Type	Description
Status	`status`	string	Create index operation status

Delete Index

Delete an index in Elasticsearch

Input	ID	Type	Description
Task ID (required)	`task`	string	`TASK_DELETE_INDEX`
Index Name (required)	`index-name`	string	Name of the Elasticsearch index

Output	ID	Type	Description
Status	`status`	string	Delete index operation status

# Functions

CreateIndex

mappings refer to elasticsearch documentation for more information, use dense_vector type with similarity and dims fields pre-defined mappings is mandatory for vector search, if index isnt created with mappings, vector search will not work as dense_vector type doesn't explicitly defined.

DeleteDocument