Categorygithub.com/elastic/elastic-packagetoolsreadme
package
0.102.0
Repository: https://github.com/elastic/elastic-package.git
Documentation: pkg.go.dev
Overview
Versions
1
Dependencies
7
Dependents
0
Files
90 SLOC

# README

elastic-package

elastic-package is a command line tool, written in Go, used for developing Elastic packages. It can help you lint, format, test and build your packages. Learn about each of these and other features in Commands below.

Currently, elastic-package only supports packages of type Elastic Integrations.

Please review the integrations contributing guide to learn how to build and develop packages, understand the release procedure and explore the builder tools.

Getting started

Download latest release from the Releases page.

On macOS, use xattr -r -d com.apple.quarantine elastic-package after downloading to allow the binary to run.

Alternatively, you may use go install but you will not be able to use the elastic-package version command or check updates.

go install github.com/elastic/elastic-package@latest

Please make sure that you've correctly setup environment variables - $GOPATH and $PATH, and elastic-package is accessible from your $PATH.

Change directory to the package under development.

cd my-package

Run the help command and see available commands:

elastic-package help

Development

Even though the project is "go-gettable", there is the Makefile present, which can be used to build, install, format the source code among others. Some examples of the available targets are:

make build - build the tool source

make clean - delete elastic-package binary and build folder

make format - format the Go code

make check - one-liner, used by CI to verify if source code is ready to be pushed to the repository

make install - build the tool source and move binary to $GOBIN

make gomod - ensure go.mod and go.sum are up to date

make update - update README.md file

make licenser - add the Elastic license header in the source code

To start developing, download and build the latest main of elastic-package binary:

git clone https://github.com/elastic/elastic-package.git
cd elastic-package
make build

When developing on Windows, please use the core.autocrlf=input or core.autocrlf=false option to avoid issues with CRLF line endings:

git clone --config core.autocrlf=input https://github.com/elastic/elastic-package.git
cd elastic-package
make build

This option can be also configured on existing clones with the following commands. Be aware that these commands will remove uncommited changes.

git config core.autocrlf input
git rm --cached -r .
git reset --hard

Testing with integrations repository

While working on a new branch, it is interesting to test these changes with all the packages defined in the integrations repository. This allows to test a much wider scenarios than the test packages that are defined in this repository.

This test can be triggered automatically directly from your Pull Request by adding a comment test integrations. Example:

This comment triggers this Buildkite pipeline (Buildkite job).

This pipeline creates a new draft Pull Request in integration updating the required dependencies to test your own changes. As a new pull request is created, a CI job will be triggered to test all the packages defined in this repository. A new comment with the link to this new Pull Request will be posted in your package-spec Pull Request.

IMPORTANT: Remember to close this PR in the integrations repository once you close the package-spec Pull Request.

Usually, this process would require the following manual steps:

  1. Create your elastic-package pull request and push all your commits
  2. Get the SHA of the latest changeset of your PR: 
     $ git show -s --pretty=format:%H
1131866bcff98c29e2c84bcc1c772fff4307aaca
  3. Go to the integrations repository, and update go.mod and go.sum with that changeset: 
    cd /path/to/integrations/repostiory
go mod edit -replace github.com/elastic/elastic-package=github.com/<your_github_user>/elastic-package@1131866bcff98c29e2c84bcc1c772fff4307aaca
go mod tidy
  4. Push these changes into a branch and create a Pull Request
    • Creating this PR would automatically trigger a new Jenkins pipeline.

Testing with Elastic serverless

While working on a branch, it might be interesting to test your changes using a project created in Elastic serverless, instead of spinning up a local Elastic stack. To do so, you can add a new comment while developing in your Pull request a comment like test serverless.

Adding that comment in your Pull Request will create a new build of this Buildkite pipeline. This pipeline creates a new Serverless project and run some tests with the packages defined in the test/packages/parallel folder. Currently, there are some differences with respect to testing with a local Elastic stack:

  • System tests are not executed.
  • Disabled comparison of results in pipeline tests to avoid errors related to GeoIP fields
  • Pipeline tests cannot be executed with coverage flags.

At the same time, this pipeline is going to be triggered daily to test the latest contents of the main branch with an Elastic serverless project.

Commands

elastic-package currently offers the commands listed below.

Some commands have a global context, meaning that they can be executed from anywhere and they will have the same result. Other commands have a package context; these must be executed from somewhere under a package's root folder and they will operate on the contents of that package.

For more details on a specific command, run elastic-package help <command>.

elastic-package help

Context: global

Use this command to get a listing of all commands available under elastic-package and a brief description of what each command does.

elastic-package completion

Context: global

Use this command to output shell completion information.

The command output shell completions information (for bash, zsh, fish and powershell). The output can be sourced in the shell to enable command completion.

Run elastic-package completion and follow the instruction for your shell.

{{ .Cmds }}

Elastic Package profiles

The profiles subcommand allows to work with different configurations. By default, elastic-package uses the "default" profile. Other profiles can be created with the elastic-package profiles create command. Once a profile is created, it will have its own directory inside the elastic-package data directory. Once you have more profiles, you can change the default with elastic-package profiles use.

You can find the profiles in your system with elastic-package profiles list.

You can delete profiles with elastic-package profiles delete.

Each profile can have a config.yml file that allows to persist configuration settings that apply only to commands using this profile. You can find a config.yml.example that you can copy to start.

The following settings are available per profile:

  • stack.apm_enabled can be set to true to start an APM server and configure instrumentation in services managed by elastic-package. Traces for these services are available in the APM UI of the kibana instance managed by elastic-package. Supported only by the compose provider. Defaults to false.
  • stack.elastic_cloud.host can be used to override the address when connecting with the Elastic Cloud APIs. It defaults to https://cloud.elastic.co.
  • stack.geoip_dir defines a directory with GeoIP databases that can be used by Elasticsearch in stacks managed by elastic-package. It is recommended to use an absolute path, out of the .elastic-package directory.
  • stack.kibana_http2_enabled can be used to control if HTTP/2 should be used in versions of kibana that support it. Defaults to true.
  • stack.logsdb_enabled can be set to true to activate the feature flag in Elasticsearch that enables logs index mode in all data streams that support it. Defaults to false.
  • stack.logstash_enabled can be set to true to start Logstash and configure it as the default output for tests using elastic-package. Supported only by the compose provider. Defaults to false.
  • stack.self_monitor_enabled enables monitoring and the system package for the default policy assigned to the managed Elastic Agent. Defaults to false.
  • stack.serverless.type selects the type of serverless project to start when using the serverless stack provider.
  • stack.serverless.region can be used to select the region to use when starting serverless projects.

Useful environment variables

There are available some environment variables that could be used to change some of the elastic-package settings:

  • Related to docker-compose / docker compose commands:

    • ELASTIC_PACKAGE_COMPOSE_DISABLE_VERBOSE_OUTPUT: If set to true, it disables the progress output from docker compose/docker-compose commands.
      • For versions v2 < 2.19.0, it sets --ansi never flag.
      • For versions v2 >= 2.19.0, it sets --progress plain flag and --quiet-pull for up sub-command`.

  • Related to global elastic-package settings:

    • ELASTIC_PACKAGE_CHECK_UPDATE_DISABLED: if set to true, elastic-package is not going to check for newer versions.
    • ELASTIC_PACKAGE_PROFILE: Name of the profile to be using.
    • ELASTIC_PACKAGE_DATA_HOME: Custom path to be used for elastic-package data directory. By default this is ~/.elastic-package.

  • Related to the build process:

    • ELASTIC_PACKAGE_REPOSITORY_LICENSE: Path to the default repository license.
    • ELASTIC_PACKAGE_LINKS_FILE_PATH: Path to the links table file (e.g. links_table.yml) with the link definitions to be used in the build process of a package.

  • Related to signing packages:

    • ELASTIC_PACKAGE_SIGNER_PRIVATE_KEYFILE: Path to the private key file to sign packages.
    • ELASTIC_PACKAGE_SIGNER_PASSPHRASE: Passphrase to use the private key file.

  • Related to tests:

    • ELASTIC_PACKAGE_SERVERLESS_PIPELINE_TEST_DISABLE_COMPARE_RESULTS: If set to true, the results from pipeline tests are not compared to avoid errors from GeoIP.

  • To configure the Elastic stack to be used by elastic-package:

    • ELASTIC_PACKAGE_ELASTICSEARCH_HOST: Host of the elasticsearch (e.g. https://127.0.0.1:9200)
    • ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME: User name to connect to elasticsearch (e.g. elastic)
    • ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD: Password of that user.
    • ELASTIC_PACKAGE_ELASTICSEARCH_KIBANA_HOST: Kibana URL (e.g. https://127.0.0.1:5601)
    • ELASTIC_PACKAGE_ELASTICSEARCH_CA_CERT: Path to the CA certificate to connect to the Elastic stack services.

  • To configure an external metricstore while running benchmarks (more info at system benchmarking docs or rally benchmarking docs):

    • ELASTIC_PACKAGE_ESMETRICSTORE_HOST: Host of the elasticsearch (e.g. https://127.0.0.1:9200)
    • ELASTIC_PACKAGE_ESMETRICSTORE_USERNAME: Username to connect to elasticsearch (e.g. elastic)
    • ELASTIC_PACKAGE_ESMETRICSTORE_PASSWORD: Password for the user.
    • ELASTIC_PACKAGE_ESMETRICSTORE_CA_CERT: Path to the CA certificate to connect to the Elastic stack services.

Release process

This project uses GoReleaser to release a new version of the application (semver). Release publishing is automatically managed by the Jenkins CI (Jenkinsfile) and it's triggered by Git tags. Release artifacts are available in the Releases section.

Steps to create a new release

  1. Fetch latest main from upstream (remember to rebase the branch):
git fetch upstream
git rebase upstream/main
  1. Create Git tag with release candidate:
git tag v0.15.0 # let's release v0.15.0!
  1. Push new tag to the upstream.
git push upstream v0.15.0

The CI will run a new job for the just pushed tag and publish released artifacts. Please expect an automated follow-up PR in the Integrations repository to bump up the version (sample PR).