Tracetest CLI e2e Tests

In this folder we have the End-to-end tests done on the CLI to guarantee that the CLI is working fine. The main idea is to test every CLI command against the Tracetest server with different data stores and different operating systems.

Implementation Status

Linux	Windows	MacOS
:white_check_mark:	:soon:	:soon:

Tracetest Data Store

Jaeger	Tempo	OpenSearch	SignalFx	OTLP	ElasticAPM	New Relic	Lightstep	Datadog	AWS X-Ray	Honeycomb	Dynatrace
:white_check_mark:	:soon:	:soon:	:soon:	:soon:	:soon:	:soon:	:soon:	:soon:	:soon:	:soon:	:soon:

CLI Commands to Test

Misc and Flags

CLI Command	Test scenarios
version	VersionCommand
help	HelpCommand
--help, -h	HelpCommand
--config	All scenarios

Run Tests and TestSuites

CLI Command	Test scenarios
run test -f [test-definition]	RunTestWithGrpcTrigger
run test -f [test-definition] --vars [variableset-id]	RunTestWithHttpTriggerAndVariableSetFile
run test -f [test-definition] --vars [variableset-definition]	RunTestWithHttpTriggerAndVariableSetFile
run testsuite -f [testsuite-definition]	RunTestSuite
run testsuite -f [testsuite-definition] --vars [variableset-id]
run testsuite -f [testsuite-definition] --vars [variableset-definition]

Resources: Config

CLI Command	Test scenarios
apply config -f [config-file]	ApplyConfig
delete config --id current	DeleteConfig
get config --id current --output pretty	GetConfig, ApplyConfig, DeleteConfig
get config --id current --output json	GetConfig
get config --id current --output yaml	GetConfig
list config --output pretty	ListConfig
list config --output json	ListConfig
list config --output yaml	ListConfig

Resources: Data Store

CLI Command	Test scenarios
apply datastore -f [data-store-file]	ApplyDatastore
delete datastore --id current	DeleteDatastore
get datastore --id current --output pretty	GetDatastore, ApplyDatastore, DeleteDatastore
get datastore --id current --output json	GetDatastore
get datastore --id current --output yaml	GetDatastore
list datastore --output pretty	ListDatastore
list datastore --output json	ListDatastore
list datastore --output yaml	ListDatastore

Resources: Demo

CLI Command	Test scenarios
apply demo -f [new-demo-file]	ApplyDemo
apply demo -f [existing-demo-file]	ApplyDemo
delete demo --id [existing-id]	DeleteDemo
delete demo --id [non-existing-id]	DeleteDemo
get demo --id [non-existing-id]	GetDemo, DeleteDemo
get demo --id [existing-id] --output pretty	GetDemo
get demo --id [existing-id] --output json	GetDemo
get demo --id [existing-id] --output yaml	GetDemo
list demo --output pretty	ListDemo
list demo --output json	ListDemo
list demo --output yaml	ListDemo
list demo --skip 1 --take 1	ListDemo
list demo --sortBy name --sortDirection asc	ListDemo

Resources: VariableSet

CLI Command	Test scenarios
apply variableset -f [new-variableset-file]	ApplyVariableSet
apply variableset -f [existing-variableset-file]	ApplyVariableSet
delete variableset --id [existing-id]	DeleteVariableSet
delete variableset --id [non-existing-id]	DeleteVariableSet
get variableset --id [non-existing-id]	GetVariableSet, DeleteVariableSet
get variableset --id [existing-id] --output pretty	GetVariableSet
get variableset --id [existing-id] --output json	GetVariableSet
get variableset --id [existing-id] --output yaml	GetVariableSet
list variableset --output pretty	ListVariableSet
list variableset --output json	ListVariableSet
list variableset --output yaml	ListVariableSet
list variableset --skip 1 --take 2	ListVariableSet
list variableset --sortBy name --sortDirection asc	ListVariableSet

Resources: PollingProfile

CLI Command	Test scenarios
apply pollingprofile -f [pollingprofile-file]	ApplyPollingProfile
delete pollingprofile --id current	DeletePollingProfile
get pollingprofile --id current --output pretty	GetPollingProfile, ApplyPollingProfile, DeletePollingProfile
get pollingprofile --id current --output json	GetPollingProfile
get pollingprofile --id current --output yaml	GetPollingProfile
list pollingprofile --output pretty	ListPollingProfile
list pollingprofile --output json	ListPollingProfile
list pollingprofile --output yaml	ListPollingProfile

Resources: TestRunner

CLI Command	Test scenarios
apply testrunner -f [testrunner-file]	ApplyTestRunner
delete testrunner --id current	DeleteTestRunner
get testrunner --id current --output pretty	GetTestRunner
get testrunner --id current --output json	GetTestRunner
get testrunner --id current --output yaml	GetTestRunner
list testrunner --output pretty	ListTestRunner
list testrunner --output json	ListTestRunner
list testrunner --output yaml	ListTestRunner

Resources: Analyzer

CLI Command	Test scenarios
apply analyzer -f [analyzer-file]	ApplyAnalyzer
delete analyzer --id current	DeleteAnalyzer
get analyzer --id current --output pretty	GetAnalyzer
get analyzer --id current --output json	GetAnalyzer
get analyzer --id current --output yaml	GetAnalyzer
list analyzer --output pretty	ListAnalyzer
list analyzer --output json	ListAnalyzer
list analyzer --output yaml	ListAnalyzer

Resources: TestSuites

CLI Command	Test scenarios
apply testsuite -f [new-testsuite-file]	ApplyTestSuite
apply testsuite -f [existing-testsuite-file]	ApplyTestSuite
delete testsuite --id [existing-id]	DeleteTestSuite
delete testsuite --id [non-existing-id]	DeleteTestSuite
get testsuite --id [non-existing-id]	GetTestSuite, DeleteTestSuite
get testsuite --id [existing-id] --output pretty	GetTestSuite
get testsuite --id [existing-id] --output json	GetTestSuite
get testsuite --id [existing-id] --output yaml	GetTestSuite
list testsuite --output pretty	ListTestSuite
list testsuite --output json	ListTestSuite
list testsuite --output yaml	ListTestSuite
list testsuite --skip 1 --take 2	ListTestSuite
list testsuite --sortBy name --sortDirection asc	ListTestSuite

Resources: Tests

CLI Command	Test scenarios
apply test -f [new-test-file]	ApplyTest
apply test -f [existing-test-file]	ApplyTest
delete test --id [existing-id]	DeleteTest
delete test --id [non-existing-id]	DeleteTest
get test --id [non-existing-id]	GetTest, DeleteTest
get test --id [existing-id] --output pretty	GetTest
get test --id [existing-id] --output json	GetTest
get test --id [existing-id] --output yaml	GetTest
list test --output pretty	ListTest
list test --output json	ListTest
list test --output yaml	ListTest
list test --skip 1 --take 2	ListTest
list test --sortBy name --sortDirection asc	ListTest