Test

This directory contains tests and testing docs for Knative Serving:

• Unit tests currently reside in the codebase alongside the code they test
• End-to-end tests, of which there are two types:
  • Conformance tests in /test/conformance
  • Other end-to-end tests in /test/e2e
• Performance tests reside in /test/performance

The conformance tests are a subset of the end to end test with more strict requirements around what can be tested.

If you want to add more tests, see adding_tests.md.

Presubmit tests

presubmit-tests.sh is the entry point for both the end-to-end tests and the conformance tests

This script, and consequently, the e2e and conformance tests will be run before every code submission. You can run these tests manually with:

test/presubmit-tests.sh

Note that to run presubmit-tests.sh or e2e-tests.sh scripts, you'll need kubernetes kubetest installed:

go get -u k8s.io/test-infra/kubetest

Running unit tests

To run all unit tests:

go test ./...

By default go test will not run the e2e tests, which need -tags=e2e to be enabled.

Running end to end tests

To run the e2e tests and the conformance tests, you need to have a running environment that meets the e2e test environment requirements, and you need to specify the build tag e2e.

go test -v -tags=e2e -count=1 ./test/conformance/...
go test -v -tags=e2e -count=1 ./test/e2e

Running performance tests

To run the performance tests, you need to have a running environment that meets the test environment requirements, and you need to specify the build tag performance.

go test -v -tags=performance -count=1 ./test/performance

Running a single test case

To run one e2e test case, e.g. TestAutoscaleUpDownUp, use the -run flag with go test:

go test -v -tags=e2e -count=1 ./test/e2e -run ^TestAutoscaleUpDownUp$

Running tests in short mode

Running tests in short mode excludes some large-scale E2E tests and saves time/resources required for running the test suite. To run the tests in short mode, use the -short flag with go test

go test -v -tags=e2e -count=1 -short ./test/e2e

To get a better idea where the flag is used, search for testing.Short() throughout the test source code.

Environment requirements

These tests require:

1. A running Knative Serving cluster.

2. The knative-testing resources:

   ko apply -f test/config
   

3. A docker repo containing the test images

Common Flags

• By default the e2e tests against the current cluster in ~/.kube/config using the environment specified in your environment variables.
• Since these tests are fairly slow, running them with logging enabled is recommended (-v).
• Using --logverbose to see the verbose log output from test as well as from k8s libraries.
• Using -count=1 is the idiomatic way to disable test caching

You can use test flags to control the environment your tests run against, i.e. override your environment variables:

go test -v -tags=e2e -count=1 ./test/conformance/... --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo
go test -v -tags=e2e -count=1 ./test/e2e --kubeconfig ~/special/kubeconfig --cluster myspecialcluster --dockerrepo myspecialdockerrepo

Test images

Building the test images

Note: this is only required when you run conformance/e2e tests locally with go test commands.

The upload-test-images.sh script can be used to build and push the test images used by the conformance and e2e tests. The script expects your environment to be setup as described in DEVELOPMENT.md.

To run the script for all end to end test images:

./test/upload-test-images.sh

A docker tag may be passed as an optional parameter. This can be useful on Minikube in tandem with the --tag flag:

eval $(minikube docker-env)
./test/upload-test-images.sh any-old-tag

Adding new test images

New test images should be placed in ./test/test_images.

Flags

These flags are useful for running against an existing cluster, making use of your existing environment setup.

Tests importing github.com/knative/serving/test recognize these flags:

• All flags added by knative/pkg/test
• --dockerrepo
• --tag
• --ingressendpoint
• --resolvabledomain

Overridding docker repo

The --dockerrepo argument lets you specify the docker repo from which images used by your tests should be pulled. This will default to the value of your KO_DOCKER_REPO environment variable if not specified.

go test -v -tags=e2e -count=1 ./test/conformance/... --dockerrepo gcr.myhappyproject
go test -v -tags=e2e -count=1 ./test/e2e --dockerrepo gcr.myhappyproject

Using a docker tag

The default docker tag used for the test images is latest, which can be problematic on Minikube. To avoid having to configure a remote container registry to support the Always pull policy for latest tags, you can have the tests use a specific tag:

go test -v -tags=e2e -count=1 ./test/conformance/... --tag any-old-tag
go test -v -tags=e2e -count=1 ./test/e2e --tag any-old-tag

Of course, this implies that you tagged the images when you uploaded them.

Using a custom ingress endpoint

Some environments (like minikube) do not support a Loadbalancer to make Knative services externally available. These environments usually rely on rewriting the Loadbalancer to a NodePort. The external address of such a NodePort is usually not easily obtained within the cluster automatically, but can be provided from the outside through the --ingressendpoint flag. For a minikube setup for example, you'd want to run tests against the default ingressgateway (port number 31380) running on the minikube node:

go test -v -tags=e2e -count=1 ./test/conformance/... --ingressendpoint "$(minikube ip):31380"
go test -v -tags=e2e -count=1 ./test/e2e --ingressendpoint "$(minikube ip):31380"

Using a resolvable domain

If you set up your cluster using the getting started docs, Routes created in the test will use the domain example.com, unless the route has label app=prod in which case they will use the domain prod-domain.com. Since these domains will not be resolvable to deployments in your test cluster, in order to make a request against the endpoint, the test use the IP assigned to the service istio-ingressgateway in the namespace istio-system and spoof the Host in the header.

If you have configured your cluster to use a resolvable domain, you can use the --resolvabledomain flag to indicate that the test should make requests directly against Route.Status.Domain and does not need to spoof the Host.