Azure DevOps Exporter (VSTS)

Prometheus exporter for Azure DevOps (VSTS) for projects, builds, build times (elapsed and queue wait time), agent pool utilization and active pull requests.

[!IMPORTANT] I've lost access to Azure DevOps (ADO) instance so i cannot provide proper support anymore. The exporter will still be updated but don't expect active development.

Also Microsoft seems to have lost interest in ADO in favor of GitHub, the development of ADO actions has nearly stopped, they don't fix use of deprecated Azure APIs and even features which are supported in GitHub for a long time are not backported to ADO.

Configuration

Usage:
  azure-devops-exporter [OPTIONS]

Application Options:
      --log.debug                             debug mode [$LOG_DEBUG]
      --log.devel                             development mode [$LOG_DEVEL]
      --log.json                              Switch log output to json format [$LOG_JSON]
      --scrape.time=                          Default scrape time (time.duration) (default: 30m) [$SCRAPE_TIME]
      --scrape.time.projects=                 Scrape time for project metrics (time.duration) [$SCRAPE_TIME_PROJECTS]
      --scrape.time.repository=               Scrape time for repository metrics (time.duration) [$SCRAPE_TIME_REPOSITORY]
      --scrape.time.build=                    Scrape time for build metrics (time.duration) [$SCRAPE_TIME_BUILD]
      --scrape.time.release=                  Scrape time for release metrics (time.duration) [$SCRAPE_TIME_RELEASE]
      --scrape.time.deployment=               Scrape time for deployment metrics (time.duration) [$SCRAPE_TIME_DEPLOYMENT]
      --scrape.time.pullrequest=              Scrape time for pullrequest metrics  (time.duration) [$SCRAPE_TIME_PULLREQUEST]
      --scrape.time.stats=                    Scrape time for stats metrics  (time.duration) [$SCRAPE_TIME_STATS]
      --scrape.time.resourceusage=            Scrape time for resourceusage metrics  (time.duration) [$SCRAPE_TIME_RESOURCEUSAGE]
      --scrape.time.query=                    Scrape time for query results  (time.duration) [$SCRAPE_TIME_QUERY]
      --scrape.time.live=                     Scrape time for live metrics (time.duration) (default: 30s) [$SCRAPE_TIME_LIVE]
      --stats.summary.maxage=                 Stats Summary metrics max age (time.duration) [$STATS_SUMMARY_MAX_AGE]
      --azure.tenant-id=                      Azure tenant ID for Service Principal authentication [$AZURE_TENANT_ID]
      --azure.client-id=                      Client ID for Service Principal authentication [$AZURE_CLIENT_ID]
      --azure.client-secret=                  Client secret for Service Principal authentication [$AZURE_CLIENT_SECRET]
      --azuredevops.url=                      Azure DevOps URL (empty if hosted by Microsoft) [$AZURE_DEVOPS_URL]
      --azuredevops.access-token=             Azure DevOps access token [$AZURE_DEVOPS_ACCESS_TOKEN]
      --azuredevops.access-token-file=        Azure DevOps access token (from file) [$AZURE_DEVOPS_ACCESS_TOKEN_FILE]
      --azuredevops.organisation=             Azure DevOps organization [$AZURE_DEVOPS_ORGANISATION]
      --azuredevops.apiversion=               Azure DevOps API version (default: 5.1) [$AZURE_DEVOPS_APIVERSION]
      --azuredevops.agentpool=                Enable scrape metrics for agent pool (IDs) [$AZURE_DEVOPS_AGENTPOOL]
      --whitelist.project=                    Filter projects (UUIDs) [$AZURE_DEVOPS_FILTER_PROJECT]
      --blacklist.project=                    Filter projects (UUIDs) [$AZURE_DEVOPS_BLACKLIST_PROJECT]
      --timeline.state=                       Filter timeline states (completed, inProgress, pending) (default: completed) [$AZURE_DEVOPS_FILTER_TIMELINE_STATE]
      --builds.all.project=                   Fetch all builds from projects (UUIDs or names) [$AZURE_DEVOPS_FETCH_ALL_BUILDS_FILTER_PROJECT]
      --list.query=                           Pairs of query and project UUIDs in the form: '<queryId>@<projectId>' [$AZURE_DEVOPS_QUERIES]
      --tags.schema=                          Tags to be extracted from builds in the format 'tagName:type' with following types: number, info, bool [$AZURE_DEVOPS_TAG_SCHEMA]
      --tags.build.definition=                Build definition ids to query tags (IDs) [$AZURE_DEVOPS_TAG_BUILD_DEFINITION]
      --cache.path=                           Cache path (to folder, file://path... or azblob://storageaccount.blob.core.windows.net/containername or
                                              k8scm://{namespace}/{configmap}}) [$CACHE_PATH]
      --request.concurrency=                  Number of concurrent requests against dev.azure.com (default: 10) [$REQUEST_CONCURRENCY]
      --request.retries=                      Number of retried requests against dev.azure.com (default: 3) [$REQUEST_RETRIES]
      --servicediscovery.refresh=             Refresh duration for servicediscovery (time.duration) (default: 30m) [$SERVICEDISCOVERY_REFRESH]
      --limit.project=                        Limit number of projects (default: 100) [$LIMIT_PROJECT]
      --limit.builds-per-project=             Limit builds per project (default: 100) [$LIMIT_BUILDS_PER_PROJECT]
      --limit.builds-per-definition=          Limit builds per definition (default: 10) [$LIMIT_BUILDS_PER_DEFINITION]
      --limit.releases-per-project=           Limit releases per project (default: 100) [$LIMIT_RELEASES_PER_PROJECT]
      --limit.releases-per-definition=        Limit releases per definition (default: 100) [$LIMIT_RELEASES_PER_DEFINITION]
      --limit.deployments-per-definition=     Limit deployments per definition (default: 100) [$LIMIT_DEPLOYMENTS_PER_DEFINITION]
      --limit.releasedefinitions-per-project= Limit builds per definition (default: 100) [$LIMIT_RELEASEDEFINITION_PER_PROJECT]
      --limit.build-history-duration=         Time (time.Duration) how long the exporter should look back for builds (default: 48h) [$LIMIT_BUILD_HISTORY_DURATION]
      --limit.release-history-duration=       Time (time.Duration) how long the exporter should look back for releases (default: 48h) [$LIMIT_RELEASE_HISTORY_DURATION]
      --server.bind=                          Server address (default: :8080) [$SERVER_BIND]
      --server.timeout.read=                  Server read timeout (default: 5s) [$SERVER_TIMEOUT_READ]
      --server.timeout.write=                 Server write timeout (default: 10s) [$SERVER_TIMEOUT_WRITE]

Help Options:
  -h, --help                                  Show this help message

Authentication

This exporter supports Azure DevOps PAT tokens and ServicePrincipal authentication with Client Secret and (AKS) Workload Identity.

Metrics

Metric	Scraper	Description
azure_devops_stats	live	General scraper stats
azure_devops_agentpool_info	live	Agent Pool informations
azure_devops_agentpool_size	live	Number of agents per agent pool
azure_devops_agentpool_usage	live	Usage of agent pool (used agents; percent 0-1)
azure_devops_agentpool_queue_length	live	Queue length per agent pool
azure_devops_agentpool_agent_info	live	Agent information per agent pool
azure_devops_agentpool_agent_status	live	Status informations (eg. created date) for each agent in a agent pool
azure_devops_agentpool_agent_job	live	Currently running jobs on each agent
azure_devops_project_info	live/projects	Project informations
azure_devops_build_latest_info	live	Latest build information
azure_devops_build_latest_status	live	Latest build status informations
azure_devops_pullrequest_info	pullrequest	Active PullRequests
azure_devops_pullrequest_status	pullrequest	Status informations (eg. created date) for active PullRequests
azure_devops_pullrequest_label	pullrequest	Labels set on active PullRequests
azure_devops_build_info	build	Build informations
azure_devops_build_status	build	Build status infos (queued, started, finished time)
azure_devops_build_stage	build	Build stage infos (duration, errors, warnings, started, finished time)
azure_devops_build_phase	build	Build phase infos (duration, errors, warnings, started, finished time)
azure_devops_build_job	build	Build job infos (duration, errors, warnings, started, finished time)
azure_devops_build_task	build	Build task infos (duration, errors, warnings, started, finished time)
azure_devops_build_definition_info	build	Build definition info
azure_devops_release_info	release	Release informations
azure_devops_release_artifact	release	Release artifcact informations
azure_devops_release_environment	release	Release environment list
azure_devops_release_environment_status	release	Release environment status informations
azure_devops_release_approval	release	Release environment approval list
azure_devops_release_definition_info	release	Release definition info
azure_devops_release_definition_environment	release	Release definition environment list
azure_devops_repository_info	repository	Repository informations
azure_devops_repository_stats	repository	Repository stats
azure_devops_repository_commits	repository	Repository commit counter
azure_devops_repository_pushes	repository	Repository push counter
azure_devops_query_result	live	Latest results of given queries
azure_devops_deployment_info	deployment	Release deployment informations
azure_devops_deployment_status	deployment	Release deployment status informations
azure_devops_stats_agentpool_builds	stats	Number of buildsper agentpool, project and result (counter)
azure_devops_stats_agentpool_builds_wait	stats	Build wait time per agentpool, project and result (summary)
azure_devops_stats_agentpool_builds_duration	stats	Build duration per agentpool, project and result (summary)
azure_devops_stats_project_builds	stats	Number of builds per project, definition and result (counter)
azure_devops_stats_project_builds_wait	stats	Build wait time per project, definition and result (summary)
azure_devops_stats_project_builds_success	stats	Success rating of build per project and definition (summary)
azure_devops_stats_project_builds_duration	stats	Build duration per project, definition and result (summary)
azure_devops_stats_project_release_duration	stats	Release environment duration per project, definition, environment and result (summary)
azure_devops_stats_project_release_success	stats	Success rating of release environment per project, definition and environment (summary)
azure_devops_resourceusage_build	resourceusage	Usage of limited and paid Azure DevOps resources (build)
azure_devops_resourceusage_license	resourceusage	Usage of limited and paid Azure DevOps resources (license)
azure_devops_api_request_*		REST api request histogram (count, latency, statuscCodes)

Prometheus queries

Last 3 failed releases per definition for one project

topk by(projectID,releaseDefinitionName,path) (3,
  azure_devops_release_environment{projectID="XXXXXXXXXXXXXXXX", status!="succeeded", status!="inProgress"}
  * on (projectID,releaseID,environmentID) group_left() (azure_devops_release_environment_status{type="created"})
  * on (projectID,releaseID) group_left(releaseName, releaseDefinitionID) (azure_devops_release_info)
  * on (projectID,releaseDefinitionID) group_left(path, releaseDefinitionName) (azure_devops_release_definition_info)
)

Agent pool usage (without PoolMaintenance)

count by(agentPoolID) (
  azure_devops_agentpool_agent_job{planType!="PoolMaintenance"}
  * on(agentPoolAgentID) group_left(agentPoolID) (azure_devops_agentpool_agent_info)
)
/ on (agentPoolID) group_left() (azure_devops_agentpool_size)
* on (agentPoolID) group_left(agentPoolName) (azure_devops_agentpool_info)

Current running jobs

label_replace(
    azure_devops_agentpool_agent_job{planType!="PoolMaintenance"}
    * on (agentPoolAgentID) group_left(agentPoolID,agentPoolAgentName) azure_devops_agentpool_agent_info
    * on (agentPoolID) group_left(agentPoolName) (azure_devops_agentpool_info)
  , "projectID", "$1", "scopeID", "^(.+)$"
)
* on (projectID) group_left(projectName) (azure_devops_project_info)

Agent pool size

azure_devops_agentpool_info
* on (agentPoolID) group_left() (azure_devops_agentpool_size)

Agent pool size (enabled and online)

azure_devops_agentpool_info
* on (agentPoolID) group_left() (
  count by(agentPoolID) (azure_devops_agentpool_agent_info{status="online",enabled="true"})
)