Deploys and Changes

Status200 keeps a record of what changed in your project: deploys, Kubernetes rollouts, config changes, and the fixes it ran itself. Each change is drawn as a marker on the charts of its service, and Status200 AI reads them when it investigates an incident, so "it started right after the 10:02 deploy" is the first thing you learn, not the last.

How it works#

What Status200 records on its own#

You do not have to send anything for these. Status200 records them from the telemetry it already receives, and from what it does itself.

• The first version a service reports is its baseline, not a deploy. A service that has run for months is not deployed the moment Status200 starts receiving its telemetry.
• A rollback to a version seen in the last 30 days is not recorded, because that version is not new. A rollback Status200 AI runs is recorded as a fix.
• More than six new versions or ReplicaSets in an hour stop being recorded until the hour is over. A "version" that changes with every pod, such as a build timestamp, would otherwise record a deploy for each pod.

A deploy is only recorded for a service that reports service.version. Set it, with the environment and the commit, in the resource attributes of your OpenTelemetry SDK or collector:

OTEL_RESOURCE_ATTRIBUTES="service.version=2.3.1,deployment.environment.name=production,vcs.ref.head.revision=$GIT_SHA"

A rollout needs the Kubernetes agent, which stamps each pod's telemetry with its Deployment and ReplicaSet.

Record a deploy from a pipeline#

A pipeline can record a deploy, or any other change, as it happens: a config change, a migration, a feature flag. It writes with a telemetry ingestion key of your project, the kind your applications send telemetry with. Create one under Project Settings → Telemetry & APM → Ingestion Keys.

A key pinned to one service records every change under that service.

oneuptime deploy \
  --service checkout \
  --service-version 2.3.1 \
  --commit "$GIT_SHA" \
  --environment production \
  --url https://status200.ru \
  --ingestion-key "$STATUS200_TELEMETRY_INGESTION_KEY"

- name: Record the deploy in Status200
  env:
    STATUS200_URL: https://status200.ru
    STATUS200_TELEMETRY_INGESTION_KEY: ${{ secrets.STATUS200_TELEMETRY_INGESTION_KEY }}
  run: |
    npx -y @status200/cli deploy \
      --service checkout \
      --service-version "${{ github.ref_name }}" \
      --commit "${{ github.sha }}" \
      --repository "${{ github.server_url }}/${{ github.repository }}" \
      --environment production

curl -X POST https://status200.ru/telemetry/change-events/v1/ingest \
  -H "Content-Type: application/json" \
  -H "x-status200-token: $STATUS200_TELEMETRY_INGESTION_KEY" \
  -d '{"events": [{"serviceName": "checkout", "version": "2.3.1", "commit": "1a2b3c4", "environment": "production"}]}'

On a self-hosted Status200, use your own address instead of https://status200.ru. The API answers with how many events it stored and how many it skipped:

{ "ingested": 1, "skipped": 0 }

See oneuptime deploy for every option of the CLI.

How Status200 AI uses changes#

Every AI investigation of an incident or an alert starts from the changes recorded in the two hours before it, and in the ten minutes after, for a pipeline that records its deploy once the rollout finished. The changes on the services, clusters and hosts the incident is linked to come first.

For up to four of them that name a service, Status200 reads the service's telemetry in the 30 minutes before and after the change, and marks the change a suspect when the service got measurably worse right after it:

Status200 decides which change is a suspect from these numbers, not from what a change's title says. The investigation checks a suspect first, says whether it caused the incident, and says whether rolling it back is the fastest safe fix.

When the suspect is a Kubernetes rollout on a cluster whose AI fixes are on, Status200 AI rolls it back first, with a kubectl rollout undo that Status200 builds from the recorded change. A deploy that ships code is rolled back through its commits: the investigation names the commit or pull request that broke things, so a fix pull request can revert exactly that.

Status200 also checks every deploy on its own, whether or not anything pages. Once the 30 minutes after a deploy have passed, it compares them with the 30 minutes before by the same thresholds, and when the service got worse, an AI Insight names the deploy on that service.

A fix pull request Status200 AI opened is checked the same way: once a recorded deploy contains its commit, Status200 watches there what the fix was for - the exception, the operation it made faster, the monitor behind the incident or alert, the telemetry it adds - and says whether the fix worked (see Automatic code fixes). The deployed fix is read in the deploy's version only when the service's own telemetry reports that version, and otherwise from 10 minutes after the deploy, once old instances have drained.

In an investigation and in Ask AI, recent_changes lists the changes in any window, and get_change_details shows what a deploy shipped: the commits since the service's previous deploy, their authors, the pull requests and the files they touched. That needs the commit on the change and a repository connected through the GitHub App.

Reference#

Fields#

The top-level fields are stored as attributes under the OpenTelemetry names: version as service.version, commit as vcs.ref.head.revision, repository as vcs.repository.url.full, branch as vcs.ref.head.name, pullRequest as vcs.change.id and environment as deployment.environment.name. An attribute sent under one of those names wins over its field.

Attributes whose names start with status200. are Status200's own and are dropped from what a pipeline sends. Every change a pipeline records says so in status200.change.source, which is api.

Limits#

Next steps#