Skip to content
Docs
Podman Agent

Podman Agent

View as Markdown

The Status200 Podman agent is one container you run on a Podman host. It reads the stats and log files of every container on the host and sends them to Status200, where the host appears with its containers, metrics and logs. A second container, the Podman AI agent, lets Status200 AI investigate the host. Podman monitors alert on what the agent sends.

Install the agentTwo containers, with podman run or Podman Compose.What the agent collectsMetrics, logs and an inventory of every container.Container logs need k8s-filePodman logs to journald unless you switch.TroubleshootingRestart loops, missing metrics, missing logs.

How it works

The agent is an OpenTelemetry Collector with Status200's configuration built in. Through Podman's Docker-compatible API socket it reads each container's stats every 30 seconds and lists every container, image, network and volume every 5 minutes. It tails the containers' log files and sends everything to your Status200 URL over OTLP, with the ingestion key you give it.

flowchart TB containers["Your containers"] -->|"stats, log files,<br/>inventory"| agent["Podman agent"] agent -->|"OTLP over HTTPS"| oneuptime["Status200"] oneuptime -->|"investigates an<br/>incident or alert"| ai["Podman AI agent"] ai -->|"read-only<br/>docker commands"| containers

Two containers on your Podman host

The host registers itself with its first data, under the name you give it in PODMAN_HOST_NAME. The AI agent is optional: leave it out and the collector works the same.

Before you begin

  • Podman 4.0 or later, and root access on the host.
  • Podman's API socket at /run/podman/podman.sock. Turn it on with sudo systemctl enable --now podman.socket.
  • An ingestion key. Create one under Project Settings → Telemetry & APM → Ingestion Keys and copy its value.
  • Outbound HTTPS from the host to your Status200 URL.
  • For container logs, containers that use the k8s-file log driver. Rootful Podman logs to journald by default, which the agent cannot read. See Container logs need the k8s-file log driver.

Install the agent

  1. Start the agent

    Replace YOUR_STATUS200_URL with your Status200 URL, YOUR_TELEMETRY_INGESTION_TOKEN with the ingestion key, and my-podman-host with a name for this host, such as prod-podman-01. The name is how the host appears in Status200, so keep it stable: a new name registers a new host.

    Podman CLI

    Bash 
    podman run -d \
  --name status200-podman-agent \
  --user 0:0 \
  --restart unless-stopped \
  -v /run/podman/podman.sock:/run/podman/podman.sock:ro \
  -v /var/lib/containers/storage:/var/lib/containers/storage:ro \
  -e STATUS200_URL="YOUR_STATUS200_URL" \
  -e STATUS200_SERVICE_TOKEN="YOUR_TELEMETRY_INGESTION_TOKEN" \
  -e PODMAN_HOST_NAME="my-podman-host" \
  status200/podman-agent:release

    Then start the AI agent beside it, with the same values. AI investigations are on by default: while Status200 AI investigates an incident or alert on this host, it runs read-only docker commands through the AI agent, against Podman's Docker-compatible API, and it changes nothing unless you allow fixes. To install without AI investigations, skip this command.

    Bash 
    podman run -d \
  --name status200-podman-ai-agent \
  --user 0:0 \
  --restart unless-stopped \
  --read-only --tmpfs /tmp \
  -v /run/podman/podman.sock:/run/podman/podman.sock:ro \
  -e STATUS200_URL="YOUR_STATUS200_URL" \
  -e STATUS200_SERVICE_TOKEN="YOUR_TELEMETRY_INGESTION_TOKEN" \
  -e STATUS200_AI_AGENT_RESOURCE_TYPE=podman \
  -e PODMAN_HOST_NAME="my-podman-host" \
  status200/resource-ai-agent:release

    Podman Compose

    Save this as docker-compose.yml in a folder on the host. The second service is the AI agent: leave it out to run the collector without AI investigations.

    docker-compose.ymlYAML 
    services:
  status200-podman-agent:
    image: status200/podman-agent:release
    container_name: status200-podman-agent
    user: "0:0"
    restart: unless-stopped
    volumes:
      - /run/podman/podman.sock:/run/podman/podman.sock:ro
      - /var/lib/containers/storage:/var/lib/containers/storage:ro
    environment:
      - STATUS200_URL=YOUR_STATUS200_URL
      - STATUS200_SERVICE_TOKEN=YOUR_TELEMETRY_INGESTION_TOKEN
      - PODMAN_HOST_NAME=my-podman-host
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
  status200-podman-ai-agent:
    image: status200/resource-ai-agent:release
    container_name: status200-podman-ai-agent
    user: "0:0"
    restart: unless-stopped
    read_only: true
    tmpfs:
      - /tmp
    volumes:
      - /run/podman/podman.sock:/run/podman/podman.sock:ro
    environment:
      - STATUS200_URL=YOUR_STATUS200_URL
      - STATUS200_SERVICE_TOKEN=YOUR_TELEMETRY_INGESTION_TOKEN
      - STATUS200_AI_AGENT_RESOURCE_TYPE=podman
      - PODMAN_HOST_NAME=my-podman-host

    Start it in that folder:

    Bash 
    podman compose up -d

  2. Check the agent is running

    Bash 
    podman ps --filter name=status200-podman-agent
podman logs -f status200-podman-agent

    The logs should show this line once the collector has started:

    Everything is ready. Begin running and processing data.

  3. Find the host in Status200

    Open Products → Infrastructure → Podman → All Hosts. The host appears under its PODMAN_HOST_NAME within a minute or so, with its containers, metrics and logs. The AI agent shows as connected on the host's AI agent page, and at the bottom of its Overview. The host's Topology page draws its Compose projects, containers, images, networks and volumes as one map: see Topology.

Environment variables

VariableRequiredDescription
STATUS200_URLYesYour Status200 URL, such as https://status200.ru or your self-hosted instance. Use http:// and its port if the instance is HTTP-only.
STATUS200_SERVICE_TOKENYesThe ingestion key from Project Settings → Telemetry & APM → Ingestion Keys.
PODMAN_HOST_NAMENoThe host's name in Status200. Defaults to podman-host, so give every host its own.
DOCKER_API_VERSIONNoThe Docker API version the agent speaks to the Podman socket. Defaults to 1.44. Lower it if the socket reports an older maximum, or set it to an empty value to negotiate it (see Troubleshooting).

To change a value, remove the agent and run its command again (Podman CLI), or edit docker-compose.yml and run podman compose up -d (Podman Compose).

What the agent collects

DataWhat it is
Container metricsCPU (usage, percentage, throttling), memory (usage, limit, percentage), network and block I/O, uptime, restarts and process count, per container, every 30 seconds.
Container logsThe stdout and stderr of every container, from the ctr.log file the k8s-file log driver writes.
InventoryEvery container in every state, and every image, network and volume, every 5 minutes.

Each container's metrics and logs are filed under a service named after the container. Each log line gets a severity from the level it names, such as [ERROR], level=warn or {"level":"info"}; a line that names none takes its stream's: stderr is ERROR, stdout is INFO. Lines that start with whitespace or a closing bracket join the line before, so a stack trace arrives as one record. The agent's own logs are left out.

Podman Monitor lists every metric and what it means.

Container logs need the k8s-file log driver

The agent tails /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log, the files Podman's k8s-file log driver writes (json-file is the same driver in Podman). Rootful Podman's default driver is journald, which leaves no file to read, and so does a remote driver such as syslog. Such a container's metrics still arrive; its logs do not.

Check a container's log driver, and Podman's default:

Bash 
podman inspect <container> --format '{{.HostConfig.LogConfig.Type}}'
podman info --format '{{.Host.LogDriver}}'

Run a container with k8s-file, or switch a Compose service to it:

Bash 
podman run --log-driver k8s-file --log-opt max-size=100m ... <image>
YAML 
services:
  my-app:
    image: my-app:latest
    logging:
      driver: "k8s-file"
      options:
        max-size: "100m"

Or make it the default for every container created afterwards, in containers.conf (/etc/containers/containers.conf rootful, ~/.config/containers/containers.conf rootless):

containers.confTOML 
[containers]
log_driver = "k8s-file"
log_size_max = 104857600

Then recreate the containers. A container keeps the log driver it was created with, so restarting it is not enough.

Upgrading the agent

When the agent is older than your Status200, a warning sign appears beside Agent Version on the host's Overview. Select it to see these commands.

Podman CLI

Bash 
podman pull status200/podman-agent:release
podman pull status200/resource-ai-agent:release
podman rm -f status200-podman-agent status200-podman-ai-agent

Then run both podman run commands from Start the agent again.

Podman Compose

Bash 
podman compose pull
podman compose up -d

The release tag always points at the newest release. To pin a version, use status200/podman-agent:<version>, such as status200/podman-agent:10.0.31. The same image is on the GitHub Container Registry as ghcr.io/status200/podman-agent:release.

Uninstalling the agent

Podman CLI

Bash 
podman rm -f status200-podman-agent
podman rm -f status200-podman-ai-agent

Podman Compose

Bash 
podman compose down

The host stays in Status200 with the data it sent. If you no longer need it, delete it with Delete Host in its menu.

AI agent

The commands and the Compose file above, the Podman agent's install.sh and the docker-compose.yml in its [PodmanAgent directory](<YOUR_STATUS200_URL>/docs/static/scripts/agents/PodmanAgent) all run the Podman AI agent: a second container, status200-podman-ai-agent (image status200/resource-ai-agent:release), that runs the read-only commands Status200 AI asks for while it investigates an incident or alert on this host — through Podman's Docker-compatible API, with the docker CLI: docker ps -a, docker logs --tail 200 web, docker container inspect web — and, only if you allow it, applies fixes such as restarting a named container. It reads the same STATUS200_URL, STATUS200_SERVICE_TOKEN and PODMAN_HOST_NAME as the collector, so it serves exactly this host, and it shows up on the host's AI agent page in Status200.

  • AI investigations are on by default. Once the agent connects, Status200 AI investigates incidents and alerts on this host with it, and the host's Overview shows the agent's status. To stop, set STATUS200_AI_INVESTIGATION=false where the agent runs (Change under What AI may do on the AI agent page shows how), or leave the agent out.
  • It is read-only unless you start it with STATUS200_AI_ALLOW_WRITES=true; STATUS200_AI_WRITE_TARGETS (for example web-*,api-*) limits which containers a fix may touch. Start it with STATUS200_AI_FIXES too, to say how fixes run — ask-for-approval (a person approves each one), automatic or bypass-approval — and the AI agent page shows it read-only (What AI may do, set by the agent).
  • It needs the same socket as the collector and runs as root with it mounted, because the socket is root-owned — and for rootful Podman whoever can use the socket is root on this host, :ro or not. The agent's command policy is the limit: it never runs exec, run, rm or prune, and it never changes itself, the collector or anything in STATUS200_AI_PROTECTED_TARGETS (container name globs).
  • install.sh --no-ai-agent leaves it out; podman rm -f status200-podman-ai-agent removes it. Upgrade it like the collector: podman pull status200/resource-ai-agent:release, remove it, and start it again.

What it may run, how fixes work and how to troubleshoot it: Infrastructure AI Agents.

Troubleshooting

The agent restarts with "client version is too new"

The agent's logs show:

Error: cannot start pipelines: failed to start "docker_stats" receiver:
Error response from daemon: client version 1.44 is too new.
Maximum supported API version is 1.41

A Docker-API server that enforces a maximum client version refuses a newer one, so the receiver never starts and the collector exits with it. Check the API version the Podman socket reports:

Bash 
curl -s -o /dev/null -D - --unix-socket /run/podman/podman.sock http://localhost/_ping | grep -i '^api-version'

Then add -e DOCKER_API_VERSION=1.41 (or the value you got) to podman run, or set DOCKER_API_VERSION in Compose. Newer servers still serve older API versions, so the setting stays valid after an upgrade.

Rather not look the number up? Set DOCKER_API_VERSION to the empty string. The agent then negotiates the version with the socket (one HEAD /_ping, then the maximum the socket reports) instead of pinning one:

Bash 
podman run -d ... -e DOCKER_API_VERSION= ...
Podman socket permission denied, or no socket

The agent must run as root to read /run/podman/podman.sock. Check that --user 0:0 is in your podman run command, or user: "0:0" in Compose, and that the socket is on: sudo systemctl enable --now podman.socket. Rootless Podman has its socket at /run/user/<uid>/podman/podman.sock instead (systemctl --user enable --now podman.socket): mount that path in place of /run/podman/podman.sock.

The host shows as Disconnected
  1. Check that the agent is running: podman ps --filter name=status200-podman-agent
  2. Look for errors in its logs: podman logs status200-podman-agent 2>&1 | grep -i error
  3. Check STATUS200_URL and STATUS200_SERVICE_TOKEN: the key must be a valid ingestion key that has not been disabled.
  4. Check that the host can reach your Status200 URL over the network.
No metrics appear
  1. Check that the Podman socket is mounted into the agent. Its image has no shell, so look from a throwaway container that shares its mounts: podman run --rm --volumes-from status200-podman-agent alpine:3.19 ls -la /run/podman/podman.sock
  2. Look for export errors in its logs: podman logs status200-podman-agent 2>&1 | tail -100
  3. Check that the ingestion key is valid and has not expired.
Container CPU is a flat line

An older agent sends the docker_stats receiver's own CPU figure, which needs the stats reading before the current one, and Podman's Docker-compatible API never sends it: the receiver divides all the CPU a container used since it started by all the CPU the host used since it booted. The line barely moves and does not follow the load, while container.cpu.usage.total does. Upgrade the agent: from its second reading on, it sends the CPU each container used over each 30-second interval. Readings already stored stay as they were.

Metrics arrive but no container logs

Your containers most likely use journald, Podman's default log driver. Check which files the agent watches, and which driver a container uses:

Bash 
podman logs status200-podman-agent 2>&1 | grep -E "Started watching file|no files match"
podman inspect <container> --format '{{.HostConfig.LogConfig.Type}}'

If the agent reports no files match the configured criteria, switch the containers to k8s-file and recreate them, as Container logs need the k8s-file log driver shows.

The logs are on another host, or the host is named podman-host

The host's name comes from PODMAN_HOST_NAME. An agent started without it reports podman-host, so every host set up that way looks like the same one; change the name later and Status200 registers a second host, which then holds the new data. Set PODMAN_HOST_NAME to a name of its own on each host, and recreate the agent. To check the name the agent reports:

Bash 
podman inspect status200-podman-agent --format '{{range .Config.Env}}{{println .}}{{end}}' | grep PODMAN_HOST_NAME

Next steps

Podman MonitorAlert on container CPU, memory and restarts.DatabasesDatabase containers on the host are found and get pages of their own.Kubernetes AgentMonitor a Kubernetes cluster instead.Host OpenTelemetry CollectorMonitor the machine itself: CPU, disks, processes and logs.