Build agents that heal themselves: what changed in agent 1.3

A build agent has one job: stay connected and take work. When it can't, it should say why in one line and wait for a fix. It shouldn't flood the log, and it certainly shouldn't fight the person trying to fix it.

Agent 1.2 failed that test in a way that's worth walking through. Agent 1.3 is the result. This post covers what went wrong, what changed, and the small server changes that came with it.

#The loop

Here is what a Docker agent printed, over and over:

OpsNexus agent 1.2.0 reporting to http://host.docker.internal:8080 as build-01
Pipeline runner: 2 executor(s), labels docker, work dir /data/work
Claim failed: HTTP 401
The server no longer accepts this agent (unregistered?). Register it again.

The story behind it was ordinary. Someone unregistered the agent in OpsNexus, then created a new registration token and ran the install command again. That should have been the fix. It wasn't, and two separate problems made it worse.

Problem one: the saved config always won. The container kept its agent ID and secret in a data volume. On start, the agent only registered when there was no config file. The volume still had the old config, so the new token was ignored. Every heartbeat got 401, the agent exited, and Docker's --restart unless-stopped started it again. And again.

Problem two: the address only worked on one machine. OpsNexus had been opened on localhost, so the install command was rewritten to host.docker.internal, and the Linux, macOS and Windows commands used localhost. Those addresses work on the OpsNexus host and nowhere else. The same command failed on every other machine. The obvious fix, changing PUBLIC_URL to a network address, wasn't possible, because Google SSO needs PUBLIC_URL exactly as registered.

#Fix 1: wait, don't exit

The most important change is the smallest one. On a 401, agent 1.3 no longer exits. It:

1. logs one clear message with the exact fix;
2. pauses the pipeline runner, so it doesn't claim work it can't report on;
3. checks again every 5 minutes.

The message depends on where the agent runs. In a container it says to create a new token and recreate the container with ONX_REGISTRATION_TOKEN. On an installed service it prints the full register command with the agent's own URL and name filled in. If the server starts accepting the agent again, it logs "resuming" and carries on.

The reasoning is simple. Exiting on an error that a restart can't fix just hands the problem to the supervisor, and the supervisor's only answer is to try again. Waiting keeps the process alive, the log readable, and the restart counter at zero.

#Fix 2: a new token means "register again"

Agent 1.3 remembers which token its config came from, without storing the token. At registration it saves a SHA-256 fingerprint of the token next to its ID and secret. On start:

no config                         -> register with ONX_REGISTRATION_TOKEN
config + same token fingerprint   -> run with the saved registration
config + different token          -> "New registration token found; registering again"

So running the install command again with a fresh token just works, even with the same volume. That is exactly what people try first. The Docker command on the install page also removes an old container with the same name before starting the new one:

docker rm -f build-01 2>/dev/null || true
docker run -d \
  --name build-01 --hostname build-01 --restart unless-stopped \
  -e ONX_URL=http://192.168.1.10:8080 \
  -e ONX_REGISTRATION_TOKEN=onx_reg_... \
  -e ONX_AGENT_NAME=build-01 \
  -e ONX_EXECUTORS=2 -e ONX_LABELS=docker \
  -v build-01-data:/data \
  opsnexus-agent

#And a stale token never breaks a working agent

Fix 2 created a new edge case, which 1.3.2 closes. Suppose someone runs an old command again, with a token that was already used. The agent sees a "new" fingerprint, tries to register, and the server rejects the token. In 1.3.0 the agent then exited, and Docker restarted it in a loop again.

Since 1.3.2, when registration with the given token fails and a saved registration exists, the agent logs it and keeps using the saved registration:

Registration with the given token failed (...); using the saved registration (build-01, id ...)

If the saved registration was unregistered too, Fix 1 takes over: one message, then a check every 5 minutes. The rule is that the agent never exits while it has something it can try.

#Fix 3: an address that works everywhere

On the server side there's a new setting, AGENT_URL. It's the address agents use. When it's empty, PUBLIC_URL is used.

# .env
PUBLIC_URL=http://localhost:8080          # what people open; SSO needs it unchanged
AGENT_URL=http://192.168.1.10:8080        # what build agents on other machines use

Install scripts embed AGENT_URL. A new endpoint, GET /api/agents/install-info, returns the address and whether it's local-only. The install page uses it for an editable OpsNexus address field, with a warning when the address is localhost or host.docker.internal. Every command (Linux, macOS, Windows and Docker) uses that one field. There's no automatic rewriting any more, so the command you copy is the command that works on other machines.

The agent checks too. At start it prints a warning when its address is localhost or host.docker.internal. Registration errors now say whether the token was rejected or the server couldn't be reached. Those two problems need very different fixes.

When OpsNexus moves, agents don't need to register again. ONX_URL always overrides the saved address in a container. Installed agents get a new command:

python onx_agent.py set-url https://opsnexus.example.com

#Bonus: Git errors that explain themselves

Agent 1.3.1 fixes a different kind of confusion. A run failed with remote: Repository not found, although the pipeline had a credential. GitHub and GitLab answer "not found" for private repositories a token can't see. In this case the token was a valid GitHub fine-grained token, but for a different resource owner than the repository's. Fine-grained tokens only reach repositories of the account or organization chosen when the token was created.

The agent now recognises Git access errors (not found, authentication failed, publickey, 403). It says whether the pipeline has no credential or a credential without access, and explains the fix. It also stops retrying with a full fetch, which fails the same way.

#Upgrading

• Docker: rebuild the agent image (step 1 of the Docker command on the install page), then recreate the container with a new token.
• Installed service: open Build agents, pick the agent and choose Upgrade, or run the installer again.

The Build agents docs cover installation, and Troubleshooting lists each log line with its fix.