When an agent will not run

The agent by symptom — it will not start, enrolment fails, it shows as offline, jobs fail at once — and what each line in its log means.

The agent talks into its log, one JSON line per event. Almost every answer on this page begins with reading it:

journalctl -u runjet-agent -f        # on a host
docker logs -f runjet-agent          # in a container

The lines below are quoted as the agent writes them, so they can be searched for.

The agent will not start

At startup the agent would rather stop with an explanation than run in a configuration that does not match what you think it is. Each of these sentences ends the process, and systemd tries again five seconds later with the same result, until the cause is removed.

In the log	Cause and cure
`RUNJET_URL is required`	The address is missing from `agent.env`. Usually a file `enroll` did not write because it already existed — the command printed what should be in it at the time.
`RUNJET_PUBLIC_KEYS is required: without it the agent cannot tell its scheduler from anyone else who can reach it`	The same, for Runjet's key.
`no stored identity and no ENROLLMENT_TOKEN: register this agent in Runjet and pass the one-time token it issues`	The agent has never been through enrolment: there is no `identity.json` in `/var/lib/runjet-agent` and no token in the environment. On a host, run `enroll`; in a container, the volume or `ENROLLMENT_TOKEN` is missing.
`invalid AGENT_POLL_WAIT`	The value is not a duration in the form `30s`, `1m`.
`AGENT_PASS_ENV entry "…" looks like a pattern: list exact names…`	A pattern instead of a name. List the names one by one (why).
`AGENT_PASS_ENV names …, which is this agent's own configuration…`	One of the agent's own settings is in the allowlist. Remove it.
`AGENT_JOB_USER "…": user: unknown user …`	The job account does not exist on the host. Create it (`useradd --system --create-home --home-dir /var/lib/runjet-job runjet-job`), or — if it exists only in LDAP or SSSD, which a static binary cannot see — give a numeric uid and `AGENT_JOB_GROUP`.
`AGENT_JOB_USER is the numeric id … and no account has it, so AGENT_JOB_GROUP must supply the group id`	A numeric uid with no entry in `/etc/passwd`; add the gid.
`cannot run commands as runjet-job: …`	The agent cannot switch to the account. Under systemd, `AmbientCapabilities=CAP_SETUID CAP_SETGID` is missing from the unit; in a container, `--user root` is needed. The message names both.
`AGENT_SECRETS_FILE "…" does not exist`	A path to a file that is not there. Create it, or remove the variable — unset means "no file".
`/etc/runjet-agent/secrets:12: expected KEY=value`	A line with no `=` in the secrets file (the format).

Enrolment fails

enroll ends with a sentence beginning enroll:.

POST /agent/enroll: 401 Unauthorized: {"error":"enrollment token is not valid"} — Runjet does not know the token. Either its hour has run out, or it has already been used once, or a reset has replaced it in the meantime, or -url points somewhere other than where it was issued. The response is the same for all four cases; the only way on is a new token from the dialog.
… already holds an identity, so this host is enrolled — there is already an agent on this host. What to do about it is in the reset.
enrollment response is not from a trusted scheduler — a response arrived, but it was not signed with the key from -key. A typo in the key, or something between you and Runjet is answering for it. The token is not burnt: the key is checked before anything is spent.
426 Upgrade Required — this binary speaks a protocol version outside the window Runjet serves. Download the current release (upgrading); the token stays valid.
no enrollment token: pass -token, -token-file, or feed it on stdin — the command was given empty input. Most often -token left out with the intention of feeding the token on stdin, and then run from a script with no stdin.

The agent is offline

Two things say so, and they measure differently:

The dot in the agent list goes red and says Offline — not seen recently when the agent has not reported in for roughly two minutes. It is cheap and accurate; during a deployment of Runjet it greys out and comes back.
An email, "has stopped checking in", goes to the workspace's admins and operators when the agent has been silent for ten minutes, once per outage; the agent's return resets it for next time. The longer threshold is because a restart of Runjet breaks every agent's held call at once, and a short threshold would post to the entire customer base. What else arrives by email is in notifications.

While an agent is silent, the jobs aimed at it do not run — and on its return a missed run is caught up only for the jobs that have it switched on.

The usual causes, in order of how often they happen: the machine is off; the service is stopped (systemctl status runjet-agent); the host has lost its route out (DNS, a firewall, a proxy — the agent needs only HTTPS to api.runjet.dev); and a certificate the host does not trust, which shows in the log as poll failed with a TLS error. In every case the agent tries again every five seconds and reconnects on its own as soon as it can; nothing needs restarting.

poll failed with 401 Unauthorized means something else: Runjet knows the agent, but its key no longer works — after a reset or a deletion. That one does not fix itself.

Jobs fail at once

A run that ends failed within a second usually says why in the error field on the run:

this agent has no value for DB_PASSWORD — the job names a secret this host does not have (secrets on the host). A job on several agents may therefore succeed on one and not on another.
command not found in the output — the job was given a built environment, not the agent's. PATH is the one from the systemd unit, not from your shell; JAVA_HOME and the like have to be in AGENT_PASS_ENV (isolation). In a container: the tool is not in the image.
No such file or directory on a relative path — jobs run in the runjet-job account's home, and where it has none, in the agent's working directory, where they cannot write. The agent reports this at startup (below).
Permission denied on a file "the agent can obviously see" — the job runs as runjet-job, not as the agent, and that is deliberate. Let that account read the file.

Warnings at startup

These sentences do not stop the agent. They say it is running in a configuration you may not have chosen.

In the log	What it means
`job commands run as this agent's own user, so they can read its private key and its secrets file; set AGENT_JOB_USER to separate them`	`AGENT_JOB_USER=` is empty. In the agent list this reads as not isolated (why it matters).
`the job account has no home directory, so commands run in the agent's working directory and a job writing a relative path will fail; create it to give jobs somewhere to work`	The job account exists without a home. `mkdir -m 0700 /var/lib/runjet-job && chown runjet-job: /var/lib/runjet-job`, or create the account again with `--create-home`.
`secrets file is readable by other users on this host`	The secrets file is not `0600`. `chmod 0600` it.
`some secrets are too short to mask in run output and will appear in it verbatim`	The named values are under five characters and will show in the output (masking).

What the agent says when all is well

agent ready with the name, Runjet's address, maxParallel and the job account; then silence until work arrives. run started and run finished for each run; runjet changed how much runs at once when the ceiling from the plan has changed. A log with nothing in it after startup is the log of a healthy agent, waiting.