My Agent Looked Alive for 36 Minutes. It Had Written 39 Bytes.

Everyone publishes what worked. This publishes what broke, and the rule it became.

I once launched a code-writing CLI worker and watched it stay alive for 36 minutes. The process table said it was there. The worker had not crashed. The reasonable conclusion, at the time, was that it was still working.

It was not working. It had written 39 bytes.

A PID is an identity signal. It is not a progress signal.

The prompt that never ended

The worker was started with codex exec, but its standard input was not redirected. Instead of receiving an explicit end-of-input signal, it waited on its own prompt: “Reading additional input from stdin...”

That made the failure deceptively ordinary. The parent had a PID to inspect. pgrep could find it. Nothing had thrown an exception. A dashboard that answered only “is the process alive?” reported the same shape for a healthy worker and a worker waiting for input that would never arrive.

The output file was the uncomfortable evidence. It stayed at 39 bytes while the clock moved. The process table described existence; the byte count described work.

The useful monitoring question is therefore not “is the process present?” but “what measurable artifact changed since the last observation?” That question works across logs, generated files, test reports, and structured output. It is portable because it observes work rather than a particular tool.

There is no benefit in making the monitor sound confident. Its job is to expose the evidence needed for the next decision, including the uncomfortable case where the process exists but the work is not moving.

The rule became byte-delta

The durable rule is now written into the spawn protocol as the watchdog requirement: every worker gets a heartbeat every 120 seconds, and liveness is judged by output byte-delta, never by process existence alone.

A heartbeat that only speaks when something changes is not enough. Silence can mean a healthy long operation or a stuck process. The watchdog prints on every tick so the monitor itself remains observable. On each tick it records the process state, total bytes, and the delta from the previous sample.

Zero byte-delta for 3 consecutive ticks is STALLED. That is not a polite suggestion to wait longer. It is an instruction to investigate. A worker may be alive and still fail the liveness contract.

The backstop is a ledger

A watchdog only helps when someone remembers to arm it. That is why every spawn is also recorded with its parent task identifier in a spawn ledger. The ledger is recording-only; it does not pretend a hook can create a recurring conversational monitor after the fact.

At the next preflight, the ledger can surface three useful states: a registered process that is gone, a registered process whose log has stalled, and a live worker that was never registered. The point is not to make the hook clever. The point is to make an unwatched spawn visible before it becomes a mystery.

That distinction matters because a global hook must fail open. A broken recording hook should not block every agent or shell call on the machine. Recording and enforcement are separate responsibilities.

The mechanical fix

The specific launch fix is small: when I start a CLI worker, I redirect its stdin from /dev/null. The worker cannot wait for interactive input that the orchestration contract never intended to provide.

The operational fix is larger in mindset. I no longer ask “is it alive?” as the first question. I ask “what artifact is growing, by how many bytes, and when did it last move?” Exit status comes later. A zero exit code can still describe the wrong execution, and a non-crashed process can still be functionally dead.

The lesson stored beside this hook says that every spawn must be recorded with its parent task identifier because an unwatched spawn is invisible until it hangs. That sentence is more useful than a successful demo. It tells me what evidence must exist before I trust the next run.

The operating rule

I treat process existence as a weak signal and output movement as the stronger one. The worker gets closed stdin, a same-turn watchdog, a 120-second heartbeat, byte-delta sampling, and a ledger entry. Three unchanged samples become STALLED. The system investigates instead of narrating optimism.

That is the rule I wanted from the incident: do not confuse a process that has not died with work that is still happening.

If you run agents, do this

• Redirect non-interactive workers with < /dev/null.
• Measure output byte-delta, not only PID existence.
• Heartbeat every 120 seconds, including unchanged ticks.
• Classify 3 consecutive zero-delta ticks as STALLED.
• Record every spawn with its parent task identifier.