Hermes

The agent fleet that runs itself.

Hermes is the system behind everything else on this site: five machines, one person, running agents around the clock. The interesting part isn't the agents. It's what keeps them working while I'm asleep and something breaks.

Five machines, one operator

One machine is the brain I talk to. The other four have jobs. They share one memory and one always-on gateway, so any of them can pick up where another left off.

01 · Orchestration

The machine I talk to

It plans the work, hands pieces to the other four, and reports back. This is where I live day to day.

02 · Markets

Assets and market data

Watches positions and runs the recording and monitoring jobs that can't miss a tick.

03 · Research

Long-running research

Reads, searches, and collects data over hours, then writes what it finds back to shared memory.

04 · Security

Keeps the others honest

Audits changes before they ship and scans for anything that shouldn't go out.

Shared layer

What all five stand on

Underneath the fleet: one gateway that doesn't stay down for long, a memory committed to git every five minutes so a crash costs minutes at worst, and a watchdog that assumes anything can fail. The fifth machine builds the code and self-hosts the fleet's own files.

Staying up is the hard part

Anyone can start a process. The engineering is in noticing when it's broken — especially when it looks fine. I check three separate things, because "the process is running" answers almost none of them.

Is it alive?

launchd restarts any agent the moment it exits. Ordinary crashes heal in seconds, without me.

Is the loop alive?

A process can be running and still be frozen — stuck on one blocking call, answering nothing. A separate thread watches a five-second heartbeat. When it stops updating, the loop is dead even though the process isn't.

Can it reach the world?

Every cycle it makes a real round-trip to the outside. A machine that's "up" but can't actually send or receive is still down, and this is the only check that catches it.

Long jobs are detached from any single session, so they finish even if the thing that started them dies — and a sweeper reports any that go missing. Nothing important depends on me being awake.

One real failure

Here's the failure that taught me the difference between the first check and the second.

One morning, a file upload quietly blocked the event loop on one machine. The process stayed alive the whole time, so every process-level restart stayed asleep. Every channel went silent. It sat like that for over three hours before I noticed by hand and restarted it.

So I built the second check. This is the same failure now.

~100s

to recover on its own from a freeze that used to run silent for over three hours.

t = 0s

A blocking call freezes the loop

The process is still alive, so nothing that watches for a dead process reacts.

0 – 90s

The heartbeat goes stale

A heartbeat task stops updating its timestamp. A separate thread is watching that timestamp, not the process.

90s

The watchdog calls it

It declares the loop frozen, dumps a full traceback of every thread to disk so the exact line that blocked is captured for next time, then exits the process on purpose.

~100s

launchd brings it back

It sees the exit and relaunches the gateway. Because memory was committed minutes ago, it comes back where it left off. No one had to be awake.

What I built, and what's next

I built the system around the models that makes a one-person operation behave like something with an on-call team. Running it taught me to tell a model failure apart from an orchestration or network one. Building them is what I want to do next, and the fleet is where I'd start.

A watchdog that restarts dead agents in seconds.
An event-loop watchdog that catches a frozen-but-alive process.
Jobs that survive a crash, with a sweeper that reports orphans.
Auth that renews itself and syncs across all five machines.
One shared memory, committed to git every five minutes.
End-to-end reachability checks, so "up" means actually reachable.

Forty-odd times something has broken in production. Each time, the fix became automatic recovery — so the same thing can't fail silently twice. That's the difference between a demo and something you can leave running.