# Guard consolidation: one framework, the guard mesh (Test Manager, 8 Oct 2026)

**Goal (delete audit item 4):**
- About 25 separate guard scripts become one framework.
- Each owner moves its checks into `/opt/guard/checks.json` on its box **by 15 Oct**.
- Then the old script goes off. Reversible, and the file is kept.
- Live status: http://159.223.88.168/hx-c9ois/fw/guards.html

## How the mesh works
- **Layer 1:** `/opt/guard/guard.py` on every box. It runs twice a minute (cron at :00, systemd timer at :30), checks, fixes with backoff, and writes `status.json`.
- **Layer 2:** `/opt/guard/entry.sh`. Every other box calls it every minute over SSH, with a key that can run nothing else.
  - It revives a silent local guard and its triggers.
  - It starts the units listed in `/opt/guard/units.txt` when they are down.
- **Telegram:** only the leader box sends (kookwatch, then picker-box, then research-box).
  - It sends only for a check that is still bad after both layers, 5+ minutes or the check's `escalate_after`.
  - DOWN once, BACK once, a reminder every 6 h.
- **Stopping something on purpose:** `systemctl disable --now X` or `touch /opt/guard/hold/X`. The guards leave it alone.

## Adding your check
Add one JSON object to `checks` in `/opt/guard/checks.json`, then run `python3 /opt/guard/guard.py --dry`. It prints every check and fixes nothing.

Rules for the check:
- Start the name with your tier: S (money), M (data the tests read), L (nice to have).
- Add an owner in the name, e.g. `"M crew collector (PP Feeds)"`.

| type | checks | fix | fields |
|---|---|---|---|
| `unit` | systemd unit running; optional heartbeat; crash loop (5 restarts in 15 min, escalates with no fix) | start / restart with backoff | `unit`, `hb: {file, kind: epoch\|mtime\|json:<key>, max}`, `log: {file, max}` (heartbeat old but log live = not hung), `min_up`, `backoff`, `backoff_file` (shared with your own watchdog), `env: {VAR: value}` |
| `fresh` | file age (mtime, epoch inside, or a json key) | `fix` shell command, or restart `unit` | `path`, `kind`, `max`, `fix`, `fix_timeout`, `unit`, `backoff` |
| `url_fresh` | a published JSON has a fresh `t` | none (escalates) | `url`, `key`, `max` |
| `sock_ping` | unix socket answers `{"op":"ping"}` (2 misses) | restart unit | `unit`, `path` |
| `cron_lines` | cron jobs still present | re-adds the exact line | `lines: [[match, full line], ...]` |
| `json_value` | a config value is one of the allowed values | sets it back | `path`, `key`, `allowed`, `set` |
| `py_ok` | a Python file compiles | restores the known-good copy | `path`, `good` |
| `must_be_off` | a unit Ben decided stays off | disable --now | `unit` |
| `progress` | a job's output keeps moving, until a done marker appears | restart unit | `glob`, `max`, `unit`, `until: [file, text]` |
| `system` | disk / memory | none | `disk_max`, `mem_min` |
| `paid_env`, `paid_key_scan` | paid RPC key used only by the allowed processes / found only in the opt-in files | none (escalates) | see the checks on kookwatch |

**Other fields:**
- `escalate_after` (seconds) on any check sets how long it may stay bad before Telegram. Default 300.
- A unit that must also be **started by the remote layer** when it is down goes into `/opt/guard/units.txt` (one unit per line).

**Never in a check:**
- Anything that switches live on or raises a size (D-030).
- Restarting the live bot except through its own hung rule. The `unit` check with `log` + `backoff_file` already does that.

## Old guards to retire once their checks are in (owner moves them)
super, watchdog-bots, unitfix, guardkit, alertgate, pickguard, sentinel, pwguard, judgeguard, pdguard, w2guard, dcguard, diskguard, walguard, chainfix, rotfix, giver_fix, runtrack-guard, three stale.py, fresh.py, livewatch2, ppfeedguard, ppfeedguard2. livewatch stays until its checks are in too.

**Order:**
1. Owner adds the checks and runs `--dry`. All OK, or BAD for a real reason.
2. 24 h on both side by side.
3. Owner comments out the old cron line / disables the old unit, marked "RETIRED <date> (in guard mesh)".

**What the Test Manager does:**
- Reviews each move.
- Keeps the shared code (`guard.py`, `entry.sh`, `mesh.py`).
- `install.py` is never run again (it would overwrite checks.json); code changes are copied in by hand and go into Git.