Updated 23 Oct 02:36:30 BKK by kookwatch. Alert sender (leader): kookwatch. Open problems that every layer failed to fix: 7
- research-box: history rebuild moving (one-time, paid) - no progress for 20.0 h (bad 19.0 h; 38 fix(es), last: restart hist-rh (rc 0))
- research-box: M picker learning loop ran (hourtrain, hourly) - 338.7 h old (max 4.0 h) (bad 328.3 h; no automatic fix exists)
- kookwatch: machine - disk 86% used, 5863 MB RAM free, load 7.04 (bad 29.2 h; no automatic fix exists)
- kookwatch: M runtrack snapshots freshness (Test Manager, from super) - 115.6 h old (max 75 min) (bad 114.3 h; 667 fix(es), last: restart runtrack.service (rc 0))
- kookwatch: M 10 s holder and buy/sell tape freshness (Test Manager, from super) - 100.3 h old (max 30 min) (bad 99.8 h; 582 fix(es), last: restart rhfast.service (rc 0))
- kookwatch: M Picker retrain freshness (Test Manager, from super) - 177.7 h old (max 36.0 h) (bad 141.7 h; 826 fix(es), last: restart hist-retrain.service (rc 0))
- kookwatch: M Launch catch-up (reconcile) freshness (Test Manager, from super) - 139.7 h old (max 10 min) (bad 139.6 h; 814 fix(es), last: restart reconcile.timer (rc 0))
Layer 1 = each box's own guard (twice a minute). Layer 2 = the other boxes call its entry over SSH every minute (revive guard, start dead units).
Telegram only when both fail. To stop a unit on purpose: systemctl disable --now X or touch /opt/guard/hold/X.
| firstmin-trader - local guard 20 s ago | |||
| AI pick feed reaches this box | OK | -2 s old | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 60 s old | |
| L live-change log cron job | OK | 1 jobs present | |
| LIVE bot fmmc5 (real money) | OK | running, heartbeat 2 s | |
| S stop rule: 10 losses in a row (paper/live test 2 only; Ben 8 Oct: none for fmmc5) | OK | 1 jobs present | |
| dashboard pnl.html (live) | OK | 57 s old | |
| dashboard pnl2.html | OK | 60 s old | |
| dashboard pnl2v.html | OK | 59 s old | |
| dashboard tests.json | OK | 61 s old | |
| gmgnd (live sells) running | OK | running | |
| gmgnd answers | OK | answers ping {"warmOk": 66476, "warmErr": 72, "lastWarmMs": 238} | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 37% used, 314 MB RAM free, load 5.51 | |
| outside-money card | OK | 59 s old | |
| paper test 1 fml2 | OK | running, heartbeat 0 s | |
| paper test 1b fml2v | OK | running, heartbeat 0 s | |
| safety + dashboard cron jobs | OK | 12 jobs present | |
| kookwatch - local guard 0 s ago | |||
| 60 s AI picker (feeds paper tests) | OK | running, heartbeat 1 s | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 19 min old | |
| L ROT chains.json from rot-box (Manager) | OK | 78 s old | |
| L ROT closes.json from rot-box (Manager) | OK | 31 min old | |
| L ROT map-data.json from rot-box (Manager) | OK | 16 min old | |
| L ROT rh_drill.json from rot-box (Manager) | OK | 21 min old | |
| L ROT rotation.json from rot-box (Manager) | OK | 14 min old | |
| L ROT rotmap.json from rot-box (Manager) | OK | 3.2 h old | |
| L ROT sync main<->rot-box (Manager) | OK | 21 s old | |
| L judge guard ok (PP Data) | OK | 15 s old | |
| L lane check ok (PP Data) | OK | 7 s old | |
| L lane price tracker (PP Data) | OK | 4 min old | |
| L lane stats page (PP Data) | OK | 5 min old | |
| L live-change log cron job | OK | 1 jobs present | |
| L nightly review cron job | OK | 1 jobs present | |
| L nightly review page | OK | 18.9 h old | |
| M 10 s holder and buy/sell tape freshness (Test Manager, from super) | BAD | 100.3 h old (max 30 min) | 582 fix(es): restart rhfast.service (rc 0) |
| M AI picker round 5 (paper) freshness (Test Manager, from super) | OK | 0 s old | |
| M Ben Telegram buttons (benbot) | OK | running | |
| M Ben Telegram line in cron | OK | 2 jobs present | |
| M Launch catch-up (reconcile) freshness (Test Manager, from super) | BAD | 139.7 h old (max 10 min) | 814 fix(es): restart reconcile.timer (rc 0) |
| M Live job checker (sentinel) freshness (Test Manager, from super) | OK | 5 s old | |
| M PP Feeds layer-2 guard (ppfeedguard2) | OK | running, heartbeat 13 s | |
| M PP data cron lines (PP Data) | OK | 3 jobs present | |
| M PP data liquidity table (liqinc) (PP Data) | OK | 13 min old | |
| M PP data picker-box copy (pbsync pull) (PP Data) | OK | 10 s old | |
| M PP data ponstape (PP Data) | OK | running | |
| M PP data ppexit honest-exit tracker (PP Data) | OK | 1 s old | |
| M PP data venuepick paper picker (PP Data) | OK | 0 s old | |
| M PP feed GMGN board (price-predict-rh) | OK | running, heartbeat 36 s | |
| M PP feed crewlive (crews) | OK | running, heartbeat 19 s | |
| M PP feed klines | OK | running, heartbeat 2 min | |
| M PP feed ponstape (Pons curve) | OK | running, heartbeat 5 s | |
| M PP feed rhledger | OK | running | |
| M PP feed runtrack (runners) | OK | heartbeat 115.6 h old but log live (not hung) | |
| M Picker guard (pickguard) freshness (Test Manager, from super) | OK | 15 s old | |
| M Picker retrain freshness (Test Manager, from super) | BAD | 177.7 h old (max 36.0 h) | 826 fix(es): restart hist-retrain.service (rc 0) |
| M SW engine w2coder (paper) | OK | running, heartbeat 1 s | |
| M bot watchdog freshness (Test Manager, from super) | OK | 39 s old | |
| M cleaner freshness (Test Manager, from super) | OK | 2 min old | |
| M daily miss review in cron | OK | 1 jobs present | |
| M gmgn feeds recorder freshness (Test Manager, from super) | OK | 34 s old | |
| M instant miss watch ran | OK | 75 s old | |
| M lane price tracker (lanetrack) freshness (Test Manager, from super) | OK | 4 min old | |
| M lane radar v3 (laneradar3) freshness (Test Manager, from super) | OK | 2 s old | |
| M lane radar v3 beat (PP Data) | OK | 2 s old | |
| M lane stats page (lanestats) freshness (Test Manager, from super) | OK | 5 min old | |
| M meta radar (metaradar) freshness (Test Manager, from super) | OK | 66 min old | |
| M paper test (pp-paper) freshness (Test Manager, from super) | OK | 76 s old | |
| M picker learning loop cron (hourly) | OK | 1 jobs present | |
| M ppinc hourly new pools + supply (PP Data) | OK | 10 min old | |
| M rate-limit review in cron | OK | 2 jobs present | |
| M rate-limit review ran (heartbeat) | OK | 11 min old | |
| M runtrack snapshots freshness (Test Manager, from super) | BAD | 115.6 h old (max 75 min) | 667 fix(es): restart runtrack.service (rc 0) |
| S AI picker (paper) freshness (Test Manager, from super) | OK | 1 s old | |
| S P3 mirror cross (a1pull@cross) | OK | running | |
| S P3 mirror early (a1pull@early) | OK | running | |
| S backup checker (livewatch2) freshness (Test Manager, from super) | OK | 15 s old | |
| S fast checker (livewatch) freshness (Test Manager, from super) | OK | 4 s old | |
| S fast checker livewatch | OK | running, heartbeat 4 s | |
| S live scanner rhfast (feeds Price Predict) | OK | running, heartbeat 56 s | |
| S live settings watchdog | OK | 1 jobs present | |
| S live settings watchdog ran | OK | 21 s old | |
| backup picker aipick5b stays off | OK | off (as decided) | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| machine | BAD | disk 86% used, 5863 MB RAM free, load 7.04 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 24 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| pick feed file l2picks.json | OK | 0 s old | |
| pick feed l2feed | OK | running, heartbeat 0 s | |
| published tests dashboard | OK | 18 s old | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |
| testspub cron job | OK | 1 jobs present | |
| picker-box - local guard 2 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 19 min old | |
| M lab paper books | - | on hold (/opt/guard/hold/labbooks) | |
| M shadow picker pbshadow | OK | running | |
| free RPC relay (2nd IP) | OK | running | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 33% used, 780 MB RAM free, load 0.41 | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| research-box - local guard 1 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 19 min old | |
| L kitb | OK | 38 s old | |
| L shadow grid report (Test Manager, nightly 02:30) | OK | 23.5 h old | |
| L trending feed | OK | 3 min old | |
| M KITFAST kit feed (PP Feeds) | OK | running, heartbeat 0 s | |
| M PP Feeds layer-1 guard (ppfeedguard) | OK | running, heartbeat 6 s | |
| M auditq ran | OK | 6 min old | |
| M crew tagger pretag | OK | running, heartbeat 1 s | |
| M datajob runner ran | OK | 28 s old | |
| M flowwatch ran | OK | 14 min old | |
| M ideaq ran | OK | 3 min old | |
| M killbot ran | OK | 25 min old | |
| M paper cards ran | OK | 87 s old | |
| M picker learning loop ran (hourtrain, hourly) | BAD | 338.7 h old (max 4.0 h) | |
| M v3 live feed v3live | OK | running, heartbeat 0 s | |
| M v3 mc5 paper book | OK | running, heartbeat 3 s | |
| S A1 early + cross feed (a1feed) | OK | running, heartbeat 0 s | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| history rebuild moving (one-time, paid) | BAD | no progress for 20.0 h | 38 fix(es): restart hist-rh (rc 0) |
| machine | OK | disk 89% used, 5382 MB RAM free, load 7.80 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 28 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| rebuild moves to free when done (cron) | OK | 1 jobs present | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |