---
name: alert-bot-fleet-9
description: Alert bot fleet log, part 9, from 15 Sep 2026. The PRM Watch message storm and the fleet-wide disk-full event behind it - why a full disk turns into Telegram spam, the unbounded sqlite WAL that is the prime suspect, and the three fixes (non-fatal state writes, ONLINE card cooldown, disk alerting in the watchdog). Read before touching prmwatch, the watchdog, or anything that writes to the trench databases.
sources: [cowork]
aliases: [alert bots part 9, disk full, prm spam, wal checkpoint, disk alert, errno 28]
---
Continues [[alert-bot-fleet-8]]. Droplet access in [[data-access]]. PRM itself in [[premium-prm]].

# 15 Sep 2026. "why is this sending the same message over and over"

## What Ben saw
- [stated] Ben pasted a Telegram screenshot: three identical **"✅ PRM WATCH ONLINE"** cards at 08:36, 08:37, 08:37 Bangkok (01:36, 01:37 UTC), same wallet count, same fleet block.
- **They were not duplicate sends. They were separate BOOTS.** `prmwatch.service` had `NRestarts=68`.

## The chain, and every link of it is worth keeping
1. **01:08:29 UTC the droplet ran out of disk.** First victim in the journal: `python3[1006818]: [robinhood] loop OperationalError('database or disk is full')`. 18,211 "No space left on device" lines followed.
2. **The whole fleet crash-looped, not one bot.** Restart counters afterwards: rbt3k 99, klines 99, ubermwatch 97, crypticalert 92, chjokkawatch 92, alfiewatch 92, ansem-rh 91, pixelalert 90, rhtracker-base 88, dougwatch 87, neetwatch 85, rhtracker 83, ghostwatch 83, ghostsol 83, dinerodom 83, grailwatch 79, runwatch 75, dougsol 70, prmwatch 68, crashbot 55.
3. **prmwatch's startup order is what turned a crash loop into a message storm:** `send(ONLINE card)` then `save(STATE, st)` on the next line. `save()` raised OSError 28 and killed the process, so every restart sent another card and then died again. **The card went out BEFORE the thing that crashed.**
4. **It fixed itself at ~01:37.** Roughly 14GB appeared and vanished with nothing left on disk to blame.

## PRIME SUSPECT for the 14GB: a sqlite WAL that could not checkpoint
Circumstantial, and said so rather than asserted, because the file was gone by the time anyone looked. But it fits every observation and nothing else does:
- Both trench databases are **WAL mode, `wal_autocheckpoint=1000` pages = 4MB**.
- **WAL only reclaims space when a checkpoint completes, and a checkpoint is blocked while ANY reader holds the database.** `/opt/trench/trench.db` (1.41GB) had six processes on it continuously: `trench.py` x2, `newborn.py`, `casig.py`, `xwatch.py`, plus the `paper.sh` cron every 5 minutes.
- **It was already losing that race at rest: `trench.db-wal` measured 27.6MB against a 4MB threshold.**
- A blocked WAL grows without bound; when the readers died on the full disk the last connection closed, SQLite checkpointed and truncated, and all the space came back at once. **Fills fast, frees instantly, leaves no evidence** is the WAL signature.
- Not the cause: bot logs (biggest is casig.log at 3.6MB, `/etc/logrotate.d/bots` works), the journal (64MB), `/var/log` (97MB total). Swapfiles are 1GB (12 Sep) + 3GB (13 Sep) and predate this.

## THE THREE FIXES, deployed and verified 15 Sep 01:45 UTC
Patch `/root/fix_disk_spam.py` (written on the Mac at `~/kookwatch/`, scp'd, run on the droplet). 7 self-checks, 0 failed. Backups `prmwatch.py.bak-20260915-*`, `watchdog.py.bak-20260915-*`, `/root/crontab.bak-20260915-*`.

1. **`save()` in prmwatch is NEVER FATAL.** Catches, logs `STATE WRITE FAILED`, cleans up the tmp file, returns False. **A watcher that cannot write its state should keep watching and say so, not die.** This alone ends the storm, because the crash is what caused the restart.
2. **ONLINE card cooldown, `ONLINE_COOLDOWN_S = 1800`.** At most one card per 30 minutes. Suppressed boots increment `st["boots_since_card"]`, and the next card that does go out carries *"restarted N times since the last card - something was wrong, check the droplet."* **A restart storm now produces one informative card instead of 68 identical ones.** Belt and braces behind fix 1.
3. **The watchdog now ALERTS on disk.** `check_disk()` + `DISK_WARN_PCT 85` / `DISK_CLEAR_PCT 78`, one card on the way down, one on the way back, never repeats. Wired into the main loop beside `check_credits` and `check_blocked_bots`.
   - **It had been printing disk in the daily heartbeat since it was built and never once judged it.** Exactly the 14 Sep rule in a new costume: a service is not healthy because it is running, and a number you print but never compare is not a check.
   - The card names the **biggest write-ahead logs over 50MB** (`_wal_hogs()`), so the next occurrence is diagnosable instead of guessed at. That is the whole point: this incident could only be reasoned about, not proven.
4. **`/opt/common/wal_checkpoint.py` + cron `*/5`.** Runs `PRAGMA wal_checkpoint(TRUNCATE)` on both trench databases with a 20s busy timeout. **TRUNCATE is the only mode that shrinks the file back to zero.** First run: `/opt/trench/trench.db 27.6MB -> 0.0MB`. `/opt/trench-sol/trench.db` returned `busy=1` (readers active) and will be caught next run, which is the expected normal answer and not an error.
   - **`PRAGMA journal_size_limit` is PER CONNECTION and is not stored in the database file**, so setting it from a one-off script does nothing for the bots. The periodic external TRUNCATE checkpoint is the fix that actually works without touching every bot.

## Verification, without a false alarm and without filling the disk
`/root/verify_disk.py` imports watchdog.py by spec, stubs `tg()` and `log()`, moves the thresholds either side of the live value and asserts: one alert on the way down, none on a repeat call, one all-clear on the way back, none on a repeat, and `disk_alerted` set then cleared. **PASS.** Same sandbox method as the rugscan selftest: exercise the real function with the send stubbed, never a reimplementation of the logic.
- prmwatch and watchdog-bots restarted, both active, `NRestarts=0`, `state.json` now carries `online_card_ts` and `boots_since_card`.

## Rules this produces
- **Send the card AFTER the risky work, not before.** prmwatch announced itself and then died on the next line. Any startup or success notice placed before the operation it describes will be sent once per failure.
- **Any repeated identical alert is a restart counter until proven otherwise.** `systemctl show <unit> -p NRestarts` is the first command, before reading a single line of the bot's own code.
- **One bot spamming is a fleet symptom.** Check `NRestarts` across every running unit before believing the problem is local; here 20 services were looping and only the one with a startup card was visible to Ben.
- **A full disk is silent.** Nothing in the fleet watched it, `systemctl is-active` stayed green throughout, and the only reason anyone found out is that one bot happened to announce every boot.
- **`du` after the fact cannot find a file that has been released.** When space fills and frees with nothing left behind, suspect an open-but-deleted fd or an unbounded WAL, and instrument for the next time rather than keep digging.

## THE REAL FRAGILITY, flagged and NOT fixed without asking
**The droplet is a 1GB RAM box running 34 services, 23 cron jobs and 2.4GB of live sqlite.** `free -h` at the time of writing: 961Mi total, 95Mi free, and **4GB of swapfile has been bolted on across two days** (1GB on 12 Sep, 3GB on 13 Sep) which is itself the symptom of a box that ran out of memory twice already. Low memory also makes sqlite spill to disk more, which feeds the exact failure above. **The honest recommendation is to resize the droplet rather than keep adding swap and checkpoint crons.** Ben's call; nothing done.

# 15 Sep 2026, SECOND PASS. DELTA storm, and the first fix was the wrong SHAPE

- [stated] Ben pasted a second screenshot minutes later: five identical **"✅ DELTA alert is live"** cards at 08:28-08:30 Bangkok (01:28-01:30 UTC), `RH Wallet Alerts` bot. Same disk window, different bot. `deltaalert.service` **NRestarts=23**, dying at `deltaalert.py:43 save_state -> OSError 28` immediately after the boot card, byte-for-byte the prmwatch failure.

## THE MISS, owned
I grepped `"WATCH ONLINE|ONLINE$|✅.*ONLINE"`, got one hit, and told Ben **"only prmwatch sends an ONLINE card"**. deltaalert says *"is live"*. **Seven bots send a boot card:** prmwatch, deltaalert, ansem-tracker, basewatch, crash-ton, dougfomo, runwatch.
- **The rule: never grep for one PHRASING of a behaviour and report the count as the behaviour's frequency.** The right query was the shape (a send before the main loop), not the words. Same family as the 14 Sep GMGN `launchpad_platform` miss: varying the knobs you already know about is not looking.

## THE FINDING THAT MADE THE FIRST FIX WRONG
Four of those seven **already had a guard, and every guard lived on the disk that was failing**:

| bot | its guard | stored in |
|---|---|---|
| basewatch | `booted_today()` | state file |
| runwatch | `meta_get(c,"announced")` | sqlite |
| dougfomo | `st["primed"]` | state file |
| prmwatch | `online_card_ts` | state file, **my own pass-1 fix** |

**A full disk wipes out every one of them at the same moment, which is precisely the event during which you least want seven bots shouting.** Pass 1 added a fourth guard to the same broken pile. Correct diagnosis, correct behaviour, wrong storage.

## FIX 1, and it is the mechanical one: systemd StartLimit on every fleet unit
`/etc/systemd/system/<unit>.d/ratelimit.conf` with **`StartLimitIntervalSec=600`, `StartLimitBurst=5`**, applied to **46 fleet units** (every unit carrying `Restart=always` or `on-failure`).
- **68 cards becomes at most 5, then systemd stops trying, the unit sits failed, and watchdog-bots reports it DOWN** — which is the message actually worth getting. It works for any bot, for any crash reason, including bots nobody has read.
- **`watchdog-bots.service` is deliberately excluded.** The thing that reports the others must never rate-limit itself out.
- **TRAP I CAUGHT IN MY OWN PATCH: the first run also hit `syslog.service`, `do-agent`, `droplet-agent` and two `dbus-org.freedesktop.*` units**, because the filter was "has Restart=", not "is a fleet bot". **Rate-limiting syslog would have stopped logging on the next transient failure.** Removed, `daemon-reload`, 46 drop-ins left. **A fleet-wide loop over `/etc/systemd/system/*.service` must exclude system units explicitly, never by inference.**

## FIX 2: /opt/common/bootcard.py, the guard moved to RAM
- `allow(name, cooldown=1800) -> (send_it, boots_suppressed)` and `suffix(n)`. State is a marker file under **`/run/bootcard/<name>`**.
- **`/run` is tmpfs. It is RAM, it does not fill when / fills, and it clears on reboot, which is exactly the right lifetime for "have I announced myself recently".** 97MB, 2% used.
- Keeps the ORIGINAL send time via `os.utime` when suppressing, so a restart storm cannot walk the cooldown forward one boot at a time.
- **Fails OPEN** (sends the card) if /run is unusable: a silent bot is worse than a chatty one, and /run failing is not the scenario this exists for.
- Wired into **deltaalert** and **prmwatch** (whose pass-1 disk-backed cooldown was replaced by it). The other five boot-card bots are covered by Fix 1 and are not yet on the shared guard.
- `save_state()` in deltaalert also made non-fatal, same treatment as prmwatch's `save()`.

## Verified live 15 Sep 01:55 UTC
- 11 self-checks, 0 failed. Backups `deltaalert.py.bak-20260915-*`, `prmwatch.py.bak-20260915-*`. Patch `/root/fix_bootspam.py` (written on the Mac at `~/kookwatch/`, scp'd).
- Restarted each bot a SECOND time on purpose: `deltaalert` logged *"boot card suppressed, 1 restarts since the last one"*, `prmwatch` logged *"ONLINE card suppressed, 1 restarts since the last one"*, `/run/bootcard/deltaalert` contains `1`. **No card sent.**
- `systemctl show` confirms `StartLimitIntervalUSec=10min`, `StartLimitBurst=5` on both. deltaalert, prmwatch, watchdog-bots all active, NRestarts back to 0.

## Rules this produces
- **A guard against a failure must not be stored in the thing that fails.** Anti-spam state for a disk-full event belongs in tmpfs; a retry counter for a database outage does not belong in that database. Ask what the guard depends on before choosing where it lives.
- **Prefer the fix that covers the bots you have not read.** One systemd drop-in per unit beat seven code patches, needed no knowledge of each bot, and protects against crash causes not yet seen.
- **Grep for the SHAPE, not the wording**, when counting how many places a behaviour occurs.
- **Still not done:** ansem-tracker, basewatch, crash-ton, dougfomo and runwatch send boot cards guarded only by disk-backed state. Fix 1 caps them at 5; moving them onto `bootcard.allow()` would cap them at 1.

# 16 Sep 2026. THE STALE-WALLET BLIND SPOT, and the check that closes it

- [stated] Ben: "I haven't received any OX ghost alerts recently", then "make sure other wallet alerts don't have the same issue. set the alert to the new wallet as well."

## THE FAULT, and it is a whole new class
**`ghostwatch.service` was green, at the chain tip, zero restarts for five days, and completely useless, because the wallet it watched had been drained on 11 Sep.** It missed three live subscriber-only calls (PRM, CEREBRO, INVEST). The real wallet is `0xa2f64ede5fb760498bc500306b3833c479e34211`; full evidence in [[ghostdotenv-wallet]].
- **THE RULE, and it now sits beside "a service is not healthy because it is running": A WALLET BOT IS NOT HEALTHY BECAUSE IT IS RUNNING, IT IS HEALTHY BECAUSE THE WALLET IT WATCHES STILL TRADES.** Nothing in systemd, in the log, in the socket check or in the watchdog could see this. A drained wallet and a working bot are byte-identical from the outside.
- Note the two-service trap that hid it: **`ghostsol.service` and `ghostwatch.service` both exist and both call themselves GHOST ALERT.** The Solana one was healthy and visible in Telegram, so the silence looked like "the bot is fine, he is just quiet". **When a bot looks quiet, list every unit matching the name before concluding anything.**

## FLEET AUDIT, 16 Sep. Only ONE other bot had the same fault.
Scanned all 79 watched wallets (60 Robinhood, 16 Solana, 3 TON) for last on-chain movement.
- **`dougwatch` / `0x5795f2390e9e4c1c2e4664a9271339cacca5b658` ("DOUG fomo-RH"): nothing in 14+ days, nonce 0, 0 ETH. Dormant.** Doug's other wallet `0xd42fcea321b8b0093fd78e76182dd92f588c0a9b` is live (1.6d).
- **⚠ I DISABLED IT AND BEN CORRECTLY OVERRULED ME. RE-ENABLED 16 Sep, and the reasoning is now standing policy.** "Leave the old wallet on. Why not? If he uses it again, then we have an alert on it. It'll be quiet anyways, given that he's not using it."
  - **He is right and the receipt is in the code: `tracker.py` builds `pads = [pad_addr(a) for a in wallets]` and passes the whole ARRAY into just two `eth_getLogs` calls per scan. Twenty wallets cost exactly what one costs.** So a dormant address adds ZERO RPC load and buys a free early signal the day the trader funds it again.
  - **STANDING RULE: NEVER REMOVE A DORMANT WALLET FROM A WATCHER. Relabel it "(dormant)", write why in the note, and leave it enabled.** The only thing a dead wallet must not do is nag, and that is the staleness filter's job, not the config's. Applied to both `0x5c54382981bdf5276912a68c1d1a15a5354cc6a0` (ghost old) and `0x5795f2390e9e4c1c2e4664a9271339cacca5b658` (doug fomo).
  - **Check the per-wallet cost model before pruning anything.** If a watcher batches by topic array, the wallet count is free; if it loops one call per wallet, pruning is a real saving. Read the scan function first.
  - Verified after re-enabling: both back on, both bots active, and NEITHER is in the staleness check's scope (the "(dormant)" label is on the skip list, and the transition-based logic would never have fired on an already-quiet wallet anyway). Backups `config.json.bak-20260916-*` in both `/opt/ghostwatch` and `/opt/dougwatch`.
- Everything else healthy. The other 31 "quiet" wallets are quiet BY DESIGN: the 38-wallet prmwatch overhang fleet (whose entire thesis is that it never moves), Crash's dormant wallets and TON holder wallets, ansem "payments (dormant)", the rbt3k eliminated candidates.
- **Method, cheap and reusable: ONE batched `eth_getLogs` per block chunk for the WHOLE fleet, using an ARRAY of padded addresses in topics[1] and again in topics[2]. Topic arrays are OR. 60 wallets cost the same as one.** 8 days at 250k-block chunks is ~48 calls. Solana side is `getSignaturesForAddress(limit=1)` and reading `blockTime`.

## THE MECHANICAL FIX: `/opt/watchdog/stale_wallets.py`, wired into watchdog-bots
Runs every 12h in a **daemon thread** (the scan takes 6-10 minutes and must not block the loop that restarts downed bots). Alerts ONCE when a live target goes quiet past 7 days, ONCE when it comes back.
- **Two filters make it worth reading, and getting them wrong is what I did first:**
  1. `SKIP_SERVICES = {"prmwatch", "crash-ton"}` plus label-word exclusion (`dormant, holder, whale, funder, treasury, relay, conduit, bridge landing, eliminated, receive-only, disabled, not a wallet, exchange`). Cohort and trip-wire wallets are supposed to be silent.
  2. **It only alerts on a wallet it has SEEN trading before** (`state["stale_ever_active"]`). A wallet already quiet when the check was installed is a deliberate trip-wire, not a fault, and stays silent for ever. This is the filter that does the real work, because it needs no label discipline.
- Scope went **79 wallets -> 28** after filtering; first clean run logged `stale check: 27 wallets, 0 quiet-flagged, 0 new, 0 recovered` and seeded 23 live targets. No message sent.

## ⚠ MY OWN MISTAKE, AND IT IS THE SAME ONE THIS FILE ALREADY WARNS ABOUT
**I deployed version 1 without the filters and it immediately sent Ben a 32-wallet Telegram alarm** listing the entire PRM overhang fleet and Crash's dormant wallets. Logged at 12:17:04 UTC. I cleared the bad flags, reset the state, sent one short correction card, then shipped the filtered version.
- **The rule I broke is three sections up in this same file: verify with the send stubbed before letting a new check reach Telegram.** `/root/verify_disk.py` exists precisely because of this, and I did not write the equivalent for this one. **A new alert path is guilty until proven quiet: run it once with `tg` stubbed to `print`, read what it WOULD have sent, and only then wire it to the real bot.**
- Second, smaller: the first patch attempt computed indentation from the anchor line and produced a syntax error in `watchdog.py`. Caught by `ast.parse`, restored from the backup taken in the same command. **Always `cp` the file and `ast.parse` the result inside the same one-liner; never hand-compute indentation, match the exact existing line instead.**
- Shell trap, cost three attempts: **heredocs through `ssh` mangle `\x27` and triple quotes.** For anything with nested quoting, write the complete file locally and `scp` it. That worked first time, every time.

## State of the two ghost services now
- `ghostwatch`: watching `0xa2f64ede5fb760498bc500306b3833c479e34211` (enabled), old wallet kept as a disabled row with its history. Active, 0 restarts, ~6s behind the tip. Backup `config.json.bak-20260916-*`.
- `ghostsol`: unchanged, both Solana wallets live.
- The `HTTP Error 429` traceback in `ghostwatch.log` is the HANDLED path (`Rpc.call` backs off 5 times, the run loop catches and continues). **It is noise. Do not chase it again.**

# 16 Sep 2026. NEW BOT: SOLANA UNICORN FINDER (`sunicorn`)

- [stated] Ben: "zV18RQeme9fDZUbCGf7skSWNz6Vphs5j3PQUHjHdcgi build a separate bot for this wallet and call it Solana Unicorn Finder", then "The bot should trigger in a new Telegram chat. Control my desktop to get that done."
- **Own bot `@bk_sunicorn_bot` ("Solana Unicorn Finder"), created in BotFather by driving Telegram Desktop, and its OWN chat. Deliberately not the shared RH Wallet Alerts thread.**
- Watches `zV18RQeme9fDZUbCGf7skSWNz6Vphs5j3PQUHjHdcgi` (the wallet in [[copytrade-targets-2]]: ~$29k into STONK on 4 Aug at $0.0019, now 13,410,707 STONK; ALLINU 9.5h after launch). **Only 22 lifetime buys, so every buy is a signal — `min_usd` is 200, not 1000.**
- Built as a clone of `/opt/dougsol/dougsol.py` (a mature 35KB Solana watcher), `sed`-rebranded, into `/opt/sunicorn/sunicorn.py`. Unit `sunicorn.service`, log `/opt/sunicorn/sunicorn.log`, config `/opt/sunicorn/config.json`, plus `settoken.py` and `setchat.py` helpers. `ratelimit.conf` drop-in applied (StartLimitIntervalSec=600, StartLimitBurst=5), `py_compile` OK, registered in `~/botctl/registry.json` as bot 30.
- Alerts: buys, sells, outbound transfers, SOL funding. `alert_transfers_in false` (kills airdrop spray), `alert_unknown false`. 12s poll on Helius.

## ⚠ THE BLOCKER, and it is a permanent constraint on desktop control
**Computer control in background mode CANNOT touch the system clipboard and CANNOT open a context menu.** `Cmd+C`, the Edit > Copy menu item and right-click all refuse with "would touch the system clipboard" / "would bring the app to the front". So **a Telegram bot token can never be lifted off the screen by Claude; Ben must tap it himself**, and then `pbpaste | ssh ... > token.txt` + `settoken.py` moves it without it ever being read. Plan any future BotFather work around that.
- Telegram Desktop quirks found: **plain `Return` inserts a newline, `Cmd+Return` sends.** Clicking the Send button via AXPress did nothing. `app_type` sets AXValue rather than typing, which works for the composer.
- **Confirm a new bot exists WITHOUT reading its token: `curl -s -A Mozilla/5.0 https://t.me/<username>` and read `og:title`.** It returned "Solana Unicorn Finder", which proved the username was accepted with zero secret exposure.
- Start was already pressed in the new chat, so `setchat.py` (getUpdates) resolves the chat id the moment the token lands.

## LIVE 16 Sep 2026 13:02 UTC (20:02 Bangkok)
- Ben supplied the token himself. `settoken.py` wrote it and deleted `token.txt`; `setchat.py` resolved **chat_id 357464289**. Note that is Ben's own user id, the SAME number every bot uses — **a "separate chat" in Telegram comes from a separate BOT, not a separate chat id.**
- `sunicorn.service` enabled and active, NRestarts=0, log: `baseline for UNICORN: 60 tokens, 0.1130 SOL`. Boot card landed in its own thread: *"Now watching 1 wallets: UNICORN ... Thresholds: min_usd 200, min_sol_move 5.0"*.
- **`/start` alone did NOT show up in `getUpdates`; a second plain message ("ping") did.** If setchat returns `{}`, send any ordinary message to the bot and re-run rather than assuming the token is wrong.

# 16 Sep 2026, EVENING. "kookwatch was down / crypticalert was down" — TWO UNRELATED FAULTS

- [stated] Ben pasted the watchdog cards: crypticalert recovered at 21:34 and 22:46 Bangkok, kookwatch "not active (failed), Restart: FAILED, needs a look" at 22:45.

## 1. kookwatch: the TELEGRAM ACCOUNT SESSION WAS REVOKED. Only Ben can fix it.
- Unit exits 1 after ~9s, six times, then hits the StartLimit and sits failed. Journal shows only the restart counter because the app never gets far enough to log.
- **Running it by hand gives the answer in one line: `Please enter your phone (or bot token):`** Telethon loaded `TG_SESSION` from `/opt/kookwatch/.env`, Telegram rejected it as unauthorized, so it fell back to the interactive login, got EOF under systemd and died. `TG_SESSION` is still present in `.env`, it is just dead.
- **Last healthy line in `kookwatch.log` is 13 Sep 16:58 "listening…". It has been blind for three days.** There is no `.session` file anywhere on the box; this bot is StringSession-only.
- **Fix, and it needs Ben's phone and login code, which Claude must never handle:** run `python3 setup_session.py` (works on the Mac, needs `TG_API_ID`/`TG_API_HASH` from my.telegram.org), it prints a new `TG_SESSION=` line, that replaces the old line in `/opt/kookwatch/.env` on the DROPLET, then `systemctl reset-failed kookwatch && systemctl restart kookwatch`.
- **The likely cause is someone ending the session in Telegram > Settings > Devices.** A userbot session dies silently and looks identical to a crash loop from the outside. **RULE: for any Telethon/userbot service, run it by hand before reading any code — an auth prompt IS the diagnosis.**

## 2. crypticalert: ITS OWN FALLBACK RPC WAS POISONING ITS RETRIES
- crypticalert was the only RH bot with two entries in `rpc_urls`: the real one plus `https://robinhood-rpc.publicnode.com`.
- **Tested directly: publicnode answers `eth_blockNumber` in 0.11s but returns HTTP 403 on EVERY `eth_getLogs`, at spans of 100, 2,000 and 50,000 blocks, with a browser UA and with a python UA. It simply does not serve getLogs.**
- `Rpc.call` advances `self.i` on every failure and **treats 403 exactly like 429** ("Treat all throttle signals the same", added after an earlier incident). So on a normal burst, roughly half of its five retries went to an endpoint that can never answer, and each one triggered the hard 2s-doubling backoff up to 8s. **That is why crypticalert flapped twice in an hour while fourteen other bots on the same RPC rode the same burst out.**
- **Fixed: publicnode removed from `/opt/crypticalert/config.json`, reason written into a `_rpc_note` field, backup `config.json.bak-20260916-*`, service restarted, active, no new 429 since.**
- **RULE: never add a fallback endpoint without testing it on the exact METHOD the bot depends on.** A health check that only calls `eth_blockNumber` will bless an endpoint that is useless for the real work, and a failover that fails is worse than no failover.

## Fleet RPC picture, measured rather than assumed
- **15 bots share `https://rpc.mainnet.chain.robinhood.com`.** Poll intervals: 2s (dougwatch, ubermwatch, ghostwatch, chjokkawatch, neetwatch), 3s (earlybirds, pixelalert, alfiewatch, grailwatch), 4s (dinerodom, rbt3k), 5s (ansem-rh, crypticalert, rh-tracker), 20s (prmwatch). Roughly 10 requests/second sustained.
- **429s in the last 30 minutes across the WHOLE fleet: 26 lines, thinly spread (crash-ton 8, crypticalert 4, sunicorn 3, dougsol 3, casig 2, rbt3k 2, ubermwatch 2, alfiewatch 2). That is normal handled noise, not saturation.** Checked before changing any poll interval, and nothing was changed.
- The primary RPC caps `eth_getLogs` at 10,000 matched logs per query (`-32000 logs matched by query exceeds limit of 10000`), which is why `max_block_span` matters.
- Disk 76% (18G of 24G), memory 961Mi total with 169Mi available. The 1GB-RAM fragility flagged on 15 Sep is unchanged.
- **The structural improvement still not built: ONE shared scanner that pulls Transfer logs once per tick with a batched topic array and fans out to per-bot filters, replacing 15 independent pollers. Would cut RH RPC load by roughly 80% with no loss of latency.** Proposed to Ben, not built.

# 16 Sep 2026. SECRET SWEEP ACROSS THE FLEET

- [stated] Ben: "Make sure phone number, login, password, and all that stuff are not visible going forward. Clean that all up." Standing rule now also in [[preferences]].

## What the audit found
- **71 files on the droplet held a credential. 68 of them were mode 644, world and group readable.** Every live bot config, every dated `config.json.bak-*`, plus `/opt/kookwatch/.env`. Single-user root box, so not an external breach, but it means any `cat` of a config puts a live token on screen.
- **Nine forgotten scratch copies in `/tmp` on the droplet held real bot tokens**, some from August: `/tmp/deploy_crypticalert.sh`, `/tmp/mastalert_build.sh`, `/tmp/cb_payload.json`, `/tmp/toks.txt`, and extracted config trees under `/tmp/ub/`, `/tmp/mt/`, `/tmp/rwx/`, `/tmp/coinbench/`, plus two `tmp*/watch.json`. **A build script is a credential file the moment it carries a token, and nobody ever deletes /tmp.**
- Mac was clean apart from the ssh key (600, needed) and one orphan: **`~/kookwatch/ghost_bot_token.txt`, mode 600, holding a live 46-character bot token that matches NOTHING on the droplet** (checked every `config.json` and every `.env` by sha256 prefix). Left in place rather than deleted, because a credential with no known owner should be revoked in BotFather, not silently destroyed. **Flagged to Ben.**
- Mac shell history: zero matches in `~/.zsh_history` and every `~/.zsh_sessions/*.history`. Desktop Commander spawns non-interactive shells, so nothing it runs is recorded.

## What was done
- **67 files chmod 600** in one pass (scan for secret patterns, chmod only the matches). The 11 configs left at 644 were verified to hold no secret. Final verify: 70 of 70 secret-bearing files at 600, zero exposed.
- **Every /tmp copy deleted, on both machines**, then re-scanned `/tmp /root /var/tmp /home`: clean.
- **NEW TOOL `/opt/common/showcfg.py` (755).** `python3 /opt/common/showcfg.py <file>` prints any config.json or .env with every secret fingerprinted as `8980...R_c (len 46)`. Redacts by KEY name (token, secret, password, api_key, api_hash, api_id, session, private, seed, mnemonic, cookie, auth) and by VALUE shape (Telegram token, Helius `api-key=`, `sk-`). Handles both JSON and dotenv. **Use it instead of `cat` on anything that might hold a credential.**
- `/opt/sunicorn/token.txt` was already gone (settoken.py deletes it). `/tmp/tk.txt` on the Mac gone.

## The rule this produces
- **A backup is an exposure.** `cp config.json config.json.bak-$(date)` doubles the number of files holding a live token, and nobody ever chmods a backup. Either `chmod 600` in the same command, or strip the token before saving the backup.
- **Never `cat` a config to read one field.** It is a habit that prints the token every time. `showcfg.py` exists so there is no reason to.
- **Delete the scratch file in the command that creates it**, not "later".