---
name: alert-bot-fleet-7
description: Alert bot fleet log, part 7, from 13 Sep 2026. The evhist seeding fix (bots that count from their own boot, not from the chain) and the PRM Watch burn-rank correction. Read before touching /opt/common/evhist.py or any recurring-event card.
sources: [cowork]
aliases: [evhist, event history, PRM Watch, prmwatch, burn rank, seed]
---
Continues [[alert-bot-fleet-6]]. Fleet inventory and droplet access live in [[data-access]]. PRM itself is in [[premium-prm]].

# 13 Sep 2026. "this is the first PRM burn?" and the uptime-meter bug

## What happened
- [stated] Ben pasted a PRM Watch card: "PRM FLEET BURNED, 414,120 PRM ($3,196) burned, **#1 first one on record**, 414,120 PRM in total".
- It was **burn number 10**. Ten transfers to `0x000000000000000000000000000000000000dead`, 15,516,912 PRM, 1.552% of supply, all from `0x07ec901873CFF7AD597B9b17B99f17eF04A8aF75`. The ledger is in [[premium-prm]].
- `/opt/common/evhist.py` (written 13 Sep, one day earlier, for exactly the opposite complaint) counted events from the moment the bot first saw one. **A counter that starts when the process starts is not a counter, it is an uptime meter.**

## The rule this produces
- **Never let a card claim a rank the bot has not earned from chain history.** "#1" and "first one on record" are claims about the world. "First since I booted" is a claim about the process. They must not share wording.
- Same error class as the linkscan `alerted` flag (see [[alert-bot-fleet-4]]): **a state field is evidence about the bot, never about reality.** Before printing a superlative, ask which of those two the number describes.
- Generalisation: any alert word that ranks or counts (first, biggest, most, record, never before) needs a backfill from the source of truth before it is allowed to ship.

## The fix, deployed 13 Sep
- `/opt/common/evhist.py` (backup `evhist.py.bak-20260913-*`) gained:
  - `seed(state, key, events, replace=False)` which installs prior events read off the chain, deduped inside a 2 second / 0.1% window, and marks the key under `state["_evseed"]`.
  - `seeded(state, key)`.
  - `since(gap, is_seeded=True)`: unseeded first event now reads **"first one this watcher has seen"**, never "first one on record".
  - `histline()`: unseeded keys print **"#N seen"** and **"N tokens seen by this watcher"** instead of "#N" and "in total".
- So the honest default is now the degraded one. A bot that was never seeded cannot lie, it can only be vague.
- `/root/prmseed.py` on the droplet replays the whole PRM history against the 38 fleet wallets and seeds every key (burn, sell, buy, consolidation, transfer_out, in, auction_bid, auction_refund). Dust floor 1,000 PRM. Reusable pattern for any cohort bot.

## The bigger bug the replay exposed: a size threshold cannot identify a contract
- `classify()` in prmwatch called something a venue only if `code_size > 100`. **An ERC-1967 proxy is 87 bytes.**
- **`0x08a59435c8359a45f4f5dc8d91df893cc33daf29` is an 87-byte ERC-1967 UUPS proxy and it is the venue that took 446,281,787 PRM off the fleet in 71 sells** (blocks 56176712 to 61551574, the bulk on day one). Code body is the standard `0x6080...360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc...` implementation-slot delegatecall. Selector seen on the calling tx: `0x94ac01a1`.
- **The bot had been announcing every one of those as "PRM FLEET MOVED TOKENS OUT, sent to an outside wallet, not a sale".** On a token whose entire thesis is watching for the operator to dump, that is the worst possible miss.
- **Fixed 13 Sep: the test is now `code_size > 0`**, with one hand-carved exception, an EIP-7702 delegation (exactly 23 bytes, `0xef0100` prefix) which is still a wallet. Code or no code is a fact; a byte count is a guess. Backup `prmwatch.py.bak-20260913-*`.
- **Apply this everywhere in the fleet.** Any bot that separates a sale from a transfer by code size has the same hole.

## The 39th wallet
- **`0x7cc44e0880feb0f6934d26103458b3a158ea0280`** never touched the curve. Two fleet wallets handed it **25,762,381 PRM (2.58% of supply, about $198,833 at $0.007718)** at blocks 56228282 and 56228753 on day one. Plain EOA, zero code, has not moved a token since.
- Added to `/opt/prmwatch/config.json` wallets, now 39. Backup `config.json.bak-20260913-*`.

## Chain ledger the replay produced (whole life of PRM, dust floor 1,000 PRM)
| key | n | PRM |
|---|---|---|
| curve_buy (launch, drained contract `0x0ea38acc640a159f2cebe5439a54d977ded64158`) | 42 | 875,684,584 |
| sell (via the 1967 proxy and the RH PoolManager) | 71 | 446,281,787 |
| transfer_out (all to the 39th wallet) | 2 | 25,762,381 |
| burn | 10 | 15,516,912 |
| buy (buybacks, mostly PoolManager `0x8366a39cc670b4001a1121b8f6a443a643e40951`) | 69 | 18,144,013 |
| auction_bid | 34 | 13,530,000 |
| auction_refund | 28 | 9,020,000 |
- Reconciles to the ~406.3M the fleet still holds. **Launch acquisition is split into its own `prm_curve_buy` key so a buyback card never inherits an 875M running total.**
- All nine keys are marked seeded, including the ones with zero rows, so a genuine first event can honestly say "#1 first one on record".

## Method note that matters beyond PRM
- **Two full-chain topic-filtered `eth_getLogs` per wallet (as sender `topics[1]`, as receiver `topics[2]`) reconstructs a cohort's entire transfer history without a block sweep.** 38 wallets = 76 calls.
- **The RH RPC 429s hard on batched full-chain getLogs.** Batches of 20 failed instantly. Batch size 2 with a 2 second gap and 4s-multiplied backoff is what works. Cheap per-call methods (`eth_getCode`, `eth_getBlockByNumber`) still batch at 20.

# 13 Sep 2026. UNIPCS STALKER retired, and the first clean bot removal

- [stated] Ben: **"deactivate and delete the unipcs stalker process and alert"**. Done, both halves of it.
- Two services, not one, and a single-word grep would have missed the second: **`unistalk.service`** (KuCoin hot-wallet withdrawals into fresh Solana wallets, then their first buys) and **`unistalk-rh.service`** (exchange/Relay ETH into fresh Robinhood Chain wallets, then their buys). The wallet findings they were built on survive in [[unipcs-wallets]] and [[unipcs-worldview]]; only the watcher is gone.

## THE REMOVAL CHECKLIST, now the standing one for retiring any bot in this fleet
A bot is not one thing in one place. Deleting the directory alone leaves four ghosts. In order:
1. **`systemctl stop` AND `systemctl disable` every unit.** Disable is what removes the `multi-user.target.wants` symlink; stop alone lets it come back on reboot.
2. **Archive before deleting.** `tar czf /opt/_removed/<bot>-<stamp>.tar.gz -C /opt <bot>` then `rm -rf`. Deletion is the one irreversible step in the fleet and 650KB of insurance costs nothing.
3. **`rm /etc/systemd/system/<unit>.service` then `systemctl daemon-reload` and `reset-failed`.** Without this the unit lingers as `not-found` and the watchdog can still name it.
4. **Unhook the watchdog**, `/opt/watchdog/watchdog.py` AND `/opt/watchdog/state.json`. Otherwise it alarms forever on a log file that no longer exists.
5. **Unregister from `/opt/botctl/registry.json`** so `botctl status` does not show a permanently dead row.
6. **Kill leftover Mac-side pollers.** A previous session had left an `ssh ... for i in 1..8; do sleep 20; cat /opt/unistalk/state_rh.json` loop running on the Mac (PID 24484) and it survived the droplet deletion. **`ps aux | grep <bot>` on the MAC as well as the droplet.**
7. **Move the Mac source copies out of `~/kookwatch/`** into `~/bots-mac-disabled/_removed-<bot>-<date>/`, so a future session cannot re-deploy from them. Seven files here: `unistalk.py`, `unistalk_setup.sh`, `rhstalk.py`, `rhstalk_patch{,2,4}.py`, `rhstalk_setup.sh`.

## Two traps hit, both worth keeping
- **`state.json` is NESTED. A top-level `pop()` silently removed nothing** and reported success. The strip has to recurse. The script said "dropped nothing" and the verification grep is what caught it, which is the whole argument for verifying rather than trusting the patch's own output.
- **The watchdog unit is `watchdog-bots.service`, not `watchdog.service`.** `systemctl restart watchdog` returned non-zero, the `||` fallback also missed, and the service sat `inactive` until it was restarted under the right name. **Resolve a unit name with `systemctl list-unit-files | grep` before restarting it; never guess it from the directory name.**

## State after
- Registry **28 -> 26 bots**, parses clean. Zero `unistalk` references anywhere in `/opt` outside `.bak` files, zero units, `watchdog-bots` active.
- **34 fleet services still running**: alfiewatch, ansem-rh, ansemtracker, chjokkawatch, crashbot, crashton, deltaalert, dinerodom, dougfomo, dougsol, dougwatch, earlybirds, ghostsol, ghostwatch, grailwatch, gramwatch, kookwatch, linkscan, longboard, lwserve, neetwatch, pairexmon, prmwatch, rbt3k, rhtracker, rhtracker-base, rugwatch, strongtrader, ubermwatch, xdeck, watchdog-bots.
- Archive kept at `/opt/_removed/unistalk-20260913-070618.tar.gz`. Backups `watchdog.py.unistalkbak-*`, `state.json.unistalkbak2-*`, `registry.json.bak-*`. Removal script `/root/rm_unistalk.sh`.
- **`botctl.py` run ON the droplet fails**: it SSHes to `159.223.88.168` using `/root/kookwatch/id_kookwatch`, which only exists on the Mac. **botctl is a Mac-side tool by design.** Verify registry edits with a direct `json.load` on the droplet instead.

# 13 Sep 2026. STRONG TRADER retired, and RH SWINGTRADER 1 found (it was never the same bot)

- [stated] Ben, in one message: **"Remove the strong trader bot alert. Everything that's in there, I don't need it anymore."** AND **"I used to have this original trader that I was watching... He was buying on breakdowns when things were dumping. He was originally buying Stomp, Broker, and Cashcat... he would sell it in small clips when it would go up 10%, 15%, 20%. He would ladder out... Where is that bot? I can't find it anymore."**
- **THE TWO HALVES NEARLY COLLIDED AND THAT IS THE LESSON: the delete request and the "where is it" request could have been the same bot.** `strongtrader` held nine wallets, one of them "THE FLUSH BUYER" `0x4671fb2fb50a8001e13523a8a3df37bc67777777`, described in [[copytrade-targets]] as laddering limit clips into a crash, which matches his description closely enough to be dangerous. **So the config was read and the ladder wallet identified BEFORE anything was stopped.** Standing rule: when a delete and a find land in the same message, resolve the find first.

## THE ANSWER: it is `rhtracker.service`, and it was never missing
- **`0x688F9499f748053b14541F5c596c53A1b002F52D`**, labelled **"RH Swingtrader 1"** in `/opt/rh-tracker/config.json`. Second wallet in the same config is its Base leg, **`0xf5483bc0F2f89A22282c57E028d4D6C2F63958a4`**.
- Service `rhtracker.service`, directory `/opt/rh-tracker`, log `/opt/rh-tracker/rhtracker.log`. **Active the whole time.**
- **THE LADDER SIGNATURE, verified on chain lifetime, and it is unmistakable. Sells outnumber buys by 4x to 7x on every name:**

| token | buys | sells | sells per buy | contract |
|---|---|---|---|---|
| STONKBROKER | 11 | 69 | **6.3** | `0xe934e36A439C94017B64a3FecE66AF12099aBF50` |
| CASHCAT | 14 | 82 | **5.9** | `0x020bfC650A365f8BB26819deAAbF3E21291018b4` |
| WOOD | 3 | 20 | **6.7** | `0xf8bc08092c06db6148114dcf82af881f1085f92b` |
| UP | 24 | 109 | **4.5** | `0x57c0e45cb534413d1c20a4240955d6bb250bb4f1` |
| GWOOD | 14 | 40 | 2.9 | `0x24d8657e10af588b12de3e102a116f77b9e35ee8` |
| COPPERINU | 16 | 34 | 2.1 | `0x5317c0d077d2eeb639448939b930d49c4984b63b` |
- **A few concentrated buys, many small sell clips, is the whole trade shape in one ratio.** Worth reusing as a screen: `sell_legs / buy_legs >= 4` on a token a wallet has fully exited identifies a laddering-out swing trader without any pricing work at all.
- Lifetime 2,760 in-legs, 886 out-legs, **676 distinct tokens**, first buy 7 Jul 2026 19:18 UTC. CASHCAT first buy 8 Jul 15:38, STONKBROKER first buy 18 Aug 11:25.
- **"Stomp" does not exist in its book.** The only RH token named STOMP (`0x9067CbdE73bF25e3fEbd2E51E6767fC60AeC2493`, mcap ~$3.9k) has zero legs from this wallet and zero from the flush buyer. Most likely Ben half-remembered STONKBROKER as two names, or meant WOOD / GWOOD. Not resolved, and said so rather than fitted.

## WHY HE STOPPED SEEING IT: the trader went quiet, the bot did not break
- **Last real swap: 3 Sep 2026 08:48 UTC. Ten days of silence.** Last sells per name: CASHCAT 21 Aug 12:37, STONKBROKER 30 Aug 08:16, UP 2 Sep 15:54. It holds **zero** CASHCAT and **zero** STONKBROKER now, so those positions are fully closed.
- Everything since is inbound dust: the last twelve distinct txs in seven days are all `transfer (no-dex-evidence)` on two tokens, correctly suppressed.
- **The classifier is not the problem. Its `SWAP_TOPICS` set is byte-identical to strongtrader's and carries the Uniswap v4 PoolManager topic `0x40e9cecb9f5f1f1c5b9c97dec2917b7ee92e57ba5563708daca94dd84ad7112f`.** A silent bot on a quiet wallet is the correct output, and the temptation to "fix" it would have been wrong.
- **TWO CONFIG ITEMS THAT WOULD BITE IF HE RESUMES, not changed without asking:** `min_usd_buy` is **5000** (strongtrader ran 1000, and this wallet ladders in small clips, so most entries would be silent), and `quote_tokens` holds only WETH `0x0bd7d308f8e1639fab988df18a8011f41eacad73`, **missing USDG `0x5fc5360d0400a0fd4f2af552add042d716f1d168`**, which is the chain's main quote. The USDG gap only matters for fallback check (4) since a real swap event is caught first, but it is still wrong.

## strongtrader removal, second run of the checklist above
- Nine wallets went with it, all documented in [[copytrade-targets]] so nothing is lost: `0x0428ef4923513f73d53fd6c28875574fc7ed0413`, `0x7e3ba68c49561aae7c23c1d20fef0f1d7615a3ad`, `0x44df085447dbebcf69c6675c3b8a795c7fdeb3f4` (fullturbo.eth), `0xde52c75041371b5e08d8a50b82c63734850628c1` (already disabled), `0xfa8e161b288386c38eced61ebdea0693de5184ff`, `0x56e95b47a07210f1ea9fa418c7c5dc164c419ac0`, `0xff2bcd8e3a12d6930518a45519da1c5e67129948`, `0x4671fb2fb50a8001e13523a8a3df37bc67777777`, `0xF86a7c853B60C36f79F6E1dE0E7868d109E97004`.
- Registry **26 -> 25 bots**. Archive `/opt/_removed/strongtrader-20260913-075406.tar.gz`. Backups `watchdog.py.stbak-*`, `state.json.stbak-*`, `registry.json.bak-*`, `pairexmon.py.stbak-*`. Script `/root/rm_strongtrader.sh`.
- **NEW STEP 8 FOR THE CHECKLIST: grep the fleet for the bot's name in ALERT TEXT, not just in config and code.** `pairexmon.py:348` had the string "xdeck, linkscan and strongtrader" inside the X-blackout card, so the blackout alert would have kept naming a bot that no longer exists. Patched to "xdeck and linkscan", pairexmon restarted, active.
- `/opt/setup_ansem_rh.py` still mentions it, left alone: it is a one-off setup script that already ran, not a service.
- 33 fleet services running afterwards, watchdog-bots active, rhtracker active.

# 13 Sep 2026. RUNWATCH built — the migrated-to-500k runup alert

- [stated] Ben: freshly migrated Pons V2 coins run from $30k-ish to $500k+ inside 15 minutes to 4 hours, then dump 70%+, **and the dump is where he buys**. He asked for an alert "as soon as a freshly migrated coin passes 500k market cap". Named examples: **BLAST 0x724a163f0081fa1771590dda9b55edde2c5b823a**, **CME 0xe2324FF2a59F8eCBa8c321c6466e59121C00e795**, **PAIREX 0xB8Cf4ad387cfd607C66a207CFFFC46498aACd9d6**. All three verified as Pons V2 `PoolGraduated` events (blocks 61711984, 57289683, 60873584), so one trigger covers the whole pattern.
- **Service `runwatch.service`, `/opt/runwatch/runwatch.py`, sqlite `/opt/runwatch/runwatch.db`, log `/opt/runwatch/runwatch.log`, 45s loop.** Registered in `/opt/botctl/registry.json` (25 -> 26 bots) and in watchdog `UNITS` with `max_silence 600` (it logs every cycle, so silence IS judgeable here, unlike the websocket bots). **Shares the coinbench bot token and chat 357464289** — same thread, no new BotFather step.
- Two alerts, in Ben's plain-action format:
  - **RUNUP** on first cross of `runup_mcap` (500,000) -> headline **WATCH. Do not chase.**
  - **BUY ZONE** once it has run and then falls to `dump_keep` (0.30) of its own peak -> headline **BUY.** This is the entry he actually wants and it only ever fires on something that ran first.
- Config knobs in `/opt/runwatch/config.json`: `runup_mcap` 500000 · `dump_keep` 0.30 · `window_hours` 6 (never ran -> dropped, never alerts) · `hold_hours` 48 · `min_liq` 5000 · `poll_seconds` 45 · `backfill_hours` 6 · `muted`.
- **THE FIX THAT MATTERS, and it is the evhist lesson again in a new costume: the first version set the base market cap to the FIRST PRICE THE BOT SAW.** On the backfill that made BLAST read "up 1.0x" at $1.15M, because the bot's first sight was not the migration. **A base taken at boot describes the bot, not the token.**
  - Correct base comes free from the event itself: `PoolGraduated(address indexed token, uint256 positionId, uint256 tokenAmount, uint256 pairTokenAmount)` carries the LP amounts seeded at migration. `price_at_migration (in quote) = pairTokenAmount/10^qd ÷ tokenAmount/10^18`, then `base_mc = current_mc × price_at_migration ÷ priceNative`. **Pure ratio, so no USD price of the quote token is needed** and no `totalSupply()` call either.
  - Verified: BLAST base **$41,160** against $1.13M live = 27.5x, which matches Ben's "from 30k" description. Before the fix it read $1,153,336.
  - Reusable rule: **any "since launch" or "since migration" multiple must be anchored to an on-chain event field, never to the watcher's first observation.**
- Quote-token decimals resolved by `decimals()` (`0x313ce567`) with an in-process cache, defaulting to 18 on failure. RH quote tokens are mixed (USDG 6, WETH 18, tokenized stocks vary) so assuming 18 would misprice every USDG pair by 1e12.
- DexScreener `tokens/v1/robinhood/{a,b,...}` 30 per call is the mcap source (standing rule: DexScreener marketCap for RH tokens). ~57 live candidates at any time = 2 calls per cycle. Tokens DexScreener has not indexed count `misses` and are dropped at the window edge rather than retried forever.
- Deploy path was the standing one: write on the Mac at `~/runwatch/`, `scp` to the droplet, because **the Cowork container has no outbound route to 159.223.88.168**. Patches applied as python replace-scripts (`patch1.py`, `patch2.py`, `wdpatch.py`) rather than heredocs — **a zsh heredoc through `ssh` mangles `\x27` escapes and the watchdog patch failed that way once.**
- Backup files left: `/opt/runwatch/runwatch.py.bak-*`, `/opt/botctl/registry.json.bak-*`, `/opt/watchdog/watchdog.py.runwatchbak-*`.
- **Open item:** the two RUNUP cards sent at 08:34 UTC (BLAST, 奶蛙 0xbc18d0b1d039107fc74ea97c333da6323b87723d) carried the wrong "up 1.0x" line from the pre-fix build. Everything after 08:36 is correct.

## RUNNERS BOARD, same day — the multi-chart screen Ben asked for
- [stated] Ben: he cannot watch 6 to 8 fresh runners' charts at once. Wants "the GMGN surface with all the details and the quick buy buttons" but as a multi-chart that auto-populates on criteria (migrated in the last 3 hours, went above $500k).
- **GMGN ALREADY HAS NATIVE MULTICHARTS and it was found by opening it, not assumed.** Token page -> chart toolbar -> **Multicharts** -> a layout picker offering **1 through 8 panes** in several arrangements. Picking one splits the chart area immediately and `localStorage["tradingview-main.layout-type"]` reads `6v`. **So the feature exists; the only thing missing is auto-fill.** Tell him this before offering to build anything.
- **GMGN CANNOT BE IFRAMED. Tested, not assumed:** an iframe to `gmgn.ai/robinhood/token/...` from `example.com` renders Chrome's refused-to-connect placeholder. **DexScreener CAN be iframed** and its embed renders fine: `https://dexscreener.com/robinhood/{PAIR}?embed=1&theme=dark&info=0&trades=0&interval=5`. Note it needs the **pair address**, not the token address.
- **Auto-filling GMGN's own panes is possible in principle but was abandoned as a rabbit hole.** The layout lives in `localStorage["tradingview-main.chart"]`, keyed by a chart GUID, and the per-pane symbols sit inside a JSON **string** nested at least two levels deep (`content` containing another `content`). Symbol format is `robinhood/0x.../USD/MCAP`. A future session could finish this as a bookmarklet; do not start it without budgeting for the nesting.
- **BUILT INSTEAD: http://159.223.88.168/runners.html**, served by the existing `lwserve` out of `/opt/launchwatch/pub/`. Auto-populating grid, no typing, no tab juggling.
  - Feed `runners.json` written every cycle by `write_board()` in `/opt/runwatch/runwatch.py`. Rows = tokens with a `runup_ts` and `done=0`, newest first, max 12.
  - Per tile: ticker, stage badge, market cap, **x from migration**, peak and peak x, **% off peak**, age, liquidity, the **buy_at** price (peak x dump_keep), a live DexScreener chart, an **open in GMGN** button and a copy-CA button. Full 42-character address printed on every tile.
  - Stage is derived, not stored: `BUY ZONE` once the buyzone fired, `FADING` at >=30% off peak, else `RUNNING`.
  - Controls: show 4/6/8/12, filter by stage, and a manual "refresh charts".
  - **The render is diffed by address, not rebuilt.** A full re-render every 20s would reset every iframe and wipe the charts. Tiles that persist get only their header and stats replaced; the iframe is left untouched. Any future board with embedded charts must do the same.
- Schema additions to `tok`: `pair`, `liq`, `v24`. **`sqlite3` CLI is NOT installed on the droplet** — ALTER TABLE has to go through a python `sqlite3` snippet, and a missing-column error surfaces as a traceback in the bot log, not at restart.
- **The Claude-in-Chrome extension refuses screenshots and navigation on `159.223.88.168`** ("Permission denied for this action on this domain"), so the board could not be eyeballed from this side. Verified instead by: HTTP 200 from Ben's Mac, a valid `runners.json` (BLAST 55.5x at $2.28M, 奶蛙 27.9x at $1.1M), and `node --check` on the extracted inline script. **Curl from the Mac is the fallback verification for anything served on that IP.**

### Runners board v2, same day, after Ben used it
- [stated] Ben: "it keeps reloading the charts", sort by market cap, and drop the stats row ("I can see that all on the chart"), he just wants them up fast once they cross 500k.
- **THE RELOAD BUG, and it is a general rule: MOVING a DOM node that contains an iframe reloads that iframe.** v1 reordered tiles every 20s with `prev.after(el)` / `GRID.prepend(el)`, so every chart restarted on every tick. **Fix: append each tile ONCE and never move it again; reorder with CSS `order` on the grid item.** Anything holding an embed (chart, video, map) must be positioned by CSS, never re-parented. Also do not `replaceWith()` a parent that contains the embed; write to the leaf text nodes instead.
- v2 tile is now: colour dot (green running, amber fading, red buy zone), ticker, age, market cap, chart, GMGN button, copy-CA button, full address. The x-from-migration / peak / drawdown row is gone from the UI but still in `runners.json` and still drives the stage colour and the Telegram cards.
- Default sort is **market cap desc**, with a "newest run" option kept in the sort dropdown.
- `poll_seconds` 45 -> **20** so a fresh cross lands within ~20s; board polls the feed every 15s. DexScreener cost is 2 batch calls per cycle, nothing.
- Verification without the Chrome bridge (it is permission-blocked on that IP): extract the inline `<script>` with a regex and run **`node --check`** on it, then curl the page from the Mac for a 200. Both are Mac-side.
- [stated] Ben: "the copy CA button isn't working." **Cause: `navigator.clipboard` only exists in a secure context, and the board is served over plain `http://159.223.88.168`, so the handler threw silently on every click.** Fixed with a hidden-textarea `document.execCommand('copy')` fallback plus a "copied" flash on the button. **Standing rule for anything served off that droplet: no `navigator.clipboard`, no getUserMedia, no service worker, no Notification API. It is http, so every secure-context API is unavailable and fails quietly.**

### Runners board v3 — own charts, 30s, market-cap axis. The DexScreener embed was thrown away.
- [stated] Ben: default 30 second candles, **y axis in market cap not price**, no "Tracked by DexScreener" footer, no OHLC legend, no volume row, buttons at the TOP so the chart gets the space.
- **None of that is reachable through DexScreener's embed URL.** Its toolbar does have a 30-second option and a Price/Mcap toggle (checked by opening the real page and the resolution dropdown: SECONDS = 1s / 15s / 30s), but they are UI state, not URL params, and the footer and the TradingView legend are always painted. Guessing `chartType=mcap` / `priceScale=mcap` just left the embed stuck on "Loading pair...".
- **So the board now draws its own charts** with TradingView **lightweight-charts 4.1.3**, fed by GMGN's OpenAPI.

**GMGN kline, the piece that made it possible (a FOURTH correction to "the API is thin on this chain"):**
- `gmgn-cli market kline --chain robinhood --address <addr> --resolution 30s --raw` returns `{list:[{time(ms),open,high,low,close,volume,amount}]}`. **30s resolution exists on Robinhood Chain.** Resolutions: 30s / 1m / 5m / 15m / 1h / 4h / 1d.
- Also present and not yet used: **`gmgn-cli market trenches`** (new creation / near completion / completed) and `market signal`, `market trending`, `market hot-searches`, `market search`. The earlier note that Trenches has "no REST list endpoint" is true only of the website; the OpenAPI has it.
- **The droplet has no node, so gmgn-cli could not be installed there. The auth was reimplemented in ~15 lines of python instead, read straight out of `dist/client/signer.js` and `dist/client/OpenApiClient.js` on the Mac:**
  - host `https://openapi.gmgn.ai`, kline path `/v1/market/token_kline`
  - **market/token/portfolio routes are "exist auth": API key only, NO signature.** Only swap and order routes sign.
  - exist auth = header `X-APIKEY` plus query params `timestamp` (unix seconds, server allows ±5s) and `client_id` (a UUID, replays rejected within 7s).
  - signed auth, for later: `X-Signature` = base64 Ed25519 over the message `{sub_path}:{sorted_query_string}:{body}:{timestamp}`, query sorted alphabetically, array values sorted and repeated as `k=v`.
- API key copied Mac -> droplet by piping it straight into `ssh ... 'cat > /opt/runwatch/gmgn_key'` (chmod 600), so the value never passed through chat.

**Service `klines.service`, `/opt/runwatch/klines.py`, 30s loop.** Reads `runners.json`, pulls 30s candles for the top 12, converts price to market cap with `totalSupply()` (`0x18160ddd`) cached per token, writes `/opt/launchwatch/pub/k/{addr}.json` and prunes files for tokens that left the board. 0.6s spacing between tokens keeps it inside the leaky bucket.

**Board v3:** top bar is dot + ticker + age + GMGN + copy CA + market cap, then the chart fills everything else. Candles only, no volume series, no legend, `localization.priceFormatter` renders the axis as $1.62M / $308K. Height selector 200/270/340/420. Charts are created once per token and only `setData` afterwards, so nothing reloads.
- **`lightweight-charts` is NOT on cdnjs (404). jsdelivr has it.** It is now self-hosted at `http://159.223.88.168/lw.js`, downloaded on the droplet itself, so the board has no CDN dependency. **paste.rs refused the 160KB file** — for anything that size, curl it down on the droplet directly rather than routing through the container.
- Symptom to recognise: the tiles rendered with a header and an empty box and the status read "feed unreachable", because `createChart` threw inside the same try/catch as the fetch. `typeof LightweightCharts === 'undefined'` in the console was the actual diagnosis. **Do not wrap library init and network fetch in one try/catch.**
- [stated] Ben, after v3: "it is now not showing me the full history... let me still choose the timeframes." **Root cause is a hard API limit, not a setting: `token_kline` returns AT MOST 100 candles, and passing `--from`/`--to` over a 6 hour window at 30s still returns exactly 100 rows.** Tested both ways on BLAST. **So on GMGN the timeframe IS the history length:** 30s = 50 min · 1m = 100 min · 5m = ~4.3h · 15m = ~25h.
- Fix: `klines.py` now fetches **all four resolutions** per token and writes them into one file as `{"res":{"30s":[...],"1m":[...],"5m":[...],"15m":[...]}}`, keeping the previous array for any resolution whose call fails. Loop moved 30s -> 45s, spacing 0.35s per call, 12 tokens x 4 = 48 calls per cycle and no rate-limit hits observed.
- Board gained a **`tf` dropdown, default 1m**, first control in the header. Changing it resets `rec.seen` and redraws every tile from the already-downloaded file, so switching timeframe costs zero network and never reloads a chart. `rec.seen` holds the last-drawn resolution string rather than a boolean, which is what makes `fitContent()` re-run on a switch but not on a data refresh.

### Runners board v4 — the DexScreener embed broke mid-session, so the board now has two pages
- **Full history at every timeframe is SOLVED: `token_kline` paginates backwards.** Pass `to=<oldest timestamp you already have>` and you get the 100 candles before it. Verified on BLAST: page 1 ended 08:52:30, page 2 with `to=` that returned 08:03:00 to 08:52:30. `klines.py` now **backfills a token page by page to its migration time once** (MAX_PAGES 14, stops early when a page returns <90 rows), then only ever fetches page 1 and merges. History is immutable, only the last candle moves, so steady state is still 4 calls per token per cycle.
- [stated] Ben then said the DexScreener embed was better because **lightweight-charts has no drawing tools and no indicators**, and asked for "proper plugins that I can still use, or switch over to DexScreener". Correct: lightweight-charts is a renderer, it has no toolbar at all. TradingView's full Advanced Charts needs their licensed Charting Library plus a custom datafeed, which is not a same-session build.
- **So the board was switched back to DexScreener embeds, and they then stopped rendering.** What was actually observed, in order:
  1. `?embed=1&theme=dark&info=0&trades=0` in an iframe: worked earlier today (Ben's own screenshot proves it), then hung on "Loading pair..." every time.
  2. The same URL opened TOP-LEVEL in his browser: also hung.
  3. `?embed=1&theme=dark` (no info/trades params) top-level: **rendered fine**, with the left drawing rail, indicators, resolution picker and the Price/Mcap switch.
  4. The same plain embed inside my iframe: **blank**.
  5. The NON-embed pair page in an iframe: blank, it sends X-Frame-Options.
  So framing works only for `embed=1`, and `embed=1` is being throttled or blocked for us right now. **Most likely cause: repeated embed loads from one IP during testing. Do not conclude the feature is gone; retest from a cold browser before rebuilding anything.**
- **Shipped as two pages with a one-click link between them, because neither is reliable alone:**
  - `http://159.223.88.168/runners.html` — own charts (lightweight-charts, 30s/1m/5m/15m, market-cap axis, full history, no tools). Works always.
  - `http://159.223.88.168/ds.html` — DexScreener embeds with the full toolbar and drawings, a `view` switch between "full tools" and "chart only", and the footer cropped with an oversized iframe inside `overflow:hidden`.
- **Chrome restores `<select>` values across a same-URL reload**, so a changed default in the HTML does not show up for someone who already picked an option. Set it from JS, or version the URL, when a default must change.

# 13 Sep 2026. RUNWATCH retired the same day it was built, and it nearly took XDECK down with it

- [stated] Ben: **"Delete the runup ex alert. I don't need that anymore. Stop the bot. Delete the process."** Done. `runwatch.service` stopped, disabled, unit removed, `/opt/runwatch` deleted, watchdog unhooked, registry **26 -> 25 bots**. Archive `/opt/_removed/runwatch-20260913-112232.tar.gz`. Script `/root/rm_runwatch.sh`.
- Everything built for it earlier today is in the RUNWATCH and RUNNERS BOARD sections above. **The runners board pages `runners.html` and `ds.html` and `klines.service` were NOT touched**, so the board still exists but its `runners.json` feed is now dead. If Ben wants the board gone too, that is a separate removal: `klines.service`, `/opt/launchwatch/pub/runners.html`, `ds.html`, `lw.js` and `/opt/launchwatch/pub/k/`.

## STEP 9 FOR THE REMOVAL CHECKLIST, and this one would have caused a silent outage
**A bot's directory can hold a SHARED SECRET that another live bot reads by absolute path.**
- `/opt/xdeck/xdeck.py:127` did `open("/opt/runwatch/gmgn_key")`. Deleting `/opt/runwatch` deleted xdeck's GMGN API key.
- It was caught because step 6 of the checklist greps `/opt` for the bot's name afterwards, and `/opt/xdeck/xdeck.py` came back in the "live refs left" line. **That grep is the whole reason this was a five-minute fix instead of a week of quietly broken xdeck.**
- **So: before `rm -rf`, grep the fleet for the bot's DIRECTORY PATH (`/opt/<bot>/`), not just its name.** A path reference to a key, a db, a cache or a shared module is invisible to a name grep if the other bot never says the word.
- Fixed by recovering the key from the tarball to a neutral home: `/opt/common/gmgn_key` (38 bytes, chmod 600), and repointing xdeck. Backup `xdeck.py.rwbak-*`. xdeck restarted and active, zero `runwatch` references left anywhere in `/opt` outside `.bak` files.
- **Generalisable: shared secrets and shared data belong in `/opt/common/`, never inside one bot's folder.** `/opt/common/evhist.py` already set that precedent; `gmgn_key` now lives beside it. Worth a sweep for other cross-bot absolute paths.

## A deploy trap that cost one failed run
- The first attempt patched xdeck with a python heredoc inside an `ssh '...'` single-quoted command and tried to escape the inner quotes as `'`. **zsh does not expand `'` inside single quotes, so python received a bare unquoted path and died with `SyntaxError: invalid syntax`.** It was harmless only because the script compiles before writing.
- **The reliable path, now used twice today: `Desktop_Commander.write_file` the patch to the Mac, then `ssh ... 'cat > /root/patch.py' < /tmp/patch.py`, then run it.** No quoting layer at all. Never hand-escape quotes through an ssh heredoc again.
- The patch script itself should always `ast.parse` the new source BEFORE `open(p,"w")`, so a broken patch leaves the live file untouched. That is what made the failed attempt a non-event.

## CORRECTION, same hour: RUNWATCH WAS THE WRONG BOT. Restored.
- [stated] Ben, immediately after: **"I was talking about the runup Telegram alert. That's completely different than what we're doing with GMGN and the dashboards and the alerting when something goes over 250k and all of that stuff. That is completely different. We're basically replacing the old Telegram alert. That one was older, and we don't need that anymore because of the new process that we're just building now with xdeck and the multi-chart dashboards."**
- **I deleted the thing he told me to keep.** `runwatch`'s config at delete time read `runup_mcap = 250000`, and "the alerting when something goes over 250k" is the exact phrase he used for what stays. **The clarification contained the answer and the delete contradicted it.**
- **THE RULE, and it is the real lesson of the day: a delete request whose target is named by DESCRIPTION and not by service name must be resolved to a service BEFORE anything is stopped, and if two candidates fit, ask.** "The runup alert" matched one directory by name (`/opt/runwatch`), so the name match was taken as identification. **A name match is a hypothesis, not an identification.** The confirming step that was skipped: check whether the candidate is the OLD thing or the NEW thing by its build date and by whether the user's own description of what-to-keep overlaps it. runwatch was built the same morning; "that one was older" ruled it out in one clause.
- **Restore path, and it worked completely because the archive step exists:** `tar xzf /opt/_removed/runwatch-20260913-112232.tar.gz -C /opt`, rewrite `/etc/systemd/system/runwatch.service` from scratch (the unit file is NOT in the tarball, only `/opt/<bot>` is, so **the removal checklist should also archive the unit file**), `daemon-reload`, enable, start, re-add to `registry.json`, re-hook watchdog after the linkscan row. Live again inside 3 minutes, `watching=72`, `runners.json` writing at 11:26. The sqlite `runwatch.db` came back intact so no state was lost.
- `gmgn_key` now exists at BOTH `/opt/common/gmgn_key` and `/opt/runwatch/gmgn_key`, and xdeck points at the `/opt/common/` copy. Keep it that way; `/opt/common/` is the correct home.
- **Interim action taken instead of a second guess: `muted: true` in `/opt/runwatch/config.json`.** That stops the RUNUP and BUY ZONE Telegram cards while the engine keeps feeding `runners.json` and the boards. Fully reversible with one flag, backup `config.json.mutebak-*`. If he wanted the Telegram half gone and the board kept, this is already done; if he meant a different bot, flip it back.

### The search for the actual "old runup Telegram alert", and what it turned up
- **No active service other than runwatch has a runup-style market-cap threshold.** Checked every active bot's config for tag/mcap/threshold/multiple keys: they are all wallet watchers (`max_alert_usd 5000000`, `max_block_span 50000`) plus linkscan's `min_mcap`/`max_mcap` band and pairexmon's peg drift.
- **`botctl.py ping` from the MAC is the fastest way to enumerate which Telegram bot each service posts as** without ever printing a token. It fails on two legacy rows (`neetwatch`, `ghostwatch`) that carry a `tg` key instead of `config`, so it needs a `.get("config")` guard. The registry's own `tg` field already names most of them.
- **`longboard` is NOT an alert**: 216 lines, watches the long.xyz launcher `0x22e99278308b393ea1260859b181ad7e78f5eeed` topic `0xadc6f1f726f7c710f77ec06adc75f3bb964e5be19581b072c67f7b9b4039267b`, filters on MIN_BUYS_24 15 / MIN_LIQ 5000 / MAX_MCAP 1500000, and writes to `/opt/launchwatch/pub`. Zero Telegram sends.
- **The strongest remaining candidate is `/opt/pullback/pullback.py`**, and it is CRON-DRIVEN not systemd-driven, which is why it reads as "inactive": `0 22 * * * cd /opt/pullback && python3 pullback.py`. It sends a daily Telegram card headed **PULLBACK & COMEBACK** with sections TOP OF THE SCORE, NEW IN THE TOP BAND TODAY, REPEAT ROCKETS IN A PULLBACK, DO NOT BUY THE BOTTOM N, and it **computes a `runup` metric** (`runup=ath/base`, line 217). It is from 5 Sep, so it IS older, and the new BUY ZONE alert plus the runners board replace exactly what it does.
- **`systemctl is-active` is not a liveness test on this droplet. Eight bots run from crontab, not systemd**: rhtop (04:20), pullback (22:00), launchwatch (*/3), deptab (03:00), fillwatch (*/30), parupdate (01:00), stonkfun (05:20), linkscan backfill (*/5). **Any "which bots are running" answer must read the crontab as well, or it will miss a third of the fleet.**

## Watchdog alarmed on a HEALTHY runwatch, hours after the restore. Two separate causes, both created by the delete.
- Symptom: watchdog cards "runwatch was down, log silent 12 min and no live socket, Restart: FAILED" then "keeps failing, restarted 3 times in the last hour, so I have stopped restarting it".
- **The bot was fine the whole time.** `active (running)`, Main PID 757960, watching 78 tokens, logging every ~12s. It was logging to the JOURNAL.
- **Cause 1: the unit file I rewrote from scratch during the restore omitted `StandardOutput=append:/opt/runwatch/runwatch.log` and `StandardError=append:...`.** `/opt/runwatch/runwatch.log` froze at 11:22, the moment of the delete. The watchdog reads log mtime, so it correctly reported silence.
  - **RULE: a hand-rebuilt systemd unit must carry the `Standard*=append:` lines, or the watchdog alarms on a perfectly healthy bot.** And the archive step of the removal checklist must include `/etc/systemd/system/<bot>.service`, since `tar czf ... -C /opt <bot>` does not capture it.
  - **RULE: verify a restored bot by the LOG FILE mtime, not by `systemctl is-active`.** `is-active` was green through the entire outage.
- **Cause 2: `rm -rf` of a directory leaves any running service holding an `append:` fd writing to a DELETED INODE.** `klines.service` had `/proc/<pid>/fd/1 -> /opt/runwatch/klines.log (deleted)`. Restoring the directory creates a NEW file the old process never sees, so its output goes nowhere forever.
  - Symptom to grep for: `ls -l /proc/<pid>/fd/1` ending in `(deleted)`.
  - **RULE: after restoring (or deleting) a directory, restart every service that had an open fd into it.** Fixed by restarting klines; candles in `/opt/launchwatch/pub/k/` fresh again (7 tokens).
- Watchdog give-up state cleared: `/alerted/runwatch`, `/restarts/runwatch`, `/gaveup/runwatch` stripped from `/opt/watchdog/state.json` (backup `state.json.rwfixbak-*`), `watchdog-bots` restarted. **The give-up keys persist, so fixing the bot is not enough; the state must be cleared or it stays silent on that unit.**
- Final state verified: runwatch, klines, xdeck, lwserve, linkscan, pairexmon, watchdog-bots all active; runwatch log moving (`head=61937852 new=1 watching=79`), `runners.json` written 11:56:52.
- Also noted, not acted on: `longboard` and `lwserve` units also lack `Standard*` redirects (harmless, they are not in watchdog `UNITS`).

# 13 Sep 2026. XDECK: the "reload" button was a CLEAR button, and an empty column could never fill
- [stated] Ben, on the `$prm` column: "waiting for posts... when i click reload nothing happens."
- **Nothing was broken. Not the key, not the bot.** twitterapi.io had 1,645,330 credits, xdeck was `active`, its rule `$prm -filter:retweets` was live, and `/api/poll` held exactly one matching post.
- **Two causes, both design, and together they make a dead-looking column:**
  1. **The `↻` icon in the column header was `BUF[c.id]=[];counts=0;paint()`, i.e. CLEAR, not refresh.** Clicking it on an empty column is a genuine no-op. The tooltip said "clear"; Ben (and anyone) reads the glyph as reload.
  2. **The live buffer only holds what arrived since the PAGE OPENED, and backfill (`/api/older`) only fires on scroll-to-bottom.** An empty column has nothing to scroll, so **an empty column can never fill itself**. Any column whose query is quiet, or that was edited (a `rule ~` update resets it), sits on "waiting for posts..." forever.
- **Fix shipped:** `ics[1]` now re-runs the column's search through `/api/preview` (the same one-shot search used when a column is created), resets the scrollback cursor, and paints. Tooltip is now "refresh this column now". Backup `index.html.bak-refresh-20260913-122239`, patch `/root/xd_refresh.py`, xdeck restarted, page 200. Verified: preview on `cmtzs3yry` returns **19 posts** for `$prm -filter:retweets`.
- **RULE: a UI control whose glyph implies "fetch" must fetch.** A destructive or no-op action behind a reload glyph reads as a broken product every time. Same class as the copy-CA button failing silently on the http board.
- **RULE: any feed pane that only accumulates live arrivals needs an explicit fill path that does not depend on scrolling.** "Load more on scroll" is not reachable from empty.