---
name: alert-bot-fleet-8
description: Alert bot fleet log, part 8, from 13 Sep 2026. The xdeck filter layer - why exclude words looked dead, the smart-apostrophe bug, the mined anti-spam word list, and the rule that a noisy column is usually a bad QUERY not a missing filter. Read before touching /opt/xdeck.
sources: [cowork]
aliases: [alert bots part 8, xdeck, xdeck filters, exclude words, spam words, XStack]
---
Continues [[alert-bot-fleet-7]]. Fleet inventory and droplet access in [[data-access]].

# 13 Sep 2026. XDECK: the global exclude filter "did not work". Three separate bugs.

- [stated] Ben: "Global filter in XStack isn't working. I added 'vote' and 'votes', and it still shows. It should live update, obviously, when I click Save. Even when I reload it, it didn't matter." And: find spam words actively, without killing good content.
- **The save was fine. "vote" and "votes" were sitting in `config.json` the whole time.** The words were never the problem.

## Bug 1 (the big one): exclude words were applied ONLY at ingest, never at render
- `blocked()` ran on the raw tweet inside `push()`. Anything already in the buffer stayed visible after a save, and `/api/poll?back=3000` on page load **replayed the same posts out of `FEED`**, so a reload changed nothing either. The filter was structurally incapable of removing a post it had already let through.
- **RULE: a filter that only runs at ingest is not a filter the user can change.** Any user-editable filter must also run at RENDER, or every edit is retroactively inert.
- Fixed in two places:
  - **Client** `/opt/xdeck/index.html`: new `EXRX` / `EXH` / `CEXRX`, `buildEx()`, `exBlocked(d,cid)`, and `vis(d,cid)` now tests global excludes, excluded handles and per-column excludes before the spam score. `buildEx()` is called in `loadCfg()` and `saveCfg()`. Call sites updated: `shown`, `hiddenTotal`, `fresh.filter`, `older.filter`.
  - **Server** `/opt/xdeck/xdeck.py`: new `item_blocked(it, cid)` (re-tests a SLIMMED feed item, since `blocked()` only understands a raw tweet) and `purge_feed()`, called from `recompile()`. Every POST to `/api/config` now strips newly-excluded posts out of `FEED` under the `CV` lock.
  - Verified live: added a word present in 66 of 141 feed items, feed dropped to 75, log printed `purge_feed: dropped 66 now-excluded items`, then reverted.

## Bug 2: smart apostrophes silently defeated any multi-word exclude containing one
- X posts use U+2019, not `'`. An exclude typed as `let's connect` could never match `Let's connect`. Same hole in `spam_score`, whose `shill` list contains `don't miss out`.
- Fixed with one normaliser on BOTH sides: `SMART` / `nrm()` in xdeck.py (used in `compile_excludes`, `blocked`, `item_blocked`, `spam_score`) and `nrm()` in index.html (used in `_rx` and `exBlocked`). Maps '' -> `'`, "" -> `\"`, en/em dash -> `-`, nbsp -> space.
- **RULE: any user-typed pattern matched against social text must be unicode-normalised on both the pattern and the haystack.**

## Bug 3 (the earlier one, same file): the column `↻` icon was a CLEAR button, not refresh
- Covered in [[alert-bot-fleet-7]]. Now re-runs `/api/preview`.

# The anti-spam word mining, done properly rather than guessed
Method, reusable: **harvest, apply the CURRENT filters, then mine only what SURVIVES, then print every hit of every candidate and eyeball it before shipping.**
- Harvested **1,403 posts** from all 8 live column queries via twitterapi.io straight from the Cowork container (12 pages each, `advanced_search` + `next_cursor`). Applied `min_followers 150` + the existing 34 excludes: **908 survivors**. Mined 2- and 3-grams over survivors with a distinct-handle count beside each (a 1-handle n-gram is one bot, not a pattern).
- **28 words added, 69 of 1,403 posts killed, ZERO false positives** (every hit was printed and read):
  - collab farm: `collaboration` `collaborations` `collaborate` `collab` `let's connect` `lets connect` `let's discuss` `lets discuss` `lets collaborate` `check inbox` `caught my attention` `caught my eye` `follow me back` `send me dm` `send dm` `message me` `get featured` `marketing partnership` `promote your project` `promote this` `grow together` `excited to have found` `shows promise`
  - paid-caller brag: `from my call` `from our call` `since our call` `since the call` `up profit` `profit from my` `my call on`
  - `whatsapp` (the "I created a dedicated WhatsApp group for $X $Y $Z shareholders" pump farm, 3 handles, identical text)
- Exclude list 34 -> 64 words. Backups `config.json.bak-spam-*`, `xdeck.py.bak-spam-*`, `index.html.bak-spam-*` / `.bak-excl-*`. Scripts `/root/xd_excl.py`, `/root/xd_spam.py`.
- The ONLY risky entries are `message me`, `collab` and `promote this`. Both hits of each were spam, but they are the first to remove if a real reply disappears.

## THE BIGGEST FINDING, and it is not a filter finding at all
- **The 中国制造 column's query was `("中国制造" OR "Made in China" OR 0xe339a891d14fafd37af44a3c9f227856268a1620)`. 114 of its 176 surviving posts were ordinary English chatter about Chinese manufacturing** (Teslas, eBay plush toys, tank explosions, restaurant menus). Not spam. Not blockable by any word that would not also kill the token's own posts.
- **RULE: when a column is full of junk, check whether the QUERY is wrong before adding a single exclude word.** A ticker or a project name that is also a common English phrase (`$B`, "Made in China") floods a column and no filter list can fix it.
- Fix shipped: the English phrase now requires a crypto anchor.
  `("中国制造" OR 0xe339a891d14fafd37af44a3c9f227856268a1620 OR ("made in china" (pons OR robinhood OR mcap OR "market cap" OR memecoin OR token OR chain))) -filter:retweets`
  Measured on page 1: on-topic went **5 of 20 -> 19 of 20**. A tight query with the English phrase removed entirely scored the same 19 of 20, so the anchored version is kept because it still admits English crypto posts.
- The `$B` column is `($B OR "B案" OR 0x19c8fcee15af82c583516a9c0e062e51f200b5b2)` and is **82% on-topic (133 of 162)**, so it was left alone. Worth re-checking if `$B` starts trending for another reason.

## Facts about the xdeck filter stack, for next time
- Three independent layers, easy to confuse: **global `exclude_words`** (hard hide), **per-column `exclude_words`**, and the **`spam` scorer** (scores 0..n, the page hides at `threshold`, default 3). A word in the scorer's `phish`/`shill` lists only ADDS POINTS, it does not hide on its own.
- `compile_excludes` is whole-word: `(?<![a-z0-9])word(?![a-z0-9])`. So `votes` does NOT match `vote`, and **`t.me/` can never match** because the char after `/` is a letter. Plurals need their own entry.
- The haystack is **text + display name + handle**, so an exclude word also kills an account whose NAME contains it.
- `min_followers` is 150 globally and does most of the heavy lifting: it alone removed 495 of 1,403 harvested posts.
- `spam.allow` (currently `longlaunches`, `ghostdotenv`) short-circuits the scorer entirely for those handles, but **does NOT bypass `exclude_words`**.
- twitterapi.io from the container is the right harvesting tool for this: 1,403 posts across 8 queries cost about $0.20 and under a minute.

# 13 Sep 2026. XDECK IDLE PAUSE. Shipped and verified live.

- [stated] Ben: "Add a feature on Xdeck that pauses the feed because I don't want to waste credits while I'm not at my laptop. If I don't use my mouse for 10 minutes, then Xdeck goes into pause mode. Only when I start moving my mouse again does it reactivate right away."

## Where the credits actually go, which decided the whole design
- `use_websocket` is **false** in config.json, so the websocket path (`ws_loop`, `sync_rules`, filter rules) is dead and `resync_loop` just calls `drop_all_rules`. **Every credit xdeck spends is a `do_search` inside `top_poll_loop`: one `advanced_search` per enabled column, every 45 seconds** (5 columns live). `credit_loop` (`/oapi/my/info` every 180s) and `backfill` (one search per column per boot, ~20 tweets x 15 credits x N) are the only other spenders.
- **Therefore pausing the BROWSER would have saved nothing.** The pause had to gate the server loop. A client-only "stop rendering" feature would have looked like it worked and cost exactly the same.

## The mechanism: browser heartbeat, server gate
- **Server `xdeck.py`**: new `ACT = {"last", "paused"}` and `WAKE = threading.Event()` next to `TOPSEEN`, plus `mark_active()` and `idle_paused()`. `top_poll_loop` now does `WAKE.wait(45); WAKE.clear(); if idle_paused(): continue` instead of `time.sleep(45)`. **`WAKE` is what makes resume instant: without it a mouse move would still wait out up to 45s of sleep.**
- New route **`GET /api/active`** calls `mark_active()` and returns `{ok, paused, idle_pause_minutes}`. `/api/poll` now also returns `paused`, and `/api/stats` returns `paused` and `idle_sec`.
- **Client `index.html`**: a self-contained block appended at the end of body. Listens on `mousemove, mousedown, wheel, keydown, touchstart, scroll` (window, capture, passive) plus `visibilitychange`, throttles the heartbeat to one `GET /api/active` per 30s, and stops beating entirely once idle. An amber `#pz` pill is inserted into `#top` before `.sp` reading "PAUSED - idle, spending nothing". `window.XIDLE()` returns `{idleSec, paused}` for debugging from the console.
- `credit_loop` sleeps 900s instead of 180s while paused.
- **Closing the laptop or the tab pauses it too**, because the heartbeat simply stops. That is a free side effect of heartbeat-gating rather than timing mouse events server-side, and it is the better behaviour.

## THE TRAP THIS DESIGN HAD TO AVOID, and it is not obvious
- Polls are scoped `since_time:TOPSEEN[cid]`, and `TOPSEEN` only advances when a poll runs. **So a paused deck leaves `since_time` frozen, and the first poll after an overnight pause would ask for the entire gap and bill for all of it** - turning a credit-saving feature into a credit-spending one. `mark_active()` clamps every `TOPSEEN` entry forward to `now - resume_max_backlog_minutes` (default 30) on resume. **RULE: any pause on a since_time-scoped poller must cap the catch-up window, or the saving is handed straight back on resume.**

## Config keys added (config.json)
- `idle_pause_minutes` = **10** (0 disables the feature entirely)
- `resume_max_backlog_minutes` = **30**

## Verified live, not assumed
- Set `idle_pause_minutes` to 0.2 temporarily and restarted. With no heartbeat: `/api/stats` returned `paused: True, idle_sec: 62`, log printed `idle 0.2 min, no client activity: polling PAUSED, spending nothing`, and no `poll` lines appeared. One `GET /api/active` at 18:38:49 printed `activity seen: polling RESUMED, catch-up capped at 30 min` **in the same second**, and stats flipped to `paused: False`. Threshold restored to 10 and the service restarted. Credits at 1,155,352.
- `python3 -m py_compile xdeck.py` passes; the new JS block passes `node --check` (node is NOT installed on the droplet, it was checked in the Cowork container instead).
- Backups: `xdeck.py.bakidle-*`, `index.html.bakidle-*`, `config.json.bakidle-*`. Patch script kept at `/root/xd_idle.py` (idempotent, asserts on every anchor and refuses to run twice).

# 14 Sep 2026. TWO BOTS SHUT DOWN: PAIREX MONITOR and RH EARLY BIRDS.

- [stated] Ben: "turn off the pairex monitor completely" and "shut down the RH early birds alert completely".
- `pairexmon.service` (PAIREX MONITOR, `/opt/pairexmon/pairexmon.py`, the dev and platform watcher built for [[pairex-token]]) and `earlybirds.service` (RH EARLY BIRDS, `/opt/earlybirds/tracker.py`, early holder and early flipper wallets on Robinhood Chain) are both **stopped and disabled** as of 14:12 UTC.

## ⚠ THE STEP THAT MAKES "COMPLETELY" ACTUALLY COMPLETE, and it is easy to miss
- **`systemctl stop` + `disable` is NOT enough on this droplet. `watchdog-bots.service` (`/opt/watchdog/watchdog.py`) holds a `UNITS` dict and runs `systemctl restart <unit>` on anything in it that reads inactive, every 300 seconds.** Stopping a bot without removing its `UNITS` entry means the watchdog brings it back within five minutes, and then fires a restart-storm alert to Telegram about it.
- **STANDING PROCEDURE for turning any fleet bot off: (1) delete its line from `UNITS` in `/opt/watchdog/watchdog.py`, (2) `py_compile` the file, (3) `systemctl stop` + `systemctl disable` the unit, (4) `systemctl restart watchdog-bots`, (5) confirm the watchdog's startup line prints a LOWER unit count.** The count is the receipt: it went 26 -> 25 (pairexmon) -> 24 (earlybirds), logged in `/opt/watchdog/watchdog.log` as `watchdog starting - N units, check every 300s`.
- **`systemctl mask` FAILS on this droplet's bots: "File /etc/systemd/system/pairexmon.service already exists."** Mask wants to write a `/dev/null` symlink at that exact path and the real unit files live there, not in `/lib`. Masking would mean moving the unit file, which makes restoring harder for no gain. **Disable plus the watchdog removal is the complete answer: nothing starts it at boot, nothing restarts it on a timer, only a manual `systemctl start` can.**

## What was left in place deliberately
- **Nothing was deleted.** `/opt/pairexmon/` and `/opt/earlybirds/` keep their code, config, logs and `state.json`, so either bot is one `systemctl enable --now` plus one line back in `UNITS` away from running again.
- `CREDIT_SOURCES` in watchdog.py still lists `/opt/pairexmon/config.json` as a fallback place to read the shared twitterapi.io key. Harmless, the file still exists, and it is only a key lookup. The credit-alarm message text was updated to stop naming pairexmon as a live watcher.
- Backups: `/opt/watchdog/watchdog.py.pairexoff-20260914-141050` and `.ebofF-20260914-141149`. Patch scripts `/root/pairex_off.py` and `/root/eb_off.py`, both assert their anchor appears exactly once and refuse to run twice.

## Checked and clean
- No cron entry for either bot. No other systemd unit references them (only their own unit files under `/etc/systemd/system/`). The `/opt/casig/*` grep hits for "pairex" are unrelated test fixtures and cached run data, not the monitor.
- Last line written by each before stopping: pairexmon `14:10:23 watcher issuer 8.7s`, earlybirds `13:38:24 tx 0x765cfb432f -> transfer (no-dex-evidence)`.
- twitterapi.io credits healthy at the time: **1,996,518 recharge + 133,918 bonus**. The watchdog's `credits alarm sent, balance 1951` at 10:45 UTC that day was a genuine near-empty reading before Ben topped up, not a bug.

# 14 Sep 2026. PIXEL ALERT built, and a THREE-BOT SILENT OUTAGE found while building it.

- [stated] Ben: "Build a separate Telegram alert bot for him. We call it Pixel Alert. It is in a separate bot." Target is @spacepixel, see [[spacepixel-wallet]].
- [stated] Ben, mid-build: "I stopped and deleted some of these bots, so you need to make sure that they are ready and available in my Telegram."

## ⚠⚠ THE FINDING THAT MATTERS MOST TODAY: A BLOCKED TELEGRAM BOT IS INVISIBLE TO EVERY HEALTH CHECK IN THIS FLEET
- **`getMe` passes on a blocked bot. `getChat` ALSO passes on a blocked bot, returning HTTP 200.** Only an actual send fails, with `403 {"ok":false,"error_code":403,"description":"Forbidden: bot was blocked by the user"}`.
- Every bot in the fleet wraps its own send in try/except and swallows the error (`tracker.py` line ~421, and watchdog's own `tg()` logs and returns False). **So a blocked bot is `active` in systemd, writing a healthy log, green in botctl, green in the watchdog, and delivering nothing at all.**
- **A first pass using `getChat` reported all 20 fleet bots LIVE and was completely wrong.** The correct probe is **`sendChatAction` with `action=typing`**: it posts nothing visible to the user and fails with exactly the same 403. Use it, never `getChat`.
- **THREE BOTS ARE BLOCKED, found 14 Sep 16:47 UTC:**

| bot | services it silently kills |
|---|---|
| **@xalertben_bot** | **coinbench, runwatch, parupdate, x-alert** — all four LIVE and shouting into the void |
| @rhearlybird_bot | earlybirds (already retired today, harmless) |
| @bk_pairexmon_bot | pairexmon (retired today), and now pixelalert |

- **@xalertben_bot is the real outage: four running services whose alerts have been going nowhere.** Only Ben can undo it; a bot cannot unblock itself. He has to open each bot in Telegram, press Unblock, then Start.
- The other 17 tokens are confirmed deliverable: @OxUberM_bot, @OxUberMendes_bot, @bk_alfiealert_bot, @bk_ansem_tracker_bot, @bk_chjokka_bot, @bk_dougalert_bot, @bk_ghostenv_bot, @bk_grailalert_bot, @bk_gramwatch_bot, @bk_linkscan_bot, @bk_nostradamus_bot, @bk_prmwatch_bot, @bk_rhwatch_bot, @crash_accum_bot, @crashtonbuy_bot, @dinerodom_bot, @kookwatch_bot.
- Standing scripts: `/root/tgcheck2.py` (probe every token found under `/opt/*/` with sendChatAction, writes `/root/tgcheck2.json` and `.txt`), `/root/tgshow.py`. **Do not resurrect `/root/tgcheck.py`, it uses the broken getChat probe.**

## THE MECHANICAL FIX, per Ben's repeat-error rule: the watchdog now catches this by itself
- **`check_blocked_bots()` added to `/opt/watchdog/watchdog.py`**, called from the main loop right after `check_credits()`, backup `watchdog.py.blockbak-20260914-*`, patch `/root/wd_blockcheck.py` (asserts its anchors, `ast.parse`s before writing, refuses to run twice).
- Scans every `\d{8,10}:[A-Za-z0-9_-]{30,}` under `/opt/*/` (skipping `.bak` and `RETIRED`), probes each with `sendChatAction`, and on any 403 sends one card naming **the bot AND every service it silently kills**, with the fix ("open the bot in Telegram, press Unblock, then Start").
- Runs at most every 6 hours, remembers the previous blocked set so it alerts on CHANGE only, and sends a recovery card when the set empties. **Reports through the watchdog's own @crashtonbuy_bot, which is the one bot that has to stay unblocked for any of this to work.** That dependency is worth stating to Ben: if he ever blocks @crashtonbuy_bot, nothing can tell him anything.

## PIXEL ALERT itself
- **`pixelalert.service`, `/opt/pixelalert/`, log `/opt/pixelalert/pixelalert.log`, active and enabled 16:52 UTC.** Registry 27 -> **28 bots**. Hooked into watchdog `UNITS` right after alfiewatch.
- **Built by cloning `/opt/alfiewatch/`**, not written fresh. `tracker.py` (1,075 lines) is shared byte-identical by alfiewatch, grailwatch and chjokkawatch and is the proven multi-wallet Robinhood Chain buy/sell watcher, and it ships with `pnl.py`, `backfill.py`, `report_pnl.py`, `bookreport.py`, `settoken.py`. **Cloning a live watcher is the right move for any new wallet bot on this chain; do not write a new one.**
- **`tracker.py` hardcodes its own name in the startup line ("ALFIE ALERT starting") even though the card header reads `config["tag"]`.** Patched to PIXEL ALERT, backup `tracker.py.brandbak-*`, script `/root/pixel_brand.py`. **Check for a hardcoded bot name on every future clone.**
- Config: tag `PIXEL`, `min_usd_buy` / `min_usd_sell` **25** (deliberately lower than alfiewatch's 50, because candidate A trades in a few hundred dollars at a time and 50 would silence most of it), `poll_seconds` 3, transfers off, settlement contract `0xb92fe925dc43a0ecde6c8b1a2709c170ec4fff4f`.
- **Watches ONE wallet: `0x1050cecf1eec272e8ecd5e0dbfa8599407b7e416`, labelled "PIXEL A (best fit)" and marked unconfirmed in its note**, because [[spacepixel-wallet]] did not close the identification.
- Built watching two, but [stated] Ben at 17:13 UTC: "Remove Pixel B wallet. I only care about the best-fit wallet." `0x3e32b434a9d4ad1853244bec120244bdb56b8b33` removed from the wallets list, service restarted, config backup `config.json.bak-dropb-20260914-*`, script `/root/pixel_dropb.py`.
- **Telegram: it reuses the @bk_pairexmon_bot token**, free because Ben retired pairexmon the same day. That gives a genuinely separate bot and a separate thread with no BotFather step, at the cost of the handle still reading `bk_pairexmon`. Renaming the display name (not the @handle) is a BotFather `/setname` away.
- **DELIVERY CONFIRMED 17:0x UTC.** Ben unblocked it; a real `sendMessage` returned `ok: true, message_id 123`, then the live card went out as 124. Re-probe of the whole fleet: **18 live, 2 still blocked (@xalertben_bot and @rhearlybird_bot)**.
- ⚠ **`/root/tgcheck2.py` takes about 100 seconds to walk 20 tokens (two API calls each, no concurrency), so a 50-second `nohup ... & sleep 50` read returns the PREVIOUS run's file and reports stale results.** It did exactly that here and contradicted a send that had just succeeded. **Delete `/root/tgcheck2.txt` and `.json` before launching, then poll for the file to reappear.** General rule for any background check on this droplet: verify by the output file's mtime, never by elapsed sleep.

## ⚠ pairexmon CAME BACK after being disabled, and it is not the watchdog
- Found `active` again at 16:53 with `ExecMainStartTimestamp 15:28:41`. Journal shows it stopped at 15:08, started 15:14, stopped 15:20, started 15:28 — **all long after the 14:12 shutdown, and it is NOT in watchdog `UNITS` any more** (verified). Something or someone else started it three times. Stopped again 16:56; still `disabled`, so it will not survive a reboot.
- **This matters beyond pairexmon: it shares the token Pixel Alert now uses, so a running pairexmon posts into the Pixel Alert thread.** If it reappears, find the starter before doing anything else.