# PP Feeds status (8 Oct 2026, 19:45 Bangkok, handover)
To-do (incl. parked items): http://159.223.88.168/hx-c9ois/ppfeeds/todo.md
Owner: PP Feeds (live data that bots and paper books read). History and research data: PP Data, see ../ppdata/ppdata-status.md
Live guard view (JSON, every 20 s): http://159.223.88.168/hx-c9ois/ppfeeds/guard2.json

## Feeds and who guards them
| Feed | Where | Heartbeat limit | Layer 1 (same box, every 10 s) | Layer 2 (other box, every 20 s) |
|---|---|---|---|---|
| v3live (v3 + Pons grad swaps) | research-box | 90 s | ppfeedguard restarts it | feedguard2 on kookwatch |
| pretag (crew pre-launch tags) | research-box | 90 s | ppfeedguard | feedguard2 |
| v3mc5 paper book | research-box | 180 s | ppfeedguard | feedguard2 |
| P3 a1feed (FW A1 early + cross lines) | research-box -> kookwatch mirror | 20 s | ppfeedguard | feedguard2 (also checks the mirror) |
| trending | research-box cron | 10 min | ppfeedguard runs it | feedguard2 |
| kitb | research-box timer | 25 min (kitb.ok) | ppfeedguard | feedguard2 |

- Every unit has systemd Restart=always and no start limit (crash -> back in seconds).
- A feed is judged from max(last heartbeat, unit start). After a restart it gets a grace window before another restart. This stops the restart death-spiral that kept v3live and pretag down for about 6 h this morning (old hbwatch restarted them every 2 min, before they could write a first heartbeat).
- Layer 1 and layer 2 restart each other. cron restarts each guard if its service stops or goes silent.
- Live feeds get CPU and IO priority over research (CPUWeight/IOWeight 1000).
- Tested 8 Oct: a frozen a1feed was restarted by layer 1 in 20 s. A frozen layer 1 was restarted by layer 2 in 65 s.
- Telegram: only when every layer has failed (a feed still stale 15 min after both guards acted, or research-box unreachable for 5 min), through alertgate. **Note:** alertgate's chat is muted by tgguard, so nothing reaches Ben yet. Waiting on Ben for the alert chat.

## RPC when paid sources are out
- v3live and pretag use /opt/sw/rpcfall.py: paid dRPC, then free dRPC, then the public node (direct, plus a second IP through the picker-box relay 10.104.0.4:8545). Alchemy was removed 8 Oct. A billing or quota answer parks that source for 60 s. The public routes are paced and slow down by themselves on 429.
- v3live reads block times from the logs (blockTimestamp), so it no longer makes one call per block.
- 8 Oct 16:20 to 17:02: dRPC was out of balance (Alchemy is still over its monthly cap), and live ran on the public node. Ben topped up dRPC at 17:02, and the feeds went back to paid by themselves.

## RPC outage 8 Oct: what was paused and what is left
- Research jobs that starved the live feeds of public RPC were paused 16:30 to 17:10 (chainlogs x4, hist-rh, gold_backfill_v3). ppfeedguard resumed them by itself at 17:10, once paid had answered for 5 min. Log: research-box /opt/sw/feedguard/paused.txt.
- v3live gap 83206609..83219980 (about 16:25 to 16:47 Bangkok): filled by v3gap, done 17:16.
- pretag gap 83203840..83219980 was skipped (no pre-launch tags for that window).
- v3live rows written 16:52 to 17:00 with time 0 (the node sends blockTimestamp 0x0) were repaired at 17:00: 1,713 swaps fixed, none left. v3mc5 read those rows while they were wrong, so its book may show odd ages for coins in that window.

## P3 (FW A1 request 8 Oct): live early-feature feed
- kookwatch /opt/launchwatch/pub/hx-c9ois/fw/a1feed/early.jsonl and cross.jsonl, append only. Source: research-box /opt/sw/a1feed/ (a1feed.service). Mirrored by a1pull@early / a1pull@cross (ssh tail, under 1 s).
- early: one line at +60 s and one at +300 s after the first swap, for every new v3 WETH pool. Lines start from pools first swapped after 8 Oct 16:15 Bangkok (the feed's epoch). All values are as of the check time. lag_s = seconds after the check time that the line was written, and src_stale=true means v3live was behind.
  - The launch-block buyer count is the number of buy swap logs in the launch block. A bundle buying for 11 wallets in 1 tx shows 11, with launch_block_buy_signers 1.
  - distinct_buyers counts distinct tx senders, so a bundle counts once. buy_swaps counts every buy.
- creator = the sender of the pool-creation tx. cross lines also carry pool_weth_at_cross: dust pools fake $1M+ caps at age 0, so filter on it.
- cross: the first time each pool's cap reaches $300k, $500k, $700k and $1M: cross time, age at cross, buy USD at or above the level so far (the crossing buy itself), total buys and sells, cap at cross.
- Latency: under 5 s when v3live is current. v3live's own lag is the limit while paid RPC is out.

## Parked by Ben 8 Oct 17:36
- Coin brain (AI) and deepread: stopped and disabled, crons off. X API (twitterapi.io): X-only services and crons off, and api.twitterapi.io blocked on kookwatch and research-box. Alchemy removed. Undo steps: /root/parked-2026-10-08/UNPARK.md.

## 60 s / 30 s picker hourly shadow (PP Data ask, 8 Oct 18:51)
- aipick.py: every scored coin's features go into scored.feats. The hourly shadow model (aimodel_h.pkl / aimodel30_h.pkl, from PP Data) books into its own aipaper_h.db / aipaper30_h.db (model champion_h); the champion and its DB are unchanged. Details: to-ppdata-shadow-done.md

## dRPC batch limit (8 Oct 19:25)
- The dRPC key refuses batches of more than 3 ("free plan"). rpcfall splits batches and sends the parts in parallel. v3live and pretag keep pace, about 1 min behind. Ben is checking the key's plan; guard checks for these two stay at 180 s until it is fixed. pretag gap 83256245..83311754 skipped.

## Known issues
- v3mc5 leaks read connections to v3live.db (33 open). Harmless for now; watching it.