---
name: trench-model-6
description: Price Predict part 6, from 15 Sep 2026 15:30 UTC after trench-model-5 neared its cap. Gate A's fourth defect (holes in the tape), the settlement starvation I caused with the tape cache, the self-sizing settle budgets, and the stable repaired ledger at 3x sample. Read with trench-model-5.
sources: [cowork]
aliases: [trench model part 6, price predict 6, patch204, patch205, patch206, patch207, patch208, kl cache, gate A]
---
Continues [[trench-model-5]]. Everything there still applies.

# GATE A IS DONE. FOUR DEFECTS, NOT THREE.
Fixes 1 to 3 (corrupt `high`, look-ahead anchor, dust prints with no volume) are in [[trench-model-5]]. Tripling the cache exposed a fourth.

## FIX 4: A TAPE WITH A HOLE IN IT IS NOT A CONTINUOUS PRICE
Going from 782 to 2,356 Solana positions tripled the returns, which is the signature this project has learned to distrust. It was one row:
```
2qP1Avi76Kvktu19xFJ4B1GAoDYx8jCgJEf8Pz3HS2vh, entry $900,924.15
  the ENTIRE tape after entry is four candles:
    +1008s   $429,401,603
    +5208s   $628,146,980
    +9288s   $892,208,024
    +9828s   $2,855,073,692,403
  obs witness: 105 readings, maximum $1,271,192
```
**23.8% of the whole book in one position.** The coin did not trade for seventeen minutes after we bought it and the next print is 340x the obs maximum: the price basis changed across the hole. **The volume rule passes it, because four candles ARE their own median volume.**

Two guards, both asking the same question:
- **MAXGAP (600s).** A candle is usable only if the previous usable candle was within ten minutes. At the first larger hole the tape ENDS and the position marks out at the last price before it. A position you cannot sell for ten minutes is not one you can claim a price for, and nothing guarantees the basis survived the gap. A real run trades continuously, so this deletes nothing that was running.
- **FIT_SPREAD (4x).** The scale is the median of obs_mcap / kline_close over matched timestamps. If those ratios disagree among themselves by more than 4x (p90 over p10), the basis was not constant over the overlap and the row cannot be priced at all. Counted and excluded, not quietly scaled by a median of two incompatible regimes.

Concentration after the guards: top single row **2.6% (SOL)** and **2.5% (RH)** of the book, from 23.8%.

# THE STABLE REPAIRED LEDGER, ~3x the first sample, untrimmed
Range is closes-only (honest) to candle-high fills (optimistic).
| exit | Robinhood 3,568 pos / $153,944 | Solana 2,161 pos / $201,669 |
|---|---|---|
| A hold 1h | $63.25 - $65.39 | $84.56 - $84.86 |
| B sell all at 60s | **$86.33 - $104.41** | **$97.54 - $119.58** |
| C 2x or -50%, else 10m | $81.53 - $87.52 | $89.77 - $103.01 |
| D half at 2x, rest at $500k | $76.49 - $79.86 | $90.71 - $99.64 |

Bars climbed to (entries already above a bar excluded from that bar):
| bar | Robinhood | Solana |
|---|---|---|
| $100k | 2.50% (2.04-3.07) | 5.37% (4.47-6.43) |
| $500k | 0.25% (0.13-0.48) | 0.91% (0.59-1.42) |
| $1M | **0.14%, 5 of 3,567** | **0.77%, 16 of 2,088** |

**Every figure is within a few dollars of the run at a third the sample, with a new class of defect removed in between. That is the strongest evidence this project has produced, and the conclusion is unchanged: NOTHING CLEARS BEN'S $110 HURDLE ON THE HONEST FILL.** B and C clear $100 only on the optimistic one, and the width of that band is entirely the intra-candle ordering, which one-minute candles cannot resolve.

# I STARVED SETTLEMENT WITH MY OWN BACKFILL. patch206
The tape cache was given `gmgn.RESERVE = 5.0`. Settlement runs at 6.0. **On this limiter a HIGHER reserve means LOWER priority**, so the backfill had a better claim on the budget than the job it exists to serve:
```
[sol] settle[paper]: 3 priced, 0 peaks corrected upward, 18820 still to do
UNHEALTHY: solana/paperx settle queue  342 waiting  *** FALLING BEHIND ***
```
Cache moved to 10, the documented backfill tier. Settlement recovered to 139 / 76 / 48 rows a pass.

**The real fix: the two jobs were fetching the SAME kline twice.** Settlement pulls a full tape for every row it prices, at exactly the two moments that tape is longest (70 minutes and 25 hours), and was throwing it away after one use. `settle.py` now writes every tape it fetches into the cache (separate database file, every write wrapped so a cache failure can never cost a settled row, `ON CONFLICT ... WHERE excluded.n >= kl.n` so a shorter tape never overwrites a longer one). The cache went from 1,281 to 3,210 Solana rows in half an hour on settlement's fetches alone.

# patch207: SETTLEMENT BUDGETS NOW FOLLOW THE QUEUES
patch197 hardcoded 200s to `paper` and 40s to `paperx`, sized to the queues that day. The queues moved and the numbers did not: Robinhood paper had 23 rows waiting and 200 seconds, while Robinhood paperx had 162 and 40. The split is now COMPUTED at the top of every tick from the live queue depths, per chain, proportionally, with a 40s floor per book. Measured immediately after: RH 78/162, SOL 180/60. **Same failure class as boxlift keeping its own box list: a measurement taken once and then trusted forever.**

# patch208: THE FOURTH TIME A GATE CHECK BROKE ON A CORRECT FILE
GATE RED on "settlement runs one writer per database". Two ways patch200's own shape-based rewrite still broke:
1. `_sh.count("leg env") == 2` - patch207 gives each chain its database path, so the call sites read `leg /opt/trench/trench.db env &` and the literal "leg env" appears zero times. **The check was asserting a spelling again while its own comment said to assert the shape.**
2. `_sh.split("leg() {")[-1].split("}", 1)[0]` - patch207's leg() contains `${s% *}`, so the parser stopped at that brace and found no SETTLE_TABLE lines.

Fixed on both sides: leg() no longer uses brace expansions (`read -r pa px < <(split "$db")`), and the check reads the body to a `}` ALONE ON A LINE and counts BACKGROUNDED CALLS to leg rather than one spelling. **patch111, patch120, patch200, patch208: four times. The rule is not "assert the shape", it is "the check must fail only when the INVARIANT is violated, and every parser you write is a spelling".**

# NEVER OVERWRITE A SHELL SCRIPT THAT MAY BE RUNNING
After patching settle.sh, the log showed `line 47: mport: command not found` and a syntax error, while `bash -n` on the same file passed. bash reads a script incrementally and resumes at its previous BYTE OFFSET in the new file, so the running pass landed mid-heredoc. Self-correcting on the next invocation, and alarming for the ninety seconds it takes to work that out. Write to a temp file and `mv` it into place.

# patch209: THE PAPER BOOK'S FOUR LEGS RAN IN SERIES INSIDE A 300s TICK
`UNHEALTHY: solana paper entries  35 min ago *** STALE ***`. paper.sh took the shared writer lock and ran Robinhood early, Robinhood late, Solana early, Solana late one after another: `timeout 280 + 200 + 280 + 200` is 960 seconds of worst case inside a 300 second cron, **with Solana always last**. 190 of the last 691 lines in /tmp/paper.log were "previous run still going, skipping" - 27% of ticks never started.

**This is patch151 and patch198 a THIRD time.** patch151 fixed exactly this shape for settlement ("four 200s slots need 800s and Solana paperx was always the one that never ran") and paper.sh was never given the same treatment. patch198 established that **entries are the one thing that cannot be caught up later**, because the book buys at 60 to 240 seconds of a coin's life and that window does not come back.

Fix, the same shape settlement already has: the two CHAINS run in PARALLEL, each under its OWN lock (`/tmp/trenchw-rh.lock`, `/tmp/trenchw-sol.lock`), and the two LEGS within a chain stay in series (same file, and late.py needs paper.py's minute-one row to exist first). The shared `/tmp/trenchw.lock` stays for the hourly batch jobs (rollup, scoreall, daily) and **the book no longer takes it at all**, so a 39-minute `paper.py --rescore` can no longer stop entries. Verified live: both per-chain locks appear, two paper.py processes run together, Solana now reports FIRST in the log instead of last, both chains 5 minutes fresh.

**Rule: whenever a cron chain's worst case exceeds its own tick, the last job in the chain does not run slowly, it does not run.** Check the arithmetic on every serial runner: sum the timeouts and compare to the interval.

# THE WITNESS SPLIT: DOES THE RESULT REST ON ROWS NOBODY CAN CHECK?
The scale is fitted against the obs tape where the two overlap and falls back to a pre-entry kline anchor where they do not. That fallback row has nothing checking it. From the audit at 4,487 Solana positions:
- `AFZhFM8GRAfDGXsKJAqk55jq7EFjQtqgxvmyMrb9pump`, entry $7,734.31, two candles, first at +22s reading $750,937 (a 97x in twenty-two seconds), **obs witness: 0 readings.** 2.0% of the book on an 85x nobody can confirm.
- Beside it in the same audit, `6NTPz7iSpz1b363uT8TtRvH84aYxR68EEkGHgJN2Urew`, entry $453,412.31, kline maximum $54,409,782, **obs witness maximum $57,886,498 - the board saw it too.** A real 120x.

Rather than invent another threshold, `FIT_ONLY=1` drops every row whose scale could not be fitted against obs, and the ledger is run BOTH ways. **Restricting to witnessed rows moves every figure UP, not down**, so the result does not rest on unverifiable rows: Solana all rows 4,487 / witnessed 3,785, A $91.46 -> $92.30, B $99.86 -> $100.60, C $93.38 -> $96.46, D $96.12 -> $98.21, $1M 0.55% -> 0.63%.

# STABILITY AT 16:40 UTC, roughly double the earlier sample
| exit | Robinhood 4,256 pos | Solana 4,487 pos | Solana witnessed only, 3,785 |
|---|---|---|---|
| A hold 1h | $63.00 - $64.88 | $91.46 - $91.76 | $92.30 - $92.71 |
| B sell at 60s | $86.62 - $104.88 | $99.86 - $124.63 | **$100.60 - $126.58** |
| C 2x or -50%, else 10m | $81.63 - $88.00 | $93.38 - $105.37 | $96.46 - $110.04 |
| D half at 2x, rest at $500k | $76.45 - $79.82 | $96.12 - $104.41 | $98.21 - $107.45 |

**Robinhood has not moved a dollar across three runs and a 2.4x sample growth** ($63.25 -> $63.00, $86.33 -> $86.62, $81.53 -> $81.63, $76.49 -> $76.45). Solana rose a few dollars as its sample doubled and the top row is now a confirmed runner at 3.0% rather than a defect at 23.8%. $1M climbed to: RH 0.12% (5 of 4,255), SOL 0.63% witnessed (23 of 3,635). **Still nothing at $110.**

# patch210: THE ENTRY WRITE WAS BARE. A LOCK KILLED THE WHOLE ENTRY PASS.
/tmp/paper.log carried this repeatedly:
```
[robinhood] paper: 67 new entries, 67 scored, field n=1434
    c.execute(
sqlite3.OperationalError: database is locked
```
An AST sweep of both entry paths (walk the tree, find `execute`/`commit` calls whose SQL starts with a write verb, check whether any ancestor is a `Try`) found every write in the ENTER path bare: `paper.py` line 706 INSERT and lines 722/723 commit, `late.py` line 249 INSERT and lines 265/266 commit.

patch154 set the rule for the SCORER ("a lock must cost ONE row, not the rest of the pass"), patch157 applied it to the late scorer and patch192 to settlement's dead-coin branch. **Nobody ever applied it to enter().** That is the worst possible place for it: a missed settlement is redone next pass, but the book buys between 60 and 240 seconds of a coin's life and **a missed entry is gone forever**. Wrapped; INSERT OR IGNORE is idempotent so a skipped coin still inside its window is entered next pass. Verified: `[robinhood] paper: 1 rows skipped on a locked database` replaced the traceback.

**The AST sweep is the reusable tool here.** Grepping for bare writes by eye had already missed this three times.

# FIX 5: A CENSORED HORIZON WAS BEING RESOLVED IN OUR FAVOUR EVERY TIME
```
FcjHXfhiAgdMkbb6iZnmmxH3G8pSuVxrRHS9vdB9wCGL, entry $673,226.72
  the whole tape after entry is three candles:
    +545s   $54,017,046      +665s   $81,696,244      +725s   $166,915,020
  OBS WITNESS: 3 readings over 84 seconds, maximum $720,209
```
Booked as **247.93x, 3.7% of the Solana book**. A coin genuinely worth $166,915,020 trades every minute; three candles in an hour means the kline is not showing that coin's tape.

**The asymmetry is the real defect.** `at(H)` falls back to the LAST close when the tape ends before the horizon. For a coin that died that is correct and conservative. For a coin whose tape ends high it hands us the high, and **tapes do not end at random: a coin that migrates or changes pool stops returning candles exactly when it runs.** Every censoring error was in our favour. Same shape as the ripeness bias round three found, in a new place.

Rule: a horizon is CENSORED when the last usable candle sits more than MAXGAP before it, and a censored horizon is marked at `min(last kline close, obs maximum)`. It does not touch real winners because obs is what confirms them: `6NTPz7iSpz1b363uT8TtRvH84aYxR68EEkGHgJN2Urew` has a kline maximum of $54,409,782 against an **obs maximum of $57,886,498**, higher than the kline, so it keeps its 120x in full.

**The effect has exactly the right signature: it hits the long horizons and leaves the short ones alone.**
| | before | after |
|---|---|---|
| SOL A hold 1h | $93.88 | **$87.10** |
| SOL B 60s | $100.63 | $100.50 |
| SOL C | $92.96 | $92.90 |
| SOL D | $96.59 | $94.49 |
| RH A hold 1h | $62.52 | $61.88 |
| RH $1M climbed | 0.11% (5) | **0.07% (3)** |
Two of Robinhood's five "$1M" rows were censored tapes.

# THE LEDGER HAS NOT MOVED ACROSS SIX CHECKS AND A 3x SAMPLE. 16 Sep 06:50 Bangkok
Robinhood 5,339 positions, Solana 14,164, roughly triple the sample the censoring fix was measured on. Closes-only, per $100:
| exit | Robinhood | Solana |
|---|---|---|
| A hold 1h | $61.22 (was $62.76, $62.07) | $84.81 (was $86.12, $86.00) |
| B sell at 60s | **$87.71** (was $87.26, $87.33) | **$100.27** (was $99.06, $99.37) |
| C 2x or -50% else 10m | $81.72 | $91.00 |
| D half at 2x | $76.x | $92.33 |
$1M climbed to: RH 0.09% (5 of 5,338), SOL 0.64% (88 of 13,827). Top single row down to 1.7% (RH) and 1.9% (SOL).

**Every figure sits inside its earlier interval. The conclusion has survived a tripled sample, five defect fixes and six independent re-runs: nothing clears the $110 hurdle on the honest fill, and B is the only exit near $100.** patch210 confirmed working: 281 log lines and several hours with zero tracebacks, locks now printing "N rows skipped" instead.

Watch item, not yet a problem: Solana's `pre` route (rows with no obs witness) rose from 20.3% to 25.9% as settlement's own fetches filled the cache with tokens obs never covered. `FIT_ONLY=1` prices the witnessed subset and has always read HIGHER, so the drift is not flattering the result, but if `pre` keeps climbing the witnessed-only run becomes the number to quote.

# THE COLLECTORS ARE STARVED. THE DROPLET HAS ONE CPU. 16 Sep 14:05 Bangkok
`UNHEALTHY: solana observations`. Chased it and found something bigger than the alarm.

**`nproc` = 1.** kookwatch is a single-vCPU droplet running two Price Predict collectors, settlement, the paper book, the late leg, callbook, casig, xwatch, newborn, the rugscan and rhx crons, 17 alert bots, and whatever analysis I run. Load average **12.3 on one core, 0.0% idle**.

Both collectors target a 10-second poll (360/hour). Measured distinct obs timestamps per hour:
| | Robinhood | Solana |
|---|---|---|
| 15 Sep, daytime | 111-148/h | 150-260/h |
| 16 Sep 07:00 UTC | **~36/h** | **~24/h** |
Solana went from a 10-second cadence (measured early in this session: p50 gap 10s) to **gaps of up to 635 seconds and 1 to 2 polls per five minutes.**

**This is not a bug, it is capacity**, and it matters more than any number in the ledger: obs is the ground truth for the 60-to-240-second entry window AND for the price-to-market-cap scale fit. It also explains the Solana `pre` route (rows with no obs witness) climbing 20.3% -> 25.9% -> 27.5% -> 29.0%: **fewer obs readings, fewer rows with a witness.** I had been treating those two facts as unrelated.

**My share of the blame, owned: it was ALL mine.** I ran the kl cache fetcher continuously plus three or four pp.py passes and a gate_paper run every check-in, for seventeen hours, on that one core. Stopped: klrun.sh killed, the klcache cron removed (settlement still feeds the cache for free via patch206, and it already holds 25,514 Solana and 7,276 Robinhood tapes, which is plenty).

**AND I THEN MISREPORTED THE FIX, BECAUSE I MEASURED FIVE MINUTES AFTER IT.** At +5 min the poll rate had not moved and the load had barely moved, so I told Ben "I was making it worse rather than causing it". Two hours later, with nothing of mine running:
| | at the low | two hours after I stopped |
|---|---|---|
| Solana | ~24/hour | **234/hour** |
| Robinhood | ~36/hour | **150/hour** |
| load | 12.3 | 8.98 |
A ten-fold recovery on Solana. **I was the cause of the acute collapse, not a contributor to it.**

**The rule: on a box with a run queue twelve deep, a measurement taken five minutes after an intervention is measuring the QUEUE, not the intervention.** Wait for the backlog to drain before concluding anything, and say "too early to tell" rather than guessing. Exactly the same error class as everything else in this file: a number read before it meant anything.

**What remains true after the correction, at a smaller size:** one vCPU is still a real constraint. Both chains sit at 42% (Robinhood) and 65% (Solana) of the 360/hour target, which is roughly where they were throughout Gate A, so the ledger stands. Resizing would buy better coverage for Gate C; it is no longer urgent or blocking.

Top consumers on the one core: `casig.py` 26% continuous, `paper.py --rescore` 13%, the two `trench.py` collectors 7% and 5%, `callbook.py` 7%. Aggregate CPU is only ~65% while load is 11.5, so most processes are BLOCKED rather than running: single-core disk and lock contention, not compute.

**Two levers, both Ben's call because both cost something:** resize the droplet past one vCPU, or turn services off. **Gate C cannot be run honestly on a collector polling at 7% of its intended rate**, so this is now the blocking item, ahead of picking a candidate to freeze.

**STANDING RULE THIS EARNED: never run the analysis on the box that collects the data, and check `nproc` before assuming a machine can absorb a background job.** Eight consecutive flat check-ins produced no new information and cost the live system real coverage. Monitoring that damages what it monitors is worse than not monitoring.

# THE "SETTLEMENT IS FALLING BEHIND" ALARM WAS READING A SAWTOOTH. 17 Sep 14:00 Bangkok
Ben forwarded the Telegram card: `solana/paper settle queue 211 waiting *** FALLING BEHIND ***`, whose own text guesses "usually the shared GMGN budget is exhausted".

**Every part of that guess was wrong, and checking beat guessing three times over.**
- NOT the budget: limiter showed tokens 6.0, until 0, **consec 0**, no ban.
- NOT the priority order (my first suspect, because patch194 was exactly that bug): the queue splits **18 final-horizon, 90 never-settled, 12,064 hourly-refresh**, so the urgent work is small and sits at the FRONT of `due()`.
- It is a **SAWTOOTH**. Sampled every 33 seconds through a pass: **107, 37, 9, 9, 19.** Solana takes ~1,000 rows an hour past the 70-minute mark, about 85 arrive between five-minute passes, and a pass clears 280. Capacity beats arrivals better than three to one. The ten-minute health tick just lands wherever it lands on the teeth.

## patch212: judge the FLOOR, not the sample
`_grew = now > 200 and now > (sample an hour ago) * 1.15` compares two random points on a sawtooth. A queue is falling behind when its **floor** rises, because the floor is what a pass failed to clear. Now: `min(last 3 ticks) > 200 and min(last 3) > min(first 3) * 1.15`, and the printed line shows the floor. On the sequence that fired (96,110,195,141,156,211) the floors are 96 and 141, so it no longer fires; a real backlog still does, one tick later at worst. Immediately after: `solana/paper 32 waiting, floor 32, ok (floor was 141 an hour ago)` and **ALL HEALTHY**.

## patch211: settlement already had the tape and was buying it again
patch206 made settlement WRITE every fetched tape to kl.db. It never read them back. Measured on the live overdue queue: 174 rows, 89 with any cached tape, **71 with one fetched at or after the horizon being settled**. So 41% could be priced with no API call.
`series()` now takes `need_ts` and uses the stored tape when it was fetched at or after the moment the horizon needs, converting the stored `[t,o,c,h,l,v]` rows back into the shape `gmgn.kline` returns so both paths run the SAME parser. Every cache read wrapped, patch206's rule. patch211b prints the split with the pass summary.
Result: Solana paper went **280 priced -> 697 priced (569 cached, 128 fetched)** in one pass, and Robinhood paperx 212 priced with 15 fetches. **This REDUCES load on the one-vCPU box rather than adding to it**, which after 16 Sep is the only kind of fix worth shipping here.

## patch211c: MY OWN BUG, caught by the counter I had just added
`need = min(ts + STAGE1, now)` is in the PAST for a refresh row (settled=1, under 25h), so **the hourly refresh started serving itself the stage-one tape** - which is precisely the censoring patch142 and patch194 removed. The refresh exists to catch a coin that ran AFTER stage one.
The log said so within one pass:
```
before:  280 priced,  42 peaks corrected upward   (15.0%)
after:   697 priced,  22 peaks corrected upward   ( 3.2%)
```
Five times fewer upward corrections on 2.5x the rows. Rule restated properly: **a stored tape is allowed only when the horizon has already CLOSED.** Final (age>=25h) -> `ts+STAGE2`; first settle (never settled) -> `ts+STAGE1`; **refresh -> `now`, so it always fetches.** Verified: `[sol] settle[paperx]: 27 priced (0 cached, 27 fetched), 3 peaks corrected upward`.

**STANDING RULE: when you add a fast path, add the counter that proves how often it fires IN THE SAME PATCH.** Without patch211b's "(N cached, M fetched)" the correction-rate collapse would have looked like a quiet week, and a five-fold drop in upward peak corrections is exactly the kind of silent pessimism this project has been fighting all along.

# STATE, 15 Sep 2026 22:40 Bangkok
- GATE GREEN at 96 checks. One NOTE (the launchpad box is selective, which is correct, see patch205 in [[trench-model-5]]).
- Cache: Robinhood 4,205 tapes, Solana 3,391 and climbing on settlement's own fetches.
- Brief published at **v6** with section 0.9 as the headline.
- Solana paper settle queue still ~7,900 and Solana paperx ~1,070: large, pre-existing, and the reason Ben chose KEEP_RATE=0.5.

# STANDING RULES THIS ROUND EARNED
- **A tripled sample that triples the return is a defect until proven otherwise.** Audit the top contributors before believing any improvement.
- **A guard computed from a token's own data fails on tokens with almost no data.** The volume rule uses the median of the token's own candles, so four candles pass it trivially. Every self-referential threshold needs a minimum-sample condition beside it.
- **Check the priority direction before giving a background job a share of a shared budget.** On this limiter a higher reserve is a LOWER priority, and I read it backwards.
- **When two jobs fetch the same thing, make one of them keep it.** The fix for contention was not scheduling, it was removing the duplicate work.