---
name: model-blind-tests
description: Scored out-of-sample tests of the Robinhood Chain price model, with the 1 Sep 2026 RETRACTION at the top. Read this before quoting any accuracy figure for the model, because most numbers from Tests 2 to 7 were inflated by a price-construction artifact and the honest figures are the CLOSE-TO-CLOSE ones below. Companion to [[rh-model-spec]] and [[screen-log]].
sources: [cowork]
aliases: [blind-test-log, model-scorecard, backtest-log]
---

# Model tests, Robinhood Chain

- [stated] Ben asked for a blind test on 31 Aug 2026 (four tokens he held), then for repeated rounds of testing and improvement, then on 1 Sep for a full recheck for key errors.

## THE RETRACTION, 1 Sep 2026. Read this before anything below.
**From Test 2 onward the entry price for every return was a 3-DAY VWAP (days O-2 to O), and the outcome was measured from it.** Buying pressure over days O-2..O pushes price up inside that window, so the 3-day VWAP sits below the end-of-day price, and the "return" to day O+1 or O+2 is positive mechanically. The 3-day imbalance feature is therefore correlated with the target BY CONSTRUCTION. The young cohort had the same defect at smaller scale (6-hour VWAP entry on a 24-hour horizon).
This is exactly why Test 2 appeared to "overturn" Test 1: Test 1 used a single-day price and found direction at 0.518; Test 2 switched to the 3-day VWAP and found 0.763. **Test 1 was right. Test 2 was wrong.** Everything that followed inherited the defect.

### What the defect was worth, same rows, same features, only the price definition changed
| target | 3-day VWAP entry (as reported) | day-O VWAP entry | CLOSE-to-CLOSE (last 2 UTC hours) |
|---|---|---|---|
| OLD up in 2d | 0.776 | 0.593 | **0.535** |
| OLD up in 7d | 0.685 | 0.596 | **0.596** |
| OLD touch +50% in 7d | 0.777 | 0.713 | **0.636** |
| YOUNG up in 24h (6h vs 1h entry) | 0.712 | | **0.654** |
| YOUNG touch +50% in 72h | 0.709 | | **0.673** |
| imb3 Spearman vs 2d return | +0.271 | +0.033 | **-0.036** |
| imbslope Spearman vs 2d return | +0.224 | +0.044 | **-0.015** |

**The "imbalance term structure", called the single biggest discovery of the project, was the artifact.** Every imbalance feature is dead close-to-close: imb1 -0.052, imb2 -0.047, imb3 -0.036, imb5 +0.002, imbslope -0.015, imbacc -0.036. Dropping the whole family costs 0.007 AUC.
**Death AUC (0.87 to 0.90) is unaffected**: death is "no trades", which does not depend on any price.

## THE HONEST NUMBERS, close-to-close, walk-forward, 6 test origins, OLD cohort (3,403 rows with a valid close, 1,644 out of sample)
| hold | up (direction) | touch +50% |
|---|---|---|
| 1d | 0.508 | 0.656 |
| 2d | 0.535 | 0.631 |
| 3d | 0.567 | 0.611 |
| 5d | 0.555 | 0.637 |
| 7d | 0.596 | 0.636 |
| 10d | 0.597 | |
| 14d | **0.645** | |
**Direction skill RISES with horizon.** Test 7's "half-life of three days" was the artifact decaying, not skill decaying. The honest curve runs the other way.
Market-relative target (does the token beat the origin's median return) at 7d: **0.613 vs 0.589 absolute**, +0.024, the one clean improvement found.

## THE 42 UTILITY TOKENS, close-to-close (Test 7 redone, 423 rows, 15 folds)
| hold | chain direction | chain touch | the 42 direction | the 42 touch |
|---|---|---|---|---|
| 1d | 0.587 | 0.634 | **0.501** | 0.509 |
| 3d | 0.549 | 0.624 | 0.484 | 0.541 |
| 7d | 0.589 | 0.648 | **0.438** | 0.482 |
| 14d | 0.618 | 0.679 | 0.477 | 0.562 |
**The model has NO measurable skill on Ben's 42 large utility tokens at any horizon.** The 0.772 one-day figure reported to him was entirely artifact. On the whole chain there is modest skill that rises with horizon.

## Category (Test 6) re-checked close-to-close
Payout-word tokens vs infrastructure-word tokens, 7-day up-rate: 61.9% vs 46.7%, **+15.2pp, permutation p = 0.158, n = 42 vs 60.** Test 6 reported +25.8pp, p = 0.0009. Directionally consistent, **not established.** Factory stocks still show near-zero touch (1.3%), which is real.

## Features that SURVIVE close-to-close (2-day return, Spearman within origin, t across 6 origins)
lhold +0.071 (t 8.1) | age +0.115 (t 7.1) | newshare1 -0.076 (t -3.7) | tak_per_hold -0.104 (t -3.7) | turn -0.114 (t -3.4) | lmc +0.075 (t 2.5) | hgrow7 -0.103 (t -2.5) | impact10k -0.110 (t -2.4) | ldepth +0.110 (t 2.4)
Slow, structural features: holder base, age, turnover, new-buyer share, depth. **Nothing in the trade-flow family survives.** Realised vol drops from -0.272 to -0.109 (t -1.9).
Young cohort, last-hour entry: rvol12 -0.139, ltak -0.128, depth_to_mc -0.099 survive; i3/i6/i12 go to -0.017/-0.050/-0.046.

## What survives from the whole project, in order of confidence
1. **Death is predictable, 0.87 to 0.90, and cross-sectional demeaning helps it (+0.025).** Unaffected by the artifact.
2. **The tail (touch +50%) is modestly predictable, 0.61 to 0.68, rising with horizon.** True pool depth is a real contributor (depth_to_mc survives on the young cohort).
3. **Direction is near coin-flip under a week (0.51 to 0.57) and reaches 0.65 at 14 days.** Use market-relative targets.
4. The young-cohort death spread (Q1 vs Q5) and the age-3h death call are likely real since death is price-independent; the age-3h TOUCH numbers were built on a 2-hour VWAP entry and are probably mildly inflated. Not re-run.
5. Conditional facts from Test 1 (single-day price, clean): at ATH with mom7 > +400%, 33% up at 7d; age-36h cohort 25.9% up at 36h; absolute first-36h volume is a negative signal.

## Test 1 (31 Aug, blind, clean prices): 2 of 4 correct on direction. DELTA down (right), GLD down (right), PONS mild down (wrong, +19%), TWO down (wrong, +34%, peak +101%). The GLD held was a copycat; real factory GLD is `0xc9a981fee1f9dec688bb123ccdecc63d0debfc4e`.

## Tests 2 to 7 (1 Sep): superseded. Their AUC figures (0.76 to 0.78 old direction, 0.72 young direction, 0.78 touch, 0.772 one-day on the 42) are NOT to be quoted. Their non-price findings stand: decimals bug (41 non-18-decimal tokens), winsorisation, `dex.pools` empty on robinhood, the true-depth route via PoolManager_evt_Initialize, the 203 factory stocks in `rwa_stock_factory_robinhood`, wallet quality and sniper block and deployer track record all dead, 5-minute bars add nothing over hourly.

## Standing rules, revised
- **Entry price for any return is a CLOSE (last 1 to 2 hours), never a multi-period VWAP. Exit price the same.** Any feature window must END at or before the entry price's window.
- Before believing any feature, re-run the target with a stricter price definition. If the rho collapses, it was the price, not the market.
- Report the AUC for the target actually claimed. Direction, touch and death are three problems.
- Winsorise means at p99, cap outcomes, join decimals, freeze forward calls before reading outcomes.