Hiss NN · 02
M0 → M6
Each milestone gates the next. The gates are the point: every one of them exists
because something downstream failed silently once, and a gate is the cheapest place to catch it.
Findings below are measured on this system, not quoted from papers.
| M | Builds | Gate |
|---|
| M0 | Sandbox + toy-game correctness | Exploitability → 0 on Kuhn/Leduc |
| M1 | Card-abstraction precompute | Bucket lookup is fast and stable |
| M2 | Abstracted six-max game + traverser | Known result reproduced on a 2p subgame |
| M3 | SD-CFR training run | Advantage loss falls; reservoirs fill; policy fits |
| M4 | Distill + export + serve | Served net matches the mixture it distilled |
| M5 | Evaluation and iteration | bb/100 vs realistic opponents, with intervals |
| M6 | Ten-pocket six-max game | Every bet size is both playable and visible |
M0 — Sandbox and the correctness gate
Build: an isolated environment (~/deepcfr, its own venv,
pip install open-spiel) with no database, no poker app and no contact with live
training. Then run OpenSpiel's tabular CFR, CFR+ and external-sampling MCCFR on
Kuhn and Leduc poker — toy games whose equilibria are known —
and confirm exploitability falls to ~0 in seconds. Then run the PyTorch deep_cfr.py on
Leduc and confirm the neural path approaches the tabular answer.
Why this is not optional
It is the only point in the entire ladder where the correct answer is known independently. Every
later stage is measured against itself. If the toolchain is wrong here you will not find out at M3 —
you will find out never, because a broken large run looks exactly like a hard problem.
What went wrong: a plateau that was not a ceiling
Leduc stalled around 1.4 exploitability and was read as the method's limit. It was
undertraining. With identical traversals, raising advantage-network steps from 64
to 250 was four times better; the cheap fix turned out to be the learning rate
rather than more steps — --adv-steps 250 --lr 1e-2 took it from 0.17 to 0.0719, with a
real ceiling nearer 0.085 by iteration 800. A plateau is a hypothesis, not a measurement.
Separately: regret matching goes pure (probability 1 on one action) at 73.6% of
infosets, because the advantage targets have negative mean. Network capacity, reservoirs
and architecture were each investigated and exonerated first. Suspect the target distribution before
the model.
M1 — Abstraction precompute
Build: a suit-isomorphism canonicaliser; equity-histogram computation
(E[HS], E[HS²]); mini-batch k-means with EMD to produce bucket tables —
169 preflop (lossless), ~1-2k per postflop street. Deliverable: a fast
(hole, board) → bucket_id lookup, in a form the live feature path can also use.
Storage is small and widely misjudged: the "10⁸-10⁹ infosets" figure describes the game's
strategic size, not a disk requirement — the entire reason for using a generalising network is that
the table is never stored. The precompute is ~0.3 GB permanent, ~4 GB peak if streamed in chunks.
What went wrong: more boards stopped helping
Solve quality against board count peaks at 24 boards and then declines — adding
boards past that made it worse, not merely slower. What paid was stratified board
selection (+8.4), i.e. choosing boards to span the texture space rather than sampling more
of them. Three separate "just scale it" hypotheses were falsified here.
Also: every solve for a period ran against the wrong villain range — 37 combos
where the correct range held 131 — which invalidated every search measurement taken before it was
caught. Assert the size and shape of a range at the point of use, not at the point of
construction.
M2 — Abstracted six-max game and traverser
Build: configure universal_poker for six players, the real blinds
and a bounded effective stack, with a discrete betting abstraction; wire an external-sampling MCCFR
traverser to the M1 buckets. Validate legal-action masking explicitly — the
half-pot abstraction has a known edge case, and an illegal action accepted by a solver produces a
policy that cannot be played.
Gate: reproduce a published result (~37 mbb/g) on a two-player flop subgame
before trusting anything six-handed. Two players is where the theory actually holds, so it is where
a bug is still visible.
M3 — The SD-CFR training run
Build: the loop — advantage networks with a card-embedding and betting-history
architecture; reservoir buffers in the tens of millions of samples; 10⁴-10⁵ external-sampling
traversals per iteration; the advantage net retrained from scratch each iteration and
snapshotted to disk, because the snapshot history is the SD-CFR average; several hundred to
a thousand iterations with linear weighting. CPU fans out the traversals; a GPU refits the net.
What went wrong: the policy net was never fitted
The chain saved an untrained policy network for five consecutive sessions. A
periodic checkpoint saved the advantage side correctly and the policy side unfitted, so every
downstream evaluation scored random weights — and a --max-hours checkpoint's policy net
evaluated worse than uniform, which is the tell. Fixed at the source rather than by
re-fitting after the fact.
A related misreading: rising entropy in a CFR average is not divergence. The
average strategy legitimately becomes more mixed as it converges toward an equilibrium that is
itself mixed. Treating that as a failure signal sends you hunting a bug that is not there.
What went wrong: profiling the wrong thing
A profile reported "84.5% of time is matmul", which argued for a GPU. Re-measured properly,
61% was Python building batches — the GPU would have idled behind the data loader.
The real split is roughly 54.6% traversal, 41.7% training. Beware ps pcpu here: it is a
lifetime average and reported the exact inverse of the live split.
And thread count is not free: 16 threads beat 32 on this box. Oversubscription past the physical
cores cost throughput.
M4 — Distill, export, integrate
Build: sample infosets, label them with the SD-CFR average, train one supervised
policy net, and export it in the existing serving contract — TorchScript
model_scripted.pt plus norm.npz — so the serving process, the bots and the
drivers need no changes. Add it as a champion slot and A/B it against the incumbent with live
bb/100 as arbiter.
Why reuse the contract rather than improve it
A new model and a new serving path at once means a regression has two possible causes and no
clean bisect. The contract is deliberately boring: same input width, same normalisation files, same
heads, same endpoint shape.
M5 — Evaluation and iteration
Build: LBR continuously during development as a cheap exploitability lower
bound; exact small-game exploitability periodically; head-to-head bb/100 against the rule engine and
prior checkpoints. Widen abstraction or iterations only where these say it is needed.
What went wrong: nearly every evaluator, at least once
An arena reported a champion "crushing" while scoring a policy that never played,
because a guardrail overwrote every bet it made. A gate manufactured a fake +31 by measuring its own
noise, and was replaced with a common-deals evaluator (SE ~5). A convergence metric reported
"declining" when it had fitted a champion swap as a trend line — now segmented by
pinned baseline. A blueprint bridge fired zero times because the fill wrote bucket
2 and the consumer read 0/1, producing a perfect, meaningless null.
The generalisation: always report how many decisions a treatment actually changed.
A null with zero changed decisions is not evidence of no effect; it is evidence of no treatment.
M6 — The ten-pocket six-max game
M6 is the departure from the plan, and it exists because of the bet-abstraction traps described in
concepts. Neither stock option worked: fcpa could not express a
half-pot bet — the derived blueprint's bet0.50 mass was exactly 0.0000 while half-pot
was the live bot's most common aggressive action — and fullgame carried every size in
its actions but not in its tensor, where a 4-chip and a 200-chip bet differ by one
bit.
Build: a custom Python game that wraps fullgame for rules
only — dealing, legality, showdown, side pots — and puts its own action space and its own
information state on top:
0 FOLD 1 CALL 2 MIN 3 QUARTER 4 HALF 5 THREEQ 6 POT 7 POT2 8 POT4 9 ALLIN
to_call = max_contrib - contrib[me]
raise_to = max_contrib + f * (pot + to_call) # clamped to [min-raise, all-in]
Two design choices carry the milestone. A pocket whose clamped amount collides
with another's is made illegal rather than silently aliased — so a short stack offers four
or five distinct pockets instead of ten labels for three amounts, and reported mass per pocket means
what it says. And the information state records which pockets each player used per street
plus the exact chips committed — the thing fullgame's tensor omits, without
which the extra pockets would be as unusable as its own.
What went wrong: reservoir starvation, and a bridge with a ceiling
Traversals per iteration were cut to 30 after an uncapped-tail fix made iterations fast. The
reservoirs then sat far below the batch size, every advantage update silently skipped, and the run
did pure traversal with zero learning while the counter advanced. Raised back to 300.
Separately, the blueprint bridge's fidelity is structural: 8× the fill hands
moved leave-one-out total variation 0.254 → 0.254. Forty-two flat readings were measuring the
bridge, not the run. The policy had in fact learned — it closed 31% of the gap.
What M6 is still not
Multiway collapse measured harmful, and a +32 bb/100 heads-up blueprint edge came
out neutral in six-max — the fleet's actual game. Heads-up results do not transfer,
and a six-max claim has to be measured six-handed.
Next — the served network, exactly →
