Hiss NN · 01

Concepts

In dependency order. Each one exists because the one before it was not enough, and that chain — not the individual definitions — is the thing worth carrying away.

1 · Poker is an extensive-form game with imperfect information

Chess is a perfect information game: both players see the entire state. Poker is not — you cannot see opponents' cards, so many distinct situations look identical from where you stand. The set of states you cannot tell apart is an information set, and it is the unit a poker strategy is defined over. You do not choose an action per state; you choose one per information set, because you cannot condition on what you cannot observe.

This single fact breaks the usual game-playing toolkit. Minimax and alpha-beta assume you can evaluate a position. Here a position is not one position, and worse, its value depends on what your opponent believes you hold — which depends on your strategy. The evaluation depends on the strategy being evaluated. That circularity is why poker needed its own theory.

2 · Equilibrium, and why the answer must be random

A Nash equilibrium is a strategy profile where no player gains by unilaterally changing. In two-player zero-sum games it has a strong guarantee: playing it means you cannot be beaten in expectation, whatever the opponent does. That is the ceiling on your losses, not on your winnings.

Equilibrium strategies in poker are necessarily mixed — probability distributions over actions, not a single choice. If you always bet your strong hands and always checked your weak ones, your bets would announce your holding. Randomness is not decoration; it is what makes a strategy unreadable. Every policy in this system therefore outputs a distribution.

3 · Regret, and regret matching

Rather than compute an equilibrium directly, we reach one by iterated self-improvement. The quantity that drives it is regret: having played a hand, how much better would you have done had you always taken action a instead? Positive regret means the action was underused.

Regret matching turns that into a strategy: play each action with probability proportional to its accumulated positive regret. Actions that would have helped get played more; actions that never would have get abandoned. The remarkable result is that if regret grows sub-linearly, the time-average of the strategies converges to an equilibrium.

4 · Counterfactual regret minimisation (CFR)

Regret matching applies to a single decision. CFR extends it to a game tree by defining counterfactual value: the value of an information set, weighted by the probability that everyone except you played to reach it, with your own contribution factored out. The "counterfactual" is precisely that removal — it asks how good this spot is when you reach it, not how often your current strategy happens to reach it. Without that, rarely-visited branches would be starved of learning signal exactly because the current strategy avoids them.

One CFR iteration walks the tree, computes counterfactual regrets at every information set, accumulates them, and updates via regret matching. Repeat; average. Two refinements are effectively standard:

• CFR+ — clamp accumulated regret at zero instead of letting it go arbitrarily negative, so an action that was bad early can be re-adopted quickly when it becomes good. Converges markedly faster.
• Linear CFR — weight later iterations more heavily in the average, on the reasoning that they were produced by better strategies.

5 · Sampling, because the tree is astronomical

No-limit hold'em has more information sets than can be enumerated, so a full tree walk is out. Monte Carlo CFR samples instead. The variant used here is external sampling: traverse every one of your actions exhaustively, but sample a single outcome for chance events and for opponents' actions. You get an unbiased estimate of counterfactual regret at a fraction of the cost, and the variance is acceptable because your own branching — the part that matters for your regret — is still covered fully.

6 · Abstraction: making the game small enough to solve

Sampling alone is not enough; the game must also be shrunk. Two independent abstractions do it, and each is a lossy compression with its own failure mode.

Card abstraction

Group hands that play alike into buckets, and solve over buckets rather than cards. Built in three steps:

• Suit isomorphism. Before anything else, collapse hands that differ only by a renaming of suits. Nothing is lost — the game genuinely cannot tell them apart — and it cuts the precompute by roughly 24×.
• Equity features. Describe a hand not by its cards but by how it performs: E[HS], its expected hand strength against a random opponent, and on earlier streets E[HS²] or a full equity histogram, which captures the shape of the distribution. A made hand with no improvement and a draw can share an average while behaving completely differently — the second moment is what separates them, and this is what "potential-aware" means.
• Clustering. k-means over those features, with Earth Mover's Distance as the metric, because the objects are distributions: EMD measures the work to turn one histogram into another, which is the right notion of "these two hands play alike". Preflop needs no clustering — there are exactly 169 strategically distinct starting hands, so it is lossless. Postflop targets roughly one to two thousand buckets per street.

Bet abstraction

No-limit allows any bet from the minimum to all-in, so the action space must be discretised to a handful of pot fractions. This is the lever with the sharpest consequences in this whole system, and M6 exists because of it.

Imperfect recall

A further compression: let the abstraction forget some history. The information state used here records the set of bet sizes each player used on each street, not the ordered list. Ordered encoding needs a fixed slot count, and measured over 3,000 random six-max hands, 4.3% of streets already reached twelve actions with the tail still climbing — a safe ordered encoding would need ~24 slots and a 1,665-wide tensor against 377 for the set form. What is lost is only the within-street order; who committed how much survives exactly in the contribution vector, which is what distinguishes a three-bet from a cold call. The theoretical cost: CFR's convergence guarantee is stated for perfect recall.

7 · Deep CFR: replacing the table with a network

Even abstracted, storing a regret value for every information set is infeasible. Deep CFR replaces the table with a neural network that generalises across information sets — similar spots get similar values without ever having been enumerated. Mechanically:

1. Traverse with external sampling. At each of your information sets, record the sampled counterfactual regrets as a training sample.
2. Store samples in a reservoir buffer — a fixed-capacity structure that keeps a uniform random sample of everything ever added, so old iterations are represented without unbounded memory.
3. Retrain the advantage network from scratch each iteration. Not fine-tuned — re-initialised — because the target is the average over all iterations and warm-starting biases toward recent ones.
4. Repeat for hundreds to thousands of iterations.

8 · SD-CFR, and why the average net was dropped

Vanilla Deep CFR trains a second network to approximate the average strategy — and that second approximation is the largest error source in the method. Single Deep CFR removes it: keep every iteration's advantage network snapshot on disk, and define the average policy as the (linearly weighted) mixture over those snapshots. The snapshot history is the average. It costs disk and sampling time at query, and it removes an entire layer of approximation error, while getting Linear-CFR-style weighting essentially for free.

9 · Distillation: from a history of nets to one servable net

A mixture over hundreds of snapshots cannot be queried in a millisecond. Distillation collapses it: sample a large number of information sets, label each with the expensive mixture's action distribution, and train one supervised network to reproduce those labels. The result has Deep CFR's serving shape without its training error — one forward pass, no search, no snapshot sampling. This is what "blueprint" means in practice, and it is the object that gets exported and served.

10 · Behavioural cloning and the teacher problem

The other track starts differently: take a corpus of decisions from a source you trust and train the net to predict them. Fast, stable, and strictly bounded by the teacher. Two teachers were used here — real hand histories, and a rule-based engine whose book is interpreted into labels — and the choice between them is instructive. Real histories carry a human's actual edge and their tilt. A rule interpreter is duller but disciplined, and crucially it folds correctly, which a corpus of one aggressive player's hands does not teach.

11 · PPO: improving a policy by playing

Once a reasonable policy exists, reinforcement learning can push it further. Proximal Policy Optimisation is a policy-gradient method: play hands, observe returns, and shift the policy toward actions that did better than expected. Three parts matter.

• Advantage — not "did this action win" but "did it do better than this state's expected value", which is why the network carries a value head. GAE trades bias against variance when estimating it.
• The clipped objective — PPO limits how far the policy may move per update. A policy-gradient step that is too large collapses the policy; clipping makes the update conservative by construction.
• Step size against signal-to-noise. Poker's reward is extraordinarily noisy, so the usable step is far smaller than the defaults suggest. Measured here: one epoch at 1e-4 beat four epochs at 3e-4 by +25 bb/100. When rewards are noise-dominated, the fix is almost always a smaller step, not a cleverer objective.

12 · Measuring: exploitability, LBR, and money

Three tools, used at three scales.

• Exact exploitability — compute the best response to your strategy and see what it wins. Only tractable in toy games (Kuhn, Leduc), which is exactly why those are the correctness gate: there the right answer is known.
• LBR (local best response) — a cheap, greedy approximation of a best response. It gives a lower bound on exploitability, so a high LBR proves you are exploitable while a low one proves nothing.
• bb/100 against real opposition — the arbiter, and the noisiest. Variance means thousands of hands can reverse a sign. The countermeasures are common random numbers (deal both policies the same cards so the comparison is paired and the deal's luck cancels), confidence intervals on every reading, and precision-weighted fits across a history rather than comparing two numbers.

Next — the milestone ladder, M0 through M6 →

Hiss NN — the poker network, its training ladder and its serving contract. Written so the whole thing can be rebuilt from an empty machine. Figures are from the live system, not from the plan; where the two disagree the plan is annotated.