evals.finance · technical reports · the third report
A runnable evaluation of language models on an options-component ETF redemption
Read it on SSRN · PDF (12 pages) · Every run: answers, raw completions, scored reports, run records · The grader review · The first report · The second report
An exchange-traded fund whose portfolio is four listed option legs is redeemed in kind. The option positions have to move at the clearing house from the fund's account to the redeeming dealer's, a step the clearing house's own rule filing of January 2026 describes as fragmented and heavily dependent on manual intervention. The task is the settlement tie-out: given the redemption order, three portfolio composition files of which one is the basket of record, the fund's final net asset value, the clearing house's record of four position moves, the custodian's positions the next morning and the agent's own status report, tie every leg to the basket, recompute the cash component and decide whether to settle.
In the gold case one leg moved at 12,000 contracts against 120 due, the order's underlying-share equivalent keyed as the contract count, while the agent's report says all four legs moved. The right answer is to hold and have 11,880 contracts returned first. A clean twin settles. Eight models from three vendors ran both cases once.
| Model | Broken order | Clean twin |
|---|---|---|
| Claude Opus 4.8 | 1.000 · all checks | 1.000 · all checks |
| Claude Sonnet 4.6 | 0.868 | 1.000 · all checks |
| Claude Haiku 4.5 | 0.543 · gate fired | 0.549 · gate fired |
| GPT-5.6-sol | 1.000 · all checks | 1.000 · all checks |
| GPT-5.5 | 1.000 · all checks | 1.000 · all checks |
| GPT-5.4 | 0.988 | 1.000 · all checks |
| GPT-5.4-mini | 0.968 | 1.000 · all checks |
| Gemini 3.6 Flash | 0.956 | 0.989 |
Scores out of 1.000 after the gates; "all checks" means every criterion met; "gate fired" means an automatic failure on the arithmetic the gate exists to catch. One run per model per case, October 2026.
Every model held the broken order and settled the clean one. Every model found the leg, the 12,000 against 120 and the 11,880 excess, named the return transfer before any release and escalated to both clearing members. Nine of sixteen runs met every criterion; three models were perfect on both cases. No model assumed what the dealer had paid for its shares, which the documents do not say.
The losses were arithmetic one layer down. A small model valued the option legs per share instead of per contract, so every leg came out at one hundredth of its value, and the gate built for that confusion fired on both cases. A mid-tier model sized the excess at ten times its value and carried the error into its hold note. Another moved the written legs' positions the wrong way. A small model converted a per-share impact to a percentage fifty times too small. The decisions were right every time; the sizes were not.
Six independent attackers spent a day trying to make the grader score a wrong answer well or a right answer badly, about 1,500 graded mutations. Every gate was then moved onto an evidence hook, so that nothing fires on a missing field or a notation, and 267 of the attackers' checks were pinned as standing regressions. Every mechanical assertion of the fixture was checked against public primary documents and labelled. Running real models then found six places where the grader rejected legitimate answers; all runs were re-graded with the original scores kept, and eleven scores moved without any decision or gate changing.
The fund, the dealer and the settlement documents are constructed, because the real ones are not public; the option contract terms are real. One run per model per case; recorded results, not rates. Not investment advice. Everything on this page is reproducible from the repository.