evals.finance · technical reports · the second report
Measuring a maker-checker protocol for LLM agents on corporate-actions processing
Read it on SSRN · PDF (14 pages) · Every run: transcripts, ledgers, worksheets, verdicts · The first report · The third report
The oldest control in operations, a maker and a checker, with both seats taken by AI. Six corporate-action cases built on real events, eight models from three vendors, one grader, run four ways: the documents in a single prompt; the same task worked through tools; the tools run handed to a second agent of the same model, which ties the work out, approves or rejects, and sends the maker into one revision round; and the same protocol with one stronger model reviewing every maker. One run per model per case, 48 cells in each arm, three repeats on one case.
| Arm | Runs that passed every check, of 48 |
|---|---|
| Documents in one prompt | 32 |
| With tools | 35 |
| A second AI of the same model as checker | 27 |
| A stronger AI as checker | 25 |
The reviewers rejected 20 of the 46 ledgers the grader scores correct, and the makers revised in all 21 rejections. The stronger reviewer over-ruled ten correct refusals, and the makers then booked numbers the documents do not support. The reviewer received the documents, the tools and the maker's work, but not the rules the maker worked to, and under a plain tie-out instruction a correct refusal looks like a missing figure.
A reviewer with the documents but without the maker's rules is a source of errors as well as a filter for them. The paper sets out the mechanism, the part of the result that traces to the environment rather than to the reviewers, and the next protocol: the maker checks each finding against the source before changing anything.
One run per model per case; recorded results, not rates. Not investment advice. Everything on this page is reproducible from the repository.