ETF & asset-management domain expert · LLM evaluation, AI governance, and GenAI model validation · Greater Boston
I build runnable evaluations that measure whether an AI can be trusted to do a specific finance job — graded the way an experienced professional grades work: against a rubric, with automatic failures for the errors that quietly ruin an answer, and credit for admitting what can't be known.
Eight runnable evaluations of real asset-management workflows. Each decomposes the workflow into scored checkpoints, grades against a point-weighted rubric with auto-fail gates, cites gold answers to source documents, and runs live models end to end.
| Eval | What it measures |
|---|---|
| Quarterly earnings analysis | Digest a 10-Q, reconcile the figures, flag what moved |
| Defined-outcome (buffer) ETF diligence | Recompute cap and buffer from the option strikes; price the protection |
| DCF valuation | Project cash flow, discount at WACC, bridge enterprise value to equity |
| ETF creation/redemption reconciliation | Tie an authorized participant's basket to the published PCF — settle only if it ties |
| OTC swap confirmation matching | Match two confirmations field by field — affirm only if the economics agree |
| Corporate-actions processing | From a document store with amendments and distractors: pin the governing version and dates, compute the entitlement — commit only what the deadline allows |
| ETF NAV oversight | Recompute a leveraged ETF's NAV from the administrator's package and the counterparty statements; find the stale swap mark and size it against the fund's error policy: hold and re-strike, or release only if it ties |
| Options-component ETF redemption tie-out | Tie out an in-kind redemption whose basket is exchange-traded options: each OCC position adjustment to the basket, the custodian's positions and the cash component — settle only if every leg ties |
The last five are back-office controls. As of October 2026 they appear to be the only public, runnable LLM evaluations of capital-markets post-trade operations, with one qualification: fund NAV work also appears, as a task family inside a general professional-work suite (Surge AI's DAYJOB, October 2026). The other prominent finance benchmarks (GDPval, Vals AI's Finance Agent, Mercor's APEX, BigFinanceBench, FrontierFinance) grade front-office analyst work.
The corporate-actions evaluation also runs as an agent environment: the same six cases with the tools a desk has (a document store, positions as of a date, a calculator, a test ledger) and, in two further arms, a second AI as checker. With tools, 35 of 48 runs passed every check, up from 32 with the documents in a single prompt; with a checker of the same model, 27; with a stronger checker, 25.
The options-redemption evaluation ran on eight frontier models from three vendors in October 2026: every model held the broken redemption and settled the clean one, nine of sixteen runs passed every check, and the losses were magnitude errors one layer down — a mid-tier model sized the excess at ten times its value, and a small model valued the option legs per share instead of per contract and tripped the gate built for that confusion.
Technical report (PDF) — TieOutBench v1.1 · On SSRN · DOI 10.2139/ssrn.7243025 · Second report: A Second AI as Checker · on SSRN (7611299) · DOI 10.2139/ssrn.7611299 · its PDF · Third report: Settle Only If Every Leg Ties · on SSRN (7655899) · DOI 10.2139/ssrn.7655899 · its PDF · Dataset snapshot (Mendeley Data) · Repository on GitHub · Live model leaderboard
On a real McDonald's FY2025 case, the correct valuation — under the case's stated, dated assumptions; a grading reference, not investment advice — lands about 20% below the market price, while the classic blunder — dividing enterprise value by share count and skipping the net-debt bridge — lands within 3% of it. To a reviewer eyeballing outputs, the wrong method looks calibrated and the right one looks aggressive. A gate on the valuation bridge catches what an averaged score rewards.
Given a creation basket that was short by $13,320, one frontier model correctly refused to settle — but its own arithmetic was $200,000 off, so it described a short basket as over-delivered. Grade only the decision and it scores perfect; in production its escalation sends the desk hunting in the wrong direction.
Every frontier model tested — eight across three vendors — correctly refused to affirm a swap whose fixed rate was off by five basis points — about EUR 25,000 a year on a EUR 50 million notional. But in the first three-model round, only one sized it right. One called it ten times too small, which a human triages as trivia; one read five basis points as fifty and reported EUR 2.5 million over the life of the trade against a correct figure of EUR 125,000, which triggers a fire drill.
Across the first three live-run evaluations, no model ever committed the catastrophic action the signature gates exist to catch. The corporate-actions evaluation changed that: one small-tier model pinned the correct corrected announcement, stated the corrected record date — and then computed the entitlement on the superseded position and booked the wrong payment. The flagships still never commit the catastrophic action; the small tier now does, and the evaluation localizes exactly where.
Giving the corporate-actions models the tools a desk has raised the runs that passed every check from 32 to 35 of 48. Adding a second AI of the same model as a checker lowered it to 27: the reviewers rejected 20 of the 46 ledgers the grader scores as correct, and the makers changed their work after every rejection. A stronger checker rejected fewer correct ledgers and still over-ruled ten correct refusals into stated values. A checker needs the rules, not only the documents.
Given a leveraged ETF's NAV package with one swap carried at the prior day's price, every one of eight frontier models held the NAV, named the swap and released the clean twin package; ten of sixteen runs passed every check. The points lost were one layer down: one small model summed the recomputed assets two million dollars short and sized the error at 25 cents a share instead of 75. It still held, because the one-cent floor was exceeded either way. Two open-weight models, running on a desktop graphics card and on a laptop, did the same work as the paid flagships.
Caveats stated rather than buried: one run per model per case; evaluations 3–8 now cover eight frontier models across three vendors (evaluation 7 also five open-weight models run locally), while evaluations 1–2 have been run only against open-weight models. The leaderboard says so on its face.
Before a firm lets an AI do analyst or operations work, it needs to know whether — and exactly where — to trust it. A blended accuracy number cannot answer that. A gated, checkpoint-level evaluation can, and it serves three purposes:
Long-form pieces on where finance LLMs hold up and where they break. Published on LinkedIn:
Evaluation is one half of the work. The other is having built the systems being evaluated: multi-agent orchestration over an event bus, pre-trade compliance screening with an immutable audit trail, regime-switching model ensembles, and a phased production delivery plan — shadow mode, user acceptance testing, runbooks, hypercare.
Those demonstration systems live at aistrategyinvest.com. They are research and portfolio demos — not offered as an investment service, and not investment advice.
Fifteen-plus years leading cross-functional programs in financial services. I ran the change-management side of an institutional ETF servicing platform, where the daily subject matter was the full ETF lifecycle — creation and redemption, basket composition, rebalancing, corporate actions — including extending the platform to process ETFs holding derivatives: options, futures, forwards, and swaps. Independent consultant since 2022, building AI-powered tools for ETF research, due diligence, and derivatives-overlay analysis.
Most of my work sits where finance domain judgment meets AI evaluation: authoring benchmarks and evaluations for AI labs and data providers, helping asset managers and financial institutions stand up GenAI evaluation and model validation, and ETF product and platform work where AI tooling genuinely fits.
If any of that is live at your firm, I'd enjoy the conversation.