Introducing PrivateMarketsAgentBench

We are releasing PrivateMarketsAgentBench, a benchmark designed to measure how well AI agents perform on the structured, convention-dependent workflows that define private-markets finance. Each task consists of a production-native data environment, a clearly defined output schema, and a grader that scores task completion the same way a portfolio operations team would: all-or-nothing, because partial credit does not exist when numbers flow into investor reports.

The benchmark evaluates two classes of system: a Specialist system (HenonMAX 2.1) purpose-built for private-markets data, and a Generalist system (Claude Opus 4.8) representing the frontier of general-purpose LLM agents. Both systems are fed data in production-native form. All 60 tasks are scored.

The goal is to provide a clear picture of where AI agents can reliably execute financial workflows today, and where they cannot. By articulating where specialist systems outperform generalist models, PrivateMarketsAgentBench helps private-markets firms measure the ROI of AI investments and identify which workflows can be safely automated versus which require human oversight.

Why we built a private-markets agent benchmark

AI adoption in financial services has reached saturation at the experimentation level. According to McKinsey's 2025 State of AI report, 88% of organizations are using generative AI in some capacity. Yet only 33% have scaled beyond localized pilots. The gap is not a lack of interest or tooling. It is the challenge of converting AI use into governed, auditable workflows that produce numbers a CFO would sign.

In private markets specifically, the problem is structural. Financial workflows depend on stored conventions, proprietary policies, warehouse-only dimensions, fiscal calendars, allocation keys, and multi-entity consolidation rules that no document contains and no general-purpose model has been trained on. When the answer requires knowledge that lives in a data warehouse rather than a PDF, file-only agents break.

PrivateMarketsAgentBench exists to make this failure mode legible. Rather than testing whether a model can answer a financial question, it tests whether a system can navigate the full complexity of a production financial workflow: reading structured data, applying stored business rules, and producing outputs that match what a human analyst would deliver.

Larger models do not necessarily outperform smaller ones

A common assumption is that larger, more expensive models will produce better results on any task. Data from Artificial Analysis shows this is not the case. When measured across intelligence, speed, and cost per task, larger models show diminishing returns at scale. The relationship between parameter count and operational accuracy is not linear, particularly on structured financial workflows where the bottleneck is data access and business-rule application rather than raw reasoning capacity.

This finding has direct implications for how private-markets firms should evaluate AI systems. A model that scores higher on general reasoning benchmarks may still fail on a multi-currency consolidation task because it lacks access to the stored allocation keys and fiscal calendar mappings required to produce a correct answer.

Furthermore, the performance gap between open-weight and proprietary models is narrowing rapidly. According to Artificial Analysis Intelligence Index v4.1, open-weight models have nearly closed the gap with proprietary systems on standard reasoning tasks. This convergence reinforces the thesis that the differentiating layer for financial AI is not the model itself, but the structured data layer, proprietary mappings, and verification pipeline that sit around it.

Source: Artificial Analysis (artificialanalysis.ai)

Where general AI fails: the structured workflow problem

The PrivateMarketsAgentBench results reveal a clear pattern. General-purpose LLM agents perform well on tasks where the answer can be derived from documents alone. They fail systematically on tasks where the answer depends on stored conventions, proprietary policies, or warehouse-only dimensions.

The pattern is consistent: when the answer requires data or a stored convention that no document contains, the file-only agent scores zero. The general LLM is wrong on every file in hand because the correct answer depends on mappings, ontologies, calendars, and allocation keys that live in a structured data warehouse.

12-month lifecycle: cost compounding

Beyond accuracy, the benchmark reveals a structural cost problem with file-only agents. A general-purpose agent re-reads and regenerates every month, so its cumulative cost climbs steadily. The longer the lifecycle, the wider the gap. Over a 12-month period, the cost difference becomes dramatic.

Henon ingests once and re-queries the warehouse. The general agent re-reads and regenerates every month. Over a 12-month lifecycle, this produces a 53x cost difference on a single task type.

PDF table extraction: where vision models break

A separate evaluation tested both systems on PDF table extraction and querying. The task: read a table from a PDF, then answer sourcing, aggregation, and cell-tracing questions. A figure off by more than 0.01% is graded as wrong.

On clean synthetic tables (30-300 rows, no text layer, real OCR), both systems score 5/5 on accuracy. The differentiator is cost: Opus Vision is billed per page-image, while Henon's Textract-and-verify pipeline is 5.7x to 8.4x cheaper and produces structured, reusable output.

The discriminating test is the dense real-world table: a real quarterly fund report with 9 near-identical columns (FYE x 4, LTM, Budget, YTD x 3), stacked headers, space-separated thousands, negatives, n/a values, and grey-on-grey formatting flattened to an image. This is where raw vision breaks.

Full benchmark: 60 tasks

Across the full 60-task benchmark (combining financial workflow tasks and PDF table extraction tasks), the aggregate results reinforce the pattern:

Why general AI fails in both expense and quality

The PrivateMarketsAgentBench results demonstrate that generalist LLM agents, while powerful on open-ended reasoning tasks, consistently fail on the structured, convention-dependent workflows that define private-markets finance. HenonMAX closes this gap by combining a purpose-built data warehouse, proprietary mappings, and a dual-AI verification layer, delivering higher accuracy at a fraction of the cost and with full traceability back to source documents.

The benchmark will continue to evolve as new task categories are added and as both specialist and generalist systems improve. We will publish updated results as new model versions are released.