Research

xMentium performance on DocILE.

We evaluated the xMentium complete-pipeline on a 500-document reshaping of the DocILE key information extraction benchmark. The system ranked first among the evaluated document extraction systems on the full 55-field schema.

01 Overview

Evaluation task and headline result.

DocILE is a public benchmark for key information localization and extraction over business documents. For this evaluation, each system received the same PDF input and the same 55-field extraction schema. xMentium's complete-pipeline achieved the highest outcome-weighted field extraction score on the evaluated set: 75.0, compared with 69.9 for the document extraction provider and 69.7 for the strongest frontier LLM baseline.

500

Documents scored

~14,300

Ground-truth field values

Field types (36 header, 19 line-item)

Systems compared

Validation subset drawn from the DocILE annotated corpus. Document mix: 68% tax invoices, 21% order forms, 4% sales orders, 4% receipts, 2% purchase orders, plus one proforma and one credit note.

02 The Dataset

Why DocILE was selected.

We reviewed several public extraction datasets before selecting DocILE. The alternatives were either too narrow, too small, too synthetic, primarily diagram-oriented, or too heterogeneous in length to support a stable comparison. DocILE offered the strongest combination of scale, document realism, fixed schema, and layout diversity.

CORD

Shop receipts with limited document variety.

FUNSD

Scanned forms, but only 199 documents.

FATURA2

Synthetic invoices with comparatively clean layouts.

Digitize-PID

Engineering drawings rather than document metadata extraction.

Enginuity-bench

Engineering diagrams, but too clean to stress practical extraction failures.

LongExtractBench-50

Long table-heavy PDFs with document lengths ranging from 22 to 11,622 pages.

DocILE contains real business documents rather than synthetic or cleaned-up samples, and its fixed 55-field schema supports consistent system-to-system comparison. Its 1,000+ layout clusters also reduce the risk that a system is being evaluated only on template memorization.

03 Evaluation Protocol

Comparator inputs were held constant.

All non-xMentium comparator systems were evaluated through the same benchmark harness. Each runner received the same document bytes, the same extraction schema, the same system instruction, and the same downstream canonicalizer and scorer.

Same schema, verbatim

Every provider receives the exact same JSON schema, with descriptions taken verbatim from the dataset. No rewriting descriptions to be more extraction-friendly for one provider.

Same system prompt

Every runner uses the shared "You are extracting structured fields from a business document..." prompt. No provider-specific coaching.

Zero shot

No worked examples are included in any prompt. Every provider sees the schema and the PDF, nothing else.

Single pass

One structured-output call per document. No self-critique loops, retry-on-low-confidence, or iterative refinement. Native multi-pass reasoning inside a provider's own endpoint counts as part of the provider; we add no passes on top.

Same preprocessing

The schema is preprocessed exactly once, and the identical result is handed to every runner. No provider-specific schema tweaks.

Same canonicalizer for scoring

Predictions and ground truth are canonicalized with the same rules that produced the ground truth itself. Providers are scored on extracted content, not on formatting.

No corpus-tuned post-processing on the leaderboard

Any variant whose cleanup, filter, or normalization rules were derived from measuring failures on the same 500 documents is excluded from the ranked table, even if it scores well.

04 xMentium System

The evaluated system is the complete-pipeline configuration.

The xMentium row reports a five-stage xADL pipeline built for the DocILE extraction task. It combines OCR, OCR repair, structured VLM extraction, deterministic normalization, and export into the DocILE schema. The pipeline uses invoice-domain field descriptions and prompt rules, but it does not use per-document tuning or worked examples.

OCR

PaddleOCR attaches text and bounding-box origins to each page.

OCR repair

A fast vision model cleans OCR text into reading order and repairs common character errors.

Extraction

A structured-output VLM call receives the page image, repaired OCR text, field descriptions, and invoice-domain extraction rules.

Normalization

Per-page outputs are consolidated, line items are deduplicated, and amounts, dates, and currencies are canonicalized.

Export

The flat extraction record is projected into the nested 55-field DocILE schema for scoring.

The reported xMentium cost is metered LLM spend for the pipeline: local PaddleOCR, a Luna OCR-repair pass, a Terra extraction pass, and non-LLM normalization/export stages.

05 Scoring

The headline score is an outcome-weighted field extraction score.

The evaluation scores extracted field values rather than bounding-box localization. Predictions and ground truth are canonicalized with the same rules, then each document-field cell is assigned to one of five outcomes. The same scorer is applied to every system.

m

matchCanonical value equals ground truth.

n

nearClose value under fuzzy, token, substring, or numeric tolerance.

w

wrongA value was emitted, but it is not close to ground truth.

mi

missedGround truth contained a value and the system emitted none.

e

extraThe system emitted a value where ground truth contained none.

F = (5m + 0.25n) / (5m + 0.25n + 2mi + 2.5e)

Wrong values receive no positive credit in the numerator. They are still reported as a distinct failure class so that value errors, omissions, and over-emissions remain separable in analysis.

Original sample invoice with no field highlights. — The same invoice without highlights (left) and with DocILE dataset annotations (right). The annotations illustrate the field locations and line-item structure used by the benchmark; this is not a visualization of xMentium predictions.

The same invoice with DocILE field and line-item annotations highlighted in color. — The same invoice without highlights (left) and with DocILE dataset annotations (right). The annotations illustrate the field locations and line-item structure used by the benchmark; this is not a visualization of xMentium predictions.

Cost is summed per document from each provider's published token rate or, for the document extraction provider, from reported credits converted at the published credit price.

06 Results

xMentium ranked first on the full 55-field evaluation.

Across the 500-document evaluation, xMentium produced the highest outcome-weighted score. It also produced the largest raw match count and fewer extra values than the next-ranked systems.

75.0xMentium F score

+5.1vs. document extraction provider

+5.3vs. best frontier LLM

Rank	Provider	Match	Near	Wrong	Missed	Extra	F	Cost
1	xMentium	9,448	1,087	1,016	2,796	2,852	75.0	$17.93
2	Document extraction provider	9,315	1,157	1,156	2,719	4,664	69.9	$37.26
3	OpenAI Terra	8,532	1,045	828	3,942	3,101	69.7	$14.38
4	OpenAI Sol	8,558	1,033	974	3,782	4,123	67.1	$34.37
5	OpenAI Luna	8,624	1,070	1,013	3,640	4,968	65.6	$6.71
6	Anthropic Sonnet 5	7,931	1,094	1,022	4,300	4,691	62.9	$10.31
7	Gemini 3 Pro	8,172	1,001	906	4,213	5,202	62.7	$19.03
8	OpenAI GPT-5 mini	8,529	1,168	1,150	3,500	6,768	61.5	$3.34
9	OpenAI GPT-5	7,168	1,117	993	5,069	3,692	61.3	$22.77
10	Gemini 2.5 Flash	7,455	1,056	1,080	3,720	5,827	60.0	$3.66
11	Gemini 2.5 Pro	7,799	1,050	1,160	3,280	7,367	58.5	$17.20
12	Gemini 3 Flash	6,920	1,007	1,035	4,552	6,158	55.8	$5.87
13	Gemini 3.7 Flash	6,307	881	898	5,920	4,086	55.3	$1.78
14	NVIDIA Nemotron Parse + Nano Omni	7,353	949	1,124	3,768	10,657	50.2	$2.30
15	NVIDIA Nemotron 3 Nano Omni	8,784	1,114	1,353	2,847	14,770	49.6	$2.04
16	Anthropic Opus 5	2,324	393	426	11,204	6,472	21.8	$51.78

07 Degradation Experiment

Robustness under controlled image degradation.

We also evaluated a deterministic 100-document subset under medium image degradation. The subset was stratified by document type, source, cluster tier, and layout cluster. Each page was rendered at 100 DPI, randomly rotated within +/-2.5 degrees, blurred with a Gaussian radius of 1.0, perturbed with additive noise, and recompressed at JPEG quality 30. All systems were scored against the same unchanged answer keys.

Rank	Provider / model	Match	Near	Wrong	Missed	Extra	F (%)	Cost
1	xMentium complete-pipeline	1,491	248	308	641	623	67.3	$3.60^†
2	OpenAI gpt-5.6-sol	1,203	208	254	1,023	639	57.7	$7.16
3	Google Gemini 2.5 Pro	1,409	277	242	760	1,320	57.1	$3.60
4	Document extraction provider (default)	1,284	289	290	825	1,086	56.5	$2.15^*
5	OpenAI gpt-5.6-luna	1,189	228	307	964	806	55.6	$1.51
6	OpenAI gpt-5.6-terra	1,090	211	246	1,141	559	55.1	$3.34
7	OpenAI gpt-5-mini	1,026	228	238	1,196	647	52.3	$0.57
8	Google Gemini 2.5 Flash	1,198	216	330	870	1,194	52.2	$0.82
9	OpenAI gpt-5	738	185	175	1,590	420	43.2	$3.82
10	Anthropic Claude Sonnet 5	759	125	128	1,676	819	38.5	$1.76
11	Anthropic Claude Opus 5	510	76	89	1,982	1,400	24.0	$9.12

Cost derived from published token rates; ^* Document extraction provider cost derived as credits × $0.015. ^† xMentium cost extrapolated from the full 500-document metered pipeline cost.

08 Limitations and Future Work

The result establishes benchmark performance, not general document intelligence.

DocILE is a useful public benchmark because it is large, annotated, schema-stable, and layout-diverse. It remains a business-document benchmark. The next research question is whether the same pipeline design continues to dominate on more visually complex, domain-specific document families.

Metric reporting

The headline table reports the outcome-weighted field extraction score used in this evaluation. It should be read separately from official DocILE localization metrics.

Dataset scope

DocILE primarily evaluates business documents such as tax invoices, orders, and receipts. It does not represent diagram-heavy engineering corpora such as P&IDs.

Robustness scope

The degradation experiment uses deterministic image transformations. It is a controlled robustness test, not a complete model of real scanner, fax, handwriting, or capture artifacts.

Future research

Next evaluations should extend beyond invoice-like documents to annotated engineering diagrams, P&IDs, and customer-specific corpora with independently constructed answer keys.

Business rules

Adding business rules improved performance across the evaluated models. xMentium came out on top because those rules worked across its complete pipeline, combining domain knowledge with OCR repair, structured extraction, and normalization. The result reflects how effectively the pipeline puts business rules to work alongside the underlying models.

System development

Additional work should evaluate arithmetic cross-checks, field-level confidence calibration, and model routing policies that preserve accuracy while reducing cost.