On this page
The short answer
Evaluate document extraction by comparing required fields and relationships with the original page evidence across representative layouts. Separate parsing, recognition, normalization, and record acceptance. A well-formed JSON object can still contain values from the wrong page, an outdated revision, or a visually adjacent but unrelated label.
What to take away
- Keep original page layout and source versions.
- Separate recognized text from correct field association.
- Report accepted documents as well as field scores.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Preserve the document as evidence
The 2024 Docling technical-report abstract describes document conversion with layout and table-structure models. It is a foundational source about the problem, not a statement of current package specifications or evidence that any parser is accurate on your files.
Retain the exact source file, page number, version, and permitted evidence reference for each case. Define required fields, acceptable missing values, and relationships between them. A text-only reference can conceal errors caused by columns, footnotes, or visual grouping, so reviewers need the original layout when those features determine meaning.
Evidence: Docling Technical Report (2024) [1]
Build layout slices from the intended workload
Include clean digital files, scans, rotations, low-resolution pages, multi-column text, repeated headers, and forms where those conditions actually occur. Add revised or duplicate pages if the workflow must select the current value. Preserve file-level families when splitting development and final tests.
Keep input failures distinct from extraction errors. A password-protected or unsupported file that the product cannot process should receive an explicit terminal outcome. Silently dropping it from the denominator can make the extractor look accurate only because difficult documents never reached scoring.
Use an invoice fixture with observable relationships
Imagine an invented two-page invoice with an account number in the header, a subtotal on page one, and a corrected total on page two. A candidate returning the subtotal as the payable total fails even when the digits were recognized perfectly. The field’s role and revision context matter.
Define a passing record with the correct account, final total, currency, and source references. Include an absent optional field as a control: the extractor should mark it unavailable according to the schema rather than infer a plausible value from unrelated text. These are proposed fixtures, not executed extraction results.
| Case or record | Expected check | Decision |
|---|---|---|
| Subtotal mistaken for final total | Digits right; semantic role wrong | Reject the payable record |
| Optional field absent | Schema permits unavailable value | Do not invent it |
| Unsupported file format | No extraction attempt completed | Report input failure explicitly |
Report field quality and usable records
Count exact or contract-normalized field matches, missing fields, extras, wrong-source values, and complete accepted documents. Show results by layout condition and field type. A frequent easy header should not hide poor accuracy on the one total that determines downstream work.
Diagnose failures at the earliest supported stage: unreadable source, parsing order, label association, normalization, or final schema mapping. Compare candidates on identical files and references. Preserve manual-review workload and abstentions so an extraction improvement cannot merely shift difficult documents into an unreported human queue.
Limits of the evidence
The invoice is fictional and no document pipeline was tested. The Docling paper was reviewed at abstract level and concerns its 2024 report. Layout samples, reference labeling, and permitted access to original pages constrain the evaluation.
Common questions
Is valid JSON a successful extraction?
It establishes output shape only. Check source support, field meaning, normalization, and any relationships required by downstream work.
Should scanned files be mixed with digital PDFs?
Report the overall workload if useful, but retain separate slices so different parsing and recognition conditions remain visible.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Docling Technical Report (2024) Auer et al. · 2024 · PreprintReviewed: abstract and bibliographic record; PDF conversion, layout analysis and table structure; foundational v1 report, not current package specifications. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.