Evaluate document parsing before tuning a RAG retriever

Define semantic parsing checks

Microsoft documents layout extraction that preserves structural information. The evaluation question is whether your representation retains the meaning required by the application. A text extraction can contain every visible word and still join a footnote to the wrong paragraph. Label source anchors for headings, lists, tables, units, and exceptions. Treat boilerplate removal as a transformation that needs inspection, especially when a disclaimer or scope qualifier resembles a repeated footer.

Evidence: Document Layout skill [1]

Use deliberately fragile documents

Construct an authored two-column policy PDF with one eligibility rule in the left column and a separate exception on the right. Add a table whose unit appears only in its heading and a footnote that changes one row. Prepare expected extracted facts rather than demanding a single serialized text layout. The parser passes when the rule, unit, and row-specific exception remain connected. Matching the document title alone is a weak check that would miss the consequential failures.

Locate the first broken representation

Save the raw file hash, parser version, extracted blocks, block positions, and final chunk text. For each failed answer, first ask whether the correct fact exists in the parsed output. If it does not, retriever tuning cannot restore it. If it exists but the chunk loses its heading, fix the transformation between parsing and indexing. Only classify the failure as retrieval after confirming that a usable evidence unit actually reached the searchable corpus.

Gate changes on preserved meaning

Compare the old and new parser on the same document fixtures before reprocessing the full corpus. Require all chosen critical facts to retain their relationships, while allowing harmless whitespace differences. Keep a separate count of documents the parser could not process so successful files do not hide dropped files. Review a sample of extraction differences visually against the original page. A parser upgrade should be reversible until those checks establish that the searchable representation still supports the intended questions.

Give extraction failures a repair owner

Create an issue record containing the original page location, expected fact, actual extracted block, and the first stage that lost the relationship. A heading attached to the wrong paragraph suggests a layout or chunk-assembly repair. A missing minus sign suggests a symbol-recognition or normalization repair. A table flattened without headers suggests a structural representation change. These are different interventions and should not all become requests for a better embedding model. Preserve at least one fixture for each repaired pattern so a later parser update cannot silently reintroduce the same defect. Include parser crashes and empty successful outputs in the document manifest; excluding them would overstate usable corpus coverage.

Common questions