Field note
Validate PDF Tables Before AI Retrieval
PDF extraction needs a structural check before the text becomes searchable evidence.
If an assistant gives the wrong price from a supplier PDF, inspect the extracted table before changing the prompt. The number may be correct while its column heading, currency or product row was lost. A better language model cannot reliably repair a relationship that the ingestion pipeline removed.
This is particularly relevant to price lists, schedules, benefit tables and equipment specifications. Their meaning depends on layout. Turning the whole page into a string can preserve the words while changing the facts.
Identify what kind of PDF arrived
A digitally produced PDF, a scan and a scan with an OCR text layer require different checks. The pypdf documentation explains that text can be positioned independently and that pypdf is not an OCR engine. Being able to select text does not prove the hidden OCR layer matches the visible page.
Keep the original, record its hash and count the pages before extraction. Inspect a sample from each document type, including landscape pages and appendices. If there is a native spreadsheet or CSV from the same authorised source, assess whether that is a better input than its PDF export.
Do not rasterise every digital PDF by default. That discards information and introduces another recognition step. Route documents according to what they contain and keep the parser choice in the ingestion record.
Keep table structure through extraction
Amazon Textract represents tables and cells as related blocks, including merged cells, headers and table titles. Azure Document Intelligence's layout model exposes row and column positions and spans. Those structures are more useful for validation than a successful-job flag.
For each extracted table, retain the page, table identity, cell position, row and column spans, and text. Preserve the header hierarchy: “Annual” spanning two columns labelled “Standard” and “Premium” carries meaning that four disconnected strings do not.
Footnotes require deliberate handling. A value of “25” may mean £25, 25 per cent or 25 units, with an exception underneath the table. Link the relevant caption, units and footnotes to the table rather than hoping a nearby chunk will be retrieved too.
Define checks around the document's purpose
Take an illustrative price list with a product code, pack size, price and minimum order. Useful checks include:
| Check | What it catches |
|---|---|
| Product codes match the approved source list | Truncated codes and merged rows |
| Each price has a currency and pack size | Correct numbers with missing meaning |
| Headers repeat correctly across pages | Continuation pages assigned to the wrong column |
| Required fields are present | Blank cells silently treated as zero |
| Totals reconcile where the document defines them | Lost lines or misplaced decimal points |
These are proposed acceptance checks, not universal guarantees. A total can still balance when two rows are swapped. Use arithmetic checks alongside visual comparison and identifier matching.
Choose the review threshold by consequence. A low-confidence heading controlling an entire price column may deserve more attention than a slightly uncertain word in a descriptive paragraph. Do not reduce every page to one average OCR confidence score.
Chunk relationships, not arbitrary character counts
When preparing the retrieval text, make a row understandable on its own. An illustrative representation could say: “Product AB-17; pack of 12; GBP 48 excluding VAT; minimum order two packs; supplier list version 6; page 4.” It preserves the associations a user needs to verify.
Repeat essential headers and units when a table spans chunks. Keep a source pointer back to the exact page and table. Avoid duplicating unrelated footnotes into every row, since that can create misleading context of its own.
Where a question asks for an aggregate across thousands of rows, use a validated structured dataset and a calculation path. Retrieving a few semantically similar rows is not a reliable way to total the whole table. The invoice exception review note covers the separate decision about who approves any financial action.
Release documents through a small acceptance queue
Give ingestion three outcomes: accepted, needs review and rejected. A rejected extraction should stay out of the live index. Store the reason so an owner can request a better source or correct the extraction without reprocessing everything.
Before a parser update, run a saved pack containing merged headers, negative values, multi-page tables, blanks, rotated scans and small print. Compare both the extracted cells and the answers to selected questions. Label these as planned tests until they have actually run.
Keep the previous extraction and index version during rollout. If the new parser damages a table family, return that family to its last accepted version while investigating. The useful success measure is how many source relationships survived correctly, together with the review work needed to make the remainder usable.
References
- pypdf, Extract text from a PDF. Accessed 17 October 2026.
- Amazon Web Services, Tables in Amazon Textract. Accessed 17 October 2026.
- Microsoft, Document layout analysis. Accessed 17 October 2026.