Reference architecture 04 Batch inference Exception routing Cost per document

Document pipelines at volume.

At scale the interesting problem is not accuracy on a clean scan. It is cost per document, and what happens to the small fraction that are damaged, rotated, handwritten or in a language nobody planned for.

THE PROBLEM

The demo works on the good scans. The project lives or dies on the bad ones.

Bills of lading, customs forms, invoices and delivery notes arrive continuously, in every condition: crisp PDFs, phone photos taken at an angle, faxes of faxes, forms where the important field is handwritten in a margin. A pilot on a curated sample reports 97% and everyone is delighted.

Then production traffic arrives and the distribution is different. The 3% that fail are not randomly distributed — they cluster on specific trade lanes, specific partners, specific document templates. And because they cluster, they are systematically the documents belonging to particular customers, who now experience the system as broken.

Two things follow. Exception handling is the primary design surface, not a fallback. And unit economics decide viability: a pipeline costing more per document than the clerk it replaces is a science project.

THE DESIGN

DOCUMENT PATH

Receive

email · EDI · upload
Triage

quality + type score
Batch

group by template
Extract

multimodal, batched
Validate

cross-field checks
Exception

typed queue
Emit

to ERP / TMS

Lower box = typed exception queue Marker = one document through the pipeline

Triage runs before inference. Scoring a document as unprocessable costs a fraction of a cent; discovering it after a model call costs orders of magnitude more.

Triage before inference. A cheap classifier scores legibility, orientation, page count and template match first. Rotated pages get rotated; unreadable pages are rejected immediately with a specific reason rather than being sent to an expensive model that will hallucinate something plausible.

Batch by template, not by arrival. Grouping same-template documents lets a single prompt carry template-specific instructions and few-shot examples, which materially improves accuracy and lets you use a smaller model. Batch APIs then cut per-token cost substantially against synchronous calls, in exchange for latency you usually do not need.

Validation is cross-field, not per-field. Individually plausible values that are jointly impossible — a total that is not the sum of its lines, a delivery date before a dispatch date — are the highest-yield check available and cost nothing to run.

Exception queues are typed. "Failed" is not actionable. illegible_scan, unknown_template, arithmetic_mismatch and unsupported_language route to different people with different fixes, and the distribution across those types tells you exactly where to invest next.

pipeline/triage.py PYTHON

def triage(doc):
                    class="text-dim"># Cheap checks first — each one avoids a model call.
                    if doc.ocr_confidence < class="text-plasma">0.55:
                        return Exception_(kind=class="text-emerald-data">"illegible_scan", page=doc.worst_page)

                    if doc.detected_rotation:
                        doc = doc.rotate(doc.detected_rotation)

                    template = match_template(doc)
                    if template is None:
                        return Exception_(kind=class="text-emerald-data">"unknown_template")

                    if template.language not in SUPPORTED:
                        return Exception_(kind=class="text-emerald-data">"unsupported_language",
                                          detail=template.language)

                    return Batchable(doc, template=template)

Triage is deliberately cheap and dumb. Its job is to keep expensive calls away from documents that cannot succeed.

WHERE IT BREAKS

Template drift

A partner silently changes their invoice layout and accuracy degrades for that partner only — invisible in the aggregate. Mitigation: track accuracy per template and per sender, and alert on a per-sender drop rather than a global one.

Exception queues that grow without bound

If exception volume exceeds what the team can clear, the backlog becomes the new bottleneck and you have moved the problem. Mitigation: treat queue depth as a production SLO with an alert, not a report nobody reads.

Batch latency colliding with operations

Batch pricing assumes you can wait. A customs document blocking a container at a port cannot. Mitigation: two lanes — synchronous for time-critical document types, batch for the rest — with the split driven by business rules.

Costs that only look good per document

Per-document inference cost omits retries, exception-handling labour and the engineering time to keep templates current. Mitigation: measure fully loaded cost per successfully emitted record, which is usually well above the naive figure.

WHAT TO MEASURE

Straight-through rate

The share of documents reaching the ERP with no human touch. The headline number, and the one that determines the business case.

Exception mix

Volume by exception type over time. This is the roadmap: whichever type dominates is what to fix next.

Loaded cost per record

Inference, retries, storage and human minutes, divided by successfully emitted records. Compare against the current per-document cost, not against zero.

These are the targets we design against and instrument for; they are not results from a specific deployment. Any figures used above are illustrative.

Keep reading

Services

Intelligent Automation

Document Intelligence

→

30-minute technical call · no deck