Technology

Multimodal AI.
Beyond text.

Multimodal models handle documents, images and audio as native inputs, which removes whole pipelines. They also fail in ways text models do not, and those failures are less visible because the input is harder for a reviewer to skim.

WHERE IT GENUINELY CHANGES THINGS

Documents as documents

A scanned invoice, a form, a technical drawing with annotations. Reading these as images preserves layout, tables and the spatial relationships that a text extraction pipeline destroys — and layout frequently carries the meaning.

Audio without a transcription stage

Processing speech directly retains tone, overlap and timing that a transcript discards. For meeting analysis and quality monitoring, those are often the signal rather than the noise.

Video understanding

Answering questions about a recording without watching it, with references back to timestamps. Genuinely new capability for media archives, inspection footage and recorded sessions.

Screenshots and interfaces

Reading a screen the way a person does. Powerful for support ticket triage and for automating work in systems that offer no API — with the caution that a UI-driven automation is fragile in exactly the way an API-driven one is not.

WHERE THE CLAIMS STOP HOLDING

Dense numeric tables. A model reading a financial statement image can transpose a digit or drift a column, and the output looks entirely plausible. For anything where the number must be right, extract with a purpose-built pipeline and use the model to check for consistency rather than as the source. This is one of the most common expensive mistakes in document AI.

Small text and poor scans. Accuracy degrades with resolution in ways that are not signalled in the output. The model does not say “this was blurry”; it produces its best guess with the same confidence as everything else. Image quality checks belong before the model, not after.

Precise spatial or dimensional reasoning. Reading a technical drawing to summarise it is achievable. Reading it to extract a tolerance you will manufacture to is not, and should not be attempted without verification.

Counting. Counting objects in an image is unreliable at anything beyond small numbers, and it fails silently. If a count matters, use a detection model built for it.

THE COST SHAPE IS DIFFERENT

Images and audio consume far more tokens than the equivalent text, and a high-resolution page can cost more than several pages of extracted text would. For high-volume document processing, that difference frequently decides the architecture: a cheap classifier routing to full multimodal processing only where layout genuinely matters, rather than sending everything through the expensive path.

It also strengthens the case for self-hosting at volume. Document and image workloads are where per-token pricing bites hardest, and where owning the hardware changes what is economically worth processing.

QUESTIONS

QUESTIONS — 3

Sometimes, and it is worth testing rather than assuming in either direction. Multimodal models handle layout and context better than traditional OCR; dedicated OCR is often more accurate on dense text and dramatically cheaper at volume. A hybrid — OCR for text, multimodal for layout and structure — is frequently the right answer.

Build an evaluation set from your real documents with the correct extractions labelled, including the awkward ones — poor scans, unusual layouts, handwriting. Aggregate accuracy on clean documents tells you very little about production, where the awkward ones are the whole problem.

Yes. Open-weight vision-language models are capable and improving, and for document workloads involving confidential material — which is most of them — that is usually the right deployment. Memory requirements are higher than text-only models of comparable size, so size accordingly.

Keep reading

Models

Building with Google Gemini

→

Reference

Document Intelligence

30-minute technical call · no deck