Scientific Evaluation Standards (2026)

The Epistemology of Accuracy: Benchmarking in 2026

The proliferation of end-to-end Vision-Language Models (VLMs) required a fundamental departure from traditional Character Error Rate (CER) and Word Error Rate (WER) metrics. Traditional string-matching algorithms fail to capture topological correctness, multi-column reading orders, and mathematical equivalence.

1. OlmOCR-Bench: Deterministic Pass/Fail Unit Testing

Published by AllenAI (Ai2), OlmOCR-Bench abandons string-distance scoring in favor of 8,413 deterministic, pass/fail unit tests executed across 1,403 highly complex PDF pages. It evaluates seven rigorous categories:

▪ArXiv Math: Verifies rendered LaTeX bounding boxes rather than raw string tokens.
▪Table Structure Preservation: Checks spatial neighboring cell relationships rather than simple text concatenation.
▪Header and Footer Suppression: Ensures repetitive page running headers do not corrupt downstream LLM context.
▪Multi-Column Reading Order: Validates natural human reading flows across complex newspaper and academic layouts.
▪Historical Degraded Scans: Tests extreme physical wear, low-DPI noise, and faded inks.

Why Edit Distance Failed: Traditional Edit Distance heavily penalizes a model that outputs \dfrac{a}{b} instead of ground-truth \frac{a}{b}, even though both render the exact same formula. OlmOCR-Bench evaluates rendered DOM bounding boxes, correctly scoring mathematically equivalent outputs as successful passes.

2. Tree-Edit-Distance-based Similarity (TEDS) for Tables

For table extraction, raw character accuracy is insufficient because cell associations and merged headers must be preserved. We evaluate table structure using the TEDS metric (ICDAR Table Recognition Standard), representing tables as HTML tree nodes and computing tree edit distance:

TEDS(Ta, Tb) = 1 - (TreeEditDistance(Ta, Tb) / max(|Ta|, |Tb|))

3. Latency & Hardware Profiling Protocols

All latency benchmarks represent single-page 300 DPI 1.2MB image/PDF uploads measured from a c6i.2xlarge AWS client instance to US-East / EU-West endpoints under 50 concurrent requests. Open-source models are benchmarked using vLLM batch inference on a dedicated NVIDIA A100 80GB GPU. We report median (p50) and 95th percentile (p95) response times.