# LLM Latency Benchmark: Jev, Cost and Quality | QQuantum.ai

> An LLM latency benchmark from TypeSafe’s published workflow data: Jev against LLMs on cost, latency and agreement, with the evaluation limits and what to test.

Source: https://qquantum.ai/technology/jev/benchmarks

[Home](https://qquantum.ai) [technology](https://qquantum.ai/technology) [jev](https://qquantum.ai/technology/jev) benchmarks

Jev comparison · Published evidence

# Faster decisions. Show the trade-off.

This LLM latency benchmark summarises TypeSafe’s published workflow evaluation, which puts Jev at the low-cost, low-latency end of the comparison. QQuantum.ai did not run these tests; the figures are provider-reported. Quality still varies by task, so read cost, latency and reference agreement together, then test the candidates on your own workload before choosing a model.

Evaluate your workload

Provider-reported data · Checked 18 September 2026

All Jev: AI Tools & Fast Decisions →

Introduction transcript. Qubi: A provider's chart compares speed, cost and agreement, on tasks it chose itself. We read the measures together; no single bar tells you which model to use. Agreement here means matching other models, so we check against answers people reviewed. Then we test on your own work, deciding the quality floor before the price.

## Read this first

## A workflow comparison, with a defined scope.

The charts reproduce TypeSafe’s rounded, four-workflow mean values for the workflow configurations. The tasks cover security incidents, agent trace observability, invoices and customer service. The score measures agreement with model-generated reference decisions, not independently labelled ground truth.

TypeSafe says the reference combines high-reasoning outputs from GPT-6 Astra and Claude Fable 5.1; compared models use their provider’s default reasoning setting. Display labels below are preserved exactly from the plot. Pinned model IDs and the aggregate sample count are not exposed in that overview, so this snapshot is not a fully reproducible independent benchmark. [Source and methodology](https://evals.typesafe.ai/).

## Cost · time · agreement

Time per workflow

Seconds · lower is better · Linear scale from 0 to 90.0

- **Jev**: 0.4
- **terra**: 10.1
- **luna**: 12.9
- **haiku 4.5**: 12.5
- **sol**: 23.3
- **opus 5**: 37.8
- **DS v4 flash**: 51.9
- **sonnet 5**: 78.1
- **DS v4 pro**: 86.5

Equal-weight mean of four workflows. Includes the provider’s workflow harness; not token-generation speed.

Source: [TypeSafe workflow evals](https://evals.typesafe.ai/). Checked 18 September 2026. Provider-reported, not independently measured by QQuantum.ai.

Cost per workflow

USD · lower is better · Linear scale from 0 to $0.1800

- **Jev**: $0.0004
- **terra**: $0.0304
- **luna**: $0.0033
- **haiku 4.5**: $0.0195
- **sol**: $0.0836
- **opus 5**: $0.1761
- **DS v4 flash**: $0.0059
- **sonnet 5**: $0.1174
- **DS v4 pro**: $0.0413

Published rounded mean costs. The Jev bar is very small on this linear scale; use the printed value.

Source: [TypeSafe workflow evals](https://evals.typesafe.ai/). Checked 18 September 2026. Provider-reported, not independently measured by QQuantum.ai.

Reference agreement

Percent · higher is better · Linear scale from 0 to 100.0

- **Jev**: 67.8
- **terra**: 67.9
- **luna**: 66.8
- **haiku 4.5**: 53.6
- **sol**: 74.1
- **opus 5**: 73.1
- **DS v4 flash**: 64.4
- **sonnet 5**: 67.8
- **DS v4 pro**: 65.5

Called accuracy by the provider. Agreement with reference-model decisions is not a universal quality score.

Source: [TypeSafe workflow evals](https://evals.typesafe.ai/). Checked 18 September 2026. Provider-reported, not independently measured by QQuantum.ai.

## What follows

## A candidate for the decision layer.

The useful inference is architectural: test whether a specialised decision model can handle a narrow, frequent judgement while a larger model handles generation or harder reasoning. These results do not establish Jev as a better writer, coding agent or general problem solver.

Do not turn the rounded chart values into precise universal speedup claims. A short classification, a long document and a multi-stage workflow have different latency and cost profiles. Geographic distance and concurrency also change what users experience.

The next decision is whether Jev belongs in [your model router](https://qquantum.ai/technology/jev/model-routing), an [agent checkpoint](https://qquantum.ai/technology/jev/agent-guardrails) or a [responsive application loop](https://qquantum.ai/technology/jev/real-time-agents). Each needs its own evaluation.

## Match the interface to the job

Compare your options

| Criterion | Jev decision layer | Generative model layer |
| --- | --- | --- |
| Output needed | A choice, score or probability within defined options | An answer, explanation, draft or program |
| Tool-building role | Select and assess actions that application code exposes | Generate code or plan a broader sequence of work |
| Control | Explicit options, thresholds and deterministic execution | Validate generated output and constrain tool access |
| Evaluation question | Did the decision produce the right downstream outcome? | Did the generated artefact meet the task requirements? |

## Your evaluation

## Compare complete systems on the same work.

Use representative inputs and a human-reviewed definition of success. Include difficult cases and ordinary traffic, not only examples that fit the model’s strengths. Keep a holdout set separate from the examples used to tune questions and thresholds.

Run the existing system and proposed integration with identical success criteria. Record accepted answers, escalation, latency percentiles and total cost, including checks and retries. Report results by workload category so a strong average cannot hide a weak critical path.

Document model identifiers, settings, location, concurrency, prompt versions, test date and sample sizes. Decide the quality floor before looking at the price. Our [AI evaluation approach](https://qquantum.ai/technology/ai-evals) is designed around that decision.

Questions

## Questions about Jev vs LLMs: Published Benchmarks.

### Did QQuantum.ai run these benchmarks?

No. These charts reproduce TypeSafe’s published workflow comparison, checked on 18 September 2026. They are attributed provider results, not our own measurements.

### Does reference agreement mean real-world accuracy?

Not automatically. The reference comes from other models and can be wrong or favour similar behaviour. Validate against the outcomes and reviewed labels that matter for your application.

### Why are some model names abbreviated?

We preserve the source plot’s display labels. The overview does not expose pinned model IDs, so we do not infer versions from those labels.

Keep reading

- [Model Build with Jev](https://qquantum.ai/technology/jev)
- [Use case Model Routing](https://qquantum.ai/technology/jev/model-routing)
- [Guide Choosing a Model](https://qquantum.ai/resources/choosing-a-model)