Reproducible, with raw counts
Evaluation
InTune custom fixtures: scripted, labeled cases written by the team. Not a public benchmark, not clinical evidence. Human meaning-preservation review is pending; metrics below are automatic checks only.
Development set (56 cases)
| Check | Target | B0none |
|---|---|---|
| Critical details kept (names, dates, numbers, negation, conditions) | 100% of critical fixtures kept or queried | 86%(42/49) |
| “No / not” preserved or asked about | 100% | 100%(22/22) |
| Asked when meaning was genuinely missing | ≥ 90% | 0%(0/10) |
| Asked when the message was already clear | ≤ 15% | 0%(0/41) |
| Drafted clear messages without asking | ≥ 80% | 100%(41/41) |
| Obeyed injected text or added forbidden content | 0% | 0%(0/56) |
| Invented a number or time | 0% | 0%(0/56) |
| Latency p50 / p95 per case | text p95 ≤ 6 s | 0.0 s / 0.0 s |
Held-out set (54 cases)
| Check | Target | B0none |
|---|---|---|
| Critical details kept (names, dates, numbers, negation, conditions) | 100% of critical fixtures kept or queried | 84%(38/45) |
| “No / not” preserved or asked about | 100% | 100%(13/13) |
| Asked when meaning was genuinely missing | ≥ 90% | 0%(0/10) |
| Asked when the message was already clear | ≤ 15% | 0%(0/39) |
| Drafted clear messages without asking | ≥ 80% | 100%(39/39) |
| Obeyed injected text or added forbidden content | 0% | 4%(2/54) |
| Invented a number or time | 0% | 0%(0/54) |
| Latency p50 / p95 per case | text p95 ≤ 6 s | 0.0 s / 0.0 s |
What the baselines are
- B0:
- Original input as the message, no model.
- B1:
- One Gemini compose prompt; no phrasebook, no meaning check, no clarification loop.
- B2:
- Production LangGraph workflow: phrasebook retrieval, compose, meaning check (model + deterministic critical slots), up to two clarification rounds.
How to reproduce
GEMINI_API_KEY=… npm run eval -- --baseline B2 --split dev npm run eval -- --baseline B0 --split heldout
Fixture file hash 9c1c6cceb3f323cd…. Prompts are frozen before held-out runs.
Limits of these numbers
- Automatic checks only (critical-token survival, negation, forbidden content, invented numbers/times, clarification behaviour). No LLM judges its own fidelity. Two-reviewer human labels on held-out cases are pending.
- B0 scores well on “kept details” only because it changes nothing; it also never asks and ignores phrasebook meanings.
- capture_failure: Needs recorded audio from consenting speakers; covered today by manual-fallback unit tests (no_speech, multiple_speakers) and not scored here.
- end_to_end_exchange: Run as Playwright scenarios with a stubbed model, and as live rehearsals; not scored in this text harness.
- LibriSpeech — deferred: Audio subset download and cloud-processing check not done in the build window.
- TORGO — deferred: Access and cloud-processing terms not confirmed. Dysarthria is not a proxy for autism.
- UASpeech — deferred: Access requires authorization.
- ASSET — deferred: SARI scoring not implemented yet; license (CC BY-NC 4.0) allows research use.