We ran production regna-1 against RouterBench, the public routing benchmark introduced by Martian. Every number below is a measurement from 2026-07-13, with the methodology and its limits stated in full.
200 questions sampled across all 57 MMLU subjects in RouterBench. Reference scores are the original benchmark's recorded results for its 2024-era models, computed on the same 200 samples.
| Model | Accuracy |
|---|---|
| regna-1 (2026-07) | 93.0% |
| gpt-4-1106-preview | 86.0% |
| gpt-3.5-turbo-1106 | 73.5% |
| claude-v2 | 68.5% |
| mixtral-8x7b-chat | 68.0% |
200 questions from the ARC-Challenge science reasoning set, same protocol.
| Model | Accuracy |
|---|---|
| gpt-4-1106-preview | 97.5% |
| regna-1 (2026-07) | 95.5% |
| claude-v2 | 90.0% |
| mixtral-8x7b-chat | 85.0% |
| gpt-3.5-turbo-1106 | 83.0% |
Dataset. RouterBench 0-shot (withmartian/routerbench). We sampled 200 MMLU and 200 ARC-Challenge prompts with a fixed random seed (42).
Grading. RouterBench does not ship gold answers, so we recovered them by majority vote over the reference models' responses that the benchmark scored as correct. Gold was recoverable for 95.5% of MMLU and 99.0% of ARC items; unrecoverable items were excluded before sampling, which removes the hardest tail for all models equally.
Run. Production regna-1 API, temperature 0, max_tokens 1500, 2026-07-13. All 400 requests succeeded. Average billed cost: 1 credit ($0.01) per question. Answers were parsed as letter choices; parse failure counted as wrong (there were none).
What this does and does not show. The reference models are the ones RouterBench shipped with, from 2024. regna-1 orchestrates newer-generation models, so this is not an era-adjusted model comparison. What it shows is what you actually buy: the delivered accuracy and the billed cost of one regna-1 call on standard benchmark questions, measured end to end through the production gateway, including routing overhead.
Reproducibility. Evaluation scripts and raw per-question results are archived in our repository and available on request: support@regnax.ai.