Hypernym Modulum vs 5 current-generation frontier inference stacks, evaluated on BABILong long-context retrieval and multi-hop reasoning.

Where Modulum lands among current-generation frontier stacks at 128k context.

BABILong: long-context retrieval and multi-hop reasoning across 32k–1M tokens.

Seven distinct inference stacks compared.

Same Gemma-4-31B-Q4 weights, with and without Hypernym's platform stack.

This is the cleanest possible isolation of platform contribution. Hypernym provisioned a vanilla mirror endpoint serving the identical Gemma-4-31B-Q4 weights via raw llama.cpp without Modulum's components. Both sides receive the same 50 BABILong prompts (idx 0–49) at temperature 0.

Interpretation — statistical honesty

At N=50 per side, only 3 of 8 mask cells reach conventional statistical significance (qa1 64k, qa2 32k, qa2 128k). The qa2 32k cell at +28pp is the strongest single piece of evidence the platform does real architectural work (p=0.003). The other 5 lifts trend positive in the right direction but are below the noise floor at N=50 — including the qa1 128k cell that v1 reports as +14.6pp / now corrected to +10pp / still not significant. The qa3 32k regression (−6pp) is also not significant at this sample size; it could be sample noise, not a real platform side-effect. To convert these into formally publishable claims, the next experiment cycle must extend vanilla N to 200+ per cell.

Accuracy as a function of context length, by task.

How does frontier accuracy hold up at 256k / 512k / 1M context?

Only Gemini 3.1 Pro natively supports context above 200k. Modulum and most other frontier APIs cap at 128k–200k. This data is Gemini-only and informs the hypothesis: does frontier decay accelerate at extreme contexts?

Honest implication (revised)

Gemini 3.1 Pro on qa1 retrieval holds at 128k (84%) and 256k (84%, 0 errors) — no decay measured from 128k → 256k. At 512k, accuracy drops to 66% across 50 attempts (12% of which hit the spending cap; the 44 that completed averaged 75%). At 1M we have no measurement — every attempt was rejected at the API budget layer. The "frontier decay accelerates at extreme contexts" claim is supported only by the 512k cell, not by 1M. Modulum cannot be tested in this range either because the Modulum API is capped at 128k by Hypernym. The cleanest publishable framing: long-context frontier comparison stops at 128k for both stacks; the 256k/512k extended probe is suggestive, not conclusive.

How fast each stack loses ground per doubling of context.

Linear fit of accuracy against log₂(context tokens) across the 32k → 128k window (Gemini extended to 1M). Reported as pp per doubling. Lower magnitude = flatter decay = stronger long-context preservation.

The First, Not Lost claim — what the slope data actually says

Hypernym's published thesis is that Modulum preserves earlier-context facts when frontier models lose them at length. The qa3 slope (−2.5 pp / doubling — better than every other tested stack except Opus 4.7) is the strongest evidence for this. The claim does NOT hold on qa1 retrieval — Opus 4.6 (0pp), GPT-5.5 (−2pp), and Opus 4.7 (−2pp) all have flatter qa1 slopes than Modulum (−8.75pp). The honest framing: Modulum preserves multi-fact reasoning state better than retrieval at this scale. On qa3 in particular, Modulum competes with hyperscaler-scale stacks on slope while running on 16 GB.

Variables a publishable benchmark must report.

The leaderboard reports accuracy. A formally publishable benchmark must also report: prefill rate, decode rate, error patterns, and within-run drift. The Modulum + Vanilla SQLite captures these natively via the llama.cpp timings block; frontier APIs do not.

Decode speed — tokens / sec (median per cell)

Prefill rate — context tokens / sec (median)

Error & failure-mode disclosure

Within-run drift — accuracy by tercile

Split each cell's sample order into 3 equal slices (early / mid / late) and measure accuracy per slice. Indicates whether a model degrades over sustained operation.

How the bench was run.

What we have NOT yet ruled out.

Listed in full because transparency about limitations is the credibility move for partner-side validation.

• C-01

  Modulum vs frontier is NOT bare-model vs bare-model

  Modulum is a proprietary inference platform on a 31B-Q4 open-weight model with minimal orchestration. Frontier comparators (GPT-5.5, Opus 4.6/4.7, Gemini 3.1 Pro, Grok 4.3) are products with internal orchestration we cannot disable: context caching, sparse attention, internal RAG (possibly), thinking-mode reasoning (confirmed for GPT-5.5 and Gemini), tool use. Compute footprint per call is materially larger on frontier than Modulum (orders of magnitude depending on model and runtime, not measured in this bench). Modulum's gap should be read in that light.

• C-02

  Sampling noise on Opus 4.7 and GPT-5.5

  Both models reject temperature=0 in their API (deprecated / unsupported). Default temperature ≈ 1.0 means outputs are sampled, not deterministic. Per-cell accuracy could shift ±3–5pp on rerun. The 13.8pp gap between Modulum and Gemini at qa3 128k N=500 is way above this noise level.

• C-03

  Mask ablation only covers idx 0..49 (N=50)

  Vanilla Gemma-4 was run at N=50 per cell vs Modulum at N=100–500. Apples-to-apples comparisons (same idx 0..49 both sides) shown in section 04. Modulum's full N=200–500 numbers are statistically more confident than the vanilla N=50 baseline.

• C-04

  Single-slot serial Modulum vs production-batched frontier

  Modulum endpoint enforces 1 in-flight request at a time (gateway-level). Frontier APIs run continuous batching with concurrent users. Per-call latency on Modulum is therefore higher than it would be at production scale; cannot be cleanly compared to frontier API latency.

• C-05

  Modulum context-window cap at 128k blocks extreme-context comparison

  Modulum and vanilla Gemma-4 mirror endpoints are documented by Hypernym as capped at 128k context. We did not run probe requests above 128k against Modulum to obtain a direct 4xx confirmation in our SQLite. Important correction: the prior "Gemini 1M = 0%" claim is RETRACTED — that cell was 50/50 HTTP 429 spending-cap failures (Google API budget layer), not Gemini-model-context failures. We have no published evidence either way about Gemini 1M actual capability. The clean comparison ceiling for both stacks today is 128k.

• C-06

  Substring scoring inflates retrieval (qa1) more than reasoning (qa2/qa3)

  qa1 has 6 possible location outputs; substring matching is lenient. Re-scoring with exact-last and unique-loc rules within ±1 row per cell — empirical scoring-bias concern not validated. Still, all reported numbers are substring-rule based.

• C-07

  Anthropic version anomaly

  Opus 4.6 (88.7% 128k avg) substantially outperforms Opus 4.7 (65.3% 128k avg) on this benchmark. Could be: real Anthropic regression 4.6→4.7 on long-context reasoning, OR 4.7's default sampling parameters differ from 4.6's (we couldn't set temperature on 4.7). Reported both transparently. Opus 4.6 is the stronger comparator.

How the data was generated, in order.

What would elevate this from MODERATE to STRONG evidence.

1. ▸ 01

   Modulum context-window extension to 256k+

   Request to Hypernym pending. Critical for testing the central hypothesis that Modulum's platform contribution amplifies at extreme contexts where frontier decays (Gemini at 1M = 0% in our test).

2. ▸ 02

   qa3 32k regression investigation

   Only cell where Modulum platform underperforms vanilla base (−6pp). Is this real (platform over-correction on short-context multi-hop) or sample-specific noise? Worth running at higher N to clarify.

3. ▸ 03

   Production-batched Modulum throughput

   Single-slot demo backend doesn't reflect production-scale serving economics. Real cost/throughput numbers require Hypernym to expose batched inference behavior.

4. ▸ 04

   Modulum + orchestration vs frontier + orchestration

   Current comparison is bare-Modulum vs full-frontier. A fairer comparison would add equivalent orchestration (RAG, chain-of-thought, caching) to Modulum and measure the closure rate.

5. ▸ 05

   Apply Modulum platform to a larger base model

   If Hypernym's platform contribution holds (+13pp avg over bare model), applying it to a 70B+ or 200B+ base could close more of the frontier gap. Currently hypothetical until partner-deployed.

What this data supports vs what it does not.

Single defensible 1-sentence pitch

"Hypernym's Modulum inference platform measurably improves long-context performance over the same Gemma-4-31B-Q4 base by +12.15 pp on average across 9 BABILong cells (3 of 8 reach p<0.05 at N=50), achieves the flattest qa3 multi-fact reasoning decay slope of any tested stack at −2.5 pp per doubling of context, and runs on a workstation-scale single-GPU deployment (Hypernym profile) — though it currently trails current-generation closed-weight frontier products (Opus 4.6, GPT-5.5) on absolute 128k accuracy by 38.0–42.7 pp, making the load-bearing value proposition multi-fact context preservation at workstation scale rather than absolute-capability parity with hyperscaler-served frontier."