How this works

The tasks

Six domains. Each task gives a premise, a set of constraints, and a required output shape — identical for every model, wrapped in an identical prompt template.

Tasks are written from the structural patterns of canonical work, never from verbatim excerpts. Roughly three in ten use premises invented for this benchmark, so the gap between novel and canonical performance is measurable rather than assumed. For the sitcom domain that split applies per show, not just per domain.

• Sitcom Scene
• Stand-Up Set
• Movie Scene
• Song Lyrics
• Meme Caption
• Book Opening

The rubrics

Each domain has one rubric. Every dimension names a question, five anchor descriptors at 0.0 / 0.25 / 0.5 / 0.75 / 1.0, and a weight; weights sum to 1. Judges pick an anchor, never a free-scale number — that is what makes the anchor text mean anything. The weighted score is computed by the harness, never by the judge.

Sitcom dimensions split in two. Portable dimensions score craft any reader could assess. Show-conditioned dimensions score fidelity to a named Show Bible, supplied verbatim in the judge prompt. Adding a show costs one document, not a corpus.

No dimension asks whether something is funny. Funniness is not a judgeable property, and a benchmark claiming to measure it would be overclaiming in its first column. The comedic dimension asks whether the humour uses the mechanism the show actually uses — which is a fidelity question, and answerable.

Every rubric was content-hashed before the sweep. The fingerprints, and what a mismatch would prove, are on the verification page.

The Show Bibles

A Show Bible is the scoring standard for one show: established character voices, the comedic mechanism, format conventions, recurring devices. It is injected verbatim into the judge prompt for every show-conditioned dimension, so a judge scores fidelity to this document rather than to its own recollection of the show.

The four shows are a deliberate 2×2 of format against voice. That is what lets the transfer analysis tell a model failing format conventions apart from one failing character voice.

Bible quality bounds every score in that show's column, which is why they are published in full rather than summarised. A weak bible should be visible to you, not hidden from you.

From an output to a number

1. Generate. One output per model per task, at temperature 0.7, capped at 12000 tokens, through an identical prompt wrapper. Empty, filtered, or truncated output gets exactly one retry, then is recorded as failed and never judged.
2. Judge, twice. Two judges from different families score every valid output independently against the rubric. Each returns per-dimension anchors and a rationale.
3. Classify. Each judge call resolves to exactly one of score, refusal, or error. A refusal gets one retry with the craft framing made explicit; if that scores, the task is still excluded and appears only in the reframe appendix.
4. Count. A task counts only if both judges scored it on the standard prompt. The task score is the mean of the two weighted scores.
5. Aggregate. A domain score is the mean over counted tasks, with a 95% percentile bootstrap interval over tasks — 10,000 resamples, seeded per model and domain so the same inputs always give the same interval.
6. Publish, or refuse to. A row publishes only if every task was attempted and at least 80% counted. Otherwise it renders as an EXCLUDED verdict with its reason. There is no partial row.
7. Band. Models whose intervals overlap share a rank band and are not ordered against one another.

There is no overall score, and no field in the published artifact could hold one. Craft in a sitcom scene and craft in a book opening are not the same quantity, and averaging them would produce a number that means nothing while looking authoritative.

The freeze procedure

Rubrics are tuned before the sweep, on a three-model development subset, with models referred to by neutral labels. The label-to-model mapping is written to the run journal encrypted, and the key is only released by the freeze event itself — so during tuning it is not merely inconvenient to learn which model is which, it is not possible.

Freezing content-hashes every rubric, every Show Bible, and the task set into the journal. Editing any of them afterwards forces a version bump and a re-sweep, because it changes what the published numbers mean.

Task set

418094d0cfd08153da83ca8406793fbe

Frozen at

2026-08-08T07:36:31.813872Z

Models and judges

Judges: z-ai/glm-5.2 (zai) and mistralai/mistral-large-2512 (mistral). Different families on purpose — two judges from one house would be one opinion counted twice. Where a model shares a family with a judge, the row carries a self-family flag; we disclose the risk rather than correcting for it, since any correction would be another unvalidated model.