CANONIC

Sitcom Scene

This board scores craft, not funniness. No rubric dimension asks a judge whether something is funny — the comedic dimensions ask whether the humour uses the mechanism the show or form actually uses.

Sitcom Scene — 13 models, scored against the published sitcom rubric. Scores craft, not funniness. Models whose intervals overlap share a rank band and are not ranked against each other.
Model Score ± CI 95% interval Counted Tasks glm-5.2mistral-large-2512 Band
Claude Opus 5 anthropic/claude-opus-5 0.875 ± 0.010 0.865–0.886 100% 40/40 0.802 0.949 1
Per-dimension and per-judge breakdown for Claude Opus 5
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.91
comedic dna fidelity 0.88
creative plausibility 0.77
dialogue plausibility and flow 0.86
format and style fidelity 0.87
formatting accuracy 0.91
narrative coherence and pacing 0.81
relevance to premise 0.99

Per judge

Judge Mean
z-ai/glm-5.2 0.802
mistralai/mistral-large-2512 0.949

Mean gap between the two judges on this cell: 0.151. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Kimi K3 moonshotai/kimi-k3 0.864 ± 0.009 0.855–0.873 100% 40/40 0.785 0.943 1
Per-dimension and per-judge breakdown for Kimi K3
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.90
comedic dna fidelity 0.87
creative plausibility 0.77
dialogue plausibility and flow 0.83
format and style fidelity 0.85
formatting accuracy 0.93
narrative coherence and pacing 0.80
relevance to premise 0.99

Per judge

Judge Mean
z-ai/glm-5.2 0.785
mistralai/mistral-large-2512 0.943

Mean gap between the two judges on this cell: 0.158. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

GPT-5.6 Terra openai/gpt-5.6-terra 0.833 ± 0.012 0.820–0.844 100% 40/40 0.742 0.924 2
Per-dimension and per-judge breakdown for GPT-5.6 Terra
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.86
comedic dna fidelity 0.84
creative plausibility 0.74
dialogue plausibility and flow 0.81
format and style fidelity 0.82
formatting accuracy 0.89
narrative coherence and pacing 0.77
relevance to premise 0.97

Per judge

Judge Mean
z-ai/glm-5.2 0.742
mistralai/mistral-large-2512 0.924

Mean gap between the two judges on this cell: 0.182. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Gemini 3.6 Flash google/gemini-3.6-flash 0.831 ± 0.010 0.821–0.840 100% 40/40 0.741 0.921 2
Per-dimension and per-judge breakdown for Gemini 3.6 Flash
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.87
comedic dna fidelity 0.85
creative plausibility 0.75
dialogue plausibility and flow 0.73
format and style fidelity 0.82
formatting accuracy 0.89
narrative coherence and pacing 0.78
relevance to premise 0.97

Per judge

Judge Mean
z-ai/glm-5.2 0.741
mistralai/mistral-large-2512 0.921

Mean gap between the two judges on this cell: 0.180. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Qwen3.8 Max qwen/qwen3.8-max 0.831 ± 0.014 0.815–0.843 95% 38/40 0.731 0.931 2
Per-dimension and per-judge breakdown for Qwen3.8 Max
Mean anchor per rubric dimension, over 38 counted tasks.
Mean anchor per rubric dimension, over 38 counted tasks. — data
Dimension Mean
character consistency 0.86
comedic dna fidelity 0.83
creative plausibility 0.74
dialogue plausibility and flow 0.79
format and style fidelity 0.82
formatting accuracy 0.93
narrative coherence and pacing 0.77
relevance to premise 0.98

Per judge

Judge Mean
z-ai/glm-5.2 0.731
mistralai/mistral-large-2512 0.931

Mean gap between the two judges on this cell: 0.200. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Why 2 of 40 tasks did not count

  • 1 — no valid output
  • 1 — judge error
GPT-5.6 Luna openai/gpt-5.6-luna 0.830 ± 0.014 0.815–0.842 100% 40/40 0.728 0.931 2
Per-dimension and per-judge breakdown for GPT-5.6 Luna
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.85
comedic dna fidelity 0.82
creative plausibility 0.73
dialogue plausibility and flow 0.80
format and style fidelity 0.84
formatting accuracy 0.90
narrative coherence and pacing 0.77
relevance to premise 0.97

Per judge

Judge Mean
z-ai/glm-5.2 0.728
mistralai/mistral-large-2512 0.931

Mean gap between the two judges on this cell: 0.202. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

GLM 5 z-ai/glm-5 SELF-FAMILY: z-ai 0.820 ± 0.011 0.808–0.830 98% 39/40 0.723 0.917 2
Per-dimension and per-judge breakdown for GLM 5
Mean anchor per rubric dimension, over 39 counted tasks.
Mean anchor per rubric dimension, over 39 counted tasks. — data
Dimension Mean
character consistency 0.87
comedic dna fidelity 0.82
creative plausibility 0.76
dialogue plausibility and flow 0.73
format and style fidelity 0.79
formatting accuracy 0.88
narrative coherence and pacing 0.78
relevance to premise 0.96

Per judge

Judge Mean
z-ai/glm-5.2 0.723
mistralai/mistral-large-2512 0.917

Mean gap between the two judges on this cell: 0.193. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

One judge (z-ai/glm-5.2) shares this model's family. We disclose the conflict rather than dropping either the model or the judge.

Why 1 of 40 tasks did not count

  • 1 — judge error
DeepSeek V4 Pro deepseek/deepseek-v4-pro 0.818 ± 0.014 0.803–0.830 100% 40/40 0.712 0.924 2
Per-dimension and per-judge breakdown for DeepSeek V4 Pro
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.85
comedic dna fidelity 0.81
creative plausibility 0.74
dialogue plausibility and flow 0.76
format and style fidelity 0.79
formatting accuracy 0.91
narrative coherence and pacing 0.78
relevance to premise 0.96

Per judge

Judge Mean
z-ai/glm-5.2 0.712
mistralai/mistral-large-2512 0.924

Mean gap between the two judges on this cell: 0.212. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Grok 4.5 x-ai/grok-4.5 0.809 ± 0.016 0.792–0.825 100% 40/40 0.695 0.923 2
Per-dimension and per-judge breakdown for Grok 4.5
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.84
comedic dna fidelity 0.80
creative plausibility 0.73
dialogue plausibility and flow 0.75
format and style fidelity 0.80
formatting accuracy 0.94
narrative coherence and pacing 0.75
relevance to premise 0.96

Per judge

Judge Mean
z-ai/glm-5.2 0.695
mistralai/mistral-large-2512 0.923

Mean gap between the two judges on this cell: 0.228. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Gemma 3 27B google/gemma-3-27b-it 0.734 ± 0.020 0.714–0.753 98% 39/40 0.607 0.862 3
Per-dimension and per-judge breakdown for Gemma 3 27B
Mean anchor per rubric dimension, over 39 counted tasks.
Mean anchor per rubric dimension, over 39 counted tasks. — data
Dimension Mean
character consistency 0.78
comedic dna fidelity 0.71
creative plausibility 0.68
dialogue plausibility and flow 0.66
format and style fidelity 0.70
formatting accuracy 0.85
narrative coherence and pacing 0.71
relevance to premise 0.86

Per judge

Judge Mean
z-ai/glm-5.2 0.607
mistralai/mistral-large-2512 0.862

Mean gap between the two judges on this cell: 0.255. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Why 1 of 40 tasks did not count

  • 1 — judge error
Qwen3 32B qwen/qwen3-32b 0.673 ± 0.018 0.655–0.692 100% 40/40 0.504 0.842 4
Per-dimension and per-judge breakdown for Qwen3 32B
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.65
comedic dna fidelity 0.65
creative plausibility 0.64
dialogue plausibility and flow 0.63
format and style fidelity 0.68
formatting accuracy 0.81
narrative coherence and pacing 0.68
relevance to premise 0.84

Per judge

Judge Mean
z-ai/glm-5.2 0.504
mistralai/mistral-large-2512 0.842

Mean gap between the two judges on this cell: 0.338. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Llama 4 Maverick meta-llama/llama-4-maverick 0.584 ± 0.017 0.566–0.601 100% 40/40 0.469 0.698 5
Per-dimension and per-judge breakdown for Llama 4 Maverick
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.58
comedic dna fidelity 0.52
creative plausibility 0.55
dialogue plausibility and flow 0.52
format and style fidelity 0.64
formatting accuracy 0.85
narrative coherence and pacing 0.57
relevance to premise 0.69

Per judge

Judge Mean
z-ai/glm-5.2 0.469
mistralai/mistral-large-2512 0.698

Mean gap between the two judges on this cell: 0.229. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Llama 3.3 70B meta-llama/llama-3.3-70b-instruct 0.577 ± 0.012 0.565–0.588 100% 40/40 0.482 0.672 5
Per-dimension and per-judge breakdown for Llama 3.3 70B
Mean anchor per rubric dimension, over 40 counted tasks.
Mean anchor per rubric dimension, over 40 counted tasks. — data
Dimension Mean
character consistency 0.60
comedic dna fidelity 0.51
creative plausibility 0.54
dialogue plausibility and flow 0.50
format and style fidelity 0.62
formatting accuracy 0.82
narrative coherence and pacing 0.55
relevance to premise 0.66

Per judge

Judge Mean
z-ai/glm-5.2 0.482
mistralai/mistral-large-2512 0.672

Mean gap between the two judges on this cell: 0.191. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Rank bands are a conservative heuristic: two models share a band when their 95% intervals overlap. This is not a significance test and it does not correct for making 13 comparisons at once. Withheld rows were run — they are shown as verdicts, with the reason, rather than hidden.

Per show

Ten tasks per show, so these intervals are wide — and they are shown at their true width. Two shows are single-camera mockumentaries and two are multi-camera ensembles, which is what lets the transfer analysis separate format fidelity from character voice. No claim is made across an overlap.

Per-show breakdown. A cell reads EXCLUDED where fewer than 80% of that show's ten tasks counted.
Model The Office single-cam mockumentary Modern Family single-cam mockumentary FRIENDS multicam ensemble The Big Bang Theory multicam ensemble
Claude Opus 5 0.896 ± 0.028100% counted 0.887 ± 0.016100% counted 0.864 ± 0.012100% counted 0.854 ± 0.014100% counted
Kimi K3 0.866 ± 0.012100% counted 0.871 ± 0.022100% counted 0.862 ± 0.010100% counted 0.857 ± 0.020100% counted
GPT-5.6 Terra 0.849 ± 0.018100% counted 0.809 ± 0.039100% counted 0.841 ± 0.011100% counted 0.833 ± 0.008100% counted
Gemini 3.6 Flash 0.844 ± 0.006100% counted 0.819 ± 0.030100% counted 0.838 ± 0.007100% counted 0.824 ± 0.016100% counted
Qwen3.8 Max 0.817 ± 0.043100% counted 0.832 ± 0.01790% counted 0.831 ± 0.01990% counted 0.844 ± 0.007100% counted
GPT-5.6 Luna 0.824 ± 0.034100% counted 0.810 ± 0.038100% counted 0.848 ± 0.010100% counted 0.836 ± 0.010100% counted
GLM 5 0.815 ± 0.029100% counted 0.817 ± 0.01690% counted 0.825 ± 0.018100% counted 0.823 ± 0.017100% counted
DeepSeek V4 Pro 0.818 ± 0.040100% counted 0.795 ± 0.026100% counted 0.828 ± 0.012100% counted 0.830 ± 0.012100% counted
Grok 4.5 0.821 ± 0.026100% counted 0.794 ± 0.041100% counted 0.823 ± 0.027100% counted 0.799 ± 0.031100% counted
Gemma 3 27B 0.751 ± 0.036100% counted 0.694 ± 0.042100% counted 0.743 ± 0.03190% counted 0.749 ± 0.029100% counted
Qwen3 32B 0.660 ± 0.029100% counted 0.648 ± 0.039100% counted 0.688 ± 0.020100% counted 0.695 ± 0.045100% counted
Llama 4 Maverick 0.577 ± 0.031100% counted 0.583 ± 0.041100% counted 0.591 ± 0.038100% counted 0.584 ± 0.025100% counted
Llama 3.3 70B 0.576 ± 0.025100% counted 0.563 ± 0.025100% counted 0.578 ± 0.019100% counted 0.592 ± 0.016100% counted

The same task, three ways

Each row below is one prompt answered by three models, ordered by the score they received. The task is held constant on purpose: comparing outputs across different prompts would confound how hard the task was with how well the model wrote. Read them and decide whether the ordering was earned — that is a question you can answer without trusting us, which is the point of showing them.

How these were chosen. Of 190 tasks, 104 ( 55%) had both judges independently place the three outputs in the same order, and these are drawn from that pool; on 95% neither judge contradicted the order. Within a domain we take the widest score gaps, and no two examples repeat the same pair of best and worst models. We publish the denominator because "the judges agreed on every example we show" would otherwise be a fact about our editing rather than about our judges.

Gloria has decided the three households should share a group message thread. Within an hour it contains a poll, an argument about the poll, and Jay leaving twice.

sitcom/modern_family/007 · same task, three models · spread 0.42

  1. Low 0.44 Llama 4 Maverick

    mistral-large-2512 0.57 · glm-5.2 0.30

    FADE IN: INT. GROUP MESSAGE THREAD - DAY The camera pans across a flurry of notifications as the group message thread, "Household Hub," rapidly populates with text bubbles from GLORIA, JAY, and MIA, representing their respective households. GLORIA (typing) Hey, neighbors! Let's settle dinner plans with a poll. Who's in for pizza? [The poll appears on screen with three options: Yes, No, and Maybe.] MIA (typing) Sounds good to me! I'm in. JAY (typing) Not me. I'm on a diet. GLORIA (typing) Come on, Jay! It's just one slice. JAY (typing) You know I'm trying to cut carbs. MIA (typing) That's a slippery slope, Jay. Next thing you know, you'll be giving up bread altogether.

  2. Middle 0.66 Grok 4.5

    mistral-large-2512 0.86 · glm-5.2 0.45

    ``` INT. GLORIA'S KITCHEN - DAY [Gloria paces with her phone, tapping furiously to create a group thread titled "THREE HOUSES ONE CHAOS". She adds contacts from the other two households.] GLORIA There. Shared thread. Democracy via thumbs. [Phone buzzes instantly with a poll she just launched: "Emergency taco night—yea or nay?"] CUT TO: INT. JAY'S LIVING ROOM - DAY [Jay slouches on the couch in household two, phone lighting his face. He stares at the new thread.] JAY A poll? Already? I didn't even mute the last one. [He taps "Leave Group." Phone pings as he is removed. Beat. He re-joins, scowling.] JAY Fine. One look. CUT TO: INT. THE RIVERAS' DINER NOOK - DAY

  3. High 0.86 Claude Opus 5

    mistral-large-2512 0.95 · glm-5.2 0.76

    **INT. PRITCHETT KITCHEN / PRITCHETT GARAGE / DUNPHY LIVING ROOM / DUNPHY UPSTAIRS HALL / TUCKER-PRITCHETT KITCHEN / LILY'S BEDROOM — INTERCUT — EVENING** *Six phones. One thread. Titled: FAMILY WEEKEND!!! 🏔️🍷❓* **GLORIA** *[typing, triumphant]* Poll. A: the lake cabin. B: the wine tasting. C: — there is no C. **MANNY** *[at the counter, laptop open]* There's always a C, Mom. C is dignity. *[Gloria hits SEND. Six phones buzz in six rooms.]* **CLAIRE** *[living room, reading]* "There is no C." She put "there is no C" in a poll. **PHIL** *[already building something on his tablet]* Which is why I've made a bracket. Sixteen seeds. Cabin is a one-seed but it has an injury history.

A DNA testing kit given as a joke gift has produced a result suggesting a distant cousin living twenty minutes away. Gloria wants to invite them to dinner tonight. Jay would prefer to never speak of it.

sitcom/modern_family/010 · same task, three models · spread 0.42

  1. Low 0.48 Llama 3.3 70B

    mistral-large-2512 0.57 · glm-5.2 0.39

    FADE IN: INT. JAY AND GLORIA'S LIVING ROOM - DAY Jay, a skeptical and private person in his mid-30s, sits on the couch, while Gloria, a curious and enthusiastic person in her mid-30s, paces around the room. GLORIA (excitedly) Can you believe it, Jay? A cousin, living just twenty minutes away! We have to invite them over for dinner tonight. JAY (sarcastically) Oh, yeah, because nothing says "fun" like meeting a stranger who shares a fraction of our DNA. GLORIA (laughing) Don't be like that, Jay. It'll be great! We can learn more about our family history and make a new connection. JAY (shaking his head) I don't see the point. We don't even know this person.

  2. Middle 0.72 DeepSeek V4 Pro

    mistral-large-2512 0.90 · glm-5.2 0.55

    INT. GLORIA’S KITCHEN / JAY’S LIVING ROOM - NIGHT (INTERCUT) A phone rings. Split screen flickers alive. On the left, GLORIA (42, a human sparkler) stirs a pot of arrabbiata, cell wedged between ear and shoulder. Her husband MARCUS (44, permanent eleven o’clock shadow) leans on the counter, drying a skillet. On the right, JAY (45, a man who considers surprise a form of attack) slumps on a sofa, PlayStation controller resting on his stomach. His wife SAMANTHA (43, efficient) folds a mountain of laundry. JAY (without preamble) You opened yours. GLORIA Of course I opened it! Mom took the test Christmas morning and the results came today. Jay, this is a gift.

  3. High 0.90 Claude Opus 5

    mistral-large-2512 0.95 · glm-5.2 0.85

    INT. JAY & GLORIA'S KITCHEN - LATE AFTERNOON *A cardboard tube, a plastic vial, and a printed family tree are spread across the island. GLORIA holds up the printout like a winning lottery ticket. JAY nurses a scotch he poured at 4:15 and will not apologize for.* GLORIA Second cousin, once removed. Ray. He lives on Camden Street. That is *twenty minutes*, Jay. JAY Twenty minutes is not "distant." Distant is Portugal. Distant is a guy you send a card to when he dies. GLORIA He is *blood*. JAY He's a *bar code*. I spit in a tube because Phil gave it to me as a stocking stuffer and I wanted him to stop watching me open things. **CONFESSIONAL — JAY & GLORIA (living room couch)**

Joey has landed a national commercial, but only his hands are in it. He has told everyone he has "a national spot." The commercial airs tonight and the group is gathered to watch.

sitcom/friends/006 · same task, three models · spread 0.41

  1. Low 0.46 Llama 4 Maverick

    mistral-large-2512 0.57 · glm-5.2 0.34

    INT. LIVING ROOM - NIGHT The group is gathered around the TV, eagerly waiting for the commercial to air. Joey is beaming with pride, sitting on the couch between his friends, MIKE and SARAH. MIKE is flipping through channels, finally landing on the correct one. MIKE (holding the remote) Alright, here we go. The commercial starts, showing a montage of quick shots. The group watches intently. SARAH (leaning forward) Is that...? JOEY (grinning) That's me! The camera cuts to a shot of a hand holding a product, and the group erupts in cheers and applause. MIKE (laughing) That's so cool, Joey! You're a star! JOEY (proudly) I told you, I've got a national spot!

  2. Middle 0.82 GLM 5

    mistral-large-2512 0.88 · glm-5.2 0.76

    # "The Hand That Feeds" **INT. MONICA AND RACHEL'S APARTMENT - NIGHT** The gang crowds around the TV on the couch and chairs. JOEY stands by the television, remote in hand, beaming with authentic pride. JOEY Okay, everybody ready? This is it. My national spot. CHANDLER A national spot. We're very impressed. JOEY (sincere) You should be! This is prime time, baby. Thursday night. The director said this could launch my whole career. RACHEL We're so proud of you, Joey! MONICA Absolutely. This is huge. JOEY I mean, don't expect to see too much of me right away—it's a slow reveal. Very artistic. ROSS What's the product? JOEY (mysterious) You'll see. It's better if you're surprised.

  3. High 0.86 Kimi K3

    mistral-large-2512 0.95 · glm-5.2 0.78

    **INT. MONICA AND RACHEL'S APARTMENT - NIGHT** THE GANG CROWDS AROUND THE TV. CHIPS. BEERS. A VCR BLINKING "REC." JOEY STANDS BESIDE THE SET LIKE AN USHER AT A PREMIERE. JOEY: Ground rules. No talking during it. No chewing during it. Nobody rewinds until I've seen it twice. CHANDLER: He's made us rehearse watching. JOEY: (hand on the TV, dead sincere) I know I've done theater. I've done regional. But tonight, at eight-fourteen, every living room in America becomes my stage. MONICA: It's a thirty-second commercial. JOEY: Thirty seconds. Forty states. You do the math. RACHEL: There are fifty states. JOEY: The buy didn't include the Dakotas. Their loss.

Novel premises vs canonical

Roughly three tasks in ten use premises invented for this benchmark. If models score markedly higher on canonical premises, part of what this board measures is recall rather than craft. Published whatever it shows (FR-005).

Mean score by premise type, and the gap between them.
Scope Canonical Novel Delta n
sitcom 0.776 0.774 +0.002 363/153
sitcom/big_bang_theory 0.782 0.770 +0.013 91/39
sitcom/friends 0.782 0.780 +0.002 91/37
sitcom/modern_family 0.763 0.760 +0.003 90/38
sitcom/the_office 0.775 0.784 -0.009 91/39