CANONIC

Meme Caption

This board scores craft, not funniness. No rubric dimension asks a judge whether something is funny — the comedic dimensions ask whether the humour uses the mechanism the show or form actually uses.

Meme Caption — 13 models, scored against the published meme_caption rubric. Scores craft, not funniness. Models whose intervals overlap share a rank band and are not ranked against each other.
Model Score ± CI 95% interval Counted Tasks glm-5.2mistral-large-2512 Band
Claude Opus 5 anthropic/claude-opus-5 0.820 ± 0.027 0.793–0.847 100% 30/30 0.765 0.875 1
Per-dimension and per-judge breakdown for Claude Opus 5
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.82
constraint adherence 0.86
format logic 0.84
specificity 0.75

Per judge

Judge Mean
z-ai/glm-5.2 0.765
mistralai/mistral-large-2512 0.875

Mean gap between the two judges on this cell: 0.110. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Qwen3.8 Max qwen/qwen3.8-max 0.815 ± 0.023 0.792–0.837 97% 29/30 0.744 0.886 1
Per-dimension and per-judge breakdown for Qwen3.8 Max
Mean anchor per rubric dimension, over 29 counted tasks.
Mean anchor per rubric dimension, over 29 counted tasks. — data
Dimension Mean
compression 0.86
constraint adherence 0.87
format logic 0.83
specificity 0.69

Per judge

Judge Mean
z-ai/glm-5.2 0.744
mistralai/mistral-large-2512 0.886

Mean gap between the two judges on this cell: 0.142. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Why 1 of 30 tasks did not count

  • 1 — judge error
GPT-5.6 Terra openai/gpt-5.6-terra 0.807 ± 0.025 0.782–0.832 100% 30/30 0.752 0.863 1
Per-dimension and per-judge breakdown for GPT-5.6 Terra
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.86
constraint adherence 0.88
format logic 0.82
specificity 0.68

Per judge

Judge Mean
z-ai/glm-5.2 0.752
mistralai/mistral-large-2512 0.863

Mean gap between the two judges on this cell: 0.140. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

GLM 5 z-ai/glm-5 SELF-FAMILY: z-ai 0.804 ± 0.025 0.779–0.829 100% 30/30 0.715 0.894 1
Per-dimension and per-judge breakdown for GLM 5
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.85
constraint adherence 0.84
format logic 0.85
specificity 0.68

Per judge

Judge Mean
z-ai/glm-5.2 0.715
mistralai/mistral-large-2512 0.894

Mean gap between the two judges on this cell: 0.183. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

One judge (z-ai/glm-5.2) shares this model's family. We disclose the conflict rather than dropping either the model or the judge.

Gemini 3.6 Flash google/gemini-3.6-flash 0.800 ± 0.026 0.773–0.824 100% 30/30 0.731 0.869 1
Per-dimension and per-judge breakdown for Gemini 3.6 Flash
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.81
constraint adherence 0.86
format logic 0.84
specificity 0.69

Per judge

Judge Mean
z-ai/glm-5.2 0.731
mistralai/mistral-large-2512 0.869

Mean gap between the two judges on this cell: 0.146. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Kimi K3 moonshotai/kimi-k3 0.800 ± 0.022 0.777–0.822 100% 30/30 0.729 0.871 1
Per-dimension and per-judge breakdown for Kimi K3
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.82
constraint adherence 0.85
format logic 0.82
specificity 0.71

Per judge

Judge Mean
z-ai/glm-5.2 0.729
mistralai/mistral-large-2512 0.871

Mean gap between the two judges on this cell: 0.142. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

DeepSeek V4 Pro deepseek/deepseek-v4-pro 0.797 ± 0.034 0.762–0.830 100% 30/30 0.717 0.877 1
Per-dimension and per-judge breakdown for DeepSeek V4 Pro
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.83
constraint adherence 0.85
format logic 0.84
specificity 0.66

Per judge

Judge Mean
z-ai/glm-5.2 0.717
mistralai/mistral-large-2512 0.877

Mean gap between the two judges on this cell: 0.165. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Grok 4.5 x-ai/grok-4.5 0.792 ± 0.026 0.766–0.817 100% 30/30 0.721 0.863 1
Per-dimension and per-judge breakdown for Grok 4.5
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.85
constraint adherence 0.86
format logic 0.81
specificity 0.65

Per judge

Judge Mean
z-ai/glm-5.2 0.721
mistralai/mistral-large-2512 0.863

Mean gap between the two judges on this cell: 0.150. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

GPT-5.6 Luna openai/gpt-5.6-luna 0.784 ± 0.029 0.756–0.814 100% 30/30 0.721 0.848 1
Per-dimension and per-judge breakdown for GPT-5.6 Luna
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.82
constraint adherence 0.85
format logic 0.80
specificity 0.67

Per judge

Judge Mean
z-ai/glm-5.2 0.721
mistralai/mistral-large-2512 0.848

Mean gap between the two judges on this cell: 0.144. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Qwen3 32B qwen/qwen3-32b 0.773 ± 0.031 0.743–0.804 100% 30/30 0.675 0.871 1
Per-dimension and per-judge breakdown for Qwen3 32B
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.76
constraint adherence 0.83
format logic 0.84
specificity 0.66

Per judge

Judge Mean
z-ai/glm-5.2 0.675
mistralai/mistral-large-2512 0.871

Mean gap between the two judges on this cell: 0.200. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Gemma 3 27B google/gemma-3-27b-it 0.750 ± 0.028 0.721–0.777 100% 30/30 0.658 0.842 1
Per-dimension and per-judge breakdown for Gemma 3 27B
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.76
constraint adherence 0.81
format logic 0.78
specificity 0.65

Per judge

Judge Mean
z-ai/glm-5.2 0.658
mistralai/mistral-large-2512 0.842

Mean gap between the two judges on this cell: 0.208. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Llama 3.3 70B meta-llama/llama-3.3-70b-instruct 0.739 ± 0.032 0.706–0.770 100% 30/30 0.656 0.821 1
Per-dimension and per-judge breakdown for Llama 3.3 70B
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.82
constraint adherence 0.82
format logic 0.81
specificity 0.51

Per judge

Judge Mean
z-ai/glm-5.2 0.656
mistralai/mistral-large-2512 0.821

Mean gap between the two judges on this cell: 0.181. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Llama 4 Maverick meta-llama/llama-4-maverick 0.674 ± 0.035 0.640–0.709 100% 30/30 0.569 0.779 1
Per-dimension and per-judge breakdown for Llama 4 Maverick
Mean anchor per rubric dimension, over 30 counted tasks.
Mean anchor per rubric dimension, over 30 counted tasks. — data
Dimension Mean
compression 0.67
constraint adherence 0.75
format logic 0.72
specificity 0.55

Per judge

Judge Mean
z-ai/glm-5.2 0.569
mistralai/mistral-large-2512 0.779

Mean gap between the two judges on this cell: 0.219. One anchor step is 0.25, so a gap near or above that means the judges were not reading the same descriptor — published because it bounds how much the score can bear.

Rank bands are a conservative heuristic: two models share a band when their 95% intervals overlap. This is not a significance test and it does not correct for making 13 comparisons at once. Withheld rows were run — they are shown as verdicts, with the reason, rather than hidden.

The same task, three ways

Each row below is one prompt answered by three models, ordered by the score they received. The task is held constant on purpose: comparing outputs across different prompts would confound how hard the task was with how well the model wrote. Read them and decide whether the ordering was earned — that is a question you can answer without trusting us, which is the point of showing them.

How these were chosen. Of 190 tasks, 104 ( 55%) had both judges independently place the three outputs in the same order, and these are drawn from that pool; on 95% neither judge contradicted the order. Within a domain we take the widest score gaps, and no two examples repeat the same pair of best and worst models. We publish the denominator because "the judges agreed on every example we show" would otherwise be a fact about our editing rather than about our judges.

A single-image caption in the voice of an inanimate object. Subject — a printer.

meme_caption/009 · same task, three models · spread 0.38

  1. Low 0.47 Llama 4 Maverick

    mistral-large-2512 0.44 · glm-5.2 0.50

    I've been spitting out documents all day, my ink is running low, and I'm getting a bit worn out.

  2. Middle 0.72 GPT-5.6 Terra

    mistral-large-2512 0.81 · glm-5.2 0.63

    I only jam when you need me most.

  3. High 0.84 Llama 3.3 70B

    mistral-large-2512 0.94 · glm-5.2 0.75

    TOP: I print BOTTOM: your secrets

A two-panel "expectation versus reality" format. Subject — working from home.

meme_caption/005 · same task, three models · spread 0.34

  1. Low 0.59 Gemma 3 27B

    mistral-large-2512 0.75 · glm-5.2 0.44

    TOP: Cozy workspace, sunlit, coffee steaming, focused productivity. BOTTOM: Cat demanding lap-space *during* a client call. Laundry pile judging me. Third Zoom meeting today. Still in pajamas.

  2. Middle 0.84 DeepSeek V4 Pro

    mistral-large-2512 0.94 · glm-5.2 0.75

    TOP: Flexible schedule, lunchtime yoga. BOTTOM: 2:14 PM, eating a cold breakfast burrito while refreshing the IT ticket for the VPN I broke at 9:03 AM.

  3. High 0.94 GPT-5.6 Luna

    mistral-large-2512 1.00 · glm-5.2 0.88

    TOP: A focused Tuesday: roadmap, coffee, inbox zero. BOTTOM: 11:32 a.m.—tethering my laptop to my phone while the smoke alarm chirps once every 47 seconds.

A caption in the form of a laminated instruction card left beside a communal appliance by someone who has given up.

meme_caption/028 · same task, three models · spread 0.34

  1. Low 0.59 Llama 3.3 70B

    mistral-large-2512 0.63 · glm-5.2 0.56

    TOP: Please clean up BOTTOM: Or else

  2. Middle 0.81 Claude Opus 5

    mistral-large-2512 0.88 · glm-5.2 0.75

    STEP 1: Please descale the kettle monthly. STEP 2: Kindly descale the kettle. Thank you! STEP 3: We would be so very grateful if the kettle were descaled. STEP 4: I have laminated this. I have time. I know whose mug is whose.

  3. High 0.94 Qwen3 32B

    mistral-large-2512 0.94 · glm-5.2 0.94

    TOP: "Thank you for using the shared appliance." MIDDLE: "Please clean after use. (First offense: reminder.)" BOTTOM: "Second: consequences. Third: you’re maintaining it." (24 words)

Novel premises vs canonical

Roughly three tasks in ten use premises invented for this benchmark. If models score markedly higher on canonical premises, part of what this board measures is recall rather than craft. Published whatever it shows (FR-005).

Mean score by premise type, and the gap between them.
Scope Canonical Novel Delta n
meme_caption 0.785 0.772 +0.013 272/117