CANONIC

Canonic · a verification company for unverifiable domains

RL environments for creative work, with a grader you can audit.

For post-training teams training agents on writing, comedy and other work with no unit test at the end of it. RL improves an agent only as far as its environment can measure. Coding and maths got graders; creative work got preference scores. We build the grader, ship it as a pinned environment, and publish where it breaks.

Not a benchmark. An instrument.

Talk to us about a pilot See the Stage 1 results

What the grader actually reads

Two models answering the same task, from the Stage 1 sweep. Not a curated pair from different prompts — that would confound how hard the task was with how good the model is.

Task The opening of a comic novel about an institution.

Llama 4 Maverick 0.50
The Institute for Unlikely Sciences stood like a sentinel on the outskirts of town, its imposing façade a testament to the enduring power of Victorian optimism. As the morning sun struggled to penetrate the grimy windows, a faint hum of activity began to emanate from within. In the foyer, the Director, Dr. Helena Anders, stood poised behind a polished wooden counter, her eyes fixed on the sweeping curves of the staircase as she awaited the arrival of her senior staff.

glm-5.2: 0.30 · mistral-large-2512: 0.70

Claude Opus 5 0.95
The Institute had been founded in 1874 by a deed of gift whose seventh clause required that the Governing Body consist of nine members, and whose eleventh clause, added by the founder's widow in a spirit of correction, required that no more than eight of them be alive at any one time. Lawyers had been consulted. The lawyers had confirmed that the clauses were both valid, and that they were consistent, and that the consequence of their consistency was that the Institute could meet but could not resolve, since a resolution required the assent of a majority of the whole number and the whole number included Chair Nine, who was, by the terms of the gift, deceased.

glm-5.2: 0.95 · mistral-large-2512: 0.95

Both judges are named above each excerpt because the claim is that they ordered these independently, and an averaged score cannot show that. Across the sweep, both judges ordered 180 of 190 eligible tasks without contradicting each other.

6 domains. 6 separate scales.

One board per domain, each on its own rubric, and no combined view anywhere. The scales do not convert: craft in a sitcom scene and craft in a book opening are not the same quantity.

On 5 of 6 boards the top band is a tie — the intervals overlap, so the board declines to order them. That is the finding, not a gap in it.

Results by domain

Sitcom Scene: mean rubric score with 95% percentile-bootstrap intervals. Columns are coloured by provider family; the top rank band is outlined in ink and ticked TOP BAND.

2 models share the top band. Their intervals overlap, so this board does not rank them against each other and neither should you.

Scores craft, not funniness.

Full board: Sitcom Scene →

Stand-Up Set: mean rubric score with 95% percentile-bootstrap intervals. Columns are coloured by provider family; the top rank band is outlined in ink and ticked TOP BAND.

2 models share the top band. Their intervals overlap, so this board does not rank them against each other and neither should you.

Scores craft, not funniness.

Full board: Stand-Up Set →

Movie Scene: mean rubric score with 95% percentile-bootstrap intervals. Columns are coloured by provider family; the top rank band is outlined in ink and ticked TOP BAND.

9 models share the top band. Their intervals overlap, so this board does not rank them against each other and neither should you.

Full board: Movie Scene →

Song Lyrics: mean rubric score with 95% percentile-bootstrap intervals. Columns are coloured by provider family; the top rank band is outlined in ink and ticked TOP BAND.

One model holds the top band here — the only domain where the evidence separates a single winner. 1 row is withheld, shown as a verdict with its reason rather than dropped.

Full board: Song Lyrics →

Meme Caption: mean rubric score with 95% percentile-bootstrap intervals. Columns are coloured by provider family; the top rank band is outlined in ink and ticked TOP BAND.

13 models share the top band. Their intervals overlap, so this board does not rank them against each other and neither should you.

Scores craft, not funniness.

Full board: Meme Caption →

Book Opening: mean rubric score with 95% percentile-bootstrap intervals. Columns are coloured by provider family; the top rank band is outlined in ink and ticked TOP BAND.

11 models share the top band. Their intervals overlap, so this board does not rank them against each other and neither should you.

Full board: Book Opening →

Per-domain — no overall score exists. Each tab is one domain measured on its own rubric. There is no view that combines them: craft in a sitcom scene and craft in a book opening are not the same quantity, and averaging them would produce a number that means nothing while looking authoritative.

All 78 rows across 13 models →

What we got wrong

Every evaluation company claims rigour. The cheap way to prove it is to publish where our own instrument failed, on the same page as the results.

2

of 8 reward-hacking probes scored above our threshold — they beat our own rubric. We published them unaltered rather than retuning until they failed. The rubric caught the other 6

1

row is withheld rather than estimated — Kimi K3, shown as a verdict with its reason, never as a low score

0

human-validation studies run. We do not publish an agreement statistic because we have not earned one. The judges are 2 language models from different families, and that is the whole of it

Next

A seventh domain asking whether a model actually knows a place, rather than defaulting to the nearest postcard. Models write Mangalore as Kochi's fishing nets and Kerala's houseboats.

Not built and not measured. Listed as the next benchmark so it is clear what we are working on, not as a capability we are selling.

The demonstration, and what does not exist yet →

Verification, not vibes.

If you are training on creative work and cannot tell whether the reward means anything, that is the problem we build for.

Talk to us about a pilot