Scoring code needs testing too

Protocol status — 2026-09-11

This is a proposed, unrun protocol dated 2026-09-11. No benchmark audit or scoring re-run has been performed for this article.

Before you use a writing-benchmark score in an editorial decision, inspect how the benchmark turns its inputs into that score. Record the source, version, rubric, fixtures, transformations, exclusions, aggregation, and rounding. If those parts are unavailable, record them as uncollected rather than filling the gaps with assumptions.

Our purpose here is narrow: define a future source-and-scoring check for two existing writing benchmarks without claiming that the check has happened. The protocol does not report a benchmark result, model ranking, accuracy measure, or product capability.

Define the question before collecting material

Start with a reproduction question:

Which documented inputs, rubric, scoring steps, and aggregation rules would be needed to reproduce the reported score?

That question keeps the review close to the published claim. It does not ask which model writes best, whether a benchmark predicts production quality, or whether one benchmark is more valid in general. Those questions need separate methods and evidence.

For each benchmark under review, write down the exact claim being checked. Then identify the material needed to inspect it:

• the original source URL or document;
• the retrieved date;
• the disclosed benchmark version;
• the scoring-code location, if code is published;
• the rubric and input mapping;
• the permitted input material;
• the fixture provenance;
• the evaluated model and version, if disclosed; and
• the reported calculation or output.

Treat these as proposed controls, not as validated best practice or measured findings. If a field is unavailable, mark it uncollected. Missing material in the current record is not evidence that a benchmark lacks documentation. It only means that the material is not available for this proposed review.

Pin the source material

A future run should begin by recording the primary source used for each claim. Save the relevant URL, document identifier, repository state, version, and retrieval date where those details are available.

Do not rely on a benchmark name alone. A name may refer to more than one release, scoring file, fixture set, or published explanation. The audit record should identify the particular material inspected so a later reader can distinguish a source change from a scoring change.

The same rule applies to code. Record the file path, commit or release identifier, execution instructions, dependencies, and permissions needed to inspect or run it when those details are published. If a scoring file is referenced but not available, leave its location and status uncollected.

Use the original source for methodological explanations. The current publication record does not include the primary benchmark materials identified in the broader research plan, so this article does not name the benchmarks or make benchmark-specific claims. A later results report should attach verified primary-source links, dates, and source identifiers on the relevant lines.

Inspect the scoring steps

Once the source material is pinned, follow the scoring path in order. The proposed sequence is:

1. List the inputs. Record every input used by the score, including text, labels, prompts, reference answers, annotations, or other permitted material.
2. Trace transformations. Note cleaning, parsing, normalization, token handling, filtering, or other transformations before scoring.
3. Record exclusions. Identify removed inputs, missing values, rejected outputs, failed runs, and the rule applied to each case.
4. Inspect aggregation. Record how item-level values become a reported score, including weighting, averaging, minimums, maximums, and rounding.
5. Define expected behaviour before testing. If a controlled fixture is proposed, state what the scoring process should do before executing it.
6. Recalculate only when permitted inputs and the required environment are available. Do not infer an output from incomplete code, an incomplete fixture set, or a description of the method.

This sequence is designed to separate what the published material says from what a future execution would show. A documented rule is not the same as an observed test output. A proposed acceptance condition is not a passed condition.

Label synthetic fixtures clearly

A synthetic fixture is a deliberately created test input. It is not, by that label alone, a customer example, a production document, or an observed writing case.

If a future review creates or uses a synthetic fixture, label it wherever it appears in the audit record. Record who created it, why it was created, which scoring behaviour it is intended to inspect, and whether it was supplied by the benchmark or introduced by the auditor.

Synthetic fixtures may help isolate a defined scoring behaviour. They do not establish production writing quality. They also do not show how a model performs across a broader population of writing tasks. Any later report should keep the fixture’s design and scope next to the observation.

No fixtures are executed or reported in this article.

A recording template for a future run

Use the table below as a blank record. The acceptance conditions are proposed checks, not passed checks. The observation cells remain uncollected because this protocol has not been executed.

Proposed protocol dated 2026-09-11; not executed; no results.

Keep the blank cells blank. Do not convert an unavailable source, fixture, or output into a pass, fail, score, or assumption.

What this protocol cannot establish

An unrun procedure establishes no benchmark correctness, model ranking, accuracy, production writing quality, or TrustGrowth or customer outcome.

It cannot establish that one model is better overall. It cannot establish that a benchmark predicts the quality of writing used by a particular team. It cannot establish general benchmark validity beyond the sources, versions, fixtures, and procedures actually checked in a future run. It cannot establish a result for a source that was not inspected or a calculation that was not reproduced.

A source-and-scoring check also should not become a ranking by implication. Differences in documentation, fixture design, or implementation may be worth recording, but they are not automatically evidence that one benchmark or model performs better. A comparative claim needs a defined comparison, comparable criteria, consistent versions, and reported results.

Version scope is another boundary. A future observation would apply to the pinned material and execution conditions recorded for that run. It would not automatically apply to later source revisions, model versions, fixture sets, or scoring-code changes.

Synthetic-fixture scope is a separate boundary. A controlled test can inspect a defined scoring behaviour, but it cannot by itself establish how a benchmark behaves on production writing or what a customer will experience.

These limits do not require dismissing an existing benchmark. They define what a careful source-level review can and cannot say. A later report should preserve the benchmark’s stated purpose, describe the inspected evidence accurately, and place its limitations beside its observations.

How a later results report would differ

A results report would be a separate publication. It would require an actual execution date, pinned primary materials, recorded inputs and outputs, disclosed limitations, and an independently reviewed evidence trail.

It should identify the benchmark sources and versions, show which checks were run, distinguish source observations from recalculated outputs, and link any third-party measured number to its original source on the same line with explicit third-party attribution.

It should also state what was not available. A missing code file, unavailable fixture, undocumented exclusion, or unrecorded model version would remain a limitation, not a silent assumption. No run or re-run date is committed in this protocol.

Apply the checklist

Before using the next writing-benchmark score in an editorial decision, work through these checks:

• Record the exact source URL, document, repository state, version, and retrieval date.
• Identify the rubric and map each score component to its permitted inputs.
• Trace transformations from raw input to scored input.
• Record exclusions, missing values, failed runs, and their treatment.
• Inspect aggregation, weighting, averaging, and rounding.
• Label every fixture as synthetic, production-derived, or otherwise defined.
• Record the model, version, configuration, and reproduction environment where disclosed.
• Define the expected behaviour before running a proposed fixture.
• Recalculate only when the required inputs, code, permissions, and environment are available.
• Keep observations separate from interpretations and broader conclusions.
• State the version, fixture, and source limits beside any future result.
• Do not turn a source check into a leaderboard or product claim.

This checklist is a recording aid for a future review. It is not evidence that any benchmark has passed these checks.

Use the recording template when inspecting your next writing-benchmark score.