Pre-registered · v1.0 · published before the run

Ungoverned chat vs. governed architecture on enterprise finance tasks

Two arms, one model. Arm A is ungoverned chat: the model does the arithmetic, the analyst pastes whatever fits, and owner questions go out by email. Arm B is the governed Artificial Financial Intelligence layer: arithmetic runs as deterministic code, data is queried, owner input is initiated and approval-gated, and every figure carries a source record.

This page publishes the protocol before either arm runs — the tasks, the scoring, the fairness rules, and the empty result tables. The dataset and its generator are published with it, so anyone can reproduce the inputs and re-run both arms.

The two arms

DimensionArm A — ungoverned chatArm B — governed AFI layer
ModelClaudeClaude
Arithmeticdone by the modeldone by deterministic code
Data accessanalyst pastes what fits in contextqueried from the data layer
Owner inputanalyst chases by emailinitiated in Teams, approval-gated, logged
Traceabilitywhatever the model citesevery figure carries a source record

Octopus AI runs on Claude, and Anthropic is a partner. The comparison is not against a model or a vendor; it is against ungoverned chat used for core finance work.

The dataset

NorthPeak Industrial is a fictional industrial group generated deterministically from seed 20260907: 6,529 GL rows spanning October 2025 to December 2026, 2,880 budget rows, monthly FX rates, and eight cost centers with named owners across three entities — NP-US (USD), NP-UK (GBP) and NP-IL (ILS). The fiscal year ends 30 September, so fiscal Q1 FY2027 is October to December 2026.

Five items are planted deliberately: a misbooked conference sponsorship, a pulled-forward campaign that only the owner can explain, a prior-year reclassification that exists nowhere in the current period, monthly FX rates that punish a single average rate, and a fiscal calendar that differs from the calendar year. Three of the five cannot be resolved from the data at any level of model capability — they can only be resolved by asking the person who knows.

The scoring key is sealed. It is held by the scorer, never exposed to either arm, and published with the results.

The three tasks

T1 — Consolidation

What is the consolidated operating expense for fiscal Q1 FY2027 across all three entities, in USD? Tests whether monthly FX rates are read rather than averaged, and whether the fiscal calendar is respected.

T2 — Variance explanation

Explain the top three material variances for fiscal Q1 FY2027 against budget. Scored on whether the true cause is named, not on whether the number is restated in words.

T3 — Reproducibility

T1 asked three times in three fresh sessions with no memory carried between them. All three answers recorded verbatim.

The metrics

IDMetricClaim under test
M1Numeric error ratethe model never does the math
M2Time to a defensible pack (wall clock and analyst minutes)capacity returned to the team
M3Traceability rateaudit-grade lineage
M4Owner response ratethe initiating loop
M5Reproducibility across three runsdeterminism

M2 reports two clocks on purpose. Ungoverned chat usually wins wall clock to a first answer, and we expect it to win here. What it loses is analyst minutes to a defensible answer — the chasing, the re-pasting, the reconciling and the writing-up a reviewer will accept. A single blended number would hide that, so we do not report one.

Fairness rules

  • Pre-register. This protocol is published before either arm runs.
  • Arm A gets a genuine best effort: a competent analyst, a well-written prompt, unlimited attempts within the time budget, and the exact prompt published with the results.
  • The dataset and generator are published, so anyone can re-run both arms.
  • Where ungoverned chat wins — setup time, cost to a first answer — it is reported as a win.
  • Same analyst, same day, both arms.
  • The scoring key never enters a prompt, a pasted excerpt, or the analyst's workspace.

Get the protocol and the dataset

The full protocol PDF and the complete NorthPeak dataset with its generator. Work email only — we send the material you asked for and the results when the run is scored.

We use your details to send the material you requested and occasional CFO governance updates. No marketing lists, and you can unsubscribe at any time.

The archive contains gl_transactions.csv, budget.csv, fx_rates.csv, cost_centers.csv, a README and the generator script. The scoring key is not included; it is published with the results.

Questions and answers

Why does the comparison arm also run Claude?

Because the variable under test is architecture, not model capability. Both arms run the same model. The difference is whether arithmetic is executed as deterministic code, whether data is queried or pasted, whether owner input is an initiated and logged loop, and whether every figure carries a source record.

Is the dataset real customer data?

No. It is synthetic data for a fictional group, NorthPeak Industrial, generated deterministically from seed 20260907 so that regeneration is byte-identical and published results are checkable. It contains deliberately planted items.

What does the benchmark not claim?

It does not claim one model is better than another — both arms run the same model. It does not claim ungoverned chat is not useful. And it does not claim these numbers reproduce on any specific customer's ledger, which has messier data and no planted items.

Have the results been published?

Yes. The run was completed on 20 September 2026 and the scored results, raw answers and deviations are published at /benchmark. This page remains the pre-registration it was before the run: the protocol, tasks, metrics, fairness rules and empty result tables were published first, so the reporting could not be reshaped around a favourable result.

Can the protocol be challenged before the run?

Yes, and that is the point of pre-registering it. If a task, a tolerance, or the Arm A setup looks unfair, tell us before the run and the protocol is amended in public and version-numbered.

Caveat

This is synthetic data with deliberately planted items. It demonstrates architectural behaviour under known conditions. It is not a claim about performance on any specific customer's books, and it is not a claim about any model's capability.