<!-- Canonical: https://myoctopus.ai/benchmark -->

Results — run 20 September 2026

# Enterprise finance AI benchmark: ungoverned chat vs. governed architecture

Same model in both arms. The variable under test is the architecture around it. The tasks, tolerances and scoring key were pre-registered and published before either arm ran; the dataset and generator are public, so anyone can re-run this.

0 of 3

sessions where ungoverned chat produced the correct consolidated figure

3 of 3

sessions where the governed architecture produced it exactly

$3,734,927.48

size of the ungoverned arm's error on a single quarter

## The measured results

Model: google/gemini-2.5-flash, temperature 0, three fresh sessions per arm, no session discarded. Dataset: NorthPeak Industrial, seed 20260907. Fiscal year ends 30 September, so fiscal Q1 FY2027 is October to December 2026.

| Metric | Scoring key | Arm A — ungoverned chat | Arm B — governed architecture |
| --- | --- | --- | --- |
| Consolidated opex, fiscal Q1 FY2027 (T1) | $13,734,927.48 | $10,000,000.00 (all three sessions) | $13,734,927.48 (all three sessions) |
| Error against the key | — | −$3,734,927.48 per session | $0.00 per session |
| Correct sessions (T1) | 3 of 3 required | 0 of 3 | 3 of 3 |
| Identical answer across three fresh sessions (T3) | required | Yes — identically wrong | Yes |
| Largest budget variance identified (T2 rank 1: 6400 Marketing / 7040, +$362,830.00) | 3 of 3 required | 0 of 3 | 3 of 3 |
| Ranks 2 and 3 identified (+$241,560.01 / −$186,920.00) | 3 of 3 required | 0 of 3 | 3 of 3 |
| Median model wall clock per session | — | 65.8 seconds | 7.9 seconds |

### Arm A — ungoverned chat

The model received the four raw CSV files and answered directly. It could ask owner questions, and received the full owner statement set as soon as it did.

### Arm B — governed architecture

Arithmetic ran as deterministic code over the same files — each GL row converted at its own period's published rate — and the model narrated the computed result against the owner records. It was instructed never to recompute or round.

## How the ungoverned arm failed

The failures are worth more than the scoreline, because each one would have survived a quick review. The answers were well formatted, cited source files, and looked finished.

### A round number where a computed one belonged

All three ungoverned sessions reported consolidated operating expense as exactly $10,000,000.00 — a figure that appears nowhere in the dataset. It was presented with a supporting table and source-file citations.

### Budget rows read as actuals

The ungoverned arm's own workings mixed budget.csv amounts into the actuals column, and in places cited budget.csv as the source of an actual figure. The citation looked valid; the number was not.

### FX applied in the wrong direction

fx\_rates.csv publishes local currency per 1 USD. The ungoverned arm multiplied by inverted rates of its own construction rather than the published ones.

### A variance that does not exist

Its largest reported variance was −$4,403,812.50 against a zero budget — an artefact of its own arithmetic. The three real discriminating variances were never surfaced.

### Reproducibly wrong is not the same as reliable

The ungoverned arm was perfectly consistent: the same incorrect figure three times, in three fresh sessions. Consistency is not evidence of correctness.

## What this does not show

Every deviation from the pre-registered protocol, and every limit on what the numbers support:

*   The operator was a script, not a human analyst. Wall-clock figures are model latency, not analyst minutes, so the protocol's time-and-effort table is reported as latency only and the analyst-minutes columns stay empty.
*   Both arms ran google/gemini-2.5-flash at temperature 0. The protocol says a frontier model; this is the model we ran, and the result should be read as applying to it.
*   Owner input was supplied to both arms as a fixed, written statement set rather than a live email or Teams thread. Arm A received the full set as soon as it asked any owner question, which is more generous than a real email chase.
*   Three sessions per arm, as pre-registered. No session was discarded, re-prompted or re-run.
*   Scoring was mechanical (exact string and figure matching against the key), which removes scorer judgement but also gives no partial credit.
*   The dataset is synthetic with deliberately planted items. This measures architectural behaviour under known conditions; it is not a claim about performance on any specific company's books.

This is not a claim that one model is better than another — both arms ran the same model, by design. It is a claim about where the arithmetic and the approval live.

## Check it yourself

The scoring key was held back until the run finished and is published with the dataset. Regenerating the dataset produces byte-identical files, so every figure above is re-checkable.

[Raw run output (prompts, answers, timings)](/downloads/prove-benchmark/RESULTS.json)[Scored summary](/downloads/prove-benchmark/RESULTS_SUMMARY.json)[Dataset and generator](/downloads/octopus-ai-prove-benchmark-dataset.zip)[Pre-registered protocol (published before the run)](/downloads/octopus-ai-benchmark-protocol.pdf)

## Questions

### What does this enterprise finance AI benchmark measure?

Whether the architecture around a language model changes the answer it gives on real finance work. Both arms run the same model on the same published dataset. Arm A answers from the raw files in chat. Arm B computes every figure in deterministic code and lets the model narrate the result and the owner context.

### Who won, and by how much?

On the consolidation task the governed architecture returned the correct figure, $13,734,927.48, in all three sessions. The ungoverned chat arm returned $10,000,000.00 in all three sessions — an error of $3,734,927.48 on one quarter. On the variance task the governed arm reported all three discriminating variances; the ungoverned arm reported none of them.

### What is governed financial intelligence?

An architecture in which arithmetic runs as deterministic code rather than model output, every outbound action is approval-gated by a named human, and every published figure traces back to its source rows. The model contributes language and context, not calculation.

### Can I reproduce these results?

Yes. The dataset generator is deterministic (seed 20260907) and regenerates byte-identical files, the dataset and the scoring key are published, and the prompts, raw answers and scoring output are downloadable from this page.

### Is the comparison fair to general-purpose AI chat?

The comparison arm received the complete dataset, a best-effort prompt published verbatim in the results file, and the full owner statement set on request. Its prompt is published so anyone can improve on it and re-run.

[Run this on your own numbers](/proof-of-concept) [Read the pre-registered protocol](/benchmark-protocol)[Trust hub](/trust)
