Skip to content
Banja Labs
HomeWorkWhat we doThe lab
Let's talk

NSW cafe quarter - scripted BAS, GST, state holiday, and a refuse turn

One task across 30 published runs. Inspect the scores and outputs together, with each run's execution method shown.

All experimentsModel resultsDesigner buildsCompare models
Task
BASQU-0001
Domain
Australian accounting
Difficulty
hard

Leading published run: claude-opus-4-8 (low reasoning) at 100.0% composite. Lowest: grok-4.3 at 0.0%. 30 published runs shown on this task.

How this task was evaluated

Results are ordered by this task's published score. API runs use one prompt and one completion; agent runs use a coding harness with tools and iteration. Inspect the execution method when comparing results.

Objective scores use the benchmark's execution checks. Judge scores use a vision panel's fixed rubric; the published score leaves out a model's own family. Missing values are labelled rather than treated as zero.

Read the brief

This is a benchmarking hypothetical, not tax advice. Figures and public-holiday dates are as at FY2025-26 (Australia). You are acting as the bookkeeper for a cafe registered in New South Wales (NSW). This is a SCRIPTED MULTI-TURN run for one quarter. You will be asked a sequence of questions, one per turn. Answer each turn on its own, and keep your earlier answers consistent across the whole run. Quarter figures (all sales and purchases are GST-taxable at the standard 10% rate, so each GST-inclusive amount contains one eleventh of GST): - Total sales including GST: $93,500 - Total purchases including GST: $44,000 - Gross wages paid to employees: $30,000 - PAYG withholding rate applied to those wages: 19% There are no PAYG instalments, fuel tax credits, or other amounts this quarter. Reminder: GST on a GST-inclusive amount is that amount divided by 11, never 10% added on top. Use the public-holiday date for the business's own state, not another state's.

Anthropicclaude-opus-4-8
Low reasoning
Agent harness2026-07-21
Composite 100.0%Objective 100.0%
Open outputFull run
Anthropicclaude-opus-4-8
Medium reasoning
Agent harness2026-07-21
Composite 100.0%Objective 100.0%
Open outputFull run
Anthropicclaude-opus-4-8
High reasoning
Agent harness2026-07-21
Composite 100.0%Objective 100.0%
Open outputFull run
Anthropicclaude-opus-4-8
Extra-high reasoning
Agent harness2026-07-21
Composite 100.0%Objective 100.0%
Open outputFull run
Anthropicclaude-opus-4-8
Max reasoning
Agent harness2026-07-21
Composite 100.0%Objective 100.0%
Open outputFull run
Anthropicclaude-sonnet-4-6
High reasoning
Agent harness2026-07-21
Composite 100.0%Objective 100.0%
Open outputFull run
Anthropicclaude-sonnet-5
High reasoning
Agent harness2026-07-21
Composite 100.0%Objective 100.0%
Open outputFull run
Anthropicclaude-fable-5
High reasoning
Agent harness2026-07-09
Composite 100.0%Objective 100.0%
Open outputFull run
Zhipuglm-5.2
default reasoning
API single-shot2026-07-17
Composite 100.0%Objective 100.0%
Open outputFull run
Moonshotkimi-k3
default reasoning
API single-shot2026-07-17
Composite 100.0%Objective 100.0%
Open outputFull run
Googlegemini-3.1-pro-preview
High reasoning
API single-shot2026-06-23
Composite 100.0%Objective 100.0%
Open outputFull run
Googlegemini-3.5-flash
default reasoning
API single-shot2026-06-23
Composite 100.0%Objective 100.0%
Open outputFull run
Googlegemini-3.1-flash-lite
default reasoning
API single-shot2026-06-23
Composite 100.0%Objective 100.0%
Open outputFull run
xAIgrok-4.5
High reasoning
API single-shot2026-07-09
Composite 100.0%Objective 100.0%
Open outputFull run
xAIgrok-4.5
Medium reasoning
API single-shot2026-07-09
Composite 100.0%Objective 100.0%
Open outputFull run
xAIgrok-4.5
Low reasoning
API single-shot2026-07-09
Composite 100.0%Objective 100.0%
Open outputFull run
xAIgrok-4.20-reasoning
default reasoning
API single-shot2026-06-25
Composite 100.0%Objective 100.0%
Open outputFull run
xAIgrok-build-0.1
default reasoning
API single-shot2026-06-25
Composite 100.0%Objective 100.0%
Open outputFull run
xAIgrok-composer-2.5-fast
default reasoning
Agent harness2026-06-25
Composite 100.0%Objective 100.0%
Open outputFull run
DeepSeekdeepseek-v4-pro
default reasoning
API single-shot2026-06-25
Composite 100.0%Objective 100.0%
Open outputFull run
DeepSeekdeepseek-v4-flash
default reasoning
API single-shot2026-06-25
Composite 100.0%Objective 100.0%
Open outputFull run
Anthropicclaude-haiku-4-5
High reasoning
Agent harness2026-07-09
Composite 0.0%Objective 0.0%
Open outputFull run
Moonshotkimi-k2.7-code
default reasoning
API single-shot2026-07-17
Composite 0.0%Objective 0.0%
Open outputFull run
OpenAIgpt-5.5
High reasoning
API single-shot2026-07-17
Composite 0.0%Objective 0.0%
Open outputFull run
OpenAIgpt-5.4-mini
High reasoning
API single-shot2026-06-23
Composite 0.0%Objective 0.0%
Open outputFull run
OpenAIgpt-5.6-sol
High reasoning
API single-shot2026-07-09
Composite 0.0%Objective 0.0%
Open outputFull run
OpenAIgpt-5.6-sol
Max reasoning
API single-shot2026-07-09
Composite 0.0%Objective 0.0%
Open outputFull run
OpenAIgpt-5.6-terra
High reasoning
API single-shot2026-07-09
Composite 0.0%Objective 0.0%
Open outputFull run
OpenAIgpt-5.6-luna
High reasoning
API single-shot2026-07-09
Composite 0.0%Objective 0.0%
Open outputFull run
xAIgrok-4.3
default reasoning
API single-shot2026-07-03
Composite 0.0%Objective 0.0%
Open outputFull run

Product thinking,
design and engineering

An independent product lab, building in Australia.

hello@banja.au
HomeWorkWhat we doThe labStart a conversation
About the labProductsExperimentsJournalWebsites & brand? Norex
© 2026 Banja LabsBuilt together in Australia
PrivacyTermsLinkedIn ↗