Locul / Research / Office AI Benchmark
Office AI Benchmark 2026: Do you need Opus to write an email?
For this office AI benchmark we gave 13 models the same 40 office tasks a real assistant gets: tricky email replies in English and German, document summaries, Excel formulas checked on real spreadsheets, and file sorting with answer keys. Three runs each, blind judging against Claude Opus 5.5.
The short answer
In the Office AI Benchmark 2026, Claude Opus 5.5 was the best model for hard work emails, but not by much: Sonnet 5.5 matched or beat it on 58% of emails and Qwen3.5-397B, the best open model, on 31%, against 77% for Opus judged against itself. For documents, file sorting and most spreadsheet work, open models come much closer for a fraction of the cost: €24.12 per 1,000 emails with Opus against €1.07 with Qwen3.5-397B.
- Email is where the expensive models earn their price.Measured against Opus 5.5 in blind pairwise judging, Sonnet 5.5 matched or beat it on 58% of emails and GPT-5.4 on 44%. The best open model, Qwen3.5-397B, managed 31%. Opus against its own second run scores 77%, so that is the ceiling. Cheap models miss the buried ask, the trap in the thread or the tone.
- For summaries and documents, big open models are already on par.Qwen3.5-397B beat Opus in 61% of document comparisons and Qwen3-235B in 54%, while keeping 94% and 91% of the key facts (Opus: 95%). Cost per document: about 1/20th (Qwen3.5-397B) and 1/130th (Qwen3-235B) of Opus.
- File sorting and labelling is a solved problem.Every model scored 94% to 100% against the answer keys on invoice extraction, receipt categorising and inbox triage. Mistral Medium, an EU provider, scored 99%.
- Excel formulas still split the field.Formulas were run on the real test sheets in LibreOffice. Opus scored 100%, Sonnet 99%, GPT-5.4 96%. gpt-oss-120b reached about 80%, Qwen3-32B 41%. The typical failure is a broken sheet reference that returns an error.
- Laptop-size models are not ready for tricky email replies.The best laptop-size model, Qwen3 8B, matched or beat Opus 5.5 on 10% of emails. Gemma 3 4B, Llama 3.2 3B and Qwen3.5 9B managed 2%. They miss the trap, add options and preambles, or get facts wrong. They are fine for structured chores: Qwen3.5 9B scored 90% on file sorting against the answer keys.
Which AI writes work emails as well as Opus?
Only Sonnet 5.5 comes close to Opus 5.5 on work emails: it matched or beat Opus on 58% of replies, GPT-5.4 on 44% and the best open model on 31%. Each model answered the same 16 email tasks 3 times. A blind judge compared every answer with Opus 5.5 in both orders. Match or beat is the share of those comparisons the model won or tied. Opus judged against its own second run lands at 77%, so that is as high as any model can realistically go.
Match or beat Opus 5.5 on email
Solid: judged better than Opus. Light: a tie. Hover a bar for the detail.
What does a good AI email cost?
Cost is what the provider billed for our runs, converted to euros. Opus 5.5 costs €24.12 per 1,000 email replies. The cheapest open models cost well under a euro for the same 1,000 replies, but they also fall furthest below the dashed line.
Email match-or-beat against cost per 1,000 emails
Up is better, left is cheaper. Cost axis is logarithmic: each step is ten times the price.
Where do cheaper models catch up?
Cheaper models catch up on documents, file sorting and labelling: Qwen3.5-397B beat Opus in 61% of document comparisons and every model scored 94% to 100% on file tasks. Emails and documents are judged against Opus 5.5. Excel formulas were run on the real test workbooks in LibreOffice and file tasks were checked field by field against answer keys, so those two columns are plain scores out of 100.
Score by task category
16 email, 8 document, 8 Excel and 8 file tasks. Darker is better.
| Model | Emailmatch or beat | Docsmatch or beat | Excelanswer key | Filesanswer key |
|---|---|---|---|---|
| Claude Opus 5.5US | 77% | 92% | 100% | 100% |
| Claude Sonnet 5.5US | 58% | 79% | 99% | 100% |
| GPT-5.4US | 44% | 83% | 96% | 100% |
| Qwen3.5-397BUS | 31% | 75% | 74% | 97% |
| Qwen3-235B (2507)US | 25% | 67% | 60% | 98% |
| gpt-oss-120b (Groq)Fast open models (US) | 23% | 42% | 78% | 94% |
| Mistral Medium 3.5EU (Mistral) | 21% | 62% | 67% | 99% |
| gpt-oss-120b (Nebius)EU company, region not pinned | 15% | 21% | 82% | 97% |
| Qwen3 8B (VL Instruct)US | 10% | n/a | n/a | n/a |
| Qwen3-32BEU company, region not pinned | 8% | 21% | 41% | 96% |
| Gemma 3 4BOn your device | 2% | n/a | n/a | n/a |
| Llama 3.2 3BUS | 2% | n/a | n/a | n/a |
| Qwen3.5 9BUS | 2% | 4% | 23% | 90% |
* Opus 5.5 row: email and document columns compare Opus with its own second run. n/a: not measured for that model.
Can a laptop-size model handle email replies?
Not the hard ones. On the same 16 tricky emails, Opus 5.5 beat the small models in 95 of 96 blind comparisons for Gemma 3 4B, Llama 3.2 3B and Qwen3.5 9B. Qwen3 8B did best at 10% match or beat. Gemma 3 4B ran on an M1 Pro MacBook Pro (16 GB) with Ollama at about 22 tokens per second, 14 seconds for a typical reply, nothing leaving the machine. Llama 3.2 3B, Qwen3 8B (the VL Instruct build) and Qwen3.5 9B were served via OpenRouter instead of local hardware: the open weights are the same, so quality carries over, but their speed and cost are cloud numbers. Qwen3 4B had no hosted endpoint we could reach, so Qwen3.5 9B took its place. Use small models for drafts you will rewrite, sorting and extraction, not for replies that have to be right.
Laptop models against the cloud on email replies
Opus 5.5 against itself, Sonnet 5.5 and the best open cloud model for reference.
The trap most models walked into
Morning! Niamh here, practice manager at Salthill Dental Care (Galway). Can you draft a quick email from me to Dr Hughes and Dr Fahy (the two partners)? Atlantic Dental Supplies are putting prices up but the good news is we have until end of December to order at the old prices, so I want to tell them no rush, we'll do the big consumables order (gloves, composite, impression material, about €6,800 at current prices) in the first week of December as usual, after the Christmas rota is sorted. Keep it short and upbeat, they hate long emails.
Dear Niamh, Following on from my message last month, please find attached our revised price list. Due to increases from our manufacturers, prices across most consumables lines will rise by 9 to 12% for orders placed on or after 1 November 2026. As a valued account, any orders placed before 1 November will be honoured at current pricing, with delivery available up to 31 December 2026 if you'd prefer to stagger stock into the new year. If you'd like to discuss volume, please give me a call.
What happened
Most models missed a deadline buried in the thread and wrote an email that would have cost the client money. Task email-15: a practice manager asks for a 'no rush, order in December' email, but the supplier's price rise applies to orders placed on or after 1 November. Opus 5.5 and Sonnet 5.5 flagged the misread before drafting in all 3 runs. Every other model wrote the wrong email in every run. Qwen3-235B even quoted the correct deadline and still told the partners there was no rush. Waiting would cost the practice about EUR 610 to 820.
How long does each model take to answer?
Measured from our machine in Europe until the whole answer arrived, across every task. The quickest was gpt-oss-120b (Groq) at 2.1 s for a typical answer.
Seconds to a complete answer
Filled dot: typical answer (p50). Ring: slow answers (p95).
Full results
Select a column to sort. Email and Docs: match or beat Opus 5.5. Facts kept: share of the 10 key source facts each document answer preserved. Excel and Files: answer-key scores. Cost averages all 40 tasks. Region is where the provider that served our runs processes data.
| Claude Opus 5.5 * | Anthropic | US | 77% | 92% | 95% | 100% | 100% | €0.032 | €24.12 | 13 s | 41 s |
| Claude Sonnet 5.5 | Anthropic | US | 58% | 79% | 99% | 99% | 100% | €0.019 | €13.02 | 11 s | 40 s |
| GPT-5.4 | OpenAI | US | 44% | 83% | 95% | 96% | 100% | €0.0099 | €5.00 | 7.8 s | 24 s |
| Qwen3.5-397B | DeepInfraServed by DeepInfra (US) as a stand-in for OVHcloud (France). | US | 31% | 75% | 94% | 74% | 97% | €0.0016 | €1.07 | 6.7 s | 23 s |
| Qwen3-235B (2507) | DeepInfraServed by DeepInfra (US) as a stand-in for Scaleway (Paris). | US | 25% | 67% | 91% | 60% | 98% | €0.0003 | €0.18 | 25 s | 82 s |
| gpt-oss-120b (Groq) | Groq | Fast open models (US) | 23% | 42% | 89% | 78% | 94% | €0.0005 | €0.24 | 2.1 s | 5.4 s |
| Mistral Medium 3.5 | Mistral | EU (Mistral) | 21% | 62% | 89% | 67% | 99% | €0.0039 | €2.15 | 2.8 s | 7.8 s |
| gpt-oss-120b (Nebius) | NebiusServed by Nebius as a stand-in for Scaleway (Paris). | EU company, region not pinned | 15% | 21% | 89% | 82% | 97% | €0.0005 | €0.25 | 2.7 s | 7.9 s |
| Qwen3 8B (VL Instruct) | ParasailLaptop-size open weights, served in the cloud by Parasail via OpenRouter, not run on device. | US | 10% | n/a | n/a | n/a | n/a | €0.0003 | €0.25 | 4.3 s | 11 s |
| Qwen3-32B | DeepInfraServed by DeepInfra (US) as a stand-in for Groq or Nebius. | EU company, region not pinned | 8% | 21% | 84% | 41% | 96% | €0.0002 | €0.096 | 12 s | 43 s |
| Gemma 3 4B | Ollama on a MacBookOllama on an M1 Pro MacBook, 16 GB | On your device | 2% | n/a | n/a | n/a | n/a | €0 | €0 | 14 s | 36 s |
| Llama 3.2 3B | ParasailLaptop-size open weights, served in the cloud by Parasail via OpenRouter, not run on device. | US | 2% | n/a | n/a | n/a | n/a | €0.0001 | €0.084 | 2.3 s | 4.4 s |
| Qwen3.5 9B | DeepInfraServed by DeepInfra (US) as a stand-in for Ollama on a MacBook. | US | 2% | 4% | 82% | 23% | 90% | €0.0002 | €0.090 | 9.6 s | 40 s |
Scroll the table sideways for cost and speed.
* Opus 5.5 is compared with its own second run. Hosting stand-ins are explained in the method below. Same open weights should score the same on an EU host such as Scaleway or OVHcloud; speed and price will differ.
How we tested
- 40 tasks: 16 emails (11 English, 5 German or Swiss/Austrian German), 8 document tasks, 8 Excel tasks and 8 file sorting or labelling tasks. Each task is a realistic, messy request with the pasted thread, notes or sheet, written to a realism bar and checked by a Jev realism gate. All names, companies and data are synthetic.
- Every model ran every task 3 times with default settings. Reasoning models ran at low reasoning effort. Qwen3 thinking was switched off.
- Email and document answers were judged blind against Opus 5.5 by Jev, TypeSafe's judge model, in both orders (A/B and B/A) to cancel position bias. A model counts as a win or loss only when both orders agree; otherwise it is a tie. Opus was also judged against its own second run to show the noise floor.
- We calibrated the judge on 20 pairs labelled by hand. Jev agreed with the human label on 70% exactly and reversed it (picked the other side) on only 1 of 20.
- Document answers were also checked fact by fact: each task lists 10 source facts and Jev marks each as preserved, missing or contradicted.
- Excel formulas were written into the real test workbook, filled down, recalculated in LibreOffice and compared cell by cell with the answer key. File tasks return JSON that is compared field by field (accuracy and macro-F1 for labels).
- Cost is what the provider billed for the run (via OpenRouter), converted at the ECB reference rate of 1 EUR = 1.1193 USD on 23 Oct 2026. Latency is wall-clock time from our machine in Europe.
- Hosting substitutions: we had no direct keys for Scaleway, OVHcloud, Groq or Nebius, so Qwen3-235B, Qwen3.5-397B and Qwen3-32B ran on DeepInfra and gpt-oss-120b on Groq and Nebius, all through OpenRouter. Output quality depends on the open weights, not the host, so the same models on an EU host such as Scaleway or OVHcloud should score the same; speed and price will differ.
- Laptop-size models: Gemma 3 4B ran locally with Ollama on an M1 Pro MacBook Pro with 16 GB of memory (Q4_K_M). Llama 3.2 3B (Parasail), Qwen3 8B VL Instruct (Parasail) and Qwen3.5 9B (DeepInfra, Parasail or Together) were served via OpenRouter, not on local hardware, because local installs were not workable on the test machine. Same open weights, so quality scores carry over to a laptop; their speed and cost are cloud figures. Qwen3 4B had no hosted endpoint and was replaced by Qwen3.5 9B. Laptop-size models ran the 16 email tasks; Qwen3.5 9B also ran every other task.
Questions people ask
Do I need Claude Opus to write work emails?
For hard emails (a buried ask, a trap in the thread, a tense relationship) Opus 5.5 was clearly best in our test; Sonnet 5.5 matched or beat it on 58% of emails. For routine replies, summaries and sorting, much cheaper models do the job.
What is the cheapest model that is good enough for office documents?
Qwen3-235B and Qwen3.5-397B matched or beat Opus 5.5 on most document tasks at a small fraction of the cost per task.
Can a local model on a laptop write email replies?
Not for tricky replies. Laptop-size models (Gemma 3 4B, Llama 3.2 3B, Qwen3 8B, Qwen3.5 9B) matched or beat Opus 5.5 on only 2% to 10% of our hard email tasks. They are useful for first drafts, sorting and extraction on the device, where nothing leaves the machine.
How much does it cost to write 1,000 emails with AI?
About EUR 24 with Opus 5.5, EUR 13 with Sonnet 5.5, EUR 5 with GPT-5.4, EUR 2 with Mistral Medium and under EUR 0.25 with Qwen3-235B or gpt-oss-120b. A model run on your own laptop costs nothing beyond electricity.
Which AI model is best at Excel formulas?
Opus 5.5 (100%), Sonnet 5.5 (99%) and GPT-5.4 (96%) on formulas recalculated on real test sheets. Open models ranged from 41% to about 80%.
Can I use the task set?
Yes. The 40 tasks, answer keys, test spreadsheets and every graded output are published under CC BY 4.0. Download the CSV below.
Download, rerun or cite the benchmark
The tasks, test spreadsheets and every graded output are free to reuse under CC BY 4.0. Link back to this page so readers find the latest run.
Cite it
Locul (2026). Office AI Benchmark 2026. https://locul.ai/research/office-ai-benchmark/