TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Surge AI has released DAYJOB: Tax, a benchmark of 50 professional tax assignments across eight practice areas. The company reports a 10.8% pass rate for the leading model, Claude Sonnet 5.5, using a threshold requiring at least 95% of each task’s grading criteria. The results and examples show performance gaps in consequential issue-spotting, but the findings come from Surge AI’s own benchmark and do not establish how agents perform across all tax work.
Surge AI has introduced DAYJOB: Tax, a benchmark that tests AI agents on 50 professional tax assignments involving returns, workpapers, reviews and client advice. The company reports that the top-scoring model, Claude Sonnet 5.5, passed 10.8% of attempts under a strict rubric threshold, highlighting how often agents fell short of near-complete performance on the test.
The benchmark spans eight areas of tax practice, including individual and family tax, small-business tax, partnerships, property and investments, estates and trusts, international tax, corporate transactions, and tax review and disputes. Agents receive client requests and supporting records and must produce a deliverable such as a tax return, technical review or advisory response. Surge says the tasks test calculations and tax knowledge as well as evidence gathering, judgment and handling missing or conflicting information.
Surge’s leaderboard reports 10.8% for Claude Sonnet 5.5, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. The company measures pass@1 against a 95% rubric threshold: an attempt counts as a pass only if it meets at least 95% of that assignment’s criteria, covering such elements as calculations, evidence, consequential issues and the completed work product. These are results on this benchmark, not a general estimate of model accuracy on tax work.
The assignments draw on 1,789 workspace files, with a median of 30 files per task and a maximum of 123. Surge lists 1,487 PDFs, 117 Excel workbooks and 38 Word documents, alongside emails, images, spreadsheets and other records. Examples include handwritten receipts that must be reconciled with inventory records, dispatch logs that may reveal work across state lines, and cross-border household records involving employment, property and cryptocurrency.
Tax Advice Depends on Issue-Spotting
The benchmark focuses on a risk that is different from a simple arithmetic error: an agent may answer the question asked while missing another fact that could materially affect the client’s decision. In one example provided by Surge, a homeowner considering an October 2026 or January 2027 sale would meet the residence test for a possible exclusion of up to $250,000 in gains on either date. The agent evaluated those options but did not flag a later eligibility deadline tied to when the homeowner moved out.
Surge reports that, across five attempts, Astra never identified that deadline, while Opus identified it in all five. The example illustrates the benchmark’s emphasis on professional judgment, but it is one task and a result reported by the benchmark publisher. More broadly, the low pass rates under the stated threshold suggest that, on these assignments, strong performance on individual parts does not often translate into an answer that satisfies nearly all grading criteria. For tax firms and clients, the distinction matters because advice can be incomplete even when the calculation or direct response is correct.
Products Worth Considering
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
Amazon Kindle Paperwhite Signature Edition (12th Generation, 2024), 32 GB, Metallic Black
Logitech MX Brio Ultra HD 4K Webcam, Graphite
As an affiliate, we earn on qualifying purchases.
How the Tax Benchmark Is Built
DAYJOB: Tax extends Surge AI’s DAYJOB benchmark series, which the company says previously covered healthcare and finance. The tax release is designed around professional workflows rather than short factual questions: some tasks ask for filing packages and supporting workpapers, while others require investigating transactions, reviewing prior filings or advising on a decision that has not yet been made.
Surge says tax experts develop assignments from real-world workflows, assemble the records and set expected outcomes and grading criteria. It describes a three-layer expert review for correctness, completeness and solvability. The company also identifies reviewers with experience in corporate taxation, partnerships, real estate, individual returns and trusts. That process provides information about how the benchmark was prepared, though readers would need the underlying task set and evaluation details to independently assess its representativeness.
“This is the kind of professional judgment we’re testing with DAYJOB: Tax.”
— Surge AI, describing the benchmark’s purpose
Limits of the Reported Scores
The supplied report does not specify its publication date, the full model-testing setup, the number of runs behind each leaderboard score, or whether the benchmark tasks and grading rubrics are publicly available. The reported pass@1 figures apply to Surge’s 95% threshold; they do not by themselves show performance at lower thresholds, on other tax benchmarks or in live client work. Surge also mentions a comparison of model performance and average cost per trial, but the supplied material gives no cost figures.
The source excerpt cuts off during a second example about a separating couple and retirement accounts. It does not provide the complete facts, the rubric outcome or the company’s analysis of that case. The benchmark results therefore cannot establish from this material whether the model’s marital-status conclusion was wrong, what evidence it overlooked, or what practical advice the couple should receive.
Further Benchmark Details Needed
Surge’s report presents DAYJOB: Tax as a new addition to its benchmark series. The supplied material does not announce a future test date, an independent evaluation or a plan for updating the model rankings. The next useful evidence for readers would include fuller methodology, task-level results, the cost data referenced in the report and access to enough benchmark materials for outside researchers to review the scoring.
Until such information is available, the leaderboard is best read as a snapshot of how named models performed on Surge’s set of 50 tax assignments under its stated grading rule. It offers examples of the types of work being tested, but it does not settle whether an AI agent can safely replace a tax professional or reliably handle a particular client’s filing or advice.
Key Questions
What is DAYJOB: Tax?
It is Surge AI’s benchmark of 50 professional tax assignments. Tasks include preparation, review and advisory work across eight areas of tax practice.
Which model scored highest?
Surge reports that Claude Sonnet 5.5 led with a 10.8% pass rate. Claude Fable 5.1 scored 9.6%, and Claude Opus 5.5 scored 9.2%.
What counts as a pass?
An attempt must meet at least 95% of the task’s rubric criteria. The criteria can cover calculations, supporting evidence, consequential issues and the final deliverable.
Do these results show that AI is reliable for tax advice?
No. They show performance on this particular benchmark under Surge AI’s grading method. The scores do not establish how agents perform across all tax situations or in live client work.
What remains unknown about the results?
The supplied report does not give full testing details, cost figures or enough information to independently assess how representative the assignments are. It also does not provide the complete outcome of the retirement-account example.
Source: surgehq.ai
