In this week’s free edition, I described my increasingly awkward arrangement with Claude and ChatGPT Astra. I have reasons to use both. I also have work to finish, and deciding which assistant gets which assignment wasn’t originally on the list.
For finance teams, that decision deserves a little structure. Otherwise, the person who watched the most recent demo tends to win, at least until somebody watches another one.
I’ve put together a workbook to help you compare two tools on three jobs your team actually does. It records the output, the corrections, and the economics of keeping your current tool, replacing it, or using both. The final decision stays with you.
Download the Finance AI Evaluation Workbook
The workbook uses A for your current tool and B for the alternative. It doesn’t contain model rankings or my own test results. The Example sheet uses invented numbers to show how the calculations work, and those numbers never feed your evaluation.
Pick three jobs you recognize
Start with work you’ve reviewed often enough to recognize a good result. You’ll need source files, an expected deliverable, and a person who can judge whether the output is usable.
The Task briefs sheet starts with three editable assignments.
Investigate a variance. Supply actuals, budget, account mappings, transaction detail, and your materiality threshold. Ask for a reconciled variance bridge, supported explanations, and a list of what remains unresolved. An unexplained difference should still be there when the model finishes.
Write management commentary. Give each tool the same approved analysis, audience, word limit, and two examples of writing you’d be comfortable sending. Using an approved analysis isolates the communication task. You can see whether the tool preserves meaning and expresses it clearly without also wondering whether it calculated revenue correctly.
Update a reporting package. Provide a copy of a workbook, new data, the relevant mappings, and a presentation template. Check the formulas, totals, and agreement between files. Use copies, with clear limits on what the tool can change. A successful demonstration shouldn’t create a surprise in the production workbook.
You can substitute reconciliations, cash reporting, or another recurring assignment. Keep the comparison to three distinct jobs. If the same work appears in two tasks, don’t count its monthly benefit twice.
Before running anything, write the acceptance criteria. Specify the period, entity, tolerance, required evidence, and output format. Decide what would count as a critical error. The workbook includes starting examples, but your reviewer should adapt and approve them.
Give both tools the same assignment
A long-running conversation with your current assistant may contain months of useful background. A new account won’t have that context. Comparing them without addressing this would mostly test how much you’ve already explained to one of them.
Build a small, shared set of inputs. Include business definitions, source files, relevant exceptions, and examples of the expected output. Use it in fresh sessions with both tools.
Record the product, plan, exact model, effort setting, integrations, and context version in the workbook. A model working through a browser has different opportunities and limitations from one receiving uploaded files. For this exercise, compare the setups your team would actually use and document those differences.
Run three paired trials per task, for eighteen runs altogether. Use the same case for A and B within each pair. Across pairs, include different representative cases, perhaps a routine month, a messier month, and a known exception. Alternate which tool goes first, and avoid giving the second tool corrections learned from reviewing the first.
Three pairs are enough to start noticing problems. They aren’t enough to establish a reliable failure rate. If performance varies widely, extend the evaluation with a fresh workbook and more cases before deciding.
Keep the review on the clock
In the Runs sheet, score the first output from one to five on correctness, supporting evidence, and usability. The scoring descriptions are in Task briefs. The workbook averages these equally so you can scan the results, but it keeps critical errors separate.
A beautiful slide can’t compensate for an invented explanation. Mark a critical error even if you eventually repair it. Separately, record whether the finished output meets your acceptance criteria after review.
Then record your time in four categories: preparation and prompting, review and editing, repair or recovery, and handoffs. Keep those categories separate so the same minute doesn’t appear twice.
Include retries in the original trial. If you give up and finish the assignment yourself, count that recovery time. Otherwise, the assistant’s worst performance disappears from the comparison, which is a generous arrangement for the assistant.
Elapsed time gets its own field. An agent running unattended for half an hour uses thirty minutes on the clock, but it hasn’t necessarily consumed thirty minutes of your attention. The workbook values active human time and records elapsed time separately. Review the latter when a deadline or turnaround requirement is part of the job.
Record interventions, reviewer names, and links or locations for the outputs. A short note about what went wrong will be more useful next month than a score you can no longer explain.
The faster draft that costs more
The workbook’s fictional example compares two tools preparing the same reporting task. Both eventually produce usable work.
Minutes to first draft: Current tool A: 10; Alternative B: 5.
Active preparation and prompting: Current tool A: 4; Alternative B: 3.
Review and editing: Current tool A: 12; Alternative B: 18.
Repair and recovery: Current tool A: 4; Alternative B: 7.
Handoffs: Current tool A: 0; Alternative B: 2.
Total active human minutes: Current tool A: 20; Alternative B: 30.
Variable usage cost: Current tool A: $1; Alternative B: $2.
B produces a draft five minutes sooner. It also requires ten more minutes of human work before the job is finished.
At an illustrative loaded labor rate of $60 per hour, the labor and usage cost comes to $21 per task for A and $32 for B. At twelve tasks a month, with monthly fixed fees of $20 and $40 respectively, the totals are $272 and $424.
That’s $152 more per month for B. Add $240 of future setup cost and there isn’t a payback while the recurring difference remains negative.
Those are invented values, not evidence about Claude or Astra. They demonstrate why the time to the first answer can be such an appealing and incomplete measurement.
Put the three arrangements through the calculation
Once your run records are complete, enter monthly task frequency and your cost assumptions on the Decision sheet. Enter zero where a cost is genuinely zero. An empty cell means you haven’t supplied it yet.
The workbook calculates labor plus variable usage for each task, then compares three arrangements:
Keep A: all three jobs remain with your current tool.
Replace A with B: all three move to the alternative.
Use both: you select A or B for each job and include extra monthly coordination time.
Fixed fees should reflect the total relevant fees for each arrangement. Don’t enter the full subscription fee and then count subscription-covered usage again as an additional cash expense. Use variable charges only where they are actually incremental, and use the same currency throughout.
The hybrid estimate needs particular care. Include task-specific handoffs in the run log. Add only the remaining monthly coordination work on the Decision sheet. If you divide a single job between tools, test that combined process as its own assignment before relying on the savings estimate.
For setup, include future migration, testing, training, and cash expenses. Time already spent evaluating the tools is sunk for the go-forward decision. You can record it separately for the project budget, but spending it doesn’t make switching a better choice now.
The workbook shows payback only when recurring savings are positive. The calculation assumes your measured performance and monthly volume continue. It values released staff capacity; it doesn’t claim that payroll will fall by the same amount.
Make the decision you can explain
Review status appears alongside cost. Missing approvals, incomplete runs, and critical or unusable outputs stay visible. A low number in the cost column doesn’t clear a failed task.
You might keep A because the alternative’s improvement is too small to justify the disruption. You might replace A because B performs consistently across the work you tested. Or B may earn one specific assignment while A remains the default for everything else.
Postponing is also a complete decision when the evidence is mixed. Write down what you still need to learn, who will review it, and what would justify reopening the comparison.
Even if all three sampled jobs favor B, check the other responsibilities of your current platform before canceling it. Three assignments can’t represent an entire department’s work.
For most teams, a specific conclusion will be more useful than a declaration about which AI company is winning: this tool gets this job, under these conditions, because it produces an acceptable result with less total effort.
That’s enough to make the next assignment easier. The assistants can wait for their annual rankings.

