# How to Evaluate AI Customer Support for B2B: A 30-Request Test

> Build a 30-request pilot from your own support queue. Test routine answers, account-specific investigations, workflows, and human handoffs with a practical scorecard.

_Manav Mehra · Oct 17, 2026 · 10 min read_

![Decimal logo and A 30-request test caption on a pale background with subtle halftone corners.](/blog-covers/b2b-ai-support-30-request-test.svg)

A customer asks why their invoice changed. Another can’t invite a teammate. A third says yesterday’s export is missing half its rows.

All three might arrive as short chat messages. The work behind them is different. One may need a billing policy and account history. Another may need a role check. The export may require someone to investigate what happened in the product.

An AI support evaluation should expose those differences. Otherwise, a strong performance on familiar help-center questions can hide the work your team will still have to do.

Start with 30 real requests from your own queue. For each one, decide what a good outcome requires, run the request through the proposed workflow, and record what remains for a person. Here’s a practical way to do it.

Thirty is a manageable starting point, not a validated benchmark. Use this test to find useful coverage and failure modes before a larger, supervised rollout. It cannot establish your future automation rate or prove that a system is safe.

Use the [pilot scorecard template](/downloads/b2b-ai-support-30-request-scorecard.txt) to record your setup, a separate answer key for each request, scores, remaining work, and rollout decisions. Save or copy the template into your team’s working document.

## 1. Choose requests before you see the demo

Take a recent, fixed window of support history, such as the last 30 days. Include the channels you expect the system to handle: chat, email, tickets, and customer Slack conversations where relevant.

Select within the groups below using a consistent method, such as a random draw. Don’t let the vendor choose the tickets, and don’t quietly replace failed cases with easier ones. Record exclusions, including requests you cannot share under your data policies.

| Primary job to test | Requests | What to include |
| --- | --- | --- |
| Routine product, account, and billing answers | 10 | Four product-use questions, three account or access questions, and three billing or plan questions |
| Account-specific investigation | 8 | Cases where documentation alone could not establish what happened: failed exports, missing records, unexpected limits, integration errors |
| A procedure with multiple steps | 6 | Requests requiring clarification, a policy decision, a lookup, routing, or an approved action |
| Human handoff or a deliberate stop | 6 | Exceptions, insufficient evidence, unavailable access, disputed charges, or issues requiring another team’s judgment |
| Total | 30 |  |

These are suggested allocations. Adjust them before the test if your queue differs substantially. Keep the difficult and high-consequence work represented.

Give each request one primary group so you don’t count it twice. Then add two separate tags:

- Context needed: documented answer, account lookup, or investigation across sources
- Allowed outcome: answer, draft for review, permitted action, or escalation

Those tags matter. A billing question can require investigation; an integration question can have a straightforward documented answer. Topic alone is a poor measure of difficulty.

Also record each group’s approximate share of your real queue. This deliberately mixed sample tests coverage. Its overall pass percentage is not a forecast of production performance.

## 2. Write the answer key before running the test

Ask an experienced support lead to review each case. The original human reply is useful evidence, but it may contain a mistake, an undocumented exception, or a workaround that no longer applies.

For each request, write down:

1. What the customer actually needed
2. The facts and sources required to answer
3. The acceptable next step, including any necessary clarification
4. What must stay internal
5. What needs human approval, and who should own an escalation

Keep this reviewer-only answer key out of the agent’s context.

For a historical replay, give the system the conversation up to the point being tested. Exclude the final resolution and internal notes that reveal the answer. Check connected knowledge for articles derived from that same case. Otherwise, you may be measuring whether the system can retrieve the solution to a ticket it has effectively already seen.

Use the account state, product version, and policies applicable to the case. If you can’t reconstruct them, mark the result inconclusive rather than treating today’s account state as evidence of what happened last month.

Reserve six of the 30 requests as a holdout: two routine, two investigative, one procedural, and one handoff case. Use the other 24 to find configuration gaps. Freeze the setup before running the six unseen cases, and report their results separately. If you tune against a holdout failure, that case becomes a regression test; select new unseen cases for the next check.

## 3. Test the workflow you intend to deploy

Decide what the pilot is evaluating. A customer-facing answer, a private investigation note, and an executed account change are different deliverables.

Give each candidate the approved sources and permissions required for its proposed deployment. Record missing integrations and setup effort. If a necessary source is unavailable, distinguish that configuration gap from an incorrect answer. The inability to complete the job still matters to the buying decision.

For the initial run, use test accounts and a controlled environment. Disable customer sends and production-changing tools where possible. Verify connected tool permissions and customer-delivery settings before running the test. Keep any consequential actions behind the human approval your team requires.

Run multi-turn requests through the whole conversation. When the agent asks a necessary question, supply only the information the customer could reasonably provide. Check whether it uses the answer, follows the appropriate policy branch, and reaches the right destination.

For a handoff, inspect the receiving queue. Did a ticket actually appear? Does it have an owner, the customer’s question, relevant evidence, and a clear next step? A message promising follow-up is not enough.

In Decimal, workspace-enabled [Policies](https://docs.decimal.app/policies) support procedures with questions, conditions, configured knowledge search, and escalation. Ticket creation requires the widget’s escalation integration to be configured. Include a policy-driven conversation in the pilot and verify its handoff end to end.

## 4. Score four things, then record the remaining work

Use the same reviewer guidance for every candidate. Have a second person review disputed results and serious failures.

| Dimension | 0 | 1 | 2 |
| --- | --- | --- | --- |
| Correctness | Wrong, misleading, or misses a material condition | Partly right; needs substantive correction | Correct for the available evidence, including a necessary clarification or escalation |
| Evidence | Unsupported, irrelevant, or wrong account/version | Useful support with an important gap | Key claims trace to appropriate, inspectable sources |
| Remaining human work | Person must redo the work or undo a mistake | Useful progress, but substantial work remains | Only the planned review, approval, or receiving team’s next step remains |
| Safe escalation and boundaries | Discloses restricted information, oversteps authority, or misses a required handoff | Stops safely, but the handoff or explanation is incomplete | Respects access and approval boundaries; completes the required handoff |

Don’t collapse these into a single winner’s score. An unauthorized action or cross-account disclosure cannot be offset by excellent answers elsewhere. Treat serious boundary failures as rollout blockers until corrected and retested.

For each case, record the outcome separately: answer ready, investigation useful, workflow completed in test, correctly escalated, failed, or inconclusive. “Correctly escalated” can be a successful result while still leaving substantial human work.

Measure active human minutes spent checking sources, correcting the response, completing the process, and preparing the handoff. Keep that separate from elapsed resolution time. If you compare against historical handling time, label it as an estimate unless you have a comparable measurement.

### Example: an invoice question with an approval boundary

Illustrative evaluation example, not a Decimal customer result.

A customer asks: “Why did our invoice increase? Please reverse the extra charge.”

The answer key says the increase came from additional seats partway through the billing period. The evidence must connect the correct account, invoice, seat-change record, and applicable billing policy. A credit requires the billing team’s approval.

A strong result explains the increase, identifies what remains to be reviewed, and creates the required billing handoff with the supporting records. It does not say the credit has been issued.

| Check | What the reviewer records |
| --- | --- |
| Correctness | Does the explanation match the account and billing period? |
| Evidence | Can the reviewer verify the seat change and invoice calculation? |
| Remaining work | What must Billing still investigate, decide, or do? How long did review take? |
| Boundaries | Was approval preserved, and did the case reach the right queue? |

The same approach works for a failed export: verify the affected run and evidence, then distinguish a confirmed cause from a hypothesis. Or for an access request: verify the requester’s authority before exposing account details or proposing a role change.

## 5. Look for value in the work between the question and the reply

This is where Decimal belongs in a B2B evaluation. Its [documented ticket workflow](https://docs.decimal.app/support/ticket) gives the team an investigation and a separate customer response, with sources that can include code, articles, and logs. That lets a reviewer test the explanation against the underlying evidence and measure how much investigation remains.

The published customer workflows give concrete examples of what to evaluate:

### Granola: routine answers and account-specific investigations

Decimal investigates incoming Plain tickets using customer context, CloudWatch logs, relevant code, and incident information. It places findings and a proposed response in a private note for a support engineer to review. The case study also reports that 70% of common questions are resolved in chat before becoming tickets. Include both routine questions and investigation-heavy cases when testing that combination. [Granola customer story](/blog/granola-decimal-case-study-2026)

### Resilinc: investigation and routing to the right team

Decimal’s deployment in Freshdesk includes private investigative notes and identifying requests that should go directly to Data Operations. Its story reports MTTR falling from 6.5 to 2.5 days as part of a broader support transformation that also included a helpdesk migration, process changes, and knowledge management improvements. Test routing quality and the work left for the receiving team alongside answer quality. [Resilinc customer story](/blog/resilinc-decimal-case-study-2026)

These are reported customer deployments, not results from this 30-request test or guarantees for your queue. Use them to choose concrete work to test: an answer grounded in policy, an account-specific investigation, a reviewable response, and a handoff that saves the next person from starting over.

## 6. Make a rollout decision by request type

Before looking at the scores, agree on what would justify expanding each workflow. Your standard for a public product explanation may differ from your standard for an account-specific billing response.

At the end, fill in this short decision sheet:

| Request group | Ready / tested | Inconclusive | Serious boundary failures | Human minutes remaining | Decision |
| --- | --- | --- | --- | --- | --- |
| Routine answers | ___ / 10 | ___ | ___ | ___ | ___ |
| Investigations | ___ / 8 | ___ | ___ | ___ | ___ |
| Procedures | ___ / 6 | ___ | ___ | ___ | ___ |
| Handoffs | ___ / 6 | ___ | ___ | ___ | ___ |

Define “ready” against the intended outcome, required evidence, and approval rules you set beforehand. Keep the six holdout results visible separately, and keep first-pass results separate from retests. Adjust the tested counts if you changed the suggested allocation. Choose among: expand under review, repair and retest, or keep human-owned.

You may find that routine answers are ready for a supervised live pilot, investigations are valuable as private notes, and billing exceptions should remain with people. That is a useful result. It tells you exactly where the product can help and what still needs work.

Then validate those decisions on new requests under supervision. Watch for repeat contacts, reopened cases, corrections, and missed handoffs. Thirty historical requests can earn the next stage of a pilot; they do not replace it.

Bring your sample to a [Decimal demo](https://cal.com/decimal-demo/30m). Ask to follow the evidence from the customer’s question through the investigation, reply, and handoff. By the end, you should be able to name the work your team no longer has to do, the work it still owns, and the evidence behind that judgment.