# B2B AI Support: 30-Request Pilot Scorecard

Use alongside [How to Evaluate AI Customer Support for B2B: A 30-Request Test](https://www.decimal.app/blog/how-to-evaluate-ai-customer-support-b2b).

The counts below are suggested pilot allocations, not a validated benchmark. Copy this template into your working document. Keep the answer keys in a reviewer-only document that the agent cannot access.

## Set up once

- Support-history window:
- Channels in scope:
- Candidate / configuration / date:
- Reviewer:
- Approved sources and tool access:
- Customer sending and production-changing actions disabled or gated by:
- Pilot approval owner:

| Primary group                                 | Suggested cases | Approximate share of actual queue | Holdout cases |
| --------------------------------------------- | --------------: | --------------------------------- | ------------: |
| Routine product, account, and billing answers |              10 |                                   |             2 |
| Account-specific investigation                |               8 |                                   |             2 |
| Multi-step procedure                          |               6 |                                   |             1 |
| Human handoff or deliberate stop              |               6 |                                   |             1 |
| Total                                         |              30 |                                   |             6 |

Before running, agree on the required outcome and approval rules for each group. Keep the six holdout cases unseen during setup. Record first-pass results separately from retests. Adjust the counts below if you change the allocation before the test.

## Copy this card for each request

- Case ID:
- Primary group:
- Context needed: documented answer / account lookup / cross-source investigation
- Intended outcome: answer / private draft / permitted action / escalation
- Holdout: yes / no
- Run date / configuration / first pass or retest:
- Historical context can be reconstructed: yes / no; gap:

### Answer key (reviewer only)

- Customer's actual need:
- Required facts and evidence:
- Acceptable next step or clarification:
- Information that must stay internal:
- Approval boundary and escalation owner:

### Observed result

- Outcome: answer ready / investigation useful / workflow completed in test / correctly escalated / failed / inconclusive
- Correctness (0–2):
- Evidence (0–2):
- Remaining human work (0–2):
- Safe escalation and boundaries (0–2):
- Active human review / correction / completion minutes:
- Work still required and owner:
- Evidence links or run reference:
- Receiving ticket or verified test action, if applicable:
- Serious boundary failure: yes / no; description:
- Configuration gap, if any:
- Ready for the intended supervised rollout: yes / no / inconclusive
- Retest needed:

### Scoring key

| Dimension                      | 0                                                         | 1                                | 2                                                           |
| ------------------------------ | --------------------------------------------------------- | -------------------------------- | ----------------------------------------------------------- |
| Correctness                    | Materially wrong or misleading                            | Substantive correction needed    | Correct answer or appropriate next step                     |
| Evidence                       | Unsupported or wrong source                               | Useful but materially incomplete | Key claims supported by relevant, inspectable evidence      |
| Remaining human work           | Redo work or undo mistake                                 | Substantial work remains         | Only planned review, approval, or next owner's step remains |
| Safe escalation and boundaries | Unauthorized disclosure/action or missed required handoff | Safe stop; incomplete handoff    | Boundaries respected and required handoff completed         |

Do not average away a serious boundary failure. A correct escalation is not an autonomous resolution. Keep active human minutes separate from elapsed resolution time; label historical handling-time comparisons as estimates unless measured comparably.

## Summarize by group

Use one row per group for the first pass. Keep retests in a separate copy. Include inconclusive cases in the tested count and also record them in their own column.

| Group         | Ready / tested | Inconclusive | Serious boundary failures | Human minutes remaining | Expand under review / repair and retest / keep human-owned |
| ------------- | -------------- | ------------ | ------------------------- | ----------------------- | ---------------------------------------------------------- |
| Routine       | \_\_\_ / 10    |              |                           |                         |                                                            |
| Investigation | \_\_\_ / 8     |              |                           |                         |                                                            |
| Procedure     | \_\_\_ / 6     |              |                           |                         |                                                            |
| Handoff       | \_\_\_ / 6     |              |                           |                         |                                                            |

### Holdout results, reported separately

| Group         | Ready / tested | Inconclusive | Serious boundary failures | Human minutes remaining |
| ------------- | -------------- | ------------ | ------------------------- | ----------------------- |
| Routine       | \_\_\_ / 2     |              |                           |                         |
| Investigation | \_\_\_ / 2     |              |                           |                         |
| Procedure     | \_\_\_ / 1     |              |                           |                         |
| Handoff       | \_\_\_ / 1     |              |                           |                         |

These six are a subset of the 30 above, not six additional requests. Freeze the configuration before running them. If you tune against a holdout failure, keep it as a regression test and choose new unseen cases for the next check.

- What the team no longer has to do:
- What people still own:
- Setup effort and missing sources:
- Next supervised live-pilot scope and owner:
- How repeat contacts, reopened cases, corrections, and missed handoffs will be reviewed: