The AI Pilot Scorecard: Ninety Days to a Decision You Can Defend
Most AI pilots in CS end the same way: nobody can say whether they worked. A five-metric scorecard, scored monthly over ninety days, with a gate that forces one of three calls — scale, extend, or kill — and the evidence to defend whichever one you make.
The most common outcome of an AI pilot in a CS org is not success or failure. It is ambiguity — ninety days of activity, a deck full of anecdotes, and a renewal decision on the tooling that nobody can defend with a number. The fix is not better AI. It is deciding, before the pilot starts, exactly what you will measure and exactly what score forces which decision.
I have now watched a dozen CS teams run AI pilots — copilots for call prep, drafting assistants for QBR decks, summarizers bolted onto the CRM — and the pattern that repeats is not technical. The tools mostly work. What fails is the decision at the end. The pilot finishes, the vendor asks about rollout, the CFO asks what it returned, and the answer is a collection of screenshots and a CSM who says it "saves a ton of time." Nobody wrote down what success meant on day one, so on day ninety every opinion is equally valid, and the loudest one wins.
That is an expensive way to make a tooling decision, and it gets more expensive every quarter as the number of tools pitched at your team grows. The discipline that fixes it is the same one that fixes renewal forecasting and health scoring: pick a small number of metrics you can actually measure, anchor the scores so a stranger would assign the same number, and pre-commit to the decision each score band forces. What follows is the scorecard I use. It fits on one page, takes about an hour a month to score, and ends every pilot with a call you can defend line by line.
Why ninety days, and why a gate
Ninety days is long enough for novelty to wear off and short enough that the org still remembers why it started. The first two weeks of any AI tool produce inflated numbers — everyone tries it, everyone has a story. The signal you care about is what usage and output look like in weeks six through twelve, once the tool has to survive on usefulness rather than curiosity. A thirty-day pilot measures enthusiasm. A ninety-day pilot measures habit.
The gate matters more than the duration. Before the pilot starts, you write down three outcomes and the score that triggers each one: scale it, extend it once under changed conditions, or kill it. The pre-commitment is the entire point. If you wait until day ninety to decide what the numbers mean, the numbers will mean whatever the person with the strongest prior wants them to mean. A gate agreed in advance converts the end of the pilot from a debate into a reading.
The five metrics
Each metric is scored 1–4 at day 30, day 60, and day 90. The day-90 score is the one the gate reads; the earlier two exist to show trajectory and to catch a pilot that is dying early enough to fix the setup rather than waste the quarter. Five metrics, maximum score 20.
Time returned. Hours per CSM per week the tool gives back, measured against the baseline you captured before the pilot — not estimated afterward. This is the metric everyone claims and almost nobody measures. The baseline is non-negotiable: time two weeks of the target workflow before the tool arrives, or accept that your headline number is folklore. Self-reported "it saves me hours" scores a 1 no matter how enthusiastic the report.
Output quality. How much of what the tool produces survives contact with a human editor. For drafting tools, track the share of AI drafts sent with light edits versus rewritten from scratch. For summarizers, spot-check a sample weekly against the source call or thread. A tool whose output is 80% rewritten is not saving time — it is adding a review step, and your time-returned number is lying to you. This metric exists to keep the first one honest.
Adoption depth. Not logins — workflow presence. What share of the team used the tool in the workflow it targets, unprompted, in the last two weeks of the scoring period? Mandated usage in week two tells you nothing. Unprompted usage in week ten tells you almost everything. If adoption needs constant management pushing past day 60, the tool has failed the only vote that counts, whatever the dashboard says.
Customer-visible impact. Did anything the customer experiences actually move — response time on inbound questions, QBR prep turning into QBR conversation, follow-ups arriving the same day instead of Thursday? This is the hardest metric to move in ninety days and the easiest to over-claim. Score it only on things you can point at: a cycle-time number, a customer remark in writing, a before/after you could show a skeptic. If nothing customer-visible moved, that is not automatically failure — but it caps how excited the rollout case gets to be.
Risk and rework. Count the incidents: the hallucinated detail that reached a customer, the summary that inverted what the champion said, the draft that leaked internal pricing language, the hour spent cleaning up any of the above. Every tool generates some. The score reflects both the count and the ceiling — one incident that reached a customer unreviewed outranks five caught in review. A pilot with great productivity numbers and an unmanaged incident stream is not a success; it is a liability with good PR.
The scorecard
| Metric | What a 4 looks like | What a 1 looks like |
|---|---|---|
| Time returned | 3+ hrs/CSM/week against a measured baseline | Self-reported savings, no baseline |
| Output quality | Most output ships with light edits; spot-checks clean | Most output rewritten; summaries miss the point |
| Adoption depth | Whole team in-workflow, unprompted, weeks 10–12 | Usage only when managers ask about it |
| Customer-visible impact | A customer-facing number moved and you can show it | Nothing the customer experiences changed |
| Risk & rework | Zero incidents reached a customer; rework trivial | Unreviewed output reached customers, or rework eats the savings |
Scores are anchored, not averaged from vibes. The test for every number: could someone who was not in the pilot read your anchor, look at your evidence, and land on the same score? If a score depends on who is scoring, it is not a 3 — it is unmeasured.
The gate
Add the five day-90 scores. Three bands, three pre-committed calls:
16 or better, nothing below 2 — scale it. Roll it out, put the time-returned number in the capacity plan, and set a date to re-score at full rollout, because numbers from an eight-person pilot do not always survive a forty-person org.
12 to 15 — extend once, change one variable. One extension, thirty days, and you must name the single thing you are changing: the workflow it targets, the segment of the team using it, the prompt and template setup, the review step. An extension that changes nothing is a delay with better branding. One extension maximum — a pilot that cannot clear the gate in 120 days with a variable changed is telling you its answer.
11 or below — kill it, in writing. Write the two-paragraph post-mortem: what was promised, what was measured, why it is being stopped. This is not bureaucracy. It is the document that saves the next team ninety days when the same category gets pitched again next year, and it is what makes "we killed it" a credible, repeatable act rather than a quiet fade-out that lets zombie tools accumulate.
The bands are deliberately unforgiving in the middle. A 13 feels like "pretty good — ship it," and that instinct is exactly what the gate exists to block. A 13 means at least two of the five things you said mattered did not happen. Extend it and find out why, or kill it — but do not scale ambiguity.
Who scores it, and how to keep it honest
The scorecard has one operational failure mode: the person who championed the pilot scores it. Enthusiasm does not survive contact with a rubric, but it bends one. Three rules keep the reading clean.
Split the scoring. The pilot owner assembles the evidence; someone who does not own the outcome assigns the scores. In a small org that is your ops lead or a peer manager. The owner argues the evidence, not the number.
Evidence goes in the sheet, not the meeting. Every score gets one line of evidence written next to it — the measured hours, the spot-check sample, the usage pull, the incident log entry. A score with no line next to it reverts to 1. This single rule removes most of the theater: it is remarkable how a 3 becomes a 2 when someone has to write down what the 3 is based on.
Score the trend, decide on day 90. The 30- and 60-day readings are diagnostic, not binding. A pilot at 9 on day 30 with a broken setup gets the setup fixed, not a eulogy. But the day-90 reading is binding — that is what pre-commitment means. If the team reserves the right to overrule the gate when they dislike the answer, there is no gate, just a longer meeting.
One more boundary worth writing down on day one: what the pilot is not measuring. A ninety-day scorecard on a call-prep copilot says nothing about NRR, and pretending it does invites the CFO conversation you cannot win. Keep the claimed outcomes inside what five metrics can actually see.
A worked example
A nine-CSM mid-market team piloted a call-prep and follow-up copilot. Baseline captured first: 4.1 hours per CSM per week on prep and follow-up across a timed two-week sample. Day-90 scores: time returned 3 (1.5 measured hours back), output quality 3 (follow-ups shipped with light edits; prep briefs solid), adoption 4 (nine of nine, unprompted, weeks 10–12), customer-visible impact 2 (same-day follow-ups doubled, but nothing a customer had remarked on), risk 2 — one summary inverted a renewal signal and reached an account team before anyone caught it. Total: 14. Extend.
The variable they changed: a mandatory fifteen-second review step on any summary touching renewal or escalation language, plus a tightened prompt template for the two call types generating the misreads. Thirty days later: quality 4, risk 3, impact 3 — a champion forwarded a recap to her own VP as "the cleanest vendor follow-up we get." Total: 17. They scaled it, with the review step kept as permanent policy and the incident log carried into rollout as a standing metric.
Without the gate, day 90 reads as a success story with one awkward incident, and the incident gets a slide titled "learnings." With the gate, 14 is 14: the team was forced to fix the actual defect before scaling it across every renewal conversation in the book. The scorecard did not slow them down. It made the version they scaled the version that deserved it.
What this changes on Monday
If you have a pilot running now with no baseline and no gate, you are not ninety days from a decision — you are ninety days from a debate. Salvage it: write the gate today, capture the best baseline reconstruction you can, and score what you have at the next thirty-day mark. An imperfect scorecard applied late still beats a clean narrative applied never.
If the pilot has not started, the sequence is: baseline first, gate in writing second, tool third. One hour before kickoff buys you a defensible answer at day ninety to the only three questions anyone senior will ask — what did it return, what did it break, and what are we doing next.
This piece opens the AI in CS pillar in the playbook library. The scorecard follows the same rule as every rubric on this site: scores you can defend, gates you don't argue with. The one-page scoring sheet is in the tools library, and The NRR Playbook sends one tactical idea like this every week.