Refining a playbook from review data

Every review leaves a trail of decisions: on each finding, the reviewer records whether they agree, partially agree, or disagree with the AI’s call (see Understanding finding verdicts). Most tools throw that signal away. Clment uses it.

When your reviewers keep disagreeing with the same playbook item, that item is usually the problem — too strict, ambiguous, or just not how your team actually works. Refine from review data surfaces those items and drafts an improvement from your reviewers’ own notes, so your playbook gets sharper the more you use it.

It’s free — refining a playbook costs no review credits.

The signal

For each item in a playbook, Clment tallies how reviewers decided on the findings it produced, across every review run against that playbook:

• Agree — the reviewer accepted the finding.
• Partial — they agreed in part.
• Disagree — they rejected it.

An item gets flagged when reviewers consistently push back — either a clear majority of decisions go against it, or it racks up several outright disagreements. A flag never changes anything on its own; it just means “worth a look.”

Refining a playbook

Open the playbook and choose Refine from review data. (A playbook with flagged items also shows a badge, and its owner gets a nudge — see below.)

1. Read the signal. The dialog lists each item with its agree / partial / disagree counts, a pushback percentage, and a sample of the reviewer notes explaining why they disagreed. This is the evidence — read it before changing anything.
2. Generate suggestions. Clment drafts a change for each flagged item — a reworded rule, an added exception, or a recommendation to drop it — each with a short rationale citing the pushback.
3. Pick what to apply. Every suggestion has a checkbox. Keep the ones you agree with.
4. Preview. See the full revised playbook the way it will actually read, or switch to a line-by-line Changes view.
5. Save as a new version. Applying the accepted changes saves them as a new version of the playbook.

Clment only ever suggests — nothing changes until you approve it, and the result lands as a normal new version you can edit further or roll back from.

Choosing which reviews count

A playbook gathers reviews over time, and some may predate your last edit — run against wording you’ve since changed. The dialog shows the provenance up front (“Based on 22 reviews · 6 since your last edit”) and lets you narrow the pool:

• All (the default) — every review. Old pushback on rules you haven’t changed is still valid signal, so nothing is hidden by default.
• Since last edit — only reviews run after you last edited the playbook.
• Last 12 months — only recent reviews.

Reworded items are handled automatically either way: a finding carries the wording it was reviewed against, so changing an item’s text starts its pushback fresh rather than dragging the old version’s history forward.

Light pushback

By default, suggestions are drafted only for items reviewers consistently push back on. If a playbook has nothing flagged but you still want to look at items with just one or two disagreements, tick Include items with lighter pushback before generating — useful for an early, lightly-reviewed playbook.

The proactive nudge

You don’t have to go looking. When a playbook item newly crosses the pushback threshold after a review is signed off, the playbook’s owner gets an in-app notification linking straight to the refine dialog. It won’t spam you — there’s a cooldown, and it only fires when something new crosses the line.

What it never does

• No automatic changes. It drafts; you approve. A flag or a suggestion never edits your playbook on its own.
• No effect on past reviews. Saving a new version doesn’t touch findings from reviews already run — those stay locked to the version they used (see Creating a playbook → Versioning).
• No credit cost. Analysing pushback and drafting suggestions is free.

See also

• Creating a playbook — write the rules in the first place, and check your playbook’s strength.
• Understanding finding verdicts — what agree / partial / disagree mean.
• Reviewing contracts — where the decisions come from.