Refining a rulebook from review data

Every review leaves a trail of decisions: on each finding, the reviewer records whether they agree, partially agree, or disagree with the AI’s call, plus a note on what they’d change (see Understanding finding verdicts). Most tools throw that signal away. Clment uses it — it’s why the finding cards remind you that your answers help improve your rulebook (sometimes known as playbooks) for future reviews.

When your reviewers keep disagreeing with the same rulebook item, that item is usually the problem — too strict, ambiguous, or just not how your team actually works. Refine from review data surfaces those items and drafts an improvement from your reviewers’ own notes, so your rulebook gets sharper the more you use it.

It’s free — refining a rulebook costs no review credits.

The signal

For each item in a rulebook, Clment tallies how reviewers decided on the findings it produced, across every review run against that rulebook:

• Agree — the reviewer accepted the finding.
• Partial — they agreed in part.
• Disagree — they rejected it.

An item gets flagged when reviewers consistently push back — either a clear majority of decisions go against it, or it racks up several outright disagreements. A flag never changes anything on its own; it just means “worth a look.”

Refining a rulebook

Open the rulebook and choose Refine from review data. (A rulebook with flagged items also shows a badge, and its owner gets a nudge — see below.)

1. Read the signal. The dialog lists each item with its agree / partial / disagree counts, a pushback percentage, and a sample of the reviewer notes explaining why they disagreed. This is the evidence — read it before changing anything.
2. Generate suggestions. Clment drafts a change for each flagged item — a reworded rule, an added exception, or a recommendation to drop it — each with a short rationale citing the pushback.
3. Pick what to apply. Every suggestion has a checkbox. Keep the ones you agree with.
4. Preview. See the full revised rulebook the way it will actually read, or switch to a line-by-line Changes view.
5. Save as a new version. Applying the accepted changes saves them as a new version of the rulebook.

Clment only ever suggests — nothing changes until you approve it, and the result lands as a normal new version you can edit further or roll back from.

Choosing which reviews count

A rulebook gathers reviews over time, and some may predate your last edit — run against wording you’ve since changed. The dialog shows the provenance up front (“Based on 22 reviews · 6 since your last edit”) and lets you narrow the pool:

• All (the default) — every review. Old pushback on rules you haven’t changed is still valid signal, so nothing is hidden by default.
• Since last edit — only reviews run after you last edited the rulebook.
• Last 12 months — only recent reviews.

Reworded items are handled automatically either way: a finding carries the wording it was reviewed against, so changing an item’s text starts its pushback fresh rather than dragging the old version’s history forward.

Light pushback

By default, suggestions are drafted only for items reviewers consistently push back on. If a rulebook has nothing flagged but you still want to look at items with just one or two disagreements, tick Include items with lighter pushback before generating — useful for an early, lightly-reviewed rulebook.

The proactive nudge

You don’t have to go looking. When a rulebook item newly crosses the pushback threshold after a review is closed, the rulebook’s owner gets an in-app notification linking straight to the refine dialog. It won’t spam you — there’s a cooldown, and it only fires when something new crosses the line.

What it never does

• No automatic changes. It drafts; you approve. A flag or a suggestion never edits your rulebook on its own.
• No effect on past reviews. Saving a new version doesn’t touch findings from reviews already run — those stay locked to the version they used (see Creating a rulebook → Versioning).
• No credit cost. Analysing pushback and drafting suggestions is free.

See also

• Creating a rulebook — write the rules in the first place, and check your rulebook’s strength.
• Understanding finding verdicts — what agree / partial / disagree mean.
• Reviewing contracts — where the decisions come from.