Scoring documents against guidelines: CoT prompting takes us from variance to consistency

Scoring documents against guidelines: CoT prompting takes us from variance to consistency

Being a brand-aware document writing and review agent, one of the tasks the agent fulfilled was to score documents sent to it measuring conformance to the organisation's brand documentation guidelines. The score provides other agents, processes, and human reviewers a sense of the quality of the document reviewed or written by the agent.

Scoring documents for guideline conformance using an LLM grader is straightforward in theory. In practice, the scoring drifts. A plain structured prompt produced scores that swung 18 points across repeated runs on the same document, 66-84 out of 100. The grading rubric assigned compounded the problem as the lower end versus the higher end of these scores lay in two different acceptance bands. That variance kills trust in the output.

After multiple rounds of tuning the structured prompt with limited results, we switched the approach to chain-of-thought prompting. By asking the model to work through the scoring step-by-step identifying applicable guidelines, analysing each one for deviation, costing them, and assigning the score from the analysis. The variance tightens to 76–84, resulting in an improvement that is acceptable.

The baseline approach

The starting point was a structured prompt that asked the model to score a submission against organisational guidelines, weigh deviations by severity of the guideline not respected and the frequency it was broken, and return an overall score from 0 to 100 plus an acceptance verdict (pass, review, or fail) and brief reasoning.

The guidelines themselves were three HTML files, 250KB total, hosted internally. The prompt included the full guideline text and a clear rubric for weighting deviations. Some representative examples follow.

• Hard guideline breaches cost far more than soft ones.
• Breaches of a guideline in many places cost more than single breaches.
• Missing required elements cost the same as guideline violations.
• Missing a soft guideline in one place costs little.

On a sample of two documents scored 20 times each, the variance was large. The same document scored anywhere from 66 to 84. Worse, the score range spanned two bands in the rubrics in their quality implications. A score under 70 could mean "the document needed a good amount of hard guideline violations fixed and was not good to publish as is" and a score above 70 indicated that "1 or 2 hard guidelines broken along with a few recommended guidelines. Hence the document needed a quick review and patch before publishing" depending on where in the 70–89 range it fell. That ambiguity left the review process without a clear direction to proceed, followed by a lack of trust in the agent.

The approach: chain-of-thought prompting

The approach that yielded results for us was to add explicit steps. Instead of asking the model to score directly, the revised prompt asked it to identify which guidelines apply to the document type, then analyse each guideline in order with hard ones first. For each deviation found, the model quoted both the rule and the passage that breaks it, reported each breach separately, and noted how widely it appears. Only after this analysis did it assign the score by weighing the deviations against the rubric, then set the verdict from the score alone.

Crucially, the prompt asked the model to write out its reasoning first, then derive the score from that reasoning. This forces the model to commit to specific deviations before scoring, rather than scoring and then rationalising the number.

The same experiment conducted with the revised prompt produced scores in the 76–84 range: a tighter band, and one that stayed within a single quality tier. The variance dropped by roughly half.

Why this works

Chain-of-thought prompting works here because the scoring task is deterministic. The guidelines are explicit and thereby the deviations are measurable. The scoring rubric based on the deviations is clear. The model's reasoning becomes auditable: you can see which deviations it found and how it weighted them. If the score seems wrong, you can trace back to the analysis and spot the error. When the evaluation criteria are concrete, asking the model to show its work tightens the output and makes it trustworthy.

Implementation note

If you use Pydantic models to capture the output, define the reasoning field before the score output field in the model. It is good practice to write it this way, though not all models need the ordering hint.

Lesson

A structured prompt is like a list of ingredients. The guidelines, severity rules and the scoring rubric are the ingredients without specifying a method to evaluate them. So, each run cooks its own score. The chain-of-thought prompting technique in this case acts as the recipe. Instructions on breaking down the guidelines by severity, checking which ones apply, costing each deviation based on patterns and then scoring based on this reasoning results in a better final dish.