What it is

In the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop. Think writer + editor: draft, critique, revise, repeat — until the evaluation passes.

How it works

Input ─▶ [ Generator LLM ] ─▶ draft
                                  │
                                  ▼
                          [ Evaluator LLM ]
                          pass? ──no──▶ feedback ─┐
                            │ yes                  │ (loop back
                            ▼                      │  to generator)
                          Output  ◀────────────────┘

When to use it

Particularly effective when we have clear evaluation criteria, and when iterative refinement provides measurable value. — Anthropic
  • You can articulate what 'good' looks like as feedback the generator can act on.
  • Each round of refinement measurably improves the output.
  • A first-pass answer is rarely good enough on its own.

Trade-offs

  • The loop costs latency and tokens; you need a stopping criterion (max rounds / good-enough threshold).
  • Only as good as the evaluator — a weak judge yields weak refinement.
  • Academic grounding: Reflexion (Shinn et al., 2023) shows verbal self-reflection improves an agent across iterations.

Concrete examples

  • Literary translation, where an evaluator captures nuances the first pass missed.
  • Complex search that needs multiple rounds of analysis and refinement.
  • Examples from Anthropic; the self-refinement idea is formalized by Reflexion.