LLM evaluation for production teams: a practical starting map

Shipping an LLM feature without evaluation infrastructure means every prompt tweak is a coin flip. You don't need a research lab to fix that — you need a minimum viable eval loop.

Level 0 → Level 1: freeze expectations

Start by capturing ten to fifty real inputs with acceptable-output criteria. Even imperfect datasets create regression detection: change a prompt, re-run, compare. promptfoo makes this local and CI-gated cheaply; DeepEval suits Python-native teams wanting metric structure inside pytest.

Choosing scorers honestly

Three families exist, each with failure modes:

Deterministic checks (contains, regex, JSON validity) — reliable but shallow. Statistical metrics (BLEU-family, embedding distance) — cheap signals, easily gamed, weak for open-ended generation. Model-graded metrics (faithfulness, helpfulness) — closest to human judgment but variance-prone. Pin judge models, temperature-zero them, and spot-check judgments weekly.

Document scorer limitations like you'd document a known bug. Teams that hide judge unreliability rebuild trust expensively later.

Level 2: wire into delivery

Gate deploys on eval results exactly like unit tests fail the build on regressions. GitHub Actions running an eval matrix per PR converts "prompt engineering" into software engineering.

Level 3: close the production loop

Sample real traffic, route failures back into datasets (Langfuse and Braintrust both support online evaluation patterns). Your eval suite becomes a living artifact reflecting actual user pain, not launch-week guesses.

Tool selection shortcut

Local-first and security-sensitive: promptfoo. Python/RAG-heavy: DeepEval. Full platform with tracing and annotation queues: LangSmith. Self-hosted telemetry-first: Langfuse. All four are catalogued here with verified-basis summaries and honest limitation notes.