In AI engineering, evaluation (evals) is evolving from a side tool to a core role. Changing a single sentence in a model prompt can lead an agent to take dangerous actions in edge cases, yet most teams still only understand evaluation as "run a test and check accuracy".

Suraj Sharma published a long thread on X outlining his perspective: if he had six months to become an AI evaluation engineer, he would follow a systematic 12-stage training plan and build 7 hands-on projects. The thread sparked widespread discussion, with many practitioners noting "Stage 9 is the most expensive lesson"—an evaluation suite that doesn't block bad deployments is nothing more than a dashboard.

Below is the organized breakdown of his roadmap by stage.

## Stages 1–3: Testing Fundamentals, Failure Modes, Datasets

Start with core testing fundamentals: pytest fixtures and parametrization, JSONL data processing, precision/recall/F1, confidence intervals, and Cohen's kappa. These are the foundational building blocks of all evaluation work. Build a pytest plugin that loads JSONL datasets and outputs a metric report on every run. Evaluation is testing, and testing is math. If you can't quantify agreement, you can't prove improvement.

Stage 2 focuses on understanding LLM failure modes: temperature, top-p sampling, seed control, tokenization boundaries, instruction drift, and hallucination classification. Fuzz a model with 500 adversarial examples, categorize every failure into your taxonomy, and document reproduction steps for each. Every evaluation suite is a map of known failure modes—you can't evaluate what you don't understand.

Stage 3 covers gold dataset engineering: annotation guidelines, stratified sampling, inter-annotator agreement, dataset versioning, and contamination detection. Build a 300-example dataset with two annotators, measure kappa, manage the dataset with git, and maintain a changelog. Evaluation quality depends entirely on dataset quality. Garbage annotations produce garbage confidence scores.

## Stages 4–6: Judges, RAG, and Trajectories

Stage 4: Scoring methods and LLM-as-a-Judge. Exact match, fuzzy match, embedding similarity, rubric design, judge selection, and common judge biases: verbosity bias, position bias, and self-preference bias. Build a rubric-based judge, calibrate it with 100 human labels, and publish an agreement score. An uncalibrated judge is just an expensive "gut feeling"—a calibrated judge is a proper measurement instrument.

Stage 5: RAG evaluation. Hit rate, MRR, NDCG, recall@k, faithfulness and relevance, citation grounding, and quality of refusals. Build a RAG evaluation harness that sets separate gates for retrieval and generation, and adds adversarial queries that require the model to return "no answer". RAG fails in two distinct places, and a single aggregated score will hide which环节 is broken.

Stage 6: Agent and trajectory evaluation. Tool call correctness, step-level scoring, trajectory distance, counterfactual replay, and sandboxed execution scoring. Build a trajectory scorer that compares every tool call against the gold standard path and blocks dangerous action sequences. Final answer evaluation hides where the agent actually broke down. The trajectory is the real unit of accountability. One commenter put it this way: final answer evaluation is like checking if a car reached its destination, not checking if it ran three red lights. A lucky correct final answer can cover up five dangerous wrong decisions made along the way.

## Stages 7–9: Frameworks, Statistical Significance, CI/CD Gates

Stage 7: Master DeepEval, promptfoo, RAGAS, Braintrust, and OpenAI Evals. Write a custom harness that runs three frameworks under a single CLI and outputs a unified metric report. Frameworks are just scaffolding—a custom harness gives you control when the scaffolding falls short.

Stage 8: Statistical rigor for evaluation differences. Bootstrap confidence intervals, paired significance testing, effect size, sample size calculation, and seed variance. Build a comparison report that outputs results like "Model B outperforms by 3.2% ± 1.1% (p<0.01)". A model comparison without confidence intervals is just rolling dice.

Stage 9: CI/CD regression gates. Run parameterized evaluations in GitHub Actions that execute automatically on every pull request, and block merging if task success rate drops more than 2% or latency spikes. An evaluation that doesn't block deployment is just a report, not a gate. One commenter noted: The dashboard is where evaluation starts, but the gate is where evaluation becomes engineering.

## Stages 10–12: Monitoring, Data Flywheels, Red Teaming

Stage 10: Production monitoring and drift detection. Use Langfuse, LangSmith, and Phoenix to sample 5% of production traffic, run offline evaluations every night, and trigger alerts when quality degrades. Gold datasets go out of date—production is the most up-to-date evaluation suite.

Stage 11: Data flywheels and evaluation-driven optimization. Automatically format user downvotes into preference pairs that trigger overnight LoRA fine-tuning. Build a feedback loop: downvote → annotated case → added to gold dataset → nightly evaluation → difference report. In AI, the compounding advantage isn't the model—it's the flywheel that turns failures into tests. One commenter warns: Before turning downvotes into preference pairs, you must do PII scrubbing and cause annotation, otherwise the model will just learn the anger of frustrated users.

Stage 12: Red teaming, public benchmarks, and portfolios. Adversarial suites (prompt injection, jailbreaking, data leakage), benchmark methodology, contamination-aware reporting. Publish a public evaluation breakdown that compares your agent against a naive baseline, with full methodology and reproducible seeds. Senior evaluation engineers get hired for their methodology, not their dashboards.

## A Real-World Case: User Edits as Ground Truth

In a reply to the thread, Abhay Sehgal shared the evaluation loop his team built for LLM extraction features. The difference between the content users edit in a form and the LLM's pre-filled content is a free, high-quality signal. They use a stronger model as a judge to distinguish between "the model was wrong" and "the user intentionally changed it", group cases by root cause, backtest prompt fixes, and only update prompts after manual approval.


The key insight of this case: if you're already collecting user corrections, you already have an evaluation dataset. Don't just stare at the accuracy number—ask when that number can mislead you.

## Closing Thoughts

Vibe-checking is dead. If you can't measure it, you can't ship it. The job of an AI evaluation engineer is to build quality gates for every autonomous system. Suraj's roadmap looks long, but as he puts it: "You can spend months building just a handful of these projects." Evaluation is not a one-off task—it's a continuously running engineering system.