← Back

Personal Notes/Guide for AI Engineering

·Bryan Lai
TechnicalMLLLM

Personal Notes for AI Engineering

If you are building LLM products without evals, you are shipping vibes.

Synthesized from insights shared by experts at Traceloop, QuotientAI, HoneyHive, Halluminate, and OpenPipe.

Evals Are The Product

LLM apps are not normal software.

The same prompt can pass nine times and fail the tenth. Agents make this worse because one bad step can poison the next five.

So evaluation is not polish.

It is the product boundary.

An eval tells you what "good" means. It tells you when a prompt change helped, when a model swap hurt, and when a demo is not ready for users.

Start Small

Do not begin with a giant eval platform.

Start with one real task and one real failure.

  1. Build a basic version.
  2. Put it in front of a small group.
  3. Log everything.
  4. Find the failures.
  5. Turn the failures into tests.

The eval suite should grow from real mistakes, not imagined ones.

Log Everything

You cannot evaluate what you did not save.

Log:

  1. User input.
  2. Model output.
  3. Retrieved context.
  4. Tool calls.
  5. Tool results.
  6. Intermediate steps.
  7. User feedback.
  8. Model version.
  9. Prompt version.
  10. Timestamp.

Real production data beats synthetic data. Synthetic data is useful for bootstrapping, but it rarely captures how users actually ask, misunderstand, rush, typo, or change their minds.

Define The Baseline

Before you fine-tune, build agents, or add routing, run the simplest strong baseline.

Use a strong prompted model. Save the outputs. Score them.

That baseline becomes the thing every later change must beat.

Without a baseline, "better" is a feeling.

Build A Golden Set

A golden set is a small set of inputs with trusted answers or trusted properties.

Start small:

  1. Critical happy paths.
  2. Known failure cases.
  3. Edge cases.
  4. Safety cases.
  5. High-value customer examples.

Do not chase coverage before you have quality.

Ten sharp examples beat 500 sloppy ones.

Evaluate One Thing At A Time

Bad evals ask:

Is this answer good?

Good evals ask:

  1. Did it answer the question?
  2. Did it use the provided context?
  3. Did it avoid unsupported claims?
  4. Did it follow the required format?
  5. Did it call the right tool?
  6. Was the tone acceptable?

One judge. One criterion.

If you evaluate five things at once, you will not know what broke.

Use The Right Eval

Code Checks

Use code when the answer should be exact.

Good for:

  1. JSON schema.
  2. Required fields.
  3. Forbidden words.
  4. Regex checks.
  5. API success codes.
  6. Output length.

Code checks are fast and cheap. They are weak for taste.

Reference Checks

Compare the output to a known answer.

Good for:

  1. Classification.
  2. Extraction.
  3. Regression tests.
  4. Tasks with clear expected answers.

Be careful with BLEU, ROUGE, and similarity scores. They can reward text that looks close while missing the point.

LLM Judges

Use a model to judge another model.

Good for:

  1. Tone.
  2. Relevance.
  3. Safety.
  4. Faithfulness.
  5. Coherence.

But an LLM judge is not truth.

It can be biased. It can prefer its own style. It can be inconsistent. It can over-score confident nonsense.

Make judges useful:

  1. Give one criterion.
  2. Ask for yes/no when possible.
  3. Require a short explanation.
  4. Randomize answer order in comparisons.
  5. Compare against human judgment early.
  6. Use multiple judges for important tasks.

Human Review

Use humans when taste, safety, money, or customers matter.

Humans are slow and expensive. They are also still the best source of judgment.

Use domain experts for domain tasks. Prefer A/B comparisons over vague scores when possible.

Agents Need Different Evals

Agents are harder because the answer is not the whole story.

The path matters.

Check:

  1. Did it pick the right tool?
  2. Did it pass the right parameters?
  3. Did it understand the tool result?
  4. Did it recover from errors?
  5. Did it loop?
  6. Did it finish the task?
  7. Did it stop when it should have asked a human?

For agents, evaluate the path and the final message.

The final answer can look fine while the agent took a dangerous path to get there.

RAG Needs Two Evals

RAG fails in two places:

  1. Retrieval.
  2. Generation.

Evaluate them separately.

For retrieval, measure whether the system fetched the right chunks.

Useful metrics:

  1. Hit rate.
  2. MRR.
  3. Precision@K.
  4. Recall@K.

For generation, measure whether the model used the retrieved context correctly.

Ask:

  1. Is the answer supported by the context?
  2. Did it answer the user?
  3. Did it add unsupported claims?

If the final answer is bad, first find whether search failed or writing failed.

Production Monitoring Is Not Optional

Models drift.

Data drifts.

Users find new ways to ask.

Track:

  1. Quality score.
  2. Task success.
  3. Latency.
  4. Cost.
  5. Refusals.
  6. Regenerations.
  7. Edits.
  8. Thumbs up/down.
  9. Tool failures.

Aggregate scores are not enough. Read the failures. Cluster them. Find the pattern.

Common Mistakes

  1. Adding evals after the system is already complicated.
  2. Using fake data for too long.
  3. Treating clicks as quality.
  4. Treating LLM judges as ground truth.
  5. Evaluating too many things at once.
  6. Ignoring production drift.
  7. Fine-tuning before proving the baseline.
  8. Skipping human review for high-stakes output.

Bottom Line

Trust does not come from a better prompt.

Trust comes from seeing the system fail, naming the failure, writing the eval, and making sure it does not fail that way again.

Start simple.

Log real data.

Build a baseline.

Evaluate one thing at a time.

Read the failures.

That is how LLM demos become products.