Skip to content
New this month 24 fresh C++, C#, F#, JavaScript, TypeScript and Lua courses just landed. Browse new releases Use code WELCOME10 for 10% off your first order · 14-day refund
WisdomCharms — home
Cart 0 items in cart
Prompt Engineering Intermediate New

Evals for Prompts: Stop Guessing Whether It Got Better

Build a small eval suite that tells you honestly whether a prompt change was an improvement.

Rated 4.0 out of 5 from 2 reviews 11 students

LM Created by Laura Mbeki

  • Last updated August 2026
  • English
  • 6h of material
  • 45 lessons

What you will learn

8 concrete outcomes

Every bullet below is something you will have built, shipped or be able to explain by the time you finish the last lesson.

  • Assemble an eval dataset from real traffic, weighted toward the hard cases
  • Choose between exact, structural, similarity and model-graded scoring
  • Calibrate a model grader against human labels and measure agreement
  • Understand sampling variance and how many examples a claim needs
  • Detect position and verbosity bias in an automated grader
  • Run evals in continuous integration and fail a build on regression
  • Track quality, latency and cost together rather than separately
  • Maintain a golden set that does not leak into the prompts it tests

Course curriculum

6 modules · 45 lessons · 6h of material

7 lessons running 52m in total. Each lesson ships with the finished source files and a short written recap, so you can follow along in your own editor and skim the module again later.

Lesson-by-lesson titles, code downloads and exercises live inside the course library you get access to straight after checkout.

8 lessons running 1h 2m in total. Each lesson ships with the finished source files and a short written recap, so you can follow along in your own editor and skim the module again later.

Lesson-by-lesson titles, code downloads and exercises live inside the course library you get access to straight after checkout.

8 lessons running 1h 4m in total. Each lesson ships with the finished source files and a short written recap, so you can follow along in your own editor and skim the module again later.

Lesson-by-lesson titles, code downloads and exercises live inside the course library you get access to straight after checkout.

8 lessons running 1h 6m in total. Each lesson ships with the finished source files and a short written recap, so you can follow along in your own editor and skim the module again later.

Lesson-by-lesson titles, code downloads and exercises live inside the course library you get access to straight after checkout.

7 lessons running 58m in total. Each lesson ships with the finished source files and a short written recap, so you can follow along in your own editor and skim the module again later.

Lesson-by-lesson titles, code downloads and exercises live inside the course library you get access to straight after checkout.

7 lessons running 58m in total. Each lesson ships with the finished source files and a short written recap, so you can follow along in your own editor and skim the module again later.

Lesson-by-lesson titles, code downloads and exercises live inside the course library you get access to straight after checkout.

6 modules · 45 lessons

6h total length

Requirements

Short list, and deliberately so. If you meet these you can start today.

  • Comfortable writing Python and running a test suite
  • An application or prompt you actually want to evaluate
  • API access to at least one model provider

About this course

Every team building with models reaches the same wall. Someone edits a prompt, tries it three times, declares it better, and ships. Two weeks later nobody can say whether the system improved or quietly regressed. Evals are the cure, and they are far more approachable than the literature suggests.

You build an eval suite from nothing. First a dataset, drawn from real inputs rather than invented ones, deliberately weighted toward the cases that hurt. Then graders in increasing order of difficulty: exact match and structural checks for anything typed, similarity and rubric scoring for open text, and model-graded evaluation with all its pitfalls, including position bias, verbosity bias and graders that agree with themselves more than with reality.

You will learn to calibrate a model grader against human labels, to compute agreement honestly, and to know how many samples you need before a difference means anything. Sampling variance gets its own module, because comparing two prompts on ten examples proves almost nothing and most teams do exactly that.

The final modules put evals into a workflow: running them in continuous integration, tracking cost and latency alongside quality, catching regressions when a provider updates a model, and keeping a golden set that does not slowly leak into your prompts.

Topics covered

Frequently asked questions

Still unsure about something? Write to misteryjj100@gmail.com and a human answers, usually the same working day.

No. The course starts with thirty examples and shows what that size can and cannot support, then grows the set as your confidence requirements grow.

You build the harness in plain Python with pytest so nothing is hidden, then map it onto existing frameworks in a final lesson if you prefer to adopt one.

Yes. A dedicated module covers retrieval-specific metrics, separating a retrieval failure from a generation failure so you fix the right half.

Checkout is handled on our provider's secure payment page. The moment your payment clears we email your personal access link and access code to the address you used at checkout, and the same link appears in your account library. There is nothing to install and nothing to wait for.

Email misteryjj100@gmail.com within 14 days of your purchase, quote your order number, and we refund the full amount to your original payment method. No form to fill in and no questions about how much of the course you watched.

What students say

Reviews are written by people who bought this course. We publish the critical ones too.

4.0

Rated 4.0 out of 5

Course rating · 2 reviews

Rating distribution

  • 5 stars 0%
  • 4 stars 100%
  • 3 stars 0%
  • 2 stars 0%
  • 1 star 0%
  • AD

    Alina Dobrescu

    ML engineer

    Feb 2026
    Rated 4.0 out of 5

    The variance chapter is the important one

    Being shown that a five-example comparison can flip its verdict on a rerun is the wake-up call most teams need, mine very much included. Not five, though: the continuous integration chapter is written against one particular runner, and adapting it to ours took longer than watching the chapter did.
  • JN

    Jasper Nieuwenhuis

    Data scientist

    Oct 2025
    Rated 4.0 out of 5

    Honest about model-graded evaluation

    It does not pretend a model judge is neutral. Position bias and length bias each get their own treatment, backed by measurements rather than assertions, and that candour is rare enough to justify the money by itself. The advice on writing a rubric is thinner than I hoped, and that happens to be exactly where I am stuck.

Your instructor

LM

Laura Mbeki

Prompt engineer and AI workflow designer

  • 506 students taught
  • 16 courses published
  • 4.3 instructor rating
  • Prompt engineering
  • Prompt libraries
  • AI workflows
  • Prompt evaluation

Laura went independent after seven years of agency work and now designs the prompt libraries that sit behind other people's products. She treats prompting as engineering: versioned prompts, a held-out evaluation set, a regression run before anything ships, and a token budget you have to hit. Her packs are the ones she uses with her own clients — briefing, rewriting, summarising, review — rather than sanitised examples, and each comes with notes on where it fails. She keeps every pack working across ChatGPT, Claude and a small open model, so the technique outlives the model.

See every course by Laura Mbeki
$39 USD

One-time payment · lifetime access

The WisdomCharms dispatch

One useful email a week. No fluff, no spam.

New course releases, discount codes before anyone else, and a short, practical breakdown of one technique — a prompt pattern, a C++ idiom, a TypeScript trick — that you can use the same day.

  • Subscriber-only launch pricing
  • Unsubscribe in one click
  • We never sell your address

By subscribing you agree to our Privacy Policy. Questions? Write to misteryjj100@gmail.com.