Learning Braintrust, on a real app

You've been hired by a dog daycare to make their AI better. This is the path.

Each step is one idea, one markdown file, and one commit. You write the code; the step file tells you what and why. Nothing here is a finished solution you paste blindly — the point is to leave with the habits, not the repo.

Before step 1

1. The app runs (bun run app) and you've read a couple of report cards.
2. .env has NOUS_API_KEY. Get a free Braintrust account and add BRAINTRUST_API_KEY too — step 1 needs it.
3. You've seen the problem: click Rufus, compare his report to the timeline underneath.

Stuck? Ask the tutor

In a Claude Code session in this repo:

/tutor

The tutor figures out where you are, explains the next idea, and checks your work when you say a step is done. Good moments to call it: "what's next", "check my work", "why isn't this showing up in Braintrust", or any time a word like span or scorer stops making sense.

It won't write the step for you. You'll get the explanation and the code to apply yourself, one idea at a time.

How to work through it

Do it all in your copy of the repo, on your own branch, so main stays clean:

git switch -c my-course

(The one-line installer already did this for you.)

Then work through the steps in order, in the same folder. Commit whenever you like — it's your branch.

Getting course updates

New steps land on main. To pull them into your branch:

git switch main && git pull
git switch my-course && git merge main

Starting over

git switch main                    # the untouched app
rm app/data/happytails.db          # forget generated reports; reseeds on next launch

Your my-course branch is still there if you want to go back to it.

The steps

Nine steps, each one leaving you able to show something, not just know it. The last column is the moment you'd put on a screen for your team.

Steps 2 onward are written as you reach them, so each one describes what actually happened in your runs — real scores, real regressions, the false alarm your first scorer raises — rather than a script written in advance.

Where it's going

Steps 1–4 are the core loop and stand alone: see it, score it, keep the hard cases, prove a change helped. Steps 5–6 handle what code can't judge and where human opinion enters. Steps 7–8 are what your colleagues will care about most — prompt changes that aren't deploys, and production that scores itself. Step 9 makes it routine, and ends by mapping all of it back to whatever prompt you're really responsible for.

The words

You'll hit these constantly, and they're easy to confuse.

Logs are what happened. Experiments are what would happen if. They're different screens in Braintrust, and looking in the wrong one is the most common early confusion.

The demo at the end

The point of the course is a working system; the point of the repo is that you can show it. Each step file ends with what you can now demonstrate, and step 9 assembles those into learn/demo.md — a run of show with timings, the commands to type, and the lines to say.

Two things make the demo land, and both are decisions made early:

• Rufus. A quiet, sad little day that the model turns into a social triumph. Everyone in the room sees the problem in five seconds without knowing anything about evals.
• The numbers move. The scores in your demo are real runs from this repo, not slides. The regressions are real too — including the one in step 4 where the fix breaks something else.

The loop you're building toward

1. Log what production does.
2. Find the bad ones.
3. Turn them into a dataset with a notion of correct.
4. Run an experiment — that number is your baseline.
5. Change the prompt, run again, compare.
6. Ship if it went up, and keep the dataset forever, so the next change can't quietly break what you just fixed.

Step 5 is the payoff. Without it, every prompt edit is a coin flip — and the classic failure is fixing one complaint while breaking three things nobody retested.