//pragmatic leaders

AI Judgment for Engineers Making Product Decisions

featured

A seven-week path for engineers, ML practitioners, and technical founders who can build the AI feature but now have to make the product call. Learn to choose the right problem, pick the smallest model that clears the bar, write the prompt like a spec, evaluate against user value, and brief leadership on cost, latency, and safety in plain English.

forEngineers, ML practitioners, and technical founders in flat teams who are being asked to scope and ship an AI feature without a dedicated PM — and who can feel themselves defaulting to benchmark scores, model taste, and build-first habits.
outcomeYou can decide which AI feature is worth building, pick the right model on product evidence rather than benchmark strength, write a prompt the way a PM writes a PRD, build a sharp eval set tied to user value, and brief leadership on cost-per-user, latency, and safety without hiding behind jargon.
7 weeks7 stages24 available items
ask if this path fits →

This path is for the engineer who no longer gets to hide behind benchmark charts. You can read the papers, build the model, and ship the code. The new job is deciding which AI feature deserves to exist, what user problem it solves, and where the smallest acceptable model actually sits.

The sequence is deliberate. Separate product value from model performance first, then force yourself to write the PRD before the prompt, build the eval set before launch, and design the UX around failure before you indulge any architecture debate. The point is not to make you less technical — it is to make the technical skill answer to product evidence.

Finish the path by writing the one-page brief and presenting it out loud. If you can explain the problem, the model rung, the eval logic, the cost and latency budget, and the failure-mode strategy in plain English, you are no longer "the engineer helping PM." You are making the call.

01

Stage 1 — Why model performance is not product value

The first trap is treating the model as the product. Users do not care that the model is technically impressive; they care that a real problem gets solved inside their workflow. Start by deciding whether AI belongs here at all, and what evidence would count as value before you start comparing models.

  1. 1When AI Is the Right Answer (and When It Isn't)manual13 min readChapter 1 is the reset engineers need — use AI for unstructured work, code for structured work, and stop wasting frontier models on problems a rule could have solved.
  2. 2Eval Before Launchmanual15 min readChapter 4 makes the uncomfortable point early — if you do not know how you will judge success before launch, you are still doing research, not product.
  3. 3Linear — AI as a Quiet Utility, Not a Chat Assistantcase14 min readLinear did not ship AI as theatre; it shipped a quiet utility that earned its slot in the workflow. Read this when you feel tempted to confuse visible magic with durable value.
02

Stage 2 — Pick the smallest model that clears the bar

Defaulting to the technically best model is how smart teams destroy their own unit economics. The discipline here is to start at the bottom of the ladder, climb only when the eval set forces you to, and understand the market well enough to know what will be commodity by the time you launch.

  1. 1The Model-Selection Laddermanual12 min readChapter 2 is the central operating rule for this whole path — the job is not to find the smartest model, it is to stop at the cheapest rung that still wins.
  2. 2The 2026 Model Landscapemanual13 min readChapter 12 gives you the board state — which labs are strong where, where open models have caught up, and where paying frontier prices is still justified.
  3. 3Cursor — The AI Code Editor That Competed with GitHubcase6 min readCursor is the clean example of product judgment beating model vanity — a sharp UX wrapped around the right model choice, not a science project hunting originality.
  4. 4DeepSeek — The Week the AI Capex Story Brokecase16 min readDeepSeek is the case you read when someone says cost will sort itself out later. It will not — cost shape is strategy, and the market reprices fast.
03

Stage 3 — Write the PRD before you write the prompt

Engineers often build before they specify. This stage forces the opposite habit. Nail the problem, the user, the edge cases, the failure posture, and the success metric before you start editing prompts or wiring tools into the stack.

  1. 1Writing PRDs That Engineers Readmanual11 min readThis is the discipline most build-first teams skip — turn the fuzzy "we should add AI" impulse into a document that survives contact with stakeholders.
  2. 2Defining the Right Problemmanual11 min readA surprising amount of prompt thrash is just bad problem framing. Fix the problem statement and half the technical debate disappears with it.
  3. 3Prompt Design as Product Designmanual12 min readChapter 3 shows why the prompt is not copywriting — it is product specification at the line-by-line level, and it deserves the same rigor.
04

Stage 4 — Build evals against user value, not lab vanity

PM-grade evaluation does not ask whether the model is generally good. It asks whether this feature helps this user complete this job with acceptable failure. Build the golden set, tie it to product metrics, and use adoption as evidence rather than vibes.

  1. 1Eval Before Launchmanual15 min readReturn to Chapter 4 here and do the actual work — a small, sharp eval set that catches regressions before users do.
  2. 2Metrics That Mattermanual12 min readThe point is not more dashboards. It is choosing the handful of product numbers that tell you whether the feature is helping, ignored, or actively eroding trust.
  3. 3GitHub Copilot — The First Real AI Product, and What Five Years Taught Uscase14 min readCopilot is what adoption looks like when the product clears the bar in the real world — not in a benchmark table, but in the daily habits of working engineers.
05

Stage 5 — Design the UX and architecture for failure modes

Once the model is in the loop, failure is part of the product. Hallucination, abstention, tool calls, retrieval, and multi-step flows are not backend details; they are the user experience. Design them that way.

  1. 1Hallucination as a Product Problemmanual15 min readChapter 5 is the harsh but useful correction — you do not "solve" hallucination, you design around it so users know when to trust, verify, or walk away.
  2. 2AI UX Patterns That Workmanual13 min readChapter 8 gives you the interaction patterns that make AI features usable instead of merely impressive — citations, regenerate, confidence cues, undo, and clear exits.
  3. 3Tool Use, Function Calling, Agents — The Maturity Laddermanual12 min readChapter 6 stops you from jumping straight to agents because it sounds advanced. Walk the ladder and earn the complexity.
  4. 4RAG, Fine-Tune, or Context Window?manual16 min readChapter 7 is the architecture decision most teams make with vibes. Use it to choose the most reversible option first and keep your future self out of a hole.
06

Stage 6 — Explain cost, latency, and safety like a product owner

If you cannot explain the economics and risk posture of the feature to a non-technical CEO, you do not own the decision yet. This stage turns engineering trade-offs into leadership language: cost-per-user, p95 latency, safety controls, and what the company is buying for the spend.

  1. 1Cost & Latency as First-Class Product Constraintsmanual13 min readChapter 9 makes the unit economics impossible to dodge — every extra model rung has a price, and every extra second on the clock changes the product.
  2. 2Safety, Privacy, Compliance for Shipping Teamsmanual15 min readSafety is not a policy appendix. It is the part of the brief that keeps a useful feature from turning into a board meeting.
  3. 3Presenting to Leadershipmanual8 min readThis is how you stop sounding like you are asking for technical permission and start sounding like you are making a business case under uncertainty.
07

Capstone — Deliver the one-page AI feature brief

The path ends when you can produce a one-page AI feature brief that covers the problem, the smallest model that clears the bar, the eval set, the cost and latency budget, and the UX strategy for failure modes. If the CEO can read it in five minutes and make the call with you, you have crossed from engineer-who-built-it to operator-who-owns-it.

  1. 1Writing PRDs That Engineers Readmanual11 min readUse the PRD discipline as the skeleton so the brief starts with the problem and the decision, not the model and the demo.
  2. 2The Model-Selection Laddermanual12 min readIf your brief cannot defend why you stopped on this rung, you have not finished the hardest product decision in the room.
  3. 3Eval Before Launchmanual15 min readThe one-pager is only real when the eval set is concrete enough that another engineer could run it tomorrow and challenge your claim.
  4. 4Presenting to Leadershipmanual8 min readThe final test is verbal — explain the trade-off in plain English, with numbers, until a non-technical leader can repeat it back to you cleanly.