Defend a counterintuitive discovery call
With a skeptical design and engineering trio
Separate signal from noise and earn agreement on the next test.
Usage data and interviews appear to point in different directions.
Start practice
Recommend killing a visible bet
With an executive sponsor whose reputation is attached to the initiative
Make a clear stop-or-pivot recommendation while preserving trust.
The programme has executive visibility and two quarters of sunk cost.
The tool failed, but the model kept going
Leadership Judgment · Medium · 8 min
The learner must design how tool failures are surfaced so the agent stops pretending success when a downstream system is broken.
Practice
Too many tools, not enough signal
Leadership Judgment · Hard · 8 min
The learner must decide whether to simplify the tool set, tighten tool guidance, or run a targeted eval loop when the model keeps choosing the wrong function.
Useful enough to buy, risky enough to pause
The learner must design a consent gate for a purchase-capable tool so the agent can act without smuggling in authorization.
Think every step or plan up front
The learner must choose between a ReAct loop and a planner-executor workflow for an agent task with real latency and reliability constraints.
Wrong answer, no clue why
The learner must scope the smallest observability layer that will make a single-agent workflow debuggable without turning into an infrastructure side quest.
The agent will not stop trying
The learner must design a halting rule for a single-agent loop that keeps spending money against an unavailable dependency.
Regulator Asked for the Decision Trail
Customer Judgment · Medium · 8 min
Compliance needs a complete customer-level model decision history, but the product has no audit-ready event model.
Redact PII Without Killing Signal
Product Strategy · Medium · 8 min
QA logging captures raw prompts with account numbers, and the team must design redaction that still preserves useful evaluation data.
The Forwarded Email That Rewrote Your Agent
An injected instruction inside forwarded customer email content overrode the assistant's behavior, exposing weak defense layers.
The model cited the wrong policy because the source was wrong
People Leadership · Hard · 8 min
An outdated SOP keeps getting retrieved and cited, forcing a decision about whether the right fix is curation, evals, UI surfacing, or refusal.
The chunker got smarter and retrieval got worse
A header-aware chunking change looks better in demos but drops retrieval@5 by eight points, forcing a ship, rollback, or hybrid call.
Should long context kill the retrieval stack?
A new million-token model makes the team want to stuff all the docs into the prompt and delete RAG, forcing a call on cost, latency, freshness, and answer quality.
Two prompt diffs, zero review system
People Leadership · Medium · 8 min
Conflicting prompt edits shipped in the same week, exposing that the team has no lightweight review process for behavior-changing prompt changes.
The 800-word prompt nobody will touch
A critical prompt has become so bloated and superstitious that nobody on the team can safely edit it.
When examples start replacing the spec
The team keeps adding more few-shot examples to fix edge cases, and token cost is creeping up without a clear rule for when to stop.
What should one agent hand to the next
The learner must choose between shared memory, a structured handoff payload, or a full-context dump when two agents need to pass state.
Four agents for a three-step job
The learner must decide when to down-scope an overbuilt multi-agent design into a simpler workflow.
Two agents, two opposite answers
The learner must design an arbitration path when specialized agents return contradictory conclusions on the same input.
One vendor on the help-centre critical path
The team is about to make a single model provider the only dependency for a support agent that now sits on a customer-critical path.
The complaints hiding in the long tail
A lightweight model looks fine in the dashboard but is failing a small, valuable intent class that only surfaced through user complaints.
The cheap task riding on the expensive model
A team is burning margin on a low-complexity classification job because nobody wants to be the person who steps down from the frontier model.
Nine MCP Servers, Eighty Tools, and a Confused Agent
You must design an MCP server consolidation policy after tool pick accuracy collapses under uncontrolled server sprawl.
Publish Your MCP Server or Keep the Moat?
You must choose whether to release your MCP server publicly for ecosystem growth or keep it private for control, monetization, and security.
Your MCP Query Tool Knows Too Much
An internal MCP database tool has no per-user scoping, and you must define the auth boundary before sensitive data leaks.
Right Answer, Fragile Reasoning
The model passes answer-key evals while using inconsistent logic, forcing the team to decide whether reasoning traces should be graded.
The Refund Policy The Bot Invented
Stakeholder Management · Medium · 8 min
A support assistant promised a refund path that does not exist, and now legal, support, and product are all pulling in different directions.
The Citation That Never Existed
A research copilot keeps producing plausible-looking URLs that do not resolve, and the team must choose where to enforce source truth.
Good answers hiding bad retrieval
The team is grading only final answers in a retrieval-augmented generation system, and the model is masking a weak retrieval layer.
The metric win with a hidden regression
A prompt change improved one important metric and quietly harmed another, forcing a real ship-or-hold decision.
Launch week with no golden set
Leadership wants to ship next week, but the team has been relying on demos and Slack anecdotes instead of a real evaluation set.
Pure vector search keeps missing the exact SKU
Identifier-heavy lookups are failing in semantic search, forcing a call on whether to add BM25 plus vector hybrid search with RRF without turning the stack into a science project.
The vector store MVP choice became a budget line
A team that picked Pinecone for speed now faces a meaningful annual bill and must decide whether moving to Qdrant is worth the migration and operational burden.
The benchmark says upgrade the embedding model
A new embedding model wins public benchmarks, but migration means a full re-index and downtime, forcing a ship, defer, or shadow-run decision.
Viral Growth, 22x Cost Spike
A sudden social-driven traffic surge exploded model spend, and the team must introduce spend controls without stalling momentum.
38% Repeat Queries, 4% Cache Hits
Traffic shows high repetition but the current cache barely helps, forcing a redesign of match strategy and eviction policy.
The 4-Second p95 Ceiling
Users love answer quality but churn on response lag, forcing the team to choose concrete latency wins instead of generic optimization talk.
Should personalization live in the prompt or in the model?
A team wants deeper user-specific behavior and must choose between per-user adapters and retrieval-augmented user context while weighing quality, cost, privacy, and operational complexity.
The answer is wrong even though the right chunk is in the prompt
Long prompts are missing facts buried in the middle, forcing a decision on reranking, prompt structure, or shrinking the context.
The agent remembers everything and the bill is exploding
A conversational agent has been stuffing every prior turn into context, pushing token cost up fourfold and forcing a summarization-versus-eviction policy decision.
Your AI Agent Wants Merge Rights, But Your Pager Pays the Bill
Your team has strong agent output quality but meaningful silent failures, and you must design a trust-budget gate before granting autonomous merges.
Vendor Says 80, Your Codebase Says 22
You must decide whether to pause, narrow, or invest when benchmark claims and internal autonomous-agent performance diverge sharply.
One Wrong Click Can Onboard the Wrong Customer
Customer Judgment · Hard · 8 min
You must set sandbox and human-in-the-loop policy for a browser agent handling third-party SaaS onboarding flows with high blast radius.
The Model Is Wrong 12% of the Time and Sounds Certain Every Time
You must design how uncertainty is surfaced so users calibrate trust correctly without abandoning the product.
Fast-Feeling Stream or Cleaner Final Draft?
You must choose response delivery mode by use-case instead of defaulting to one UX pattern across all AI tasks.
Users Want to Course-Correct Mid-Flow, Not Start Over
You must choose whether agent corrections should regenerate, edit in place, or branch into alternatives based on artifact type and review cost.
Telling your CEO the v2 she announced is wrong
Stakeholder Management · Hard · 8 min
Practice raising hard truth to a high-conviction founder — with evidence, with a real alternative, without flattery, without panic.
Your strongest IC just told you they're interviewing
Practice the conversation that holds the relationship, hears the real reason, and earns the right to keep them — without sliding into a counter-offer reflex.
Legal has been silent for three months and just blocked launch
Practice surfacing what changed with a stakeholder who has been quiet — and getting to a path forward in 30 minutes without putting them on the defensive.
Your project just became off-strategy. The skip-level wants a 1:1.
Practice positioning your team's work after a company pivot — without pretending it still fits the new strategy and without abandoning the value.
Pitching next year's platform bet to a quarterly leadership team
Practice making the long-bet case to leaders optimizing for the next quarter — without softening the timeline or selling certainty you don't have.
The feature you shipped didn't move the metric
Practice owning a clean miss with your VP — without deflection, without performative humility, and without abandoning the underlying bet.
The designer who won't let go of the redesign
Practice making a hard design judgment with a craft partner you respect — when the data says her six weeks of work didn't beat the prior version.
The angry customer who's actually right
Practice listening past frustration to find the real broken assumption — and committing to one specific fix without over-promising or hiding behind apology.
Your dashboard says success; your researcher says failure
Product Strategy · Hard · 8 min
Practice synthesizing conflicting evidence — a metric that says ship and qualitative research that says don't — and making a recommendation you can defend in either direction.
Defining 'done' with the peer who keeps shipping bugs
Practice setting a shared quality bar with a senior peer whose definition of "done" has been creating customer-visible bugs for two quarters.
The QA report came in red 36 hours before launch
Practice making the ship/slip call when QA finds a P1 edge case the night before launch and your war-room expects an answer in the next ten minutes.
Specing the AI feature your CEO just announced on earnings
Practice scoping an exec-driven feature into a shippable v1 spec without dropping ambition or making promises engineering can't keep.
Giving feedback to an underperforming peer
Practice delivering specific, actionable feedback to a peer whose work is slipping — without damaging the working relationship.
Defending a delayed launch
Practice holding the line on a launch slip without losing trust with a senior partner.