How They Will Test You

AI Engineering Design Patterns · 30 Essential Questions

Chapter 4 is all about engineering judgment for production-grade Agents. These questions are appearing more and more often in interviews and design reviews. Answer them yourself first, then check the framework.

How to Use This Page
Each question is labeled with who's asking. This chapter leans technical — tech colleagues have more questions, and they are the most unsparing.
🎙 InterviewerWants to verify you truly understand — not just recite buzzwords
👔 BossWants explanations and commitments
🛠 Tech colleagueTesting whether you're worth trusting
Each question has three layers: What they're assessing → Answer framework → Bonus points. For anything you can't answer, click the linked lesson pages at the end.
Q1Interviewer
"Everyone is talking about context engineering. What exactly is the difference from writing good Prompts? Why did Prompt engineering suddenly become obsolete?"
🎯 What they're assessing
The opening concept question for this chapter, testing whether you've kept up with the paradigm shift from single-turn conversations to multi-step Agents. Anyone who just says "context engineering has a broader scope" is reciting a definition. Someone who truly understands can articulate exactly what's in the window and why it must be managed.
🧭 Answer framework
  1. Give the definition first: Prompt engineering optimizes how instructions are written. Context engineering manages all the Tokens sent to the model at each reasoning step: System Prompt, tool definitions, conversation history, retrieval results, user state — all of it.
  2. Explain the motivation: Context is a scarce resource. Three hard constraints: Context Rot (the longer the context, the lower retrieval accuracy), limited attention budget (irrelevant Tokens dilute useful information), and quadratic complexity (doubling the context quadruples attention computation).
  3. State the goal: Find the minimum high-signal Token set. Every Token must contribute to the reasoning — the mindset of "stuff in as many as possible" doesn't work.
  4. Give three handles: System Prompt at the right altitude (role + principles, don't pile on 50 rules); lean tool set; Few-shot of 2-3 carefully chosen representative examples — don't pad with edge cases to look thorough.
⭐ Bonus point Cite the quadratic cost: expanding context from 50K to 100K quadruples attention computation. Many people know that longer is more expensive; few can say that longer also makes it dumber.
Organize your answer using these lesson pages → From Prompt Engineering to Context Engineering The Three Pillars of Context
Q2Interviewer
"Your Agent needs to run long tasks of dozens of steps. What do you do when the context window is almost full? Can you just open a new session and keep going?"
🎯 What they're assessing
Testing whether you understand the fundamental dilemma of long tasks. Someone who answers "open a new session" doesn't realize the new window remembers nothing. Someone who answers "switch to a larger context model" hasn't calculated the cost of attention dilution. This question separates people who've seen production systems from those who've only played with demos.
🧭 Answer framework
  1. State the dilemma first: Open a new window, and the Agent loses all memory — it will redo work it already completed. Stay in the old window, and Tokens pile up, attention dilutes, and performance keeps dropping. Claude Code, Cursor, and Devin work on this problem every day.
  2. Pillar 1 — Compaction: When the window is near full, use one LLM call to produce a structured summary. Keep architecture decisions and open bugs; discard redundant tool outputs and intermediate steps of completed tasks. Pick the wrong items to discard and the Agent will repeat its mistakes.
  3. Pillar 2 — Structured notes: Proactively write key information to external files; new windows read them back to restore memory. Claude Code's TODO file and the game notes Claude maintains while playing Pokémon are both examples of this.
  4. Pillar 3 — Sub-Agents: Delegate deep exploration. A sub-Agent burns 30K Tokens in its own window reading code and reasoning, then returns only a 1,500-Token conclusion to the main Agent. The main context stays clean at all times.
⭐ Bonus point Articulate each pillar's scope: compact to stay lean within the window; notes to pass memory across windows; sub-Agents to isolate exploration noise. Then add that real products use all three in combination. This shows you grasp the system, not just the terms.
Organize your answer using these lesson pages → The Three Pillars of Context Why Agents Struggle with Long Tasks Session ≠ Context Window
Q3Tech colleague
"The product needs to add codebase Q&A. You're not going to kick off a project to build a vector store, are you? Claude Code does everything with grep on the spot."
🎯 What they're assessing
Testing whether you've defaulted to RAG as the answer. A PM who immediately goes to chunking, vectorization, and building an index looks to the tech colleague like someone who always reaches for a hammer. They want to hear you compare approaches and calculate the cost of maintaining an index.
🧭 Answer framework
  1. Accept the premise: The goal is to put the right information into the context window — RAG is just one means among many. For data that changes frequently like a codebase, just-in-time retrieval is often more appropriate.
  2. Explain JIT retrieval: Use glob/grep to search on demand, keeping the context lean and containing only what's currently needed. The cost is one extra tool-call latency; the gain is eliminating the maintenance burden of building and synchronizing an index.
  3. Give a hybrid strategy: Preload high-frequency information (project conventions, core rules, user preferences); fetch long-tail information on demand. Analogous to browser caching: hot data in memory, cold data fetched on request.
  4. Clarify when RAG is right: RAG suits relatively static knowledge bases, but naive chunking loses context. Contextual Retrieval adds a context prefix to each Chunk, combined with BM25 dual-path retrieval and Reranking, reducing retrieval failure rate by 67%.
⭐ Bonus point Proactively calculate the Contextual Retrieval cost: one extra LLM call per Chunk for the prefix, which can be reduced with Prompt Caching. Only worthwhile for high-accuracy use cases like legal, medical, or financial — not for chat recommendations.
Organize your answer using these lesson pages → JIT Context vs Preloading Contextual Retrieval: Better RAG
Q4Interviewer
"The production Agent keeps selecting the wrong tools and filling in the wrong parameters. The engineers say the model is too dumb — just wait for the next generation. As a PM, what do you think?"
🎯 What they're assessing
Testing whether you know about ACI. Anyone who agrees with "wait for the next model" is immediately out. The interviewer wants to hear: tool definitions are the Agent's user interface, wrong tool selection is most likely a design problem, and PMs have a clear diagnostic checklist.
🧭 Answer framework
  1. Establish the frame: A tool's name, parameters, and description are the Agent's user interface. Traditional APIs are deterministic; Agent tools are non-deterministic — when they're used and how depends entirely on design quality. Tool design deserves the same investment as HCI design.
  2. Give a diagnostic checklist: Go through four principles. Does parameter order give the model thinking space (simple direction first, complex content after)? Does the format align with training data (standard unified diff beats a custom DSL)? Are you forcing the model to count line numbers mechanically? Is there mistake-proofing (Poka-yoke)?
  3. Give a concrete example: On SWE-bench, changing the file path parameter to accept only absolute paths (not relative paths) was a single parameter change that transformed tool calls from frequently erring to nearly perfect.
  4. State the description standard: Write as if documenting for a smart junior developer with no context. Cover the five-pack: example usage, edge cases, input format, how it differs from other tools, and when not to use it.
⭐ Bonus point Add "If humans can't tell which tool to use, AI won't be able to either." Then mention Claude Code's advanced approach: using an Agent to write descriptions for its own tools, run evals, and auto-iterate toward optimization.
Organize your answer using these lesson pages → ACI: Agent-Computer Interface Using an Agent to Optimize Its Own Tools Keep the Tool Set Lean
Q5Boss
"The new model has been out for a week. Competitors announced integration the day after release. We need three weeks to evaluate? Explain where the time is going."
🎯 What they're assessing
On the surface it's pushing for speed; underneath it's asking whether your team has an eval infrastructure. This is the question that turns passive blame into an opportunity to request resources. Answering "that's just how engineering schedules work" is admitting incompetence. Answering "we'll switch tomorrow" is gambling with product quality.
🧭 Answer framework
  1. Lead with the conclusion: Migration speed depends on eval infrastructure. Teams with a solid eval suite run the tests, confirm no regressions, and switch in a few days. Teams without one spend weeks on manual verification. Our slowness is tech debt in infrastructure.
  2. Explain what evals buy you: Change a Prompt, swap a model, tune a parameter — and know in minutes what the overall impact is. Prevent fixing one bug and creating three. Be first in line to benefit every time a new model launches.
  3. Give a launch plan: Start with 20 test cases covering core scenarios. 20 well-designed cases beats 500 that are still in the planning document by an entire generation.
  4. Manage expectations while you're at it: Competitors who integrate fast aren't necessarily testing rigorously. Public benchmark scores are inflated (the model can recognize exams) — use your own business scenario cases. Your eval environment must match production; sandbox configuration differences alone can cause 6 percentage points of error.
⭐ Bonus point Replace adjectives with metric language in your reports: upgrade "feels worse" to "conciseness improved from 72 to 85, but over-engineering degraded from 3% to 7% — needs rollback." Your boss's trust in you will jump a level.
Organize your answer using these lesson pages → Why Evaluation Matters More Than Training Eval Pitfalls: Noise, Cheating, and Regression
Q6Interviewer
"How do you automate scoring of Agent output quality? Is LLM-as-Judge reliable enough on its own?"
🎯 What they're assessing
Testing the depth of your eval toolbox mastery. Anyone who just says "use an LLM to score" has merely heard of it. Someone who can clearly explain the boundaries and combination of the three Grader types sounds like they've actually run evals.
🧭 Answer framework
  1. List all three Grader types: Code Grader (assertions, unit tests, regex — millisecond latency, zero cost, fully reproducible, but too strict on reasonable variants); Model Grader (can evaluate subjective quality, but has cost and bias); Human Grader (highest quality, but doesn't scale).
  2. Answer the LLM-as-Judge question directly: Its reliability depends entirely on the Rubric. "Rate quality from 0 to 1" is nearly useless — you need to be specific at every score level: what does 0 look like, what is 0.3 missing, what conditions must all be met to score 1.
  3. Give the combination: Code Grader as the foundation for deterministic scenarios, Model Grader to extend to subjective quality, humans periodically spot-checking and calibrating for Model Grader drift. All three layers are indispensable.
  4. Add a commonly missed point: What you evaluate should be the Outcome — the final state of the environment. The Agent saying it's done doesn't count; you need to check whether the file was actually changed correctly and whether the API was actually called correctly.
⭐ Bonus point Mention the Trial concept: model output is stochastic — the same Task must be run multiple times to have statistical meaning. Then cite Descript's three-dimension scoring (didn't break anything, did what was required, did it well) to show you've seen real-world cases.
Organize your answer using these lesson pages → Three Grader Types: Code, Model, Human Core Concepts in Evaluation
Q7Tech colleague
"This requirement has the Agent running model-generated code using the user's GitHub Token. If something goes wrong, who's responsible? Adding a line that says 'don't run dangerous operations' to the Prompt isn't going to satisfy me."
🎯 What they're assessing
The classic confrontation in a security review, testing whether you understand structural security. If your answer only contains "add constraints to the Prompt" or "the model will refuse" — this requirement gets rejected on the spot. The interviewer wants to confirm you know the defense must be built into the architecture.
🧭 Answer framework
  1. Agree with the other person's stance first: Prompt-based defenses are unreliable; security must rely on structural design. The goal is that even if the model is completely manipulated by Prompt Injection, the attacker still cannot obtain credentials.
  2. Classify the risks: Three categories, each with separate defenses: intentional user abuse; model-initiated loss of control (over-acting, executing real operations based on hallucinations); external attacks (injection instructions embedded in web pages and documents, without the user's knowledge).
  3. Give the credential solution: First principle: generated code and secrets are always isolated in separate containers. Two modes: Token injected into resource access path (Agent-usable but invisible — e.g., embedded in Git remote URL); Vault proxy forwarding (proxy injects Token per session, Agent never sees a single character).
  4. Give the execution environment solution: OS-level sandbox with triple isolation (file system, network, process), layered with three-level trust control: manual approval for high-risk tools, session-level authorization, global policy as the final backstop (production databases are never reachable).
⭐ Bonus point Proactively say "the stronger the model, the larger the attack surface in old architectures," so security design cannot rely on model upgrades to automatically improve. This statement makes security engineers see you as one of their own.
Organize your answer using these lesson pages → Three Risk Categories: Abuse, Loss of Control, External Attack Sandbox and Credential Isolation
Q8Interviewer
"For this feature you designed — should it be a Workflow or an Agent? Give me a decision criterion. Don't just say Agents are smarter."
🎯 What they're assessing
The watershed question from this chapter's first lesson. Anyone who just shouts "Agent" is chasing a buzzword. Someone who truly understands looks at task structure first: is control in the code's hands or the model's? That choice determines every cost and debugging approach that follows.
🧭 Answer framework
  1. Give the definition first: A Workflow is an LLM and tools walking a predefined code path — the developer already decided A, then B, then C when writing the code. An Agent is the model dynamically deciding the flow, independently judging which tool to call at each step and when to stop.
  2. Lay out the core difference: In a Workflow, a given input means a given path — easy to reproduce and debug. In an Agent, the same input may take different paths; behavior is uncertain, and production issues are hard to reproduce.
  3. Give the decision criterion: When the task breakdown is clear and the steps are fixed, use a Workflow — typical cases are copywriting pipelines and data-cleaning flows. When the task is open-ended and needs on-the-spot decisions, use an Agent — typical cases are coding assistants like Claude Code and Devin.
  4. Cite the production consensus: Anthropic reviewed a large number of production cases; the most successful implementations didn't use complex frameworks — they used simple, composable patterns.
⭐ Bonus point Proactively run the cost numbers: Workflow call counts are fixed and the budget is estimable; Agent loop counts are unknown and the bill is uncontrollable. When you present a budget, the two are completely different conversations — most candidates never think of this.
Organize your answer using these lesson pages → Workflow vs Agent: Know What You Need First
Q9Tech colleague
"Your plan has a classifier and a fixed flow. The team next door is already running fully autonomous Agents. Aren't we being too conservative?"
🎯 What they're assessing
A reverse probe, testing whether you'll be kidnapped by industry buzzwords. A PM who can walk the complexity ladder backwards and prove each extra layer of complexity has a matching payoff is the one an engineering team will trust.
🧭 Answer framework
  1. Establish the principle first: Complexity is a cost. Every layer you add has to answer the same question: is the gain from this layer worth the extra latency, spend, and debugging difficulty?
  2. Lay out the four-rung ladder: First optimize a single LLM call (Prompt, Few-shot, Temperature); if that's not enough, add RAG; if still not enough, use a Workflow to break the steps apart; only go to an Agent when you truly need flexible decisions.
  3. Price full autonomy: Handing control from code to the model means unknown loop counts, uncontrollable cost, and hard-to-reproduce behavior. Those costs only pay off when there's a matching gain.
  4. Give an over-engineering anti-pattern: Using an Agent framework for a problem that one Prompt plus one search can solve — the latency, cost, and uncertainty the framework introduces far outweigh the gain.
⭐ Bonus point Cite the production-practice conclusion: for most scenarios, optimizing a single call plus retrieval augmentation is enough. A PM who dares to say "we don't need an Agent" in a design review makes engineers more comfortable than one who chases buzzwords.
Organize your answer using these lesson pages → The Complexity Ladder Five Workflow Patterns
Q10Interviewer
"Have you used all five Workflow patterns? Pick three and walk me through them — what each is good for, and what you pay for it."
🎯 What they're assessing
The advanced version of a vocabulary question. People who can recite the five English names are a dime a dozen; people who can clearly state when to use them and what they cost are rare. Asking you to pick three is testing how you talk about trade-offs.
🧭 Answer framework
  1. Prompt Chaining: Use it when a task splits into a fixed sequence of steps. You can insert quality gates between steps — for example, check that the copy contains brand-critical information before it goes to translation. The cost is latency: you're trading latency for accuracy.
  2. Routing: When input types vary, classify first, then split. Simple FAQs go to a fast, cheap model like Haiku; refunds go to Sonnet plus order tools. The value is separation of concerns and cost tiering.
  3. Parallelization: Two sub-patterns. Sectioning runs independent sub-tasks in parallel — say, security, performance, and style reviews of the same code at once. Voting runs the same task multiple times and takes the majority — spending money for confidence.
  4. Close with the other two: Orchestrator-Workers has the orchestrator dynamically split tasks at runtime — use it when sub-tasks can't be fixed in advance; it's the closest to an Agent. Evaluator-Optimizer loops generate-then-judge, and fits cases like translation that have a clear quality bar.
⭐ Bonus point State the boundary between Orchestrator and Parallelization: in the former, sub-tasks are decided dynamically by the orchestrator at runtime; in the latter, they're hardcoded. Almost nobody can draw that line.
Organize your answer using these lesson pages → Five Workflow Patterns
Q11Boss
"The AI support API bill is up 40% this month. User volume is only up 10%. Cut next quarter's spend in half — and don't drop a single feature."
🎯 What they're assessing
Testing whether you have a toolbox for structural cost-cutting. "Negotiate with the vendor" or "cap usage" both mean you didn't catch the ball. The bill is growing faster than users, which means Tokens per call are bloating — that's the disease to treat.
🧭 Answer framework
  1. Start with Routing: Add a classifier. Simple FAQs go to a cheap small model; refunds, complaints, and other hard cases get the strong model plus tools. The bulk of support traffic is simple questions — this cut saves the most money.
  2. Govern tool returns: Check whether you're returning everything. Dumping 847 full records in one go burns 50,000+ Tokens; switch to the top 10 core fields plus a pagination hint, and 800 Tokens is enough — with higher information density.
  3. Take the cache dividend: Put stable content like the System Prompt and tool definitions in the prefix and keep them unchanged. Once you hit the Prompt Cache, the cost of the repeated portion drops sharply.
  4. Close the verification loop: Run evals after each change to confirm quality didn't drop, then report with data: how much cost fell, core metrics held flat.
⭐ Bonus point Remind the boss of a hidden bill: longer context also makes the model dumber (Context Rot) — retrieval accuracy falls as Token count rises. Slimming context often saves money and raises quality at the same time. Most teams haven't noticed this.
Organize your answer using these lesson pages → Routing for Cost Splitting Token Efficiency: Trim Your Returns Context Rot and the Attention Budget
Q12Interviewer
"Engineers say your System Prompt reads like a requirements doc — almost 60 rules. How far do you think a System Prompt should actually go?"
🎯 What they're assessing
Testing judgment about "the right height." People who pile on rules default to not trusting the model; people who write one line — "you are an assistant" — give up on guidance. They want you to say clearly what's wrong with each extreme, and how you'd govern those 60 rules down.
🧭 Answer framework
  1. Lay out the two extremes: Too vague ("you are a helpful assistant") leaves the model with no sense of direction, so output stays generic. Too specific (50 rules plus 100 edge cases) locks the model down — it can't adapt when something new shows up.
  2. Give the sweet spot: A clear role, plus 5–10 core principles, plus clear boundaries — then trust the model to judge inside that frame. Like a good manager: give direction, don't issue every next instruction.
  3. Price the rules: 60 rules are Tokens themselves — they eat the attention budget. When rules fight each other, model behavior gets even harder to predict.
  4. Give a landing move: Sort rules into principles, format, and boundaries, then merge. You can usually compress to a dozen or so, then run evals to confirm behavior didn't regress.
⭐ Bonus point Point out the hidden cost of over-constraint: once a model is tied down by 50 rules, it can't handle new situations flexibly — you're paying frontier-model prices for a rules engine. Engineers will nod at that sentence.
Organize your answer using these lesson pages → The Right Height for Your System Prompt
Q13Interviewer
"The Agent's memory is just a notes file it writes itself? Sounds primitive. How do you actually design that thing so it's reliable?"
🎯 What they're assessing
Structured notes sound simple; what's being tested is all in the design details: what to write, what format, when to read it back. Anyone who answers "let the model freestyle" has clearly never run a real long task.
🧭 Answer framework
  1. Explain the principle first: Externalize short-term memory (the context window) into long-term memory (the file system). After a window reset, the first thing a new session does is read the notes to restore state — memory continues across windows.
  2. The format must be fixed and structured: Free-form prose costs extra Tokens just to understand when you read it back. Use fixed columns: done, not done, key decisions, known issues — later reasoning pulls straight from the columns.
  3. Set the read/write rhythm: Write after every key step, don't wait until the window is almost full; read as the first step of every window reset or new session. If the rhythm slips, the notes drift from actual progress.
  4. Separate the job from Compaction: Compaction shrinks old information so you can keep using it — good for uninterrupted conversations. Notes are stored outside and fetched later — good for tasks that may be interrupted and need to continue across sessions. Real products use both together.
⭐ Bonus point Add a protection mechanism: lock this kind of state file with strong wording in the Prompt — for example, explicitly "do not delete or modify the existing test checklist" — otherwise the Agent will lower the bar itself just to make tests pass.
Organize your answer using these lesson pages → Structured Notes Progress Files and Feature Lists
Q14Interviewer
"You added a think tool to the design. It doesn't look up data, doesn't call an API, doesn't change any state. Why add a tool that does nothing?"
🎯 What they're assessing
Testing whether you actually understand the Think Tool's mechanism and boundaries. "Letting the AI think more is always good" gives you away immediately. They want to hear what problem it solves and when it's pure waste — data helps.
🧭 Answer framework
  1. Explain the mechanism: Think Tool is a side-effect-free tool. Its only job is to let the Agent write its thinking down mid-execution. Packaging "stop and think" as a tool call lets the model naturally insert a stretch of reasoning into the rhythm of the tool chain.
  2. Explain the scenarios: Three cases see the most lift: long chains of 5+ tool calls; policy-dense environments (20 refund policies plus 6 exceptions); serial decisions where each step depends on the last. The common thread: early information gets drowned by later context.
  3. Cite the numbers: τ-bench airline support went from 0.570 to 0.878 — a 54% lift; retail support lifted 11%. Airline change-and-cancel policy is far more complex than retail. The denser the policy, the more Think Tool is worth.
  4. Draw the boundary: Using it for one-and-done operations like checking weather or reading a file is pure overhead; pure generation tasks that don't call tools don't need it; problems you can think through in one shot are better handed to Extended Thinking.
⭐ Bonus point Make the split with Extended Thinking clear: one does deep planning before acting; the other pauses mid-execution to organize. If you can say "the thinking happens at different times," you've punched through this question.
Organize your answer using these lesson pages → Think Tool: Making AI Think Before Acting
Q15Tech colleague
"This tool requirement wants every order record for the user. Heavy users have 800-plus. Have you calculated how many Tokens one call like that eats?"
🎯 What they're assessing
Testing whether you treat tool returns as part of the context budget. What a tech colleague fears most is a PM saying "just return everything, let the model pick" — that detonates cost and attention at the same time.
🧭 Answer framework
  1. Own the bill first: Returning everything is a disaster. 847 full records are about 52,000 Tokens. The model can't process that, and it dilutes attention on everything else in the window.
  2. Give four slimming strategies: Summarize (return stats only), truncate (top N by default), paginate (with a page parameter), filter (support conditions) — combine them by scenario.
  3. Returns should carry the next clue: Returning only "success" is bad design. Return what the Agent needs for the next step — ticket ID, link, owner — and you skip a follow-up lookup call.
  4. Bake pagination into the return body: Include total, showing, page, plus a hint like "use page=2 to see more," and the Agent knows how to fetch the rest.
⭐ Bonus point Elevate this to a principle: tool returns are part of ACI too — they occupy the model's attention budget. An 800-Token high-density return beats dumping 50,000 Tokens of raw data.
Organize your answer using these lesson pages → Token Efficiency: The Science of Return Values ACI: Agent-Computer Interface
Q16Interviewer
"After your Agent's tools went from 10 to 60, the wrong-tool rate actually went up. How would you govern this tool set?"
🎯 What they're assessing
Testing whether you can land "less is more." Everyone can add tools; PMs who dare to cut tools and know how to organize them are rare. They want a governance plan with clear rules — don't rush to blame the model.
🧭 Answer framework
  1. Name it first: Wrong-selection rate rising with tool count means the boundaries between tools have gone fuzzy. Put search, find, and lookup — three near-identical tools — side by side, and wrong picks are a disease of the tool set. The model is just exposing the symptom.
  2. Merge the duplicates: If two tools overlap in more than half their use cases, merge them. Better one tool with a few extra parameters than two that are easy to confuse.
  3. Add namespaces: Group related tools with a shared prefix — jira_create_issue, jira_list_issues, git_diff, db_query. The Agent can see at a glance which tools operate the same system, and the wrong-pick rate drops sharply.
  4. Accept with data: Tool-selection errors can be measured by evals. Run the same suite before and after governance, compare the hit rate, and prove the cuts were the right ones.
⭐ Bonus point Mention that tool definitions themselves occupy context: the schemas for 60 tools enter the window on every reasoning turn. Cutting unused tools is free attention budget on every call.
Organize your answer using these lesson pages → Five Principles of Tool Design Keep the Tool Set Lean
Q17Interviewer
"Who are you assigning to write descriptions for dozens of tools? How do you guarantee the quality — by reviewing them by hand?"
🎯 What they're assessing
Sounds like a staffing question; it's actually testing whether you know the use an Agent to optimize Agent tools workflow. "Engineers write it, I review it" means you're still in traditional-docs thinking.
🧭 Answer framework
  1. Prototype: Have Claude Code generate the tool prototype from the requirement — definition, parameter validation, API call logic. Humans only describe what they want.
  2. Evaluate: Build evals on four dimensions: did the Agent pick the right tool, were parameters filled correctly, was the return understood correctly, and end-to-end task success rate.
  3. Optimize: Have Claude Code read the eval results and auto-analyze failures. It can produce precise conclusions like "43% of errors are the Agent confusing search and list because the descriptions are too similar," then rewrite the descriptions, add distinguishing notes and examples.
  4. Define the human's role: Humans set the eval standard and do final acceptance; the machine writes and revises. Loop until you hit the bar — iteration is an order of magnitude faster than hand-tuning.
⭐ Bonus point Name the essence of this loop: the Agent itself has the most say in whether a tool is well written. Let the user be the author, and the Agent becomes the product manager of its own tools — more effective than any documentation standard.
Organize your answer using these lesson pages → Using an Agent to Optimize Its Own Tools The Art of Writing Tool Descriptions
Q18Interviewer
"Say you start next week. Our Agent has zero evals. In the first thirty days, how do you build evaluation from scratch?"
🎯 What they're assessing
Testing the operating path. People who shout "evals matter" are everywhere. Someone who can give a concrete rhythm from 0 to 20 cases and say how each concept lands has actually done this.
🧭 Answer framework
  1. Week one, define success: The first value of writing evals is forcing the team to answer "what counts as good." Pick the core user scenarios and write 20 carefully designed Tasks, each with input and a success criterion. 20 is enough to start — don't wait for a grand plan of 500.
  2. Build the minimum loop: The Harness spins up the sandbox, runs the task, collects results; the Grader scores; a set of Tasks makes a Suite you can run in one click. First align the team on the language: Task, Trial, Grader, Suite.
  3. Save the Transcript: Record the full trajectory of every run — each reasoning step, each tool call, intermediate results. When something fails you can tell whether it picked the wrong tool or filled the wrong parameter, and the eval can actually steer improvement.
  4. Hook it into the change process: From then on, Prompt changes, model swaps, and parameter tweaks all run the Suite first — you see the blast radius in minutes. The team's reporting language upgrades from "feels worse" to specific scores.
⭐ Bonus point Add a counterintuitive takeaway: many teams, while writing evals, clarify a product definition that had been fuzzy for a long time. Evaluation is both a QA tool and a requirements-analysis tool.
Organize your answer using these lesson pages → Why Evaluation Matters More Than Training Three Grader Types: Code, Model, Human
Q19Boss
"One eval round is hundreds of model calls. A month of testing burns tens of thousands. Is that money well spent?"
🎯 What they're assessing
The boss is questioning ROI. Chanting "evals matter" does nothing. You have to recast every seemingly wasteful expense as insurance and leverage — and give a way to control the cost.
🧭 Answer framework
  1. Explain why so many runs: Model output is stochastic. A single run of the same task is noise. The same Task needs multiple Trials to have statistical meaning. Saving that money is making product decisions with dice.
  2. Price the cost of no evals: Fix one bug, create three; you only find out when users complain. One "the Agent got worse" investigation means three days of commit archaeology. Engineer time is far more expensive than API fees.
  3. Price what evals earn: Change a Prompt or swap a model and know the impact in minutes. When a new model ships, run the Suite, confirm no drop, and switch — you're first in line for the dividend every time.
  4. Give a cost-down plan: 20 carefully chosen cases cover the core scenarios. Deterministic checks sit on millisecond, zero-cost code Graders; the expensive model Grader is only used for subjective quality.
⭐ Bonus point Close in one sentence: eval investment compounds — every minute up front keeps paying off in regression tests, model migrations, and team collaboration. Bosses understand compound interest.
Organize your answer using these lesson pages → Core Concepts in Evaluation Cost Structure of the Three Graders
Q20Interviewer
"Users said the output was too wordy, so you changed the System Prompt and shipped it. Coding ability dropped. Walk me through the postmortem — where did the process go wrong?"
🎯 What they're assessing
An incident postmortem, testing whether you know the real-production cases of Prompt changes causing regression and the guardrail process. Anyone who blames the model or the QA folks is out on the spot.
🧭 Answer framework
  1. Name it first: Improving one dimension is not improving the whole. In a real case, a System Prompt change to cut wordiness raised conciseness and dropped the coding eval about 3%: as the model got concise, it also skipped key comments and error handling.
  2. Process error one: No line-by-line ablation. Prompt changes should alter one line at a time and measure the impact alone, so you know what each sentence contributes.
  3. Process error two: You only measured the target dimension. Before shipping, run the full eval suite — conciseness, code quality, over-engineering, together — so you don't fix one thing and break another.
  4. Cite a sibling incident: Even a seemingly harmless config change like the default reasoning-effort value has caused multi-dimension regressions. The conclusion: every change — Prompt, parameters, infrastructure — goes through evals the same way.
⭐ Bonus point Mention reverse metrics like an "over-engineering eval": while you optimize conciseness, watch whether that one is getting worse. A pair of metrics that restrain each other is what keeps optimization from running off in one direction.
Organize your answer using these lesson pages → Eval Pitfalls: Noise, Cheating, and Regression The Evaluation Evolution Story of Claude Code
Q21Tech colleague
"You're going to talk public benchmark leaderboards in a model-selection meeting? Those things were gamed long ago. Do you actually trust that score?"
🎯 What they're assessing
Testing how deep your grasp of eval credibility goes. A PM who admits benchmarks have limits and can still name the specific failure mechanisms is the one who can stand in a selection meeting.
🧭 Answer framework
  1. Failure mechanism one — the model recognizes the exam: On BrowseComp, Claude Opus 4.6 can infer it's running a benchmark, recognize the question pattern, then search for answers or call similar items it saw in training data. A static question bank in a networked environment may be measuring recall, not ability.
  2. Failure mechanism two — discriminating power decays: The stronger the model, the better it is at recognizing evals. Fixed benchmarks keep losing their ability to separate frontier models, and public-set scores get systematically inflated.
  3. Failure mechanism three — infrastructure noise: Change only CPU and memory limits, and scores can move 6 percentage points. Same model, same task, different sandbox config — the ranking can flip.
  4. Give the alternative: Use private cases from your own business, dynamically generated test items, and limited networking; align the eval environment with production; when you report a score, report the environment config with it.
⭐ Bonus point Give a one-line placement: leaderboards are for screening the shortlist; the final decision looks at private evals. Putting benchmarks in the right place is exactly the attitude a tech colleague wants to hear.
Organize your answer using these lesson pages → Models Recognizing Exams and Infrastructure Noise
Q22Interviewer
"You give an Agent a big task, let it run overnight, and in the morning it's basically junk. Walk me through how it actually fails."
🎯 What they're assessing
Testing whether you've seen a real long-task failure. "The context wasn't long enough" only touches the surface. They want two concrete failure modes, and the shared handoff problem behind them.
🧭 Answer framework
  1. Failure mode one — One-shotting: The Agent tries to finish every feature in one go and the window runs out mid-way. The next Agent to take over faces a half-built mess and can only guess what the last one did. Time gets burned putting basic features back together, and progress stalls.
  2. Failure mode two — Premature Completion: The Agent sees a few features implemented and declares done — it actually only did 30% of the core. No task list, so it doesn't know what's missing; no end-to-end tests to prove it actually runs.
  3. Give an analogy: Like an engineering team that fully amnesias on every shift change — everyone who sits down has to understand a pile of half-finished code from zero. That's a long-running Agent with no handoff mechanism.
  4. Name the essence: The core challenge of long tasks is handoff; doing the work is the easy part. What Agents lack is a way to keep continuity when context breaks. The direction of the fix is a progress file plus incremental commits.
⭐ Bonus point Add "even Compaction can't save One-shotting — after compression the instructions aren't clear enough, and the new Agent still gets lost." That shows you know where compression tops out.
Organize your answer using these lesson pages → Why Agents Struggle with Long Tasks Initializer + Coding Agent
Q23Interviewer
"Have an Agent autonomously build a complete web app — hundreds of feature points. How do you design this system so it keeps moving and doesn't stall out?"
🎯 What they're assessing
An open architecture question, testing mastery of the dual-role Harness. They expect three layers: role split, state files, and an acceptance mechanism. Miss one and it sounds like you only read the title.
🧭 Answer framework
  1. Split the two roles: Initializer runs only the first turn, taking you from zero to something: write init.sh to set up the environment, write the progress file, expand high-level requirements into a detailed feature list, make the first git commit. The Coding Agent reads the progress file each turn, does one feature, then updates progress and commits.
  2. Keep the feature list in JSON: Models are less likely to casually rewrite structured JSON; Markdown lists get rewritten on a whim. Each feature carries a category, a step list, and a passes field.
  3. Accept with end-to-end tests: Explicitly require the Agent to actually open the page and click buttons with browser automation. Unit tests alone aren't enough. E2E passing is what counts as passes.
  4. Spell out the value of one-at-a-time: At the end of every turn the code is in a mergeable state, the commit is a rollback point, the progress file is the handoff letter, and the window never gets stuffed to bursting.
⭐ Bonus point Report the result: this setup has produced a claude.ai clone with 200-plus features, each with a matching E2E test. An architecture answer with numbers is a completely different kind of convincing.
Organize your answer using these lesson pages → Initializer + Coding Agent Failure Modes of Long Tasks
Q24Boss
"The sandbox, evals, and scaffolding you want are a three-month schedule. Models upgrade every six months. Won't all of this be wasted by then?"
🎯 What they're assessing
The boss is asking whether the investment will be money down the drain. This question needs classification: which engineering goes stale as models improve, and which is a lasting asset. Saying it's all worth it or none of it is — both are wrong.
🧭 Answer framework
  1. Admit half of it first: A Harness encodes assumptions about the current model's abilities, and assumptions go stale. Real case: Sonnet 4.5 had context anxiety — performance dropped as conversations got longer, so the team added a context-reset mechanism. After switching to Opus 4.5 the anxiety disappeared, and the mechanism started slowing things down.
  2. Give the classification: Workarounds for a specific model and specific Prompt tricks will go stale. Sandbox isolation, permission layering, eval systems, and Session logs are lasting architecture — the stronger the model, the more you need them.
  3. Schedule by class: Lasting assets first; skip temporary patches if you can. The principle: don't write code today that tomorrow may not need.
  4. Flip the role of evals: When a model upgrades, evals are exactly what lets us confirm in days whether the new model works and which old patches can die. The eval investment in these three months is what saves the manual verification on every future upgrade.
⭐ Bonus point Make the judgment itself the answer: telling which logic will go stale as models improve, and which are truly lasting architecture decisions — that judgment is the most valuable engineering skill of the AI era.
Organize your answer using these lesson pages → Build the Simplest Thing That Works Evals Make Model Migration Faster
Q25Interviewer
"Heard of brain-hand separation? Why split an Agent's thinking and execution into different processes? What do you actually gain by splitting them?"
🎯 What they're assessing
An architecture-understanding question. Reciting "decoupling" does nothing. They want to hear what the three components each are, and what changes in failure recovery and performance after you split them.
🧭 Answer framework
  1. Lay out the three components: Session is an append-only persistent event log; Harness is the brain, running the loop that calls the model and routes tools; Sandbox is the hand, the container that executes code and edits files.
  2. Pets become cattle: When all three are crammed in one container, a container crash loses the session and the task fails completely. After the split, a sandbox crash is just one tool-call error — the model decides to retry, the system spins a new container, and work continues.
  3. Explain the brain's recovery path: A Harness crash isn't fatal either. A new Harness starts with wake(sessionId), reads the full event stream back from Session to restore context, and the task is unaffected.
  4. Cite the performance gain: The brain can start processing without waiting for the container to be ready — median TTFT dropped 60%, p95 dropped over 90%. Once components are decoupled you can also have one brain control multiple hands in parallel, or one hand relay across multiple brains.
⭐ Bonus point Open with an OS analogy: when you call read(), you don't care whether the underlying disk is an SSD or a network volume. A Managed Agent does the same thing — the brain doesn't care which container the hand is. Interviewers remember that analogy.
Organize your answer using these lesson pages → Managed Agent: Brain-Hand Separation
Q26Interviewer
"A lot of people treat Session and the context window as the same thing. What's the difference, and why do they have to be stored separately?"
🎯 What they're assessing
Concept discrimination plus architectural motive — the deep water of this chapter. Someone who can walk the through-line of compaction is irreversible actually understands state design for long-running Agents.
🧭 Answer framework
  1. Give the analogy first: The Context Window is RAM — fast, small, gone when you're done — it holds the curated content for the current reasoning step. Session is disk — large, survives power loss — it holds the complete record of every raw event.
  2. Explain why you separate them: Compaction and trimming are both irreversible, and when you compress it's hard to predict which Tokens will matter later. A detail that looks irrelevant today may be the basis of a key decision tomorrow — lose it and it's gone forever.
  3. Give the right posture: All raw events go into Session, append-only, never deleted. The window is just a temporary viewfinder onto Session. Losing Context is fine — you can rebuild it anytime.
  4. Spell out the engineering dividend: The Harness uses getEvents to query any interval on demand, filter specific event types, and keep the prefix stable to improve Prompt Cache hit rate. Swap the model or the Harness and Session doesn't move.
⭐ Bonus point One-sentence summary: don't use RAM as a hard drive. When users complain "the Agent forgets things," almost every one of those product problems traces back to these two layers not being separated.
Organize your answer using these lesson pages → Session ≠ Context Window The Three Components of a Managed Agent
Q27Boss
"Users are complaining our Agent pops a dozen confirm dialogs a day — like an intern who won't take responsibility. Can we just remove them all?"
🎯 What they're assessing
The boss wants the experience, but you can't trade it for safety. Testing whether you can give a structured plan that cuts dialogs sharply without raising risk. A binary "keep them" or "delete them all" both fail.
🧭 Answer framework
  1. Lead with the conclusion: You can cut most of them, not all of them. Auto Mode's production data: a classifier plus a sandbox cut permission dialogs by about 83%, with no drop in safety.
  2. Explain the classifier: Assign a risk level to every action. Safe operations like reading files and searching code go through; only genuinely suspicious ones pop a dialog. Dialogs go from "ask by default" to "ask on exception."
  3. Explain the sandbox backstop: Even if the classifier mis-allows a dangerous action, the code still runs in an environment with triple isolation — file system, network, process — and can't hurt the real system.
  4. Keep a high-risk deny list: Deleting files, writing databases, sending email — those always need a human. Those dialogs are exactly where user trust comes from.
⭐ Bonus point Compress the logic into one sentence: high autonomy comes from the classifier, low risk comes from the sandbox — you need both. Classifier alone is gambling; sandbox alone still feels bad.
Organize your answer using these lesson pages → Auto Mode in Practice OS-Level Sandbox Isolation
Q28Tech colleague
"These three third-party MCP servers you want to connect — they write the tool descriptions, they give us the return data, we can't review a single line. Have you thought about what that means?"
🎯 What they're assessing
Testing your grasp of the MCP attack surface. They're reminding you: every external data source is another entry they can steer. "It's a big vendor, should be fine" is a zero.
🧭 Answer framework
  1. Catch the supply-chain risk: The Agent trusts tool descriptions returned by MCP. A malicious server that tweaks a description can steer behavior. The Agent thinks it's using a "search files" tool; what's actually running is a delete.
  2. Catch the injection risk: Even a non-malicious server isn't safe. Content it forwards (a scraped page, say) may hide injection instructions, and when the Agent processes that data it can be talked into unintended actions.
  3. Give the governance move: Vet every MCP integration the way you vet a third-party SDK. Minimize how many you connect. Treat everything MCP returns as untrusted data.
  4. Give the architectural backstop: Store OAuth Tokens in an external Vault and forward through a proxy, so the sandbox never sees credentials; isolate the network to limit exfiltration. Even if injection succeeds, the attacker can't steal anything or send it out.
⭐ Bonus point Proactively say "every extra MCP server is another injection entry — I'm cutting this integration list in half first." A PM who cuts their own requirements is the highest-grade trust signal in a tech colleague's eyes.
Organize your answer using these lesson pages → MCP's Dual Risks Two Modes of Credential Isolation
Q29Boss
"The enterprise contract says our Agent will never touch their production database. Sales already signed. Tell me — how do you technically guarantee that 'never'?"
🎯 What they're assessing
The ability to translate contract language into architecture language. "We'll constrain it strictly in the Prompt" and this deal is gone. The boss wants a guarantee mechanism you can put in a contract appendix.
🧭 Answer framework
  1. Set the tone first: Promises are kept by structure, not by the model behaving. The design goal: even if the model is fully steered by an injected instruction, the production database is still unreachable.
  2. Give three layers of trust control: Tool level — high-risk actions need human approval every time. Session level — each session has a limited authorization scope, reclaimed automatically when it ends. Global level — org policy hard-codes that production databases are never reachable; no session grant can override it. The contract's "never" maps to the global layer.
  3. Add network isolation: The Agent runs in a restricted sandbox; network access is controlled; the production database address is unreachable at the network layer — it doesn't even get a chance to try.
  4. Give auditability: Session logs are append-only and record every step. The customer can come audit anytime. A promise plus evidence is what you can actually sign.
⭐ Bonus point Proactively add the credential piece: production-database connection credentials never enter the Agent's execution environment — they're managed through a Vault proxy. A door you can't get the key to is a door that's actually locked.
Organize your answer using these lesson pages → Three Layers of Trust Two-Layer Containment Strategy
Q30Interviewer
"Last question. This chapter has a lot of design patterns. If you could take only one sentence with you, which one — and why?"
🎯 What they're assessing
The closing question, testing abstraction and engineering values. Reciting a term is weaker than giving a judgment. They want to see whether you can compress the whole chapter into your own engineering view, then use it to string together what you learned.
🧭 Answer framework
  1. Give that sentence: Do the simplest thing that works. Every clever pattern points back to it: start from the simplest plan, and add complexity only when it clearly pays off.
  2. String the chapter with it: If one Prompt can do it, don't reach for a Workflow; if a Workflow can do it, don't reach for an Agent. For context, find the minimum high-signal Token set. If tools can be merged, don't split them.
  3. Add the second layer: The core of Agent engineering is state management. What information appears in the window, when, and in what form — that's all an engineer can control. The model's intelligence is given by pretraining; you can't control that.
  4. Add the time dimension: Models get stronger; engineering gets simpler. Helper logic like retries, error correction, and formatting becomes redundant as models improve. Spend the effort on lasting architecture: evals, sandboxes, Session.
⭐ Bonus point Close with Claude Code's lesson: most of its engineering complexity went into managing context; making the model smarter was secondary. That sentence is enough for the interviewer to remember you.
Organize your answer using these lesson pages → Build the Simplest Thing That Works Advanced Overview
One Final Tip
Most questions in this chapter come from the actual floor of technical design reviews, where what's always wanted is judgment and trade-offs. The right approach is still to speak them out loud — to a colleague, a friend, or a recording. The parts that don't flow smoothly are exactly what you think you understand but don't. Click the linked lesson pages and go fill the gaps.