How They Will Test You

AI Harness · 30 Essential Questions

Chapter 2 covers engineering in practice: context, Prompt, security, Agent, and cost. These 30 questions come from three real scenarios — answer them yourself first, then check the framework.

How to Use This Page
Each question is labeled with who's asking. They're probing the same knowledge area, but listening for different things.
🎙 InterviewerWants to verify you truly understand — not just recite buzzwords
👔 BossWants explanations and commitments
🛠 Tech colleagueTesting whether you're worth trusting
Each question has three layers: What they're assessing → Answer framework → Bonus points. For anything you can't answer, click the linked lesson pages at the end.
Q1Interviewer
"Your AI assistant starts forgetting things after 30 turns, and it can't remember what the user said in the first turn at all. Explain why, and how you plan to handle it."
🎯 What they're assessing
The entry-level dividing line in context engineering. It's testing whether you can derive an engineering solution from the window mechanism. Anyone who answers "just switch to a model with a bigger context window" gives themselves away: they haven't run the numbers, and they don't know that large windows have their own pitfalls.
🧭 Answer framework
  1. First, identify the root cause: The context window is all the Tokens the model can see in one pass. Anything beyond it is truncated, and the model has zero memory of it — not even a vague impression. Forgetting means early turns have been cut off.
  2. Give three strategies: Direct truncation (drop the earliest turns — zero cost but permanent information loss); summary compression (summarize history before storing, preserving names and preferences); selective retention (vectorize history, use semantic retrieval to inject only relevant turns).
  3. Match strategy to scenario: Single-turn tool queries like weather checks are fine with truncation. Customer service and long-term learning conversations benefit from summaries after 20+ turns. Complex Agents with very long conversations (100+ turns) should use vector retrieval — fewest Tokens, most accurate answers.
  4. Puncture the "big window" fallacy: Billing is per Token — stuffing everything in makes costs rise linearly. Long contexts also suffer attention dilution. A large window is a capability ceiling; managing the window is the actual solution.
⭐ Bonus point Anyone can recite the benefits of the three strategies. Few can articulate the costs: summaries require an extra LLM call and lose information; retrieval can miss weakly associated but important implicit context. Naming the costs is what makes it sound like you've actually done this.
Organize your answer using these lesson pages → Context Window: AI's Working Memory Context Overflow: Three Handling Strategies Context Compression: Four-Layer Defense
Q2Interviewer
"Everyone says AI PMs need to write good Prompts. For the same task, what's the difference between your Prompt and a hastily written one? Pick a technique you've actually used and explain it."
🎯 What they're assessing
Testing whether you've actually written Prompts or just read articles about them. There are plenty of people who can rattle off Few-Shot, CoT, and other buzzwords; the person who can show a good-vs-bad comparison and explain the effect difference is who the interviewer wants.
🧭 Answer framework
  1. Give the formula first: Role + Task + Context + Constraints + Examples + Format. Missing any component degrades quality. The core mindset is to treat Prompts like code.
  2. Go deep on one technique: For example, Few-Shot. For text classification without examples, the model gives you flowing prose. Give it three "input → label" examples, and it immediately learns the format and standard, outputting a single word that can go straight into your program.
  3. Have an advanced technique ready: For complex reasoning, add chain-of-thought — make the model work step by step. The reasoning process becomes transparent and accuracy improves dramatically. Breaking complex tasks into multi-step sub-Prompts, each optimized separately, produces far higher quality than asking everything at once.
  4. End with constraints: Word count, audience, tone, and forbidden words — write them clearly. Constraints are the cheapest way to control output. A Prompt without constraints produces random results.
⭐ Bonus point Say "I write test cases for my Prompts. Whenever I revise one, I run fixed inputs and check whether the outputs have regressed." PMs who manage Prompts as assets and iterate on them like code are rare. This one statement sets you apart.
Organize your answer using these lesson pages → Advanced Prompt Techniques System Prompt Core Principles Output Format Trade-offs
Q3Tech colleague
"Yesterday a user typed 'Ignore all previous instructions' and extracted our entire system Prompt. How do we defend against this? Surely just adding 'Do not reveal the system Prompt' to the Prompt is enough?"
🎯 What they're assessing
Testing whether you understand the root cause of injection, and whether you think in terms of multi-layer defense. Agreeing that "one line is enough" means when the next variant attack comes, both of you own the blame.
🧭 Answer framework
  1. Lead with root cause: Prompt injection shares its origin with SQL injection: data and instructions flow through the same channel. The system and user text in the message list are all concatenated into one string fed to the model — it can't distinguish which part is an instruction and which is user data. There is no silver-bullet fix.
  2. Give a three-layer intercept: Input layer: use regex to filter known attack patterns (patterns like "ignore.*instructions" or "DAN" trigger immediate rejection at zero Token cost). Prompt layer: write security constraints at the end of the System Prompt, declare them highest priority, and state they cannot be overridden by user input. Output layer: scan replies for System Prompt keywords and rewrite or replace on match.
  3. Standardize rejection language: Whichever layer intercepts the attack, respond in natural product-appropriate phrasing. Never expose the detection logic — this prevents attackers from using trial-and-error feedback to narrow in on a bypass.
  4. Acknowledge there's no silver bullet: Regex can't stop metaphorical bypasses; model constraints can't stop new variants. Security = layered stacking, each layer catching a portion, each successive layer seeing fewer threats.
⭐ Bonus point Proactively propose building an attack sample library, using known types such as privilege escalation, role-play, structural injection, and metaphorical disguise for regular regression testing. Add "relying solely on the model's own alignment is the most dangerous design" — your tech colleague's impression of you will change on the spot.
Organize your answer using these lesson pages → Prompt Injection: Why You Get Attacked 12 Attack Case Studies Prompt Defense: Three-Layer Intercept in Practice
Q4Interviewer
"How does an Agent call tools? Does the model itself go and call the API? If a tool deletes data, whose security responsibility is it?"
🎯 What they're assessing
Testing whether you can distinguish the boundary between the model and the framework. If you think the model is actually executing code, everything downstream about Agent permission design and risk control is a house of cards. This boundary determines where you invest in security when building Agent products.
🧭 Answer framework
  1. Cut to the essence: The model does nothing but predict text from beginning to end. A "tool call" is the model outputting a structured JSON expressing "I want to call get_weather with parameters city=Beijing, date=tomorrow." This is just text — nothing has happened yet.
  2. The framework takes over: Your code parses this JSON, performs tool whitelist validation, parameter checking, and permission control, then actually calls the API. All security logic lives in the framework layer — the model has nothing to do with it.
  3. Inject the result: The API's return data is appended to the message list as a tool_result message, and the model predicts again based on the full context to generate the natural-language reply the user sees. Complete chain: text → framework parses → API → inject result → text.
  4. Answer the responsibility question: It's the framework's. The model only makes the request; execution and interception are both the engineering code's job. That's why high-risk operations need whitelists, parameter validation, and human confirmation — these are product design decisions.
⭐ Bonus point Add one detail: the user sees 1 reply, but behind the scenes there's a chain of 5 API messages. Then add that how you write a tool description directly affects whether the model selects the right tool — good vs. bad descriptions can differ by 3×, and this is where PMs can directly contribute.
Organize your answer using these lesson pages → The Secret of Tool Calls The 5 Messages Behind One Conversation The Art of Tool Descriptions
Q5Boss
"The AI feature has been live for a month. The API bill went up 8×, but users only grew 30%. Where did all the money go? Can we cut it in half next month?"
🎯 What they're assessing
Testing whether you can break down the bill into plain language and then commit to an optimization timeline. Answering "LLMs are just expensive" tells your boss you can't manage costs. Answering "I'll ask the engineers to look into it" hands away all initiative.
🧭 Answer framework
  1. First explain the structural cause: In multi-turn conversations, each turn resends the full history. Costs grow with every turn. If users grew 30% but conversations became deeper and longer, a several-fold bill increase is mechanistically expected — and fixable.
  2. Give the fastest win: Check KV Cache hit rate. Keep the System Prompt stable and don't inject dynamic content into it. Cached history is billed at a discount — in multi-turn scenarios this is the biggest cost driver.
  3. Give the second win: Slim down the context. Summarize and compress history, drop irrelevant turns, stop using the window as a trash can. Input Tokens drop directly.
  4. Commit with numbers: Define a cost-per-session metric and report weekly. Week one: fix cache hit rate. Week two: add compression. After systematic optimization, cutting the bill in half is a grounded target.
⭐ Bonus point Describe one concrete waste on the spot: putting a dynamic timestamp in the System Prompt — a single line — permanently invalidates the cache and directly doubles costs. The boss doesn't need to understand Attention, but he can understand "one line of text burns twice as much money."
Organize your answer using these lesson pages → Why Multi-turn Conversations Keep Getting More Expensive KV Cache: Trading Space for Time Comprehensive Cost Optimization
Q6Interviewer
"What is KV Cache? I've heard that adding one line of the current time to the System Prompt can invalidate the entire cache. Why?"
🎯 What they're assessing
Testing whether you understand the prefix condition for cache hits. This is the cost optimization question PMs most often overlook, yet it best demonstrates engineering comprehension. A good answer shows your cost awareness runs all the way down to request structure.
🧭 Answer framework
  1. Explain the mechanics: Every turn, the model runs Attention over all history Tokens. KV Cache stores the K/V matrices that have already been computed; the next turn only computes new Tokens — trading space for time and money.
  2. Explain the hit condition: Cache is matched by prefix. The System Prompt sits at the very front — if even one character changes, all cache after it is invalidated.
  3. Answer the trap: A dynamic timestamp changes every second — every request has a different prefix, hit rate goes to zero, cost +100%. The correct approach is to keep the System Prompt static and pass the time in a user message.
  4. Add the engineering trap: Cloud inference is distributed. Requests may be routed to nodes without your cache, causing mysterious implicit cache misses. Production systems should use explicit caching (cache_control) to guarantee hits.
⭐ Bonus point List other cache killers: random session IDs, user ID prefixes, A/B test variables, random emoji. Sum up with one principle: anything that makes the System Prompt different on every request is burning money.
Organize your answer using these lesson pages → KV Cache: Trading Space for Time Dynamic Timestamps: The Most Expensive System Prompt Explicit Caching: Real-World Comparison
Q7Tech colleague
"For this AI feature's output, do you want JSON or Markdown? Have you thought it through? If you also want streaming output, some formats won't hold up."
🎯 What they're assessing
Checking whether you understand how format choice affects parsing and user experience. A PM who casually says "either works, you decide" will cause the engineer to write a mountain of fallback code — or users staring at a blank screen for 10 seconds after launch.
🧭 Answer framework
  1. First segment by consumer: If the output goes to a program for parsing, storage, or processing — choose JSON, with stable field structure. If the output is displayed directly to humans — choose Markdown: models handle it best and rendering is cheap.
  2. Explain the streaming difference: JSON requires the full text before it can be parsed — in a streaming scenario the user just waits. Markdown can display token-by-token, producing the best feel. This is the root cause of poor time-to-first-character in many products.
  3. Give the middle ground: If you need both structure and streaming, wrap fields in XML tags. The frontend renders each segment as its closing tag arrives — balancing structure and experience.
  4. Add the cost angle: JSON's brackets, quotes, and field names are all formatting Tokens. The same content costs more than a compact format — for high-frequency endpoints, shaving that 10-30% is worthwhile.
⭐ Bonus point Turn it back with "Will the frontend render this output directly, or will the backend parse and store it?" Use the consumer to back-derive the format. Tech colleagues hate PMs who pick formats on a whim; they love PMs who think through the boundaries for them.
Organize your answer using these lesson pages → Output Format Trade-offs Streaming Output and Format Compatibility Syntax-Level Optimization
Q8Interviewer
"Why do products like ChatGPT and Claude default to Markdown? We want our model to output pretty rich text directly — can we do that?"
🎯 What they're assessing
Whether you can derive the format choice from how the model actually outputs. "It's just industry convention" only sees the surface. The person who can walk through, step by step, why Markdown won is the one who understands the constraints of a plain-text model.
🧭 Answer framework
  1. Name the tension first: An LLM is a plain-text model. It emits text token by token — it doesn't know color, font size, or alignment. Users still expect headings, bold, and lists. The format has to solve layout inside plain text.
  2. Eliminate the alternatives: HTML tags are heavy — the same passage is about 45 tokens, half of them tags. Word/PDF are binary; you can't stream them character by character. LaTeX is easy for the model to get wrong. Markdown expresses layout with a few characters like # and **, about 20 tokens for the same content — 55% cheaper.
  3. Put rendering on the frontend: The model only outputs Markdown text. The frontend renders it into rich text with marked.js (6KB, zero dependencies, one line of code) or react-markdown. If you want rich-text polish, let the render layer do the work. The model doesn't need to care about styles.
  4. Add the streaming trap: During streaming, a code block may not be closed yet — parsing it raw will blow up. In production you detect unclosed backticks, temporarily close them, then render, with a 50ms throttle. Syntax highlighting has to re-run after every re-render.
⭐ Bonus point Name the essence: Markdown is the best fit for "plain-text model + layout needs," and every major AI product picked it. That same derivation works for every format decision.
Organize your answer using these lesson pages → Why LLMs Choose Markdown Markdown Syntax and Rendering Pipeline Output Format Trade-offs
Q9Boss
"Customers say our AI sounds stiff. Can we make it talk like our star support agent, and follow the company template? We'd have to train our own model for that, right?"
🎯 What they're assessing
Whether you can catch the request with the cheapest option, and shut down the expensive myth that "we need to train a model." Saying "we might need fine-tuning" can cost the company hundreds of thousands extra. Saying "just change the Prompt" sounds too casual — you have to explain why it works.
🧭 Answer framework
  1. Lead with the conclusion: You don't need to train a model. The System Prompt defines role, tone, and constraints. Swap the role definition on the same model and you have a different product. Every AI product is, at heart, a different role written into the System Prompt.
  2. How to land the voice: Write the star agent's style as a role spec — "warm and direct, 3 to 5 sentences, never say 'as an AI,' end with one actionable suggestion" — then add 2 or 3 real conversation examples for the model to copy. Showing examples beats explaining in words.
  3. How to land the template: Have the model output structured fields; the frontend renders them into the company template. The product owns the look. The model owns the content. Template redesigns don't require touching the Prompt.
  4. Give time and cost expectations: A Prompt change can ship to a small-traffic test the same day, with zero training cost. The one thing to watch: keep the role definition stable. Don't stuff dynamic content in, or the cache dies and costs go up.
⭐ Bonus point Add this: same model, different Prompt, wildly different output — that's the core lever for differentiating an AI product. A company voice baked into a role asset is something competitors can't copy.
Organize your answer using these lesson pages → System Prompt Core Principles Advanced Prompt Techniques Output Format Trade-offs
Q10Interviewer
"Are Prompt injection and jailbreaking the same thing? What tricks do attackers usually use? Pick a few and walk me through them."
🎯 What they're assessing
Whether your security knowledge is scattered points or a system. Anyone who can only say "ignore previous instructions" is still on lesson one. The person who can classify the attack surface and name a representative technique for each class is the one qualified to talk about defense design.
🧭 Answer framework
  1. Separate the concepts first: Injection mixes instructions into a data channel so the model executes the attacker's commands — "ignore all previous instructions." Jailbreaking coaxes the model off its safety rails into an unrestricted role, like DAN. Jailbreaking is the role-play escape flavor of injection.
  2. Name the five types: Privilege-escalation instructions (forged identity, fake auth codes, gradual multi-turn privilege upgrades); role-play escape (DAN jailbreaks, grandma-exploit emotional manipulation); Few-Shot malicious injection (planting bias in examples, hijacking the output format); structural-symbol injection (JSON disguised as admin commands, instructions hidden in HTML comments, forged system delimiters); metaphorical disguise (classical-literature wrapping, "I'm teaching programming" excuses, reverse psychology).
  3. Call out the most dangerous type: Gradual privilege escalation is the most common in the real world: the first few turns are normal questions that lower your guard, then on turn three they claim to be an admin and ask you to drop the restrictions. The defense principle: evaluate safety independently on every turn. Don't relax because the earlier turns looked fine.
  4. Land it on defense logic: Regex can catch DAN and "ignore.*instructions" and other known keywords. It cannot catch metaphors and new variants. So Prompt-layer constraints and output-layer scanning have to be the backstop.
⭐ Bonus point Cite "instructions hidden in HTML comments" — invisible to the eye, readable to the model. Then add that input preprocessing should strip comments and zero-width characters. That shows you thought about the preprocessing layer too.
Organize your answer using these lesson pages → 12 Attack Cases Prompt Injection: Why Attacks Succeed Prompt Defense: Three-Layer Interception in Practice
Q11Tech colleague
"This request is just adding an inventory-lookup tool to the Agent, right? I can finish the API this afternoon. What else do you even need to evaluate?"
🎯 What they're assessing
Probing whether you know the hidden cost of adding a tool, and what a PM is supposed to do in tool design. If you just say "sure, add it," then when call success rate and the bill blow up after launch, you won't even be able to say where the problem is.
🧭 Answer framework
  1. Acknowledge, then add the bill: The API really isn't hard. But every tool definition lives in the System Prompt — about 60 tokens per turn. A product with 10 tools and 100 conversation turns a day burns 60,000 tokens on tool definitions alone. The longer the tool list, the higher the chance the model picks the wrong one.
  2. Review the description together: Good vs. bad descriptions differ by 3× in success rate. Whether the description says "dates must be YYYY-MM-DD" decides if the model gets it right in one call (800 tokens) or guesses the format four times (3,000 tokens). Write the prerequisites too: an email tool should say "if the user gives a name, call find_contact first to look up the address," or the model will invent one.
  3. Mark the safety properties: Is this tool read-only, or does it change state? Can it run concurrently with other tools? isConcurrencySafe and the read/write flag decide the scheduling strategy and whether you need a human confirm. That's a business call — I have to make it.
  4. Set the failure policy: How long is the timeout, how many retries, what do you return on failure. 10 seconds for reads, 30 for writes, at most 2 retries, a friendly message on failure — lock those in before you ship.
⭐ Bonus point Turn it back with "Does this tool touch user data?" and start a conversation about permission tiers. Your tech colleague will realize you're helping them defuse landmines, and the posture flips from challenge to collaboration.
Organize your answer using these lesson pages → The Art of Tool Descriptions Multi-Tool Orchestration: Concurrent vs. Sequential Short-Term Memory = Context Window 5 Patterns of Agent Deadlocks
Q12Interviewer
"Everyone's talking about Agents. What's the actual difference between an Agent and a regular chatbot? Don't feed me marketing words."
🎯 What they're assessing
Whether you have architecture-level understanding and a sense of sources. Anyone who says "Agents are smarter" will get followed up into silence. The person who can name the classic architecture, break out four capabilities, and attach runtime details has clearly studied the system.
🧭 Answer framework
  1. Draw the line in one sentence: A regular LLM can only "talk" — question in, answer out, done. An Agent can "do" — it plans, calls tools, observes the result, and corrects itself until the task is finished.
  2. Cite the source: Lilian Weng's June 2023 post "LLM Powered Autonomous Agents" laid out the classic architecture: the LLM in the center as the brain, surrounded by Planning, Memory, Tools, and Action. Almost every Agent framework today is a shadow of that diagram.
  3. Attach an example to each: Plan breaks competitive analysis into search, extract, compare, write — and if competitor B has no public pricing, it dynamically switches to third-party reviews. Memory is short-term via the context window, long-term via a vector store that remembers the user across sessions. Act/Reflect is observe-after-execute: the code errors, it diagnoses, it edits, it reruns, until the tests pass.
  4. Land on the formula: Agent = LLM + tools + a loop. The core is the "think, act, observe, think again" cycle. The better you design the loop, the more reliable the Agent.
⭐ Bonus point Give a real order of magnitude: a moderately complex task (strip every console.log from a project and run the tests) actually ran 14 loop turns, 15 tool calls, about 25,000 tokens. Numbers like that are what make it sound like you've seen an Agent run for real.
Organize your answer using these lesson pages → Agent: AI That Gets Things Done ReAct Loop: Think → Act → Observe Agent Engineering Landscape
Q13Interviewer
"MCP is everywhere. What problem does it actually solve? If you owned our tool ecosystem, would you put everything on MCP?"
🎯 What they're assessing
Whether you can talk about a protocol as a business. Reciting "Model Context Protocol" means nothing. The interviewer wants to hear what cost it removes, how you pick among the three transports, and what your adoption decision is based on.
🧭 Answer framework
  1. Qualify it with an analogy: MCP is USB for AI tools. Without it, every Agent writes custom glue for every tool: 3 Agents × 3 tools = 9 custom integrations, and a new tool means 3 more. With it, you write the adapter once — 6 standard connections, and a new tool adds 1.
  2. Three transports: stdio uses a local process's stdin/stdout — simplest, lowest latency, good for local debug. SSE is one-way: the server keeps pushing and the client can't interrupt mid-stream, and it's being phased out. Streamable HTTP is bidirectional streaming, the official recommended standard — use it on new projects.
  3. Give the adoption call: Tools you want to expose outward are worth putting on MCP — one adapter gets you into every MCP-capable Agent ecosystem. Purely internal private tools can wait. Ship the business first, standardize later.
⭐ Bonus point Add a selection detail: a lot of existing MCP servers still use SSE, so you have to care about forward compatibility when you integrate. That shows you've looked at the real ecosystem, not just memorized the new standard.
Organize your answer using these lesson pages → MCP Protocol: The USB Interface for Tools The Secret of Tool Calls
Q14Boss
"The team next door pasted a customer list into an external AI for analysis and got written up. Our team uses AI every day. Tell me clearly — what are the things we absolutely cannot do?"
🎯 What they're assessing
The boss wants a boundary the team can execute on immediately — if something goes wrong, the management responsibility is his. "Just tell everyone to be careful" is a non-answer. Give a clear red line plus a judgment method, and he'll actually hand this to you.
🧭 Answer framework
  1. Name four red lines: Sensitive data stays inside the wall — customer info, trade secrets, financials don't go into external AI, and a vendor promising "we don't store it" doesn't help, because the transfer itself is already a leak. Credentials never enter the chat — if an API key, password, or token gets pasted, rotate it immediately. High-risk actions require confirmation — dropping a database, transferring money, changing permissions: AI can only recommend, a human hits confirm. Approve before use, label after — AI-generated content going public must be marked; that's a legal requirement under China's Interim Measures for the Administration of Generative AI Services (《生成式人工智能服务管理暂行办法》).
  2. Give the frontline a judgment test: Default to redaction. If you're not sure you can paste it, redact first. One self-check: if this conversation got screenshotted onto the internet, would we be in trouble? If yes, don't paste it.
  3. Pair it with tiered controls: L1 scenes like writing email and summarizing — people decide on their own. L2 scenes that affect external customers need human review plus manager approval. L3 scenes where an Agent touches production or money need sandbox testing, security approval, a kill switch, and logs kept for 180 days.
  4. Make the accountability explicit: The user is the first responsible party — "the AI wrote it, not me" doesn't fly. The approver who signs is jointly liable. A manager who "didn't know a report was using it" is not off the hook either.
⭐ Bonus point Call out the easiest trap: debugging with "can you look at why this errors" and pasting a whole chunk of code that has secrets in it. Give the team a habit: "review first, then paste; if you leaked it, rotate the credential immediately."
Organize your answer using these lesson pages → AI Safety Red Lines: Four Bottom Lines Risk Tiers & Accountability
Q15Interviewer
"The user asks the Agent to check flights, weather, and hotels in one sentence, and the model returns three tool calls in one turn. Do you run them one by one or all at once? What's your basis?"
🎯 What they're assessing
Whether you understand the safety-versus-experience trade-off behind scheduling. "Obviously run them together, it's faster" hasn't thought about data races. "Always serial, it's safer" is wasting the user's time for nothing.
🧭 Answer framework
  1. Lay out three strategies: Sequential — one after another, safest, total time is the sum (1.5 + 0.8 + 1.2 = 3.5 seconds). Concurrent — all at once, total time equals the slowest (1.5 seconds), 57% faster. Smart batching — group by safety and run in batches.
  2. Give the decision rule: Look for state changes and dependencies. Booking a flight may charge a card and change user state — run it alone. Weather and hotels are pure lookups — safe to merge and parallelize. Batch 1 runs the flight; batch 2 runs weather plus hotels together.
  3. Land it on the mechanism: The framework marks each tool with isConcurrencySafe, and the concurrent-safe ones become a batch that runs together. The mark is a business judgment. The PM has to tell engineering each tool's safety tier.
  4. Follow up on dependencies: If hotels need to filter by flight arrival time, they depend on the flight result and must be serial. Draw the dependency graph first, then talk scheduling.
⭐ Bonus point Convert the strategy into experience: 3.5 seconds vs. 1.5 seconds is a felt difference in a conversational product. Scheduling is a direct driver of wait time. That sentence shows you're looking at engineering through a product lens.
Organize your answer using these lesson pages → Multi-Tool Orchestration: Concurrent vs. Sequential The 5 Messages Behind One Conversation
Q16Interviewer
"The user asks 'what's the weather in Beijing tomorrow,' and the UI just shows one reply. How many messages actually ran in the API behind that one sentence?"
🎯 What they're assessing
Whether you've taken apart a full tool-call chain. Anyone who can only say "the model called a tool" can't design loading states or progress cues, because they don't even know how many intermediate steps they can work with.
🧭 Answer framework
  1. Read out the ledger: The user sees 1 reply. The API ran 5 messages, 2 model calls, 1 external API. The messages are, in order: system instruction, user question, model returning tool_calls, tool role filling the result back in, model giving the final answer.
  2. Explain the critical hop: In message 3 the model's content is null — only a tool_calls field. That turn the model didn't speak. It submitted a "request form." The framework is what actually calls the API.
  3. Point out the cost implication: A conversation with tools is at least two model calls. Token use more than doubles versus plain Q&A. When you budget a tool-using feature, use that multiplier. Don't quote a one-call budget.
  4. Land it on product decisions: Of those 5 messages, which ones the user should feel and which stay silent — that's product design. A spinner while querying, a progress bar on tool results, exposing failure so the user can retry: every step is a decision.
⭐ Bonus point If you can casually write the messages-array role order — system / user / assistant(tool_calls) / tool / assistant — you've actually read an API payload. That's a rare signal among PMs.
Organize your answer using these lesson pages → The 5 Messages Behind One Conversation The Secret of Tool Calls
Q17Tech colleague
"Our Agent's long conversations keep blowing the context. I'm planning to just delete the earliest messages when we're almost full. Any product objections?"
🎯 What they're assessing
Probing whether you understand that compression is layered, and a blunt cut will hit the wrong things. If you say "sure, your call," then when the user's first-turn requirement gets deleted, the blame is on product.
🧭 Answer framework
  1. Give the budget frame first: The model window is 256K. Set the real safe space at 200K, and reserve 56K for the model's reply. Every compression trigger is a percentage of that safe space.
  2. Lay out four layers of defense: At 60%, trim — drop the huge raw tool returns from early turns and keep a summary; the user feels nothing. At 75%, light compression — replace early long turns with a short summary; mild loss. At 85%, fold — merge several early turns into one session summary; details go, the main thread stays. At 95%, emergency compression — keep only system, a global summary, and the last 3 turns.
  3. Show them the gain: In the course demo, a 1,200-token raw weather-API JSON compresses to an 80-token summary. With all four layers, the same 200K window can hold 5× more conversation.
  4. Give the conclusion: Deleting everything is using the fourth-layer emergency drug as the first layer. Do it in four layers and most conversations only ever hit layers one and two. The user never notices.
⭐ Bonus point Remind them the deletion order matters: drop raw tool returns first, then touch conversation content. Tool returns have the lowest information density and take the most space — that's the highest-ROI first cut.
Organize your answer using these lesson pages → Context Compression: Four-Layer Defense Short-Term Memory = Context Window
Q18Interviewer
"Can your Agent remember a user's preferences from last month? How does it remember? The window is only so big — what happens when memory piles up?"
🎯 What they're assessing
Whether you can tell short-term and long-term memory are two different mechanisms, and whether you'll make trade-offs on retrieval parameters. "We store it in a database" already gives you away. The real question is how you pull the right memory at the right moment.
🧭 Answer framework
  1. Set the tone with an analogy: Short-term memory is the desk — the context window only holds so much. Long-term memory is the filing cabinet — a vector database stores user preferences, project config, historical bugs, and when you need them you retrieve the most relevant few and put them back on the desk.
  2. Walk the chain: Memory first goes through an embedding model into vectors and into a vector store. The course example is 768-dimension vectors in LanceDB. At question time you embed the question too, and recall by semantic similarity.
  3. Name the two key parameters: topK=5 caps each recall at 5 items so memory doesn't crowd the window. minScore=0.3 is the similarity floor — if it's not relevant, don't inject it. Those two numbers are product trade-offs you have to call.
  4. Point at the quality bottleneck: Retrieval quality depends on the embedding model. Whether "fix the login API" and "login API concurrency 500" match the same memory is what decides if this system is an assistant or a decoration.
⭐ Bonus point Turn it back: what information is even worth putting in long-term memory? High-reuse things like user preferences and project config. Dumping the entire chat into the store just lets junk occupy the topK=5 slots.
Organize your answer using these lesson pages → Long-term Memory: Vector Retrieval Short-Term Memory = Context Window
Q19Interviewer
"The production Agent freezes now and then — spins forever, no result. As the product owner, how does it usually die, and how do you plan to catch it?"
🎯 What they're assessing
Your ability to enumerate Agent failure modes. If you can name several ways it dies and pair each with a guard, you've actually operated an Agent product. Anyone who can only say "add a timeout" has probably only shipped a demo.
🧭 Answer framework
  1. Enumerate five ways it dies: Bad parameter format — the model emits illegal JSON. Hallucinated tools — it calls a tool that doesn't exist. Infinite recursion — the same action loops. Insufficient information — it's missing something critical and guesses anyway. API exceptions — the external service is down and nobody handles it.
  2. Pair each with a guard: Schema validation catches bad format. Tool verification catches hallucinated tools. Loop detection catches infinite recursion. An ask-the-user mechanism treats insufficient information. Timeouts catch API exceptions. One death, one medicine. Don't expect a single magic switch.
  3. Translate it into product language: For every failure, pre-decide what the user sees. Retry, error, or hand off to a human — the copy and the exit are product design. Users should not sit there staring at a spinner.
⭐ Bonus point Call out that "insufficient information" is the special one: the fix is teaching the Agent to ask the user. That's an interaction-design problem. The other four are engineering. This one is product.
Organize your answer using these lesson pages → 5 Patterns of Agent Deadlocks ReAct Loop: Think → Act → Observe
Q20Boss
"The new photo-recognition feature costs more than ten times a text-only feature. How do pictures even get billed? Can we make it cheaper?"
🎯 What they're assessing
The boss wants two answers: where the money went, and how to save it. You have to explain image billing in plain language, then give a cost-cut that doesn't kill the feature.
🧭 Answer framework
  1. Explain the billing: The model slices the image into pixel blocks and converts them to tokens. The formula is scaled height times width, divided by pixels-per-token, plus 2. Dimensions also snap to multiples of 32 — shrink if over the cap, enlarge if under the floor — and you pay for the aligned size.
  2. Point at the waste: A casual 4K original uploaded as-is costs tens of times a 512-square thumbnail. If the task is just "is this an invoice," most of those pixels are burning money.
  3. Give a tiered plan: Match resolution to the task. Coarse classification uses low res, scene understanding uses mid, OCR and chart reading get high res. Pick the right resolution and token count can differ by 10× to 100×.
  4. Give the shipping action: Add a preprocessing layer on the upload path that auto-compresses to the right tier by feature type. The user feels nothing. The bill drops immediately.
⭐ Bonus point Add that conversion rates differ by model. The course comparison is Qwen3-VL at one token per 1024 pixels, Qwen2.5-VL at 784. Put vision billing on the comparison table when you select a model.
Organize your answer using these lesson pages → Image Tokens: Pixels Burn Money Too Match Resolution to Task
Q21Interviewer
"Your Agent can delete files. Users complain that confirming every step is annoying. Engineering is afraid to just let it through. How do you design these permissions?"
🎯 What they're assessing
This is a product-judgment question about how you trade safety against efficiency. There is no single right answer. The interviewer is watching whether you have a tiered framework, and whether you'll keep the risk decision in product's hands.
🧭 Answer framework
  1. Lay out three modes: Confirm mode pops a dialog on every dangerous action — safest, most interruptive. Auto mode lets everything through — fastest, and one bad delete is an incident. Smart mode uses an LLM classifier to score risk: low risk goes through, high risk stops for confirm.
  2. Name the prerequisite work: Smart mode only works if every tool is marked "read-only" or "destructive." That mark is a tool-level product decision. The PM has to call them one by one. Pushing it to engineering is a dereliction.
  3. Face the new risk: If an LLM judges risk, the LLM can misjudge too. So the most destructive tier (drop a database, transfer money) never enters smart judgment — always human confirm. That's a second lock on misclassification.
  4. Compensate the experience: Put the confirm dialog in context, and let one confirm remember similar actions, so you cut the interrupt count and only keep the brake where it's actually dangerous.
⭐ Bonus point Say "the core tension in permission design is safety vs. efficiency, and product's job is to pick a defensible balance." That lifts the question from features to responsibility.
Organize your answer using these lesson pages → Permissions & Security Risk Tiers & Accountability
Q22Tech colleague
"You keep asking me to add Skills to the Agent. A Skill is just a chunk of Prompt, right? How is that different from me writing it into the System Prompt?"
🎯 What they're assessing
They're challenging whether Skill is even necessary. You have to admit it isn't a new technology, then make the difference in trigger, boundary, and constraints vs. a bare Prompt clear — and persuade them with the mechanism.
🧭 Answer framework
  1. Agree first, then distinguish: A Skill is, at heart, experience written as a document — a process note plus tool-call guidance. But it loads on demand: it only injects when a matching task is recognized. Whatever you hard-code into the System Prompt occupies tokens on every request.
  2. Take SKILL.md apart: Metadata with trigger words decides recall. Applicability conditions are the anti-misfire insurance — if you're not in the target project directory, it exits silently. Steps are an SOP that runs in strict order. Allowed tools draw the safety boundary — a release Skill, for example, disables delete_file and sub-agents.
  3. Land the value: ReAct without a Skill is a bad loop — several rounds of trial and error. With a Skill it's a good loop: the AI knows what to do first and what to do next before it walks out the door, and finishes in one pass. Shorter loops mean fewer tokens and less latency.
  4. Give the collaboration split: Step order and safety constraints in a Skill are business experience — product writes those. Load and execution are engineering. Cursor, Claude Code, and Copilot all support the SKILL.md standard. You don't have to invent one.
⭐ Bonus point Point out that constraints matter more than steps: a wrong step just makes the Agent do the wrong thing; a missing constraint makes it do something dangerous. "No force push, no release if tests fail" in a release Skill is that kind of life-saving clause.
Organize your answer using these lesson pages → Skill: Getting Agents to Take Fewer Detours The Essence of Skill Dissecting a Real Skill
Q23Interviewer
"Your Agent demo looks stunning. The moment it hits production, everything falls over. What's actually missing between a prototype and a product?"
🎯 What they're assessing
Whether you know that the bulk of Agent engineering is scaffolding. Anyone who blames the crash on "the model isn't smart enough" will keep waiting for the next model. The person who understands scaffolding knows which five boards to nail on.
🧭 Answer framework
  1. Give the core ratio: Agent engineering is 80% scaffolding plus 20% model. Most Agent projects fail because error handling isn't robust. How smart the model is is actually secondary.
  2. List five capabilities: Timeouts and retries — tool calls get a timeout plus exponential backoff. A max-step limit — max_iterations stops death loops. Input/output validation — JSON Schema blocks illegal parameters. A state machine plus rollback — checkpoint recovery, so failure doesn't restart from zero. Observability and logs — full-chain records, so production issues get located from the log.
  3. Contrast with a scene: In the course's "check flights plus book a hotel" simulation, a bare Agent dies on one API timeout and the whole order is toast. With scaffolding, it retries on timeout and rolls back to a checkpoint. The user just feels it was a bit slower.
  4. Land it on the schedule: Put those five capabilities into the engineering requirements at kickoff, on the same timeline as features. Bolting them on after launch is putting the brakes on after you've already started driving.
⭐ Bonus point Turn "demo to product" into an acceptance checklist: does every tool have a timeout? Is there a max_iterations? Can failure roll back? Three questions and you can roughly feel an Agent project's maturity.
Organize your answer using these lesson pages → Scaffolding Engineering: From Prototype to Product Observability
Q24Boss
"Competitors are all advertising the latest flagship model. Should we follow across the board? It's a bit more expensive, but the quality can't be worse, right?"
🎯 What they're assessing
The boss is being pulled by the instinct that "more expensive is better." You have to pull them back with a selection framework: spend the money where the blade is, and say clearly where the blade is.
🧭 Answer framework
  1. Break the instinct first: Capability and price are nonlinear. Ten times the price may buy you 10–20% more capability. On a lot of tasks, users cannot tell a mid-tier model from a flagship.
  2. Give the selection formula: Selection equals task difficulty × call volume × room for error. Simple, high-frequency, low-tolerance-for-error tasks use a small model. Complex, low-frequency, high-value tasks get the flagship.
  3. Give the alternative: Take a fraction of the "flagship everywhere" budget and put it into intent recognition plus model routing: 80% of simple questions go to a small model, only the hard ones escalate. You can save 40% to 60% with basically no experience loss.
  4. Talk with data: Compare on scenario evals, not vendor leaderboards. Run the same batch of real tasks on every candidate and draw the conclusion from our own scenes.
⭐ Bonus point Add the cut: when a competitor advertises a flagship, they mostly only use it on the storefront scenes. Matching their marketing line and matching their cost structure are two different things.
Organize your answer using these lesson pages → Model Selection: Capability vs. Cost Holistic Cost Optimization
Q25Interviewer
"If you had to set rules for an Agent going to production, which ones would you set? What happens without them?"
🎯 What they're assessing
Whether you can break stability into concrete defenses. "Do more testing" has no information. The interviewer wants to hear what accident each guardrail prevents, and how it dies if that guardrail is missing.
🧭 Answer framework
  1. Give the positioning first: An Agent's core capability comes from the LLM. Its stability comes from engineering guardrails. An Agent with no guardrails is a sports car with no brakes — the more capable, the more dangerous.
  2. List five guardrails: An iteration cap stops death loops — force the Agent to clock out when it's spinning in place. Output truncation stops blow-ups — chop oversized tool returns. Timeouts stop freezes — a hung external call doesn't take the whole task with it. Interrupt recovery stops corruption — if power or network drops mid-task, resume from a checkpoint. Context emergency compression stops crashes — when the window is almost full, compress to stay alive.
  3. Give the judgment standard: These five defenses decide whether an Agent is "usable" or "good." Accept an Agent product by walking these five one by one. Miss one, and a matching class of production incident is waiting.
⭐ Bonus point Pair the guardrails with the deadlock patterns: five ways of dying are the disease, five guardrails are the medicine. If you can talk about them as pairs, you understand a system. People who only recite a checklist can't make that mapping.
Organize your answer using these lesson pages → 5 Engineering Guardrails 5 Patterns of Agent Deadlocks
Q26Tech colleague
"Doesn't the platform already have KV Cache? The docs say it hits automatically. Why do we still have to write cache_control ourselves?"
🎯 What they're assessing
They think implicit cache is enough. You have to make clear why implicit cache is unreliable on a distributed architecture — that decides whether the savings plan actually saves money, or only saves it on paper.
🧭 Answer framework
  1. Break the premise: Cloud LLMs run on many GPU nodes. Requests get randomly routed by a load balancer. Your cache is on node A; the request lands on node B and it's a MISS. Implicit cache's real hit rate is under 30% — pure luck.
  2. Explain the explicit approach: Add a cache_control line to mark a cache anchor. The platform then routes the request to a node that has the cache, and hit rate approaches 100%. Anthropic, Alibaba Cloud, and OpenAI all support this pattern.
  3. Do the price math: In the course's discount comparison, an implicit hit bills at 20% of standard price; an explicit hit bills at 10% — 90% off input cost. Explicit wins on both hit rate and discount.
  4. Give the conclusion: Production must use explicit cache. Betting your savings on random routing is neither reliable nor professional.
⭐ Bonus point Remind them the anchor position has to respect prefix matching: keep the System Prompt fixed, put dynamic content after it, or the anchor is meaningless. That sentence hooks right back to the classic anti-pattern of a dynamic timestamp killing the cache.
Organize your answer using these lesson pages → Explicit Caching: Real-World Comparison KV Cache: Trading Space for Time Dynamic Timestamps: The Most Expensive System Prompt
Q27Interviewer
"Same feature, their Prompt is a few hundred tokens, yours is over two thousand. If I handed you the knife, where would you start cutting?"
🎯 What they're assessing
Whether you understand the first principle that a Prompt is an instruction written for a machine, and whether you can tell syntax-layer slimming from semantic-layer slimming.
🧭 Answer framework
  1. Cut the syntax layer first: Formatting tokens can eat 13% to 20% of a Prompt. For complex objects, YAML instead of JSON saves 15% to 30%. For flat lists, CSV instead of a JSON array — repeating field names N times is the biggest waste — saves 30% to 60%. Stripping Markdown decorations like bold and headings saves another 8% to 13%. In the course's measured case, bold markers alone ate 8.5% of the tokens.
  2. Then cut the semantic layer: Don't hard-code Few-Shot examples. Retrieve the 3 most relevant ones with vectors each time — 87.5% savings. Compress long documents with LLMLingua-2 before feeding the model — 5× to 20×.
  3. Tidy the structure while you're at it: Models pay the least attention to the middle, so put the critical information at the head and tail. This step doesn't save money, but it makes every remaining token worth more.
  4. Name the double payoff: Attention is O(N²). Double the Prompt and compute goes 4×. The money you save by slimming is one payoff. The speed and quality lift is a free second one.
⭐ Bonus point Quote the course's judgment: at tens of millions of calls, 20% of the monthly budget is spent on formatting that "makes the PM feel comfortable." When you cut a Prompt, cut the human-facing part first.
Organize your answer using these lesson pages → Syntax-Layer Optimization: Prompts Written for Machines Semantic-Layer Optimization
Q28Boss
"I had AI put together an industry report. The citations all look legit. Fine to send it straight to the client, right?"
🎯 What they're assessing
The boss is asking you for a "you're fine." You have to make them see that hallucinations hide in the places that look most real, and give them a tiered judgment they can use next time themselves — not just "be careful."
🧭 Answer framework
  1. Show the danger first: Hallucinations sit in the middle of real information, with the same format and citation style as the truth. In the course test, one fabricated fact was hidden among six real quantum-computing history points — names, project names, "Nature's top ten of the year," all invented, and more convincing than the real ones.
  2. Give the recognition pattern: AI's three forgery moves are fake names, fake projects, fake honors, often paired with real institutions, publishers, and Douban scores to look credible. Any citation specific enough to be a person plus a result — verify first.
  3. Give scene tiers: Email, brainstorming, translation and polish — use freely. Data analysis, technical research, writing code — verify, then use. Legal, medical, investing are high-hallucination zones — treat them as leads only. A client-facing report is in the verify-first tier. Check every key number and citation before you send.
  4. Give the action: Search the keywords in the report and cross-check. Delete any citation you can't source. Ten minutes. A lot cheaper than the client catching you.
⭐ Bonus point Leave the boss a memorable self-test: the more specific the citation, the more you should check. When AI is bluffing, it loves to invent extra-rich detail.
Organize your answer using these lesson pages → Can You Trust What AI Says? Spot the Hallucination Scenario Guide: When to Trust AI
Q29Interviewer
"What kind of task is worth splitting across multiple Agents? Once you split, how do you keep them from fighting each other?"
🎯 What they're assessing
Your grasp of when multi-Agent collaboration applies, and the safety rules. Anyone who blurts "multi-Agent is stronger" has heard too much marketing. The interviewer wants the split rationale and the concurrency discipline.
🧭 Answer framework
  1. Give the split model: The main Agent is the coordinator that breaks up the work; sub-agents each own a job. In the course's auth-module refactor: the researcher is read-only and maps the code; the developer can read and write and makes the changes; the tester can read plus run and verifies.
  2. Set concurrency discipline: Read-only tasks parallelize for speed. Write tasks stay serial for safety. Two sub-agents editing the same file at once is a disaster, so permissions and order follow the read/write nature.
  3. Explain isolation: Sub-agents run in independent worker threads, memory-isolated, and the parent can kill a child at any time. The event stream has three signals — subagent_start, subagent_chunk, subagent_end — and the progress bar and interrupt button hang off those.
  4. Draw the applicability line: Don't split a job a single Agent can finish in one pass. Splitting has scheduling and merge overhead. Only stand up a team when the task naturally chunks and you get a parallelism win.
⭐ Bonus point Add observability: once multiple Agents run, token spend is multiplicative. Usage events in the stream have to be billed per sub-agent, or the bill explodes and you don't know who spent it.
Organize your answer using these lesson pages → Multi-Agent Collaboration Observability
Q30Interviewer
"All this Prompt optimization, RAG, caching you're doing — does the next model release make it all wasted work? How do you decide what's worth investing in?"
🎯 What they're assessing
This is the chapter's final question. It tests whether you have first principles. If you can answer it well, every technique from earlier has already collapsed into one thing in your head. If you can't, you just memorized a pile of tricks.
🧭 Answer framework
  1. Give the essence first: Every Harness technique is, at root, constructing better context so the model understands intent more accurately. RAG, compression, Few-Shot, caching — all different faces of that one job.
  2. Give three dimensions: Quality — inject precise, high-density information. Structure — put the critical information at the head and tail, put core constraints in the System Prompt. Cost — carry the most useful information in the fewest tokens. Before you fund any Harness work, ask which dimension it lands on.
  3. Give the trade-off rule: Worth doing: complementary techniques you use to compete on cost, efficiency, and quality, and that get better when the model upgrades. Worth dropping: techniques that burn huge resources, get replaced the moment the model upgrades, and that users never feel. Before every kickoff, ask: after the next model version, do we still need this?
  4. Close on the origin: When you don't know what to do, go back to that question: is the context I'm giving the model everything it needs to do this well? Answer that, and you've got the essence of Harness.
⭐ Bonus point Use an already-obsolete technique as an example — say, a complex split you built to save window space that lost its value once long-context models arrived. That proves you actually measured your own plans with the "will a model upgrade replace this?" ruler.
Organize your answer using these lesson pages → The Simple Truth: Hold to First Principles Holistic Cost Optimization
One Final Tip
The right way to use these 30 questions is to speak them out loud — to a colleague, a friend, or a recording. Just reading them doesn't count. The parts that don't flow smoothly are exactly what you think you understand but don't. Click the linked lesson pages and go fill the gaps.