How They Will Test You

Harness & Self-Improvement · 30 Soul-Searching Questions

This chapter covers the most cutting-edge content, and those who test you on it can most easily tell the real from the fake. These 30 questions come from three real scenarios — try answering aloud first, then check the framework.

How to Use This Page
Each question is labeled with who is asking. They are all testing the same knowledge block, but each wants to hear something different.
🎙 Interviewerwants to verify whether you truly understand or are just reciting terms
👔 Bosswants explanations and commitments
🛠 Tech Colleagueprobing whether you are worth trusting
Each question has three layers: What they are testing → Answer framework → Bonus points. For parts you cannot answer, click the course links at the end to review.
Q1Interviewer
"Everyone's been talking about Harness lately — can you explain what it actually is? Why do some people say it's just as important as the model itself?"
🎯 What they're assessing
This is the opening litmus test for this chapter. It checks whether you can articulate Harness as a clear system concept, or if you only vaguely say "it's just a wrapper" or "just an Agent framework." Someone who truly understands it can explain what it governs and why it determines product success or failure.
🧭 Answer framework
  1. Define it first: Harness is the runtime system surrounding the base model. It determines how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results. The model provides the intelligence; Harness makes that intelligence work in the real world.
  2. Provide evidence: Successful products like Claude Code, Codex, and Cursor have proven that the Harness layer is as important as the raw model intelligence. A mediocre model with an excellent Harness often outperforms a stronger naked model.
  3. Explain the PM perspective: Swapping models is easy — Harness is the product's true moat. With the same model, the quality of the Harness design can produce user experience differences of several orders of magnitude.
  4. Elevate the insight: Harness is shifting from a supporting engineering role to becoming the optimization target itself. Cutting-edge research lets models improve the Harness surrounding them — this is the real-world path to recursive self-improvement.
⭐ Bonus pointDraw the Prompt Engineering analogy: manual Prompt techniques faded as models got stronger, but the need to specify goals, constraints, context, and evaluation never went away. Many Harness improvements will eventually be internalized into models, but the interface with external context and tools will always exist.
Organize your answer using these lesson pages → From Scaffolding to Self-Improving Systems Three Design Patterns for Harness
Q2Interviewer
"If you were to design a production-grade Agent architecture, what design patterns would you follow? Why do the best coding Agents today all look similar?"
🎯 What they're assessing
They are checking whether you have an architectural vocabulary. A revealing answer just lists tool names and framework names. What they want to hear is the pattern-level abstraction: why these systems converge to similar structures, and what problem each pattern solves.
🧭 Answer framework
  1. Name the three patterns: workflow automation, filesystem as persistent memory, and sub-Agents with background tasks. These cover 90% of the architectural decisions in today's most capable Agent systems — they are structural requirements with no real optionality.
  2. Pattern one — the loop: An Agent is a goal-oriented loop: Plan → Execute → Observe → Improve → Execute again. Failure is the trigger for self-correction: a failing test or a command error causes the Agent to analyze its trajectory and adjust.
  3. Pattern two — memory: Artifacts from long tasks quickly overflow the context window. The right approach is to persist state to the filesystem and let the Agent read and write as needed. One-sentence principle: context is working memory; filesystem is long-term memory.
  4. Pattern three — parallelism: The parent Agent acts as a process manager: spawning sub-Agents, polling progress, canceling failed branches, and merging results. Sub-Agent outputs must be persisted to files so they can be recovered after interruption.
⭐ Bonus pointArticulate the key tradeoff in sub-Agent parallelism: each sub-Agent works in an isolated sandbox and outputs to explicit file paths; the parent Agent coordinates by polling file state rather than sharing memory. This design dramatically simplifies concurrency control — being able to articulate it signals you have seen real production systems.
Organize your answer using these lesson pages → Three Design Patterns for Harness From Scaffolding to Self-Improving Systems
Q3Boss
"That Agent of ours — once the task runs long it starts making mistakes, forgetting everything said earlier. Can this be fixed? How long will it take?"
🎯 What they're assessing
The boss wants diagnosis + solution + timeline. A revealing answer is "just use a model with a bigger context window" — that shows you haven't understood the root cause. The real problem is in the context management strategy, and only someone who can explain that and provide a phased plan is worth trusting.
🧭 Answer framework
  1. Diagnose the root cause first: Most likely it is naive append-only context management. Stuffing all tool responses and history into the context fills the window as the task runs long, pushes out early information, and causes output quality to plummet. This is a strategy problem — a bigger context window only delays the symptoms.
  2. Phase 1 solution: Introduce filesystem persistent memory. After each round, write progress, error logs, and intermediate results to files, freeing the context; read selectively on the next round. Context usage goes from continuously growing to constant, enabling stable runs of dozens of rounds.
  3. Phase 2 solution: Introduce ACE-style structured context maintenance. Generator does the work, Reflector reviews and distills insights, Curator organizes insights into itemized playbooks, incrementally merging and deduplicating. Experience gets sharper with use while the context gets leaner.
  4. Timeline commitment: Phase 1 is an engineering refactor — results visible within a week. Phase 2 requires building an evaluation set to verify convergence; report changes in long-task success rates bi-weekly using metrics.
⭐ Bonus pointAdd an industry observation: long-context capability and context engineering are two separate legs. No matter how large the model's window, you still need a management layer to build structured, lean context. Every team building Agents will face this problem — solving it early is a competitive advantage.
Organize your answer using these lesson pages → Filesystem as Persistent Memory Context Engineering: From Manual to Autonomous Evolution
Q4Interviewer
"Isn't context engineering just about writing better prompts? I saw a paper saying it can self-evolve now — are you familiar with ACE and MCE?"
🎯 What they're assessing
They are checking your depth and sense of layering for cutting-edge concepts. Someone memorizing terms will lump ACE and MCE together. Someone who truly understands can articulate an evolution line: the optimization target upgrades layer by layer — from content to mechanism to system code — and what each layer solves.
🧭 Answer framework
  1. Correct the premise first: Manual Prompt writing is just the starting point. The optimization target follows an evolution line: instruction Prompt → structured context → workflow → Harness code → optimizer code. The stronger the model, the more complex the target it can optimize.
  2. Explain ACE: It optimizes context content. Maintains a structured bullet-point playbook; Generator executes tasks, Reflector distills insights from success and failure trajectories, Curator incrementally merges entries using deterministic logic. Never rewrites in bulk — avoids context collapse. The limitation is that update rules still require manual design.
  3. Explain MCE: Separates "how to manage context" from "what is in context" — two-layer optimization. The inner layer finds the optimal context for a given skill; the outer layer compares different skills to select the optimal mechanism. Both what you remember and how you remember evolve together.
  4. Explain Meta-Harness: One layer deeper — the optimization target is the code itself that determines how information is stored, retrieved, and presented. The Proposer is a coding Agent that outputs a set of Harness candidates on the Pareto frontier. Most general, but each revision requires a full trial run to score, making it the most computationally expensive.
⭐ Bonus pointSummarize the three differences in one sentence each: ACE optimizes the note content, MCE optimizes how notes are taken, Meta-Harness optimizes the source code of the entire workbench. Add the core lesson: once Harness design becomes an executable search space, a capable coding Agent can explore the same design space as a human engineer.
Organize your answer using these lesson pages → Context Engineering: From Manual to Autonomous Evolution Workflow Design: From Manual to Automated Search
Q5Tech Colleague
"You want the Agent to modify its own Harness? What if it also modifies the evaluator and gives itself a perfect score?"
🎯 What they're assessing
The tech colleague uses a sharp counterexample to probe whether you understand the safety design of self-improving systems. Answering "add a prompt telling it not to modify" will immediately lose their respect. What they want to hear is boundary design: what must stay outside the improvement loop, and how each change is validated.
🧭 Answer framework
  1. Acknowledge the problem is real: This is reward hacking — the most dangerous anti-pattern in self-improvement loops. Optimizing unit tests leads to overfitting test cases; optimizing the judge model teaches gaming; optimizing benchmark scores leads to exploiting benchmark flaws.
  2. Give the boundary principle: Evaluators and permission controls must sit outside the evolution loop, maintained by humans or tamper-proof independent mechanisms. The exam setter and the grader must be independent of the student being tested. The editable surface must have boundaries — editing at the OS system-config level breaks the abstraction boundary.
  3. Give the validation mechanism: Reference the Self-Harness three-stage loop. Weakness Mining clusters failure patterns from failure trajectories; Harness Proposal submits bounded edits; Proposal Validation uses held-in and held-out datasets to verify — only accepting edits with no regression.
  4. Give the capability prerequisite: STOP experiments show that a recursive structure does not guarantee improvement by itself. GPT-4 can continuously improve; weaker models amplify noise and actually regress. Before deploying this system, evaluate whether the base model can support meta-level optimization.
⭐ Bonus pointAdd a risk they may not have thought of: diversity collapse. Evolution loops naturally exploit known high-reward patterns; when all candidates become minor variants of the same solution, innovation stops. You need diversity rewards and archive preservation mechanisms as a safety net. If you can articulate both risks, you win this conversation.
Organize your answer using these lesson pages → Let Harness Improve Itself Future Challenges: Seven Gates of Self-Improvement Evolutionary Search: Survival of the Strongest Harness
Q6Interviewer
"Where exactly is recursive self-improvement right now? AI can modify itself now — is it about to get out of control?"
🎯 What they're assessing
They are checking whether you can evaluate cutting-edge progress without either mystifying or panicking. There are two revealing answers: amplifying "AI is about to awaken," or flatly saying "it's all hype." What they want is industry knowledge grounded in facts, with clear boundaries and judgment.
🧭 Answer framework
  1. Give the conceptual coordinates first: RSI was envisioned from Good (1965) to Yudkowsky (2008) — a system using its current intelligence to improve the very mechanism that produces intelligence. It remained a theoretical concept for decades; only in the past two years has a real-world path emerged.
  2. Clarify the real-world path: Models do not directly rewrite their own weights. What they improve is the Harness surrounding them: context management, workflows, tool orchestration, evaluation. A better Harness produces a stronger model; a stronger model in turn simplifies the Harness — a positive-feedback flywheel.
  3. Give the true state of progress: STOP proved that an improver can recursively improve itself, and automatically rediscovered classic strategies like genetic algorithms and Beam Search. Self-Harness lets Agents improve their own configurations by mining weaknesses, proposing edits, and validating against regressions. But all of this happens inside bounded, verifiable loops.
  4. Directly answer the out-of-control question: There are still seven gates before "out of control": weak evaluators making feedback signals noisy, poor memory lifecycle management, reward hacking, diversity collapse, and difficulty measuring long-term health. These are fundamental system-design challenges, and the shared solution points toward keeping humans in the loop and providing oversight at the right level of abstraction.
⭐ Bonus pointClose with a reality check on automated research: AI Scientist can produce formally complete papers, but being able to write a paper falls far short of doing science — false citations, implementation drift, and over-optimism recur repeatedly. Use this case to demonstrate that your judgment is grounded in primary sources.
Organize your answer using these lesson pages → From Scaffolding to Self-Improving Systems Let Harness Improve Itself Future Challenges: Seven Gates of Self-Improvement
Q7Interviewer
"When you optimize an Agent, what exactly are you optimizing? Isn't it just tweaking the prompt?"
🎯 What they're assessing
They are checking your grasp of the layers of what gets optimized. People who only talk about tuning the prompt are stuck at the bottom layer. Someone who truly understands knows there is a five-level evolution line, and can explain why the stronger the model, the higher the layer you can optimize.
🧭 Answer framework
  1. Name the five-level ladder: instruction Prompt → structured context → workflow → Harness code → optimizer code. Tuning the prompt is only Level 1; the end of this line is optimizing "the code that writes the optimizer."
  2. Explain the upward shift: The smarter the model, the more complex the target it can optimize and the more general the method. ACE optimizes context content, AFlow searches workflows, Meta-Harness edits Harness source, STOP optimizes the improver itself — each level has representative work.
  3. Give the PM judgment: First see which level the team is stuck at. Most teams are still between Level 1 and Level 2; every step up steeply raises the demands on the evaluation system.
⭐ Bonus pointPoint out the shared lesson of this ladder: once Harness design becomes an executable search space, a capable coding Agent can explore the same design space as a human engineer. The higher the level, the more the human role shifts from executor to exam setter.
Organize your answer using these lesson pages → Context Engineering: From Manual to Autonomous Evolution Let Harness Improve Itself
Q8Interviewer
"You said an Agent can spawn sub-Agents in parallel — what exactly does the parent Agent have to manage? What if a sub-Agent dies halfway?"
🎯 What they're assessing
They are checking your engineering details of the sub-Agent pattern. A revealing answer is "just spin up a few and run them together." What they want to hear is the process-manager view: what the parent Agent manages, how state is checked, and how failures are covered.
🧭 Answer framework
  1. Locate the role first: The parent Agent is a process manager and does four things: spawn sub-tasks, check logs and progress, cancel failed branches, and merge successful results. This is operating-system-level thinking.
  2. Explain explicit inspectability: Parallelism cannot be fire-and-forget. The parent Agent must be able to inspect each sub-Agent's status, output, and errors at any time.
  3. Explain output persistence: Sub-Agent results should be stored as files, logs, or state records — not only passed back into the parent Agent's context. That way they can be recovered even after an interruption.
  4. Explain the fault-tolerance net: Background tasks will time out, crash, or produce low-quality results. The Harness must pre-design retry strategies and graceful degradation.
⭐ Bonus pointGive a sense of scale: the payoff of parallelism is turning serial waiting into multi-core speedup. In the course demo, 5 parallel tasks finish in 3 rounds (serial would take 5); the main Agent only handles dispatch and merge, and context usage stays under 30%.
Organize your answer using these lesson pages → Three Design Patterns for Harness
Q9Tech Colleague
"For memory you're just stacking a filesystem? No vector store, no dedicated memory module — is that laziness, or is there a real reason?"
🎯 What they're assessing
Probing whether you are following fashion or understanding the tradeoff. If you cannot state the design rationale and the division of labor for the filesystem approach, you will be taken for cutting corners. What they want to hear is the structural reason behind this "crude method."
🧭 Answer framework
  1. Give the division-of-labor principle first: Context is working memory; the filesystem is long-term memory. Current-task instructions, immediate tool results, and the last 2–3 turns of conversation go in context; historical experiment results, error logs, summaries of finished sub-tasks, and long-term strategy go in files.
  2. Explain the artifact reality: Artifacts from long tasks (experiment logs, code diffs, paper abstracts, error traces, full execution trajectories) quickly overflow the context window. Stuffing them all into the prompt is a structural dead end.
  3. Give the design rationale: File read/write is a basic LLM skill — it needs no complex external toolchain, and it benefits from base-model gains: the smarter the model, the more efficient the file management. A bolted-on memory system does not get that dividend.
  4. Explain the work habit: A good Agent maintains its own scratchpad, todo list, and experiment notes, managing the workspace the way a human programmer does.
⭐ Bonus pointBack it with a frontier method: MCE simply instantiates the context function as a collection of files in a directory — skill.md stores knowledge, dynamic files store rollout records; Meta-Harness also relies on filesystem access to execution history. Filesystem-as-memory is already the shared foundation of frontier self-improvement methods.
Organize your answer using these lesson pages → Filesystem as Persistent Memory Context Engineering: From Manual to Autonomous Evolution
Q10Boss
"I keep seeing news that Agents can self-evolve now. What if a competitor ships this first? Should we spin up a project and chase it?"
🎯 What they're assessing
The boss wants a decision standard — help them tell what is worth chasing from what is still stuck in papers. Saying yes or no outright is dangerous; only someone who can give a fitness check is worth authorizing.
🧭 Answer framework
  1. Give the three conditions of fit: Evolutionary search works only when all three hold: the search space is large and discrete, gradients are unavailable but evaluation is easy, and fitness can be quantified as a number. Only then is it worth adopting.
  2. Give the unfit list: A single evaluation takes hours, the criteria are fuzzy and subjective, you mainly rely on heuristic judgment, compute budget is limited, or a human-review step is required. Hit any one of these and stay away for now.
  3. Map it to our own business: The key question is whether our Agent tasks have an evaluation set that can be scored automatically. If not, step one is building evaluation — that priority is far higher than adopting an evolutionary algorithm.
  4. Give a reference frame: DGM used evolution to lift SWE-bench Verified from 20% to 50%, but that is a coding task where pass/fail is naturally auto-evaluable. First confirm whether your own tasks have that kind of "scale."
⭐ Bonus pointProactively flag the risks: even when every condition is met, evolution loops still have two traps — reward hacking and diversity collapse — and the evaluator must sit outside the loop, maintained by humans. Spelling out the risks before you greenlight a project beats explaining them after something blows up.
Organize your answer using these lesson pages → Evolutionary Search: Survival of the Strongest Harness Future Challenges: Seven Gates of Self-Improvement
Q11Interviewer
"Why does ACE insist on incrementally updating that playbook one entry at a time? Wouldn't it be much simpler to just let the model rewrite the whole prompt?"
🎯 What they're assessing
They are checking whether you understood ACE's most central design decision. Someone reciting the pipeline will only chant the three names Generator, Reflector, Curator. Someone who truly understands can say why the update method must be incremental merge.
🧭 Answer framework
  1. Lay out the mechanism first: ACE maintains a structured bullet point playbook; each entry has an identifier and a description. Generator works from the playbook, Reflector distills insights from success and failure trajectories, and Curator writes those insights back into the playbook.
  2. Explain the key design: Curator outputs structured (identifier, description) entries and merges them into the playbook with deterministic logic — it never rewrites the entire prompt.
  3. State the reason: Letting the model iteratively rewrite a whole blob produces context collapse and conciseness bias; useful details get squeezed out pass after pass. Incremental merge plus periodic refine-and-dedup makes experience sharper with use, without making the context thicker.
  4. Name the limitation: ACE's update rules are still hand-designed — that is exactly the problem MCE then takes on.
⭐ Bonus pointPut it in one sentence: context is a continuously evolving playbook. Let the prompt grow without bound and it overflows; let the model rewrite the whole blob and it collapses. ACE took a third path: itemized, incremental, and deduplicable.
Organize your answer using these lesson pages → Context Engineering: From Manual to Autonomous Evolution
Q12Interviewer
"MCE calls itself two-layer optimization — what is each layer actually optimizing? What is the essential difference from ACE?"
🎯 What they're assessing
A frontier-method discrimination question, checking your ability to take a mechanism apart. If you cannot answer what a skill is and what the inner and outer layers each do, you have only read the title and the recap posts.
🧭 Answer framework
  1. Define skill first: An MCE skill defines a context function c = F(x; ρ). ρ is the static component (prompts, knowledge bases, codebases); F is the dynamic operator (search, select, filter, format). The method of managing context itself is formalized.
  2. Split the two layers: The inner layer, given a skill, finds the optimal context on the training set; the outer layer compares different skills on the validation set and selects the optimal mechanism. Optimize content first, then method, in alternation.
  3. Explain the evolution method: The system maintains a skill database recording each (skill, context, train score, validation score) tuple; a Meta-agent uses agentic crossover to hybridize new skills from history.
  4. Contrast with ACE: ACE's update rules are hand-designed and fixed; MCE puts "how to remember" into the optimization loop as well, so what you remember and how you remember evolve together.
⭐ Bonus pointPoint out why the inner and outer layers use the training set and validation set separately: a skill is scored on data that did not participate in the inner-layer optimization, which prevents the management mechanism from overfitting a handful of tasks — the same anti-overfitting idea as in machine learning.
Organize your answer using these lesson pages → Context Engineering: From Manual to Autonomous Evolution
Q13Tech Colleague
"Meta-Harness iterates so many versions of a harness that the history just keeps growing — if you feed it all to the model, won't you blow out its own context first?"
🎯 What they're assessing
Using engineering feasibility to probe whether you actually read the method or are only reciting the concept. What they want to hear is how history is stored, how the optimizer reads it, and which step is the real cost bottleneck.
🧭 Answer framework
  1. Explain the storage structure: Each proposed harness is a dictionary on the filesystem: source, scores, trajectories, state updates. History does not go into the prompt — it all lands on disk.
  2. Explain the access method: The Proposer itself is a coding Agent; it uses grep and cat to read execution history on demand, fetching only the slice it needs, so nothing is stuffed wholesale into context.
  3. Explain the output shape: What it produces is a set of harness candidates on the Pareto frontier — under multiple objectives it keeps a family of solutions each strong in different ways, so you can pick by scenario.
  4. Admit the real bottleneck: What does not scale is the evaluation side: every harness revision needs a full trial run to score. Of the three context-optimization methods, Meta-Harness is the most general and the most computationally expensive.
⭐ Bonus pointAdd a design insight: the grep and cat the Proposer uses are all basic skills of a coding Agent — Meta-Harness invented no new mechanism for history access. This confirms that "code is the universal language for defining a Harness," so a model with strong coding ability is a natural fit as the optimizer.
Organize your answer using these lesson pages → Context Engineering: From Manual to Autonomous Evolution Workflow Design: From Manual to Automated Search
Q14Interviewer
"AFlow says it uses a game-playing algorithm to search workflows — how does it actually search? When does the search count as done?"
🎯 What they're assessing
They are checking your mechanical understanding of automatic workflow search. Only someone who can describe the graph representation and the search loop has actually read it; the termination condition is the detail most likely to stump you.
🧭 Answer framework
  1. Explain the representation first: A workflow is represented as a directed graph: nodes are LLM call actions, edges are logical operations in code (conditional branches, loops, data passing). The design problem thereby becomes a tree-search problem.
  2. Explain the search loop: The initial workflow is the root; a soft mix of score and uniform exploration picks the node to expand, balancing exploitation and exploration; the LLM generates modified variants (add/delete/edit nodes and edges); execute and evaluate, and if there is improvement, add it back to the search tree.
  3. Answer the termination condition: Loop until the top-k average score stabilizes, or the compute budget is exhausted.
  4. Give the positioning: Contrast ADAS, which relies on a meta-agent self-refine to improvise freely; AFlow uses MCTS for systematic search, converges more stably, and also beats hand design and ADAS in experiments.
⭐ Bonus pointClose with the course demo: starting from "Plan → Execute" at 52 points, a few rounds of search found a reflection-plus-verification combo that scored 78 — 50% above the hand-designed starting point. The optimal structure it found happens to look like a senior engineer's work habit.
Organize your answer using these lesson pages → Workflow Design: From Manual to Automated Search
Q15Interviewer
"AI Scientist claims it can do research and write papers on its own — what does its pipeline look like? How does it get through the review stage?"
🎯 What they're assessing
They are checking whether you can both describe the pipeline and stay sober. People who only marvel at full automation, and people who only say it is all hype, both fail.
🧭 Answer framework
  1. Name the pipeline: Propose a research idea → write code → run experiments → analyze results → write the paper → peer review. All six steps are LLM-driven, running a complete research loop end to end.
  2. Explain the review stage: Review brings in LLM-as-judge for quality control; the papers it produces are formally complete.
  3. Give the reality check: Being able to write a paper falls far short of being able to do science. System tests found six recurring failure modes: default-value preference from training data, implementation drift under execution pressure, memory and context degradation, over-optimism, insufficient domain intelligence, and weak scientific taste.
  4. Deliver the judgment: These are structural bottlenecks. An expert-designed Harness can indeed coordinate most stages of a research loop, but the quality of judgment inside those stages is still far behind.
⭐ Bonus pointPick one and go deep — that shows the most skill: "over-optimism" means the system claims the experiment significantly beats the baseline when the actual results are all noise; "insufficient domain intelligence" means it lacks the tacit craft that never gets written into papers but everyone in the lab knows. Those two are exactly what LLM-as-judge also struggles to catch.
Organize your answer using these lesson pages → Workflow Design: From Manual to Automated Search Future Challenges: Seven Gates of Self-Improvement
Q16Interviewer
"Everyone is doing synthetic data — why does Autodata claim its synthetic data is high quality? How does it know a question was well written?"
🎯 What they're assessing
They are checking your understanding of data-quality signals. Plenty of people talk vaguely about data quality; very few can name an operational criterion.
🧭 Answer framework
  1. Name the role system: Four roles collaborate: Challenger writes questions, Weak Solver and Strong Solver each try to solve them, Verifier / Judge arbitrates.
  2. Give the core criterion: The difficulty gap is the quality signal: keep only questions the strong solver can solve and the weak solver cannot.
  3. Explain why: Questions both can solve have no training value; questions neither can solve may themselves be bad questions. Questions that sit on the capability boundary have the most value for improving the model.
  4. Give the PM transfer: This idea of "using two capability bands to pinch out a difficulty belt" can be moved directly into evaluation-set construction and training-data tiering.
⭐ Bonus pointGive the positioning: Autodata and AI Scientist both belong to the representative works of hand-designed workflows, showing that before automatic search matures, expert-written multi-role workflows are still the main source of production value.
Organize your answer using these lesson pages → Workflow Design: From Manual to Automated Search
Q17Tech Colleague
"That STOP recursive improver — isn't it just letting the model edit its own prompt? Can this thing actually get better the more it edits?"
🎯 What they're assessing
Probing your mechanical understanding of the earliest example of recursive self-improvement. They expect you to say clearly what is actually being improved, and when recursion works versus when it fails.
🧭 Answer framework
  1. Correct the object first: STOP does not directly improve the solution s; what it repeatedly improves is the "improver" I itself that produces better solutions. The seed improver takes three inputs — an initial solution, a utility function, and a black-box model — and returns an improved solution.
  2. Explain the key to recursion: The improver itself is also text (a prompt or a piece of code), so the same improvement logic can act on the improver: I_t = I_{t-1}(û, I_{t-1}; M), letting today's self upgrade yesterday's self.
  3. Give the experimental answer: It really can get better — but there is a threshold. Driven by GPT-4 it improves continuously, and even automatically discovered classic optimization strategies such as genetic algorithms, decomposition improvement, multi-armed Prompt Bandit, simulated annealing, and Beam Search; driven by GPT-3.5 and Mixtral it actually regresses.
  4. Give the conclusion: A recursive structure only supplies the possibility of improvement; convergence is not guaranteed. Weaker models lack programming intuition at the meta level and only amplify noise.
⭐ Bonus pointSpell out the deeper meaning of "automatically discovering classic strategies": nobody preset genetic algorithms or Beam Search — the improver grew these structures itself. A strong enough model plus a recursive structure can reinvent decades of human optimization lore. That is the imaginative space of meta-level optimization.
Organize your answer using these lesson pages → Let Harness Improve Itself
Q18Interviewer
"Those trajectories where the Agent failed — what do you usually do with them? Just throw them away, or can you squeeze something out?"
🎯 What they're assessing
They are checking whether you have the awareness and methodology to treat failure as an asset. Answering "write an error log" is too shallow; what they want to hear is how structured failure mining is done.
🧭 Answer framework
  1. Name the method: The first phase of Self-Harness is called Weakness Mining: cluster failure trajectories into verifier-grounded failure modes, rising from individual cases to patterns.
  2. Explain the record spec: Each failure record is a three-piece set: the terminal-verifier-level failure reason, the causal state of the relevant Agent behavior, and the abstract Agent mechanism the trajectory exposed. With those three, a failure becomes addressable.
  3. Explain the downstream use: The proposal stage prioritizes addressable, repeating error patterns — fix one place, repair a whole class, instead of treating symptoms.
  4. Give the PM landing point: How to build a failure-trajectory library and along which dimensions to cluster it is infrastructure a PM can push right now; the method can be copied as-is.
⭐ Bonus pointPoint out the weight of the qualifier "verifier-grounded": failure attribution is anchored to a verifiable terminal signal, so the clustered patterns are not etiologies the model invented for itself. Without that anchor, the bigger the failure library, the more it misleads.
Organize your answer using these lesson pages → Let Harness Improve Itself
Q19Interviewer
"If you let an Agent modify its own configuration, how do you make sure that fixing this place didn't quietly make somewhere else worse?"
🎯 What they're assessing
They are checking the quality-gate design of the improvement loop. This is the critical link for whether self-improvement can go to production; anyone who cannot describe the validation mechanism has only heard of Self-Harness.
🧭 Answer framework
  1. Give the validation mechanism: Self-Harness phase three, Proposal Validation, validates candidate edits on held-in and held-out datasets and only accepts edits with no regression. Improvement is not allowed at the cost of existing capability.
  2. Explain the edit scope: The proposal itself is a bounded Harness edit. What the model receives is the editable surface, a summary of failure modes, records of passing behavior, and a history of already-tried edits — it is boxed in before it even acts.
  3. Give the experimental evidence: Experiments on Terminal-Bench-2 with MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 showed Self-Harness learning model-specific harness instructions for each model.
  4. State the product implication: The same framework produces different optimization paths for different bases, which means harness improvement is context-sensitive. When you swap models you cannot copy the harness config as-is — you have to rerun a round of optimization.
⭐ Bonus pointPoint out the role of the "history of already-tried edits" input: it stops the proposer from looping on the same idea, equivalent to giving self-improvement a dedup memory. Small designs like this often decide whether the loop converges.
Organize your answer using these lesson pages → Let Harness Improve Itself
Q20Boss
"If this keeps going, Agents will all modify themselves — what are we keeping all these engineers for, and you the product manager, going forward?"
🎯 What they're assessing
The boss is half joking, half serious, checking whether you can articulate the new place of humans in the loop — and whether you panic. The answer has to be specific about how job duties change; just shouting "humans are important" is useless.
🧭 Answer framework
  1. Give the direction first: The answer is in the last of the seven gates: humans move up the stack and stay in the loop. AI takes over the execution layer; human value moves up to setting goals, judging direction, and holding the line.
  2. Give concrete duties: Several things must be maintained by humans: evaluators and permission controls sit outside the improvement loop; the boundary of the editable surface is drawn by people; which problems are worth solving is defined by people.
  3. Give the evidence: Among the six failure modes of automated research, "weak scientific taste" and "insufficient domain intelligence" are exactly human strengths: judging whether a question is worth asking, and holding tacit knowledge that never makes it into documents.
  4. Close with a stance: Quote the line that ends the course: humans are an indispensable steering wheel in the system — never the bottleneck to be replaced. Oversight has to happen at the right time and the right level of abstraction.
⭐ Bonus pointTranslate it into a talent strategy: moving up the stack means evaluation systems and goal-definition ability become the team's scarcest assets, so hiring and training should shift toward "people who can set the exam and accept the work." That is the kind of org-level judgment the boss wants.
Organize your answer using these lesson pages → Future Challenges: Seven Gates of Self-Improvement
Q21Interviewer
"Darwin Gödel Machine is a flashy name — how does it actually evolve an Agent? Do you have any numbers you can put on the table?"
🎯 What they're assessing
They are checking your command of the most radical case of harness self-rewriting. Anyone can recite the name; what they want to hear is the action at each step of the evolution loop, and experimental numbers you can actually quote.
🧭 Answer framework
  1. Explain the object first: DGM explicitly evolves an editable harness code repository; the Agent is allowed to modify its own harness code — more radical than AlphaEvolve editing someone else's program.
  2. Explain the loop: Start from a coding agent in the pool; pick a parent by performance probability; the parent agent inspects its own benchmark evaluation logs and proposes an improvement; mutate a new agent; after evaluation, only enter the pool if performance is high enough; loop until a stopping condition.
  3. Quote the numbers: Based on Claude 3.5 Sonnet, SWE-bench Verified went from 20% to 50%, Polyglot from 14.2% to 30.7%, with no human intervention throughout.
  4. Add the tool surface: The toolset is startlingly plain: bash plus an editor (view / create / edit). The power is in the loop design — you do not need fancy tools.
⭐ Bonus pointCall out the inverse design of parent selection on its own: selection probability is inversely proportional to the number of children, down-weighting individuals that already have many descendants and encouraging exploration of neglected branches. That is equivalent to baking an anti-diversity-collapse mechanism into the evolution loop.
Organize your answer using these lesson pages → Evolutionary Search: Survival of the Strongest Harness
Q22Interviewer
"Prompt auto-optimization tools have been around for years — Promptbreeder, GEPA, those older works. Are they still worth reading?"
🎯 What they're assessing
They are checking your sense of technical history. Someone who can place older work on an evolution line is far more credible than someone who only chases new terms.
🧭 Answer framework
  1. Explain Promptbreeder: Uses a rich set of mutation operators to evolve task-specific prompts. The key innovation is meta-evolution: the mutation prompt itself is also improved through evolution — even the method of change is evolving.
  2. Explain GEPA: Combines reflection-based prompting with evolutionary search: the Agent first reflects on the shortcomings of the current prompt, then uses evolutionary operators to produce candidate improvements, and finally picks the best.
  3. Give the historical position: Both works are pioneers of prompt evolution and laid the foundation for later, larger-scale Harness evolution (AlphaEvolve, DGM). They proved that text can evolve; later work expanded the evolvable object to code and the entire harness.
  4. Answer whether they are worth it: Yes. Once you understand the kernel of meta-evolution, every new paper feels familiar.
⭐ Bonus pointDraw this thread: Promptbreeder evolves the mutator, STOP improves the improver, MCE optimizes the management mechanism — three eras, three names, the same kernel of putting the improvement method itself into the improvement loop. Anyone who can see the isomorphism will always learn new things fastest.
Organize your answer using these lesson pages → Evolutionary Search: Survival of the Strongest Harness Let Harness Improve Itself
Q23Tech Colleague
"Since a harness can already be auto-optimized, why not just put model weights into the same loop and train them together — one step to the finish. Do you think that's sound?"
🎯 What they're assessing
Probing whether you will get high on the radical proposal with them, or can point out the risks. Answering "sound" and answering "totally unworkable" both lose points; you need a balanced assessment of mechanism plus risk.
🧭 Answer framework
  1. Explain the mechanism first: That is exactly SIA's approach: three roles enter the same optimization loop. A Meta-Agent proposes a new harness design, a Task-Specific Agent executes tasks under the new harness, and a Feedback-Agent decides from the results whether the next step updates the harness or the model weights.
  2. Give the evaluation: The direction is interesting, but the evidence is still tentative. Two open challenges are unsolved: training stability, and the Goodhart effect — optimizing a proxy metric until the true objective degrades.
  3. Explain the risk with an analogy: Rewriting the race-car engine while also rewriting the track. Both sides change at once, and when something goes wrong you cannot attribute it to either.
  4. Give the recommendation: In the near term it is more stable to run harness optimization and weight training separately; only talk about joint optimization after each single-sided loop has converged and is controllable.
⭐ Bonus pointPoint out the essential difference between SIA and every prior method: earlier routes deliberately left the weights untouched and confined improvement to the Harness layer — that is exactly why they are verifiable and rollback-able. Putting weights into the loop is tearing out that safety pad, and that is the part that most needs scrutiny.
Organize your answer using these lesson pages → Evolutionary Search: Survival of the Strongest Harness From Scaffolding to Self-Improving Systems
Q24Interviewer
"AlphaEvolve discovering a new algorithm got a lot of attention — from a product perspective, what in its system design is worth copying?"
🎯 What they're assessing
They are checking whether you can extract reusable design elements from a star case. Anyone who can only tell the story of "AI discovered a new algorithm" will not pass this one.
🧭 Answer framework
  1. Explain the main loop: Maintain a pool of candidate programs; use a frozen LLM to generate code diffs that improve the programs; repeatedly evaluate subprograms and keep the best-performing ones. The model is never trained; all gains come from the search loop.
  2. Break down the prompt design: The evolution prompt is jointly composed of the parent program, evaluation results, instructions, and meta-information; EVOLVE-BLOCK markers explicitly circle the improvable region, constraining search to the designated range.
  3. Explain the meta-prompt: The instructions and context themselves also co-evolve; they are not fixed.
  4. Cite the ablation: The evolutionary process, the context prompt, the meta-prompt, whole-file evolution, and a stronger LLM — ablation showed each has an independent contribution. What is worth copying is exactly this practice of verifying every component's necessity.
⭐ Bonus pointEndorse the result, then add a boundary: Google really did use it to find a matrix-multiplication algorithm faster than anything humans knew. But the EVOLVE-BLOCK markers mean humans first draw where it is allowed to move; however strong the evolution, it still runs inside a circle people drew.
Organize your answer using these lesson pages → Evolutionary Search: Survival of the Strongest Harness
Q25Boss
"All those auto-optimization methods you mentioned last time sounded great — so how much compute do they actually burn? Give me the numbers."
🎯 What they're assessing
The boss wants cost structure and investment pacing; reciting papers is useless. What they want to hear is which step the money is spent on, which methods are expensive, and whether you start with the cheap ones or the expensive ones.
🧭 Answer framework
  1. Explain the cost bulk: Search-style methods concentrate spend on evaluation: every round is LLM inference plus code execution plus a benchmark, and the more generations you run the more you burn. How to balance compute efficiency (how many evaluations per generation) against evolutionary gain (how much each generation improves) is still an open question even in the literature.
  2. Give a cheap-to-expensive ranking: ACE-style structured context maintenance is the lightest — it runs inside the normal task flow; MCE's two layers require an extra skill-evolution loop; Meta-Harness needs a full trial run to score every revision, the heaviest of the three computationally.
  3. Give the investment pacing: First ship ACE-style maintenance to stabilize long tasks, then build an evaluation set that can run automatically; only after evaluation is standing do you assess which tier of search method to adopt.
  4. Give the stop-loss line: A single evaluation takes hours, the metric is subjective, or the budget is limited — hit any of those three and stay at the lightweight tier. These are also the unfit scenarios for evolutionary search that the course lists explicitly.
⭐ Bonus pointOffer a money-saving trick: Meta-Harness's TerminalBench-2 experiment was initialized from an already-strong harness; continuing the search from a good baseline is far cheaper than searching from scratch. The harness-tuning work we have already done by hand is the launch asset for future auto-optimization.
Organize your answer using these lesson pages → Evolutionary Search: Survival of the Strongest Harness Context Engineering: From Manual to Autonomous Evolution
Q26Interviewer
"Everyone stores success cases as an experience library — should we also store the failed dead ends? Won't storing them steer the Agent off course?"
🎯 What they're assessing
They are checking the depth of your memory-content design. Most people only think of storing successes; the value of negative results and how to store them is one of this chapter's most counterintuitive test points.
🧭 Answer framework
  1. Take a position first: Yes, store them. Knowing what does not work is as important as knowing what does; a research Harness should make failed attempts and dead ends easy to save and retrieve.
  2. Explain why it is hard: Scientific literature is heavily biased toward success stories; LLMs may be poor at deciding when to abandon a hypothesis or honestly report a negative result. Automated research systems inherit that bias as-is.
  3. Explain how to store without steering off course: The use of negative results is pruning: label clearly what was tried, how it failed, and why, so when the Agent retrieves it it does not walk the same dead end again. Self-Harness feeding the proposer a "history of already-tried edits" is the same logic.
  4. Tie it back to a failure mode: Automated research's "over-optimism" problem (claiming a significant beat over the baseline when it is all noise) is exactly the symptom of a system that lacks the habit of honestly recording negative results.
⭐ Bonus pointClose with a counterintuitive insight: a negative-result library is both a map that prevents repeat exploration and a control group for the review stage. A team whose experience library only stores successes is forcing the Agent to crash into every dead end all over again.
Organize your answer using these lesson pages → Future Challenges: Seven Gates of Self-Improvement Let Harness Improve Itself
Q27Interviewer
"Suppose tomorrow you are put in charge of a self-improving Agent project, and the first thing you write is a risk list — which items would you put on it?"
🎯 What they're assessing
They are checking your systematic command of this chapter's closing framework. Naming two or three scattered risks is easy; what they want to hear is a complete classification map, and the solution direction for each class.
🧭 Answer framework
  1. Give the classification framework first: The seven gates fall into four classes: evaluation-related, data and memory, safety and stability, and the human role. Report by class and you are less likely to miss one.
  2. Cover evaluation and memory: Weak and fuzzy evaluators (many goals have no fast, precise verifier, so the feedback signal is blurry); context and memory lifecycle (memory demand explodes with autonomy, and context engineering should become a core part of intelligence itself); negative results (failed attempts must be easy to save and retrieve).
  3. Cover the three safety gates: Diversity collapse (candidates crowd into minor variants and innovation stops), reward hacking (overfitting tests, gaming the judge model, exploiting benchmark loopholes), and long-term success (chasing test passes while ignoring maintainability, ownership boundaries, migration cost, and backward compatibility).
  4. Close on the human role: The seventh gate is the place of humans: provide oversight at the right time and the right level of abstraction. These seven are fundamental system-design challenges — do not treat them as ordinary engineering issues and just drop them into the sprint.
⭐ Bonus pointState the inner order of the four classes: the evaluator is the source of the feedback signal; if the signal is wrong, everything downstream is wrong, which is why "weak evaluators" sits first among the seven gates. The risk list should be ordered by that priority: first solve how you score, then talk about how you improve.
Organize your answer using these lesson pages → Future Challenges: Seven Gates of Self-Improvement
Q28Tech Colleague
"These papers you keep citing get more mystical one after another. Honestly — which of them can we use next quarter, and which shouldn't we even think about for five years?"
🎯 What they're assessing
The tech colleague wants a reliable maturity layering; hyping everything or dismissing everything both lose trust. Get this question right and the collaboration after it goes smoothly.
🧭 Answer framework
  1. The "use it now" layer: The three design patterns (automation loop, filesystem memory, sub-Agent parallelism) are already standard on Claude Code, Codex, and Cursor, covering 90% of the architectural decisions in the strongest Agent systems — just build to that.
  2. The "try next quarter" layer: ACE-style structured context maintenance. Generator, Reflector, and Curator are a clear engineering pipeline; the prerequisite is having an evaluation set first to verify convergence.
  3. The "adopt if conditions hold" layer: Search methods like AFlow and DGM require automated evaluation, quantifiable fitness, and sufficient compute. Coding tasks benefit first; DGM's SWE-bench numbers were produced under exactly those conditions.
  4. The "keep watching" layer: SIA joint optimization still has unsolved training stability, and Meta-Harness is the most computationally expensive; both are still in a tentative-evidence stage — following the papers is enough.
⭐ Bonus pointGive a general criterion: how far a method is from landing basically depends on how deeply it relies on a "fast, reliable evaluator," and how mature the evaluator in your hands already is. Evaluation infrastructure is the hidden critical path.
Organize your answer using these lesson pages → Three Design Patterns for Harness Context Engineering: From Manual to Autonomous Evolution Evolutionary Search: Survival of the Strongest Harness
Q29Interviewer
"This diversity collapse you mentioned — what does it actually look like? The system is still gaining score. How do I know it has already quietly collapsed?"
🎯 What they're assessing
They are checking your understanding of an evolutionary system's chronic disease. This failure mode has no short-term symptoms; only if you can describe its appearance, its antidote, and how to monitor it do you truly understand it.
🧭 Answer framework
  1. Explain the pathology: Evolutionary algorithms and RL loops naturally tend to exploit known high-reward patterns. Short-term scores look great, and the problem is hidden.
  2. Explain the symptoms: Every candidate in the population collapses into a minor variant of the same solution; each new generation looks more like the last, and innovation stops. The course analogy is apt: if the whole class copies the top student's homework, scores look good, but no one will ever come up with a new solution again.
  3. Give the antidote: You need dedicated mechanisms to keep the solution space from collapsing: diversity rewards, archive preservation.
  4. Give a monitoring suggestion: Watching the score curve alone will not reveal collapse; you also have to measure how different the solutions inside the population are. Difference trending to zero is the alarm — it arrives far earlier than a score drop.
⭐ Bonus pointDistinguishing it from reward hacking shows the most skill: reward hacking is an inflated score with no real capability gain; diversity collapse is a real score with stalled innovation. The two diseases have opposite symptoms and the same root: optimization pressure only recognizes one number.
Organize your answer using these lesson pages → Future Challenges: Seven Gates of Self-Improvement Evolutionary Search: Survival of the Strongest Harness
Q30Boss
"The code AI writes passes every test and nothing broke in production — can we skip code review? Human review is too slow."
🎯 What they're assessing
The boss is looking for a cost cut, checking whether you can articulate the gap between "tests passed" and "repo health" and still offer an alternative. Just saying no is useless.
🧭 Answer framework
  1. Give the root cause first: This is "long-term success" among the seven gates: current optimization targets are too short-term. A coding Agent can finish the task in front of it, but is not clear enough on how to protect the long-term health of the repo.
  2. List what tests cannot catch: Standard sandbox RLVR training rarely captures maintainability, ownership boundaries, migration cost, or backward compatibility. Those are exactly the gates human review is holding.
  3. Give the risk picture: An Agent that only chases passing tests may bury time bombs in technical debt. The day they go off will not be written in any test report.
  4. Give a compromise: Review can change shape: light review for routine changes, focused review for architectural ones. Humans move up, watching ownership boundaries and long-term structure, and leave line-by-line checking to tools.
⭐ Bonus pointHelp the boss recalibrate expectations: even the frontier of self-improvement research lists the difficulty of measuring long-term health as one of the fundamental challenges, which means this is not just a process problem on our team. In the near term human review cannot be skipped; what you can save is the layer — spend human time on the part machines cannot measure.
Organize your answer using these lesson pages → Future Challenges: Seven Gates of Self-Improvement
One Last Tip
This chapter has the most cutting-edge knowledge, and it is also the easiest to bluff with. The right way to use these 30 questions is to say your answer out loud — to a colleague, a friend, or a recording. Where you stumble is where you thought you understood but didn't. Click the linked course pages to fill those gaps.