20 Questions — Domain 5 : Context Management & Reliability

Expand any question to reveal the correct answer and explanation.

1. 1 An architect is designing a high-volume support system using Claude 4 Sonnet. The system prompt is 850 tokens, and a set of static policy documents adds 300 tokens. If prompt caching is enabled, how will the first 500 requests be billed for these specific tokens?

   Consider the cumulative token count of the stable prefix and the model-specific threshold requirements.

   The 1,150-token prefix will be billed at $100\%$ input cost for the first request and $~10\%$ for the subsequent 499 requests.

   Since the combined prefix of 1,150 tokens exceeds the 1,024-token minimum threshold for Sonnet, it is eligible for caching after the first hit.

   • ✗ All 500 requests will be billed at $100\%$ input cost because the system prompt itself is under 1,024 tokens.

     Caching applies to the total stable prefix provided, not just individual components like the system prompt, as long as the total exceeds the threshold.

   • ✗ The first request will be billed at $100\%$ and subsequent hits at $10\%$, but only for the policy documents since they are separate from the persona.

     The cache breakpoint is placed at the end of the entire stable block; separate components are cached as a single prefix if ordered correctly.

   • ✗ All 500 requests will be billed at $100\%$ because Sonnet requires a minimum of 2,048 tokens to trigger prompt caching.

     The 2,048-token threshold applies specifically to the Haiku model tier, whereas Sonnet and Opus require 1,024 tokens.

2. 2 A production agent is processing a 180,000-token technical manual. Developers observe that while the agent accurately summarizes the introduction and the conclusion, it consistently misses critical specifications located in the middle of the document. Which architectural adjustment is most likely to resolve this?

   Identify a strategy that manages how the model's attention is distributed across massive amounts of data.

   Decompose the processing into focused, sequential passes over smaller segments of the document.

   This addresses the 'lost in the middle' effect where LLMs exhibit lower retrieval accuracy for information positioned in the center of very large context windows.

   • ✗ Increase the $max\_tokens$ parameter to ensure Claude has enough output space to list every specification found.

     While $max\_tokens$ controls output length, the issue is an input attention retrieval failure, not a truncation of the response.

   • ✗ Enable 'Extended Thinking' on the model to allow for deeper internal reasoning during the document scan.

     Extended thinking improves complex logic but does not inherently solve the architectural attention bias found in long-context retrieval.

   • ✗ Switch from a synchronous API call to the Message Batches API to allow more time for the model to process the middle section.

     The Message Batches API offers cost savings and higher throughput but does not change the underlying model attention mechanisms or context retrieval performance.

3. 3 An automated claims agent encounters a customer who explicitly states, 'I want to talk to a person,' even though the request is within the agent's capability and policy guidelines. According to the CCAR-F framework, what is the best architectural response?

   Focus on the priority given to explicit user requests versus model-driven autonomy.

   Escalate the session immediately to a human agent without further autonomous investigation.

   The framework mandates honoring explicit customer requests for human intervention immediately to maintain trust, regardless of the agent's self-assessed capability.

   • ✗ Acknowledge the frustration and offer a resolution first, escalating only if the customer repeats the request for a human.

     While common in some workflows, the CCAR-F standard prioritizes immediate escalation upon the first explicit human agent request.

   • ✗ Perform a self-reported confidence check; if confidence is above $90\%$, proceed with the automated resolution to reduce human load.

     Explicit user preference overrides autonomous confidence scores, which are often poorly calibrated proxies for whether an escalation is necessary.

   • ✗ Use a sentiment analysis hook to verify if the user is truly angry before triggering the expensive human escalation path.

     Sentiment-based escalation is considered an unreliable proxy for case complexity and user needs compared to explicit statements.

4. 4 In a multi-agent research system, a web search subagent encounters a 403 Forbidden error while attempting to access a specific source. How should this failure be propagated to the coordinator agent to ensure reliability?

   Look for the approach that maximizes the coordinator's ability to diagnose and recover from the failure.

   Return a structured error object containing the $errorCategory$, the specific URL that failed, and any partial findings retrieved from other sources.

   Structured context allows the coordinator to make intelligent decisions, such as retrying an alternative source or proceeding with partial data.

   • ✗ Suppress the error and return an empty successful result so the synthesis subagent can proceed without interruption.

     Silently suppressing errors is an anti-pattern that hides valuable context and leads to incomplete or inaccurate final research reports.

   • ✗ Throw a top-level exception that terminates the entire workflow to prevent the synthesis of potentially biased or incomplete data.

     Terminating the entire workflow is often unnecessary if recovery strategies or partial findings are available to fulfill the user's request.

   • ✗ Return a generic 'Search Unavailable' string in the tool output to trigger the model's natural language error handling.

     Generic error messages lack the technical specificity (like distinguishing transient vs. permission errors) needed for optimal coordinator recovery logic.

5. 5 A developer sets a cache breakpoint on a 5,000-token documentation block for an agent. After the first hit, the cache is unused for 7 minutes. Which statement accurately describes the status of the cache when the next request arrives?

   Recall the specific Time-To-Live duration and the refresh logic for ephemeral caching.

   The cache has expired and the developer will be billed at $100\%$ input cost to re-cache the block.

   The ephemeral cache TTL is 5 minutes and is only refreshed upon a successful cache hit; otherwise, it is evicted.

   • ✗ The cache is still valid because the TTL for Sonnet/Opus tier models is 30 minutes.

     The documentation specifies a 5-minute TTL for cached blocks, which is consistent across the supported model tiers.

   • ✗ The cache is valid, as the 5-minute TTL only applies if the total cache capacity of the regional cluster is exceeded.

     The TTL is a standard time-to-live parameter for the ephemeral cache and is not dependent on overall cluster capacity in the exam context.

   • ✗ The cache is expired, but the developer is billed at $50\%$ cost because the 'warm' storage persists for 24 hours.

     There is no 'warm' discount tier mentioned; once the ephemeral cache expires, the next request is billed as a standard full-cost input.

6. 6 You are evaluating an extraction system that has achieved an overall aggregate accuracy of $98\%$. Before automating the process, which action is most critical to prevent 'error leakage' in production?

   Think about how high overall percentages can hide significant problems in specific subsets of data.

   Segment the accuracy metrics by document type and specific fields to detect if certain categories have catastrophic failure rates.

   Aggregate metrics can mask poor performance in minority segments (e.g., $99\%$ on invoices but $50\%$ on handwritten forms), making stratified analysis essential.

   • ✗ Implement a 3nd-pass LLM review where a second Claude instance checks the first instance's output against the aggregate success rate.

     Reviewing against an aggregate rate does not identify segment-specific failures; only stratified data analysis can reveal those risks.

   • ✗ Trust the aggregate score but set the model's self-reported confidence threshold to $\ge 95\%$ for all extractions.

     Self-reported confidence is often poorly calibrated and does not substitute for empirical validation across different document segments.

   • ✗ Perform a simple random sample of 100 documents daily to ensure the $98\%$ accuracy remains stable over time.

     Simple random sampling under-represents rare document types, potentially missing high error rates in specific high-risk categories.

7. 7 During an extended development session with Claude Code, the conversation context becomes bloated with verbose terminal outputs and redundant logs. Which command should the architect use to preserve the current state while minimizing token usage for future turns?

   Look for a terminal command specifically intended for session log optimization.

   Use the $/compact$ command to summarize the active session history into a more token-efficient format.

   The $/compact$ command reduces the active session log by discarding redundant tokens while retaining summaries of past modifications and current goals.

   • ✗ Execute $/memory$ to move all current session details into a long-term vector database.

     While $/memory$ manages persistence, the $/compact$ command is specifically designed for context window optimization within an active session.

   • ✗ Run $fork\_session$ to create a new session that only inherits the last three turns of conversation.

     Forking creates a new branch of context but summary-based compaction is a more integrated way to manage the current session's growth.

   • ✗ Apply a $cache\_control$ header to the entire conversation history to prevent re-processing tokens.

     Prompt caching reduces cost and latency for processed tokens but does not address the problem of approaching the context window limit.

8. 8 An architect is implementing a 'Progressive Summarization' pattern to manage a long-running customer interaction. What is the primary reliability risk associated with this technique according to the CCAR-F syllabus?

   Consider the trade-off between reducing token volume and maintaining exact data precision.

   The model may condense critical deterministic values like dates, amounts, and specific customer expectations into vague generalizations.

   Summarization inherently loses granularity; specific figures or constraints vital for business logic are often sacrificed for brevity.

   • ✗ Summarization increases the 'lost in the middle' effect because the summary block is always placed in the center of the prompt.

     Summarization actually mitigates the 'lost in the middle' effect by reducing the total volume of tokens the model must attend to.

   • ✗ The model will lose the ability to identify the user's current persona if the original persona description is summarized.

     User personas are typically kept in the system prompt or static prefix, which would not be part of the dynamic conversation summarization.

   • ✗ Summarized text is billed as 'Output Tokens' at a higher rate, negating the cost benefits of context reduction.

     The cost of creating a summary is a one-time charge; the long-term benefit is reducing the input token count of every subsequent turn.

9. 9 Which mechanism is recommended for calibrating human review thresholds for an extraction system where accuracy is critical?

   Identify the technique that uses empirical performance data to map model output to human intervention needs.

   Generate field-level confidence scores and calibrate them using labeled validation sets to determine routing thresholds.

   Calibrating confidence scores against ground-truth validation data ensures that 'high confidence' actually correlates with 'high accuracy'.

   • ✗ Set the escalation threshold based on the model's self-reported probability distribution for the entire JSON object.

     LLM self-reported probability or confidence is often uncalibrated; field-level validation is more granular and reliable.

   • ✗ Route the first $10\%$ of all incoming documents to humans and automate the remaining $90\%$ regardless of confidence.

     This arbitrary sampling does not account for the complexity or risk of individual documents, potentially allowing high-risk errors to pass.

   • ✗ Use a sentiment analysis tool on the source document to detect if the text is 'confusing' or 'ambiguous' for the model.

     Sentiment analysis is designed for tone and emotion, not for technical ambiguity or the likelihood of extraction error.

10. 10 A multi-agent system uses a 'Scratchpad' file to maintain state. What is the primary architectural purpose of this scratchpad when transitioning between an investigation phase and a synthesis phase?

    Think about how an agent can keep track of essential info when moving between different tasks or agents.

    To persist key findings across context boundaries and prevent them from being lost due to context degradation or subagent isolation.

    Scratchpads serve as a localized 'memory' that persists specific, high-value findings even when context windows are cleared or subagents are swapped.

    • ✗ To provide a secure, encrypted storage area for customer PII that subagents cannot access.

      Scratchpads are generally for state and finding persistence; security isolation is handled through subagent delegation and tool permissions.

    • ✗ To act as a temporary cache for prompt prefixes to reduce the costs of the synthesis turn.

      Prompt caching is an API-level feature; a scratchpad is an agent-level pattern for managing reasoning and data persistence.

    • ✗ To allow the coordinator to store its entire raw conversation history for auditing purposes.

      Conversation history is usually handled by the application database or logging; the scratchpad is for summarized findings used during execution.

11. 11 An architect needs to implement prompt caching for a multi-turn conversation. Where should the `cache_control` block be placed to maximize efficiency?

    Recall how the cache breakpoint defines what content is stored and reused.

    At the very end of the stable prefix (e.g., system prompt and standard documents) before any dynamic conversation turns.

    Placing the breakpoint at the end of the static content ensures that all subsequent dynamic messages can leverage the cached prefix.

    • ✗ On every individual message in the conversation history to ensure the entire session is cached.

      Placing breakpoints on every message is inefficient and unnecessary, as only the prefix leading up to a breakpoint is cached.

    • ✗ Only on the very first user message, as the system prompt is automatically cached by Anthropic's backend.

      Caching is not automatic; it must be explicitly defined using $cache\_control$ blocks on the desired content.

    • ✗ At the beginning of the system prompt to allow the model to process instructions faster.

      A breakpoint at the beginning of the prompt would cache nothing, as it only caches content that appears before the breakpoint in the request structure.

12. 12 In a research synthesis scenario, two subagents return conflicting figures for a company's revenue ($500M$ vs $550M$). What is the recommended 'grounded synthesis' approach?

    Consider the principle of preserving source provenance rather than forcing a single, potentially incorrect answer.

    The coordinator should preserve both values with their respective source attribution and methodological context in the final report.

    Grounded synthesis avoids arbitrary selection and preserves provenance, allowing the end-user to evaluate the conflicting evidence.

    • ✗ The coordinator should average the two figures ($525M$) to present a neutral, high-probability estimate.

      Averaging fabricates a data point not supported by either source, destroying data integrity and source attribution.

    • ✗ The coordinator should use a secondary Haiku instance to pick the figure from the more reputable source automatically.

      Automatically picking one value without presenting the conflict leads to data loss and ignores potential validity in the discarded source.

    • ✗ The coordinator should prioritize the value with the more recent publication date and discard the older one.

      Recency is a heuristic, not a guarantee of accuracy; both values should be presented to maintain a reliable audit trail.

13. 13 When building an escalation workflow, a support agent encounters a customer query that is not covered by any existing policy in the company knowledge base. What is the most reliable architectural trigger for human intervention?

    How can the agent be taught to recognize when it lacks the information needed to proceed safely?

    Identify the 'policy gap' as an explicit escalation criterion within the system prompt.

    Explicitly instructing the model to escalate when policy is ambiguous or silent is more reliable than relying on the model to 'guess' or hallucinate a fix.

    • ✗ Monitor the model's self-reported confidence score; a gap will naturally trigger a low score ($< 50\%$).

      Self-reported confidence is poorly calibrated and the model may confidently hallucinate a policy that does not exist.

    • ✗ Set a hard cap of 3 turns; if the issue isn't resolved by then, assume a policy gap exists and escalate.

      Turn caps are arbitrary and may escalate simple issues that just take time, while failing to escalate complex policy gaps quickly enough.

    • ✗ Analyze the query for keywords like 'manager' or 'supervisor' to determine if an escalation is required.

      Keyword analysis is reactive and doesn't address the structural issue of a missing policy that the agent cannot resolve autonomously.

14. 14 A team processes $10,000$ requests daily. By implementing prompt caching on a $2,500$-token system prompt for Sonnet, they achieve a $90\%$ reduction in input costs for cached tokens. If the base cost is $3.00$ per million input tokens, what is the approximate daily savings for this cached portion?

    Calculate total daily tokens (in millions) and multiply by the per-million savings offered by the cache discount.

    $~\$67.50$

    $2,500$ tokens $\times$ $10,000$ requests is $25$ million tokens. $90\%$ savings on $3.00$ per million is $2.70$ saved per million. $25 \times 2.70 = 67.50$.

    • ✗ $~\$6.75$

      This calculation likely misses a decimal place or undercounts the total volume of daily tokens processed across the $10,000$ requests.

    • ✗ $~\$250.00$

      This value represents a significant overestimate and does not align with the provided $3.00/M$ pricing and $90\%$ discount model.

    • ✗ $~\$2.70$

      This figure represents the savings per million tokens, not the total daily savings for the $25$ million tokens being processed.

15. 15 In a data extraction workflow for chronological medical records, an architect adds a mandatory 'publication_date' field to the JSON schema. What is the primary purpose of this from a reliability standpoint?

    Think about how an agent might view different values for the same metric across different points in time.

    To provide temporal metadata that prevents synthesis agents from misinterpreting chronological progression as data contradiction.

    Without temporal context, a newer record (e.g., 'Weight: 180lbs') might be seen as conflicting with an older one (e.g., 'Weight: 195lbs') rather than a change over time.

    • ✗ To enable the prompt caching mechanism to prioritize the most recent documents in the server's ephemeral storage.

      Prompt caching is based on token string matching and hierarchy, not on the semantic date values contained within the text.

    • ✗ To verify that the document was created after the model's knowledge cutoff date to prevent hallucinations.

      The knowledge cutoff is a model property; extracting a date helps with document processing but does not change the model's training data.

    • ✗ To reduce the 'lost in the middle' effect by sorting documents by date before passing them to the context window.

      Sorting may help with logic but doesn't inherently solve the architectural attention retrieval problems found in very long prompts.

16. 16 A coordinator agent needs to resume a research task after a system crash. Which data structure is recommended to allow the agent to resume accurately without redundant execution?

    Look for a method that provides a high-level, structured overview of the work already performed.

    A structured 'Manifest' or state export containing completed subtasks, extracted findings, and the next planned steps.

    Injecting a structured manifest into the new session allows the agent to understand exactly what was finished and where to restart.

    • ✗ The entire raw log of the previous session's API requests and tool outputs.

      Passing raw logs is token-intensive and inefficient; a summarized or structured manifest is more effective for state recovery.

    • ✗ A list of the URLs that were successfully crawled by the search subagent.

      While helpful, a list of URLs doesn't capture the actual findings or the coordinator's reasoning state, leading to redundant analysis.

    • ✗ The final checkpoint of the coordinator's self-reported confidence score.

      Confidence scores do not provide the factual state or data required to resume a multi-step engineering task.

17. 17 An architect notices that a tool for looking up customer orders returns $50$ different data fields per order, even though the agent only needs $5$ for the current task. What Context Management principle is being violated here?

    Think about the efficiency of the data being passed into the model's limited context space.

    Avoid context bloat by trimming verbose tool outputs to only include relevant information.

    Returning unnecessary fields consumes tokens, increases latency, and can distract the model from the most relevant information.

    • ✗ Always pass the complete raw output to ensure the model has full context for any potential follow-up questions.

      This is an anti-pattern; excessive information degrades performance and increases costs. Follow-ups should be handled with new targeted tool calls.

    • ✗ Implement prompt caching to mitigate the costs of the redundant $45$ fields.

      Prompt caching reduces cost for static prefixes, but tool outputs are dynamic and vary per order, making them unsuitable for caching.

    • ✗ Use the $/compact$ command to summarize the order data as soon as it enters the context window.

      Summarizing at the application layer before sending to Claude is more efficient than forcing Claude to process and then summarize the bloat.

18. 18 Your Claude-powered chatbot occasionally drifts outside its defined topic constraints. What is the most reliable architectural control for ensuring reliability?

    Look for a solution that provides a deterministic check on the model's generated content.

    Implement an output classifier using a second, low-latency model to verify the response stays within scope before showing it to the user.

    An independent verification pass (output guardrail) is significantly more reliable than relying solely on instructions in the system prompt.

    • ✗ Add a redundant section to the system prompt repeating the topic constraints in all-caps.

      Prompt-based constraints have a non-zero failure rate; repetitive instructions often provide diminishing returns in reliability.

    • ✗ Increase the $temperature$ parameter to $1.0$ to give the model more creative room to interpret the scope.

      Higher temperature increases randomness and variability, which typically decreases reliability and adherence to strict scope constraints.

    • ✗ Reduce the $max\_tokens$ to $50$ to force the model to be brief and stay on topic.

      Forced brevity does not prevent the model from going off-topic; it only results in truncated off-topic responses.

19. 19 A developer wants to use prompt caching for a collection of few-shot examples that change every few days. How many examples must be included to meet the caching threshold if each example is $150$ tokens and the system prompt is $500$ tokens?

    Calculate how many total tokens are needed to pass the minimum cache threshold for higher-tier models.

    At least $4$ examples ($600$ tokens) to bring the total prefix to $1,100$ tokens.

    The combined prefix ($500 + 600$) exceeds the $1,024$-token threshold required for Sonnet and Opus caching.

    • ✗ Only $1$ example, because individual $150$-token blocks can be cached if they are marked as ephemeral.

      Individual blocks must meet the minimum size ($1,024$ or $2,048$ tokens) unless they are part of a continuous prefix that meets the total threshold.

    • ✗ At least $11$ examples to reach the $2,048$-token minimum required for standard caching across all Claude models.

      The $2,048$ threshold is only for the Haiku model tier; $1,024$ is sufficient for the higher-tier Sonnet and Opus models.

    • ✗ None, as few-shot examples are dynamic content and are never eligible for caching.

      Few-shot examples are eligible for caching as long as they are part of the static prefix provided before the dynamic user message.

20. 20 When monitoring a support agent, you notice it frequently asks for customer clarification when multiple matches are found for a single identifier (e.g., three customers named 'John Smith'). Why is this considered a 'Reliability' best practice?

    Focus on the risks of the model making a 'best guess' in a situation with multiple valid options.

    Heuristic selection by the LLM is unreliable and leads to severe data integrity errors; explicit clarification is required.

    Models should not guess when multiple valid matches exist; they must be instructed to request more identifiers to ensure the correct record is accessed.

    • ✗ Asking questions reduces the input token count of the next turn by clearing the search results from the buffer.

      Requesting clarification actually increases the turn count and total token consumption, but is necessary for accuracy.

    • ✗ Each question asked by the agent refreshes the prompt cache TTL for another 5 minutes.

      Refreshing the cache is a side effect; the primary goal of clarification is ensuring accuracy and preventing incorrect operations.

    • ✗ Clarification reduces the model's 'temperature' for the next turn, making the eventual selection more deterministic.

      Clarification doesn't change model parameters like temperature; it simply provides the model with the necessary data to make a correct decision.