Pass the Claude Certified Architect Professional Exam (CCAR-P)

1. A customer-facing assistant streams its answers token by token. After a release, end-to-end completion time is statistically unchanged, but users consistently report that the new version 'feels slower.' The team must add one latency metric to the service SLA.Which latency metric most directly captures what these users are experiencing?

   • A. Time to first token (TTFT), measured from request submission to the first streamed token.
   • B. Baseline latency for the complete response, averaged across all requests over the release window.
   • C. Output tokens per minute (OTPM) consumed against the organization's rate limit for this model class.
   • D. Average input tokens per request, tracked per endpoint.

   Answer: A. With streaming, perceived responsiveness is governed by how quickly the first token appears, which is exactly what TTFT measures; a longer preamble, a bigger cached prefix miss, or added pre-screening can push TTFT out while leaving total time flat. B is the whole-response measure the team has already confirmed is unchanged, so it cannot explain the complaint. C is a throughput figure used for rate-limit accounting, not a user-perceived latency measure. D is an input-size measure that correlates loosely with latency but is not a latency metric at all.

   Exam domain: Evaluation, Testing & Optimization

2. An internal agent has a stated objective of 'responses in under four seconds.' Mean latency is 2.1 seconds and the team keeps lowering it, yet a steady stream of complaints continues from users who wait far longer.How should the latency objective be expressed so that it reflects the experience users are complaining about?

   • A. Keep tracking mean latency and continue driving it down.
   • B. Track total tokens generated per response as a proxy for how long each response took the model to produce.
   • C. Express the objective as a percentile, such as p95 under four seconds, and track the tail explicitly.
   • D. Express the objective as the maximum latency observed anywhere in the month.

   Answer: C. Complaints come from the tail of the distribution, and a percentile objective is the standard way to make the tail a first-class, actionable target. A keeps optimizing the statistic that is already hiding the problem. B is a proxy that ignores queueing, retrieval, tool round trips, and time to first token. D is technically a tail measure but a single outlier failing the whole month makes it unactionable and encourages gaming rather than improvement.

   Exam domain: Evaluation, Testing & Optimization

3. An invoice-extraction pipeline emits fourteen fields per document. The only accuracy metric is whole-document exact match, currently at 61 percent. The team cannot tell where to spend its next sprint.Which change to the metric set best restores diagnostic value?

   • A. Replace exact match with an LLM-graded one-to-five quality score for each document.
   • B. Keep document-level exact match but sample far fewer documents so a human can review each one closely.
   • C. Report per-field accuracy for all fourteen fields alongside the existing document-level pass rate.
   • D. Replace exact match with a fuzzy string-similarity threshold applied to all fields concatenated together.

   Answer: C. Decomposing a composite pass rate into per-field rates shows immediately which two or three fields are dragging the document score down, while keeping the document-level number as the business-facing metric. A swaps a deterministic, code-gradable signal for a noisier and more expensive one on a task where exact match works. B reduces volume and leans on human grading, both of which run against documented eval guidance. D concatenates the fields, which destroys exactly the per-field attribution the team needs.

   Exam domain: Evaluation, Testing & Optimization

4. A team proposes that its only safety control be a carefully written system prompt instructing the model to refuse prohibited requests.What is the principal weakness of that design?

   • A. It is a single layer, where guidance calls for chained safeguards: input validation, screening, action gating, and monitoring.
   • B. System prompts cannot contain safety instructions, which must instead be supplied through tool definitions and their descriptions.
   • C. System prompt instructions are ignored entirely once tool use is enabled, so the policy never applies.
   • D. A long system prompt raises cost, and cost is the primary safety concern for a high-volume design.

   Answer: A. Prompt-level instruction is a legitimate layer and the documented guidance includes it, but it is one layer and it is the layer an adversarial input is specifically trying to talk past. The documented posture is chained safeguards in depth: validate input, screen it, gate sensitive actions, and monitor continuously. B and C are simply untrue. D confuses a cost consideration with a safety argument.

   Exam domain: Governance, Safety & Risk Management

5. Monitoring shows a small number of accounts repeatedly submitting variations of the same prohibited request, probing for a phrasing that gets through.Which documented control addresses this pattern specifically?

   • A. Raise the model's effort setting so it reasons more carefully about each request.
   • B. Rewrite the system prompt after each incident to name the specific phrasing that was attempted and forbid that exact phrasing explicitly.
   • C. Disable the assistant for all users until the pattern stops.
   • D. Respond to repeat offenders by throttling or banning the accounts and informing them that their actions violate the usage policies.

   Answer: D. Adjusting responses for repeat offenders, including throttling or banning them and telling them why, is a documented mitigation aimed precisely at systematic probing, which is an account-behavior problem rather than a per-message one. A does not change what the model is permitted to do. B is an endless game of catch-up against paraphrase and grows the prompt without bound. C punishes every legitimate user for the behavior of a few.

   Exam domain: Governance, Safety & Risk Management

Want more? Finish the quiz above to unlock 15 more free questions, or open the full CCAR-P practice app.