APIAdvancedClaude API
Before an A/B test, define the primary metric and sample size
Comparing 50 sessions against 50 is a confirmation, not a test: without a metric fixed in advance you will always find a 'winner'.
Pre-specify the primary metric and compute the minimum sample according to the effect you expect and the confidence you need. A small change to a low-risk FAQ tolerates less rigor than one that touches sensitive decisions. Complement it with per-request observability: aggregate metrics can look healthy while a small fraction spends almost the whole budget.
Example
Design template: - Hypothesis: ... - Primary metric (just one): ... - Minimum effect that matters: ... - Minimum sample and duration: ... - Decision criterion: ...