Tips // Claude in practice

Tips for working better with Claude

One short tip at a time, with an example you can copy. Move through them at random or search for what you need.

APIAdvancedClaude API

Before an A/B test, define the primary metric and sample size

Comparing 50 sessions against 50 is a confirmation, not a test: without a metric fixed in advance you will always find a 'winner'.

Pre-specify the primary metric and compute the minimum sample according to the effect you expect and the confidence you need. A small change to a low-risk FAQ tolerates less rigor than one that touches sensitive decisions. Complement it with per-request observability: aggregate metrics can look healthy while a small fraction spends almost the whole budget.

Example
Design template:
- Hypothesis: ...
- Primary metric (just one): ...
- Minimum effect that matters: ...
- Minimum sample and duration: ...
- Decision criterion: ...
#ab-testing#observability#metrics