APIBasicClaude API
Do not test the exact text of a Claude response
The same prompt can give different wordings that are all correct; assert properties (field present, value in range), not strings.
The model samples each token from a distribution, so there is no guarantee of identical output, not even at temperature 0. A test that compares exact text will fail intermittently. Check structure and ranges in code; to judge meaning, use an eval with a model-based judge.
Example
# Instead of: assert resp == "It is a billing problem."
data = json.loads(resp)
assert "category" in data
assert data["category"] in {"BILLING", "TECHNICAL", "ESCALATION"}