ab-test-setup

A/B Test Setup

1️⃣ Purpose & Scope

Define an experiment that can answer a specific product question, and verify its assumptions before exposing users. This procedure cannot guarantee validity by itself.

• Documents the stopping rule
• Estimates sample needs under stated assumptions
• Makes the hypothesis and decision criteria reviewable

---

2️⃣ Pre-Requisites

You must have:

• A clear user problem
• Access to an analytics source
• Roughly estimated traffic volume

Hypothesis Quality Checklist

A valid hypothesis includes:

• Observation or evidence
• Single, specific change
• Directional expectation
• Defined audience
• Measurable success criteria

---

3️⃣ Hypothesis Lock (Hard Gate)

Before designing variants or metrics, you MUST:

• Present the final hypothesis
• Specify:
• Target audience
• Primary metric
• Expected direction of effect
• Minimum Detectable Effect (MDE)

Use the hypothesis already agreed in the task. If a launch-critical choice is missing, present the concrete choice for confirmation while continuing independent analysis. Do not repeatedly request approval for a decision already authorized.

---

4️⃣ Assumptions & Validity Check (Mandatory)

Explicitly list assumptions about:

• Traffic stability
• User independence
• Metric reliability
• Randomization quality
• External factors (seasonality, campaigns, releases)

If assu