Key Concepts
Before running experiments, understand these core concepts.
Hypothesis
A hypothesis is a specific, testable prediction about what change will improve a metric.
Good hypothesis: "Changing the CTA from 'Learn More' to 'Get Started Free' will increase button clicks by 15% because it reduces perceived commitment."
Bad hypothesis: "Let's try a different button."
A strong hypothesis includes:
What you're changing
What metric will improve
Why you believe this will work
Variants
Variants are different versions of your page that you test against each other.
Control: Your original page (unchanged)
Variant A, B, C...: Modified versions
In an A/B test, you have one control and one variant. In a multivariate test, you test multiple variants simultaneously.
Statistical Significance
Statistical significance tells you whether your results are real or just random chance.
Fibr calculates this automatically. When an experiment reaches statistical significance (typically 95% confidence), you can trust that the winning variant actually performs better.
Before significance: Results might flip. Don't make decisions yet.
After significance: Results are reliable. Roll out the winner.
Traffic Allocation
Traffic allocation determines what percentage of visitors see each variant.
50/50 split: Half see control, half see variant. Fastest results but higher risk.
90/10 split: Most visitors see control, small group sees variant. Safer but slower.
Start with 50/50 unless you're testing something risky.
Last updated