Key Features

Eligibility filtering (guardrails)

A test is evaluated only if live, set to continue_testing, and each group has ≥100 sessions for the current window. Keeps decisions from being made on thin data.

Significance engine (correct method for each case)

Ads Personalization: Frequentist—Z-test when binomial conditions hold; otherwise Fisher’s Exact. One-tailed at 95% confidence with clear decision rules.

A/B & Multivariate: Bayesian—10,000-draw Monte Carlo; stop if any variant crosses 95% probability of being best.

Dynamic traffic allocation (finish faster, waste less)

Phase 1 (warm-up): Until every variant hits 100 sessions, variants ≥100 sessions are capped at 10% each; remaining traffic goes to under-threshold variants so all reach eligibility quickly.

Phase 2 (post-eligibility):

• If a winner exists → winner gets the bulk of traffic; others at ~5% each for light monitoring.

• If no winner and similar rates (<10% CVR diff) → keep traffic equal.

• If no winner and meaningful separation (≥10% CVR diff) → allocate proportionally to observed CVR to accelerate a conclusion.

Toggle at account level (default ON).

Timely alerts & clear UI

• Automatic emails on winner/control/no conclusion with action prompts (apply winner, stop experiment, or revise and restart).

• In-product labels and tooltips explain status in plain language (“More data needed,” “Statistically significant—apply this version”).