What is Statistical Significance in Fibr?

Statistical significance is essential to conclude and draw inferences from an experiment

What is Statistical Significance

Statistical significance is a crucial metric used to determine the validity and meaningfulness of test or experiment results. It measures the probability that observed differences between multiple versions of a website are not due to random chance but are actually significant and reliable. In the context of Fibr, understanding statistical significance is essential for evaluating the effectiveness of different variations and making informed decisions based on reliable data. Let's delve into the various concepts related to statistical significance and explore how they impact your experiments in Fibr.

Conversion Lift

Conversion lift is a measure to compare the results of an A/B Test or a Personalization with holdout. It is defined for each variant that you create. It is defined as the percentage difference between the conversion rates of all visitors who see a variant and the conversion rates of all visitors who see the original version of the page.

A positive conversion lift signifies that the variant is performing better than the original version of the page. Whereas a negative conversion lift implies that the original variant is better than the other.

Its important to remember that conversion lift does not necessarily mean that your test was statistically significant. It simply shows the percentage difference in conversion rate between the two groups.

Significance

Significance level is a measure of how confident we are that the results of a test or experiment are accurate. It tells us how likely it is that the results we see in our sample are representative of the entire population.

Fibr uses Bayesian Statistics to determine the statistical significance of a particular test. Bayesian methods differ from traditional frequentist approaches by providing probabilities of one variation outperforming another directly, which is more intuitive and actionable for decision-makers. This approach addresses common pitfalls in A/B testing by allowing continuous updating of beliefs about the effectiveness of variations based on accumulating data, thus providing a dynamic assessment of test results. Bayesian statistics offer a more nuanced understanding of uncertainty through credible intervals, and employ prior distributions which, though subjective, converge on agreement as more data is collected. The process ultimately leads to more robust decision-making in marketing strategies and web design optimizations.

Interpretation of Statistical Significance