تقرير
Anytime-Valid Confidence Sequences in an Enterprise A/B Testing Platform
| العنوان: | Anytime-Valid Confidence Sequences in an Enterprise A/B Testing Platform |
|---|---|
| المؤلفون: | Maharaj, Akash V., Sinha, Ritwik, Arbour, David, Waudby-Smith, Ian, Liu, Simon Z., Sinha, Moumita, Addanki, Raghavendra, Ramdas, Aaditya, Garg, Manas, Swaminathan, Viswanathan |
| المصدر: | Companion Proceedings of the ACM Web Conference 2023 (WWW '23 Companion) |
| سنة النشر: | 2023 |
| المجموعة: | Statistics |
| مصطلحات موضوعية: | Statistics - Applications, G.3 |
| الوصف: | A/B tests are the gold standard for evaluating digital experiences on the web. However, traditional "fixed-horizon" statistical methods are often incompatible with the needs of modern industry practitioners as they do not permit continuous monitoring of experiments. Frequent evaluation of fixed-horizon tests ("peeking") leads to inflated type-I error and can result in erroneous conclusions. We have released an experimentation service on the Adobe Experience Platform based on anytime-valid confidence sequences, allowing for continuous monitoring of the A/B test and data-dependent stopping. We demonstrate how we adapted and deployed asymptotic confidence sequences in a full featured A/B testing platform, describe how sample size calculations can be performed, and how alternate test statistics like "lift" can be analyzed. On both simulated data and thousands of real experiments, we show the desirable properties of using anytime-valid methods instead of traditional approaches. Comment: 15 pages, 12 figures. Expanded version of ACM Web Conference Proceedings paper |
| نوع الوثيقة: | Working Paper |
| DOI: | 10.1145/3543873.3584635 |
| URL الوصول: | http://arxiv.org/abs/2302.10108 |
| رقم الأكسشن: | edsarx.2302.10108 |
| قاعدة البيانات: | arXiv |
| DOI: | 10.1145/3543873.3584635 |
|---|