industry-applications.md

Causal Inference Applications in the Industry

The following is a list of applications of causal inference in the industry, sorted by topic and date.

• Experimentation Platform
• Variance Reduction
• Conditional Average Treatment Effects
• Quantile Testing
• Interference
• Continuous Testing
• Ranking
• Quasi-experiments
• Mediation Analysis
• Trustworthy Experiments
• Misc

Experimentation Platform

• Decision Making at Netflix - Netflix (2021)

• Spotify’s New Experimentation Platform - Spotify (2020)

• Under the Hood of Uber’s Experimentation Platform - Uber (2018)

• Democratizing online controlled experiments at Booking.com - Booking (2017)

Variance Reduction

• Deep Dive Into Variance Reduction - Microsoft (2022)

• Increasing the sensitivity of experiments with rank transformation - Booking.com (2020)

• Improving Experimental Power through Control Using Predictions as Covariate (CUPAC) - Doordash (2020)

• How Booking.com increases the power of online experiments with CUPED - Booking.com (2018)

• 📝 Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix - Neflix (2016)

• 📝 Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data - Microsoft (2013)

Conditional Average Treatment Effects

• Leveraging Causal Modeling to Get More Value from Flat Experiment Results - Doordash (2020)

• Free Lunch! Retrospective Uplift Modeling for Dynamic Promotions Recommendation within ROI Constraints - Booking (2020)

Quantile Testing

• Comparing quantiles at scale in online A/B-testing - Spotify (2022)

  • 📝 Resampling-free bootstrap inference for quantiles - Spotify (2022)

• How Wish A/B tests percentiles - Wish (2021)

• 📝 Large-Scale Online Experimentation with Quantile Metrics - LinkedIn (2019)

• 📝 Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas - Microsoft (2018)

• Analyzing Experiment Outcomes: Beyond Average Treatment Effects - Uber (2018)

• Streaming Video Experimentation at Netflix: Visualizing Practical and Statistical Significance - Netflix (2018)

Interference

• 📝 Network experimentation at scale - Facebook (2020)

• Unbiased Experiments in Congested Networks - Netflix (2021)

• Budget-split testing: A trustworthy and powerful approach to marketplace A/B testing - Linkedin (2021)

• Detecting interference: An A/B test of A/B tests - LinkedIn (2019)

  • 📝 Detecting Network Effects: Randomizing Over Randomized Experiments - LinkedIn (2017)

• Spreading the Love in the LinkedIn Feed with Creator-Side Optimization - LinkedIn (2018)

  • 📝 Using Ego-Clusters to Measure Network Effects at LinkedIn - LinkedIn (2018)

• The Design of A/B Tests in an Online Marketplace - Ebay (2018)

• A/B testing in a long-distance carpooling marketplace - BlaBlaCar (2018)

• Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash - Doordash (2018)

• Simulating a ridesharing marketplace - Lyft (2016)

• Interference Across a Network - Lyft (2016)

• 📝 Design and analysis of experiments in networks: Reducing bias from interference - Facebook (2014)

• 📝 Why Marketplace Experimentation Is Harder than it Seems: The Role of Test-Control Interference - Ebay (2014)

Continuous Testing

• Sequential Testing at Booking.com - Booking (2023)

• Choosing Sequential Testing Framework — Comparisons and Discussions - Spotify (2023)

• Improving Experimentation Efficiency at Netflix with Meta Analysis and Optimal Stopping - Netflix (2019)

• 📝 Peeking at A/B Tests - Optimizely (2017)

Ranking

• Partial Blockout Experiments for a Two Sided Marketplace - Booking (2022)

• Beyond A/B Test : Speeding up Airbnb Search Ranking Experimentation through Interleaving - Airbnb (2022)

• Innovating Faster on Personalization Algorithms at Netflix Using Interleaving - Netflix (2017)

Customer Lifetime Value

• Estimating long-term effects when only short-run experiments are available - Spotify (2023)

• 📝 Beyond Customer Lifetime Valuation: Measuring the Value of Acquisition and Retention for Subscription Services - Netflix (2022)

• Estimating the long-run value we give to our users through experiment meta-analysis - Meta (2022)

Geo-Experiments

• Market Segmentation for Geo-Testing at Scale - Expedia (2023)

• Measuring Ad Effectiveness Using Geo Experiments - Google (2011)

Quasi-Experiments

• Gaining confidence in synthetic control causal inference with sensitivity analysis - Spotify (2023)

• There’s more to experimentation than A/B - Booking (2020)

Mediation Analysis

• 📝 Mediation Analysis in Online Experiments at Booking.com: Disentangling Direct and Indirect Effects - Booking (2018)

Trustworthy Experiments

• Patterns of Trustworthy Experimentation: During-Experiment Stage - Microsoft (2021)

• More Trustworthy A/B Analysis: Less Data Sampling and More Data Reducing - Microsoft (2021)

• Data Quality: Fundamental Building Blocks for Trustworthy A/B testing Analysis - Microsoft (2021)

• Patterns of Trustworthy Experimentation: Post-Experiment Stage - Microsoft (2021)

• Patterns of Trustworthy Experimentation: Pre-Experiment Stage - Microsoft (2020)

Misc

• 📝 Design-Based Inference for Multi-arm Bandits - Netflx (2023)

• Answering counterfactual “what-if” questions in a trustworthy and efficient manner - Spotify (2023)

  • 📝 Estimating categorical counterfactuals via deep twin networks - Spotify (2021)

• 📝 Science of price experimentation at Amazon - Amazon (2023)

• Quantifying Efficiency in Ridesharing Marketplaces - Uber (2023)

• Increase A/B Testing Power by Combining Experiments - Ebay (2022)

• Overtracking and trigger analysis: reducing sample sizes while INCREASING the sensitivity of experiments - Booking (2022)

• Don’t be seduced by the allure: A guide for how (not) to use proxy metrics in experiments - Meta (2022)

• Leveraging proxy variables for causal inference - Booking (2021)

• Trapped in the Present: How engagement bias in short-run experiments can blind you to long-run insights - Pinterest (2019)

• Measuring Success with Experimentation - Ebay (2019)

• Analyzing Switchback Experiments by Cluster Robust Standard Error to Prevent False Positive Results - Doordash (2019)

• Experiment Rigor for Switchback Experiment Analysis - Doordash (2019)

• Using Causal Inference to Improve the Uber User Experience - Uber (2019)

• Significance Testing for Ratio Metrics in Experiments - Ebay (2016)

• 📝 Automatic Detection and Diagnosis of Biased Online Experiments - Linkedin (2011)