Vignesh Ananth

Vignesh Ananth

Economist - Econometrics - Data Analytics - Machine Learning - Data Science

A/B Testing and Econometrics - Part 1

5 minute read

Published:

This is part one of a two part series on A/B testing and Econometrics. This is a link to Part 2.

A/B testing is one of the primary tests tech companies use before they roll out new features or changes to their products. If you pay close attention you might be able to point out that you’re part of an A/B test by noticing a subtle change on an app or website that you use. At its core an A/B test is nothing but a controlled experiment. A fraction of users are randomly assigned to a control and another fraction randomly assigned to a treatment. The control is the status quo, where there is no change in the product. The treatment is the case where the users see a change in the product. The difference in the metric of interest, known more formally as the outcome variable, between the control group and the treatment group is the causal effect of the change.

The goal of any A/B test is to quantitatively ascertain, with statistical significance, the causal effect of a change measured by some pre defined metric - known in some cases as a KPI (Key Performance Indicator). For example, a company may be interested in quantifying the change in click through rate by altering the position of a button on its website or app. Companies also test more complex things like sign-up flows, landing page designs. etc with a focus on tracking conversion via a specific pre defined metric.

The advantage of an A/B test is the fact that the design allows for causal identification. Causal identification is important because correlation doesn’t imply causation. For example, if a company were to simply make a change on a product and after a certain amount of time look at the value of the pre defined metric, what they will be picking out is a correlation between the the change and the predefined metric. The reason for this is the fact that multiple things are likely happening at the same time that could affect the metric. However, the strategy of control and treatment uses the control as a benchmark to estimate the effect of this change on the treatment.

From the perspective of identifying the causal effect of a change in a product setting, the A/B test is the gold standard. Tech companies typically have users in the millions, therefore the sample sizes for these tests are easily large enough to get an answer with a high level of statistical significance. In addition to this, random assignment is easy - this prevents selection bias and allows for true causal inference.

A/B testing, while admittedly the gold standard, isn’t always something that is feasible. There are ethical concerns that come up when running an experiment and more pertinent to the case of a product company is actually alienating users through these experiments. The other case where A/B tests can’t help are looking at something retrospectively or trying to see if something out of the control of the product team affects usage - you can’t run experiments on things that you can’t control. This begs the question: Are there other ways to ascertain causal inference when experiments are not feasible?

The answer to this question lies in the Econometricians tool box. Econometrics as a field is primarily focused on causal inference and a lot of the questions that economists ask can’t be tested by running an experiment. Experiments run in the real world are very expensive and if you’re asking questions like - what the causal effect of an extra year of schooling is on wage earning - there is no good way to construct an experiment to test this. Randomly assigning students who want to complete school to a control group where they are forced out of school one year before the treatment group is simply infeasible.

Econometricians, however, have designed tools to answer these questions. The successful use of these tools is by using what are known as ‘natural experiments’ to approximate conditions that would have resulted from designing and running experiments.

In the schooling example that I cited earlier economists Angrist and Krueger used a creative method where they used the season of a child’s birth and state laws that mandate compulsory school attendance to identify the causal effect of an extra year of schooling. The season of a child’s birth influences when a child can start school which in turn, many years down the line, alters the date that the same child can drop out of school by a year. This provides them with the control group that were born in the same year, within months of each other, but could drop out one year before the treatment group. An impossible condition to achieve in an experimental setting.

These methods are more widely applicable to answer other questions around causal inference where A/B testing is infeasible. These methods, while not as robust as an experiment, do a great job of identifying causal effects. They require a bit of creativity and some simplifying assumptions, but they work when an experiment is infeasible.

Part 2 of this post focuses on an explanation of some of these methods.

References

Angrist, J. D., and A. B. Krueger. “Does Compulsory School Attendance Affect Schooling and Earnings?” The Quarterly Journal of Economics, vol. 106, no. 4, Jan. 1991, pp. 979–1014., doi:10.2307/2937954.

Angrist, Joshua D., and Pischke Jörn-Steffen. Mostly Harmless Econometrics: an Empiricist’s Companion. Princeton University Press, 2009.

You May Also Enjoy

The costs keep mounting in the gig economy

9 minute read

Published:

The last decade has been the decade of the gig economy. Companies such as Uber, Lyft, Doordash and Airbnb that occupy this space have had access to unprecedented amounts of venture capital that has fueled tremendous amounts of unprofitable growth. Airbnb has raised $4.8 billion dollars since inception and Uber has raised $24.5 billion dollars since inception, neither has been consistently profitable. These companies have hit many speed bumps along the way, some as a result of their - grow at any cost attitude and other regulatory pressures that have altered overly optimistic profitability predictions and valuations. However, COVID-19 and the impending fallout from this crisis is the ultimate test of the robustness of their business models and valuations they command. It is nothing like these companies, let alone anyone in the world - perhaps with the exception of Bill Gates, ever imagined in their wildest dreams was coming.

Do ride sharing services make traffic congestion worse?

11 minute read

Published:

The introduction of ride sharing platforms like Uber and Lyft have made our lives significantly better. Gone are the days where not owning a car meant meticulous examination of public transport schedules or expensive cab rides. They have so seamlessly integrated into our lives that it’s hard for me to even think about life without access to a ride on demand, wherever I am, at whatever time of the night.

A/B Testing and Econometrics - Part 2

10 minute read

Published:

This is the second post in a two part post on A/B testing and causal inference using Econometric methods. The first post can be found here.

Sentiment Analysis of Airbnb Reviews

4 minute read

Published:

The ubiquity of the multi side sharing economy platforms has gotten most of us accustomed to assigning a ‘star-rating’ post usage of a service. In the case of Uber it is a 5 star scale that both driver and passenger assign each other. In the case of Airbnb it is a 10 point scale for a variety of criterion that both hosts and renters assign each other.