New Statistical Tool Can Help Workaround For Randomized Experiments

A new statistical tool can help researchers get meaningful results when a randomized experiment, considered the gold standard, is not possible.

Randomized experiments split participants into groups by chance, with one undergoing an intervention and the other not. But in real-world situations, they can’t always be done. Companies might not want to use the method, or such experiments might be against the law.

Developed by a researcher at The University of Texas at Austin, the new tool called two-step synthetic control adapts an existing research workaround, known as the synthetic control method.

The traditional synthetic control method creates synthetic control groups from the data, in place of real ones. The groups are weighted statistically and compared with a group undergoing an intervention.

But the synthetic control method does not perfectly apply to all situations, especially ones in which the intervention group is different from control groups, according to Kathleen Li, an assistant professor of marketing at the McCombs School of Business. In these scenarios, the method’s lack of flexibility could lead to less accurate results.

“Our framework allows managers and policymakers to estimate effects they previously weren’t able to estimate accurately,” said Li, who developed the tool along with Venkatesh Shankar of Texas A&M University. “They get a more precise estimate that can help them make more informed decisions.”

The study, published in advance online in the journal Management Science, offers a two-step synthetic control approach:

First, it determines whether the traditional synthetic control method applies to a given case.
If it does not, the second step uses a more flexible framework that allows weighted controls to differ from 100% or to shift the control group up and down.
The researchers tested the new method on a real-world situation by looking at sales of tampons: how they responded in 2016, when New York repealed a sales tax on them.

Sales taxes on tampons have been a contentious issue, with many countries abolishing or reducing them. Proponents say feminine hygiene products are basic necessities and should not be taxed. But by 2019, only 13 U.S. states had repealed them. For policymakers, a key point of contention has been how repeal would affect tampon sales.

To find out, Li and Shankar gathered 52 weeks of sales data before New York’s repeal and 17 weeks after. Their control group included 35 states that did not repeal the tax. They found the traditional synthetic control method probably overestimated the actual increase in weekly sales, showing a 2.5% rise in New York.

When they applied their more flexible method, it estimated New York’s repeal caused a more modest increase in weekly tampon sales, only 2.08%. That figure better matches the actual sales figures before the intervention.

Li and Shankar will be making their new method available online for all to use.