Skip to main content
SpringerLink
Log in

Significant Placebo Results in Difference-in-Differences Analysis: The Case of the ACA’s Parental Mandate

  • Original Article
  • Published:
Eastern Economic Journal Aims and scope Submit manuscript

Abstract

The Affordable Care Act (ACA) lets young adults stay on their parents’ insurance. Several papers use age–time difference-in-differences strategies to argue this causes health insurance and labor effects. I show that difference-in-differences over “placebo” dates also produces statistically significant “effects” before ACA implementation, even with conservative adjustments. This suggests the effects attributed to the ACA could instead reflect dynamics in the age-structure of the health insurance and labor markets. Reducing the age bandwidth yields more reliable estimates of the increases in parental and overall insurance coverage. The key problem in this literature is therefore potentially overstating the ACA’s “effects” in other dimensions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4

Similar content being viewed by others

Notes

  1. The regression that generates the coefficients and standard errors for the month-year or quarter-year dummy variables does not cluster standard errors at the state level (as all results below do) because there are fewer clusters (51) than conditions tested (186 or 59). Still, calculating a conservative p value from F(186,51) instead of F(186,593,938) gives P=0.0186<0.05 and so the result that the trends are not equal is robust. This consistency is the case for all F-statistics in this paper.

  2. AMS does include limited placebo analysis by randomizing the implementation month between September 2008 and January 2010, and find no significant results beyond expected Type I errors. Still, trend stability in this short pre-period does not necessarily imply parallel trends in post-period

  3. These placebo tests are not entirely independent since I am rolling a multi-year window backward 1 year at a time. Still, the time frame over which I am running placebo regressions is long enough that this overlap cannot entirely explain the placebo results.

  4. This analysis is in the spirit of Ham et al. [2009] which dealt with non-comparable control groups and changes in public policy albeit in another context.

  5. The March Supplement in 2011 underwent a significant change to its imputation procedure so that any non-policy holder in the household can now be coded as a dependent on another household member’s plan. Compared with the old routine, estimates derived from the new one reduced the uninsurance rate by 0.5 percentage points (1.5 million people) and increased the rate of any private coverage by 0.5 percentage points (1.7 million people) [Boudreaux and Turner 2011]. Microdata going back to only the 2000 survey (reference year 1999) was re-released under this new procedure.

  6. Consistent SIPP data on source of health insurance for dependent individuals is only available from 2001–2011. In addition, because of the fact that the SIPP is primarily designed as a panel survey, there are significant gaps in the data between the end of one panel and the beginning of the next (i.e., 2000, 2008). As a result, any multi-year placebo time period covering either of these years is incomplete, resulting in fewer potential regressions for comparison.Furthermore, the 1996 panel does begin until March, and so there is no data for January and February of that year. Rather than dropping the 1993–1996, 1994–1997, and 1995–1998 placebo regressions, I include those with the missing months omitted. The lack of these months should not have a differential effect on those in the affected age group compared with the comparison age groups and therefore should not bias the results.

  7. As a complement to the SIPP, I also use the basic monthly Current Population Survey for 1994–2011 [Census Bureau 2013a, 2013b, 2013c] for additional labor supply placebo regressions (see the Online Appendix). The CPS covers every month in the entire sample, and as a result allows for several more placebo regressions. Furthermore, whereas the primary purpose of the SIPP is to quantify numerous outcomes for a longer panel of individuals, the basic CPS is designed to quantify labor supply, making it better suited to this analysis.A minor downside is that the labor variables have small definitional differences compared with the ones in the SIPP. These discrepancies, though, are orthogonal to the age and time dimensions of my difference-in-differences strategy and so should not affect the comparison of different CPS placebo regressions to the main regression.

  8. Seehttp://www.census.gov/content/dam/Census/programs-surveys/sipp/questionnaires/2001/SIPP%202001%20Panel%20Wave%2009%20-%20Core%20Questionnaire.pdf

  9. See http://www.census.gov/prod/techdoc/cps/cpsmar10.pdf

  10. I am using March CPS data for reference year 2010 as the affected year to be comparable to SK and Cantor. This decision does not affect the placebo results which use earlier years.

  11. Eligibility requirements and effective dates for state mandates are as described in Cantor et al. [2012a].

  12. In the data for each of 1999–2002, MEPS pools approximately 10 of the least populated states. Therefore, for placebo regressions including any of these years, these states are assigned the average value for all of the pooled states as opposed to the respective value for the individual state.

  13. AMS also includes a triple-difference specification, with the third dimension being young adults whose parents do and do not have employer sponsored health insurance (ESI). See the Online Appendix for methodology and placebo results.

  14. As described above, the CPS ASEC data for reference year 2010 is considered “affected” as the respondents were answering questions in March 2011 about the previous year and so likely answered with reference to after September 2010 (when the ACA parental insurance mandate took effect).

  15. As described above, the March CPS was significantly revised in 2010 and only 1999–2009 data was updated to this new procedure, and so placebo regressions with earlier microdata would not be comparable.

  16. Results from placebo regressions on a triple difference specification, utilizing one’s parents ESI status, are consistent with Table 3. See Online Appendix Table 1, corresponding to Table 5 in AMS. Here the bias is more difficult to untangle, as the placebo results could result from anywhere from one to four pairs of non-parallel trends (i.e.,, 19–25 with and without parental insurance, 16–18 and 27–29 with and without parental insurance, different ages with parental insurance, and different ages without parental insurance) and the affect group is even more endogenously defined (those 19–25 whose parents have ESI). These multiple factors could easily explain, for example, why in 2001–2003 vs 2003–2004 the “implementation” coefficient on parental coverage has the same sign as the 2008–2010 vs 2010–2011 whereas the “enactment” coefficient has the opposite sign.

  17. Given this result under such a stringent adjustment, there is no need to apply the less-conservative methods for multiple hypotheses that are detailed in Finkelstein et al. [2012] and Kling and Liebman [2004].

  18. Comparable placebo regressions on labor outcomes using the basic monthly CPS can be found in Online Appendix Table 2. The positive coefficients on hours worked in the 1990s could be because of the fact that during the economic expansion of the 1990s young adults would be more likely to take on more hours than those older (who were already working full time) and those younger (who were mostly still in school). The negative coefficient found during the ACA implementation period could also be the result of young adults’ hours decreasing more than those older (who have more entrenched jobs) and those younger (who were already working relatively few hours).

  19. As these methods ultimately do not solve the statistically significant placebo results problem, I do not also apply them or the one that follows to the Cantor and SK regressions using the March CPS. This is also because of the fact that these methods rely on more precise implementation timing than what the annual, backward looking CPS can reasonably accommodate.

  20. Online Appendix Table 3 shows the comparable adjusted labor results, corresponding to Table 4 above.

  21. As the labor supply data was used to calculate the weights for this approach, it will not also be used as a testable regression outcome, and so there is no comparable AG table for Table 5.

  22. Unfortunately, what I gain in robustness, I lose in external validity, as the results below are arguably inapplicable to those in the lower ages of the original affected group (e.g., 19–23).

  23. F=0.90 (P=0.8407, employment, month level), 0.90 (P=0.6907, employment, quarter level), 0.77 (P=0.9909)/0.7100 (P=0.9532) (full time), 1.00 (P=0.4798)/1.02 (P=0.4368) (any health insurance), and 1.12 (P=0.1974)/1.26 (P=0.1494) (dependent employer coverage through parents).

  24. The alternative specifications are not repeated here, as reducing to a two-by-two difference-in-differences regression with 4 observations has zero degrees of freedom [DL]. The synthetic control group results are also not repeated as it is inapplicable with only one affected age and one comparison age.

  25. Online Appendix Tables 6 and 7 parallel Tables 2 and 3 for SK and Cantor’s March CPS analysis, respectively, whereas Online Appendix Table 8 parallels Online Appendix Table 1 for DDD on SIPP Health Insurance. These results are consistent with those above in Table 8.

  26. It is also doubtful that this drop in employment could be driving the net increase in coverage for 27 years shown in Table 8 (and shown to be robust by Monte Carlo Analysis below), as the expected sign would be reversed (i.e., overall coverage decreasing because of a drop in full-time employment).

References

  • Abadie, Alberto. 2005. Semiparametric Difference-in-Differences Estimators. Review of Economic Studies, 72(1): 1–19.

    Article  Google Scholar 

  • Abadie, Alberto, and Javier Gardeazabal. 2003. The Economic Costs of Conflict: A Case Study of the Basque Country. The American Economic Review, 93(1): 113–132.

    Article  Google Scholar 

  • Akosa Antwi, Yaa, Asako S. Moriya, and Kosali Simon. 2013. Effects of Federal Policy to Insure Young Adults: Evidence from the 2010 Affordable Care Act’s Dependent-Coverage Mandate. American Economic Journal: Economic Policy, 5(4): 1–28.

    Google Scholar 

  • Akosa Antwi, Yaa, Asako S. Moriya, and Kosali Simon. 2015. Access to Health Insurance and the Use of Inpatient Medical Care: Evidence from the Affordable Care Act Young Adult Mandate. Journal of Health Economics, 39: 171–187.

    Article  Google Scholar 

  • Akosa Antwi, Yaa, Asako S. Moriya, Kosali Simon, and Ben Sommers. 2015. Changes in Emergency Department Use Among Young Adults After the Patient Protection and Affordable Care Act’s Dependent Coverage Provision. Annals of Emergency Medicine, 65(6): 664–672.

    Article  Google Scholar 

  • Anderson, Michael, Carlos Dobkin, and Tal Gross. 2012. The Effect of Health Insurance Coverage on the Use of Medical Services. American Economic Journal: Economic Policy, 4(1): 1–27.

    Google Scholar 

  • Barbaresco, Silvia, Charles J. Courtemanche, and Yanling Qi. 2015. Impacts of the Affordable Care Act Dependent Coverage Provision on Health-Related Outcomes of Young Adults. Journal of Health Economics, 40: 54–68.

    Article  Google Scholar 

  • Bell, David N. F., and David G. Branchflower. 2011. Young People and the Great Recession. Oxford Review of Economic Policy, 27(2): 241–267.

    Article  Google Scholar 

  • Bertrand, Marianne, Esther Duflo, and Sendhil Mullainathan. 2004. How Much Should We Trust Difference-in-Differences Estimates? Quarterly Journal of Economics, 119(1): 249–275.

    Article  Google Scholar 

  • Boudreaux, Michael, and Joanna Turner. 2011. Modifications to the Imputation Routine for Health Insurance in the CPS ASEC: Description and Evaluation. State Health Access Data Assistance Center, http://www.census.gov/hhes/www/hlthins/data/revhlth/SHADAC.pdf (accessed August 31, 2013).

  • Bureau of Labor Statistics. 2013. Local Area Unemployment Statistics, http://www.bls.gov/lau/home.htm (accessed August 31, 2013).

  • Busch, Susan H., Ezra Golberstein, and Ellen Meara. 2014. ACA Dependent Coverage Provision Reduced High Out-Of-Pocket Health Care Spending For Young Adults. Health Affairs, 33(8): 1361–1366.

    Article  Google Scholar 

  • Cantor, Joel C., Dina Feblloff, Alan C. Monheit, Derek DeLia, and Margaret Koller. 2012a. Lessons for the ACA: Expanding Dependent Coverage for Young Adults: Lessons from State Initiatives. Journal of Health Politics, Policy, and Law, 37(1): 99–128.

    Article  Google Scholar 

  • Cantor, Joel C., Alan C. Monheit, Derek DeLia, and Kristen Lloyd. 2012b. Early Impact of the Affordable Care Act on Health Insurance Coverage of Young Adults. Health Services Research, 47(5): 1773–1790.

    Article  Google Scholar 

  • Card, David, and Thomas Lemiuex. 2000. Adapting to Circumstances: The Evolution of Work, School, and Living Arrangements among North American Youth. in Youth Employment and Joblessness in Advanced Countries. edited by David G. Blanchflower and Richard B. Freeman. Chicago: University of Chicago Press, 171–214.

    Google Scholar 

  • Carlson, Daniel L., Ben Lennox Kail, Marlaina Dreher, and Jamie L. Lynch. 2014. The Affordable Care Act, Dependent Health Insurance Coverage, and Young Adults’ Health. Sociological Inquiry, 84(2): 191–209.

    Article  Google Scholar 

  • Census Bureau. 2013a. Annual Social and Economic Supplement to the Current Population Survey, http://www.census.gov/hhes/www/hlthins/data/revhlth/extracts.html (2000–2010) and http://thedataweb.rm.census.gov/ftp/cps_ftp.html#cpsmarch (2011-present) (accessed August 31, 2013).

  • Census Bureau. 2013b. Current Population Survey, http://thedataweb.rm.census.gov/ftp/cps_ftp.html (accessed August 31, 2013).

  • Census Bureau. 2013c. Survey of Income and Program Participation, http://thedataweb.rm.census.gov/ftp/sipp_ftp.html (accessed September 2, 2013).

  • Chua, Kao-Ping, and Benjamin D. Sommers. 2014. Changes in Health and Medical Spending Among Young Adults Under Health Reform. JAMA, 311(23): 2437–2439.

    Article  Google Scholar 

  • Congressional Budget Office. 2004. What Is Happening to Youth Employment Rates? http://cbo.gov/sites/default/files/cbofiles/ftpdocs/60xx/doc6017/11-18-youthemployment.pdf. (accessed 25 September 2015).

  • Depew, Briggs, and James Bailey. 2015. Did the Affordable Care Act’s Dependent Coverage Mandate Increase Premiums? Journal of Health Economics, 41: 1–14.

    Article  Google Scholar 

  • DiCecio, Riccardo, Kristie M. Engemann, Michael T. Owyang, and Christopher H. Wheeler 2008. Changing Trends in the Labor Force: A Survey. Federal Reserve Bank of St. Louis Review, 90(1): 47–62.

    Google Scholar 

  • Dinkelman, Taryn, and Vimal Ranchhod. 2012. Evidence on the Impact of Minimum Wage Laws in an Informal Sector: Domestic Workers in South Africa. Journal of Development Economics, 99(1): 27–45.

    Article  Google Scholar 

  • Donald, Stephen G., and Kevin Lang. 2007. Inference with Difference-in-Differences and Other Panel Data. Review of Economics and Statistics, 89(2): 221–233.

    Article  Google Scholar 

  • Dunn, W. 2013. The US Labour Market Recovery Following the Great Recession. OECD Economics Department Working Papers, No. 1015, OECD Publishing, http://dx.doi.org/10.1787/5k4ddxp3xlvf-en (accessed July 7, 2013).

  • Finkelstein, Amy, Sarah Taubman, Bill Wright, Mira Bernstein, Jonathan Gruber, Joseph P. Newhouse, Heidi Allen, and Katherine Baicker, the Oregon Health Study Group. 2012. The Oregon Health Insurance Experiment: Evidence from the First Year. Quarterly Journal of Economics, 127(3): 1057–1106.

    Article  Google Scholar 

  • Fujita, Shigeru, Christopher J. Nekarda, and Garey Ramey. 2007. The Cyclicality of Worker Flows: New Evidence from the SIPP. Working Paper 07–5, Federal Reserve Bank of Philadelphia.

  • Gruber, Jonathan, and Brigitte C. Madrian. 1997. Employment separation and health insurance coverage. Journal of Public Economics, 66: 349–382.

    Article  Google Scholar 

  • Gruber, Jonathan, and Kosali Simon. 2008. Crowd-Out Ten Years Later: Have Recent Public Insurance Expansions Crowded Out Private Health Insurance? Journal of Health Economics, 27(2): 201–217.

    Article  Google Scholar 

  • Ham, John C., and Lara Shore-Sheppard. 2005. The effect of Medicaid Expansions for low-Income Children on Medicaid Participation and Private Insurance Coverage: Evidence from the SIPP. Journal of Public Economics, 89: 57–83.

    Article  Google Scholar 

  • Ham, John C., Xianghong Li, and Lara Shore-Sheppard. 2009. A Reexamination of the Impact of Welfare Reform on Health Insurance Among Less-Skilled Women. In Welfare Reform and Its Long-Term Consequences for America’s Poor, Edited by James P. Ziliak. Cambridge University Press.

  • Han, Xuesong, K. Robin Yabroff, Anthony S. Robbins, Zhiyuan Zheng, and Ahmedin Jamal. 2014. Dependent Coverage and Use of Preventive Care under the Affordable Care Act. New England Journal of Medicine, 371(24): 2341–2342.

    Article  Google Scholar 

  • Hernandez-Boussard, Tina, Carson S. Burns, N. Ewen Wang, Laurence C. Baker, and Benjamin A. Goldstein. 2014. The Affordable Care Act Reduces Emergency Department Use By Young Adults: Evidence From Three States. Health Affairs, 33(9): 1648–1654.

    Article  Google Scholar 

  • Jhamb, Jordan, Dhaval Dave, and Gregory Colman. 2015. The Patient Protection and Affordable Care Act and the Utilization of Health Care Services Among Young Adults. International Journal of Health and Economic Development, 1(1): 8–25.

    Google Scholar 

  • Kling, Jeffrey, and Jeffrey Liebman. 2004. Experimental Analysis of Neighborhood Effects on Youth, Mimeo, at http://www.nber.org/mtopublic/483.pdf. (accessed April 9, 2015).

  • Lazear, Edward P., and James R. Spletzer. 2013. The United States Labor Market: Status Quo or A New Normal? In The Changing Policy Landscape: A Symposium Sponsored by The Federal Reserve Bank of Kansas City, 405–451.

  • Levine, Phillip B., Robin McKnight, and Samantha Heep. 2011. How Effective Are Public Policies to Increase Health Insurance Coverage Among Young Adults? American Economic Journal: Economic Policy, 3(1): 129–156.

    Google Scholar 

  • Madrian, Bridgette C. 1994. Employment-Based Health Insurance and Job Mobility: Is There Evidence of Job-Lock? Quarterly Journal of Economics, 109(1): 27–54.

    Article  Google Scholar 

  • Medical Expenditure Panel Survey-Insurance Component. 2013a. Table II. B.2.b.(1)(2010) Percent of Private-Sector Enrollees That are Enrolled in Self- Insured Plans at Establishments That Offer Health Insurance by Firm Size and Selected Characteristics, http://meps.ahrq.gov/mepsweb/survey_comp/Insurance.jsp (accessed August 31, 2013).

  • Medical Expenditure Panel Survey-Insurance Component. 2013b. Table II. A.2 Percent of Private-Sector Establishments That Offer Health Insurance by Firm Size and State, http://meps.ahrq.gov/mepsweb/survey_comp/Insurance.jsp (accessed August 31, 2013).

  • Mulcahy, Andrew, Katherine Harris, Kenneth Finegold, Arthur Kellermann, Laurel Edelman, and Benjamin D. Sommers. 2013. Insurance Coverage of Emergency Care for Young Adults under Health Reform. New England Journal of Medicine, 368(22): 2105–2112.

    Article  Google Scholar 

  • O’Hara, Brett, and Matthew W. Brault. 2013. The Disparate Impact of the ACA-Dependent Expansion across Population Subgroups. Health Services Research, 48(5): 1581–1592.

    Google Scholar 

  • O’Higgins, Niall. 2012. This Time It’s Different? Youth Labour Markets during “The Great Recession”. Comparative Economic Studies, 54(2): 395–412.

    Article  Google Scholar 

  • Saloner, Brendan, and Benjamin Lê Cook. 2014. An ACA Provision Increased Treatment For Young Adults With Possible Mental Illnesses Relative To Comparison Group. Health Affairs, 33(8): 1425–1434.

    Article  Google Scholar 

  • Savin, Nathan E. 1984. Multiple Hypothesis Testing. in Handbook of Econometrics, Volume II. edited by Zvi Griliches and Michael D. Intriligator. Amsterdam: Elsevier Science Publishers BV, 827–879.

    Chapter  Google Scholar 

  • Shane, Dan M., and Padmajga Ayyagari. 2015. Spillover Effects of the Affordable Care Act? Exploring the Impact on Young Adult Dental Insurance Coverage. Health Services Research, 50(4): 1109–2.

    Article  Google Scholar 

  • Sommers, Benjamin D., and Richard Kronick. 2012. The Affordable Care Act and Insurance Coverage for Young Adults. Journal of the American Medical Association, 307(9): 913–914.

    Google Scholar 

  • Sommers, Benjamin D., Thomas Buchmueller, Sandra L. Decker, Colleen Carey, and Richard Kronick. 2013. The Affordable Care Act Has Led To Significant Gains In Health Insurance And Access To Care For Young Adults. Health Affairs, 32(1): 165–174.

    Article  Google Scholar 

Download references

Acknowledgements

The author would like to thank Janet Currie, as well as Donna Ginther, David Dranove, Elaine Hill, Tom Vogl, Alex Mas, Doug Miller, Dan Zelter, Diane Alexander, Josephine Duh, Judd Cramer, Julia Sonier, Richard Frank, Mark Duggan, Thomas DeLeire, Amitabh Chandra, other participants in the CHW-RPDS informal seminar for their help and comments, seminar participants at Binghamton University, the University of Kansas, and the APPAM and SEM conferences, and Bobray Bordelon for data support. Funding support from the Princeton University Fellowship, the Department of Economics, the Center for Health and Wellbeing and the Griswold Center for Economic Policy Studies (all at Princeton University). All remaining errors are my own.

Author information

Authors and Affiliations

  1. Department of Economics, University of Kansas, Snow Hall, Lawrence, 66045, KS, USA

    David J G Slusky

Authors
  1. David J G Slusky
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Supplementary information accompanies this article on the Eastern Economic Journal website (www.palgrave-journals.com/eej)

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

J G Slusky, D. Significant Placebo Results in Difference-in-Differences Analysis: The Case of the ACA’s Parental Mandate. Eastern Econ J 43, 580–603 (2017). https://doi.org/10.1057/eej.2015.49

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/eej.2015.49

Keywords

Keywords

Search

Navigation