The Trouble with Misleading Studies


    On Jan. 4, 2022, the Journal of the American Medical Association (JAMA) published a study compiled by the American Cancer Society (ACS) that claims to analyze the impact of a Massachusetts law banning the sale of menthol cigarettes and other flavored tobacco products. The study focused on comparing cigarette sales in Massachusetts from January 2017 to July 2021 to cigarette sales in 33 other states that did not enact a similar statewide menthol cigarette ban.

    In the discussion section of the published study, the ACS researchers claim that “the comprehensive flavor ban in Massachusetts was associated with a statistically significant decrease in state-level menthol and all cigarette sales.” This claim is misleading, and for this reason the study and its conclusions could misinform the public as well as lawmakers about the actual impact of a statewide flavored tobacco ban on cigarette sales.

    Study Claims
    The ACS study, which was based on an analysis of Nielsen Company Retail Scanner data from retail stores in Massachusetts and 33 other states, asserts the following impacts from the Massachusetts menthol cigarette ban.

    • Sales of menthol-flavored and nonflavored cigarettes in Massachusetts declined by up to 33 percent from June 2020 to July 2021, which constitutes the first year after the law went into effect on June 1, 2020.
    • Sales of menthol-flavored and nonflavored cigarettes in the 33 other states decreased by 8 percent from June 2020 to July 2021, with menthol cigarettes in particular declining by 3 percent.

    Study Findings Misleading
    The methodology of the study and the noninclusion of publicly available state cigarette tax stamp sales data raises a serious concern that the study findings are misleading. In fact, these questionable aspects of the study lead to a conclusion that the findings are not an accurate representation of how a statewide menthol cigarette ban impacts cigarette sales. These questionable aspects of the study include the following:

    • The main conclusion that the Massachusetts flavored tobacco sales ban led to a decline in cigarette sales to Massachusetts residents is a misrepresentation. If a state bans the legal sale of a product, the sales of that product by retailers in that particular state will, of course, decline to zero. However, given that cigarettes are a nationally sold product and that consumers are mobile and will seek out other sources of a banned product, a study of one state’s cigarette sales should focus on a more regional analysis, especially considering the short distance to adjacent states where the product is legally sold. That is, the impact of the Massachusetts menthol cigarette sales ban should be analyzed in conjunction with cigarette sales data from immediately adjacent states because consumers will travel to the nearest state to purchase what has been banned in their home state.
    • Significant increases in cigarette tax stamp sales in some of these neighboring states demonstrate the improper focus of the ACS study. In fact, the study researchers confirmed that “[l]imitations of the study include that cross-border or online cigarette sales in Massachusetts were not accounted for ….” This admission by the study authors raises the question of whether excluding changes in cigarette sales in adjacent states—specifically New Hampshire, Rhode Island and Vermont—was a choice to support their eventual claim that a statewide menthol cigarette sales ban is associated with a significant decline in all cigarette sales to residents of the subject state.
    • The Nielsen Company Retail Scanner data that was relied upon for the study is only a sampling of some retail store sales and does not capture total cigarette sales in any given state, including Massachusetts. Even the study authors admit that their “findings should be interpreted cautiously as sales data may not fully capture cigarette consumption.” That is, the Nielsen Company sampling of cigarette sales data in Massachusetts would not capture cigarette purchases made by Massachusetts residents who travel to adjacent states to buy their preferred cigarettes.
    • A more accurate source for cigarette sales is cigarette tax stamp sales data from state revenue departments. All states, except North Carolina and North Dakota, require a special cigarette tax stamp to be affixed to each package of cigarettes by a wholesaler to show that the state cigarette tax has been paid. A state revenue department maintains data on every tax-paid stamp sold to wholesalers, and this data is publicly available. The decision to not use accurate and publicly available state cigarette tax-paid stamp sales data calls into question the credibility of the study and the very conclusions made by the ACS researchers.
    • According to Massachusetts cigarette tax stamp data, state tax-paid stamp sales to wholesalers declined by 24 percent (33,076,000 fewer stamps/packs), not the 33 percent projected from Nielsen Company sales sampling data. This 24 percent decline occurred from June 2020 to May 2021. During this same 12-month period, three adjacent states experienced significant cigarette tax stamp sales increases led by New Hampshire with a 22.1 percent increase (+23,380,000 stamps/packs), Rhode Island with an 18 percent increase (+5,355,000 stamps/packs) and Vermont with a 6 percent increase (+1,143,000 stamps/packs). This means that while Massachusetts recorded 33,076,000 fewer cigarette stamps sold during the first year after the flavor ban law was in effect, the three bordering states reported an increase of 29,878,000 cigarette tax stamps sold, which equates to 90 percent of the lower Massachusetts cigarette tax stamp volume.

    Evidence-based data in the form of state cigarette tax stamp sales demonstrates that the study’s conclusion is not fact-based as Massachusetts residents changed their purchasing behavior by crossing state borders and patronizing retailers in New Hampshire, Rhode Island and Vermont to purchase cigarettes.