Policy Brief—Facilitating Retrospective Analysis of Environmental Regulations

Policy Brief—Facilitating Retrospective Analysis of Environmental Regulations Abstract Prospective or ex ante studies of the costs, benefits, and distributional impacts of new environmental regulations are now commonly performed in many countries. Retrospective analyses, which aim to document actual outcomes, are far less common. The purpose of this policy brief is to illustrate the value of retrospective analysis of environmental regulations, discuss the main challenges of conducting such studies, and make suggestions for facilitating the conduct of retrospective analyses. We examine recent examples of ex post analyses of three sets of U.S. regulations—the Environmental Protection Agency (EPA) Cluster Rule, the NOx Budget Program, and federal gasoline content regulations—and British Columbia’s carbon tax. Based on this review, we offer some lessons for facilitating future retrospective analysis of environmental regulations. Why Is Retrospective Analysis Important? It is common practice for governments to assess the efficiency and effectiveness of prospective environmental regulations through ex ante analysis prior to implementation. In the United States, for example, environmental regulations are evaluated using regulatory impact analysis (RIA), which projects both the costs and benefits associated with a proposed rule. For regulations designed to limit pollution emissions or discharges (hereafter referred to as emissions), RIAs generally entail four steps: (1) estimating emissions with and without the rule, (2) translating the change in emissions into environmental outcomes, (3) estimating the benefits associated with the expected change in environmental outcomes, and (4) estimating the costs of the regulation relative to the no-regulation baseline. Conducting an RIA requires predicting the state of the world with and without the regulation, both of which are inherently uncertain. After a regulation is implemented, the methods used by firms, consumers, and other regulated entities to comply with the rule, as well as actual emissions and costs, can, in principle, be observed. Although the counterfactual no-regulation baseline can never be observed for regulated entities, there are cases in which it can be inferred from a control group of similar, unregulated entities. Thus, in principle, it is possible to evaluate some (or all) of the four steps in RIAs retrospectively (i.e., ex post). Why is retrospective analysis important? First, it is important to investigate whether the regulation had the intended impact on emissions and environmental outcomes. Establishing a causal impact on environmental outcomes is a necessary condition for ensuring that regulations are achieving their desired objectives. Moreover, if a regulation was not fully effective in achieving its intended objectives, retrospective analysis can help to reveal the factors that were responsible for the regulation’s failure. This information can be used to improve both the design of future regulations and ex ante RIAs. Second, it is important to measure the actual costs of a regulation. Actual costs may differ from ex ante estimates because of unforeseen behavioral adaptations by consumers or firms. Understanding these adaptations can also help to improve the design of future regulations and ex ante RIAs. Third, retrospective analysis may make it possible to obtain causal estimates of a regulation’s benefits or its effects on outcomes other than economic efficiency—for example, the impacts of the regulation on employment. Finally, over the long term, retrospective analysis can support the type of institutional learning needed to improve both the effectiveness and efficiency of environmental management. The purpose of this policy brief is to illustrate the value of retrospective analysis of environmental regulations, discuss the main challenges of conducting such studies, and make suggestions for facilitating the conduct of retrospective analyses.1 With this in mind, we examine recent examples of ex post analyses of three sets of U.S. regulations—the Environmental Protection Agency (EPA) Cluster Rule, the NOx Budget Program, and federal gasoline content regulations—and British Columbia’s carbon tax. For a retrospective analysis to be successful, researchers must identify a group targeted by the regulation and an otherwise similar control group that is not affected by it. This means that there must be data for the two groups—both before the regulation was implemented and afterwards. Indeed, these are the main challenges facing a retrospective analysis. We consider how each of the ex post studies we examine addressed these challenges. We conclude by providing lessons for conducting future retrospective analyses. EPA’s Cluster Rule EPA’s 1998 Cluster Rule was the agency’s first integrated, multimedia regulation designed to reduce hazardous air and water pollutants from pulp and paper mills.2 The rule imposed stringent maximum achievable control technology (MACT) standards3 to reduce emissions of benzene and other hazardous air pollutants, sulfur dioxide, particulate matter 10 µm or less in diameter (PM10), and volatile organic chemicals (VOCs)4 on 155 mills that used chemical pulping techniques. The 335 mills that used mechanical pulping techniques or purchased pulp faced less stringent standards on air emissions. Ninety-six of the 155 chemical pulping plants were also subject to best available technology economically achievable (BAT) standards5 to reduce water discharges of chloroform, dioxin, and furans.6 Estimating the Impact on Emissions and Employment Because not all pulp and paper firms were subject to the same requirements under the Cluster Rule, unregulated or less stringently regulated firms provided a control group for estimating the rule’s impact on air and water emissions and employment. Using a subset of the 335 plants subject to neither MACT nor BAT standards as controls, Gray and Shadbegian (2015) find that being subject to both types of standards effectively eliminated chloroform emissions to both air and water and reduced air emissions of VOCs by 28 percent, but had no statistically significant impact on PM10. MACT standards alone, however, had no statistically significant impact on air emissions—a potentially troubling result.7 Using different control groups and a different data source, Fraas and Egorenkov (2017) find that plants subject to MACT plus BAT regulations did not reduce their air emissions as much as the EPA projected ex ante. They use plants in the plywood industry as a control group for pulp and paper plants and find that the reduction in organic hazardous air pollutants associated with the Cluster Rule is smaller than the EPA’s ex ante prediction of a 58 percent reduction.8 A similar result (i.e., smaller reductions than projected ex ante by the EPA) is found using a control group of plants in six unrelated metals and plastics parts manufacturing industries. These studies of the Cluster Rule suggest that the results of ex post analysis can be highly sensitive to the control group used. The ex post analyses also suggest that in some cases the particular method regulated firms adopt in order to comply with a rule may be important. For example, using plants subject to MACT-only standards as a control group for those subject to both MACT and BAT standards, Elrod and Malik (2017) find that plants subject to water as well as air regulations were somewhat more likely to switch from bleached to unbleached paper products. This suggests that one method plants used to comply with the BAT rules was to switch the composition of output rather than the nature of the production process—an unexpected result. To study the impact of the Cluster Rule on employment, Gray et al. (2014) included a subset of unregulated plants that use technologies similar to the technologies used by regulated plants (i.e., some form of pulping process). Overall, they found mixed results for employment. At plants subject to both water and air pollution regulations, total employment declined by 6–7 percent, with no impact on wages. At plants subject only to air regulations, there was no significant change in total employment or in the employment of production workers, although wages increased by approximately 5 percent compared with unregulated plants. Addressing the Challenges Facing Retrospective Analysis These ex post studies of the Cluster Rule illustrate some of the main challenges facing retrospective analysis. As noted at the outset, for an ex post study to be successful, the analyst must have a control group of firms that is similar to the regulated firms and data on both sets of firms before and after the rule was implemented. In addition, the behavior of regulated firms and controls must be similar prior to implementation of the regulation. In the case of the Cluster Rule, some of the unregulated firms used different pulping processes and also differed in size from the regulated firms. To address these challenges, Gray et al. (2014) were careful to select a control group that consistently used a particular pulping technology (although it may have differed from the technology used by regulated firms) and to control for differences in plant size. Moreover, they confirmed that the employment trends of regulated and control plants were similar prior to regulation. Elrod and Malik (2017) also confirmed that key characteristics of the regulated and control plants were similar prior to implementation of the Cluster Rule. All of the Cluster Rule studies discussed here face important data challenges. In particular, a lack of data on emissions reduced the size of both the regulated and control groups in Gray and Shadbegian (2015) as well as Fraas and Egorenkov (2017). In fact, due to insufficient time series data on emissions, both studies were able to include only about half of the 155 plants regulated under the Cluster Rule in their analyses. Elrod and Malik (2017) faced similar data problems in their study of the impact of the Cluster Rule’s water regulations on the output choices of regulated plants. The NOx Budget Program The NOx Budget Program, which operated from 2003 through 2008, was a cap-and-trade program designed to reduce ground-level ozone by reducing emissions of nitrogen oxides (NOx) from electricity generating units in the midwestern and eastern United States.9 Because ozone formation occurs in the presence of sunlight, the program operated only between May 1 and September 30. This is also the period during which NOx emissions peak, due to increases in electricity demand for cooling. Control Groups The structure of the NOx program provides two natural control groups for retrospective analysis. More specifically, because regulated entities were subject to the program only in the summer months, a comparison of outcomes in regulated states in the winter months versus the summer months provides one set of controls. Because electricity generators in the western United States were not covered by the program, these generators provide a second set of controls. Estimating the Impact on NOx Emissions, Air Quality, and Health Using county-level data for 2001–2007, Deschênes, Greenstone, and Shapiro (2017) conduct an ex post analysis of the NOx Budget Program that examines the program’s impact on NOx emissions, ambient pollution concentrations, and health outcomes (mortality rates, hospital admissions, and medical expenditures). They find that the NOx Budget Program reduced NOx emissions from electric utilities by 35 percent, which yielded a modest reduction in summertime ambient ozone levels (6–7 percent relative to the baseline average) but a more significant reduction (34–42 percent) in the number of ozone days exceeding 65 parts per billion (ppb)—that is, levels close to or exceeding the EPA’s current ambient ozone standard of 70 ppb. The study also estimates that the NOx program reduced annual summertime mortality by roughly 2,500 deaths—75 percent of which were for the population 75 years of age or older—and reduced annual expenditures on medications by approximately $800 million per year (in 2015 dollars). In fact, these impacts on ozone-related mortality are larger than the estimates in the original RIA (U.S. Environmental Protection Agency 1998). Moreover, the study also measured an important category of benefits—reduced medical expenditures—that was not considered in the RIA. One could argue that these findings justify a stronger program and should be used in future RIAs. The ex post analysis also indicates a causal link between the NOx program and emissions, ambient air quality, and important categories of benefits. Addressing the Challenges Facing Retrospective Analysis Although the design of the NOx Budget Program provided natural control groups for a retrospective analysis, the analysis also required data on the relevant endpoints (i.e., mortality by cause and location, expenditures on drugs and hospital admissions). Data were readily available on emissions for electricity generators through the EPA’s Continuous Emissions Monitoring System (CEMS),10 and on ambient pollution through the Aerometric Information Retrieval System (AIRS) database.11 However, obtaining information on emergency room visits and medical expenditures required purchasing data from a private provider. Thus greater availability of data on relevant health endpoints would facilitate future retrospective analyses of regulatory benefits. Federal Gasoline Content Regulations An important strategy for reducing ground-level ozone in urban areas is to reduce VOCs in gasoline and to inhibit their ability to vaporize at high temperatures (U.S. Environmental Protection Agency 1993). U.S. federal gasoline regulations include limits on gasoline volatility and requirements for reformulated gasoline, which govern emissions of NOx, VOCs, and toxic air contaminants (TACs) from gasoline fueling and combustion.12 Volatility regulations are imposed only in the summer months and are more stringent in ozone nonattainment areas.13 Reformulated gasoline regulations are imposed only in nonattainment areas.14 Regulations on NOx and TACs are imposed throughout the year, whereas VOC regulations are imposed only in the summer months, because ozone formation is more of a problem in the summer. The spatial variation in federal gasoline regulations (i.e., the difference in regulations between attainment and nonattainment areas) and the fact that VOC regulations apply only in the summer facilitate an ex post analysis of their impacts. Retrospective analysis of these gasoline regulations provides insights concerning their impact on ground-level ozone levels (and thus their benefits) as well as their costs. Evaluating the Benefits of Gasoline Regulations To evaluate the benefits of federal gasoline regulations, Auffhammer and Kellogg (2011) examine their impact on ambient ozone concentrations using data from ozone monitoring stations for 1989–2003. Although the regulations are applied differently in nonattainment versus attainment counties, which creates a natural control group, the preferred approach of Auffhammer and Kellogg (2011) is to estimate the impact of the regulations by focusing on changes in ozone concentrations around a narrow window of time—that is, when VOC regulations begin (and end) each summer. They also control for NOx limits in electricity generating stations on the U.S. East Coast, which were implemented at the same time as the reformulated gasoline regulations. The estimates in Auffhammer and Kellogg (2011) suggest that more stringent gasoline volatility standards have not had a significant impact on ozone concentrations. They find that the reformulated gasoline regulations have a small, statistically significant impact in reducing ozone in some models, but not when using the authors’ preferred approach. They attribute this “non-effect” to the substantial flexibility in the design of gasoline volatility and reformulated gasoline regulations, which allows refiners to choose which VOCs to reduce in response to the regulation. They suggest that refiners complied with the regulation by targeting only those VOCs that could be reduced cost-effectively (primarily butane). Unfortunately, the VOCs that were more expensive to eliminate were also those that contribute most to ozone formation.15 Thus the findings in Auffhammer and Kellogg (2011) would appear to have clear implications for the design of federal gasoline regulations: increased regulatory flexibility is desirable, but not when it undermines the achievement of regulatory objectives. Evaluating the Costs of Gasoline Regulations On the cost side, ex post analyses have focused on the impacts of the regulations on the market for gasoline. In an assessment of the impact of reformulated gasoline regulations on the wholesale price of gasoline, Brown et al. (2008) focus on the spatial heterogeneity in the regulations. Consistent with EPA’s original RIA, Brown et al. (2008) find that reformulated gasoline increases the average wholesale gasoline price by about 3 cents per gallon. However, they also find substantial heterogeneity in the impact across regions, which they attribute to increases in market power due to market segmentation. In particular, because reformulated gasoline regulations differ in stringency according to ozone attainment status, they divide the U.S. gasoline market into segments that cannot arbitrage away price differentials. In market segments with few refineries, this provides the opportunity to exercise market power. Chakravorty, Nauges, and Thomas (2008) estimate a similar model, finding that the reformulated gasoline standards increase gasoline prices by 16 percent relative to a region with no regulations. Again, they find that a portion of the price increase is due to the segmentation of the market imposed by the regulations, which affords refiners the opportunity to exercise market power. These findings led the U.S. Congress (in the Energy Policy Act of 2005) to constrain the EPA’s ability to issue gasoline regulations that could exacerbate market segmentation (Auffhammer and Kellogg 2011). Addressing the Challenges Facing Retrospective Analysis These retrospective analyses of federal gasoline content regulations were facilitated by the fact that regulations were imposed with different stringency in attainment versus nonattainment areas and gasoline volatility and VOC content were regulated only in the summer. In addition, the endpoints examined by Auffhammer and Kellogg (2011) (ambient ozone concentrations) and by Brown et al. (2008) and Chakravorty, Nauges, and Thomas (2008) (gasoline prices) are ones for which data are readily available. British Columbia’s Carbon Tax In 2008, the Canadian province of British Columbia became the first region in North America to implement a substantial carbon tax (Murray and Rivers 2015). The tax was phased in between 2008 and 2012, eventually reaching Can$30 per ton of carbon dioxide (CO2) and covering all greenhouse gas emissions in the province released as a result of fuel combustion. Although the tax was not subject to a formal RIA, an ex ante analysis conducted to support the introduction of the tax suggested that the carbon tax—in the absence of any other policies—could reduce emissions in the province by 3 megaton of CO2 annually, or about 4 percent of total projected emissions (Government of British Columbia 2008). Retrospective studies on the impact of this policy on emissions are particularly challenging because all combustion greenhouse gas emissions in the province are covered with equal stringency, which means there is no natural within-province control group. Notwithstanding the challenge presented by the all-inclusive design of the carbon tax, retrospective studies have been conducted to determine the impact of the tax on gasoline consumption and employment within the province. Estimating Impacts on Gasoline Consumption Rivers and Schaufele (2015), Antweiler and Gulati (2016), and Lawley and Thivierge (2018) all estimate the impact of the carbon tax on gasoline consumption by using other provinces as a control group. Rivers and Schaufele (2015) and Antweiler and Gulati (2016) use province-level monthly data on gasoline sales in British Columbia and other provinces and find that the carbon tax has reduced gasoline consumption by an estimated 6–8 percent. In addition, using data on vehicle fuel economy in British Columbia and other provinces, Antweiler and Gulati (2016) find that approximately half of the reduction in gasoline consumption is due to the purchase of vehicles with higher fuel economy, while the other half is due to reduced transport demand. Lawley and Thivierge (2018) estimate the impact of the carbon tax on gasoline sales using data for households in British Columbia and other provinces (as a control group)16 and find results that are similar to those in Rivers and Schaufele (2015) and Antweiler and Gulati (2016). All three studies find that the impacts of the carbon tax on gasoline consumption are greater than what the literature on the price elasticity of gasoline demand suggests (see, e.g., Hughes, Knittel, and Sperling 2008). Specifically, based on their results, Antweiler and Gulati (2016) estimate a gasoline demand elasticity of −1.3, while the elasticity implied by the results in Rivers and Schaufele (2015) and Lawley and Thivierge (2018) is approximately −1.0.17Rivers and Schaufele (2015) suggest that the larger impact is due to the fact that the carbon tax was more visible to consumers (due to widespread media attention surrounding the tax) than other changes in gasoline prices and thus resulted in a larger behavioral response.18Antweiler and Gulati (2016) note that the carbon tax is less volatile than other components of the gasoline price and, as a result, is more likely to encourage consumers to improve the fuel economy of their vehicles than other changes in the gasoline price. Estimating Employment Impacts Yamazaki (2017b) estimates the impact of the carbon tax on employment using data on aggregate employment in more and less emissions-intensive sectors in British Columbia and other provinces both before and after the introduction of the carbon tax. He finds that the carbon tax resulted in a shift in employment from more energy-intensive to less energy-intensive sectors of the economy. Interestingly, however, he finds that the increase in employment in less emissions-intensive sectors outweighed the loss of employment in energy-intensive sectors, which means the net effect of the carbon tax was a small increase in overall employment in British Columbia. Using manufacturing plant-level data on employment and wages, and output data from the manufacturing sector (both before and after the introduction of the carbon tax and from British Columbia and other provinces), Yamazaki (2017a) estimates the impact of the carbon tax on these outcomes at the plant level.19 Consistent with his earlier, more aggregate analysis, Yamazaki (2017a) finds a reduction in production-line employment (but not employment of nonproduction workers) in the manufacturing sector. Yip (2018) also examines the employment effects of the carbon tax in British Columbia, using data on individual responses to a monthly labor force survey, and compares workers in British Columbia to those in other provinces before and after implementation of the carbon tax. He finds that the carbon tax is associated with an increase in the unemployment rate for less-educated males. In another study of the employment impacts of British Columbia’s carbon tax, Azevedo, Wolff, and Yamazaki (2017) compare industry employment in the province to a control group of other provinces that offers a close match of pretreatment trends. The results in Azevedo, Wolff, and Yamazaki (2017) suggest that the carbon tax had little measurable impact on overall employment and that even among energy-intensive sectors, employment impacts were typically too small to be precisely measured. The Challenges of Evaluating British Columbia’s Carbon Tax All of these retrospective studies of British Columbia’s carbon tax highlight the challenges associated with evaluating the impact of such a broadly based policy, especially one that is applied uniformly across an entire jurisdiction and thus does not provide a natural control group. These studies show that in such a context, the choice of control group and the research design can influence the results. Lessons for Facilitating Retrospective Analysis Sunstein (2011) has emphasized the need for a “culture of retrospective analysis and review.” The regulations and studies we have examined here illustrate both the usefulness of retrospective analyses of regulations and the challenges of conducting them. How could future retrospective analyses be facilitated? Identify Control Groups The biggest challenge for retrospective analyses of regulations is the choice of an appropriate control group. In the studies we have reviewed here, the identification of control groups was relatively straightforward when regulations were implemented with different stringency in different geographic regions (the NOx Budget Program, gasoline content regulations, and British Columbia’s carbon tax) or at different times of the year (the NOx Budget Program and gasoline content regulations). For the Cluster Rule, control groups reflected the different standards across firms within the same industry. However, the studies also show that the choice of control group can have an important impact on estimates of the impact of the policy. As a rule, it may be possible to use firms in an industry (or jurisdiction) that are not subject to the regulation as a control group, provided they follow trends similar to those of regulated firms in terms of key outcome variables before implementation of the regulation and they face similar shocks (aside from the regulation). Improve Data and Data Collection The choice of control group—and the ability to conduct a retrospective analysis—also depends on data availability at the micro and/or more aggregate levels. As noted earlier, studies of the impact of the Cluster Rule have been hampered by a lack of emissions data on both regulated and control plants. In contrast, data on emissions for electricity generating units facilitated the analysis of the NOx Budget Program. The availability of data on ambient air pollution made it possible for studies to estimate the impact of the NOx program and federal gasoline content regulations on ambient ozone levels. In the case of British Columbia’s carbon tax, studies have used both aggregated and microdata on gasoline sales and employment to study the impact of the policy on these outcomes. However, difficulty in obtaining data has hampered ex post analyses of compliance costs (Kopits et al. 2014; Cropper, Fraas, and Morgenstern 2017). In some cases (e.g., CEMS and AIRS), the government simply needs to continue existing data collection efforts. However, in many other cases (e.g., data on pollution control expenditures), new data are required. In the United States, the Office of Management and Budget could streamline data collection under the Paperwork Reduction Act, as has already been done for research-related information (Sunstein 2011). Design New Rules to Institutionalize Retrospective Analysis More proactively, and in an effort to further institutionalize retrospective analyses of environmental regulations, new rules could be designed to facilitate such analyses. This could involve, for example, the regulatory agency identifying at the time of rulemaking the outcomes to be chosen for retrospective analysis. It could also involve stipulating at the outset the types of measurable outcomes to be targeted in individual regulations, including a relevant control group, the associated data requirements to measure both compliance costs and environmental impacts, a power calculation of a minimum sample size to identify regulatory outcomes, and the time period for the evaluation. In some cases, this might involve coordinating data collection with other agencies or entities. Random phase-in of regulatory requirements, as occurred in the case of California’s RECLAIM20 program (Fowlie and Perloff 2013), might be feasible in some cases. Who Should Undertake Future Retrospective Analyses? Ideally, government agencies would devote resources to both conducting high-quality studies and ensuring independent peer review. In fact, the EPA has already started moving in this direction in the ex post analysis of regulatory costs (U.S. Environmental Protection Agency 2011; Kopits et al. 2014). At the same time, as indicated by the results reported here, academic research has an important role to play in this area. Thus, although obtaining funding is always an issue, academic research could clearly enhance and/or substitute for government agency analyses. Footnotes 1 This article focuses on ex post analyses of individual regulations. There is a parallel body of literature that evaluates the impacts of the Clean Air Act (CAA). However, this literature does not focus on specific rules. Rather, it uses spatial and temporal variation in attainment status under the CAA as a proxy for stricter regulation. Examples include Greenstone (2002), Chay and Greenstone (2005), Auffhammer, Bento and Lowe (2009), and Walker (2013). 2 See Federal Register 63, no. 72, p. 18575. 3 MACT represents the best demonstrated technology, defined as the average of the top 12 percent of performers in terms of control technology or practices within the regulated industry. 4 VOCs are organic chemicals that have a high vapor pressure at ordinary room temperature. 5 BAT represents the best available economically achievable performance of plants in the industrial subcategory or category. 6 Dioxins and furans are part of a family of toxic substances that share a similar chemical structure. Most dioxins and furans are not man-made or produced intentionally, but are created when other chemicals or products are made. 7 See table 5 in Gray and Shadbegian (2015). 8 Gray and Shadbegian (2015) also found smaller reductions in VOCs and PM than the EPA’s ex ante projections. However, they did find substantial reductions in chloroform discharges and emissions (consistent with the EPA’s projection), but these reductions began well before the final Cluster Rule was issued. Thus these reductions appear to have been stimulated by other EPA and state regulations. 9 Ground-level ozone is formed when NOx combine with VOCs in the presence of sunlight. Beginning in 2009, the NOx Budget Program was effectively replaced by the ozone season NOx program under the Clean Air Interstate Rule, which required further summertime NOx reductions from the power sector. See https://www.epa.gov/airmarkets/nox-budget-trading-program-historical-reports. 10 CEMS monitors power plants at hourly intervals to track emissions of sulfur dioxide, NOx, and other pollutants. See https://ampd.epa.gov/ampd/. 11 The AIRS database provides information about ambient air pollution in the United States. See https://cfpub.epa.gov/si/si_public_record_Report.cfm?dirEntryID=2779. 12 TACs include benzene, 1,3-butadiene, polycyclic organic hydrocarbons, formaldehyde, and acetaldehyde. All of these are carcinogens. 13 Gasoline volatility is measured by Reid vapor pressure, the absolute vapor pressure exerted by a liquid at 100°F. 14 Reformulated gasoline is gasoline blended to burn more cleanly than conventional gasoline. 15 This result appears to have been (in part) anticipated in the RIA (U.S. Environmental Protection Agency 1993), in which the EPA estimates that the reformulated gasoline regulations will actually increase the emissions of olefins, a particularly potent ozone-forming VOC. 16 Their detailed microdata from repeated expenditure surveys allows them to control for a number of household characteristics that could influence gasoline demand. 17 The Rivers and Schaufele (2015) estimate is derived from the coefficient in column 1 of table 2 in combination with the mean gasoline price in British Columbia reported in table A1. The Lawley and Thivierge (2018) estimate is derived from their “most conservative” semielasticity of −0.01 in combination with the mean gasoline price in British Columbia reported in table 1. 18 This is consistent with results obtained by Li, Linn, and Muehlegger (2014), who find that the response to a gasoline tax is three times larger than the response to a gasoline price change. They attribute this to the perceived persistence and greater salience of the tax. 19 He identifies an appropriate out-of-province control group to compare with the British Columbia plants covered by the tax and estimates the impact of the tax using this matched control group. 20 Regional Clean Air Incentives Market (RECLAIM) -- an emissions trading program operating in California since 1994 -- requires hundreds of polluting facilities to cut their emissions of NOx and sulfur oxides. References Antweiler W. , Gulati S. . 2016 . Frugal cars or frugal drivers? how carbon and fuel taxes influence the choice and use of cars. May 11, 2016. https://ssrn.com/abstract=2778868 or http://dx.doi.org/10.2139/ssrn.2778868. Auffhammer M. , Bento A. , Lowe S. . 2009 . Measuring the effects the Clean Air Act Amendments on ambient PM10 concentrations: the critical importance of a spatially disaggregated analysis . Journal of Environmental Economics and Management 58 1 : 15 – 26 . Google Scholar CrossRef Search ADS Auffhammer M. , Kellogg R. . 2011 . Clearing the air? The effects of gasoline content regulation on air quality . American Economic Review 101 6 : 2687 – 722 . Google Scholar CrossRef Search ADS Azevedo D. , Wolff H. , Yamazaki A. . 2017 . Do carbon taxes kill jobs? Evidence of heterogeneous impacts from British Columbia. December 20, 2017. Brown J. , Hastings J. , Mansur E. T. , Villas-Boas S. B. . 2008 . Reformulating competition? Gasoline content regulation and wholesale gasoline prices . Journal of Environmental Economics and Management 55 1 : 1 – 19 . Google Scholar CrossRef Search ADS Chakravorty U. , Nauges C. , Thomas A. . 2008 . Clean air regulation and heterogeneity in US gasoline prices . Journal of Environmental Economics and Management 55 1 : 106 – 22 . Google Scholar CrossRef Search ADS Chay K. Y. , Greenstone M. . 2005 . Does air quality matter? Evidence from the housing market . Journal of Political Economy 113 2 : 376 – 424 . Google Scholar CrossRef Search ADS Cropper M. , Fraas A. , Morgenstern R. . 2017 . Looking backward to move regulations forward . Science 355 6332 : 1375 – 76 . Google Scholar CrossRef Search ADS PubMed Deschênes O. , Greenstone M. , Shapiro J. S. . 2017 . Defensive investments and the demand for air quality: evidence from the NOx Budget Program . American Economic Review 107 10 : 2958 – 89 . Google Scholar CrossRef Search ADS Elrod A. A. , Malik A. S. . 2017 . The effect of environmental regulation on plant-level product mix: a study of EPA’s Cluster Rule . Journal of Environmental Economics and Management 83 : 164 – 84 . Google Scholar CrossRef Search ADS Fowlie M. , Perloff J. M. . 2013 . Distributing pollution rights in cap-and-trade programs: are outcomes independent of allocation? Review of Economics and Statistics 95 5 : 1640 – 52 . Google Scholar CrossRef Search ADS Fraas A. , Egorenkov A. . 2017 . Retrospective analyses are hard: a cautionary tale from EPA’s air toxics regulations . Journal of Benefit–Cost Analysis , 1 – 20 . doi:10.1017/bca.2017.8. Government of British Columbia . 2008 . Climate Action Plan. http://www.gov.bc.ca/premier/attachments/climate_action_plan.pdf. Gray W. B. , Shadbegian R. J. . 2015 . Multimedia pollution regulation and environmental performance: EPA’s Cluster Rule. Discussion paper DP 15-26, Resources for the Future, Washington, DC. Gray W. B. , Shadbegian R. J. , Wang C. , Meral M. . 2014 . Do EPA regulations affect labor demand? Evidence from the pulp and paper industry . Journal of Environmental Economics and Management 68 1 : 188 – 202 . Google Scholar CrossRef Search ADS Greenstone M. 2002 . The impacts of environmental regulations on industrial activity: evidence from the 1970 and 1977 Clean Air Act Amendments and the Census of Manufactures . Journal of Political Economy 110 6 : 1175 – 219 . Google Scholar CrossRef Search ADS Hughes J. E. , Knittel C. R. , Sperling D. . 2008 . Evidence of a shift in the short-run price elasticity of gasoline demand . Energy Journal 29 1 : 93 – 114 . Google Scholar CrossRef Search ADS Kopits E. , McGartland A. , Morgan C. , Pasurka C. , Shadbegian R. , Simon N. B. , Simpson D. , Wolverton A. . 2014 . Retrospective cost analyses of EPA regulations: a case study approach . Journal of Benefit–Cost Analysis 5 2 : 173 – 93 . Google Scholar CrossRef Search ADS Lawley C. , Thivierge V. . 2018. Refining the evidence: British Columbia’s carbon tax and household gasoline consumption. Energy Journal 39 2 : 147 – 71 . Li S. , Linn J. , Muehlegger E. . 2014 . Gasoline taxes and consumer behavior . American Economic Journal: Economic Policy 6 4 : 302 – 42 . Google Scholar CrossRef Search ADS Murray B. , Rivers N. . 2015 . British Columbia’s revenue-neutral carbon tax: a review of the latest “grand experiment” in environmental policy . Energy Policy 86 : 674 – 83 . Google Scholar CrossRef Search ADS Rivers N. , Schaufele B. . 2015 . Salience of carbon taxes in the gasoline market . Journal of Environmental Economics and Management 74 : 23 – 36 . Google Scholar CrossRef Search ADS Sunstein C. 2011 . Memorandum for the Heads of Executive Departments and Agencies: Facilitating Scientific Research by Streamlining the Paperwork Reduction Act Process. M-11-07. December 9, 2010. https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2011/m11-07.pdf. U.S. Environmental Protection Agency . 1993 . Final regulatory impact analysis for reformulated gasoline . EPA 420-R-93-017. Washington, DC : U.S. Environmental Protection Agency . U.S. Environmental Protection Agency . 1998 . Regulatory impact analysis for the NOx SIP call, FIP, and Section 126 Petitions, vols. 1 and 2. EPA-452/R- 98-003A and EPA-452/R-98-003. Washington, DC: U.S. Environmental Protection Agency. U.S. Environmental Protection Agency . 2011 . Improving our regulations: final plan for periodic retrospective reviews of existing regulations. Washington, DC : U.S. Environmental Protection Agency . Walker W. R. 2013 . The transitional costs of sectoral reallocation: evidence from the Clean Air Act and the workforce . Quarterly Journal of Economics 128 4 : 1787 – 835 . Google Scholar CrossRef Search ADS Yamazaki Akio. 2017a . The effect of climate policy on performance of manufacturing plants. Working paper. April 2017. Yamazaki Akio. . 2017b . Jobs and climate policy: evidence from British Columbia’s revenue-neutral carbon tax . Journal of Environmental Economics and Management 83 : 197 – 216 . Google Scholar CrossRef Search ADS Yip C. M. 2018 . On the labor market consequences of environmental taxes . Journal of Environmental Economics and Management 89 : 136 – 52 . Google Scholar CrossRef Search ADS © The Author(s) 2018. Published by Oxford University Press on behalf of the Association of Environmental and Resource Economists. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Review of Environmental Economics and Policy Oxford University Press

Policy Brief—Facilitating Retrospective Analysis of Environmental Regulations

Loading next page...
 
/lp/ou_press/facilitating-retrospective-analysis-of-environmental-regulations-aDmQV0302w
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of the Association of Environmental and Resource Economists. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
1750-6816
eISSN
1750-6824
D.O.I.
10.1093/reep/rey011
Publisher site
See Article on Publisher Site

Abstract

Abstract Prospective or ex ante studies of the costs, benefits, and distributional impacts of new environmental regulations are now commonly performed in many countries. Retrospective analyses, which aim to document actual outcomes, are far less common. The purpose of this policy brief is to illustrate the value of retrospective analysis of environmental regulations, discuss the main challenges of conducting such studies, and make suggestions for facilitating the conduct of retrospective analyses. We examine recent examples of ex post analyses of three sets of U.S. regulations—the Environmental Protection Agency (EPA) Cluster Rule, the NOx Budget Program, and federal gasoline content regulations—and British Columbia’s carbon tax. Based on this review, we offer some lessons for facilitating future retrospective analysis of environmental regulations. Why Is Retrospective Analysis Important? It is common practice for governments to assess the efficiency and effectiveness of prospective environmental regulations through ex ante analysis prior to implementation. In the United States, for example, environmental regulations are evaluated using regulatory impact analysis (RIA), which projects both the costs and benefits associated with a proposed rule. For regulations designed to limit pollution emissions or discharges (hereafter referred to as emissions), RIAs generally entail four steps: (1) estimating emissions with and without the rule, (2) translating the change in emissions into environmental outcomes, (3) estimating the benefits associated with the expected change in environmental outcomes, and (4) estimating the costs of the regulation relative to the no-regulation baseline. Conducting an RIA requires predicting the state of the world with and without the regulation, both of which are inherently uncertain. After a regulation is implemented, the methods used by firms, consumers, and other regulated entities to comply with the rule, as well as actual emissions and costs, can, in principle, be observed. Although the counterfactual no-regulation baseline can never be observed for regulated entities, there are cases in which it can be inferred from a control group of similar, unregulated entities. Thus, in principle, it is possible to evaluate some (or all) of the four steps in RIAs retrospectively (i.e., ex post). Why is retrospective analysis important? First, it is important to investigate whether the regulation had the intended impact on emissions and environmental outcomes. Establishing a causal impact on environmental outcomes is a necessary condition for ensuring that regulations are achieving their desired objectives. Moreover, if a regulation was not fully effective in achieving its intended objectives, retrospective analysis can help to reveal the factors that were responsible for the regulation’s failure. This information can be used to improve both the design of future regulations and ex ante RIAs. Second, it is important to measure the actual costs of a regulation. Actual costs may differ from ex ante estimates because of unforeseen behavioral adaptations by consumers or firms. Understanding these adaptations can also help to improve the design of future regulations and ex ante RIAs. Third, retrospective analysis may make it possible to obtain causal estimates of a regulation’s benefits or its effects on outcomes other than economic efficiency—for example, the impacts of the regulation on employment. Finally, over the long term, retrospective analysis can support the type of institutional learning needed to improve both the effectiveness and efficiency of environmental management. The purpose of this policy brief is to illustrate the value of retrospective analysis of environmental regulations, discuss the main challenges of conducting such studies, and make suggestions for facilitating the conduct of retrospective analyses.1 With this in mind, we examine recent examples of ex post analyses of three sets of U.S. regulations—the Environmental Protection Agency (EPA) Cluster Rule, the NOx Budget Program, and federal gasoline content regulations—and British Columbia’s carbon tax. For a retrospective analysis to be successful, researchers must identify a group targeted by the regulation and an otherwise similar control group that is not affected by it. This means that there must be data for the two groups—both before the regulation was implemented and afterwards. Indeed, these are the main challenges facing a retrospective analysis. We consider how each of the ex post studies we examine addressed these challenges. We conclude by providing lessons for conducting future retrospective analyses. EPA’s Cluster Rule EPA’s 1998 Cluster Rule was the agency’s first integrated, multimedia regulation designed to reduce hazardous air and water pollutants from pulp and paper mills.2 The rule imposed stringent maximum achievable control technology (MACT) standards3 to reduce emissions of benzene and other hazardous air pollutants, sulfur dioxide, particulate matter 10 µm or less in diameter (PM10), and volatile organic chemicals (VOCs)4 on 155 mills that used chemical pulping techniques. The 335 mills that used mechanical pulping techniques or purchased pulp faced less stringent standards on air emissions. Ninety-six of the 155 chemical pulping plants were also subject to best available technology economically achievable (BAT) standards5 to reduce water discharges of chloroform, dioxin, and furans.6 Estimating the Impact on Emissions and Employment Because not all pulp and paper firms were subject to the same requirements under the Cluster Rule, unregulated or less stringently regulated firms provided a control group for estimating the rule’s impact on air and water emissions and employment. Using a subset of the 335 plants subject to neither MACT nor BAT standards as controls, Gray and Shadbegian (2015) find that being subject to both types of standards effectively eliminated chloroform emissions to both air and water and reduced air emissions of VOCs by 28 percent, but had no statistically significant impact on PM10. MACT standards alone, however, had no statistically significant impact on air emissions—a potentially troubling result.7 Using different control groups and a different data source, Fraas and Egorenkov (2017) find that plants subject to MACT plus BAT regulations did not reduce their air emissions as much as the EPA projected ex ante. They use plants in the plywood industry as a control group for pulp and paper plants and find that the reduction in organic hazardous air pollutants associated with the Cluster Rule is smaller than the EPA’s ex ante prediction of a 58 percent reduction.8 A similar result (i.e., smaller reductions than projected ex ante by the EPA) is found using a control group of plants in six unrelated metals and plastics parts manufacturing industries. These studies of the Cluster Rule suggest that the results of ex post analysis can be highly sensitive to the control group used. The ex post analyses also suggest that in some cases the particular method regulated firms adopt in order to comply with a rule may be important. For example, using plants subject to MACT-only standards as a control group for those subject to both MACT and BAT standards, Elrod and Malik (2017) find that plants subject to water as well as air regulations were somewhat more likely to switch from bleached to unbleached paper products. This suggests that one method plants used to comply with the BAT rules was to switch the composition of output rather than the nature of the production process—an unexpected result. To study the impact of the Cluster Rule on employment, Gray et al. (2014) included a subset of unregulated plants that use technologies similar to the technologies used by regulated plants (i.e., some form of pulping process). Overall, they found mixed results for employment. At plants subject to both water and air pollution regulations, total employment declined by 6–7 percent, with no impact on wages. At plants subject only to air regulations, there was no significant change in total employment or in the employment of production workers, although wages increased by approximately 5 percent compared with unregulated plants. Addressing the Challenges Facing Retrospective Analysis These ex post studies of the Cluster Rule illustrate some of the main challenges facing retrospective analysis. As noted at the outset, for an ex post study to be successful, the analyst must have a control group of firms that is similar to the regulated firms and data on both sets of firms before and after the rule was implemented. In addition, the behavior of regulated firms and controls must be similar prior to implementation of the regulation. In the case of the Cluster Rule, some of the unregulated firms used different pulping processes and also differed in size from the regulated firms. To address these challenges, Gray et al. (2014) were careful to select a control group that consistently used a particular pulping technology (although it may have differed from the technology used by regulated firms) and to control for differences in plant size. Moreover, they confirmed that the employment trends of regulated and control plants were similar prior to regulation. Elrod and Malik (2017) also confirmed that key characteristics of the regulated and control plants were similar prior to implementation of the Cluster Rule. All of the Cluster Rule studies discussed here face important data challenges. In particular, a lack of data on emissions reduced the size of both the regulated and control groups in Gray and Shadbegian (2015) as well as Fraas and Egorenkov (2017). In fact, due to insufficient time series data on emissions, both studies were able to include only about half of the 155 plants regulated under the Cluster Rule in their analyses. Elrod and Malik (2017) faced similar data problems in their study of the impact of the Cluster Rule’s water regulations on the output choices of regulated plants. The NOx Budget Program The NOx Budget Program, which operated from 2003 through 2008, was a cap-and-trade program designed to reduce ground-level ozone by reducing emissions of nitrogen oxides (NOx) from electricity generating units in the midwestern and eastern United States.9 Because ozone formation occurs in the presence of sunlight, the program operated only between May 1 and September 30. This is also the period during which NOx emissions peak, due to increases in electricity demand for cooling. Control Groups The structure of the NOx program provides two natural control groups for retrospective analysis. More specifically, because regulated entities were subject to the program only in the summer months, a comparison of outcomes in regulated states in the winter months versus the summer months provides one set of controls. Because electricity generators in the western United States were not covered by the program, these generators provide a second set of controls. Estimating the Impact on NOx Emissions, Air Quality, and Health Using county-level data for 2001–2007, Deschênes, Greenstone, and Shapiro (2017) conduct an ex post analysis of the NOx Budget Program that examines the program’s impact on NOx emissions, ambient pollution concentrations, and health outcomes (mortality rates, hospital admissions, and medical expenditures). They find that the NOx Budget Program reduced NOx emissions from electric utilities by 35 percent, which yielded a modest reduction in summertime ambient ozone levels (6–7 percent relative to the baseline average) but a more significant reduction (34–42 percent) in the number of ozone days exceeding 65 parts per billion (ppb)—that is, levels close to or exceeding the EPA’s current ambient ozone standard of 70 ppb. The study also estimates that the NOx program reduced annual summertime mortality by roughly 2,500 deaths—75 percent of which were for the population 75 years of age or older—and reduced annual expenditures on medications by approximately $800 million per year (in 2015 dollars). In fact, these impacts on ozone-related mortality are larger than the estimates in the original RIA (U.S. Environmental Protection Agency 1998). Moreover, the study also measured an important category of benefits—reduced medical expenditures—that was not considered in the RIA. One could argue that these findings justify a stronger program and should be used in future RIAs. The ex post analysis also indicates a causal link between the NOx program and emissions, ambient air quality, and important categories of benefits. Addressing the Challenges Facing Retrospective Analysis Although the design of the NOx Budget Program provided natural control groups for a retrospective analysis, the analysis also required data on the relevant endpoints (i.e., mortality by cause and location, expenditures on drugs and hospital admissions). Data were readily available on emissions for electricity generators through the EPA’s Continuous Emissions Monitoring System (CEMS),10 and on ambient pollution through the Aerometric Information Retrieval System (AIRS) database.11 However, obtaining information on emergency room visits and medical expenditures required purchasing data from a private provider. Thus greater availability of data on relevant health endpoints would facilitate future retrospective analyses of regulatory benefits. Federal Gasoline Content Regulations An important strategy for reducing ground-level ozone in urban areas is to reduce VOCs in gasoline and to inhibit their ability to vaporize at high temperatures (U.S. Environmental Protection Agency 1993). U.S. federal gasoline regulations include limits on gasoline volatility and requirements for reformulated gasoline, which govern emissions of NOx, VOCs, and toxic air contaminants (TACs) from gasoline fueling and combustion.12 Volatility regulations are imposed only in the summer months and are more stringent in ozone nonattainment areas.13 Reformulated gasoline regulations are imposed only in nonattainment areas.14 Regulations on NOx and TACs are imposed throughout the year, whereas VOC regulations are imposed only in the summer months, because ozone formation is more of a problem in the summer. The spatial variation in federal gasoline regulations (i.e., the difference in regulations between attainment and nonattainment areas) and the fact that VOC regulations apply only in the summer facilitate an ex post analysis of their impacts. Retrospective analysis of these gasoline regulations provides insights concerning their impact on ground-level ozone levels (and thus their benefits) as well as their costs. Evaluating the Benefits of Gasoline Regulations To evaluate the benefits of federal gasoline regulations, Auffhammer and Kellogg (2011) examine their impact on ambient ozone concentrations using data from ozone monitoring stations for 1989–2003. Although the regulations are applied differently in nonattainment versus attainment counties, which creates a natural control group, the preferred approach of Auffhammer and Kellogg (2011) is to estimate the impact of the regulations by focusing on changes in ozone concentrations around a narrow window of time—that is, when VOC regulations begin (and end) each summer. They also control for NOx limits in electricity generating stations on the U.S. East Coast, which were implemented at the same time as the reformulated gasoline regulations. The estimates in Auffhammer and Kellogg (2011) suggest that more stringent gasoline volatility standards have not had a significant impact on ozone concentrations. They find that the reformulated gasoline regulations have a small, statistically significant impact in reducing ozone in some models, but not when using the authors’ preferred approach. They attribute this “non-effect” to the substantial flexibility in the design of gasoline volatility and reformulated gasoline regulations, which allows refiners to choose which VOCs to reduce in response to the regulation. They suggest that refiners complied with the regulation by targeting only those VOCs that could be reduced cost-effectively (primarily butane). Unfortunately, the VOCs that were more expensive to eliminate were also those that contribute most to ozone formation.15 Thus the findings in Auffhammer and Kellogg (2011) would appear to have clear implications for the design of federal gasoline regulations: increased regulatory flexibility is desirable, but not when it undermines the achievement of regulatory objectives. Evaluating the Costs of Gasoline Regulations On the cost side, ex post analyses have focused on the impacts of the regulations on the market for gasoline. In an assessment of the impact of reformulated gasoline regulations on the wholesale price of gasoline, Brown et al. (2008) focus on the spatial heterogeneity in the regulations. Consistent with EPA’s original RIA, Brown et al. (2008) find that reformulated gasoline increases the average wholesale gasoline price by about 3 cents per gallon. However, they also find substantial heterogeneity in the impact across regions, which they attribute to increases in market power due to market segmentation. In particular, because reformulated gasoline regulations differ in stringency according to ozone attainment status, they divide the U.S. gasoline market into segments that cannot arbitrage away price differentials. In market segments with few refineries, this provides the opportunity to exercise market power. Chakravorty, Nauges, and Thomas (2008) estimate a similar model, finding that the reformulated gasoline standards increase gasoline prices by 16 percent relative to a region with no regulations. Again, they find that a portion of the price increase is due to the segmentation of the market imposed by the regulations, which affords refiners the opportunity to exercise market power. These findings led the U.S. Congress (in the Energy Policy Act of 2005) to constrain the EPA’s ability to issue gasoline regulations that could exacerbate market segmentation (Auffhammer and Kellogg 2011). Addressing the Challenges Facing Retrospective Analysis These retrospective analyses of federal gasoline content regulations were facilitated by the fact that regulations were imposed with different stringency in attainment versus nonattainment areas and gasoline volatility and VOC content were regulated only in the summer. In addition, the endpoints examined by Auffhammer and Kellogg (2011) (ambient ozone concentrations) and by Brown et al. (2008) and Chakravorty, Nauges, and Thomas (2008) (gasoline prices) are ones for which data are readily available. British Columbia’s Carbon Tax In 2008, the Canadian province of British Columbia became the first region in North America to implement a substantial carbon tax (Murray and Rivers 2015). The tax was phased in between 2008 and 2012, eventually reaching Can$30 per ton of carbon dioxide (CO2) and covering all greenhouse gas emissions in the province released as a result of fuel combustion. Although the tax was not subject to a formal RIA, an ex ante analysis conducted to support the introduction of the tax suggested that the carbon tax—in the absence of any other policies—could reduce emissions in the province by 3 megaton of CO2 annually, or about 4 percent of total projected emissions (Government of British Columbia 2008). Retrospective studies on the impact of this policy on emissions are particularly challenging because all combustion greenhouse gas emissions in the province are covered with equal stringency, which means there is no natural within-province control group. Notwithstanding the challenge presented by the all-inclusive design of the carbon tax, retrospective studies have been conducted to determine the impact of the tax on gasoline consumption and employment within the province. Estimating Impacts on Gasoline Consumption Rivers and Schaufele (2015), Antweiler and Gulati (2016), and Lawley and Thivierge (2018) all estimate the impact of the carbon tax on gasoline consumption by using other provinces as a control group. Rivers and Schaufele (2015) and Antweiler and Gulati (2016) use province-level monthly data on gasoline sales in British Columbia and other provinces and find that the carbon tax has reduced gasoline consumption by an estimated 6–8 percent. In addition, using data on vehicle fuel economy in British Columbia and other provinces, Antweiler and Gulati (2016) find that approximately half of the reduction in gasoline consumption is due to the purchase of vehicles with higher fuel economy, while the other half is due to reduced transport demand. Lawley and Thivierge (2018) estimate the impact of the carbon tax on gasoline sales using data for households in British Columbia and other provinces (as a control group)16 and find results that are similar to those in Rivers and Schaufele (2015) and Antweiler and Gulati (2016). All three studies find that the impacts of the carbon tax on gasoline consumption are greater than what the literature on the price elasticity of gasoline demand suggests (see, e.g., Hughes, Knittel, and Sperling 2008). Specifically, based on their results, Antweiler and Gulati (2016) estimate a gasoline demand elasticity of −1.3, while the elasticity implied by the results in Rivers and Schaufele (2015) and Lawley and Thivierge (2018) is approximately −1.0.17Rivers and Schaufele (2015) suggest that the larger impact is due to the fact that the carbon tax was more visible to consumers (due to widespread media attention surrounding the tax) than other changes in gasoline prices and thus resulted in a larger behavioral response.18Antweiler and Gulati (2016) note that the carbon tax is less volatile than other components of the gasoline price and, as a result, is more likely to encourage consumers to improve the fuel economy of their vehicles than other changes in the gasoline price. Estimating Employment Impacts Yamazaki (2017b) estimates the impact of the carbon tax on employment using data on aggregate employment in more and less emissions-intensive sectors in British Columbia and other provinces both before and after the introduction of the carbon tax. He finds that the carbon tax resulted in a shift in employment from more energy-intensive to less energy-intensive sectors of the economy. Interestingly, however, he finds that the increase in employment in less emissions-intensive sectors outweighed the loss of employment in energy-intensive sectors, which means the net effect of the carbon tax was a small increase in overall employment in British Columbia. Using manufacturing plant-level data on employment and wages, and output data from the manufacturing sector (both before and after the introduction of the carbon tax and from British Columbia and other provinces), Yamazaki (2017a) estimates the impact of the carbon tax on these outcomes at the plant level.19 Consistent with his earlier, more aggregate analysis, Yamazaki (2017a) finds a reduction in production-line employment (but not employment of nonproduction workers) in the manufacturing sector. Yip (2018) also examines the employment effects of the carbon tax in British Columbia, using data on individual responses to a monthly labor force survey, and compares workers in British Columbia to those in other provinces before and after implementation of the carbon tax. He finds that the carbon tax is associated with an increase in the unemployment rate for less-educated males. In another study of the employment impacts of British Columbia’s carbon tax, Azevedo, Wolff, and Yamazaki (2017) compare industry employment in the province to a control group of other provinces that offers a close match of pretreatment trends. The results in Azevedo, Wolff, and Yamazaki (2017) suggest that the carbon tax had little measurable impact on overall employment and that even among energy-intensive sectors, employment impacts were typically too small to be precisely measured. The Challenges of Evaluating British Columbia’s Carbon Tax All of these retrospective studies of British Columbia’s carbon tax highlight the challenges associated with evaluating the impact of such a broadly based policy, especially one that is applied uniformly across an entire jurisdiction and thus does not provide a natural control group. These studies show that in such a context, the choice of control group and the research design can influence the results. Lessons for Facilitating Retrospective Analysis Sunstein (2011) has emphasized the need for a “culture of retrospective analysis and review.” The regulations and studies we have examined here illustrate both the usefulness of retrospective analyses of regulations and the challenges of conducting them. How could future retrospective analyses be facilitated? Identify Control Groups The biggest challenge for retrospective analyses of regulations is the choice of an appropriate control group. In the studies we have reviewed here, the identification of control groups was relatively straightforward when regulations were implemented with different stringency in different geographic regions (the NOx Budget Program, gasoline content regulations, and British Columbia’s carbon tax) or at different times of the year (the NOx Budget Program and gasoline content regulations). For the Cluster Rule, control groups reflected the different standards across firms within the same industry. However, the studies also show that the choice of control group can have an important impact on estimates of the impact of the policy. As a rule, it may be possible to use firms in an industry (or jurisdiction) that are not subject to the regulation as a control group, provided they follow trends similar to those of regulated firms in terms of key outcome variables before implementation of the regulation and they face similar shocks (aside from the regulation). Improve Data and Data Collection The choice of control group—and the ability to conduct a retrospective analysis—also depends on data availability at the micro and/or more aggregate levels. As noted earlier, studies of the impact of the Cluster Rule have been hampered by a lack of emissions data on both regulated and control plants. In contrast, data on emissions for electricity generating units facilitated the analysis of the NOx Budget Program. The availability of data on ambient air pollution made it possible for studies to estimate the impact of the NOx program and federal gasoline content regulations on ambient ozone levels. In the case of British Columbia’s carbon tax, studies have used both aggregated and microdata on gasoline sales and employment to study the impact of the policy on these outcomes. However, difficulty in obtaining data has hampered ex post analyses of compliance costs (Kopits et al. 2014; Cropper, Fraas, and Morgenstern 2017). In some cases (e.g., CEMS and AIRS), the government simply needs to continue existing data collection efforts. However, in many other cases (e.g., data on pollution control expenditures), new data are required. In the United States, the Office of Management and Budget could streamline data collection under the Paperwork Reduction Act, as has already been done for research-related information (Sunstein 2011). Design New Rules to Institutionalize Retrospective Analysis More proactively, and in an effort to further institutionalize retrospective analyses of environmental regulations, new rules could be designed to facilitate such analyses. This could involve, for example, the regulatory agency identifying at the time of rulemaking the outcomes to be chosen for retrospective analysis. It could also involve stipulating at the outset the types of measurable outcomes to be targeted in individual regulations, including a relevant control group, the associated data requirements to measure both compliance costs and environmental impacts, a power calculation of a minimum sample size to identify regulatory outcomes, and the time period for the evaluation. In some cases, this might involve coordinating data collection with other agencies or entities. Random phase-in of regulatory requirements, as occurred in the case of California’s RECLAIM20 program (Fowlie and Perloff 2013), might be feasible in some cases. Who Should Undertake Future Retrospective Analyses? Ideally, government agencies would devote resources to both conducting high-quality studies and ensuring independent peer review. In fact, the EPA has already started moving in this direction in the ex post analysis of regulatory costs (U.S. Environmental Protection Agency 2011; Kopits et al. 2014). At the same time, as indicated by the results reported here, academic research has an important role to play in this area. Thus, although obtaining funding is always an issue, academic research could clearly enhance and/or substitute for government agency analyses. Footnotes 1 This article focuses on ex post analyses of individual regulations. There is a parallel body of literature that evaluates the impacts of the Clean Air Act (CAA). However, this literature does not focus on specific rules. Rather, it uses spatial and temporal variation in attainment status under the CAA as a proxy for stricter regulation. Examples include Greenstone (2002), Chay and Greenstone (2005), Auffhammer, Bento and Lowe (2009), and Walker (2013). 2 See Federal Register 63, no. 72, p. 18575. 3 MACT represents the best demonstrated technology, defined as the average of the top 12 percent of performers in terms of control technology or practices within the regulated industry. 4 VOCs are organic chemicals that have a high vapor pressure at ordinary room temperature. 5 BAT represents the best available economically achievable performance of plants in the industrial subcategory or category. 6 Dioxins and furans are part of a family of toxic substances that share a similar chemical structure. Most dioxins and furans are not man-made or produced intentionally, but are created when other chemicals or products are made. 7 See table 5 in Gray and Shadbegian (2015). 8 Gray and Shadbegian (2015) also found smaller reductions in VOCs and PM than the EPA’s ex ante projections. However, they did find substantial reductions in chloroform discharges and emissions (consistent with the EPA’s projection), but these reductions began well before the final Cluster Rule was issued. Thus these reductions appear to have been stimulated by other EPA and state regulations. 9 Ground-level ozone is formed when NOx combine with VOCs in the presence of sunlight. Beginning in 2009, the NOx Budget Program was effectively replaced by the ozone season NOx program under the Clean Air Interstate Rule, which required further summertime NOx reductions from the power sector. See https://www.epa.gov/airmarkets/nox-budget-trading-program-historical-reports. 10 CEMS monitors power plants at hourly intervals to track emissions of sulfur dioxide, NOx, and other pollutants. See https://ampd.epa.gov/ampd/. 11 The AIRS database provides information about ambient air pollution in the United States. See https://cfpub.epa.gov/si/si_public_record_Report.cfm?dirEntryID=2779. 12 TACs include benzene, 1,3-butadiene, polycyclic organic hydrocarbons, formaldehyde, and acetaldehyde. All of these are carcinogens. 13 Gasoline volatility is measured by Reid vapor pressure, the absolute vapor pressure exerted by a liquid at 100°F. 14 Reformulated gasoline is gasoline blended to burn more cleanly than conventional gasoline. 15 This result appears to have been (in part) anticipated in the RIA (U.S. Environmental Protection Agency 1993), in which the EPA estimates that the reformulated gasoline regulations will actually increase the emissions of olefins, a particularly potent ozone-forming VOC. 16 Their detailed microdata from repeated expenditure surveys allows them to control for a number of household characteristics that could influence gasoline demand. 17 The Rivers and Schaufele (2015) estimate is derived from the coefficient in column 1 of table 2 in combination with the mean gasoline price in British Columbia reported in table A1. The Lawley and Thivierge (2018) estimate is derived from their “most conservative” semielasticity of −0.01 in combination with the mean gasoline price in British Columbia reported in table 1. 18 This is consistent with results obtained by Li, Linn, and Muehlegger (2014), who find that the response to a gasoline tax is three times larger than the response to a gasoline price change. They attribute this to the perceived persistence and greater salience of the tax. 19 He identifies an appropriate out-of-province control group to compare with the British Columbia plants covered by the tax and estimates the impact of the tax using this matched control group. 20 Regional Clean Air Incentives Market (RECLAIM) -- an emissions trading program operating in California since 1994 -- requires hundreds of polluting facilities to cut their emissions of NOx and sulfur oxides. References Antweiler W. , Gulati S. . 2016 . Frugal cars or frugal drivers? how carbon and fuel taxes influence the choice and use of cars. May 11, 2016. https://ssrn.com/abstract=2778868 or http://dx.doi.org/10.2139/ssrn.2778868. Auffhammer M. , Bento A. , Lowe S. . 2009 . Measuring the effects the Clean Air Act Amendments on ambient PM10 concentrations: the critical importance of a spatially disaggregated analysis . Journal of Environmental Economics and Management 58 1 : 15 – 26 . Google Scholar CrossRef Search ADS Auffhammer M. , Kellogg R. . 2011 . Clearing the air? The effects of gasoline content regulation on air quality . American Economic Review 101 6 : 2687 – 722 . Google Scholar CrossRef Search ADS Azevedo D. , Wolff H. , Yamazaki A. . 2017 . Do carbon taxes kill jobs? Evidence of heterogeneous impacts from British Columbia. December 20, 2017. Brown J. , Hastings J. , Mansur E. T. , Villas-Boas S. B. . 2008 . Reformulating competition? Gasoline content regulation and wholesale gasoline prices . Journal of Environmental Economics and Management 55 1 : 1 – 19 . Google Scholar CrossRef Search ADS Chakravorty U. , Nauges C. , Thomas A. . 2008 . Clean air regulation and heterogeneity in US gasoline prices . Journal of Environmental Economics and Management 55 1 : 106 – 22 . Google Scholar CrossRef Search ADS Chay K. Y. , Greenstone M. . 2005 . Does air quality matter? Evidence from the housing market . Journal of Political Economy 113 2 : 376 – 424 . Google Scholar CrossRef Search ADS Cropper M. , Fraas A. , Morgenstern R. . 2017 . Looking backward to move regulations forward . Science 355 6332 : 1375 – 76 . Google Scholar CrossRef Search ADS PubMed Deschênes O. , Greenstone M. , Shapiro J. S. . 2017 . Defensive investments and the demand for air quality: evidence from the NOx Budget Program . American Economic Review 107 10 : 2958 – 89 . Google Scholar CrossRef Search ADS Elrod A. A. , Malik A. S. . 2017 . The effect of environmental regulation on plant-level product mix: a study of EPA’s Cluster Rule . Journal of Environmental Economics and Management 83 : 164 – 84 . Google Scholar CrossRef Search ADS Fowlie M. , Perloff J. M. . 2013 . Distributing pollution rights in cap-and-trade programs: are outcomes independent of allocation? Review of Economics and Statistics 95 5 : 1640 – 52 . Google Scholar CrossRef Search ADS Fraas A. , Egorenkov A. . 2017 . Retrospective analyses are hard: a cautionary tale from EPA’s air toxics regulations . Journal of Benefit–Cost Analysis , 1 – 20 . doi:10.1017/bca.2017.8. Government of British Columbia . 2008 . Climate Action Plan. http://www.gov.bc.ca/premier/attachments/climate_action_plan.pdf. Gray W. B. , Shadbegian R. J. . 2015 . Multimedia pollution regulation and environmental performance: EPA’s Cluster Rule. Discussion paper DP 15-26, Resources for the Future, Washington, DC. Gray W. B. , Shadbegian R. J. , Wang C. , Meral M. . 2014 . Do EPA regulations affect labor demand? Evidence from the pulp and paper industry . Journal of Environmental Economics and Management 68 1 : 188 – 202 . Google Scholar CrossRef Search ADS Greenstone M. 2002 . The impacts of environmental regulations on industrial activity: evidence from the 1970 and 1977 Clean Air Act Amendments and the Census of Manufactures . Journal of Political Economy 110 6 : 1175 – 219 . Google Scholar CrossRef Search ADS Hughes J. E. , Knittel C. R. , Sperling D. . 2008 . Evidence of a shift in the short-run price elasticity of gasoline demand . Energy Journal 29 1 : 93 – 114 . Google Scholar CrossRef Search ADS Kopits E. , McGartland A. , Morgan C. , Pasurka C. , Shadbegian R. , Simon N. B. , Simpson D. , Wolverton A. . 2014 . Retrospective cost analyses of EPA regulations: a case study approach . Journal of Benefit–Cost Analysis 5 2 : 173 – 93 . Google Scholar CrossRef Search ADS Lawley C. , Thivierge V. . 2018. Refining the evidence: British Columbia’s carbon tax and household gasoline consumption. Energy Journal 39 2 : 147 – 71 . Li S. , Linn J. , Muehlegger E. . 2014 . Gasoline taxes and consumer behavior . American Economic Journal: Economic Policy 6 4 : 302 – 42 . Google Scholar CrossRef Search ADS Murray B. , Rivers N. . 2015 . British Columbia’s revenue-neutral carbon tax: a review of the latest “grand experiment” in environmental policy . Energy Policy 86 : 674 – 83 . Google Scholar CrossRef Search ADS Rivers N. , Schaufele B. . 2015 . Salience of carbon taxes in the gasoline market . Journal of Environmental Economics and Management 74 : 23 – 36 . Google Scholar CrossRef Search ADS Sunstein C. 2011 . Memorandum for the Heads of Executive Departments and Agencies: Facilitating Scientific Research by Streamlining the Paperwork Reduction Act Process. M-11-07. December 9, 2010. https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2011/m11-07.pdf. U.S. Environmental Protection Agency . 1993 . Final regulatory impact analysis for reformulated gasoline . EPA 420-R-93-017. Washington, DC : U.S. Environmental Protection Agency . U.S. Environmental Protection Agency . 1998 . Regulatory impact analysis for the NOx SIP call, FIP, and Section 126 Petitions, vols. 1 and 2. EPA-452/R- 98-003A and EPA-452/R-98-003. Washington, DC: U.S. Environmental Protection Agency. U.S. Environmental Protection Agency . 2011 . Improving our regulations: final plan for periodic retrospective reviews of existing regulations. Washington, DC : U.S. Environmental Protection Agency . Walker W. R. 2013 . The transitional costs of sectoral reallocation: evidence from the Clean Air Act and the workforce . Quarterly Journal of Economics 128 4 : 1787 – 835 . Google Scholar CrossRef Search ADS Yamazaki Akio. 2017a . The effect of climate policy on performance of manufacturing plants. Working paper. April 2017. Yamazaki Akio. . 2017b . Jobs and climate policy: evidence from British Columbia’s revenue-neutral carbon tax . Journal of Environmental Economics and Management 83 : 197 – 216 . Google Scholar CrossRef Search ADS Yip C. M. 2018 . On the labor market consequences of environmental taxes . Journal of Environmental Economics and Management 89 : 136 – 52 . Google Scholar CrossRef Search ADS © The Author(s) 2018. Published by Oxford University Press on behalf of the Association of Environmental and Resource Economists. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Review of Environmental Economics and PolicyOxford University Press

Published: Jul 1, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off