REVIEW OF EPA'S 2014 RFS PROPOSED RULE
Analysis and Critical Review of the Monte Carlo Simulation and Decision Analysis Used in EPA's 2014 RFS Proposed Rule
In proposing the 2014 RFS rule, the EPA, for the first time, utilized Probabilistic modeling in determining the standards. Probabilistic modeling involves assessing ranges and frequency distributions for input data and using random sampling techniques, such as Monte Carlo simulation (MCS) to develop a range of possible outcomes. While MCS could be an appropriate method for modeling uncertainties and evaluating the (RFS) volume estimates, these reports (Clemen report and Decision Strategies report) find mistakes and issues with EPA’s analysis, and discuss how the analysis is highly dependent upon modeling inputs. These independent analyses conducted by experts find shortcomings and make suggestions for EPA to improve their analysis.
Professor Clemen and Decision Strategies independently were able to replicate EPA’s Monte Carlo Simulation (MCS) results using EPA’s assumptions and probability distributions as inputs to the model. They concluded, however, that, while EPA performed the modeling process correctly from a technical point of view, there are significant concerns with the EPA assumptions
Below are excerpts from the Executive Summary of Professor Clemen’s report:
- ... There are two major problems.
- First, is that the probability that a particular facility would produce no fuel was not directly specified. The approach of assigning a lower bound (5th percentile) of zero implies that the probability of producing no fuel is the same (5%) across all facilities with the zero lower bound. This is an important modeling mistake. Especially for new facilities that were not yet producing, the probability of producing no fuel in 2014 should have been separately assessed.
- Second, the probability distributions assigned appear to ignore recent experience with cellulosic producers. In particular, the smooth six-month ramp-up period from startup to a stable volume appears to be inconsistent with information from the two facilities that began producing in 2013, both of which appear to have experienced wide variation in production levels from month to month. Moreover, neither appears to have exceeded 10% of its capacity utilization in its first year.
- The MCS output distribution for total cellulosic fuel produced can be sensitive to the input probability distributions assigned. The proposed rule indicates 5th and 95th percentiles of 8 and 30 million gallons, respectively. However, applying more realistic probability distributions for Abengoa, DuPont, and Poet – probability of producing no fuel set to 20% for Abengoa and 40% for both DuPont and Poet; and if fuel is produced, a distribution with the 95th percentile set at 20% of the plant’s nameplate capacity, prorated over months the plant is expected to be open – results in 5th and 95th percentiles of 4.6 and 15.4 million gallons, respectively.
In order to improve the input probability distributions for new cellulosic facilities, EPA would benefit greatly by engaging the services of professional business analysts and experts that specialize in new-technology start-ups, especially in the renewable fuel industry. In addition, for these experts the EPA may benefit by using a more formal probability elicitation procedure.
- A specific issue in the total renewable fuel model is that the amount of ethanol used in E10 is taken to be a fixed value, based on EIA’s forecast. Incorporating uncertainty into this forecast could have an impact on the output distribution.
- The analysis appears to have been done in a straightforward, “no frills” manner. The proposed rule says nothing about whether or what kind of sensitivity analysis might have been performed. Sensitivity analysis is typically a key part of any analysis and can reveal important insights about the model. In this case, sensitivity analysis identifies Abengoa as the key driver in the cellulosic biofuel model, and biomass-based diesel as the key driver in both total renewable fuels and advanced biofuel. The extent of the potential impact of small changes in the distributions for these variables is demonstrated.
- Given an output distribution from the MCS process, EPA requested comment on what value to choose as the standard (mean, median, mode, or another percentile.) The choice of a particular value to use as a standard should be recognized as a decision, and a “neutral methodology” would require proper cost-benefit analysis for all affected parties. Whether to use the mean, median, mode, or some other value boils down to this: EPA should do the economic analysis that would lead to a specific optimum value that can, in turn, be justified by the analysis. The agency appears to have the ability, and should be provided with adequate budgetary support, to perform such analysis as part of the proposed rule. The selected value would then be more than an arbitrary point chosen from the distribution but would be defensible on economic grounds.”
Below are excerpts from the Executive Summary of the Decision Strategies report:
- “From the information provided by the EPA and in the Federal Register, Decision Strategies was able to reproduce the results of the proposed RFS. The Monte Carlo simulation and basic algorithms used in the EPA model appear to have been correctly applied.”
- “Decision Strategies believes that probabilistic modeling using Monte Carlo simulation is an industry best practice and could be an appropriate method for modeling uncertainties and evaluating the Renewable Fuel Standard (RFS) volume estimates. However, the utility of the model is highly dependent on the assumptions used and the assessment of the ranged data.”
- “The EPA did not use best practice Subject Matter Expert interviewing techniques with the individual companies or with the other sources of data. Specifically with cellulosic biofuel producers, they discussed possible ranges of production and likely start-up dates, but they did not make probabilistic assessments with the experts or try to deal with their biases. Much of the data manipulation was done post-interview based on the expertise and experience within the EPA. The EPA could improve their assessments through the use of de-biasing techniques with the Subject Matter Experts.”
- “While the use of a 90 percent confidence range (P5 – P95) is fine, the EPA did not use best practice when creating ranges. They combined all of the variables associated with a volume into a single range plus distribution. A more appropriate modeling technique would have been to disaggregate uncertainties into two distinct groups for each company or category: one uncertainty would cover the risks related to plant completion and the second would determine the amount of production given a successful plant completion. This would improve the process without adding too much additional burden.”
- “The EPA’s method used a P5 of zero combined with the Half-Normal distribution to account for a plant that did not have commercial production in 2013. This applies the same probability of zero 2014 production to all plants. The probability of not achieving production is understated in the 5% to 8% range.”
- “The assumption to combine a six-month ramp-up (best case) with a plant capacity to develop the P95 is overly optimistic. The plant capacity is typically the volume a plant could produce if operated at nameplate capacity indefinitely. The availability factor influenced by maintenance, unplanned shut downs, etc. reduces this value. Therefore, the EPA set the P95 values at best case for the ramp-up and absolute best case for ultimate volume. Even if both the ramp up and the plant capacity were considered to be P95 values individually, to have both occur simultaneously would create a P99.75 value.”
- “Using the EPA data of the cellulosic biofuel volume shows that Abengoa has the largest impact (47.5%) on the variance of the total proposed volume and the combination of the two largest impacts (Abengoa and KiOR) is 71%. This would indicate that if there are any significant delays during completion of construction (for Abengoa) or ramp up problems with either of these producers the 2014 produced volume of cellulosic biofuels could be significantly impacted. The issues mentioned earlier in this section, combined with the dominating influence of these two plants, likely produces estimates for Cellulosic Biofuels that are overly optimistic.”
- “A sensitivity analysis should be performed to test assumptions and ranges. Decision Strategies has conducted a sensitivity analysis on the cellulosic biofuel assumptions.”