Jenny Riesz1 and Michael Milligan2
1 Centre for Energy and Environmental Markets and School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, Australia
2 National Renewable Energy Laboratory, Transmission and Grid Integration, Denver, CO, USA
Renewable technologies are often characterized as being somewhat different to “conventional” generating technologies in three ways, each with different implications for electricity markets. Firstly, some have highly variable and somewhat uncertain availability, meaning that electricity markets must be designed to elicit adequate flexibility. Second, many have very low short‐run marginal costs (SRMCs) (operating costs), meaning that the mechanisms for managing resource adequacy must be carefully considered. Third, some are nonsynchronous, meaning that grid codes and regulatory requirements must be appropriately designed. Access to flexibility can be enhanced by a range of market design choices, such as short dispatch intervals, short delays from gate closure to dispatch, large balancing areas, high demand‐side participation (DSP), and exposing renewable technologies to market price signals commensurate with other technologies. The design of markets for frequency control ancillary services (FCAS) also provides opportunities to increase access to flexibility, by creating active real‐time markets for a wide range of FCAS, allowing renewable technologies to provide FCAS, and determining FCAS reserve requirements dynamically in real time. Mechanisms for managing resource adequacy are a source of ongoing debate, with many of the key issues having been exacerbated by the entry of renewables. Rapid market change makes investment decisions difficult, regardless of the market model applied. Ultimately, given the existence of arguably successful examples of both energy‐only and capacity market designs, the choice of market model may be less important than the quality of governance with which it is implemented and maintained.
Electricity markets around the world are at the cusp of a dramatic transformation. One of the most visible aspects of this change is the rapidly growing penetration of renewable technologies. Already, renewables (including hydro) contribute more than 26% of global generating capacity and more than 21% of global electricity, and the capacity of renewables installed each year now exceeds that of all other fuel sources combined[1]. Thus, over the next few decades, many electricity market operators can expect to be dealing with much higher amounts of renewable generation. Owing to their relatively lower cost, much of this new capacity will be variable renewables, such as wind and solar photovoltaics (PV).
The technical characteristics of a system undeniably affect its ability to cost‐effectively integrate variable renewables. For example, larger power systems with strong transmission interconnections to neighbors are likely to find it lower cost to integrate higher quantities of variable renewable generation, because variability can be smoothed over a larger geographic area, and managed with a larger selection of plant. High proportions of flexible generation such as hydropower, aero derivative turbines, or reciprocating engines are also likely to alleviate many of the challenges.
However, it is becoming increasingly apparent that the technical parameters of a system are only one part of the story, and the design of electricity industry arrangements can also have an immense influence[2–4]. For many industries, these arrangements include wholesale electricity markets seeking to establish competitive operation of generation and investment decision making. In some cases, poor market design dramatically increases the costs and challenges of integrating renewables[5, 6]. Thus, before investing in expensive physical infrastructure it makes sense to examine the ways in which market design may be facilitating or inhibiting the most efficient system response. In some cases, comparatively simple changes to market design may provide cost‐effective opportunities to increase the ease and efficiency with which emerging renewable technologies can be integrated into power systems.
In considering the operation of electricity markets with a high penetration of variable renewables, it is instructive to first define the characteristics of renewable technologies that make them different. Three key characteristics are identified here[7]:
These properties of renewable technologies are illustrated in Figure 28.1, with a characterization of some of the common renewable generator types. Note that not all renewable technologies exhibit all of these characteristics, and also that the characterization illustrated here may not apply in every case. For example, biomass availability can depend on the supply of waste products that provide the fuel, which may not be easy to store in large quantities, and run‐of‐river hydro may be somewhat variable and uncertain. Thus, the intention of the framework illustrated in Figure 28.1 is not to conclusively “pigeon‐hole” renewable technologies, but, rather, to illustrate their widely varying characteristics, and to emphasize that each technology and system should be considered on a case‐by‐case basis, based on its particular properties.
The categories outlined in Figure 28.1 provide a framework for exploring the aspects of electricity market design that will best facilitate efficient and low‐cost integration of those technologies, as illustrated in Figure 28.2. Markets that incorporate large quantities of variable renewable technologies will need to ensure adequate access to flexibility. Those that include a large share of generation with low SRMCs will need to carefully consider the way in which generation investment signals are managed. Systems with a significant proportion of nonsynchronous generation will need adequate grid codes and regulatory frameworks to ensure appropriate provision of system stability requirements. Each of these aspects is discussed in more detail in the following sections.
Many of the aspects discussed could be considered good market design even in markets that do not include any renewable generation. As discussed below, all markets already include some degree of uncertainty and variability, all market designs face challenges around creating accurate investment incentives, and all systems must ensure appropriate grid codes and regulatory planning frameworks. However, the addition of renewable technologies will often exacerbate these issues, and thus the value of good design choices is enhanced as the proportion of renewables increases[5].
Variability and uncertainty are features of all power systems. Demand is inherently variable and uncertain to predict. Aggregate residential loads vary significantly over the duration of a day, and industrial loads such as electric arc furnaces can exhibit extreme fluctuations of large capacities[5, 8]. Similarly, all generators, including fossil fuel and nuclear generators, have the potential for unplanned forced outages at any time. In the case of large conventional generators this can very suddenly remove hundreds to thousands of megawatts of capacity from the grid without notice.
Thus, over many decades, power system engineers have developed sophisticated mechanisms for managing considerable amounts of variability and uncertainty. At low levels of penetration, variable renewable technologies can often be absorbed into the system without adding to its flexibility requirements[9]. However, beyond a certain point the aggregate degree of variability and uncertainty in the system is likely to increase as variable renewables are added, and these mechanisms may need to be expanded and enhanced (Box 28.1).
Power systems must always maintain balance between supply and demand for electricity within small prescribed tolerances, to maintain system frequency within the narrow bands required for safe and stable operation. This must be managed over time frames ranging from less than a second to years. As more variable renewable generation is added to the system, the degree of flexibility required from the “balance of system” (the other generators and loads) will often increase.
Flexibility in this context is used to refer to aspects such as:
Most power systems have sufficient technical flexibility to integrate high proportions of variable renewables; analysis by the International Energy Agency suggests that existing power systems may be able to integrate 20–63% variable renewables based on a purely hardware point of view[2, 11]. With innovation, the flexibility of existing systems may exceed initial expectations. For example, a recent case study demonstrated that coal plants can become flexible resources; a coal‐fired power station originally designed to operate as a base‐load plant in North America now operates successfully in a highly flexible manner, often undertaking complete shutdown and startup cycles multiple times per day. This was achieved with limited hardware modifications, but extensive modifications to operational practice[11].
Despite the high degree of technical flexibility available in many power systems, poor market design may limit access to that flexibility in a range of ways[3, 5, 6]. Market design to encourage efficient levels of flexibility needs to be explored in two dimensions[5]:
The latter may, but does not always, follow from the former, depending on the interaction of energy spot markets with mechanisms designed to manage investment (such as capacity mechanisms).
Balancing of supply and demand is managed over operational time frames firstly through general market design, usually with a real‐time wholesale energy market, in which generators make offers to sell electricity, and retailers (also known as suppliers or load serving entities) make offers to buy electricity. The market settles at the point where demand and supply are balanced, with the price of the marginal generator (the last generator dispatched) typically setting the price for all market participants in that trading interval. The design and operation of this wholesale market can have significant implications for the degree of flexibility that generators and loads willingly offer to the system.
The speed of the market is one of the most important characteristics. Very fast markets that feature short (five‐minute) dispatch intervals can integrate much higher quantities of variable renewables at lower cost[5, 6]. Shorter dispatch intervals allow more frequent re‐dispatch of the whole system, providing near constant access to the full degree of flexibility available from all plants and loads. By contrast, long dispatch intervals artificially “lock in” generators and loads to preset levels for extended periods of time, preventing adjustments that may be possible at very low cost[5]. These markets must maintain larger quantities of reserves (plant set aside for managing variability and uncertainty within dispatch intervals), thus increasing system costs[5, 12].
A short delay from gate closure (the time when the last offers and bids are made) to dispatch is also important. A short delay allows incorporation of the most up‐to‐date wind and solar PV forecasts, and therefore more accurate scheduling[10, 13, 14]. This means that smaller quantities of reserves can be maintained, reducing costs[15, 16]. Needless to say, the quality of wind and solar forecasts is also important, in addition to integration of forecasts with the full market dispatch process.
Managing dispatch over larger areas is also of significant benefit. In many markets, significant physical transmission interconnection exists, but supply and demand are balanced over an artificially much smaller area. Aggregating markets (or balancing areas) allows greater per unit smoothing of generation and load variability, access to a larger number of generators and loads for providing flexibility[17], and sharing of reserves, which reduces costs[5, 10]. In many cases the barriers to aggregation (and thus cost reduction) are political and regulatory in nature, rather than technical[18].
Many markets have only limited DSP. Because loads are usually not exposed to real‐time market prices, they usually have no incentive to respond to real‐time signals limiting the degree of flexibility they make available to the system. The potential may be very large, and possible in many cases at low cost. For example, the value of DSP potential in the European Union has been estimated at 53 billion euros[19]. Access to new technology such as advanced metering infrastructure may assist in alleviating some of the many and varied barriers to DSP. However, a shift to full DSP represents a fundamental change in the way electricity markets operate, and a complete paradigm shift in the way that customers interact with their electricity supply. Many of the barriers are likely to be social and regulatory, rather than technical or economic, and thus a concerted effort is likely to be required to unlock this potential.
Ideally, markets are designed to be “technology neutral,” such that renewable technologies fully participate in the wholesale market, and are managed on a level playing field with other technologies. Some markets have applied “priority dispatch” for renewable technologies, such that they are dispatched ahead of other technologies regardless of system conditions at the time. At low levels of penetration, this can effectively assist the entry of renewables. However, at higher penetration levels, the shielding from market price signals becomes problematic, because it inhibits an economically efficient market response. Many renewable generators (such as wind and PV) are extremely flexible, being able to ramp very rapidly from full capacity to complete shutdown with very little notice, but they are unlikely to do so unless they are appropriately incentivized by exposure to the same wholesale price signals as other generators. Because many renewable technologies have very low SRMCs, full participation in the wholesale market without priority dispatch is likely to still result in their full dispatch in most normal circumstances.
“Make Whole” payments, applied in some markets, add complexity to the issue. These payments ensure that generation resources recoup costs by providing out of market payments if necessary. Because the resource is providing a service on behalf of the system and should be compensated to cover costs, these payments are “out of market” and thus may not ensure economic optimality. In addition, they shield market participants (in this case, typically large incumbent thermal units) from important pricing signals, potentially not providing sufficient incentives to provide long‐term flexibility. Some markets have introduced new categories that allow variable renewables to participate more fully in the wholesale market, while recognizing their unique characteristics. For example, the Midcontinent Independent System Operator (MISO) introduced the “Dispatchable Intermittent Resources” category[20], and the Australian National Electricity Market (NEM) introduced the “Semi‐Scheduled” category[21]. The New York ISO has also extended its market‐based Security Constrained Economic Dispatch process to optimize the scheduling of wind plants, based on their offers, as it does with other generating resources[22].
Some system designers have decided that the price signals provided by the market are insufficient to elicit an adequate flexibility response, and have added further incentives. For example, the MISO and California Independent System Operator (CAISO) have both introduced supplementary mechanisms to reward generators for providing ramping services over time frames longer than a single dispatch interval.
FCAS are provided by generating capacity (or demand) that is reserved for responding to variations in the system balance on time periods shorter than a dispatch interval, or that were not forecast at the time of scheduling. In markets with long dispatch intervals and long delays from gate closure to dispatch, much larger quantities of reserves must be maintained. Thus, the design of the wholesale market can significantly affect the cost of providing FCAS.
Some systems have real‐time markets for a range of FCAS that operate alongside the wholesale energy market. Through these FCAS markets, suitably capable generators and loads can submit offers to provide various ancillary services in each dispatch interval, along with their energy offers and bids. This allows generators and loads to flexibly provide the maximum amount of FCAS they have available in any interval, without having to make long‐term commitments.
Dividing FCAS into distinct categories allows a wider range of generators and loads to provide only the particular services that they find cost‐effective. Common categories of FCAS are illustrated in Figure 28.3, and include:
Good market design principles suggest an approach that is tailored to the specific system, selecting categories of FCAS that allow the maximum number of generators and loads to provide the components that they find economically and technically efficient, without introducing excessive and unnecessary complexity. This will change as the mix of generators and loads evolves over time, making regular review of the FCAS market design worthwhile.
Wind and PV generators would face a significant opportunity cost from providing a contingency raise service, because this would necessitate their curtailment over extended periods of time, and therefore mean lost revenue in the energy market. However, they may find it cost‐effective to provide contingency lower services (being ready to rapidly reduce generation in the rare event of a load suddenly experiencing a forced outage). These generators are technically capable of providing this service[23] with minimal opportunity cost, meaning that in future markets this service might be provided essentially for free in any period where wind and PV are operating at sufficient levels. Thus, if wind and PV are to participate cost‐effectively in the FCAS market, these services must be divided into different categories[24, 25].
Similarly, many types of loads may find it cost‐effective to provide contingency raise services. Contingency events are relatively rare (called upon perhaps several times per year) and typically short (less than an hour), such that providing a contingency raise service would simply require demand response units to be available to rapidly reduce consumption in the rare event of a generator suddenly experiencing a forced outage. However, demand response may not be equally able to provide a contingency lower service, because this would require being able to rapidly increase consumption at short notice. Electric vehicle (EV) charging, for example, may be well suited to providing contingency raise services. Owing to the rarity of reserves being called on, the impact on EV battery cycling could be negligible, and customers minimally affected.
Of course, to achieve these benefits it is a prerequisite that renewable generators and loads are freely allowed to participate in providing all types of FCAS, as long as they can demonstrate that they are technically capable of doing so (they are in many cases [23]). This is not yet the case in many markets.
Another important aspect of FCAS market design is the manner in which the quantity required of each type of reserve is determined. Many systems apply a “static” approach, based on examination of typical system variability at different times of day, or by season, peak versus off‐peak, and so on[26]. However, the operation of variable renewable generation at any particular point in time has a significant influence on the degree of system variability, allowing a more dynamic approach that takes into account the system state in a particular dispatch interval[27–30]. Some systems set the reserve requirement in a completely dynamic fashion based upon system properties such as the time error (closely related to the system frequency)[31, 32]. This allows the reserve requirement to evolve dynamically in real time as the system changes, such that the scheduled reserve in each interval is the minimum required (reducing costs).
The interaction of markets for ancillary services and energy requires careful attention, particularly as the number and sophistication of ancillary services markets increases over time. In some cases, poor consideration of these interactions has led to detrimental perverse incentives. For example, in some eastern US systems, the energy market rules financially penalize generators that deviate from their energy schedules, aiming to provide incentives for generators to operate at their instructed levels. However, many generators are designed to automatically assist in maintaining system frequency during contingency events. If a generator's governor is tuned appropriately, a drop in system frequency will cause it to increase output, assisting in arresting system frequency decline. Despite this being behavior that is helpful to the system, the energy market rules in this example would penalize this generator for deviating from its energy schedule. This has likely contributed to many generators in these markets detuning their governor responses, and exacerbating frequency response issues[33]. These perverse incentives can be removed by only penalizing deviations from the energy schedule when those deviations are exacerbating frequency problems, such as is implemented in the sophisticated “causer pays” methodology in the Australian NEM[31].
In theory, short‐term price signals for generators to provide flexibility to a system also translate into long‐term investment signals. With sufficient foresight, developers of new generation can anticipate when the system is likely to experience a shortfall in flexibility, and therefore where there are opportunities to install new generation that can profitably provide that service. However, this may not eventuate in practice. First, during a period of rapid market transformation developers may struggle to anticipate future market needs, and may underestimate the value of increased flexibility. Second, many markets include explicit capacity remuneration mechanisms (CRMs) that supplement generator revenue. Where capacity payments constitute a large proportion of generator revenue, the design of these CRMs can have significant implications for generator incentives around new technology decisions. It is also possible that some FCAS markets are thin – i.e. even in the face of high price spikes, little additional FCAS are needed to ensure a sufficient supply and mitigate price spikes. If this is the case, a new entry (or entrants) into the market could result in a price collapse that could result in revenue insufficiency for the entrant.
In a properly operating energy‐only market, generators are exposed to extremely high prices at times of scarcity, and very low (negative) prices during times of oversupply. In some circumstances, these prices can occur suddenly, creating incentives for generators to rapidly adjust their output. Thus, in this market there is an incentive to install more flexible plant that can respond to these price signals and reap the benefits[5]. By contrast, in markets where a CRM is applied, the market price cap is typically much lower. This reduces the benefit of flexible operation.
Some markets provide added incentives for flexibility by distinguishing between different kinds of capacity in the capacity payment process. For example, in Spain only generators of certain types are eligible for capacity payments[34]. In the German market a “focused” capacity market has been proposed, involving multiple tiers of capacity credits that result in differentiated capacity payments depending upon the flexibility of the plant[35].
For most renewable technologies the majority of the expense is in the capital cost of constructing the generator, potentially with some fixed maintenance costs that do not vary significantly with the amount of energy generated. Given their very low SRMC, in a competitive market it is rational for renewable generators to offer their energy at a price close to zero. Generation at any level above their SRMC increases operational profitability (although profitability in the long‐run requires sufficient revenue to recuperate fixed costs).
In markets where renewable generators supply large quantities of energy the so‐called merit order effect has led to falling electricity prices, reducing the profitability of incumbent generators[36–39]. The entry of low SRMC renewables pushes other generators up the dispatch merit order, causing lower prices and reduced dispatch for incumbents.
This effect applies not only to renewable generators. Many conventional generating technologies also have low operating costs. Nuclear power is similarly capital intensive, and some Australian mine‐mouth coal‐fired power stations can have an SRMC at a similar level to the estimated variable operations and maintenance costs of wind generators[40]. In present power systems, these low SRMC technologies are typically combined with a proportion of much higher SRMC technologies, such as gas‐fired generation. When these intermediate and peaking generators operate, all generators are paid the higher marginal price (in a marginal pricing market) and thus recoup fixed and capital costs.
Not all renewable technologies have a very low SRMC. For example, some biomass technologies are likely to have a cost associated with collection, processing, and storage of biomass fuel. Hydro generation is also complex to value in short‐run terms, because it is energy limited and therefore often carries a substantial “opportunity cost,” which will strongly influence the way that hydro capacity is offered to the market. Thus, power systems composed of a mixture of technologies involving a proportion of these higher SRMC renewables may find that they exhibit pricing profiles not dissimilar to present power systems, even at very high renewable penetrations.
However, as illustrated in Figure 28.1, many renewable technologies do exhibit a very low SRMC, and a future power system could conceivably be composed entirely of these technologies. In this situation, it is logical to foresee market prices falling to close to zero for the majority of the time. The questions then arise: How do generators make money in such a market? Why would anyone invest? These questions relate to what are termed resource adequacy or system adequacy mechanisms.
The ideal mechanisms for managing resource adequacy have been debated for decades, without resolution. Creating accurate long‐term investment incentives via an electricity market is challenging, even in the absence of variable renewable generation. Some proponents favor the economically “pure” theory of energy‐only markets, where market participants receive revenue only through the wholesale energy and ancillary services markets[41, 42]. Others remain convinced that energy‐only markets are fundamentally flawed for a variety of subtle reasons, and support the introduction of explicit CRMs[43, 44]. These introduce a more certain revenue stream for investors in firm capacity by providing a payment for capacity availability (MW) in addition to energy supplied to the market (MWh). This can create stronger incentives to invest in new capacity, and may allow investors to acquire capital at a lower cost by reducing risk and uncertainty. They can also support higher generation adequacy and security of supply for the whole system, and allow lower price caps to be introduced in the wholesale market, reducing price volatility and therefore reducing risks for customers and other market participants[24].
However, given that CRMs represent a regulatory intervention, they also have some drawbacks[24]. The choice on the amount of reserve margin maintained in the system must be made via a regulatory decision, which can incline the system toward a conservative oversupply of capacity. Some have argued that this may be of benefit, because it may be better to err on the side of oversupply and limit the risk of extended periods of high prices for customers[45]. Design and implementation of CRMs can also be very complex[24], and many have required substantial adjustments over time as the market evolves. Care must also be taken regarding coordination with neighboring markets; implementing different market models without care for cross‐border effects can cause distortions and hinder functioning[24].
There are a wide range of different types of CRMs, and some of the most innovative, such as reliability options, may be remarkably close to properly functioning energy‐only markets in practice. Ultimately, it is likely that most types of market designs can work well if they are implemented well; perceived failures are often related to flawed implementation or excessive government intervention[46].
Renewables do not undermine the theoretical operation of energy‐only markets and do not fundamentally change the debate, but they are bringing many of the long‐discussed issues to prominence[7]. With lower wholesale prices occurring more often, an increasing proportion of generator revenues in energy‐only markets will need to be concentrated in rare scarcity periods, exacerbating uncertainty and risks for market participants[7]. In theory, market participants can manage these risks with increased levels of forward contracting, but there may be reasons why sufficient contracting does not occur.
In many ways, the most challenging aspect of resource adequacy is likely to be related to the rate of change that markets need to manage. All models are likely to struggle during periods of high uncertainty, and with rapidly changing political and regulatory drivers. These characteristics fundamentally make investment decisions very challenging, especially when planning long‐lived and lumpy infrastructure [24].
Proponents of energy‐only markets suggest that they could operate successfully, if all the barriers preventing proper market operation were removed, including low price caps, regulated retail prices, and a lack of DSP[24]. However, the proportion of renewable technologies is rapidly increasing in many markets, and addressing all these issues may take too long, given political orientations and the lengthy processes likely to be involved in increasing DSP. A CRM may assist in maintaining investment incentives in the interim, implemented as a temporary measure with eventual return to an energy‐only market model being the ultimate goal[24]. However, once introduced, CRM policies may be hard to reverse[47].
Some renewable technologies, such as wind and PV, are nonsynchronous. This means that they are technically quite different to most conventional technologies (such as coal, gas, hydro, and nuclear) which are synchronously connected to the grid. A synchronous connection contributes to frequency stability, and also assists with various technical aspects of grid operation. Increasing displacement of synchronous generation may eventually lead to the requirement for additional market dispatch constraints that require a minimum proportion of synchronous generation to remain in operation at all times (leading to spilling of nonsynchronous renewable generation). For example, in the Australian NEM it has been suggested that it may be necessary to maintain a minimum of 15% synchronous generation at all times[48], although far more detailed modeling and analysis are required. Modern wind turbines can provide synthetic inertia, which is implemented in the control software. This is an emerging issue that is being examined so that the performance of synthetic inertia can be better understood and metrics can be developed to help determine the need for inertia.
This characteristic of renewable technologies is highly technical in nature, and is more related to grid codes, network regulation, and the engineering requirements placed on new‐entrant generators. It is generally a step removed from the operation of the wholesale electricity market, although it will interact with the provision of FCAS. One of the most important messages from current research is that grid codes must be forward looking to anticipate the requirements of the future grid, but not overly onerous or discriminatory against new entrant technologies[49]. It can be challenging to strike this balance correctly.
One of the most important findings from recent research on the integration of renewable technologies into electricity markets is that market design really matters. The costs and challenges of integrating renewables can vary significantly, depending on apparently minor differences in market design. This makes cross‐comparison of integration studies and experiences challenging, and highlights the importance of context specific analysis for the particular system under consideration. It also points to an immensely important opportunity: by adjusting market design alone there is the potential to significantly reduce the costs of integrating renewable technologies into power systems.
There is no denying that system operators and market designers have a plethora of interesting challenges ahead. Questions about the degree of system flexibility required and how best to encourage it are only beginning to emerge. Similarly, it is unlikely that debates about the best way to manage resource adequacy will be resolved any time soon. Regardless of the frameworks implemented in each system, the best outcomes are likely to be achieved by rigorous analysis, facilitated by ongoing international collaboration.
3.141.47.221