To meet the hourly, daily, and seasonal variations in demand, midrange units were dispatched. Peaking units were sparingly dispatched only for limited duration to cover peak demand, such as on a hot summer day when air condition load would be high in hot regions of the world or on cold winter days when electric heating load would be high in cold regions of the world. The concept of traditional merit order is schematically illustrated in Fig. 17.1.

Historically, many networks had little renewable generation, with the exception of hydro units, which were used depending on the availability of water and the capacity of reservoirs to complement the output of thermal plants. Pumped hydro storage, when available, was also utilized to manage fluctuations in demand and to store excess generation.

Fast forward to 2017 in places like Germany, Denmark, California, Texas, or South Australia, where renewables now comprise more than half of total generation on many hours and many days, occasionally exceeding total demand on the network. In Germany, for example, the proportion of renewables has been on the rise and now dominates the energy mix on many days—a fact that will become even more pronounced as the existing fleet of nuclear plants with almost 11 GW of installed capacity will be phased out by 2022 and lignite/coal capacities of more than 7 GW will leave the market in response to recently introduced carbon reduction policies (Fig. 17.2). These numbers are significant, considering Germany’s peak load is about 80 GW.

In such renewables-dominated energy markets, the traditional notion of merit order dispatch no longer applies. Renewables such as solar and wind tend to be variable and they must be utilized when they are available—otherwise they would have to be curtailed, which means wasting low-cost, carbon-free electricity. Run-of-the-river hydro falls in the same category. If the grid operator does not use such resources when they are available, they are wasted.

Another big difference with the traditional dispatch paradigm, of course, is the fact that many renewable resources—with the exception of geothermal, hydro with massive reservoirs, biomass, and biogas with flexibility—are not dispatchable. They are available when they are available, and when they are not, the grid operator cannot order them to produce energy. This suggests that as more countries gradually move toward a future where the bulk of generation will be renewable, they must come up with alternative schemes to balance the load and generation and to keep the grid stable and reliable.

One fundamental way to think about this challenge is to think of a radically different operating and dispatching paradigm. Instead of taking load as a given and adjusting generation to chase it at all hours, why not devise a paradigm where both demand and supply are flexible and can be adjusted to dance together, rather than forcing one to follow the other.

As it turns out, this is precisely the approach that has been developed by a number of innovative enterprises who increasingly see the future of electricity markets and networks as a choreography of flexible load and variable generation—and in some cases distributed or centralized storage¹—that work in unison to keep the grid secure and reliable while maximizing the use of variable generation resources.

This approach, sometimes referred to as price-responsive demand, demand response (DR), or demand-side management (DSM) is increasingly applied in Virtual Power Plants (VPPs) to intelligently shift demand based on wholesale power prices. Industries have long since been managing their electricity demand to reduce network charges. Now the upcoming VPPs show promise as a superior alternative to balance variable demand with variable generation, the emerging reality of the future of most electricity markets.

This chapter describes the fundamentals of this promising approach by focusing on the successful business model of Next Kraftwerke, a German digital utility and VPP operator that has grown substantially in the past few years, applying these principles in the context of Europe’s challenged electricity market. By late-2016 they have aggregated more than 4000 electricity producing and consuming participants with a combined capacity of over 2700 MW—the equivalent of two large coal-fired power stations. The chapter argues that the future of electric systems will increasingly rest on aggregating flexible load and variable and flexible renewable generation—rather than dispatching thermal plants to meet an inflexible demand.

The chapter is organized as follows: Section 2 describes the rising significance of flexibility in the context of variable renewable generation. Section 3 introduces the role of aggregators and the concept of VPPs and describes how they work and what services they offer. Section 4 gives an outlook on the future evolution of VPPs followed by the chapter’s conclusions.

A related problem of renewable—especially wind—generation is the increasing frequency and costs of thermal redispatch and renewable curtailment. These measures are more and more often taken by the grid operators when on- and off-shore wind turbines in the windy North of Germany produce so much electricity that the grid transporting it to the industrial South gets congested. The problem is intensified by thermal power plants in the North and East that cannot be switched off, even when prices are negative. In these instances, the grid operators pay thermal or renewable plants in congested areas extra money to switch off; while at the same time, they pay thermal units in the South extra to ramp up.

This phenomenon has become common over the last few years and is expected to get worse in the future. In 2014, for example, the German grid operators—there are four of them—had to take action on 330 days to redispatch as described above resulting in €187 million cost compared to 232 days in 2013 and a cost of €132 million.²

The compensation costs for curtailing or switching off renewable plants rose from €43.7 million in 2013 to €82.7 million in 2014. While these costs are still quite small compared to the €24 billion that were paid out to renewable generators in 2015 in total,³ the relative cost increases in some years have been significant as illustrated in Fig. 17.4.

Germany, of course, is not alone in facing such a challenge. California and New York, for example, have 50% renewable targets by 2030 while Hawaii is striving to reach a 100% renewable goal by 2045. In South Australia, Denmark, and Texas, the transmission network is routinely overwhelmed during windy periods. A number of other countries including Norway and Iceland are virtually 100% renewable—mostly with hydro though which does not necessarily put stress on the network since water can be stored in reservoirs for generation as needed.

It is important to note that a high share of variable renewables does not necessarily mean less grid reliability—certainly not in Germany or Denmark, both countries with large and growing renewable supplies. In 2013 the lights went out for 12 min on average in Denmark and for 15 min in Germany; the corresponding figure for 2014 was a little over 12 min.⁴ In nuclear-dominated France, by contrast, it was 68 min in 2013. In America and many other countries grid service interruptions exceed one or more hours per year, on average.

Moving forward, the most noticeable expected impact of so much variable renewable generation on the network is twofold:

The first expected impact can easily be observed in the German wholesale market as illustrated in Fig. 17.5 where rising renewable generation—accompanied by excess conventional capacities with minimum loads—has depressed wholesale prices in recent years with grave consequences for the four major German power utilities.

The rise of renewables not only impacts overall prices in wholesale markets but changes the merit order of plants, which means that zero marginal cost renewables—principally solar and wind—displace more expensive base load plants including nuclear, lignite/coal, and gas, by pushing them to the right and eventually out of the picture entirely as illustrated in Fig. 17.6. This means that, all else being equal, thermal plants are dispatched less frequently, and for fewer hours. Many of them cannot be ramped down to zero so they run on minimum loads more frequently, further depressing prices. And when they do get dispatched, they receive lower prices on average. This explains why thermal generators in Germany, for example, have had such a difficult time in recent years.

The trend is likely to get worse. Recent studies estimate that up to 230 GW of fossil capacity in Europe will remain unprofitable through 2020.⁵ Due to the massive overcapacities of thermal generators, market prices cannot adequately reflect scarcity situations. Therefore, the value of flexible peak load assets is currently very low which reduces the incentive to invest in decentral, green flexibility. Still, a lot of nuclear and coal/lignite capacities are kept in the market through direct or indirect subsidies—a paradoxical outcome at odds with the EU’s goal to decarbonize the energy sector by 2050.⁶ If the EU goal is to be reached, investments in sufficient green flexibility options are needed soon to balance out the variable generation of wind and solar for the decades to come.

The second expected impact—wholesale prices becoming more volatile—is more difficult to detect in the market. It is intuitive to assume that with an increasing share of variable renewables the volatility of wholesale prices should increase, too. However in the case of Germany such a trend cannot be observed in wholesale price volatility—at least not if measured by the standard deviation of average prices on the day-ahead market where the bulk of wind and solar power is traded (Fig. 17.7). Rather, the overall trend suggests that the volatility of day-ahead prices has gone down since 2012. One of the reasons for this development is that the liquidity of the day-ahead market has increased significantly since the introduction of a market model for renewables—the so-called market premium model—in Germany in 2012. Since then, renewable electricity is regularly traded on day-ahead and intraday markets where forecasting errors can be corrected as forecasts improve shortly before delivery.

Consequently, in Germany at least, it is the case that electricity spot markets can very efficiently deal with the fluctuating nature of wind and solar so that variable renewables do not necessarily cause prices to become more volatile.

Nonetheless, renewables have immensely altered the price structures of wholesale markets. This is being felt, for example, in the collapse of midday prices when solar generation is most pronounced which has virtually eliminated the peak/off-peak paradigm of the old electricity world. As shown in Fig. 17.8, average day-ahead prices over one day have converged in the past five years—now peaks usually occur in the morning and evening hours with midday prices being closer to night prices. Another case in point is the well-known California duck curve, which refers to the daily dip in midday net load on the network caused by the rising solar generation, further described in Chapter 9 by Baak.

Variations of the California duck curve are now routinely experienced in other parts of the world including sunny Australia or windy Texas and Denmark where negative prices during periods of excess solar/wind generation are frequent.

Combined, the net effect of these two phenomena—wholesale prices plummeting and the peak/off-peak differential eroding—is to:

The next section will primarily look into the second effect by offering a solution, namely how to balance variable renewable generation and variable demand on networks with increasing amounts of variable renewable generation, such as in Germany.

A major problem caused by big swings in variable solar or wind generation is that thermal plants have to turn down to their minimum operating levels for most of the day, only to resume generation at full capacity in the late afternoon hours to meet the evening’s peak demand. In the case of California and its Duck Curve, this up and down ramping is approaching 13–14 GW and has to be accomplished during a 3-h window. This not only puts a lot of wear and tear on the thermal plants but is hugely inefficient, expensive, and polluting.

Along with this vexing challenge faced by California and Germany alike comes the so-called minimum load problem, which refers to the fact that on many cool sunny days or windy nights, the grid operator simply runs out of thermal plant capacity that can be turned down during midday or night hours.⁷ Many thermal units cannot be turned down for a variety of reasons, beyond a reasonable limit, so they have to run on minimum load during the day when they are not needed. This issue is likely to become more serious over time, certainly in California, Germany, and possibly elsewhere.

Which is why experts are looking beyond the grid operators’ traditional tool box for practical, cost-effective, and non-polluting alternatives to balance variable generation and load.

One obvious solution, of course, is to:

Next Kraftwerke, hereafter referred to as NK for brevity, a German digital utility, is among innovative companies who are trying to do all of the above, and do it in a way that is win-win-win for the grid operator, for consumers with flexible demand, and for generators who can adjust their output.

The company’s basic business model⁸ is to create a portfolio of flexible generation and loads by aggregating a large number of participants who are willing and able to respond to price signals in their VPP.

As illustrated in Fig. 17.9, NK have developed a profitable business by bringing power producers, consumers, wholesale markets, and the grid operators together using a remote control unit called the “Next Box” that allows the parties to transact with one another in real time. By late-2016 they had networked more than 4000 electricity-generating and -consuming participants with a combined capacity of over 2700 MW.

The Next Box connects the parties, enabling supply and demand to respond to price signals through highly automated machine-to-machine (M2M) communications as schematically shown in Fig. 17.10.

This is particularly relevant for offering ancillary services or control reserves. Control reserves are tendered by transmission system operators (TSOs) to stabilize the grid between 49.8 and 50.2 Hz in Europe. In general, power generators and consumers can bid positive and negative capacity to the tender. When selected, this capacity must not be marketed elsewhere so the TSO can call it when the grid is in imbalance.

In times when power markets were dominated by a small number of large thermal power plants, this balancing function used to be a rather trivial task. With millions of decentral power plants in the system today, a much larger set of variables is involved, requiring a complex optimization problem.

To solve this problem, the Next Box⁹ collects operating data of each decentral power producer and consumer in the VPP, for example data on the availability, current capacity level, and flexible capacity. These data are then encrypted and sent to the NK control system where it is decoded and processed.

The control system¹⁰ validates the data sent in by all Next Boxes so it always knows the number of megawatts available in the pool. Along with weather data and price signals, the validated data from the Next Boxes are fed into an optimization scheme that determines the optimal output value for each unit. The control system then sends a signal with those numbers back to the Next Boxes and the decentral units adapt their output accordingly.

The value of NK’s assembled portfolio of flexible resources comes from the diversity of its client base, which includes generators with flexible output, such as biomass, biogas, and hydro, customers with flexible onsite microgeneration, as well as large commercial and industrial customers with flexible loads (DSM customers) as illustrated in Fig. 17.11. They also aggregate volatile solar and wind generators, forecast and market their electricity on power exchanges, and curtail their output if market conditions mandate it.

Over time, NK have expanded their service offerings to include power trading,¹¹ balancing services for utilities or volatile power plants, and (flexible) power supply to industrial and commercial power consumers. The basic idea behind NK’s business model is simple: produce electricity when it is scarce and consume electricity when it is plentiful. Due to the increasing shares of solar and wind power in European electricity systems, scarcity and excess situations are more and more determined by the output of solar and wind plants, which makes weather forecasts a very relevant component of predicting price movements.

Based on its own weather forecasts, NK suggest ideal hourly schedules for the producers and consumers in their VPP one day before delivery. They then trade the corresponding electricity volumes for every hour on the day-ahead markets of European power exchanges. Deviations from feed-in forecasts due to forecasting errors or unexpected downtimes as well as within-the-hour gradients of wind and solar power are traded on continuous intraday markets for every quarter hour of the day. In turn, the price signals on day-ahead and intraday markets provide incentives for flexible power producers and consumers to shift their generation and consumption to those quarter hours with the best prices—for producers these are periods of power scarcity with high prices and for consumers these are periods of excess supply with low prices.

How NK manage the nuances of the interactions among the various participants in the scheme is complex. The fundamentals, however, are straightforward. When demand and prices are high:

The incentives are reversed when prices and demand are low, namely:

This is accomplished automatically through M2M communication as clients respond to price signals provided by NK—who are constantly monitoring supply and demand conditions on the wholesale market and forecasting prices and a number of other critical variables in real time.

Many consumers have ample flexibility in when they use electricity, particularly customers with inert processes, such as pumping, heating, melting, crushing, processing, etc. These customers can adjust their heavy consumption periods and shift them to times of low cost electricity without compromising their operations. These are the types of customers that NK specifically recruit on the consumption side. And these customers learn how to better respond to price signals over time—it is a classic case of learning by doing. Participants get better at it over time, saving more and are less inconvenienced.

For NK, DSM means to shift demand from high-price to lower-price times of the day in a continuous optimization process—rather than only shaving off demand peaks during high-price periods as is the traditional approach to DSM. Consequently customers rarely if ever reduce the total amount of electricity consumed, while actively and dynamically optimizing when they use electricity as illustrated in Fig. 17.12.

One of NK’s customers, for example, is an association for coastal management with large pumping loads to maintain water levels in dikes and embankments in Germany’s low-lying coastal areas. So long as these customers maintain water levels by pumping rain back into the sea, the pumps need not run constantly. This offers valuable flexibility.

With its optimization scheme, NK determine those quarters of every day in which it is cheapest to pump given a predefined set of restrictions—for example, water levels must be maintained within certain limits to avoid flooding. Once these customers agree to change their pumping schedules to take advantage of lower prices, their pumping load is adjusted and their electricity bill is reduced without any adverse impact on the management of water levels. This particular product is called “Best of 96” because a day consists of 96 quarter hours and NK pick the best, that is the cheapest, for these consumers.

Fig. 17.13 illustrates how the scheme works where customers take advantage of low prices to schedule their heavy loads. In this example, the pumps used to be operated between 5:30 and 7:30 a.m. NK’s optimization analysis, however, suggested that pumping should be done at 5:00 a.m., 12:30 p.m., 1:30 p.m., and 4:00 p.m. for half an hour each. By following this procedure the customer reduced his energy costs by 30% in 2015.

The same principle can be applied to most pumping loads virtually anywhere. These customers typically use large industrial pumps consuming large amounts of electricity. Any reduction in their power usage ends up as savings in the bottom line with no impact on their operations. While this is an easy-to-understand scheme, the same applies to many industrial and commercial loads where the exact time of when electricity is consumed is not critical.

For those electricity consumers who do not have so much flexibility that they can shift demand in quarter-hour periods, NK offer set price zones for a whole year in advance—a product called “Take your Time.” Fig. 17.14 shows how a customer’s required flexibility increases with the different options of Take your Time, peaking with Best of 96.

The same principle applies to clients with flexible generation, such as those with biomass or hydro plants. One customer on the supply side is a biogas plant whose operator optimized its design with respect to flexibility, a practice that is incentivized by recent revisions of the Renewable Energy Sources Act in Germany. For doing this, the plant owner receives a flexibility premium but in return only gets a feed-in premium for half a year’s production at full throttle. As a consequence, on each day, the plant operator wants to choose those 12 hours, or 48 quarter hours, with the highest prices to switch on the generator. This is what NK do for this particular customer, as shown in Fig. 17.15.

Of course not every plant is able to do that, just as not every consumer is able to change her electricity-consuming processes on a quarter-hourly basis. Some biogas plants have contracts for heat supply that are linked to a minimum electricity production or their biogas storage tanks are relatively small. All of the processes involved in the plant’s operation need to be analyzed to determine the range of flexibility of that particular power plant.

When bottlenecks or constraints are identified, they can frequently be overcome by investing in larger tanks or seamless operation of generators. For a plant operator, these investments need to pay off rather sooner than later. With a more volatile market, the payback periods generally become more difficult to estimate. But with the 18 GW of conventional nuclear and coal/lignite capacity leaving the market by 2025 in Germany alone, it is rather clear that price spikes will occur more often in the years to come. This is because even with more installations of wind and solar, there will be times in the year when there is hardly any wind or sun. And in these times flexible renewables that can ramp up and flexible consumers that can ramp down will be in high demand.

The business model of companies, such as NK, in other words, are robust today and are likely to become more valuable in the future.

How such a future evolves, of course, depends on where you are and what sort of regulatory policies you operate under. What may happen in Germany or New York or California—say over the next decade—may not apply to Indonesia, Saudi Arabia, or Mali.

These realities, however, are already at the front and center of priorities to be addressed by regulators in New York,¹³ California, and parts of Australia, to name a few. Other parts of the world are likely to confront similar issues in the near future.

As noted in an article in Energy Central,¹⁴ the rapid rise of renewables in places like Germany and Denmark has already made the job of the grid operators more challenging than in the past. And as the example of NK and their cohorts suggests, this challenge has created opportunities to offer services that are sought by generators, consumers, and the grid operator. The question is, how will such new business models emerge, evolve, and how fast are they likely to grow? In this context, it is also important to ask what role will incumbent utilities play—whether generators, distribution utilities, or other stakeholders?¹⁵

Clearly, a growing number of companies, such as NK will emerge to aggregate decentralized generation capacity and loads to form VPPs while offering a myriad of products and services including ancillary services to TSOs (Fig. 17.16).

If the experience of NK is any indication, such enterprises are likely to grow exponentially, and as they do, offer enhanced services and expand into other markets.

Founded in 2009, Next Kraftwerke has enjoyed explosive growth. In 2013 they aggregated 1 GW of capacity from 2400 installations and traded 2.5 TWh, mostly from biogas, biomass, and CHP plants, up from 1 TWh and 400 installations only a year previously. According to the latest statistics, the company now has over 4000 decentral units of virtually every renewable technology and loads under management, trading 9 TWh (Fig. 17.17).

NK’s rapid rise and commercial success can be traced to two valuable services increasingly sought after in future markets with high renewable penetration:

NK’s highly automated central control room in Cologne, Germany, adjusts the aggregated generation and loads according to current market prices, grid congestion, and weather data using its proprietary Next Box system.

Both producers and consumers benefit from participating in NK’s scheme—the typical savings depend on how flexible they are and how well they respond to price signals.

For the balancing market, NK are active in secondary and tertiary control reserve, which pays for availability as well as supply. In Belgium and Germany, they recently started offering primary control reserve or frequency response as well.

The availability payments eventually come out of the consumer’s wallet via grid charges, but the supply costs are covered by the market players who caused the imbalance—those who have not met their forecasts for generation or consumption.

NK’s customers typically get to keep a share of the value of the balancing contract, depending on their flexibility.

When the company was founded in 2009, NK were expecting the volume of ancillary services requested by the TSOs to rise because the feed-in of volatile renewables increased so rapidly. Seven years later they were surprised to find that they had underestimated the exploding significance of the intraday power market, which did not even exist in 2009. In effect, through the doing of the likes of NK, the amount of control reserve called by the TSOs has decreased (dark gray bars in Fig. 17.18) while the amount of electricity traded on the intraday market of EPEX SPOT has increased over-proportionately [light gray bars (green bars in web version)].

The volatility brought into the electricity market by wind and solar is now balanced on the intraday market, reducing the overall imbalance of the grid and thus reducing control reserve calls. The company interprets the development as a sign that the (spot) market can most efficiently deal with volatile renewables while ensuring the high level of energy security Germany is used to.

What is the expected growth trajectory for companies such as NK? The company’s co-founder, Hendrik Samisch, is confident in NK’s business model. He envisions that VPPs, such as his, will enable Germany to reach its ultimate goal of “100% renewable energy at a reasonable price.”

Highlighting his belief in market-oriented solutions for the integration of renewables into the energy system, Samisch says:

Next Kraftwerke’s ambitions, of course, do not stop at Germany’s borders. Samisch expects European electricity market designs to converge over the next five years, offering a huge opportunity to sell NK’s grid services across the continent. As European Commission regulation mandates it, cooperation between several TSOs in Europe, for example, on secondary control reserve, is intensifying, so balancing regions are growing larger—and NK want to play their part in the new market designs.

This explains NK’s recent expansions into neighboring markets in Austria, Belgium, France, Poland, the Netherlands, Switzerland, and most recently Italy. According to Samisch, TSOs are welcoming the opportunities of cooperation, such as lower redundancy requirements within each balancing area, and it gives NK a chance to grow. Primary reserve is already contracted transnationally in the ENTSO-E Regional Group Continental Europe.

What about the competition? Samisch expects increasing competition for grid services, not just from other VPPs, but also from Germany’s municipal utilities and even its Big Four utilities, RWE/Innogy, E.ON/Uniper, EnBW, and Vattenfall.

How will this game play out? No one can be sure but NK’s CEO says:

What NK believe is that when nuclear and coal/lignite plants are gradually closed down and market prices will move freely, there will be sufficient market incentives for setting up decentral, green flexibility options of renewable generation and running loads more flexibly to absorb scarcity and excess situations and supply the short-term flexibility in-between.

Considering the volatility of wind and solar generation over time and space, there will be quite high uncertainties as to how often a specific technology will be needed. High fixed cost assets, such as conventional power plants will likely not get sufficient security over their amortization periods—at least without government support.

Instead, the assets with low fixed costs and low or high variable costs will be most efficient to provide the bulk of flexibility in the future: the plants that are already in place, that are amortized, or have other primary use-cases outside the energy market. These assets could be CHP or biogas plants, (small-scale) hydro plants, or batteries used in electric vehicles, homes and industrial and commercial applications, and of course DSM in general.

According to NK, the flexibility principle of the future energy market will be: first shift, then store.

This complex and delicate dance of millions of decentral units will have to be choreographed by a digital utility in the center, ensuring the balance of supply and demand at all time while maintaining the grid’s reliability.

Next Kraftwerke have sought out this role and developed a successful business model around it. It is thrilling to speculate how NK’s green, decentralized vision might unfold and how the concept of variable demand responding to prices might evolve over time.