1.2.1 Electricity markets and mechanisms

• • Wholesale market: The UK wholesale market is based on direct and unrestricted bilateral trades between different actors (e.g., generators and retailers). This mechanism allows the trade of electricity from several years in advance until 1 h before delivery (gate closure). As bilateral trading is done without regard for network constraints, National Grid, which takes the role of both system operator and transmission network operator, is responsible for assessing and approving or rejecting every transaction [10].
• • Balancing market: As some actors may not deliver or consume the exact volume of energy agreed upon in the wholesale market, additional energy services are traded in the balancing market to match electricity supply and demand. The balancing market, which is also operated by National Grid, operates between gate closure and the time of delivery. In this market, generators offer to increase or decrease their output and retailers offer to increase or decrease the consumption of their customers if requested by the TSO.
• • Imbalance settlement: As mentioned above, the balancing market allows the energy market to cope with actors that do not generate or consume the exact amount of power that they traded. As this results in additional costs for the system, some actors (i.e., balancing responsible actors) might be penalized for introducing imbalances to the market. Balancing penalties are calculated ex-post based on the position of the market (i.e., long or short) and the balancing responsible actors as well as the costs for balancing the market [11].
• • Ancillary service market: As part of the grid code, some actors are obliged to provide specific ancillary services (e.g., reserve, frequency response, and so forth) [12]. Nevertheless, these (or other actors) may be flexible enough to provide services beyond their obligation, which can be traded in the ancillary service market. This market is managed by the TSO, which is the single buyer [13].

In addition to these markets, there are mechanisms in place to manage the electricity transmission and distribution businesses. The Office of gas and electricity markets (Ofgem), which is the UK regulator, imposes price controls on the TSO as well as on the different distribution network operators (DNOs) across the country. These mechanisms are revised regularly (roughly every 5 years). The latest iteration of the price controls, namely Revenue = Incentives + Innovation + Outputs (RIIO), sets the network prices and incentives from 2015 to 2023 [14]. These prices are defined based on a wide range of mechanisms, tools, and considerations, such as uncertainty mechanisms, quality of supply incentives, broad measure of customer satisfaction, loss reporting and incentive mechanisms, an innovation stimulus package, operation and capital expenditure incentives, and workforce renewal incentives, among others [15].

1.2.2 Actors

• • TSO: The UK TSO, namely National Grid, is a regulated entity that is responsible for the management and development of the grid transmission and for the operation of most electricity markets and mechanisms (without selling electricity to customers). National Grid provides physical access to the electricity market to different actors (e.g., generators) according to nondiscriminatory and transparent rules, and takes the role of market operator. In addition, the TSO ensures the security of supply, the safe operation and maintenance of the system, and a generation-consumption balance via services provided by different actors in accordance to the grid code or contracted in the balancing and ancillary markets [13]. In addition, balancing service and network costs are passed to the relevant actors based on specific mechanisms and system charges [16].
• • DNOs: DNOs are regulated entities that are responsible for the transport of the electrical power on the distribution networks. DNOs provide physical access to the distribution network to customers according to nondiscriminatory and transparent rules. In addition, they are also responsible for the safe, reliable, and economic operation of the network and for investments in new infrastructure. Distribution costs are externalized to customers in the form of system and connection charges [17]. Currently, there are 14 licensed DNOs (and several independent DNOs) in the United Kingdom.
• • Retailer: The electricity retailer acts as an intermediate agent between the wholesale and balancing energy markets and customers. The normal operation of the retailer involves buying energy from the wholesale market at a variable price and selling the energy to customers at flat prices. These retail tariffs include a profit margin that retailers charge for “protecting” customers from the variable market prices and signals. Retailers are balancing responsible actors and are penalized for introducing imbalances to the markets (i.e., from consuming a different amount of power than contracted), and are allowed to participate in the balancing market.
• • Generators: Bulk generation comprises traditional large generators connected to the transmission system, such as nuclear and large-scale coal- and gas-fired generation. These generators can participate directly in the wholesale, balancing and ancillary markets. Due to economies of scale, this type of generator generally has an advantage in the wholesale market over DER. However, this is not necessarily the case on the balancing and ancillary markets where specific types of technologies may have the advantage regardless of economies of scale.
• • Customers: Customers can comprise a variety of entities such as households, residential and commercial buildings, and small businesses, among others. In a traditional context, these customers passively consume electricity. However, as will be further discussed below, customers in a smart grid context may possess DERs such as PV systems, combined heat and power (CHP) boilers, and so forth, which can enable customers to actively participate in the different energy markets and mechanisms.

2.1.1 Technologies

As part of the UK network trials, selected distribution feeders were enhanced to enable improved network operation and the provision of DSR within relevant commercial and technical arrangements. Robust and low-cost remote controls were installed at the NOPs to enable automatic response as well as the option for normally closed operation. Moreover, several switches and automatic controls (e.g., 132 kV/6.6 kV/240 V switches and molded case circuit breakers) were installed throughout the networks to improve automatic network response and, potentially, isolate contingencies in smaller sections of the networks. In addition, new technologies were developed to enable the use of different network configurations. Automated and controllable low-voltage switches (called LYNXs) were developed to allow reconfiguring or meshing low-voltage networks and, from the protection perspective, fuses at the low-voltage side of the distribution transformers were replaced with smart breakers called WEEZAPs [29]. Both LYNX and WEEZAP devices can be operated remotely and can record valuable technical data (i.e., phase power, voltage, current, and so forth).

In order to enable DSR, circuit breakers fitted with automation were installed for new customers or, in the case of existing customers, available devices were upgraded with a retrofitted actuator. The devices were fitted with a remote terminal unit to provide the required communication link. The trial circuits were also enhanced with additional monitoring equipment (e.g., supervisory control and data acquisition systems). This enabled the collection of real-time network data for the analysis of real performance and to inform a series of network simulations and modeling exercises.

The automation algorithms within ENWL's control room management system, which is in charge of the normal operation and restoration of the networks, were updated with so-called automatic restoration sequence algorithms to take advantage of the enhanced automation levels, network reconfiguration capabilities, and DSR. These algorithms are based on predefined actions informed by the conditions of the network as estimated based on offline power flow estimations (using PowerOn Fusion software [30]) informed by measured demand data.

2.1.2 Commercial arrangements

New commercial agreements were developed for active customers who were willing to provide DSR. After testing several permutations of these contracts, two types of contracts (for existing and new customers) based on variations of the existing national terms of connection agreement were selected. These contracts define the conditions under which the customers are willing to provide flexibility, such as, for example, protected days, maximum number of calls per year, contract length, and other parameters. This work has led to the creation of managed contracts for the provision of DSR in the United Kingdom, which has now become BAU practices [31].

2.1.3 Network security and reliability

In the United Kingdom, distribution network security standards are based on P2/6 engineering recommendations, which are analogous to N-1 conditions but also include considerations for time and network demand [18]. Reliability levels are regulated in terms of customer interruptions (CI) and customer minutes lost (CML) for interruptions exceeding 3 min. As both security and reliability levels are based on infrequent interruptions, it was impractical to wait for occurrences in every trial network to obtain relevant security and reliability data. Therefore, network security limits and reliability levels were estimated based on simulations, which were updated in light of the five contingencies that were recorded in the trial networks throughout the duration of the C₂C project [32].

The security limits were simulated based on P2/6 considerations and AC power flows to identify the firm capacity of the network [20,25]. More specifically, contingencies in every section of the network were simulated to identify the potential network configurations during emergency conditions. AC power flows were used to identify thermal and voltage constraints and assess whether the networks were able to supply customers, considering daily load curves and emergency ratings (i.e., up to 20% overload for 2 h). The studies considered five demand growth scenarios proposed by ENWL as well as different options to upgrade the feeders and substations and deploy network automation and DSR.

Network reliability was assessed via Monte Carlo simulations while assuming, (i) real failure rates for the available trial networks (a rate of 0.05 failures/km/year was assumed when real data was unavailable [20]), (ii) the existing number of customers in each feeder (distributed across the network based on the demand at each load point), (iii) manual operation of the NOP of 1 h on average and distributed based on an exponential probability density function, (iv) no NOP automation in the existing networks, (v) mean time to failure, mean time to repair, and mean time to switch of 175.2, 5, and 1 h, respectively, and (vi) that the automated network can restore supply within 3 min after a contingency occurs. Randomly allocated faults are repeatedly simulated thought the networks to estimate average interruptions, also considering the conditions mentioned above. The results are the expected CI and CML for each network subject to different demand growth scenarios and interventions (i.e., reinforcements, enhanced automation, DSR, and so forth).

2.1.4 Economics

The economic evaluations of smart grid solutions at the distribution level, such as the use of DSR, should be consistent with existing UK regulations on distribution network assets built, namely the cost-benefit analysis (CBA) framework introduced as part of the RIIO price control [33]. This CBA framework proposed by Ofgem, herby called Ofgem's CBA framework, provides a set of mathematical tools to assess all investments in distribution assets. More specifically, as denoted by Eq. (1), investments in distribution network interventions are assessed in terms of the net present cost (NPC) criterion, which sums the discounted investment (Investment_costs_y) and social (Social_costs_y) costs associated with the intervention during every year throughout predefined discount rates (d = 3%) and lifetime (T = 45 years).

$\mathrm{NPC}={\sum }_{y=1}^T\frac{\mathit{Investment}\_{\mathit{costs}}_y+\mathit{Social}\_{\mathit{costs}}_y}{{\left(1+d\right)}^y}$

It is worth noting that investment costs are attributed to all capital expenditures associated with the intervention (e.g., investment costs, annual payments, maintenance, etc.) whereas social costs are based on losses, emissions, CI, and CML, among other factors. Further details on Ofgem's CBA framework and its applications can be found in the available literature [2,19,34].

2.2.1 Technologies

Several technologies were required for the DIMMER field trials and studies, including multienergy technologies, data acquisition systems, and a portfolio of mathematical tools to simulate and optimize the behavior of the community energy system. The multienergy technologies under consideration were those that were already in place in the Manchester demonstrator (e.g., PV panels and gas boilers) and those that the university is exploring as a means to improve energy efficiency in the coming years (e.g., CHP boilers, PV panels, and electric heat pumps (EHP)). An example of how these technologies may interact is presented in Fig. 7.

Historical energy consumption data for all buildings throughout the University of Manchester is available with a half-hour resolution from the Coherent data system [39]. An interface was built between the DIMMER platform and the Coherent data system to acquire information in real time. The mathematical tools developed for the project included, (i) multienergy operation, simulation, and optimization models [40,41], (ii) integrated multivector district energy network models [42], (iii) an integrated district energy management system framework [24], and (iv) a CBA and multiflow mapping model [38]. These tools allow the assessment of the energy performance of community energy systems, also considering smart coordination of DERs for the minimization of energy costs and emissions, and trade of services in different markets and mechanisms. The operation of the communities takes into consideration multienergy exchanges between different buildings, which are analyzed using dedicated technical models. The business case of different energy services is evaluated from the perspective of the community as well as considering impacts to generators, DNOs, the government, and other actors (see Section 1.2.2 for a list of relevant actors).

2.2.2 Commercial arrangements

In a traditional energy system context, passive customers only interact with retailers, which protect the customers from the dynamic prices and signals in the energy system by only exposing them to flat retail prices (see Section 1.4). As a result, customers are unable to see and react to the multiple components that constitute the customer price, such as system fees for the distribution and transmission networks, wholesale electricity prices, various taxes, capacity, the retailer margin, and so forth [38,43]. This is a reasonable approach for customers who are unable to react to these dynamic signals, but not for smart customers such as those within smart community energy systems who could use their flexible resources to benefit from these signals and relevant markets.

Allowing some of the variable components of these customer prices to be passed through to customers, which is known as a transactive energy approach [36,44], is crucial to incentivize the smart operation of communities [45]. More specifically, community energy systems could actively respond to variable signals as a means to minimize energy costs and emissions as well as to maximize revenue from participating in different markets. This smart community behavior where flexibility is used to trade services in different markets can, in principle, enhance the efficiency and security of the energy system. Accordingly, as part of the DIMMER trials, the operation of the smart community system is simulated and optimized under the consideration that different combinations of flat retail and dynamic market prices are passed down to customers.

2.2.3 Energy efficiency

The use of smart solutions and DERs at the building level are generally seen as attractive options to improve energy efficiency by reducing energy consumption. This traditional definition of energy efficiency focuses on energy savings (Energy_savings) that, in principle, lead to lower energy costs and emissions. That is, improving energy efficiency based on this traditional view simply implies reducing energy consumption (Current_Energy) compared with a baseline (Baseline_Energy) while considering externalities, as shown in Eq. (2). It is worth noting that externalities can be associated with weather, occupancy levels, and so forth.

$\mathit{Energy}\_\mathit{savings}=\mathit{Baseline}\_\mathit{Energy}-\mathit{Current}\_\mathit{Energy}\pm \mathit{Externalities}$

The main disadvantage of this definition of energy efficiency is that it cannot consider the different dynamic costs and carbon intensities associated with different energy vectors. For example, based on the traditional energy efficiency concept, reducing electricity consumption has the same value regardless of whether this occurs when the market is saturated with highly costly and dirty fossil fuel-base energy, or clean and cheap renewable energy. This is not reasonable, as customers should not make active efforts to reduce their consumption when there is surplus renewable energy [36].

Based on the above, the traditional energy efficiency is only adequate for passive customers who cannot respond to dynamic signals. However, this concept undermines the flexibility of smart community energy systems that can actively respond to dynamic prices and carbon intensities and exploit diversity by allowing multienergy trading between buildings. Thus, in order to properly capture the flexibility inherent in smart community energy systems, new energy concepts have been proposed, such as the one denoted by Eq. (3) [5,36].

$\mathit{Energy}\_\mathit{Consumption}=\sum _m\sum _b\sum _t{\mathit{Imports}}_{v,b,t}\times \mathit{Imports}\_{\mathit{Externalities}}_{v,b,t}-\sum _m\sum _b\sum _t{\mathit{Exports}}_{v,b,t}\times \mathit{Exports}\_{\mathit{Externalities}}_{v,b,t}$

This new concept of dynamic environmental efficiency explicitly considers multienergy (m) exchanges between buildings (b) through time (t) as well as variable externalities relevant to imports or exports. As before, these externalities can account for weather and occupancy levels, but now variable energy market conditions and the physical constraints of the integrated (electricity, heat, and gas) energy networks can also be considered. This is vital for smart community energy systems as it allows the consideration of the consumption side for the provision of flexibility to the energy system [36,46,47]. Discussion on the proposed dynamic environmental efficiency and technoeconomic models developed to implement this improved efficiency paradigm in smart multienergy communities can be found in Ref. [36].

4.1.1 Smart distribution network applications

The consideration of smart grid solutions based on the above-mentioned strategies is illustrated here with applications to one of the 36 trial networks that have been monitored in detail as part of the C₂C project [60]. The network selected for this study is the Chamber Hall system (see Fig. 11), which is a 6.6 kV distribution network connected to a 20.96 MW substation. The two feeders selected to form the open or closed rings are the Europa House and Peel Mill No.2 feeders. The Europa House feeder is formed by 21 lines connecting 22 nodes and supplies 1950 customers (mainly urban). The Peel Mill No.2 feeder is formed by 22 lines connecting 23 nodes and supplies 1660 customers (mainly urban).

The study considers the implementation of one or combinations of the traditional and smart interventions presented in Table 1. These interventions are deployed based on different strategies (i.e., baseline, smart, and optimal) to ensure that the network limits are not exceeded under the different demand growth scenarios presented in Fig. 12.

Based on studies informed with data taken from UK trials, upgrading the substation costs 338£ × 10³ and takes 3 years whereas reinforcing the feeders costs 110£ × 10³ and takes 2 years. The network automation levels can be enhanced at a cost of 19 k£, and up to two 0.5 kW peak DSR blocks can be contracted. For this purpose, increasing network automation levels costs 19 k£ per year whereas customer automation costs 13.5£ × 10³ per block, and there are additional 6.3£ × 10³ costs in billing and management fees per block. Overall, it is considered that enhancing the automation levels of the network and contracting DSR takes roughly a year. The recommended interventions and associated NPC associated with each investment strategy are presented in Tables 2 and 3, respectively.

The study shows that, even under current planning practices, the introduction of smart nonasset-based solutions generally leads to lower investment costs. This is mainly due to deferral or avoidance of traditional asset-based investments, particularly costly substation upgrades. The smart solutions can also bring about attractive social benefits if ring operations are allowed, regardless of the potential increase in short-term interruptions (short-term interruptions are not penalized under current regulation). However, due to inherent uncertainty in the demand growth forecasts, current practices may not properly address smart solutions and could actually lead to increased network costs, as can be seen in the application of the smart strategies on Scenario 4.

The use of smart nonasset-based solutions becomes more valuable when improved network practices, based on proper optimization tools, are considered (i.e., optimal strategy). As presented, optimal strategies tend to combine traditional and smart solutions as a means to minimize costs. This makes sense considering that DSR can be an attractive option to defer or avoid costly investments, but the annual DSR payment may also become pricey. Accordingly, DSR can be used to postpone costly investments until it is clear that the intervention will be required (avoiding sunk costs) while less-expensive interventions (IF1 in this case) can also be used to reduce the need for DSR, allowing the DNO to contract fewer DSR blocks. It is important to note that these smarter network practices can also identify conditions where traditional interventions are still cost-effective (i.e., Scenario 4) or when smart solutions are preferred (i.e., Scenario 5).

These results highlight that smart grid applications can bring about attractive technoeconomic benefits. However, proper tools, or at least a proper understanding of the conditions when these solutions make sense, are needed for exploiting the full potential of smart grids. Based on these premises, the optimization engine used for the aforementioned studies has been made available as open source material [25].

4.1.2 Wider network implications

A more comprehensive analysis of smart grid applications to distribution networks requires the consideration of a wide range of networks under different conditions. For this purpose, in this section, the smart solutions discussed above are addressed here in light of 36 real UK distribution networks (see Fig. 13 for the schematics of some of these networks). The following considerations were taken for the studies:

• • Costs and assumptions: All economic criteria (e.g., discount rates, planning horizon, and price for power losses, among others) were taken from the trial networks, were recommended by the DNO, or were based on existing regulations [3]. However, these conditions may change in the future, which must be quantified. To this end, cost variations, particularly making traditional asset-based solutions more cost-effective, are explored.
• • Substation headroom: Smart solutions are attractive options to defer or avoid costly interventions, such as substation upgrades. This conclusion is tested by varying substation headroom so that conditions arise where investments in additional transformers may not be required or may be needed in the short or long term.
• • DSR availability: Depending on customer preferences, different levels of DSR may be available in different networks. Accordingly, the study is extended to consider conditions where only one DSR block, and up to five blocks, are available.
• • Demand scenarios: Demand is scaled up so that line reinforcements will be triggered after a further 3% demand increase, which guarantees that at least one intervention will be required in every study. As in the previous study, demand is modeled based on the five demand growth scenarios proposed by the relevant DNO (see Fig. 12).

An overview of the average NPC values associated with the 36 networks and different studies is presented in Fig. 14. The results corroborate the initial observation that smart grid applications generally lead to cost reductions, especially if environmental, reliability, and other so-called social benefits are also internalized (e.g., Smart—Ring strategy). Nevertheless, it also becomes evident that certain conditions make traditional asset-based solutions more convenient than smart solutions under current planning practices that are not able to effectively deploy smart solutions. This is the case for conditions where traditional asset-based solutions become cheap or costly investments are only envisioned in the far future. These effects can be seen in Fig. 14A where the baseline becomes cheaper on average than one of the smart solutions when costs of traditional solutions drop by more than 60% of their current value. This can also be seen in Fig. 14B where the baselines outperform the smart solutions in cases with high substation headroom.

The studies also highlight that the use of proper optimization tools to capture the features of smart solutions can bring about significant benefits. This is clear by comparing the performance of the optimal strategy with any other strategy in Fig. 14. Taking the analysis further, Figs. 15 and 16 provide comparisons between the baseline and smart and optimal strategies in terms of the number of cases when one solution outperforms the other. More specifically, instead of only presenting the average values as in Fig. 14, the box plots in Figs. 15 and 16 show how often and by how much the smart or optimal strategies outperformed the baseline in terms of minimum and maximum values, first and third quantiles, and median values.

These studies provide further evidence that current planning practices could frequently cause the suboptimal use of smart solutions. This may occur even when the average value of smart strategies is greater than that of the baseline. Conversely, the use of smart solutions combined with enhanced planning practices based on optimizations resulted in economic savings (or the same costs as the baseline) in all cases considered in this study. These findings provide strong justifications for DNOs to pursue the use of improved network investment and operation planning practices.

4.2.1 Smart community applications

The portfolio of tools discussed in Section 3.2 was used to assess the energy performance of the community based on the different cases and considering BAU and optimal operation. More specifically, in a BAU context, all controllable energy devices within the district are operated in heat-following mode, as is typically done in the United Kingdom. In a smarter context where the operation of the community is optimized, all energy infrastructure is operated with the objective of minimizing energy costs and emissions while, at the same time, partaking in the available energy markets and mechanisms (see Section 1.2). An example of the energy performance of the district under both BAU and optimal operation (considering all markets and mechanisms and prioritizing economic benefits) is presented in Table 4. A comparison of expected economic and environmental benefits compared with the reference case can be seen in Table 5.

It is important to note that, in a smart context where the operation of all energy devices can be optimized, the energy performance of the community can be flexibly customized based on customer preferences and participation in different markets. The operational flexibility of the smart community is illustrated in Fig. 18. As shown, under BAU practices, the energy performance of the community is static. This does not change much in cases where little or no controllable technologies are available, as is the case of the Reference and Conservative cases. However, as shown in the Modest and Extreme cases, the intelligent coordination of controllable devices brings about significant economic and environmental benefits. For example, in this study, it can be seen that the 30% reduction targets can only be met in the extreme case under BAU practices. However, the environmental targets can be reached while also providing attractive economic gain in the modest case when the operation of the community is optimized.

The study also shows that the community can achieve different combinations of economic and environmental benefits by changing its operation regime and without compromising customer comfort (i.e., all demand is always met). This allows valuable headroom to accommodate customer preferences and unforeseen conditions, such as higher prices and carbon intensity, than originally forecast. The latter is particularly valuable if, for example, the forecast environmental benefits from the decarbonization of electricity are not as significant as expected. In such a case, the district may still meet the environmental targets just by updating its operation and without the need for investments in additional energy infrastructures. Another example is the provision of different services such as DSR just by changing the operation of the district. This could allow the community to provide distribution network support (as presented in Section 4.1) without causing customer disutility. This makes the provision of DSR for distribution network support particularly attractive for smart community energy systems.

Assuming typical UK prices for PV, EHP, and CHP technologies of 1300, 240, and 600£/kW, respectively, the energy savings attributed to the smart community energy system would pay back relevant investments in 6 (in the conservative case) to 8 years (in the extreme case). If customers are focused on net savings and revenue, the extreme case provides the highest benefits under optimistic economic conditions (i.e., a net present value of 5 M£ considering a discount rate of 3% and a planning horizon of 20 years). Assuming more pessimistic conditions where customers (or other actors) may demand higher returns before investing in energy technologies (e.g., considering a higher discount rate of 10%), the Modest case may become the preferred one due to its lower cost and relatively similar benefits to those in the Extreme case (see Fig. 18). The baseline case only becomes attractive under highly pessimistic conditions where technologies become too pricey and investments in energy infrastructure require significantly large expected return of investments. This provides strong evidence that smart community multienergy systems are generally economically attractive and highly profitable.

4.2.2 Wider system implications

The emergence of smart communities that, besides consuming energy also actively participate in different markets and mechanisms, can have significant impacts on the energy system and the business case of different actors (see Section 1.2). For example, as the Manchester district optimizes its operation more and more using controllable CHP boilers, the community will become less dependent on grid imports (the community will now also export energy to the different markets) while increasing gas consumption as shown in Fig. 19.

The new operation strategy of the community leads to a reduction in annual energy costs for customers. However, as shown in Fig. 20, the emergence of smart community multienergy systems has a clear effect on the business case of generators, the government, DNOs, and the TSO, who are expected to accrue less revenue. This information is critical for policy makers, who must decide whether or not each of these specific impacts are beneficial or not for the energy system and update regulations accordingly. For example, negative impacts on generators may be deemed reasonable if they lead to decommission of inefficient generation, but DNO losses may pose an issue as distribution network upgrades may be needed to enable the smart operation of communities (see Section 2). In order to address the latter, policymakers may update regulations so that, for example, some of the economic benefits accrued by smart communities are shared with DNOs and used for the improvement of the distribution networks.