6.3 Frequency Synchronization in Packet Networks

Since the dawn of internet protocol (IP), IP packets have been transported over telecom networks. Until the end of the last century, the physical layer of inter-office connections was one of the traditional telecom technologies, analog, PDH, or SDH/SONET. However, the introduction of optical Ethernet interfaces towards the end of the 1990s started to change the scene. Now, the most common interface in telecom routers is Ethernet. Mobile networks long remained the last fortress of legacy transport since the bandwidth requirement of cellular base stations remained low. However, then HSPA (high-speed packet access) combined with fixed monthly fees exploded the bandwidth demand and PDH access did not scale up. Ethernet came as the saviour in the price per bit problem – but associated with another problem: synchronization.

Ethernet has been designed as a low-cost plug and play technology. Combined with IP friendly adaptation to higher layers, Ethernet has become the dominant link layer and physical layer technology in packet networks. Within the Ethernet scene bridges have almost completely overcome repeaters. Correspondingly, Ethernet itself spans just a single link, terminating the clock at the receiving port. The idea of connecting the received clock to outputs (as in SDH/SONET) had been around since the early years of the past decade and standardizing Synchronous Ethernet began mid-way through. However, it takes a long time from starting a standardization project until all links in a network segment support the standardized protocol. Luckily, regarding the use of Ethernet for cellular transport, the development of packet timing technologies for telecom had already started earlier, such as adaptive clock recovery (ACR), precision implementations of Network Time Protocol (NTP), and telecom oriented options for Precision Time Protocol (PTP). These protocols can operate independently of the physical layer and thus carry synchronization without on-path support.

6.3.1 ACR (Adaptive Clock Recovery)

Ethernet was first introduced into cellular backhaul when the endpoints, for example base stations were still PDH based. As a consequence there was a need to carry PDH frames over packet network. Pseudowire specifications such as SaToP [10] and CESoPSN [11] are used. In both cases, the packet rate is fixed, typically 1000 pps, which is enough information for recovering synchronization at the receiving end. Both specifications have the option of using RTP (Real Time Protocol) for time stamping the packets. However, this option is not necessary for clock recovery because of the fixed packet rate.

6.3.2 NTP

NTP is a quarter of a century old protocol designed for synchronization of time over the internet. It has been updated several times. Version 4 was published in June 2010 [12]. NTP has been designed to run in software that has similar access to the network layer as other software applications. According to the latest version the new algorithms extend the potential accuracy over LANs to tens of microseconds.

This accuracy would consume a meaningful proportion of the whole uncertainty budget considering frequency synchronization of base stations. Therefore, the traditional NTP is a slightly too inaccurate method with which to consider synchronization of mobile networks. However, NTPv4 mentions the possibility of accessing the physical layer and defines a location in the frame where the timestamps are associated. In recent years, NTP equipment with hardware assist has emerged.

The protocol stack of a telecom implementation is shown in Figure 6.2. The base stations have, in practice Ethernet interfaces. NTP utilizes UDP transport layer over unicast IP. The three top layers are carried end to end while the Ethernet layer is typically terminated in IP and MPLS routers.

Figure 6.2 NTP protocol stack.

Figure 6.3 depicts the messaging. If the NTP algorithm is used, the slave polls the master in bursts of eight messages where the individual messages are separated by two seconds from each other. The algorithm selects the best information available from the burst for maximizing accuracy. The interval between the bursts can vary between 16 s and 36 h. The time stamps correspond to the transmit and receive moments of the two messages.

Figure 6.3 NTP timing.

Assuming that the forward and reverse delays are equal the time error of the slave is

(6.1)

The time error is used by the clock tuning algorithm. As the packet rates are low, there may be proprietary implementations that use higher packet rates for achieving adequate accuracy for base stations.

6.3.3 PTP Protocol

PTP (Precision Time Protocol) [13] was originally designed for reaching very high time synchronization accuracy, for example, for industrial automation and measurement technology. NTP could not adjust to the requirements and therefore PTP protocol was designed. The first version of PTP included the necessary definitions for hardware time stamping (essentially time stamp point in the message), on-path support (boundary clocks, BC), automatic clock hierarchy build-up (best master clock algorithm, BMCA), and faster maximum packet rate than in NTP, 1 pps.

When the telecom synchronization industry started to develop packet timing for cellular backhaul, NTP was a relatively good option. However, for remaining truly conformant with the NTP standard, the packet rate and algorithm had constraints. Therefore, several telecom companies participated in the development of PTPv2. On the one hand, PTP protocol messaging could be tailored for high packet rate required for optimal performance. On the other hand, PTP has no restrictions on the actual clock algorithm, allowing the development of algorithms specifically designed for example for base station synchronization. Having said that, proprietary NTP algorithms could still achieve the same accuracy targets, however, as PTP standard has the means to also reach high time accuracy by using on-path support defined by the standard, PTP proved to be more future proof than NTP.

PTPv1 used a reserved multicast address, which is convenient when all nodes support the protocol (full on-path support). The nodes do not need to know their neighbours' IP addresses and clock hierarchy can be created based on physical connectivity instead of protocol addresses. This single-link multicast scheme is familiar from several other protocols that include hierarchy build-up, such as routing protocols on Layer 3 and spanning tree protocol on Layer 2. At the time when a packet timing protocol was needed for telecom, on-path support could not be dreamt of. As a consequence, IP unicast operation was developed and the packet rate was increased to cover all telecom needs, typically up to 64 pps. As there were many different interests associated with PTPv2, the standard became flooded with options, almost doubling the page count compared to version 1. For helping telecom implementers, an informative Annex A.9, ‘Recommendation for implementations in unicast networks or networks with non-PTP bridges and routers’, was prepared. It was a last-minute addition, but successful, as the first telecom implementations to reach mass roll-outs of PTP clocks were based on the annex.

Figure 6.4 depicts an example of the protocol stack in the case of frequency transport without on-path support. In this case PTPv2 Annex D, ‘Transport of PTP over User Datagram Protocol over Internet Protocol Version 4’, is utilized. As in NTP, the three topmost protocols are carried end-to-end. In the example, the base station transport of the mobile operator is based on Ethernet. The mobile operator has leased L2VPNs from a transport provider, which runs an MPLS network. Often the mobile operator has a small number of IP routers on the way, in which case the Ethernet layer does not span end-to-end.

Figure 6.4 Protocol stack for frequency synchronization without on-path support.

The synchronization principle in PTP is the same as in NTP. There is one time stamped message upstream and one downstream, see Figure 6.5. From the beginning, NTP has utilized a client-server scheme of operation whereas PTP has not. Originally, instead of waiting for a request from a slave, a PTP master sends Sync packets to the multicast address as long as it does not encounter another master with higher priority. For enabling client-server operation for unicast PTP, a new message class, signaling messages, was utilized to request unicast messages from the master. The Announce messages advertise the clock class of the master, and reception of these messages is necessary before a slave's state machine may enter the slave state. After entering the slave state, the slave starts to use the timing messages for synchronization.

Figure 6.5 PTP protocol messaging.

The three unicast requests may be contained in a single signaling message if found appropriate since the requests are in the form of 10 byte TLVs (type length value) and a number of them can be inserted in a single message.

The Follow_up messages are needed if the master does not have a real-time HW timestamper. When utilizing Follow_Ups, it is adequate to detect when the timing packet transmission occurred and then enter the information into a Follow_Up message by means of a software process. There is no follow-up for the Delay_Req message since it is adequate that the slave knows the accurate transmission time of the Delay_Req by the time the algorithm is ready for the time error calculation. The two messages whose transmission and receiving times have relevance are event messages and the rest are general messages. Even though the Follow_Up messages contain time stamps they are not event messages since the time they themselves are sent or received is not needed.

The protocol allows some creativeness in using the timing messages. One can decide to use all four time stamps or just two time stamps contained in one of the two event messages. The least bandwidth and processing power is consumed if only Sync messages are used. Using both directions usually yields the best result but consumes the most resources. Using upstream only may be attractive since this direction usually has a substantially lighter load than the downstream direction.

The error functions that a phase feedback servo tries to minimize to zero in downstream, upstream, and two-way cases are given in Equations 6.2, 6.3, and 6.4, respectively.

(6.2)

(6.3)

(6.4)

In the one-way cases a constant time error of one-way delay remains when the error function is minimized. This does not matter, as long as only frequency synchronization is targeted.

Table 6.3 summarizes the messages used in the unicast model. The UDP port number 319 indicates the event messages and port number 320 maps to general messages.

Table 6.3 PTP message types and protocol layer information.

6.3.4 ITU PTP Telecom Profile for Frequency Synchronization

The telecom profile is very similar to Annex A.9. The messaging is the same as shown in Figure 6.5. The protocol stack is PTP/UDP/IPv4. There are some differences, which are listed in Table 6.4.

Table 6.4 Differences in PTP messages between the two PTP profiles used in telecom.

In the default profile described in IEEE 1588-2008 the redundancy of masters is based in a system where only one Grandmaster is active at a time. If the preferred GM fails, then the second one in priority order takes over. In the unicast model where each slave separately loads a Grandmaster, there is a need to operate multiple Grandmasters simultaneously to serve all base stations, see Figure 6.6. Further, it might be decided that an extra Grandmaster operates as standby for the active Grandmasters. G.8265.1 defines a redundancy architecture where the slaves have a precedence table of GMs. They poll all GMs for Announce messages in order to know whether the clock quality of each potential master is adequate. If the preferred GM fails, then the slave starts to request timing messages from the redundant Grandmaster.

Figure 6.6 Redundancy architecture.

The polling of Announce messages by two groups of slaves may burden a GM too much if the groups are large. Therefore, the standard allows the option of not polling the spare master unless the preferred master fails. The redundancy mechanism described in G.8265.1 can be implemented in Annex A.9 slaves. Thus, functionally, the two versions could be identical.

The telecom profile specifies exclusively the use of IPv4. Thus, IPv6 is not allowed. However, it is straightforward to implement the protocol using IPv6 based on Annex E of IEEE 1588-2008.

6.3.5 Synchronous Ethernet

Synchronous Ethernet has been designed to a certain extent along the same principles as SDH. Therefore, similar synchronization network structures as in the case of SDH, see Figure 6.1, can be expected.

IEEE 802.3 working group standardizes Ethernet and has specified limits, for example 100-ppm clock accuracy for most Ethernet interfaces. ITU has worked within this space when specifying increased accuracy and forwarding synchronization from input to output. For indicating clock accuracy, synchronization status messages are needed as in the case of SDH. Using IEEE802.3 slow protocol family is a suitable option since 802.1 bridges do not forward these messages. Nowadays, IEEE802.3 allows the creation of organization specific slow protocols (OSSP). ITU has specified one for carrying the status messages and it is called ESMC (Ethernet Synchronization Messaging Channel). The ESMC principles are the same as for SSM for SDH. The synchronization protection switching and possible spanning tree protocol protection switching for the data traffic run independent of each other.

G.8261 defines network limits in terms of MTIE and TDEV, which are described later. EEC-Option 1 clock (Synchronous Ethernet equipment clock) has the same specification as SEC (SDH equipment clock). EEC-Option 2 clock has the same limits as SEC Option 2. Recommendation G.8262 Specifies the EEC clock and G.8264 describes the ESMC messages.

The allowed number of consecutive synchronous EEC Option 1 nodes in the synchronization path is the same as in the case of SEC Option 1 clocks, 20. Then, in principle an SSU should be added to connect to the next EEC.

6.3.5.1 Over Optical Transport Network, OTN

Similar to SDH as a client signal, OTN was made transparent for Ethernet clients and defined in such a way that Synchronous Ethernet timing signals remained within acceptable jitter and wander limits. These aspects are discussed in [14]. It has turned out that Synchronous Ethernet can traverse tens of OTN nodes without accumulating too much jitter or wander.

6.3.5.2 One-way Synchronous Ethernet Links

Most of the Ethernet interfaces are such that the direction of synchronization can be changed quickly as required by the synchronization protection switching described in Figure 6.1. However, there are some exceptions. In the 1-Gbit/s and 10-Gbit/s copper interfaces (1000BASE-T and 10GBASE-T) the two ends of the links will have the same frequency. One end will always be selected as the master and the other as a slave. The direction of the synchronization can be managed by setting the auto-negotiation parameters of the interfaces. However, changing the direction would require setting the parameters in a different way and resetting the interface. The traffic might be interrupted for many seconds when several such interfaces in a chain would be reset. Therefore, the directions are set semi-permanently. Note, the one-way issue does not concern the corresponding optical interfaces. However, PON (passive optical networks) links are also one-way in terms of synchronization because the head-end is always synchronization master for running the PON transport protocol. This is not a problem, however, since PONs are used in the edge of the network oriented in the right direction concerning the synchronization hierarchy.

6.3.6 Chaining Different Synchronization Technologies

When chaining different synchronization technologies, e.g. packet timing and physical layer timing, one must take into account that each island consumes a proportion of the total wander budget. When considering chaining different synchronization technologies, the reader is encouraged to study the different deployment cases in G.8261.

6.3.7 Summary of ITU Recommendations Related to Frequency Synchronization in Packet Networks

6.3.7.1 G.8260, Definitions and Terminology for Synchronization in Packet Networks

Considers mostly packet timing since G.810, ‘Definitions and terminology for synchronization networks’, contains most of the definitions related to physical layer frequency synchronization applicable also for Synchronous Ethernet. G.8260 includes an appendix about packet timing metrics.

6.3.7.2 G.8261/Y.1361, Timing and Synchronization Aspects in Packet Networks

‘The handbook of timing in packet networks’. The recommendation describes timing in circuit emulation service (CES), Synchronous Ethernet, and packet timing including network limits for different deployment cases. Further, it defines test cases for packet timing.

6.3.7.3 G.8261.1, Packet Delay Variation Network Limits Applicable to Packet Based Methods (Frequency Synchronization)

This specifies the network limits on packet delay variation depending on reference model. The first version bases the limits on Hypothetical reference model 1 (HRM-1), a network of 10 nodes between the master and slave. The links are otherwise 1 Gbit/s fiber optic links except three links that are 10 Gbit/s fiber optic links.

6.3.7.4 G.8262/Y.1362, Timing Characteristics of Synchronous Ethernet Equipment Slave Clock (EEC)

The recommendation is very similar to G.813, Timing characteristics of SDH equipment slave clocks (SEC). The intent of synchronous Ethernet is to interoperate with existing synchronization networks based on G.813.

6.3.7.5 G.8263, Timing Characteristics of Packet Based Equipment Clocks (PEC) and Packet Based Service Clocks (PSC)

The recommendation describes PEC-M, Packet master clock, PEC-S, Packet slave clock, and PEC-B, Combined packet slave clock and packet master clock.

6.3.7.6 G.8264/Y.1364, Distribution of Timing Information Through Packet Networks

The recommendation describes the Ethernet Synchronization Messaging Channel and reference source selection mechanism.

6.3.7.7 G.8265/Y1365, Architecture and Requirements for Packet-based Frequency Delivery

Recommendation describes the general architecture of frequency distribution using packet-based methods. This is applicable to PTP and NTP.

6.3.7.8 G.8265.1/Y.1365.1, Precision Time Protocol Telecom Profile for Frequency Synchronization

ITU PTP profile definitions for frequency delivery

6.3.7.8 Synchronous Ethernet Over Optical Networks and Testing

G.709/Y.1331, Interfaces for the Optical Transport Network (OTN) about mapping clients to OTN (optical transport network) and the corresponding jitter and wander specifications G.8251, The control of jitter and wander within the optical transport network (OTN), have been developed to allow passing synchronous Ethernet across OTN with minimal jitter and wander accumulation. O.174 Jitter and wander measuring equipment for digital systems which are based on synchronous Ethernet technology completes the specifications related to Synchronous Ethernet.

6.3.8 TICTOC

The TICTOC (Timing over IP Connection and Transfer of Clock) working group in IETF has been running since 2006. The working group is concerned with highly accurate time and frequency distribution over native IP and MPLS-enabled IP packet switched networks. The charter overlaps with the agenda of ITU-T Question 13 (Network synchronization and time distribution performance) of study group 15. It has taken some time to find topics that naturally fall into the realm of an IETF working group and the early documents have expired. Currently TICTOC has two active working group documents ‘Transporting PTP messages (1588) over MPLS Networks’ and ‘Precision Time Protocol Version 2 (PTPv2) Management Information Base (MIB)’.

The first one defines the method for transporting PTP messages over an MPLS network. The method allows for the identification of PTP messages at the port level to allow for port level processing of these PDUs in MPLS equipment. Setting up corresponding pseudowires and label switched paths are described in related Internet-Drafts. In the case of frequency transport the method does not usually need to be used, because adequate synchronization quality can be achieved by carrying PTP over the same pseudowires as data. In the case of time/phase transport or very strict frequency synchronization needs, the method may be useful for ensuring symmetric paths and filtering PDV on the way. However, in the case every node supports the protocol, then the forthcoming ITU PTP time profile, see Section 6.7.2, is probably a better choice. The MIB document defines managed objects used for managing PTP devices. The managed objects can then be used with network management protocols.