One-way Delay Measurement Based on
Reference DelayChina MobileBeijing 100053Chinayanghongwei@chinamobile.comChina MobileBeijing100053Chinayaokehan@chinamobile.comEPFLIC Station 14Lausanne EPFL1015Switzerlandjean-yves.leboudec@epfl.ch
Transport
Network Working Groupreference delay; delay measurementThe end-to-end network one-way delay is an important performance
metric in the 5G network. For realizing the accurate one-way delay
measurement, existing methods requires the end-to-end deployment of
accurate clock synchronization mechanism, such as PTP or GPS, which
results in relatively high deployment cost. Another method can derive
the one-way delay from the round-trip delay. In this case, since the
delay of the downlink and uplink of the 5G network may be asymmetric,
the measurement accuracy is relatively low. Hence, this document
introduces a method to measure the end-to-end network one-way delay
based on a reference delay guaranteed by deterministic networking
without clock synchronization.With the gradual promotion of new-generation network technologies
(such as 5G networks) and their application in various industries, SLA
guarantees for network quality become more and more important. For
example, different 5G services have different requirements for network
performance indicators such as delay, jitter, packet loss, and
bandwidth. Among them, the 5G network delay is defined as end-to-end
one-way delay of the network. Real-time and accurate measurement of the
end-to-end one-way delay is very important for the SLA guarantee of
network services, and has become an urgent and important
requirement.As shown in figure 1, 5G network HD video surveillance service is a
common scenario having requirement of end-to-end one-way delay
measurement. In this case, one end of the network is a high-definition
surveillance camera in the wireless access side, and the other end of
the network is a video server. The end-to-end one-way delay from the
surveillance camera to the video server is the sum of T1, T2, T3 and T4,
which is composed of delay in wireless access network, optical
transmission network, 5G core network, and IP data network.The existing one-way delay measurement solutions are divided into two
types. One type of mechanism to calculate one-way delay is based on the
measurement of round-trip delay. However, for example, because upstream
traffic and downstream traffic do not share the same path in 5G network,
the accuracy of the end-to-end one-way delay calculated from the
round-trip delay is low. Another type of mechanism is in-band OAM with
accurate network time synchronization mechanism , such as NTP or PTP.The one-way delay measurement solution based on precise network time
synchronization requires the deployment of an end-to-end time
synchronization mechanism. The current time synchronization accuracy
based on the NTP protocol can only reach millisecond level, which cannot
fully meet the measurement accuracy requirements. The time
synchronization accuracy based on the GPS module or the PTP protocol can
meet the requirements. However, because many data centers are actually
located underground or in rooms without GPS signals, so GPS clock
information cannot be continuously obtained for time synchronization.
For time synchronization solutions based on the PTP protocol, each
device in the wireless access network, 5G transport network, and 5G core
network must support the PTP protocol, which is unrealistic at the
moment. So the one-way delay measurement solution based on precise
end-to-end time synchronization is expensive and difficult to be
deployed.This document introduces a one-way delay measurement mechanism for
Deterministic Networking (DetNet) . The one-way
delay measurement is based on a stable one-way delay of a reference
DetNet packet, named as reference delay, which is known in advance and
has extremely low jitter. We can use the reference delay provided by the
reference DetNet packet to derive the one-way delay of other common
service packets.NTP Network Time ProtocolPTP Precision Time ProtocolSLA Service Level AgreementThe key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 when, and only
when, they appear in all capitals, as shown here.The end-to-end one-way delay of a packet with bounded delay that's
sent through a deterministic network path can be used as a reference
delay, which is known in advance and has extremely low jitter. This
section will describe the end-to-end one-way delay measurement method
based on reference delay in details . Assume that the end-to-end one-way
delay of a target packet is being measured, as shown in figure 2, the
target packet is transimitted through a normal network path while the
reference packet is sent through a deterministic network path. At the
meantime, we assume that there is a global clock which could offer very
precisive timing capabilities, and we denote its current time to be true
time, t. That is to say, for local clocks at sender and receiver, their
current time are Cs(t) and Cr(t) respectively. The reference packet is
sent at first from the sender with its local timestamp Cs(Ts1) marked
inside, and at true time Tr1, the reference packet arrives at the
receiver, and the receiver shows Cr(Tr1). Similarly, the departure and
arrival timestamps of the target packet are Cs(Ts2) and Cr(Tr2). Since
clocks at sender and receiver are not time synchronized, target delay
can not be directly measured by making subtraction. However, the boundedness of the reference delay can be leveraged for
measurement, as regulated in . The boundedness of
the reference delay can be formulated by equation 1. Equation 1: L - J <= DTrue_ref <= LIn equation 1, L is the maximum value of the reference delay and J is
the peak-to-peak value of the reference delay. L and J are usually
measured in tens of microsecond level precision. DTrue_ref refers to the
true reference delay, and it is the difference between Tr1 and Ts1,
which can not be directly measured. DTrue_target denotes the true target
delay. They follow equation 2 and 3 respectively. Equation 2: DTrue_ref = Tr1 - Ts1Equation 3: DTrue_target = Tr2 - Ts2Now we can get a relationship between the reference delay and the
target delay by equation 4.Equation 4: DTrue_target = (Tr2 - Tr1) + DTrue_ref - (Ts2 - Ts1) Here we follow the clock model proposed by to formulate the time variation of clocks at
sender and receiver. The clock model states that for a TSN-grade clock,
its local time Ci(t) always follows equation 5.Equation 5: (Ci(T2) - Ci(T1) - eta) * (1/rho) <=T2 - T1 <=
(Ci(T2) - Ci(T1))*rho + eta (T2 >= T1)In equation 5, rho refers to the time stability bound, i.e. 1.0001
for TSN-grade clock, and eta is the timing jitter bound, i.e. 2ns for
TSN-grade clock. The model can be adopted for analyzing behaviors of
clocks at the sender and receiver, because they are the both ends of a
deterministic network path and their clocks are TSN-grade. In this way,
inequality 6 for sender clock Cs(t) and inequality 7 for receiver clock
Cr(t) are formulated below. Equation 6: (Cs(Ts2) - Cs(Ts1) - eta) * (1/rho) <=Ts2 - Ts1 <=
(Cs(Ts2) - Cs(Ts1))*rho + eta (Ts2 >= Ts1)Equation 7: (Cr(Tr2) - Cr(Tr1) - eta) * (1/rho) <=Tr2 - Tr1 <=
(Cr(Tr2) - Cr(Tr1))*rho + eta (Tr2 >= Tr1)Now, equation 4 can be extended with known values to express its
upper and lower bound. Upper bound is shown in equalition 8 and lower
bound is shown in equation 9.Equation 8: DTrue_target <= Cr(Tr2) - Cr(Tr1) - Cs(Ts2) + Cs(Ts1)
+ L + eta * (1+(1/rho)) + (Cr(Tr2)-Cr(Tr1)+Cs(Ts2)-Cs(Ts1))(rho - 1)Equation 9: DTrue_target >= Cr(Tr2) - Cr(Tr1) - Cs(Ts2) + Cs(Ts1)
+ L - J - eta * (1+(1/rho)) - (Cr(Tr2)-Cr(Tr1)+Cs(Ts2)-Cs(Ts1))(rho -
1)Accordingly, a point estimate of DTrue_target is expressed by
equation 10, and its corresponding inaccuracy is shown by equation
11.Equation 10: DEst_target = Cr(Tr2) - Cr(Tr1) - Cs(Ts2) + Cs(Ts1) + L
- J/2Equation 11: delta DEst_target = J/2 + eta * (1+(1/rho)) +
(Cr(Tr2)-Cr(Tr1)+Cs(Ts2)-Cs(Ts1))(rho - 1)The derivation above can be used as a theoretical proof for the
one-way delay measurement approach based on the characteristics of
reference delay within deterministic network. The measurement steps are shown in figure 3, which describe the
measurement steps at the sender side and receiver side respectively. For
the sender side, a reference packet is sent. In the first step, the
sender gets ready to send a reference packet; in the second step, the
sender marks an egress timestamp Cs(Ts1) for the reference packet; in
the third step, the sender encapsulates the egress timestamp of the
reference packet in the measurement header of the reference packet; in
the fourth step, the sender sends the reference packet. For the target
packet, the sender side procedures are the same,we omit it for
simplicity. The sending time of the target packet is according to the
traffic model of real applications. Reference packets are sent for many
times at first, in order to get accurate bounds of reference delay,
until which the target packet can not be sent for measurement. For the reference packet, the processing steps at the receiver are
shown in figure 3. In the first step, the reference packet arrives at
the receiver, and the receiver receives the reference packet; in the
second step, the receiver timestamps the reference packet at the
entrance, which is denoted as Cr(Tr1); in the third step, the receiver
decapsulates the measurement header of the reference packet to obtain
the sender side timestamp Cs(Ts1); in the fourth step, the receiver
records the timestamp information of Cs(Ts1) and Cr(Tr1); in the fifth
step, the receiver uses the source/destination pair obtained by
decapsulation in the third step as the search key, queries the reference
delay table and records the reference delay search result, upper bound L
and peak-to-peak value J.For the target packet, the processing steps at the receiver are also
shown in figure 3. In the first step, the target packet arrives at the
receiver, and the receiver receives the target packet; in the second
step, the receiver timestamps the target packet at the entrance, which
is denoted as Cr(Tr2); in the third step, the receiver decapsulates the
measurement header of the target packet to obtain the sender side
timestamp Cs(Ts2); in the fourth step, the receiver records the
timestamp information of Cs(Ts2) and Cr(Tr2); in the fifth step, the
receiver calculates the target one-way delay, which we want to measure,
according to the recorded timestamp information Cs(Ts1), Cs(Ts2),
Cr(Tr1), Cr(Tr2) and reference delay with its upper bound and
peak-to-peak jitter. The upper and lower bound of target one-way delay
can be caculated by equation 8 and 9, and at the meantime, a rough
exstimation can be made by using equation 10. The sender encapsulates the timestamp information and sender-receiver
pair information in the measurement header of the sent packet, as shown
in figure 4. The position of measurement header is in the option field
of the TCP protocol header. The delay measurement option format is
defined in figure 5. The Length value is 8 octets, which is in
accordance with TCP option. The sender ID is one octet, and the receiver
ID is also one octet. The sender side timestamp is 4 octets, which can
store accurate timestamp information.The end-to-end one-way delay includes three parts, namely the
transmission delay, the internal processing delay of the network
devices, and the internal queueing delay of the network devices. Among
them, fixed parts of the delay include transmission delay and internal
processing delay. The transmission delay is related to transmission
distance and transmission media. For example, in optical fiber, it is
about 5ns per meter. With transmission path and media determined, it is
basically a fixed value. The internal processing delay of a network
device includes processing delay of the device's internal pipeline or
processor and serial-to-parallel conversion delay of the interface,
which is related to in/out port rate of the device, message length and
forwarding behavior. The magnitude of the internal processing delay is
at microsecond level, and it is basically a fixed value related to the
chip design specifications of a particular network device. Variable part
of the delay is the internal queueing delay. The queueing delay of the
device internal buffer is related to the queue depth, queue scheduling
algorithm, message priority and message length. For each device along
the end-to-end path, the queueing delay can reach microsecond or even
millisecond level, depending on values of the above parameters and
network congestion state.With the continuous development of networking technologies and
application requirements, a series of new network technologies have
emerged which can guarantee bounded end-to-end delay and ultra small
jitter. For example, deterministic network, by
leveraging novel scheduling algorithms and packet priority settings, can
stabilize queuing delay of network device on the end-to-end path. As a
result, the end-to-end one-way delay is extremely low and bounded. So
packets transmitted by a deterministic network with delay guarantee can
be used as reference packets, and their end-to-end one-way delay can be
used as reference delays. The acquisition method of reference delay is
not limited to the above method based on deterministic network
technology.TBD.This document requests IANA to assign a Kind Number in TCP Option to
indicate TCP Delay Measurement option.IEEE Standard for a Precision Clock Synchronization Protocol
for Networked Measurement and Control SystemsIEEE On Time Synchronization Issues in Time-Sensitive Networks
with Regulators and Nonideal ClocksL. Thomas and J.-Y. Le BoudecDetNet Bounded LatencyN. Finn, J-Y. Le Boudec, E. Mohammadpour, J. Zhang,
B. Varga