Multicast On-path Telemetry using IOAMFuturewei Technologies2330 Central ExpresswaySanta ClaraUSAhsong@futurewei.comFuturewei Technologies2330 Central ExpresswaySanta ClaraUSAmmcbride@futurewei.comZTE Corp.gregimirsky@gmail.comVerizon Inc.gyan.s.mishra@verizon.comNational Institute of Information and Communications Technology4-2-1 Nukui-KitamachiKoganeiTokyo184-8795Japanasaeda@nict.go.jpHuawei TechnologiesBeijingChinazhoutianran@huawei.com
OPS
MBONEDMulticast, TelemetryThis document discusses the requirements of on-path telemetry for multicast traffic using In-situ OAM.
Applying In-situ OAM for multicast telemetry presents some unique challenges. This document provides the solutions
based on the In-situ OAM trace option and direct export option to support the telemetry data correlation and
the multicast tree reconstruction without collecting redundant data.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 when, and only when, they appear in all
capitals, as shown here.Multicast traffic is used across operator networks to support residential broadband customers,
private MPLS customers and used with corporate intranet internal customers. Multicast provides real time interactive
online meetings or podcasts, IPTV and financial markets real-time data, which all have a reliance on UDP's unreliable
transport. End to end QOS, therefore, should be a critical component of multicast deployment in order to provide a good
end user viewing experience. If a packet is dropped, and that packet happens to be a reference frame (I-Frame)
in the video feed, the client receiver of the multicast feed goes into buffering mode resulting in a frozen window.
Multicast packet drops and delay can severely affect the application performance and user experience.It is important to monitor the performance of the multicast traffic. New on-path telemetry techniques such as
In-situ OAM (IOAM),
IOAM Direct Export (DEX)IOAM Marking-based Postcard (MBP), and
Hybrid Two-Step (HTS)
are useful and complementary to the existing active OAM performance monitoring methods, provide promising means to directly monitor the network experience of multicast traffic. However, multicast traffic has some unique characteristics which pose some challenges on efficiently applying such techniques. The IP Multicast (S,G) data is identical from one branch to another on its way to multiple receivers. When adding IOAM
trace data to multicast packets, redundant data will be collected for the same original multicast packet. This is because each replicated packet keeps the telemetry data for its entire forwarding path and the replicated packets all share some common path segments.
Such redundancy consumes more network bandwidth unnecessarily. Alternatively, it could be more efficient to collect the telemetry data using solutions such as IOAM DEX to eliminate the data redundancy. However, the current postcard-based direct export solution does not have a branch identifier, making telemetry data correlation and multicast-tree reconstruction difficult.This draft proposes a set of solutions to the IOAM data redundancy problem. The requirements for multicast traffic telemetry
are discussed along with the issues of the existing on-path telemetry techniques. We propose modifications
to make these techniques adapt to multicast in order for the original multicast tree to be correctly reconstructed while
eliminating redundant data. Multicast traffic is forwarded through a multicast tree. With PIM and P2MP (MLDP, RSVP-TE) the forwarding
tree is established and maintained by the multicast routing protocol. With BIER, no state is created in the network
to establish a forwarding tree; instead, a bier header provides the necessary information for each packet to know
the egress points. Multicast packets are only replicated at each tree branch node for efficiency. There are several requirements for multicast traffic telemetry, a few of which are: Reconstruct and visualize the multicast tree through data plane monitoring. Gather the multicast packet delay and jitter performance. Find the multicast packet drop location and reason. Gather the VPN state and tunnel information in case of P2MP multicast. In order to meet these requirements, we need the ability to directly monitor the multicast traffic and derive data from the multicast packets. The conventional OAM mechanisms, such as multicast ping and trace, are not sufficient to meet these requirements. On-path Telemetry techniques that directly retrieve data from multicast traffic's live network experience are ideal to
address the above mentioned requirements. The representative techniques include
In-situ OAM (IOAM) Trace option,
IOAM Direct Export (DEX) option, and
IOAM Marking-based Postcard (MBP). However,
unlike unicast, multicast poses some unique challenges to applying these techniques. Multicast packets are replicated at each branch node in the corresponding multicast tree. Therefore, there are
multiple copies of the original multicast packet in the network. If the IOAM trace option is used for on-path data collection, the partial trace data will also be replicated into the copy for each branch.
The end result is that, at the multicast tree leaves, each copy of the multicast packet has a complete trace. Most of the data, however, is redundant. Data redundancy introduces unnecessary header overhead, wastes network bandwidth, and complicates the data processing.
In case the multicast tree is large, and the path is long, the redundancy problem becomes severe. The postcard-based solutions, including the IOAM DEX and MBP, can be used to eliminate such data redundancy, because each
node on the tree only sends a postcard covering local data. However, they cannot track and correlate the tree branches properly so it can bring confusion about the multicast tree topology. For example, in a multicast tree, Node A has two branches, one to Node B and the other to node C, and Node B leads to Node D and Node C leads to Node E.
From the received postcards, one cannot tell whether or not Node D(E) is the next hop of Node B(C). The fundamental reason for this problem is that there is not an identifier (either implicit or explicit) to correlate the
data on each branch. We provide two solutions to address the above issues. One is based on IOAM DEX and requires an extension to the instruction header of the IOAM DEX Option. The second solution combines the IOAM trace option and postcards for redundancy removal.One way to mitigate the postcard-based telemetry's tree tracking weakness is to augment it with a branch identifier field. Note that this works for
the IOAM DEX option but not for MBP because the IOAM DEX option has an instruction header which can be used to hold
the branch identifier. To make the branch identifier
globally unique, the branch node ID plus an index is used. For example, Node A has two branches, one to Node B and the other to
Node C. Node A will use [A, 0] as the branch identifier for the branch to B, and [A, 1] for the branch to C. The identifier is unchanged for each
multicast tree instance and carried with the multicast packet until the next branch fork node. Each node MUST export the branch identifier in the received IOAM DEX header in the postcards it sends.
The branch identifier, along with the other fields such as flow ID and sequence number, is sufficient for the data collector to
reconstruct the topology of the multicast tree. shows an example of this solution. "P" stands for the postcard packet. The square brackets contains the branch identifier. The curly brace contains the telemetry data about a specific node. Each branch fork node needs to generate a unique branch identifier (i.e., branch ID) for each branch in its multicast tree instance and include
it in the IOAM DEX option header. The branch ID remains unchanged until the next branch fork node. The branch ID contains two parts:
the branch fork node ID and an interface index. Conforming to the node ID specification in IOAM, the node ID is a 3-octet unsigned integer.
The interface index is a two-octet unsigned integer. As shown in , the branch ID consumes 8 octets. The three unused octets must be set to 0. shows that the branch ID is carried as an optional field after the flow ID and sequence number optional fields
in the IOAM DEX option header. A bit "M" (The third bit in the Extension-Flags field) is reserved to indicate the presence of
the optional branch ID field. If "M" is set to 1, the optional multicast branch ID field is present; otherwise it is absent. Once a node gets the branch ID information from the upstream, it MUST carry this information in its
telemetry data export postcards, so the original multicast tree can be correctly reconstructed based on the postcards. The second solution is a combination of the IOAM trace option and the postcard-based telemetry.
To avoid data redundancy at each branch node, the trace data accumulated, to that point, is exported
by a postcard before the packet is replicated. In this case, each branch still needs to maintain some identifier to help correlate the postcards
for each tree section. The natural way to accomplish this is to simply carry the branch node's data (including its ID) in the trace of each branch.
This is also necessary because each replicated multicast packet can have different telemetry data pertaining to this particular copy (e.g., node
delay, egress timestamp, and egress interface). As a consequence, the local data exported by each branch node can only contain partial
data (e.g., ingress interface and ingress timestamp). shows an example in a segment of a multicast tree. Node B and D are two branch nodes and they will export
a postcard covering the trace data for the previous section. The end node of each path will also need to export the data of the last section as a
postcard.There is no need to modify the IOAM trace option header format as specified in . We just need to configure the branch nodes to export the postcards and refresh the IOAM header and data (e.g., clear the node data list and reset the Remaining Length filed).Mtrace version 2 (Mtrace2) is a protocol that allows the tracing of an IP multicast routing path. Mtrace2 provides additional information such as the packet rates and losses, as well as other diagnostic information. Unlike unicast traceroute, Mtrace2 traces the path that a packet would take from a source to a receiver. It is usually initiated from an Mtrace2 client by sending an Mtrace2 Query to a Last-Hop Router (LHR) or to a Rendezvous Point (RP). The LHR/RP turns the Query packet into an Mtrace2 Request, appends a Standard Response Block containing its interface addresses and packet statistics to the Request packet, and forwards the packet towards the source/RP. In a similar fashion, each router along the path to the source/RP appends a Standard Response Block to the end of the Request packet and forwards it to its upstream router. When the First-Hop Router (FHR) receives the Request packet, it appends its own Standard Response Block, turns the Request packet into a Reply, and unicasts the Reply back to the Mtrace2 client.New on-path telemetry techniques will enhance Mtrace2, and other existing OAM solutions, with more granular and realtime network status data through direct measurements. There are various multicast protocols that are used to forward the multicast data. Each will require their own unique on-path telemetry solution.PIM-SM is the most widely used multicast routing protocol deployed today. Of the various PIM modes (PIM-SM,
PIM-DM, BIDIR-PIM, PIM-SSM), PIM-SSM is the preferred method due to its simplicity and removal of network source discovery complexity. With all
PIM modes, control plane state is established in the network in order to forward multicast UDP data packets. All PIM modes utilize network based
source discovery except for PIM-SSM, which utilizes application based source discovery. IP Multicast packets fall within the range of 224.0.0.0
through 239.255.255.255. The telemetry solution will need to work within this address range and provide telemetry data for this UDP traffic. A proposed solution for encapsulating the telemetry instruction header and metadata in IPv6 packets is described in
. Multipoint Label Distribution Protocol (mLDP), P2MP RSVP-TE, Ingress Replication (IR), PIM MDT SAFI with GRE Transport, are commonly
used within a Multicast VPN (MVPN) environment utilizing MVPN procedures Multicast in MPLS/BGP IP VPNs
and BGP Encoding and Procedures for Multicast in MPLS/BGP IP VPNs. MLDP LDP
Extension for P2MP and MP2MP LSPs provides extensions to LDP to establish point-to-multipoint (P2MP) and multipoint-to-multipoint
(MP2MP) label switched paths (LSPs) in MPLS networks. P2MP RSVP-TE provides extensions to RSVP-TE RSVP-TE
for P2MP LSPs for establish traffic-engineered P2MP LSPs in MPLS networks. Ingress Replication (IR) P2MP
Trees Ingress Replication Tunnels in Multicast VPNs using unicast replication from parent node to child node
over MPLS Unicast Tunnel. PIM MDT SAFI Multicast in BGP/MPLS IP VPNsutilizes PIM modes PIM-SSM,
PIM-SM, PIM-BIDIR control plane with GRE transport data plane in the core for X-PMSI P-Tree using MVPN procedures.
Replication SID SR Replication Segment for Multi-point Service Delivery
replication segments for P2MP multicast service delivery in Segment Routing SR-MPLS networks. The telemetry solution will need to be able to
follow these P2MP and MP2MP paths. The telemetry instruction header and data should be encapsulated into MPLS packets on P2MP and
MP2MP paths. A corresponding proposal is described in . BIER adds a new header to multicast packets and allows the multicast packets to be forwarded
according to the header only. By eliminating the requirement of maintaining per multicast group state, BIER is more scalable than the
traditional multicast solutions. OAM Requirements for BIER lists many of the requirements for OAM at the
BIER layer which will help in the forming of on-path telemetry requirements as well.
There is also current work to provide solutions for BIER forwarding in ipv6 networks. For instance, a solution,
BIER in Non-MPLS IPv6 Networks, proposes a new bier Option Type codepoint
from the "Destination Options and Hop-by-Hop Options" IPv6 sub-registry. This is similar to what IOAM proposes for IPv6 transport.Depending on how the BIER header is encapsulated into packets with different transport protocols, the method to encapsulate the
telemetry instruction header and metadata also varies. It is also possible to make the instruction header and metadata a part of the BIER
header itself, such as in a TLV.No new security issues are identified other than those discovered by the IOAM trace and DEX options.The document requests a new extension flag registration in the "IOAM DEX Extension-Flags" registry, as described in Section 4.1.Bit 2 "Multicast Branch ID [RFC XXXX] [RFC Editor: please replace with the RFC
number of the current document]". The authors would like to thank Frank Brockners, Nils Warnke, Jake Holland, and Dino Farinacci for the comments and suggestions.