INTERIM MEETING REPORT Minutes of an Interim Meeting on IP Provider Metrics Benchmark Methodology Working Group (BMWG) Pittsburgh Supercomputer Center 12 September 1995 Reported by E. Paul Love. Guy Almes opened the meeting with a brief background of the combination of IPPM within the BMWG. This was the preference of the Area Directors. The co-chairs of BMWG are Guy Almes and Jim McQuaid. Agenda for the afternoon Review of IPPM to date Quick run through of the Stockholm slide presentation Settings of priorities & assignments Technical presentations by Vern Paxson - LBL Matt Mathis - PSC IPPM to date Pre-Danvers. Initial work in IRTF's End to End - Winter 94 through Winter 95. BOF chaired by Matt Mathis at Danvers Stockholm meeting has about 2 dozen participants Mailing list now has about 300 This interim meeting On schedule for meeting at Dallas Stockholm slides (also in Stockholm Proceedings and via ftp at [ftp://ftp.advanced.org/pub/IPPM/Jul-95/IPPM-Slides.ps]) Context Growing complexity of the Internet exponential Criteria of IP Provider Metrics Concrete & repeatable Useful to both users & providers Avoid inducing artificial performance goals No bias for equal technologies Fair bias for non-equal technologies Hierarchy of Metrics Vague, gut level concept - e.g. delay Precise, but possibly not a measurable concept - e.g. delay of 3000 octet packet between 2 points Precise & measurable An Initial Set of Needed Metrics Path Performance Delay Expected flow capacity for a single application Expected aggregated flow capacity Reliability Delay - packet loss as they enable flow control Delay - jitter as it enables flow control - multicast Routing Stability/robustness End to end Stability Quickness of recovery Coverage CIDR effectiveness DNS Performance Security of Infrastructure NOC Responsiveness Existing Tools Trace route Routing logs Ping ttcp Traffic logs Needed New Tools Capacity tools that don't swamp production traffic Long distance AS-path analysis tools Better instrumentation of TCP Relation to Emerging Infrastructure Potential use of NAP's as locus for placing measurement tools Benefits To users - when to pressure providers To providers - knowing how to better communicate to users, vendors & competition Administrative Issues IPPM is within BMWG for now Meetings at IETF's Between IETF's as needed Separate WG perhaps later Typical Agenda Evolve consensus of metrics Share results on tools that measure them Share results on applications of these tools Keep community mindful of best practices Needed Development General - Criteria for metrics Pre-metric developments Draft metric definitions Experience ID's An experience with tool development An experience with tool application [floor comment] - Do we partition routing from path (flow & capacity) metrics? Make sure metric definitions are same across platforms, even if tools are different (or just similar) High Priority Metrics IP packet delay across paths - mean & variance Flow capacity across paths Single high speed flow Aggregation of flows Floor Comments Who should run what? Joe user? Just providers on themselves? What is the path to be measured? Definitions of terms (rfc1242 can serve as a start) Users don't care about packets/sec while providers may well New things to test Stockholm/Hawaii T3 delay-bandwidth product Can a provider reach all portions of the global Internet? A new tool for "Coverage." How to publish a "Consumer's Reports for the Internet"? Collusion from providers may be needed for routing, though not for paths. More metrics to consider Packet loss Availability of service - up time Access - probability of a busy signal for dial in customers Bits/sec - raw, total vs. capacity above an average, background load Means of measuring need to increase a provider's infrastructure - aggregation, muxing, etc. Costs with performance metrics. Also, "good enough", cheapest, etc. Presentations Matt Mattis - Top Down vs. Bottom Up [ftp://ftp.advanced.org/pub/IPPM/Sep- 95/TrenoTal.ps] Vern Paxson - Endpoint Measurement of Network Condition [ftp://ftp.advanced.org/pub/IPPM/Sep-95/paxson- slides.ps]