Benchmarking Methodology WG Minutes WG Chair: Kevin Dubray Minutes reported by Kevin Dubray. The BMWG met at the 46th IETF in Washington, D.C., on Wednesday, November 10, 1999. There were over 20 attendees. The proposed agenda follows: 1. Administrivia 1.1 Status of . 2. Methodology for ATM Benchmarking. (Jeff Dunn/Cynthia Martin) http://www.ietf.org/internet-drafts/draft-ietf-bmwg-atm-method-00.txt 3. Terminology for Frame Relay Benchmarking. (Jeff Dunn/Cynthia Martin) http://www.ietf.org/internet-drafts/draft-ietf-bmwg-fr-01.txt 4. Benchmarking Methodology for LAN Switching Devices (Jerry Perser) http://www.netcomsystems.com/perser/draft-ietf-bmwg-mswitch-01a.txt The agenda was left unmodified. 1. Administrivia. Draft-ietf-bwmg-atm-term-00.txt was announced to have gone through a WG Last Call and was in the hands of the IESG. [It has since been forwarded from the IESG to the RFC Editor.] Jeff Dunn was then introduced to lead a discussion regarding the first two agenda items. 2. Terminology for Frame Relay Benchmarking. Jeff began by stating that an ATM ABR Terminology draft was is in progress and a first draft would be submitted shortly after the meeting. Jeff stated that he would start the presentations slightly out of order from the agenda by tackling the Frame Relay (FR) terminology draft first. Jeff's presentation followed his slides (submitted with these minutes). The presentation communicated the motivation, focus, and anticipated milestones of this effort. Jeff reiterated that the goal of the work was to identify metrics that would characterize IP or higher layer performance over a Frame Relay infrastructure. Said another way, produce metrics that would help characterize the effects of impairments in the lower layers on the upper layers' performance. In general, the authors feel that the draft needs focus and clarification - this is where they turn to the WG for help. A question came up on congestion. Was the focus on DUT congestion or congestion external to the DUT? Jeff indicated this was exactly the type of insight that the authors needed from the WG. Cynthia Martin indicated another area of focus is whether to characterize single Virtual Circuit (VC) performance versus multiple VC performance. There was some discussion about the notion of "spectral" coverage - the implication was the testing was "real world" versus clinical. It was communicated that the IPPM handled more "real world" network characterizations, albeit from a more generic perspective. Jeff indicated that he believe the notion of spectral was associated more to a DLCI than time. Discard Eligibility (DE) was a Frame Relay term that might be expanded or more closely bound to the "upper layers". Additionally, it may be necessary to identify new terms to help draw out the appropriate associations. David Newman offered to make some proposals and send to the WG mailing list. Newman also thinks that the draft will need to present the necessary context, where appropriate. The notion of VCs and their associated performance may be differ if taken from a vendor's perspective than from a provider's perspective. It was thought that DS-3 channels should be given the same level of consideration in the draft as T1/E1. In general, Jeff indicated that he ignored things plesiochronous. He stated that they might have to reconsider this - especially with regards to alarm conditions. Jeff commented that it was the authors intent to get the drafts solidified by its third revision. He reiterated that the authors could only do that with WG participation. With that Jeff opened the floor for general comments. A suggestion was offered that the draft would make more sense if the terminology were more logically group as opposed to being presented alphabetically. It was also thought that a Table of Contents would be very useful. It was noted that the BECN definition needed some work. It was also stated that the FCS section might give consideration to 32 bit checksums, not just 16. It was also thought that while 2 byte headers are most common, shouldn't mention be made of 3 and 4 byte headers? Cynthia said due to methodological complications, they might like to strictly focus on the common application, documenting that other header sizes are outside the scope of this document. It was asked whether the draft should address LMI version issues as they may affect link formation. Jeff asked the whether the draft should address LMI at all? If so, are we looking to come up with related benchmarks? Right now, he considers it a more methodological issue which would be outside the scope of the draft. Jeff thanked the group for input on this draft and asked that the discussion spill over to the mailing list. He then turned what time he had remaining to the ATM Benchmarking Methodology document. 3. Methodology for ATM Benchmarking. In a similar manner as the FR presentation, Jeff presentated the motivation, focus, and anticipated milestones of the ATM Benchmarking Methodology work. The bulk of the discussion departed from addressing the ATM methodology document and centered around the futures slide. Brought up were such things as Packet Over Sonet and Routing Protocol Benchmarking. The chair asked folks to move the discussion of possible BMWG work to the mailing list. 4. Benchmarking Methodology for LAN Switching Devices Jerry Perser was next introduced to discuss the LAN Switch Methodology draft. Jerry's presentation addressed changes made to the draft, its improved organization, as well as some outstanding discussion points. Much of the discussion, however, revolved around congestion related issues. Was Head of Line Blocking (HOL) a condition or a test? Perser stated that it was a test. There was confusion over the HOL and the Backpressure indicators as both employed the same methodology. Additionally, it was offered that it wasn't valid to use the same configuration for both indicators. At the end of the discussion, some recommendations were offered: - Address HOL and Backpressure under the common heading of "Congestion Control." - Offer insight as to why HOL and backpressure are related methodologically. There was discussion as to whether it was appropriate to declare a test successfully executed versus passing judgement over a successfully executed test's results. Most thought it was a good thing to identify guidelines that determine whether the results were properly collected. A question was made as to the appropriateness of psuedo-code in a methodological statement. Some thought it wasn't OK; others thought it was fine. A suggestion was to describe the methodology using prose but offer the pseudo-code in an Appendix. Jerry indicated that he wasn't happy with the wording of Intended Load as a percentage. He offered a solution. (See slides.) He'll present the solution to the list. It was stated that we wanted to move this draft towards last call, so folks should take issues directly to WG mailing list as soon as possible. In other work, Dubray noted that the Multicast Methodology was revised since the last meeting. He noted that one of the other editors (R. Daniels) did not make his additions to the draft in time for discussion at this meeting. That notwithstanding, Dubray asked if anyone had outstanding issues with the draft. Some comments offered were: - The Mixed Class Throughput section indicates that the DUT must be set up in one of two ways. It wasn't clear as to the difference between the two approaches. Also, this methodology seems very LAN centric. Could it be made more generic? - The reference to the 20 nanosecond granularity was questioned. Was it really needed? - Forwarding Burdened Group Join Delay and Forwarding Burdened Group Latency weren't clear. - Appendix A offers many examples, but it states no algorithm. With time running out, folks were requested to mail remaining comments to the WG reflector.