Editor's note: These minutes have not been edited. Minutes of the Benchmarking Methodology Workgroup Reported by Kevin Dubray Over 50 people signed the attendance list for this meeting. This was an abbreviated meeting of the BMWG due to the extended introductory IETF general meeting. 1. Agenda. Kevin Dubray opened the meeting and offered the proposed agenda for this session of the BMWG. The agenda was accepted as follows: - Discuss recent modifications to the LAN Switch terminology draft. Resolve any outstanding issues so that the draft can be forwarded for consideration as an informational RFC. - Introduce the new cell/call benchmarking terminology draft. Ascertain if direction is correct. Provide the editor preliminary feedback on initial drafting of terms. - Introduce multicast benchmarking terminology draft. - Review and update BMWG milestones. 2. LAN Switch terminology draft. Dubray turned the floor over to the draft's editor, Bob Mandeville. Mandeville led the discussion on the latest version of the switch terminology draft, . Bob began the discussion by addressing areas of the text for which he had received feedback. Bob indicated that the most vociferous comments on the draft addressed the definition of "forward pressure". Concern was expressed that the concept, as presented, seemed to advocate this type of behavior in order to mitigate frame loss in the face of congestion, specifically collisions. It was feared that, by extension, some interpretations of this terminology may lead one to conclude that adherence to a networking standard was optional. Bob countered that was not his intent; rather, he was just trying to define a behavior. Doug Ruby questioned whether forward pressure was actually measurable as defined. Mandeville indicated that his experience was that forward pressure was, indeed, measurable. He further thought the concept was useful in drawing distinction between the behaviors of various switches. A question was raised as to whether the pertinent components of the IEEE 802.3 standard should be cited. Scott Bradner went on record to say that he did not think so. He believed that it was a good thing to keep a concept as generic as possible in order to retain the ability to use that concept in other scenarios. The group reached consensus in suggesting that the current wording of "forward pressure" reflect that not adhering to an approved Standard is not an acceptable practice. Another term for which Bob received feedback was "multidirectional traffic." Many thought that the term's name was not straightforward. Scott Bradner commented that he has used the term "meshed" to refer to the traffic orientation suggested by multidirectional. The group agreed that "meshed" should be used in lieu of term, "multidirectional." Furthermore, Bradner pointed out, the concept of "fully meshed" should be dropped from the definition component. Again, the group agreed. Some confusion was expressed with regards to the term "burst" - specifically, references to the notion of a "single frame burst." Mandeville replied that it is useful to describe a "packet in isolation." He further commented that addressing burst just extends the discussions presented in RFC 1242 and RFC 1944. A discussion of "backpressure" followed. Doug Ruby articulated that "jamming" was just one technique of backpressure. There are other techniques both active and passive. An example of active backpressure was the PAUSE flow control technique offered by IEEE 802.3x draft. Further, Doug was not sure of the utility of the throughput metric in active backpressure scenarios. Doug advocated that test gear needs to responded to active backpressure scenarios. Scott warned that it was his experience that different test gear did not necessarily respond consistently to same stimulus. Scott further noted that reliance on any one test device may be problematic. Doug countered that unless the offered terminology reflects consideration for full duplex, flow control, etc., test gear vendors will not build supporting tool sets. Dubray seconded this thought, offering that it appeared that tool vendors sometimes consider BMWG work as functional specifications for their products. Bradner asked Ruby if Doug was offering to supply wording to convey the ideas that he was advocating. Both Ruby and Mandeville agreed to work towards this end. On another topic, Mandeville raised the idea that it might be appropriate to move some of terminology that Dubray has offered in his initial Multicast Terminology draft into the current LAN Switch draft. Bob indicated that Kevin was not against moving the generic terminology into the switch draft. Scott commented that generic concepts didn't really fit into a multicast specific discussion, anyway. Kevin pointed out that his goal was not to introduce a battery of unrelated terminology. Rather, in the course of discussing multicast related terminology, it became evident that concepts alluded to in previous BMWG work were either undefined or implied. Kevin didn't particularly care where the items were concretely defined per se; however, inclusion in the multicast draft seemed most expedient and helped the multicast discussion. He further went on to say that the tactic wasn't unprecedented, citing the previously described term of "meshed". "Meshed" traffic went beyond the context of switch testing, but it was being defined in the switch draft. Someone offered that the power of the BMWG effort did not rest in any one document, but the collection of documents taken as a whole. The group agreed. Some of the terms from the Multicast Terminology Draft that Bob thought would have use in the LAN Switch draft were Device Under Test (DUT) and System Under Test (SUT). Scott reflected that DUT was fast becoming an uninteresting term. Dubray countered that the notion of a single device interacting only with the test tools was important. Moreover, it was a simple concept from which other relationships could be built. Again, the group agreed. Bob furthered the discussion on the LAN switch draft by contrasting how the LAN switch draft defined the concepts of "nominal load" and "real load" while the Multicast Draft conveyed the notion of "target rate" and "offered rate". A question from the floor queried why the need for distinction a between nominal/target versus real/offered. Kevin offered that it was his experience that in the course of testing, there were occasions when the test tool may be unable to generate frames at the requested rate. The reasons for this may be DUT related (e.g., collisions as a function of flooding conditions) or tool related (e.g. the tool unable to utilize the maximum usable bandwidth of medium). It is useful, then, to draw distinction between these two datapoints. After some small discussion the group agreed to use the terms "intended load" and "offered load". A short discussion followed on the notion of "speed" as defined in the Switch draft versus "forwarding rate" as presented the Multicast draft. Bob stated that speed only considered the forwarding ability of at the points of egress from the tested system. Bob continued that the importance of "speed" was in the metric's ability to describe a system's behavior in the face of congestion control. Dubray offered that the forwarding rate described the egress component as a function of the ingress component. In other words, forwarding rate is the observed rate at which a tested device forwards frames for a specific offered load. A question from the floor pondered whether "maximum speed" and "maximum forwarding rate" were equivalent. Dubray thought not. Maximum speed conveyed the maximum number of frames a DUT could forward in a fixed time frame (a second), without consideration for the offered load. This differs from the presented definition of maximum forwarding rate, which described the DUT's ability to forward frames in response to the maximum, tested offered load. Many thought this not to be intuitive. Scott echoed that most people seem to consider "maximum frame rate" to be the "largest" forwarding rate taken from a set of forwarding rate measurements. Dubray agreed the title of the term was not intuitive as presented. However, he continued, it was his experience that comparisons between the "largest" forwarding rate and the forwarding rate measured at the highest offered load gave significant insights into a device's behavior. Scott concurred and suggested the benchmark be given the name "forwarding rate at maximum offered load." The group thought this to be very intuitive. Dubray stated that this was consistent with email that he had received on the topic. Dubray interrupted the discussion due to time constraints and said that he and Bob would discuss further consolidation of terms off-line. In assessing the overall shape of the LAN Switch draft, the group concluded that with the discussed enhancements and other minor modifications folded into the draft, there were no major blocking factors to the document's advancement. Bob indicated that he would issue a revised draft as soon as possible. 3. Cell/Call Terminology Draft. Dubray announced that the draft's author, Robert Craig, was unable to attend the BMWG session. In his place, Bob Mandeville had cheerfully volunteered to lead the discussion on this initial draft. Bob immediately began a rigorous discussion of the draft, starting with section 3.1, Virtual Circuits. Questions were raised as to the scope of the call specific items, such as Call Setup Time and Call Setup Rate. Were these benchmarks end-to-end measurements or network element-to-network element measurements? Scott chimed in that one of the questions that he is often asked to answer is how to qualify the overhead of a tested system in that system's delivery of a requested function. Scott communicated that defining a generic system-level response benchmark may have merit. Such a benchmark would have utility, whether it was used in measuring call setup up time or an HTTP transfer. Some questioned whether such a generic benchmark would capture the specific nature of the transaction being measured. More pointedly, another person asked about the nature of the question that the draft was trying to answer. Dubray halted the discussion to focus on the group on item 2 of the agenda: ascertain whether the direction of the current draft was correct with respect to the previously agreed upon workplan. That workplan, Dubray reminded, was to develop a set of metrics and terminology that helped characterize a system of devices that forwarded frames from "legacy" based technologies via a cell-based/connection-oriented infrastructure. Dubray added that he thought the scope of this effort was to provide the terminology surrounding a test scenario not unlike that conveyed in section 19 of RFC 1944. The focus of the exercise was not to concentrate on a particular network element, but rather a system of devices comprised, perhaps, of edge devices and switches. The group thought that in its current form the draft was slightly off-mark by focusing on specific network elements. However, there was an overwhelming agreement to continue the draft in its current direction. The discussion that followed was targeted in giving the draft's editor general feedback. The most prevalent comments on the draft reflected the fact that the draft introduced terms that may be outside the scope of benchmarks and related terminology. For example, terms like "Buffering Strategy" or "Queuing Strategy," maybe outside the intended scope of the draft as they may be more architectural in nature. Dubray noted that email by Jim McQuaid on the BMWG mailing list reflected this fact and Robert agreed to drop terms of this type. On the proposed term "non-blocking factor," most thought that the concept was interesting, but they also thought the formal definition was weak and needed clarification. Mike O'Dell thought it appropriate for the draft to consider the notion of "blocking probability." With regards to section 3.4 (Buffering), someone recalled that there was some work done in this area by the ATM Forum, and that it may be prudent to survey that work. On the topic of forwarding measurements, Mike O'Dell thought it would be helpful to introduce some idea of "load dependencies". Mike went on to say that he thought it would be very useful to determine how invasive are the changes to the forwarding path on forwarding performance. In response to item 3.2.1 and 3.2.2 (Packet disassembly/reassembly time, peak and sustained, respectively), some conveyed that it would be impossible to measure the SAR function explicitly as a black box style test. The group generally thought section 3.7.1, traffic management, needed reworking. It wasn't clear what was being tested, policing or admission control. Scott Bradner offered a suggestion that the editor contact Dr. Raj Jain of Ohio State and solicit feedback and suggestions. With that, Dubray closed the discussion on the cell/call terminology draft noting that the minutes would be forwarded to the editor. He thanked Bob Mandeville for being the discussion moderator. He also noted that due to the abbreviated session, there would not be an opportunity to discuss the Multicast Terminology draft beyond what had previously been discussed. He encouraged people to read the draft and comment via the BMWG mailing list. Goals for the next BMWG session in Memphis. 1. Edit LAN switch draft to reflect the agreed upon modifications. Distribute document to mailing list for comment. If no blocking issues are identified, ascertain consensus on whether to forward draft for consideration for RFC status. 2. Continue progress on the Cell/Call Terminology Draft: address issues and concerns raised during the BMWG session; revise and reissue draft and resubmit for comment. 3. Review and comment on Multicast Benchmarking Terminology Draft; revise and reissue draft as necessary.