[BEEPbuilders] Defining Beep Interoperability Tests (Echo first?)
Thu, 29 Aug 2002 13:28:17 -0700 (PDT)
You read my mind, Bob.
BTW, your proposal actually tests much more than the Echo profile (at
least steps 2 and 3).
Essentially what your proposing is a form of functional regression
conformance testing, which is a great place to start.
Are there any other relatively simple profiles that are already well
specified we could import as functional test cases? One thing we want to
isolate is application-level issues (such as correctly processing and
forming message content-level XML) from library/infrastructure issues
(which we are interested in). ECHO probably doesn't work all the message
patterns (e.g. MSG/RPY, MSG/ANS/NUL, MSG/ERR) so we should also (in fairly
short order) find another profile (or invent a simple one) to get at least
minimal coverage of the message types. Again, the aim is to avoid
requiring a lot of application level implementation and complexity while
relatively fully excercising the libraries.
On Thu, 29 Aug 2002, Bob Wyman wrote:
> As Gabe Wachob points out  defining precise tests of
> on-the-wire Beep streams is going to be a challenge since permitted
> variances in Beep implementation means that "there is no standard 'on
> the wire' stream." Given this problem, I would like to suggest that
> initial efforts at interoperability testing are pursued at a higher
> level of abstraction.
> Most users of Beep will be using one of the implementations of
> the Beep stack and are most interested in their practical ability to use
> a particular stack in writing profiles that can interact with Beep peers
> written with other Beep stacks. In general, I think it is safe to say
> that most users won't really care whether or not the protocol
> implementations underlying the Beep stacks actually conform to the
> precise wording of the RFCs. They just want to know if interoperability
> can be achieved. Thus, it seems to me that we would be safe in at least
> temporarily putting aside the apparently "hard" problem of testing
> conformance to the RFC's. We should focus first on testing what users
> care about: Whether or not interoperating profiles can be written.
> The easiest way to determine if interoperating profiles can be
> written is to actually write some profiles which exercise Beep
> implementations in the ways that actual profiles will.
> The simplest pattern of interaction is probably going to be that
> which is represented by the "Echo" profile, some form of which is
> already commonly included in a number of Beep stack implementations and
> is discussed at length in Marshall Rose's book "BEEP: The Definitive
> What I suggest is that we agree that an important test of
> interoperability is the ability of two Beep stacks to support
> interoperating implementations of an Echo profile. We would then write
> up a formal description of the Echo profile and define a test case that,
> if successfully passed, would allow one to make the statement that two
> implementations pass the "Echo Test." This would tell people that
> whatever the details of the on-the-wire streams may be, they should have
> some confidence that profiles similar to the Echo profile can be made to
> interoperate using the tested Beep stacks.
> The Echo Test Script would be something like the following:
> 1. Echo a series of messages where each message is some number
> of bytes larger than the last until some maximum size is reached.
> (testing window size, buffer size limits, etc.)
> 2. Perform the first test but using TLS.
> 3. Perform the first test but use various SASL mechanisms for
> 4. etc.
> 5. Perform all tests above, but reverse the roles of initiator
> and listener.
> After running the test, it should be possible to itemize the
> test steps and state success or failure for each step.
> Does such an approach make sense and would it generate useful
> results? Should I elaborate?
> bob wyman
> BEEPbuilders mailing list
Gabe Wachob email@example.com
Founder, WiredObjects http://www.wiredobjects.com