[BEEPbuilders] Defining Beep Interoperability Tests (Echo first?)

Bob Wyman bobwyman@firstrain.com
Thu, 29 Aug 2002 16:08:58 -0400


	As Gabe Wachob points out [1] defining precise tests of
on-the-wire Beep streams is going to be a challenge since permitted
variances in Beep implementation means that "there is no standard 'on
the wire' stream." Given this problem, I would like to suggest that
initial efforts at interoperability testing are pursued at a higher
level of abstraction. 
	Most users of Beep will be using one of the implementations of
the Beep stack and are most interested in their practical ability to use
a particular stack in writing profiles that can interact with Beep peers
written with other Beep stacks. In general, I think it is safe to say
that most users won't really care whether or not the protocol
implementations underlying the Beep stacks actually conform to the
precise wording of the RFCs. They just want to know if interoperability
can be achieved. Thus, it seems to me that we would be safe in at least
temporarily putting aside the apparently "hard" problem of testing
conformance to the RFC's. We should focus first on testing what users
care about: Whether or not interoperating profiles can be written.
	The easiest way to determine if interoperating profiles can be
written is to actually write some profiles which exercise Beep
implementations in the ways that actual profiles will.
	The simplest pattern of interaction is probably going to be that
which is represented by the "Echo" profile, some form of which is
already commonly included in a number of Beep stack implementations and
is discussed at length in Marshall Rose's book "BEEP: The Definitive
Guide." 

	What I suggest is that we agree that an important test of
interoperability is the ability of two Beep stacks to support
interoperating implementations of an Echo profile. We would then write
up a formal description of the Echo profile and define a test case that,
if successfully passed, would allow one to make the statement that two
implementations pass the "Echo Test." This would tell people that
whatever the details of the on-the-wire streams may be, they should have
some confidence that profiles similar to the Echo profile can be made to
interoperate using the tested Beep stacks. 

	The Echo Test Script would be something like the following:
	1. Echo a series of messages where each message is some number
of bytes larger than the last until some maximum size is reached.
(testing window size, buffer size limits, etc.)
	2. Perform the first test but using TLS.
	3. Perform the first test but use various SASL mechanisms for
authentication.
	4. etc.
	5. Perform all tests above, but reverse the roles of initiator
and listener.

	After running the test, it should be possible to itemize the
test steps and state success or failure for each step.

	Does such an approach make sense and would it generate useful
results? Should I elaborate?

		bob wyman

[1]
http://lists.beepcore.org/pipermail/beepbuilders/2002-August/000001.html