[BEEPbuilders] Interoperability Testing

Kevin Kress kkress@myslo.net
21 Aug 2002 10:59:14 -0700


Just my 2c, 

It seems that a set of test cases based off of the RFC should be
developed. Each should be numbered based on the RFC section it covers
(example  2.2.1.1a and 2.2.1.1b could test frame header compat.) 

Once a set of test cases are developed each implementation could
implement each test, a "compatibility test kit" it you will.   Each test
case would require multiple "servers" for however many roles are
required (ie client, server, or peers).  The impl could then test
against itself and one other impl. initially.

Then the tricky part comes in how do you "certify" a implementation as
interoperable?   Maybe pit X implementation against itself and
implementations W Y and Z. 

This system would require a common UI of sorts to allow easy running of
different tests on different impls and some sort of uniform output to
indicate PASS/FAIL on each test case.  Also a common test case format
would need to be developed that would cover all the requirements for
each test.

--Kevin

On Wed, 2002-08-21 at 11:26, Gabe Wachob wrote:
> One challenge I see is that there is no standard "on the wire" stream.
> Like TCP, beep connections have lots of state on either end of the wire
> and that state is not neccesarily "deterministic" from the point of view
> of the "other" peer (e.g. buffers may empty out at different rates, thus
> causing the window sizes change). Thus, I don't think we could write out a
> set of binary streams we'd could always *expect* to see on the line.
> 
> This rules out some of the interoperability tests you might see in the
> SOAP interop testing.
> 
> I'd be curious to hear from folks who were around back then how TCP/IP
> interop testing was done..
> 
> 	-Gabe
> 
> On Wed, 21 Aug 2002, Bob Wyman wrote:
> 
> > Being able to test interoperability of our various Beep implementation
> > would, of course, be very useful. We've done some informal work by
> > downloading other's implementations as they are announced, but I think
> > we would all be well served if there was some sort of a basic validation
> > suite that we could independently test against. Thus, I would be curious
> > to know if anyone can propose an existing validation suite that one
> > might use as a model for doing this sort of work. Or, in the absence of
> > a formal validation suite, if there are any standard procedures for
> > interoperability testing that we should try to propagate through the
> > Beep community. Of course, my hope is that anything done in the Beep
> > community would lay the foundation for similar work to be done with
> > Apex, Soap over Beep, and other Beep related efforts.
> >
> > 		bob wyman
> > _______________________________________________
> > BEEPbuilders mailing list
> > BEEPbuilders@lists.beepcore.org
> > http://lists.beepcore.org/mailman/listinfo/beepbuilders
> >
> 
> -- 
> Gabe Wachob                       gwachob@wachob.com
> Personal                       http://www.wachob.com
> Founder, WiredObjects    http://www.wiredobjects.com
> 
> _______________________________________________
> BEEPbuilders mailing list
> BEEPbuilders@lists.beepcore.org
> http://lists.beepcore.org/mailman/listinfo/beepbuilders
>