1 When worlds collide and then dance: Test::LectroTest meets Test::Builder {anchor:When worlds collide and then dance: Test::LectroTest meets Test::Builder} It seems that LectroTest is picking up in popularity because I am starting to get regular requests and feedback. Recently, two related requests came in regarding something that I had put off: Figuring out how to merge the world of specification-based testing that relies upon thousands of trials per property check with the world of case-based testing where there is one trial per case. The rub is that case-base testing in Perl is managed by Test::Builder and a related family of modules that are designed to make that kind of testing easy. One convenience they offer is that calling test functions like ~~cmp_ok~~ not only performs a test, in this case a general comparison, but also reports the result of the test to the test harness. So what happens if somebody wants to perform a ~~cmp_ok~~ test from within a LectroTest property specification? When the property is checked, LectroTest will test whether the property holds in one thousand random trials (by default), each of which will end up calling ~~cmp_ok~~. Can you see where this is going? Yup, the test harness will end up seeing one thousand separate tests instead of the single test of the property. The solution to this problem, as to all problems of distinction, is hackery. The Test::* family of modules ends up filtering all calls to test functions such as ~~cmp_ok~~ down to the Test::Builder module's ~~ok~~ method. This method does two things. First, it reports the result of the test to the test harness without giving us a chance to say otherwise. (Naughty!) Second, it returns the result of the test back to the original caller. As far as property checks are concerned, the first part is bad, and the second, good. So, I basically redefine the ~~ok~~ method during property checks to throw out the first part. In a little more detail, here is how it works. I created a new module Test::LectroTest::Compat that exports a single function ~~holds~~. This function is used to inject a property check into a plain-old Test::Simple- or Test::More-style test plan. For example: {code:none} use Test::More tests => 2; use Test::LectroTest::Compat; my $prop_nonnegative = Property { \##[ x <- Int, y <- Int ]## cmp_ok(my_function( $x, $y ), '>=', 0); }, name => "my_function output is non-negative" ; __holds( $prop_nonnegative );__ # assert that the prop holds cmp_ok( 0, '<', 1, "trivial 0<1 test" ); # a "normal" assertion \# ... and so on ... {code} What, then, does ~~holds~~ do when called? It redefines Test::Builder's ~~ok~~ method, runs the property-check trials, restores ~~ok~~, and finally reports the property check's results via the newly-restored ~~ok~~. From there, Test::Builder takes over and does the magic necessary to incorporate the result into our run-of-the-mill test plan per the test harness's expectations. The actual code is as follows: {code:none} sub holds { my ($diag_store, $results) = check_property(@_); my $success = $results->success; (my $name = $results->summary) =~ s/^.*?- /property /; $Test->ok($success, $name); my $details = $results->details; $details =~ s/^.*?\\\n//; # remove summary line $details =~ s/^\\\# / /mg; # replace commenting w/ indent $Test->diag(@$diag_store) if @$diag_store; $Test->diag($details) if $details; return $success; } sub check_property { no strict 'refs'; no warnings; my $diag_store = \[\]; my $property = shift; local *Test::Builder::ok = \\\&disconnected_ok; local *Test::Builder::diag = sub { shift; push @$diag_store, @_ }; return ( $diag_store, Test::LectroTest::TestRunner->new(@_)->run($property) ); } sub disconnected_ok { $_\[1\] ? 1 : 0 } {code} You can see that there is one extra bit of hackery going on. I also redefine Test::Builder's ~~diag~~ method to capture any diagnostic output that may be emitted during the trials. Typically, this would occur only when a trial fails, and in this case the output is almost certainly worth passing back to the user in context. To ensure that the user sees it in context, I hold on to the captured output until the property check is complete and then roll it into the normal LectroTest diagnostic output. It looks great: {code:none} not ok 3 - property 'x is a natural number' falsified in 2 attempts \# Failed test (t/compat.t at line 32) \# '0' \# > \# '0' \# Counterexample: \# $x = 0; {code} In the first part you see the typical ~~cmp_ok~~ output that the assertion 0 &gt; 0 failed. The second part is the LectroTest counterexample that shows at what part of the test space the assertion failed. It takes some complicated footwork, but the resulting dance is beautiful. To see two markedly different testing systems &#8211; and in some ways markedly different testing philosophies &#8211; working in step with one another is gratifying. I suspect that most real-world testing problems can be solved better by a combination of the two approaches than by either alone. Thus I am especially happy about this integration. After some more refinement, I am going to incorporate Test::LectroTest::Compat into the main distribution. For now, you can get a copy in the [LectroTest/FAQs] section of the site. 2005-02-08 20:56:50.34 2005-02-08 01:28:39.298