AUTOMATED TESTING -- WHY AND HOW
by Eugene Volokh, VESOFT
Presented at 1990 INTEREX Conference, Boston, MA, USA
Published by INTERACT Magazine, Dec 1990.
Everyone knows how important testing is, and, with luck, everyone
actually does test the software that they release. But do they really?
Can they? Even a simple program often has many different possible
behaviors, some of which only take place in rather unusual (and hard
to duplicate) circumstances. Even if every possible behavior was
tested when the program was first released to the users, what about
the second release, or even a "minor" modification? The feature being
modified will probably be re-tested, but what about other, seemingly
unrelated, features that may have been inadvertently broken by the
modification? Will every unusual test case from the first release's
testing be remembered, much less retried, for the new release,
especially if retrying the test would require a lot of preliminary
work (e.g. adding appropriate test records to the database)?
This problem arose for us several years ago, when we found that our
software was getting so complicated that testing everything before
release was a real chore, and a good many bugs (some of them very
obvious) were getting out into the field. What's more, I found that I
was actually afraid to add new features, concerned that they might
break the rest of the software. It was this last problem that really
drove home to me the importance of making it possible to quickly and
easily test all the features of all our products.
AUTOMATED TESTING
The principle of automated testing is that there is a program (which
could be a job stream) that runs the program being tested, feeding it
the proper input, and checking the output against the output that was
expected. Once the test suite is written, no human intervention is
needed, either to run the program or to look to see if it worked; the
test suite does all that, and somehow indicates (say, by a :TELL
message and a results file) whether the program's output was as
expected. We, for instance, have over two hundred test suites, all of
which can be run overnight by executing one job stream submission
command; after they run, another command can show which test suites
succeeded and which failed.
These test suites can help in many ways:
* As discussed above, the test suites should always be run before a
new version is released, no matter how trivial the modifications
to the program.
* If the software is internally different for different
environments (e.g. MPE/V vs. MPE/XL), but should have the same
external behavior, the test suites should be run on both
environments.
* As you're making serious changes to the software, you might want
to run the test suites even before the release, since they can
tell you what still needs to be fixed.
* If you have the discipline to -- believe it or not -- write the
test suite before you've written your program, you can even use
the test suite to do the initial testing of your code. After all,
you'd have to initially test the code anyway; you might as well
use your test suites to do that initial testing as well as all
subsequent tests.
Note also that the test suites not only run the program, but set up
the proper environment for the program; this might mean filling up a
test database, building necessary files, etc.
WRITING TEST SUITES
Let's switch for a moment to a concrete example -- a date-handling
package, something that, unfortunately, many people have had to write
on their own, from scratch. Say that one of the routines in your
package is DATEADD, which adds a given number of days to a date, and
returns the new date. Here's the code that you might write to test it
(the dates are represented as YYYYMMDD 32-bit integers):
IF DATEADD (19901031, 7) <> 19901107 THEN
BEGIN
WRITELN ('Error: DATEADD (19901031, 7) <> 19901107');
GOT_ERROR:=TRUE;
END;
IF DATEADD (19901220, 20) <> 19910109 THEN
BEGIN
WRITELN ('Error: DATEADD (19901220, 20) <> 19910109');
GOT_ERROR:=TRUE;
END;
...
As you see, the code calls DATEADD several times, and each time checks
the result against the expected result; if the result is incorrect, it
prints an error message and sets GOT_ERROR to TRUE. After all the
tests are done, the program can check if GOT_ERROR is TRUE, and if it
is, say, build a special "got error" file, or write an error record to
some special log record. This way, the test suites can be truly
automatic -- you can run many test suites in the background, and after
they're done, find out if all went well by just checking one file, not
looking through many large spool files for error messages.
The first thing that you might notice is that the DATEADD test suite
can easily grow to be much larger than the DATEADD procedure itself!
No doubt about it -- writing test suites is a very expensive
proposition. Our test suites for MPEX/3000, SECURITY/3000, and
VEAUDIT/3000 take up almost 30,000 lines, not counting supporting
files and supporting code in the actual programs; the total source
code of our products is less than 100,000 lines. Often, writing a test
suite for a feature takes as long or almost as long as actually
implementing the feature. Sometimes, instead of being reluctant to add
a new feature for fear of breaking something, I am now reluctant to
add a new feature because I don't want to bother writing a test suite
for it.
Fortunately, the often dramatic costs of writing test suites are
recouped not just by the decrease in the number of bugs, but also by
the fact that test suites, once written, save a lot of testing time.
It's much easier for someone to run an already-written test suite than
to execute by hand even a fraction of the tests included in the suite,
especially if they require complicated set-up. Since a typical program
will actually have to be tested several times before it finally works,
the costs of writing a test suite (assuming that it's written at the
same time as the code, or even earlier) can be recouped before the
program is ever released.
Also, test suites tend to have longer lives than code. A program can
be dramatically changed -- even re-written in another language -- and,
assuming that it was intended to behave the same as before, the test
suite will work every bit as well. Once the substantial up-front costs
of writing test suites have been paid, the pay-offs can be very
substantial.
But even though we should be willing to invest time and effort into
writing test suites, there's no reason to invest more than we have to.
In fact, precisely because test suites at first glance seem like a
luxury, and people are thus not very willing to work on them, creating
test suites should be as easy as possible. What can we do to make
writing test suites simpler and more efficient?
One goal that I try to shoot for is to make it as easy as possible to
add new test cases, even if this means doing some additional work up
front. I try to make every new test case, if possible, to fit on one
line. The reason is quite simple: I want to have as little
disincentive as possible to add new test cases. A really fine test
suite would have tests for many different situations, including as
many obscure boundary conditions and exceptions as possible; also, any
time a new bug is found, a test should be added to the test suite that
would have caught the bug, just in case the bug re-surfaces (a
remarkably frequent event). If we grit our teeth and write some
convenient testing tools up front, we can make it much easier to
create a full test suite.
Here, for instance is one example:
PROCEDURE TESTDATEADD (DATE, NUMDAYS, EXPECTEDRESULT: INTEGER);
BEGIN
IF DATEADD (DATE, NUMDAYS) <> EXPECTEDRESULT THEN
BEGIN
WRITELN ('Error: DATEADD (', DATE, ', ', NUMDAYS, ') <> ',
EXPECTEDRESULT);
GOT_ERROR:=TRUE;
END;
END;
...
TESTDATEADD (19901031, 10, 19901110);
TESTDATEADD (19901220, 20, 19910109);
TESTDATEADD (19920301, -2, 19920228);
...
By this model, each procedure that you test would have a test
procedure like this one written for it; then the main body of your
test program would just be calls to these test procedures.
This is especially useful for procedures that require some special
processing before or after being called; for instance, they might have
reference parameters that need to be put into variables before they're
passed, record structure parameters to be filled, multiple
by-reference output parameters that all need to be compared against
expected values, and so on.
You can make up other, even more general-purpose testing tools, such
as the following procedure:
PROCEDURE MUSTBE (TAG: stringtype; RESULT, EXPECTEDRESULT: INTEGER);
BEGIN
IF RESULT<>EXPECTEDRESULT THEN
BEGIN
WRITELN ('Error: ', TAG, ': ', RESULT, ' <> ', EXPECTEDRESULT);
(* error handling code *)
END;
END;
This procedure can be used to check the result of any function that
returns an integer value, e.g.
MUSTBE ('DATEADD #1', DATEADD (19901031, 10), 19901110);
MUSTBE ('DATEADD #2', DATEADD (19901220, 20), 19910109);
MUSTBE ('DATEADD #3', DATEADD (19920301, -2), 19920228);
Other, similar, procedures might be written to help test functions
that return other types (REALs, STRINGs, etc.). On the other hand, for
functions that can't easily be called in one statement (because they
take by-reference or specially-formatted parameters), you might want
to consider writing a special test procedure.
Finally, one other alternative (which I personally prefer) is writing
a special "shell" program that asks for a procedure name, its
parameters, and the expected result, calls the procedure, and checks
the result:
PROGRAM TESTSHELL ...
...
READLN (PROCNAME, P1, P2, EXPECTEDRESULT);
WHILE PROCNAME<>'EXIT' DO
BEGIN
IF PROCNAME='DATEADD' THEN RESULT:=DATEADD (P1, P2)
ELSE IF PROCNAME='DATEDIFF' THEN RESULT:=DATEDIFF (P1, P2)
ELSE IF PROCNAME='DATEYEAR' THEN RESULT:=DATEYEAR (P1)
...
IF RESULT <> EXPECTEDRESULT THEN
... output error ...
READLN (PROCNAME, P1, P2, EXPECTEDRESULT);
END;
...
This way, your actual test suite could be a job stream, to which you
can add as many test cases as you like -- one line per test case --
without having to recompile anything:
!JOB TESTDATE, ...
!RUN TESTSHEL
DATEADD 19901031 10 19901110
DATEADD 19920228 2 19920301
DATEYEAR 19920228 0 1992
...
!EOJ
Whenever you make a change to your procedures, you just rerun the
TESTDATE job, and you'll either find some bugs or be reasonably
confident (though, of course, never 100% confident) that the software
works.
TESTING PROGRAMS THAT DO I/O
It's rather easy to test a procedure whose only inputs are its
parameters and whose only output is its result (or even a by-reference
parameter). The more places a program derives its input from, or sends
its output to, the harder it becomes to test.
Let's take a simple I/O program, one which reads a file, reformats it
in some way, and writes the result to another file. Obviously, to test
it, we should fill up the input file, run the program, and compare the
output file against the expected output file. As we discussed in the
previous section, it would be nice if we could build a program -- it
might be a 3GL or 4GL program, or even an MPE or MPEX command file --
that takes as parameters the input data and the expected output data,
so that we can easily add new test cases.
A first try on this might be a job stream like the following:
:PURGE TESTIN
:FCOPY FROM;TO=TESTIN;NEW
LINE ONE
LINE TWO
LINE THREE
:FILE MYPROGI=TESTIN
:PURGE TESTOUT
:FILE MYPROGO=TESTOUT
:RUN MYPROG
:PURGE TESTCOMP
:FCOPY FROM;TO=TESTCOMP;NEW
PROCESSED LINE A
PROCESSED LINE B
:SETJCW JCW=0
:CONTINUE
:FCOPY FROM=TESTCOMP;TO=TESTOUT;COMPARE=1
:IF JCW<>0 THEN
: handle error
:ENDIF
or, if the commands are put into a separate command file or UDC,
:TESTCMDS
LINE ONE
LINE TWO
LINE THREE
:EOD
PROCESSED LINE A
PROCESSED LINE B
:EOD
(the data would go as input to the :FCOPY commands in the command
file).
Note how the :FILE equations come in handy to redirect the program's
input and output files. Not only does this avoid the need to overwrite
the production input and output files, but it makes it possible for
several test suites which test programs that normally use the same
files (e.g. this program, the program that created this program's
input file, and the one that reads this one's output file) to run at
once. If for some reason your programs don't allow :FILE equations
(e.g. they issue their own :FILE equations to refer to these files),
try to change them so they do, or at least so they have a special
"test" mode that will read :FILE-equatable files. Note also that the
job stream regenerates the input and comparison files every time it
runs. I recommend this, since then each job stream would be a more or
less self-contained unit (if it uses a special command file that no
other test job uses, I suggest that you build even this command file
inside the test job). It is easier to move or maintain, and is less
likely to suffer from "software rot" (a condition that causes software
that's been left on the shelf too long to stop working, largely
because some outside things that it depends on have changed).
Back to our example. One problem with it is that :FCOPY ;COMPARE= is
rather finicky about the files it's comparing -- for instance, they
must both have exactly the same record size. TESTCOMP, built by an
:FCOPY FROM;TO=TESTCOMP would normally have the same record size as
the job input device, so you might need a :FILE equation to work
around this.
A more serious problem is that :FCOPY FROM;TO= can only be easily used
for creating files that contain ASCII data. What if some of the
columns of the file need to contain binary data?
Here is where I think you ought to grit your teeth and write a special
program (unless, of course, you have a 4GL that can do this for you).
Yes, I know that it seems like a pain to write code that will never be
run in production, but is only needed to test other code, but this
rather simple program could, if designed right, prove to be a highly
reusable building block.
The program would first prompt for some sort of "layout" of the file
-- a list of the starting column numbers, lengths, and datatypes of
each field in the file. Then, it would prompt for each record in the
file, specified as a list of fields, separated by, say, commas; it
would format the fields into the file record, and write it into the
file. Thus, you'd say:
:RUN BLDFILE
S,1,8, I,9,2, S,11,10, P,21,8 << string, integer, string, packed >>
SMITH, 100, XYZZY, 1234567
JONES, 55, PLUGH, 554927
...
Once you write this program, incidentally, you might find that it has
other uses, say, to do manual testing of your program once you've
already found that it has a bug and are trying to isolate it. And, of
course, if you make it general enough, it should be usable in all of
your test suites.
Also, your input file had to have been created by some program, and
your output file must be intended as to input to some other program;
there's nothing that says that you can't run those programs in the
test job to create the input file from data you've input and then
format the output file into readable text. The problems come in if the
other programs are too hard to run in batch (e.g. require block mode),
or if you'd like to be able to test each program separately from the
others, perhaps because you want to see how your program reacts to
illegal data in its input file, data that shouldn't normally appear in
the input generated by the other program.
What if your programs reads and writes an IMAGE database? This is in
some ways simpler to test and in other ways more difficult. You can
use QUERY to fill the input sets and create output (using >LIST or
>REPORT) will be usable by :FCOPY ;COMPARE=. Be sure, though, that you
sort any master sets that you dump using >REPORT -- since the order of
the entries in the master set depends on the hashing algorithm, which
depends on the capacity, unsorted output will make the test suite find
an "error" every time you change the capacity.
However, the setup of the IMAGE database might also be a bit more
cumbersome, largely since you probably want to have your own special
test database built by the job (for the reasons discussed above --
independence from the production data, from other test suites' data,
and self-containedness). You might want to create a simple program or
command file that takes a schema file, lowers the dataset capacities
on it, runs DBSCHEMA, and then does a DBUTIL,CREATE -- you'll find
that a lot of your test suites can use it.
ADJUSTING FOR ENVIRONMENT INFORMATION
Our pass-input-and-compare-output-against-expected-result strategy
works just fine if the same input is always supposed to yield the same
output, but what if the output can vary? The most common variables are
based on current date and time -- reports that contain this
information in headers, output files that have each value
date-stamped, a date-handling procedure that returns today's
day-of-week, and so on. Another related problem is with programs that
check whether they're being run online or in batch, and do different
things in these cases -- how can your batch test suite make sure that
the online features work properly?
What we really have here is a different sort of input, input not from
a file or a database, but from the system clock or the WHO intrinsic.
There are a few ways of handling this; for example, instead of doing
an FCOPY ;COMPARE=, which demands exact matches, you can have your own
comparison program that lets you specify that some particular field --
e.g., the date on a report header -- will not get compared. Even
better, your comparison program can let you specify that a particular
field should be equal to, say, the current year, month, or day,
calculated at the time the comparison program runs.
However, more flexible still -- and necessary for things like
pretending you're online rather than in batch -- you can try to
redirect this "input" from the environment, just as you redirected
input from files and databases using :FILE equations.
Now how are you going to do this redirection? Believe it or not, after
having the gall to ask you to write test suites that are as long as
your source code, I'm suggesting that you change your programs to
accommodate testing requirements. Instead of calling CALENDAR or
DATELINE, for instance -- or using whatever language construct may
give you this information -- you might write your own procedure:
FUNCTION MYCALENDAR: SHORTINT;
BEGIN
get the value of the "PRETENDCALENDAR" jcw;
IF the value is 0 THEN
MYCALENDAR:=CALENDAR
ELSE
MYCALENDAR:=value of jcw;
END;
This way, your program would normally get the current date from the
CALENDAR intrinsic, but when the PRETENDCALENDAR JCW is non-zero, will
use that value instead. You might, for efficiency's sake, want to get
the JCW value only once, and then save it somewhere; for ease of use,
you might want to look at the PRETENDYEAR, PRETENDMONTH, and
PRETENDDAY JCWs, and assemble the CALENDAR-format value from them
(possibly using the date-handling package that we so thoroughly tested
a few pages ago).
A similar procedure might be written to determine whether the program
is running online or in batch -- it'll check the PRETENDONLINE JCW,
and if it doesn't exist, or set to some default value, will call the
WHO intrinsic. If your program does different things depending on the
user's capabilities or logon id, you might want to have similar
procedures for them, too (wrapped around the WHO intrinsic call) --
although it's possible for your test suite to actually be several
jobs, each of which logs on under a different user, with different
capabilities, it may be more convenient for you if one job can make
itself look like each one of these users in turn. In fact, it might
even be convenient for your own manual debugging (say, when you want
to duplicate the program's behavior as a particular user id, or on a
particular date, but don't want to re-logon or change the system
clock).
Of course, the drawback to this approach is that you're not actually
testing the program as it really behaves, but rather as it behaves
with the testing flag set; the code you're executing in testing mode
is somewhat different than is normally executed, and if, say, there's
a bug in the CALENDAR call or the WHO call, your test suite won't
catch it, since in testing mode the intrinsics aren't called at all.
Unfortunately, this seems to be a necessary evil; the only solution is
to minimize the amount of code whose execution depends on whether or
not you're in testing mode.
One thing that you might do -- if you want to be really fancy -- is
create a library of procedures called CALENDAR, CLOCK, WHO, etc.,
which would, depending on some testing flag, either call the real
CALENDAR, CLOCK, or WHO, or return "pretend" values; you can then put
these procedures into an RL, SL, or XL, and not have to change your
source file. Once you debug your library procedures, you should have
more confidence that your testing in test mode actually simulates what
the program will really behave like in production. One thing that you
may have to do, however, is intercept not just the intrinsics that you
call directly, but also whatever procedures might be called by
language constructs (like COBOL's facility for returning today's
date).
WHAT TEST CASES SHOULD YOU USE?
So far, we've talked a lot about how to write tools that make it easy
for you to add test cases to your test suites, but not much about what
your test cases should be. Say that you're testing a DATEADD
procedure, one that returns a date that is X days after date Y. (Let's
assume that X could be negative -- X = -5 means a date that is 5 days
before date Y.) What test cases should you use? Think about this
before reading the answers!
Well, it seems to me that there are quite a few:
* Add days so that it stays in the same month (e.g. 1990/05/10+7).
* Add days so that it changes months (e.g. 1990/05/10+30).
* Add days so that it changes years (e.g. 1990/05/10+300).
* Add days so that it changes months or years over February 28th in
a non-leap year (e.g. 1990/02/10+30).
* Add days so that it changes months or years over February 29th in
a non-leap year (e.g. 1992/02/10+30).
* Handle years that are divisible by 100 but not by 400 (like 1900
or 2100), which are not leap years (did you know this?).
* Add 0 days.
* Add days so that it goes outside of your accepted date range (e.g.
beyond 1999, or whatever other date is your limit).
* Add to an invalid date -- one with an invalid year, month, or day.
* All the above, but with subtracting days.
Wow! That's a lot of work. But, you'll have to admit, all the above
are things that you really should test for (unless they're not
relevant to your particular interpretation, e.g. if your date range
doesn't extend to 1900 or 2100, or if you've consciously decided not
to check for certain error conditions), manually if not automatically;
it's especially important to test for "boundary conditions" (did you
know that, in the DATELINE intrinsic, the next day after DEC 31, 1999
is JAN 1, 19:0?), for cases that require special handling (like leap
year), and for proper handling of errors.
These are the obvious tests -- tests for bugs that you expect might
happen. As other bugs come up, however, you ought to add test cases
that would have caught these bugs: firstly, you'll have to test your
fix anyway, and if you add the test case before implementing it, it
won't cost you anything extra; secondly, the same bug (or a similar
one) may come up later, but this time will get caught.
Still, there's no need to get extreme about things; shortcuts are
still possible. Say, for instance, that DATEADD works by converting
the date into a "century date" format (number of days since a
particular base date), adding the number of days, and then converting
back into a year/month/day format -- if you're sure that this is all
it does, you might just have one test case (preferably one that seems
to exercise as much of the internal logic as possible, such as one
that changes months and years). Of course, you'd still have to
properly test the date conversion routines.
In general, what you test should depend on how your code works.
Whenever you know your code treats two different types of input
differently, you should test both. If you're fairly certain that a
single test will test many features, you can just use that one test;
if, for instance, you know that testing one module or routine will
also adequately test the module or routines that it calls, you can
make do with just testing the top-level one. However, try to resist
this temptation; firstly, the top-level module probably doesn't
exercise all the functions of the bottom-level one, and secondly, it's
very convenient to have a test suite for the bottom-level module --
that way, if you're making substantial changes to your system and you
know the top-level module is broken, you can still test the
bottom-level one independently.
Finally, an obvious point, but one that it often neglected -- it's
better to test a little than not at all. If you find something that's
hard to test in all possible ways, test it in at least a few; if, for
instance, its results are hard to automatically verify, at least make
sure that they're in the right format, or even that they're returned
at all (i.e. that the program doesn't just abort). There's really no
90-10 rule in testing -- 10% of the effort won't get you much more
than 10% of the benefit -- but you can at least avoid some of the more
obvious (and more embarrassing) bugs. Then, once the groundwork is
laid, you might try to get back to it periodically, adding a new test
case here or there. Don't let perfectionism get in the way of doing at
least something.
VERIFYING DATA STRUCTURES
Most sufficiently complicated data structures -- anything from your
data stored in an IMAGE database to your own linked lists, hash files,
or B-trees, if you write such things yourself -- have internal
consistency requirements. Certain fields in your databases may only
contain particular values; other fields must have corresponding
records in other datasets or in other databases. If any of these
internal consistency requirements are not met, you know you have a bug
somewhere.
You can get a lot of benefit out of writing a verification routine for
each such data structure that checks it for internal consistency. This
is somewhat different from the test suites we discussed before, which
check for the validity of the ultimate results, but it can still be
very useful, since internal inconsistency must have, by definition,
been caused by a program error, and is likely to eventually lead to
incorrect results (incorrect results that your test suites might not
otherwise check for).
You should call this verification routine at the end of each test
suite to verify the consistency of the structures (again, usually data
in the database) that the program being tested built; you might even
run it after each step in the test suite, to isolate exactly where an
error might be sneaking in. You may also want to run the verification
routine against your production database every night, to check your
programs as they run in the real world, not just the controlled test
environment; and, you can run it whenever you suspect that something
may be wrong (either in testing or in production), to figure out if an
internal inconsistency might be causing it.
The verification routine shouldn't be hard to write; if you can do it
using a 4GL or some other tool (like Robelle's fast SUPRTOOL -- speed
is important, since you want to make it as quick and easy as possible
to verify your data), all the better. Simply put, check all the fields
for which at least one possible value would be invalid, whether it is
because it's not one of a list of allowable values for this field, or
because it's out of range, or because it's inconsistent with some
other values in this record, or some other values in the
dataset/database. Possible checks include:
* Flag fields may contain only certain allowable values.
* Numbers, like salaries or prices, must be within certain ranges
(e.g. non-negative, below a certain amount, etc.).
* Dates must be valid (valid year, month, day).
* Strings must at least not include non-alphanumeric characters.
* Some fields must have corresponding entries in a different
dataset or database (do a DBGET mode 7, for instance, to make
sure they're there).
* Some fields are calculated from other fields, and must match
(e.g. a total price field in an invoice that must be the sum of
the price fields in the line items).
Not only can this check for bugs in your programs, but can also check
for invalid production data that your programs might not have detected
(e.g. garbage characters in string fields, bad states, state codes
that don't match phone numbers, etc.). And, again, if written as a
QUERY >XEQ file, or as a 4GL program, it can be very easily created,
and used over and over again.
TESTING COMMAND-DRIVEN AND CHARACTER-MODE INTERACTIVE PROGRAMS
As we discussed before, the key to successful automated testing is to
have the proper tools that make adding test cases easy. One in
particular -- which takes some work to construct but can make writing
test suites much simpler -- is very much worth discussing.
This test-bench lets you run another program under its control, with
the son program's $STDIN and $STDLIST redirected to message files. The
test-bench can let you specify input to be passed to the son program,
and the expected output that the son program should display; for
instance, a typical test suite might look like:
:RUN TESTBENC
RUN SONPROG << command to start son process >>
I CALC 10+20 << command for son to execute >>
O 30 << expected result >>
DOIT << tells test-bench to compare the results >>
I SQUARES 3 << command for son to execute >>
O 1 << expected result >>
O 4 << expected result >>
O 9 << expected result >>
DOIT << tells test-bench to compare the results >>
...
What are the advantages of this approach?
* It lets you specify the expected output right after the input;
this makes the test suite much easier to write (and read and
maintain) than if you had to specify all the input up front and
all the expected output (from all the commands) at the end.
* It lets you compare the output and the expected output much more
flexibly than a simple :FCOPY ;COMPARE= would; you can specify
special commands that indicate, say, that the output needn't be
exactly as you specified, but might include some variations here
and there (e.g. date- or environment-dependent information).
* It tells you exactly which commands got errors, rather than just
telling you an error was found.
Note, however, that all this test-bench does is feed input into the
son process' $STDIN and read output from its $STDLIST; what about
other output, say output to files, databases, JCWs, or MPE/XL
variables, and input from the same places?
Fortunately, output to one of those places can easily be converted
into output to $STDLIST simply by executing an MPE command, like
:PRINT (or :FCOPY on MPE/V), :SHOWJCW, :RUN QUERY, etc. If our program
can not only check the output of the son process and feed input to a
son process, but execute MPE commands and check their output and feed
them input, our problems will be solved.
Let's say your program is supposed to build a file, and you want to
make sure that the file is built with the right structure and the
right contents. Then, your test suite might look like this:
:RUN TESTBENC
RUN SONPROG << command to start son process >>
I BUILDFILE XXX << command to test >>
O File was built. << expected output to $STDLIST >>
DOIT
MPE :LISTF XXX,2 << MPE command to execute >>
O ...
O XXX 123 32W ... << expected :LISTF output >>
O ...
DOIT
MPE :PRINT XXX << MPE command to execute >>
O ... << expected :PRINT xxx output >>
DOIT
How does this program work? Well, as we said before, it runs the son
program with $STDIN and $STDLIST redirected to message files; "I"
commands write stuff to the input message file, "O" commands write the
expected output records to a special temporary file, and "DOIT"
commands read the output message file and compare it against the
O-command temporary file.
One problem is how "DOIT" will recognize that the son program is done
with its output and has issued another input prompt. If the son
program always uses the same prompt (or one of a few prompts), DOIT
can check for this; if, however, the son program's prompt isn't easily
distinguishable from normal output, you should make the son program
print a special line (e.g. "***INPUT***") before doing any input when
it's in "testing mode"; so long as this happens before any input, DOIT
can recognize these lines and realize that the output is done.
What about the MPE commands that can be used to "convert" output to
files, databases, etc. into output to $STDLIST? When we test MPEX (the
test-bench I'm describing is essentially the one that we use to test
all of our software), this is no problem, since we can just pass MPEX
an MPE command as input, and MPEX will execute it. You might do the
same yourself -- make sure that the program you're testing executes
MPE commands -- or you can have TESTBENC have two son processes, one
the program being tested, and the other a simple program that prompts
for an MPE command and executes it. The only other problem that you'll
face here is executing :FCOPY or :RUN commands on MPE/V (where they
can't be done with the COMMAND intrinsic); however, if you're an MPEX
customer, you can actually use MPEX as this MPE-command-executing son
process -- MPEX can execute :FCOPYs, :PRINTs, :RUNs, etc.
This test-bench $STDIN-and-$STDLIST-redirection solution, it seems to
me, would work quite well for any command-driven or interactive
character-mode programs. If you want to use it to test procedures,
you'll have to write a simple shell program that prompts for input
parameters, calls the procedure, and prints the output parameters, and
run this program as a son of the test-bench. Testing block-mode
programs, I suspect, would be much more difficult; I'll have a few
words about it later, but it's still an unsolved problem as far as I'm
concerned.
Of course, the more complicated your test-bench is, the more important
it is to write a test suite for it! (A bug in the test-bench that
keeps it from properly checking things could be almost unnoticeable,
since it will falsely tell you that your test suite ran fine.) We test
our test-bench by feeding a lot of test commands, some of which are
calculated to produce errors and others to succeed, and check the
results of these operation (not using the test-bench itself, of
course) to see if they're as expected.
AUTOMATIC TEST SUITE GENERATION
No matter how sophisticated your test-bench, you'll still have to
write your test cases. For simple one-line-input, one-line-output
operations, there's little that you need to do beyond specifying the
input and the expected output; however, for things that require a lot
of set-up, or are actually conversations, with many output prompts and
many inputs, you'd like a better way.
One idea that some automated testing people like is having you run the
program once, specifying all the right inputs, and making sure that
all the outputs are correct; these inputs and outputs will then be
saved, ready to be "re-played" by the test-bench, which will resubmit
exactly the same inputs, and expect to get exactly the same outputs.
In effect, then, the test suite will check that all subsequent
executions of the program behave exactly the same way as the initial
one (which was presumably correct). The way you'd do this is by
modifying the test-bench program we discussed above (what? you mean to
say you haven't written it yet?) to have a special "data-collection"
mode that will accept user inputs, pass them to the program, collect
the outputs, and create a file that can later be used by the normal
mode of the test-bench.
Now, there are a few problems with this, which lead me to conclude
that, even if this data-collect feature is used, the test suite that
it generates must be easy to modify. Firstly, the user will doubtless
make errors while entering the original inputs; since the data-collect
feature doesn't know what's a user error and what should be part of
the test suite, there needs to be some way of editing out these
errors. (Technically, you need not do this, since exactly the same
inputs should yield exactly the same error outputs in the future, but
if you don't edit them out, the test suite will be very unreadable and
unmaintainable.) Secondly, the future output probably won't exactly
match the current output -- dates and other environment information
(version numbers, etc.) will doubtless change. There needs to be some
way to edit the generated test suite to replace the expected output
with some sort of "wildcard" characters that tell the test-bench that
any output would be acceptable in this case.
However, taking into account that some editing will be needed, a
data-collect feature can be quite convenient for testing features that
involve complicated I/O sequences.
A SAMPLE TEST ENVIRONMENT
Besides having the right test tools and the right test suites, it's
important to internally set up your test suites and your test
environment so that it is as easy as possible to run all your tests
and check whether or not they succeeded. Here are a few tips that
we've found handy ourselves:
* Have one test suite for each major feature, not one big one for
your entire system. When you're working on a particular feature
and want to see if it works, you'll probably want to re-run only
that test suite after each change, and re-run all the test suites
only at the very end.
* As we mentioned before, have each test suite as self-contained as
possible, but also try to have each test case within each test
suite be relatively self-contained. The more a test case depends
on the results of the test cases that preceded it, the harder it
will be for you to fully understand what the state of your
internal files, databases, etc. is at the time the test case is
executed, and the harder it will be to maintain it, or even
understand why the "expected results" you have for it in your
test suite are really what should be expected. Of course, some
test cases must not be self-contained precisely because you want
to make sure that they work properly when done together, rather
than separately.
* Have each test suite run in its own group, with all the files
needed by the program being tested redirected to the files in
that group. The first thing that the test suite should do is
purge the group (if it's logged on to it, this will merely purge
all the files); this way, you'll be sure that this test run is
not influenced by previous runs of the same test suite, and since
the test suite runs in its own group, it will not be influenced
by concurrently-running other test suites.
* All the actual test suites and permanent support files should be
in their own group, separate from the groups in which the test
suites run; this way, the test suites will be able to purge their
own groups, as discussed above. If the test suite files are all
in a particular fileset (e.g. "T@.TEST"), they can be submitted
in MPEX using a %REPEAT/%STREAM/%FORFILES construct.
* Each test suite should signal that it completed successfully by
building a file called TESTOK, and that it failed by building a
file called TESTERR (or, even better, TESTE###, where ### stands
for the number of errors discovered). Then, a :LISTF
TESTERR.@.TESTACCT,6 will show you which jobs had errors in them.
In case you're afraid that a job might abort without building
either a TESTOK or TESTERR file, you can build the TESTERR file
at the very beginning and only purge it at the end if all went
well.
* Finally, if you use a test-bench program, the test-bench should
send all its output, especially an indication of all the errors
(what the input was, what the output was, and what the expected
output was), to a disc file called, say, TESTLOG, which can
easily be read, and will remain on the system even if the spool
file is deleted.
Thus, the configuration we use in VESOFT is:
C@.TEST.VESOFTD -- command files used by the test suites.
M@.TEST.VESOFTD -- MPEX test suites.
S@.TEST.VESOFTD -- SECURITY test suites.
A@.TEST.VESOFTD -- VEAUDIT test suites.
TESTPROD.TEST.VESOFTD -- a command file that purges the VETEST
account and %STREAMs M@.TEST+S@.TEST+A@.TEST.
@.MALTFILE.VETEST -- group used by test suite
MALTFILE.TEST.VESOFTD.
@.MBATCH.VETEST -- group used by test suite MBATCH.TEST.VESOFTD.
...
TESTING SEEMINGLY HARD-TO-TEST PROGRAMS
Some things are easier to test than others; procedure calls are
simplest, command-driven programs are rather straightforward,
"conversational" character-mode programs are a bit harder. Much
depends on how easy it is to feed the program input and intercept the
program's output; for example, if a program does input from tape, you
might redirect it by a :FILE equation to a disc file, but how will you
test the code in the program that tries to handle tape errors? If a
program is supposed to submit a job, how can you tell whether the job
was properly submitted?
There are several general tricks that you can use to solve these
problems, though these are more examples of ingenious solutions for
you to emulate, not specific instructions that should always be
followed:
* Inputs: Have ways to "fake" hard-to-trigger input conditions,
like bad tapes, control-Y, I/O errors, etc. For instance, have a
"***BAD TAPE***" record in a file be interpreted by the program
as a tape error; if the program expects a tape to contain several
things separated by EOF markers (which can't normally be emulated
by disc files), have it treat an "***EOF***" as an EOF marker.
Again, this has the same problem as the PRETENDDATE/PRETENDONLINE
features that we suggested above -- what you'll really be testing
is not the actual execution of the program, but the execution of
the program in testing mode. However, though problems with, say,
the actual condition code check that detects the tape error will
not be found, all the other aspects of tape error handling will
be properly tested.
* Outputs: Find commands or programs that can convert an output
that is hard to test for into one that is easy to test for; for
instance, if your program is supposed to do a :DOWN, test it by
doing a :SHOWDEV afterwards to see if it is DOWNed or has a DOWN
pending. If the program is supposed to do an :ABORTJOB, PAUSE for
some time (since an :ABORTJOB may not immediately take effect)
and then do a :SHOWJOB of that job number to make sure that the
job no longer exists. If your program is supposed to shut down
the system, you're out of luck...
* More outputs: But maybe you're not out of luck in the system
shut-down case, after all; analogously to the "fake input"
suggestion above, you might have your program check to see if
it's in testing mode, and if it is, print a message instead of
shutting down the system (or doing something equally
uncheckable-for). Again, this won't make sure that the ultimate
operation is done properly (since it won't be done in this case
at all), but at least it'll make sure that all of the
preliminaries will be handled correctly.
* General: Find ways of taking care of timing windows; for
instance, if your program submits a job, a simple :SHOWJOB in the
test suite won't be a proper check (since a small job might have
finished by the time the :SHOWJOB is done), and having the job
build a file or leave some such permanent file won't work either,
since the job might still not have started up. Instead, your test
suite might build an empty message file and then make sure that
the job writes a record to this message file (possibly by setting
up a logon UDC for that user). Your test suite can then read the
message file, waiting until a record is written to it, no matter
when the job actually gets around to executing.
Message files are also quite useful when the test suite has to
check something at a particular point in the son program's
execution, and if it checks it too early or too late the results
will not be quite right. This is particularly so when your code
is supposed to properly handle concurrent access in a
non-standard way (i.e. not by simply using FLOCK/FUNLOCK or
DBLOCK/DBUNLOCK). You might want to have your programs, when run
in testing mode, try to read records from message file at
critical points, which will let you control when each program
will hit a particular piece of code.
Again, these are some sample solutions to some (though by no means
all) testing problems. The $65,536 question of testing, however, still
remains: How do you test VPLUS block-mode applications? Some of the
above tricks might be usable -- instead of calling the VPLUS
intrinsics, call procedures that, in testing mode, will do normal,
unformatted terminal I/O (i.e. the input fields are to be input simply
as a data string, with all the fields run together), which can then be
run under test-bench control. Unfortunately, it seems that this would
leave too much out of the testing (for instance, the correctness of
the VPLUS calls themselves, and the correctness of any edits specified
in the VPLUS forms), and the test suites would also be quite
unreadable and unwritable. Someone might do something to intercept the
terminal I/O from within VPLUS itself, but that's getting too
complicated for me. Any ideas?
CONCLUSION
To sum up, a few testing maxims:
* Automate testing -- both the input and the checking of the
output.
* Write test suites before or while you're writing the program --
that way, you can use them to do even the initial testing.
* Figure out the testing tools that you need and don't skimp in
building them; they can save you a lot of effort.
* Make it as easy as possible to add new test cases (try to make it
one test case per line), even if it means extra work up front.
* Have your test cases be in job streams, not in source files, so
that you can add new ones without recompiling.
* Change your programs so that they can "pretend" that today's
date, the batch/online flag, your logon information, and such,
are something other than what they really are. Do the same for
hard-to-reproduce conditions, like I/O errors, control-Y, etc.
* Think about your code and come up with test cases that exercise
as much of it as possible; as new bugs arise, add test cases that
would have caught them.
* Write verification routines for all your complicated data
structures, especially including the data in your files and
databases.
* If feasible, write some sort of test-bench program in which you
can test the behavior of other programs by feeding them input and
checking their output.
* Think creatively about testing features that at first glance seem
difficult to check the results of. Use message files to control
timing problems.
* Be prepared to spend a lot of time and effort (and therefore
money) on automated testing, but expect to save a lot more
effort, and come out with much fewer bugs, if you do it right.