THE TRUTH ABOUT MPE/XL DISC FILES
by Eugene Volokh, VESOFT
Presented at 1989 INTEREX Conference, San Francisco, CA, USA
(BEST PAPER AWARD For an Outstanding Paper Presentation)
Published by INTERACT Magazine, Sep 1989.
ABSTRACT
Several years ago, I wrote a paper called "The Truth About Disc
Files". In it, I tried to describe some aspects of files that, I
felt, inquiring minds wanted to know -- things like extent
considerations, blocking, file locking, etc. Some of those things
remained the same under MPE/XL's file system, but many have
substantially changed; this paper will try to describe some of the key
differences and similarities between the MPE/XL and MPE/V file
systems, and explain some of their practical implications.
HOW FILES ARE STORED -- EXTENT CONSIDERATIONS
One of the key limitations of MPE/V was that you had to know a file's
maximum size when you were building the file. If you guessed too low,
your programs would eventually abort with an END OF FILE error; if you
guessed too high, you could waste a lot of disc space.
Actually, technically speaking, you didn't have to know a file's true
maximum size; you could always build the file with a very large file
limit, e.g.
:BUILD MYFILE;DISC=1000000
and you'd be rather certain never to overflow it. The trouble, of
course, is that the file's disc space was not allocated on a simple
as-needed basis, but rather one extent at a time. Since a file by
default had a maximum of 8 extents the above :BUILD would build a file
that was split into up to 8 chunks of contiguous disc space; the very
act of building the file would allocate one chunk of this space, which
would occupy 1,000,000 / 8 = 125,000 (contiguous!) sectors. (Remember
that a sector is 256 bytes -- we used to say 128 words, but not any
more; since a word means 2 bytes on Classics but 4 bytes on Spectrums,
I will try to use "word" as infrequently as possible in this paper.)
Even if you said
:BUILD MYFILE;DISC=1000000,32
to build the file with up to 32 extents, the file would initially be
allocated with over 31,000 sectors. In other words, it wasn't so much
selecting the right file limit that was the problem, but rather that
selecting a file limit that was too high would cause prohibitive
consumption of disc space.
This may seem somewhat nitpicky but it is actually quite relevant to
MPE/XL. MPE/XL also requires you to specify the maximum size for a
file. However -- and this is a big "however" -- it lets you specify a
very large file limit without using huge quantifies of disc space. If
in MPE/XL you said
:BUILD MYFILE;DISC=1000000
the file would be allocated 2,048 sectors at a time, even if this
would require it to have almost 500 extents when full. Thus you get
the best of both worlds -- the file can grow to up to 1,000,000
records but will never have more than 2,047 sectors of wasted disc
space. You'll find that MPE/XL often builds files (e.g. XL's built
by LINKEDIT's -BUILDXL command) that have file limits of 4,096,000 --
more than they'll ever need but what does it matter? In fact, the
highest "maximum maximum file size" -- i.e., the highest (file limit *
record size) value that you can have -- is 8,388,607 sectors.
One of the reasons why MPE/V had the 32-extent limit was that the disc
addresses of all the extents were kept in the 256-byte file label.
Since each disc address was 4 bytes, and a little bit less than 128
bytes were required for other file information (e.g. file name,
creator id, file code, etc.), that left room for only about 32 extent
pointers.
MPE/XL didn't make the same mistake of keeping extent pointers in a
single fixed-size array. Instead, each file has a linked list of
"extent descriptor blocks", each of which has 20 12-bytes entries that
point to the extents of the file. Thus, a file on MPE/XL will have:
* a file label, which points to
* an extent descriptor block which contains the disc addresses of
20 extents and also points to
* a second extent descriptor block which contains the disc addresses
of 12 extents.
Granted, this is 3 sectors (as opposed to the 1 sector, which is all
that is needed for the file label on MPE/V), but think of the
flexibility -- new extents can be added to the file with no
difficulty. To avoid possible performance problems with access to
files that have many extents, MPE/XL builds a special "extent
descriptor B-tree" whenever you open a file; that way, it can very
quickly find the address of an extent even in a a many-hundred-extent
file.
All right you've said:
:BUILD MYFILE;DISC=1000000
and now you do a :LISTF MYFILE,2. What do you get?
FILENAME CODE ------------LOGICAL RECORD---------- ------SPACE-----
SIZE TYP EOF LIMIT R/B SECTORS #X MX
MYFILE 128W FB 0 1000000 1 0 0 *
There are, obviously, three unusual things in this picture:
* The number of sectors allocated to the file is 0. If you recall on
MPE/V, even an empty file always has at least one sector allocated
to it, and usually more. This is because on MPE/V, file labels were
kept as the first sector of the first extent of the file so each
file always had to have at least one extent allocated to it. In
MPE/XL, file labels are kept separately from the file data (in a
special portion of the disc called the "file label table" -- extent
descriptor blocks are also kept there), so no data extents are
allocated and thus 0 sectors are actually allocated for the data.
Of course, the file label still takes up 1 sector of space, but that
doesn't get budgeted to the file in MPE/XL.
* The number of extents allocated to the file is O. See above
paragraph.
* The maximum number of extents for the file is "*". This means
that the file was built without a maximum number of extents (though,
as we'll explain later, even if it were built with a maximum number
of extents, this number still wouldn't really be a maximum!). For
extra credit try to guess what an "*" in the "# of extents" (as
opposed to "maximum extents") column means; we'll discuss that
later.
Now, say that we write one record into this file and then do a
:LISTF. We'd see:
FILENAME CODE ------------LOGICAL RECORD---------- ------SPACE-----
SIZE TYP EOF LIMIT R/B SECTORS #X MX
MYFILE 128W FB 1 1000000 1 2048 1 *
One 2,048-sector extent has been allocated to this file -- it will be
enough for the first 2,048 records to be written into MYFILE. When we
try to write the 2,049th record, another 2,048-sector extent will be
allocated, and so on. As we mentioned before, this means that no more
than 2,047 sectors in this file will ever be wasted (allocated but
unused).
This is quite nice, but if each such file wastes an average of 1,000
sectors (an average between those that waste 0 and those that waste
all 2,047) and you have 2,000 such files, we're talking about 500
megabytes of wasted space about the size of one disc drive. Looking
at it this way, saying that "at most 2,047 sectors can be wasted" is
small comfort.
It would have been nice if we could build the file indicating what
we'd like its extent size to be; we might then build our files with
huge file limits but tell MPE/XL that they are to be allocated only,
say, 512 or 256 sectors at a time.
Unfortunately this is (to the best of my knowledge) not possible.
MPE/XL determines the size of the extents that it will try to allocate
for a file using a (rather bizarre) formula based on the maximum
number of sectors the file can ever contain:
MAXSECT = (userlabelflimit*256+recordsize*flimit)/(16*256)*16
(i.e. the total number of sectors in the file's data portion,
rounded up to the next multiple of 16 sectors)
DEFAULTEXTENTSIZE =
IF MAXSECT<= 127 then MAXSECT rounded up to the next highest
multiple of 16
IF MAXSECT<=255 then MAXSECT rounded down to the lowest
multiple of 16
IF MAXSECT<=4095 then 128 sectors
IF MAXSECT<=65535 then (MAXSECT/32) rounded down to the next
lowest multiple of 16
IF MAXSECT>=65536 then 2048 sectors
Don't ask me why this is the case -- it just is. (It certainly makes
some sense for extent size to vary as a function of the file size, but
I'm not sure why it varies exactly in this unusual way.)
Note, however that one other great new feature of MPE/XL is that if it
can't find extents of the size that it wants (e.g. there is no empty
chunk of 2048 extents), it will just allocate smaller extents. MPE/XL
will not report an "OUT OF DISC SPACE" condition unless there really
isn't enough disc space but it's reserved for virtual (transient)
memory. Disc fragmentation does not appear to harm things except that
it may make files be built with smaller extents.
So, this all shows (among other things) that any files with a maximum
of 65,536 or more sectors will have space allocated for them in
2,048-sector chunks. If you have one of those files how do you save
the 1,000 sectors or so that will, on the average, be wasted?
On MPE/V, you could always "squeeze" the file (FCLOSE it with
disposition 8 or MPEX %ALTFILE;SQUEEZE), which would set the file's
file limit to be equal to its end of file and thus save the wasted
space. Unfortunately, it would also prevent any more records from
being added to the file (unless you rebuild it or %ALTFILE;FLIMIT=
it).
MPE/XL has a very nice alternative to this that we call "trimming".
It lets you tell MPE (using a new FCLOSE disposition) to deallocate
any unused space allocated to the file without changing the file's
file limit. In other words, this will save space without making files
any less expandable; an operation such as MPEX's
%ALTFILE @.@.@; XLTRIM
can save you hundreds of thousands of sectors (it did for us and we
only have a 925LX with two disc drives). Actually, before doing this
we called PICS and asked whether there were any files that should not
be trimmed and they told us that they didn't know of any; I then did
the trim of all the files in the system and nothing seemed to fail.
Be warned though -- it's certainly possible that some program
(probably a heavily privileged one, since normal programs have no way
of knowing whether a file has been trimmed or not) doesn't like its
files trimmed.
How does this work? Well, say that you have a file with five
2,048-sector extents the last of which contains only 537 actual
sectors of data. When you tell MPE to trim the file, it will
deallocate the last 1,504 sectors of the last extent, leaving it with
only 544 sectors. (544 = 537 rounded up to the next highest multiple
of 16; files are always allocated in multiples of 16 sectors.)
Now, the file has four 2,048-sector extents and a 544- sector extent.
If you start adding more records to it, more 2,048-sector extents will
be allocated to it; you may then again want to trim the file.
What makes this whole process work is that MPE/XL allows you to have
extents of different sizes. If, like in MPE/V, all extents (except for
the very last of the possible extents) had to be the same size, you
wouldn't be able to throw away unallocated data because that would
leave the last allocated extent with a different extent size. MPE/XL,
however, is not bothered by this -- I've often seen files with many
different sizes for many different extents. The "extent descriptor
blocks" that I mentioned earlier actually contain several pieces of
information for each extent:
* the disc number on which the extent resides (actually, the volume
table index);
* the starting sector address of the extent;
* the size of the extent, in sectors;
* and, the sector number of the first sector of data (relative to the
start of the file) that resides in this extent.
Actually with a structure this flexible it's even possible for records
#0 to #999 to be located in the second extent of a file and record
#1000 to #1999 to be located in the first extent! (Of course when you
read the file, the records will come out in the right order, but
internally inside the extent descriptor blocks the extent information
will be kept out of order.)
What are the disadvantages of trimming files? To the best of my
knowledge, there are very few. Trimming files will increase disc
space fragmentation, but it's not clear to me that this is a problem
on MPE/XL, especially since MPE/XL seems to handle correctly
situations where it can't find extents of the size that it wants (if
necessary, it just allocates more smaller ones).
Trimming files does cause the file to have more extents, which may
have a slightly adverse effect on performance. The file system
usually tries to read 128 or more sectors (16,384 bytes) at a time, so
if a file has all 16-sector extents (the smallest size possible), you
will lose the advantages of these large I/Os since you'll never have
64 contiguous sectors. However, if a file has, say, 512-sector
extents rather than 2,048-sector extents, this should cause minimal
performance penalties (if any at all).
On the other hand, if you feel that a trimmed file isn't giving you
the performance you'd like, you can copy its data into a new copy of
the file, and all the extents will then be the same size (whatever the
extent-size algorithm we showed above dictates). You can even build
the new file with a higher file limit (to increase the extent size)
and then trim the file to save as much space from the last extent as
possible. MPEX users can do this by saying
%ALTFILE filename;FLIMIT=4096000;XLTRIM
-- this will rebuild the file to have all its extents be 2,048 sectors
each except for the last one, which will only be as large as
necessary. This will give you the maximum disc space savings as well
as the maximum possible extent sizes (if that's what you want).
(Note that files with EOF = FLIMIT do not require trimming; the MPE/XL
file system automatically allocates the last extent to be just large
enough to fit all the records up to the FLIMIT, even if this makes the
extent smaller than the other extents.)
In light of all this, why does MPE/XL still support the maximum number
of extents parameter of the FOPEN intrinsic and the :BUILD command?
Well, compatibility is one reason. There are programs out there that
might, for instance, do an FGETINFO or FFILEINFO of a file's maximum
number of extents -- they should be able to get the value specified
when the file was :BUILDed (:BUILt?). In fact, MPE goes so far as to
return 8 as the maximum number of extents when you do an FFILEINFO for
a file that was built without a maximum number of extents -- all this
just to make sure that the program will get a value (though an
incorrect one) that it will be able to handle.
Another reason involves moving files from MPE/V to MPE/XL and back.
If you move a file from MPE/V to MPE/XL, it will have the same
"maximum extents" value that it did on MPE/V (even if it will be
ignored by MPE/XL). Then, if you move the file back to an MPE/V
system, it will have the same maximum-extents value that it originally
had. If the MPE/XL file had no maximum extents, MPE/V will select a
"reasonable" value for this (based on the number of sectors the file
uses).
Finally, the maximum number of extents is used (though in a very
strange way) when you build a file specifying both the maximum number
of extents and the number of extents to initially allocate. For
instance, say that you enter
:BUILD MYFILE;DISC=100000,32,4
What do you suppose will happen? here's what a :LISTF of the file
will show:
FILENAME CODE ------------LOGICAL RECORD---------- ------SPACE-----
SIZE TYP EOF LIMIT R/B SECTORS #X MX
X 128W FB 0 1000000 1 12512 1 32
The file was built with 1 extent of 12512 sectors! MPE/XL decided
that what you wanted is a file with 4/32nd (= one eighth) of its space
allocated, so it built you one like that, although with all that space
allocated as one contiguous extent. From then on, the file will be
allocated in normally-sized (in this case, 2,048-sector) chunks.
Eventually, the file will need more than 32 extents (this file would
if full, need more than 40), and MPE/XL will just blithely ignore the
maximum extents value and allocate as many as it needs. Thus, you'll
often see files with more extents than the maximum (rather perplexing
when you first see them).
Incidentally, to answer the question we asked earlier: if a file has
100 or more extents, MPE/XL will show an * in the number of extents
column of a :LISTF ,2 listing.
Finally, remember that IMAGE databases must still be fully allocated
when they are built -- I believe that IMAGE does this not because of
any file system limitation but rather for data consistency's sake; it
doesn't want to run out of disc space in the middle of a DBPUT. I
have, however, heard a hot rumor that a future version of
TurboIMAGE/XL will allow a detail dataset to be expanded when it runs
out of space (so that you can initially allocate it with less space
than it will eventually need); but (or so the rumor says), each
dataset can only be expanded once in its life -- once it's expanded,
it better not overflow again!
Seems bizarre but that's what I've heard. Believe it or not.
HOW FILES ARE STORED -- BLOCKING CONSIDERATIONS
In discussions of MPE/V, much was said about blocks, blocking factors,
and their effects on speed and disc space. Just when we had all taken
the time and trouble to learn all of their intricacies, MPE/XL has
made them (almost) completely irrelevant.
In MPE/XL, all physical disc I/O is done in multiples of one page,
which is 4096 bytes. This is not to be confused with the pages that
are 2,048 bytes -- yes, that's right, there are two kinds of pages,
each of different size, and both of which are called pages. One,
which is 2,048 bytes long, is the unit in which the hardware sees the
world; the other which is 4,096 bytes long, is the unit used by the
operating system from the memory manager on up (including the file
system).
In any event, physical I/O is done some number of 4,096-byte pages at
a time (it's good that it's a 4,096-byte page because you want to do
I/Os in fairly large chunks). Since each 4096-byte page consists of
16 256-byte sectors, a file is always allocated, read, and written in
multiples of 16 sectors at a time.
Remember that the whole point of MPE/V blocks was that a block was the
unit of file transfer for this particular file. This may have made
sense in the very earliest HP3000s, which often had as few as 128
Kilobytes of memory, and which couldn't afford huge chunks of this for
file buffering.
However, since on MPE/XL file transfer is always done 4,096 bytes at a
time, the concept of a block becomes irrelevant. Each page has as
many records in it as will fit; there are no inter-record gaps (as
there used to be on MPE/V when the block size was not a multiple of
one sector). In fact, records can even straddle pages -- if your
file's records are 1,000 bytes long, then
* the first 4,096-byte page will have 4 full records and the first
96 bytes of the fifth record;
* the second 4,096-byte page will have the last 904 (1,000-96) bytes
of the fifth record, the next 3 full records, and the next 192
bytes of the ninth record;
* and so on.
There's never any space wasted in a page (except of course, in the
allocated-but-not-written portion of the last extent) -- not because
of bad blocking factors and not even because of records with odd
record length. If you build an ASCII file with 1-byte records,
exactly 4,096 of them will fit into each 4,096-byte page.
A curious thing, incidentally, is the lengths to which MPE/XL must go
to make this efficient, reasonable, straightforward system compatible
with MPE/V's baroque and inefficient mechanisms. If you read an
odd-record-length file MR NOBUF, MPE/XL will actually insert padding
bytes at the end of each record to be compatible with MPE/V; when do
you an MPE/XL :STORE;TRANSPORT of a file whose blocks (in MPE/V)
wouldn't be multiples of 256 bytes, MPE/XL will also insert padding at
the end of each block to correspond to MPE/V's inefficient
end-of-block padding.
The blocking factor of a file is, like the maximum number of extents
of a file, specifiable but largely ignored. It's relevant only for
compatibility, for transporting files to MPE/V machines, and for NOBUF
file accesses, in which a program written for MPE/V would expect to
get data in units of MPE/V blocks.
OTHER DISC SPACE CONSIDERATIONS
So, if MPE/XL can build large files without wasting space and do file
blocking more efficiently and trim wasted space without changing file
limits, one question remains: Why does it use so much disc space?
There is one philosophical explanation (it's somebody or other's Law,
but I forgot whose): "The amount of disc space required will increase
until it meets and exceeds the amount of disc space available". This
is actually not just a facetious statement; as disc space use
algorithms become more efficient and disc space becomes more
plentiful, people will take advantage of this by building more and
more files that are larger and larger. You'll get more bang for a
buck's worth of disc space, but eventually you will exhaust it all the
same.
There are, however a few more pragmatic explanations:
* The operating system uses much more disc space than on MPE/V.
The groups MPEXL.SYS and PUB.SYS use 570,000 sectors (150
megabytes!) on our 925/LX -- PUB.SYS on our MICRO/3000 uses only
100,000 sectors.
* Code -- programs and SLs -- uses a lot more space than it did
before (this is actually a big part of the reason why the
operating system uses more disc space).
Why is this the case? Well, remember that all this "Reduced
Instruction Set" means that it takes several RISC instructions to do
the job of one Classic instruction. Thus, a program of 10,000 16-bit
Classic instructions might be replaced by one of 50,000 32-bit RISC
instructions -- a ten-fold increase. This is true of Native Mode code
and of OCTCOMPed code. Compatibility Mode code still takes the same
amount of space as it did under MPE/V.
* Although trimming files is possible, to the best of my knowledge,
few things in MPE do it routinely. Compatibility mode USLs, it
seems, are pretty substantial culprits (using far more space than
they would if trimmed), and other files should probably be
periodically trimmed, too.
Thus, our recommendations for saving disc space would be:
* Purge old unused files. This #1 space-saving feature from MPE/V
days is still as important as ever on MPE/XL (and will probably be
for a long time to come). Discs inevitably get filled up with
junk, data that the owner no longer uses, no longer wants, and has
probably already forgotten about; not only does it waste disc
space, but it also makes your full backups take more time and more
tapes. If you periodically archive and then purge all the files
(except, say, IMAGE datasets) that haven't been accessed in 120
days, you will save a lot of a disc space with minimal user
complaints
* Trim files (e.g. using MPEX's %ALTFILE @.@.@;XLTRIM) periodically.
As I mentioned before, trimming seems to be safe for all files in
the system.
* Remember that native mode and OCTCOMPed program files are now big
disc space hogs -- multiple unneeded copies of programs (which
used to be rather harmless on MPE/V) may now substantially
contribute to your disc space problems.
MAPPED FILES
Mapped files have been heralded (and correctly so) as a powerful and
valuable new feature of MPE/XL. It has been discussed in a number of
places, including chapter 11 of HP's "Accessing Files Programmer's
Guide", and also, coincidentally, chapter 11 of SRN, Inc.'s excellent
"Beyond RISC!" book. (I heartily recommend Beyond RISC! to anybody
who's at all interested in Spectrums -- call SRN at 206-463-3030).
At the RISC of beating a dead horse, I d like to go over some of the
key points of mapped files in this paper, too.
First of all, a Mapped File is actually not a type of file but rather
a type of file access. Almost any file can be opened as a mapped
file; once your program opens a file with the mapping option, it will
be able to access the file as if it were an array in its own data
area. Instead of accessing a file using FREAD and FWRITE (or the
equivalent language constructs, such as PASCAL's READLN or WRITELN),
you'll be able to access the data of the file just as you'd access any
array (or record structure).
MPE/XL will, behind your back, realize that this isn't a normal array
but is rather a mapped file; whenever you access a piece of this
array, MPE/XL will, if necessary, go out to disc to get the
appropriate data. (This is actually true for your stack, data
segments, etc., as well, but it's especially important for mapped
files.)
(Not very important note: Actually, the file system opens all files
with mapped access for its own internal purposes; however, when I talk
about mapped file access , I refer to file access that is mapped from
the users point of view.)
Let's look at what might be the perfect application for mapped files
keeping a large array of data that must survive from one execution of
a program to another.
Say that you have a large number of payroll codes (numbered, say, from
0 to 99), each of which has various attributes (such as code name, pay
scale, tax identifier, etc.) that your program must know about. Your
program has to look payroll codes up in this file and extract the
relevant data.
Without mapped files, here's what your program might look like (in
PASCAL):
TYPE PAYROLL_CODE_REC = CRUNCHED RECORD
CODE_NAME: PACKED ARRAY [1..20] OF CHAR;
PAY_SCALE: INTEGER;
TAX_ID: INTEGER;
...
END;
VAR PC_REC: PAYROLL_CODE_REC;
...
FNUM:=FOPEN (DATAFILE, 1 (* old file *));
...
FREADDIR (FNUM, PC_REC, SIZEOF(PC_REC), PCODE);
IF PCODE.TAX_ID=... THEN
...
With mapped files, you'd say:
TYPE PAYROLL_CODE_REC = CRUNCHED RECORD
CODE NAME: PACKED ARRAY [1..20] OF CHAR;
PAY SCALE: INTEGER;
TAX ID: INTEGER;
...
END;
PAYROLL_CODE_REC_ARRAY = ARRAY [0..99] OF PAYROLL_CODE_REC;
VAR PC_FILE_PTR: ^PAYROLL_CODE_REC_ARRAY;
...
DOMAIN:=1 (* old file *);
HPFOPEN (FNUM, STATUS, 2, FILENAME, 3, DOMAIN, 18, PC_FILE_PTR);
...
IF PC_FILE_PTR^[PCODE].TAX_ID=... THEN
...
Instead of doing an FREADDIR (using PASCAL's READDIR statement), we
directly access the file as if it were an array. The HPFOPEN call
(more about its unusual calling sequence later) indicates that the
file is to be opened mapped and that the pointer PC_FILE_PTR is set to
point to its data; then, whenever we refer to PC_FILE_PTR^ we get
access to the entire file as an array of records.
Why would we want to use mapped files? One reason is convenience --
in a situation like this one, it's easier (and makes more sense in
light of the logic of the program) to view the file as an array rather
than as a file. Instead of having to do a READDIR every time we want
to get a record, we just access the record directly.
Another reason is performance. Avoiding the extra READDIRs not only
makes the program smaller and cleaner, but also saves the CPU time
that would otherwise be taken by each READ, READDIR, WRITE, or
WRITEDIR. Each file system intrinsic (which are ultimately called
READ, READDIR, WRITE, and WRITEDIR, and all the similar constructs in
the other languages) has to do a lot of work finding control blocks,
checking file types, etc., even before a disc I/O is actually done.
This can take many thousands of instructions, amounting to up to a
millisecond per call (or more). Access to a mapped file can take as
little as one instruction one memory access.
As we will discuss later, mapped file access actually has some
performance penalties, too, especially when we're doing sequential
accesses to files that are not likely to be already in memory It is
actually quite possible with mapped file access to lose much more on
disc I/O increases than you would gain on CPU time savings. However,
if you're accessing files that are already likely to be in memory --
which often includes many heavily-accessed files -- mapped I/O can
give you very large performance gains (again, more about that later).
Beyond convenience and optimization, I think that there are many more
very interesting things that mapped files can let us do things that
have rarely been contemplated in the past precisely because they were
so difficult to do in the past. There is one idea that I have along
these lines; I've never tried it in a production program, but I feel
that it could very well prove quite useful.
One of the things that mapped files can give us is shared variables.
By this I don't mean global variables that are shared among all the
procedures in a program, but rather variables that are shared among
multiple programs and processes.
For example, let's say that you have a program that runs in a job
stream. The program might run for a long time, and you may want to
check on its progress -- see which phase of processing it's in, what
was the last record it processed, and so on.
With mapped files, you can do the following:
* Keep some crucial variables -- the current processing phase, the
current record being processed, etc. -- in fields of a record
structure. (This is a bit more complicated than having them all be
separate variables, but not much.)
* Have the record structure be associated with a mapped file
by HPFOPENing the file (with shared access) and using the pointer
that HPFOPEN returns as a pointer to the record structure.
* Have another program that you can run online that will open the
mapped file and print its contents for you.
Whenever the background program modifies one of the fields of this
mapped-file-resident data structure, the field will be automatically
updated in the file (even though this almost certainly require, on the
average, far less than one disc I/O for each field modification).
Then, the online program can at any time look at the contents of the
file and tell you what's going on; and, if the batch program aborts,
you'll be able to see where it was in its processing when it aborted
(since the data is saved in the permanent mapped file).
This would also be an excellent tool if you'd like to write a debugger
for some interpreter program that you have. As long as the
interpreter keeps all its control variables in a mapped-file-resident
area, then a debugger program (running in the same session or in a
different one) can look at these variables and figure out exactly what
the interpreter is doing. It can even change the variables, for
instance, setting some debugging flag, changing the value of a user
variable, or whatever; and, if all the important data is actually kept
in this file, it would permit dump analysis in case the program
aborts, and even interruptions and restarts (since the entire state of
the program would be automatically saved).
Another possible application is to have a program periodically (e.g.
for every record that it processes) check a mapped-file-resident
variable and terminate cleanly if it is set. Then, if we want to
terminate all the processes running this program, we just set the
variable, and all of them will stop. (Something like this could be
done before with message files and soft interrupts, but it would
require one record to be written to the message file for each process
accessing it.)
Of course, this could all have been done before mapped files instead
of accessing the mapped-file-resident variables directly we could just
do FREADs or FWRITEs to read or write the appropriate record from the
file. However, this would have been prohibitively expensive and clumsy
-- imagine that you had to do an intrinsic call every time you wanted
to access a particular variable; it would badly slow things down and
make your program much more complicated. As I said, all of the above
are relatively untested ideas, but I feel that much can be gained by
doing something along those lines.
The really sad thing about mapped files -- something that I think is
likely to drastically reduce their utility -- is that they can only be
accessed from PASCAL/XL, C/XL, and SPLash!. FORTRAN/XL and COBOL/XL
programs cannot access mapped files, not because of any file system
limitation, but because those languages do not support pointers In
FORTRAN and COBOL, all variables are preallocated when you run the
program or enter a procedure; to use mapped files you have to be able
to assign a particular address to a variable.
Actually, if you really wanted to use mapped files from FORTRAN or
COBOL, you could write a PASCAL, C, or SPLash! procedure that lets you
access pointers; however, this would most likely cancel any
convenience advantages that mapped files can give you.
A few other notes about mapped files -- they're all documented in
various places, but they're worth repeating:
* There are two ways of opening a file for mapped access -- "long
mapped" and "short mapped". Long-mapped access lets you access any
size file but requires you to use long (64-bit) pointers; in PASCAL,
they have to be declared as $EXTNADDR$. Short-mapped access only
lets you access a file of at most 4 megabytes; furthermore, you may
have no more than 6 megabytes worth of short-mapped files open for
each process. On the other hand, short-mapped access lets you use
32-bit pointers, which are faster to operate with than the 64-bit
ones.
* Because of the restriction on short-mapped file size and the fact
that you can't open a file short-mapped if it's already opened by
somebody else without mapping, your general-purpose programs (e.g.
copiers, editors, etc.) should probably open files long-mapped
rather than short-mapped -- it seems to be a more versatile, less
restrictive access method.
* You can not access RIO, CIRcular, or MSG files as mapped files; you
can access variable-record-length and KSAM files, but you'll see
their internal structure (i.e. what you'd see if you read a variable
file NOBUF or a KSAM file with COPY access) rather than their normal
appearance.
This may not seem to be such a big problem, and it often isn't;
however, I've found that one of the most useful features of the MPE
file system is its ability to substitute one type of file for
another -- for instance, give a message file to a program that
expects input from a standard file, or a variable-record-length file
to a program that expects input from a fixed-record-length file.
This interchangeability will be lost for files that you open with
mapped access.
* Remember that writing to a mapped file only writes the data; it does
not increment the EOF. Even if you write data that ends up in record
1000 of the file, if the EOF is 200 it will stay 200. You have to do
an FPOINT to the right record and then an FCONTROL mode 6 (as
documented by the Accessing Files manual and by the Beyond RISC!)
book to set the EOF to the right place.
An interesting aspect of this is that the data that you write beyond
the EOF will actually be written there and will remain readable the
next time you open the file for mapped access. However, it will not
be readable when you open the file normally, and will almost
certainly disappear if the file is copied, :STOREd/:RESTOREd, or
%ALTFILE;XLTRIMed.
Thus, if you write to a mapped file and forget to adjust the EOF,
your programs might very well keep working just fine until the the
file is next :STOREd/:RESTOREd or %ALTFILE;XLTRIM. You can get some
truly bizarre bugs that way.
Don't even dare think that this is a useful feature and try to
exploit it (e.g. to have some place to put hidden data that will
appear not to actually be there)! Imagine trying to maintain or
manage a system on which seemingly empty files were actually
chock-full of data.
* If you have processes share a mapped file, you may have to do
appropriate locking to prevent problems (especially if you have more
than one person writing to the file). For instance, you might write
all your programs so that they FLOCK the file before making any
updates to it; unfortunately, this will make your program more
complicated after all, the whole point was to treat the file data
just as if it were normal program variables. Furthermore, the very
fact that it's so easy to modify one of these shared variables (just
assign to it or pass it as an output parameter to a procedure) may
make it easier for you to forget to put in an FLOCK in the right
place.
HOW THE FILE SYSTEM DOES I/O
One rule that we learned under MPE/V is: always do disc I/Os in as
large chunks as possible. If a file has 256-bye records, don't read
it from (or write it to) disc one record at a time; read it ten
records at a time or, even better, thirty or sixty at a time.
The reason for this was, of course, that as the transfer count (the
number of words of data read or written) on a particular disc I/O
increases, the time to do the disc I/O increases much more slowly.
Thus, it might take you 30 milliseconds to read 256 byes, but 100
milliseconds to read 8192 bytes; if you were planning to read those
8192 bytes anyway (and weren't just going to read one 256-byte
record), you could read them ten times faster by reading them in one
8192-byte chunk than by reading them in 32 256-byte chunks.
Furthermore, you'd incur the CPU overhead (which can be pretty
substantial) of only one FREAD call rather than of 32 FREAD calls.
On MPE/V, the file system would always do disc I/Os in units of one
block. The default blocking factor (the number of records per block)
was usually not well-chosen by the operating system; for instance any
file whose record size was 65 words or more would, by default, have a
blocking factor of 1. This might have made sense in the earliest
HP3000s (on which memory was a very scarce resource), but not on
8-Megabyte series 70s, which tended to end up being quite disc
I/O-bound.
Thus, on MPE/V the recommendation was to raise the blocking factors of
MPE (and KSAM) files that you frequently access, especially serially;
this could save you a large fraction of the file's I/Os (increasing a
blocking factor from 1 to 10 could cut by 90% the number of I/Os
needed to read the file).
When disc caching was introduced, this became somewhat less important,
since the file system would pre-read from 16 to 96 sectors (4K bytes
to 24K bytes) whenever you'd do a serial disc I/O; thus even a file
with a low blocking factor could be read with relatively few disc
I/Os. However, it still paid to have the blocking factor be high since
going to cache was still more expensive than getting the record from
the file system buffer (though not as expensive as going to disc).
Finally beyond increasing the blocking factor, it was often a good
idea to read or write the file NOBUF (so that each FREAD returned an
entire block) or MR NOBUF (so that each FREAD returned several
blocks). Reading a file NOBUF caused you to do the same number of disc
I/Os (since the file system also read the file a block at a time);
however you would save the CPU overhead of all those FREADs (which
could be quite a lot). Reading a file MR NOBUF was even better, since
it let you do even fewer disc I/Os (though increasing the blocking
factor to a high enough value and then using plain NOBUF or even
normal access could accomplish the same purpose).
The trouble with reading files NOBUF or MR NOBUF is that your program
had to do its own deblocking , i.e. it had to, by itself, separate
each record in the block from the next not a very difficult task, but
not a trivially easy one, either.
To summarize (again, remember that this is on MPE/V), here are the
ways you might read a file of 1024 256-byte records (depending on the
file's blocking factor and access method):
Blocking factor Type of access # of disc I/Os # of FREAD calls
1 Normal 1024 1024
4 Normal 256 1024
16 Normal 64 1024
16 NOBUF 64 64
16 MR NOBUF 32 32
reading 8192
bytes at a time)
OK, enough re-cap. What's new in MPE/XL?
Well, the good news is that all file system disc I/O is now done in
units of several 4,096-byte pages, often 8 pages (32,768 bytes),
though the number seems to vary rather unpredictably. That's a lot of
data (probably close to the optimum from the disc drives point of
view, since at some point the beneficial effects of reading larger and
larger chunks of data will peter out), and it will substantially
decrease the amount of disc I/O that will be done. Of course what
makes this possible is all those megabytes of memory that you had to
buy to make your Spectrum run; they allow HP to go straight after
performance optimization without having to optimize memory usage as
well (or so we hope).
This means that blocking factors are now quite irrelevant to file
performance (just as they are, as we mentioned before, irrelevant to
disc space usage). You may set them high or set them low, but the file
transfer unit will not change.
The bad news is that each FREAD and FWRITE call still takes a good
deal of time, about .25 or so milliseconds running stand-alone on my
925/LX. (This may not seem like much, but remember that not everybody
gets to use a Spectrum as a personal computer! On heavily-loaded
systems the FREADs and FWRITEs will take even longer to execute, and
will adversely impact other users response times.)
What can we do about all this file system CPU overhead? Well, we could
access the files NOBUF or MR NOBUF the MR NOBUF would now be needed
not so much to decrease disc I/Os (which at one I/O per 16,384 bytes
can't really be decreased much further) as to decrease the number of
file system calls.
Alternatively, we could access these files as mapped files. Once we
open the file, we could then access all the data in the file using
simple memory accesses -- when a disc I/O is required, it'll be done
for us by the memory manager, but no file system overhead will be
required!
The only problem -- and this is a really big one -- is that, together
with the substantial CPU time savings that mapped files give us,
they can also substantially increase the amount of disc I/O that is
done. While the file system accesses the disc several 4,096-byte pages
at a time (my observations showed me that it usually accesses 8 pages,
or 32,768 bytes, in one shot), the memory manager (and thus mapped
files) accesses the disc only 4,096 bytes at a time. Thus, while we
can totally eliminate file system CPU overhead by using mapped files,
we could at the same time quadruple the amount of disc I/O that needs
to be done!
Now, as it happens, this disc I/O increase only becomes an issue if
the file is not already in memory; to the extent that it is in memory
(and many parts of your most heavily used files will be), the disc I/O
size is irrelevant because no disc I/O will be needed. However, if a
file is entirely or largely not in memory, you could suffer a very
serious performance penalty by using mapped files.
To revisit our little table of the ways you can read a file of 1024
256-byte records, this time on MPE/XL (we'll assume that the file's
blocking factor is 32 -- it's quite irrelevant except for NOBUF
access):
Type of access Maximum # of disc I/Os # of FREAD calls
Normal 8 1024
NOBUF 8 64
MR NOBUF (reading 16384 8 16
bytes at a time)
Mapped 64 0
(This assumes that the file system reads 8 4,096-byte pages at a time,
something that experiments on MPE/XL version 1.2 seem to indicate.)
Of course, the actual number of disc I/Os will vary depending on how
much of the file is in memory. Stan Sieler of Allegro Consultants
(co-author of Beyond RISC! and one of the foremost experts on MPE/XL
and the RISC architecture) ran some experiments that showed that a
mapped read of a file that was 100% non-memory-resident took more than
3 times the elapsed time (though only about half the CPU time) of a
file-system read; a mapped read of a 100%- memory-resident file took
less than 1/9th the elapsed and CPU time of a file-system read.
The moral of the story:
* Blocking factors are no longer relevant to performance.
* NOBUF and MR NOBUF can still be a good idea.
* Mapped file access is much faster for memory-resident files, much
slower for non-memory-resident files.
Finally (as Stan Sieler discusses in his paper and as we'll discuss
more in the NM FILES VS. CM FILES chapter), KSAM access can be
faster from Compatibility Mode than from Native Mode -- it's faster
still from OCTCOMPed code.
Oh, yes, one other thing: FOPEN calls are much faster on MPE/XL
than they were on MPE/V (they typically take from 25 to about 100
milliseconds running stand-alone on our 925/LX, compared to about 300
to about 500 milliseconds on a stand-alone Micro/3000). This may not
seem like much, but this can be very important for programs that open
some files, do a few checks, and then terminate (e.g. logon UDC
security programs). These programs can now take a lot less time than
before.
CM FILES VS. NM FILES
Every so often, you'll hear people talk about CM (Compatibility Mode)
files and NM (Native Mode) files. There are a few things that are
worth saying about this distinction.
The first thing that might come to mind is that a CM file is somehow
accessible only from CM and an NM file only from NM. This is not so;
both kinds of files are equally accessible from both modes (and, of
course, from OCTCOMPed code, too); in fact the access is completely
transparent -- nothing behaves any differently (at least externally)
from one mode to the other.
The distinction between CM files and NM files is purely internal. CM
files are those for which the internal file system code is implemented
in CM. KSAM files, message files, circular files and RIO files -- the
code that handles these files has simply never been rewritten by HP in
PASCAL/XL; whenever you access these files MPE/XL will execute the CM
code that pertains to these files even if this requires switching from
NM to CM. NM files of course are those whose internal code is
implemented in NM -- they include all the "vanilla" files, including
both fixed and variable-record length files and IMAGE databases
The main way in which this CM/NM difference manifests itself is in
speed of file access. As we said, if you try to access a CM file from
NM (or an NM file from CM), the system will have to switch into the
other mode in order to execute the appropriate file system code.
In addition to the switch from, say your NM program to the CM file
handling procedures, the CM file handling procedures will then have to
switch to the NM internal file system procedures to do the actual file
I/O. All these switches can take a non-trivial amount of time; for
instance it took a CM program more than 23 times longer to read a
circular file than an otherwise identical non-circular file; however
even with this the CM program ran 20% faster than an NM program
reading the same circular file! A bizarre incident indeed -- NM code
running slower than CM code.
This would not be that much of a problem if the only files that were
slower in NM were message files, circular files, and RIO files --
after all, how much are these rather esoteric file types used in
production? Unfortunately, the same thing applies to KSAM files,
which can indeed often be quite performance-critical. My tests (and
Stan Sieler's as well) showed that KSAM file accesses from NM were
over 10% slower than from CM and about 20% slower than from OCTCOMPed
code.
This might very well mean that KSAM users ought not migrate their
programs to Native Mode for now (presumably, HP will come out with an
NM KSAM soon). It seems that converting to NM will slow your KSAM file
accesses down by about 20% (compared to OCTCOMPed code -- if you're
still in CM, you should probably be OCTCOMPing all your code); you'll
have to balance this against whatever performance improvement you
expect to get on your other, non-KSAM-file-access code.
HPFOPEN
A number of the new features of the MPE/XL File System (including
mapped file access and a few others that we'll talk more about
shortly) have been implemented in the new HPFOPEN intrinsic, a
successor to the old well-loved FOPEN.
Why a new intrinsic? Because the old FOPEN intrinsic, with its limited
number of parameters (13 of them), just didn't have enough room for
all the data that needed to be passed. By the time MPE/XL came around,
* 14 of the 16 foptions bits and 13 of the 16 aoptions bits were
used up;
* The "device" parameter was actually used to pass no less than 4
different values (the device, the environment file, the tape
density and the ;VTERM parameter);
* The "forms message" parameter was used to pass 3 different values
(the forms message, the tape label, and the KSAM file
characteristics).
The MPE/V designers squeezed every last bit (almost) out of the FOPEN
intrinsic because it was designed in an inherently non-expandable way;
there was no way HP could have fit in the new parameters required to
support the new features of the MPE/XL file system.
Much like the CREATEPROCESS intrinsic supplanted the CREATE intrinsic
before it, HPFOPEN was designed to be a much more expandable (albeit,
in some respects harder to use) version of FOPEN. The general calling
sequence of HPFOPEN is
HPFOPEN (FNUM, (* 32-bit integer by reference *)
STATUS, (* 32-bit integer by reference *)
ITEMNUM1, (* 32-bit integer by value *)
ITEM1, (* by reference *)
...
ITEMNUMn, (* 32-bit integer by value *)
ITEMn,); (* by reference *)
HPFOPEN takes as input a list of item numbers and item values; it
returns the file number and the error status. A typical call might be
(naturally we hope that you define constants for the item numbers --
2, 3, 11, 12, 13, etc. -- and for the possible domain, access type,
exclusive state, etc. values):
LOCK:=1;
DOMAIN:=1 (* old *);
ACCESS_TYPE:=4 (* input/output *);
FILENAME: = '/MYFILE.MYGROUP.MYACCT/';
EXCLUSIVE:=3 (* shared *);
HPFOPEN (FNUM, STATUS, 2, FILENAME, 3, DOMAIN, 11,
ACCESS_TYPE, 12, LOCK, 13, EXCLUSIVE);
IF STATUS<>O THEN
PRINTFILEINFO (0);
The same call with the FOPEN intrinsic would be:
FIlENAME:='MYFILE.MYGROUP.MYACCT ';
FNUM:=FOPEN (FILENAME, 1, OCTAL('344'));
IF CCODE<>2 (* condition code equal *) THEN
PRINTFILEINFO (0);
As you see, the HPFOPEN intrinsic call is actually rather more verbose
than the FOPEN intrinsic call, and may be argued to be harder to
write, especially since you actually have to declare the variables
DOMAIN, LOCK, ACCESS_TYPE, and EXCLUSIVE (since they, like all HPFOPEN
item values, must be passed by reference). On the other hand, it does
keep you from having to remember what all the positional parameters
are (quick -- which FOPEN parameter is the file code?). More
importantly, HPFOPEN lets you do things that FOPEN won't:
* Open a file for mapped access (items number 18 and 21).
* Open a file given the entire right-hand side of a file equation
(item number 52). This way you don't have to worry about all the
other items or any magic numbers -- just say something like:
FILEEQ:='%MYFILE.MYGROUP.MYACCT,OLD;ACC=INOUT;SHR;LOCK%';
HPFOPEN (FNUN, STATUS, 52, FILEEQ);
This can be a lot cleaner than the normal HPFOPEN approach,
especially if all the parameters are constant (rather than having
one be a variable in which case you'd have to assemble the FILEEQ
string using a STRWRITE or, in C, an sprintf). Unfortunately, not
all file parameters are supported with this syntax -- exceptions
include mapped files, user labels, disallowing file equations and
several others.
Note that the value of FILEEQ started and ended with a "%"; it
actually didn't matter what character it started and ended with as
long as it was the same character. Rather than rely on terminators
such as blank, semicolon, or whatever, HPFOPEN lets you specify your
own string terminator as the first character in the string. In
almost all the HPFOPEN items that are strings (including the
filename parameter itself) must be passed this way.
* Open a file as a new file and immediately save it as a permanent
file (item number 3, value 4); this avoids the MPE/V headache of
having an FOPEN succeed and then at the very end of the program
having the FCLOSE fail because a file with this name already exists.
* Specify, when a file is opened, what disposition it is to be closed
with (item number 50). In other words, if you open a file that you
know should be purged when you're done with it, you can indicate
this on the HPFOPEN call; then, even if the program aborts before
FCLOSEing the file, the file will get deleted.
* And a few other, less important things. In the future, though, new
features that are added to the file system will be added through the
HPFOPEN intrinsic, not through the already overloaded FOPEN, so this
list will probably grow with time.
One other important point: how are you to interpret the STATUS
variable that HPFOPEN returns to you? The manual tells you that the
low-order 16 bits are always 143 (the code indicating that this is a
File System error) and the high-order 16 bits are values from a
16-page table in the Intrinsics Manual. Naturally, rather than
referring the user of your program to this manual, you really ought to
format the message yourself, using the HPERRMSG intrinsic:
HPERRMSG (2, O, O, STATUS);
Easy enough to do, but I'll bet you that half the programs you run
won't do this. If they don't and you have MPEX Version 2.2.11 or
later, you can just say
%CALC WRITEMPEXLSTATUS (statusvalue)
and get the text of the error message. Of course both the HPERRMSG
call and the %CALC WRITEMPEXLSTATUS will work for all
"MPE/XL-standard" 32-bit status results.
Finally, a few other interesting points:
* Whenever you create a file using HPFOPEN and do not specify a file
limit, it will be built with a file limit of 8,388,607 records
(not the measly little 1,023 that are the default on MPE/XL).
This may be a good idea in theory, but in practice it means that
you must dose your files with disposition 16 (the "trimming"
disposition) since otherwise your file will be allocated in chunks
of 2,048 sectors each, so you could easily have some 2048-sector
files with one or two records.
* It has been said that HPFOPEN is not callable from CM programs.
HPFOPEN is indeed an NM procedure so it can't be called from CM
programs as simply as, say, FOPEN can be; however, using
HPSWTONMNAME (or HPLOADNMPROC and HPSWTONMPLABEL), one can
relatively easy switch directly to HPFOPEN, passing to it whoever
parameters you please. You don't have to write any native mode
code to do this, nor do you have to put anything into any SLs or
NLs -- it's a bit trickier than a direct call, but not by all that
much.
It is, however, true that there is no (documented) way of directly
manipulating virtual pointers in CM, so mapped file access from CM
is pretty much out.
ACCESSING FILES
While internal file structure and disc space considerations have
changed dramatically with MPE/XL, the rules for accessing and sharing
files have not (except for the addition of mapped files and the
decrease in importance of NOBUF and MR NOBUF access). There's no
reason to go into them in much detail now; I'll just go through a few
of the key items that are worth repeating:
* If you want to have multiple writers appending to a (non-KSAM) file,
use ;SHR;GMULTI;ACC=APPEND access. If you do this, you will not need
to lock the file.
* If you want to have multiple writers doing any sort of writing other
than appending, be sure that you lock the file, not just before the
write but before any read done in preparation for the write. Thus,
if a process needs to read a record, calculate a new value for a
field in the record, and then write the record back, it must lock
before the read and unlock after the write; otherwise, it risks the
record being modified by somebody eke between the read and the write
and then having this other person's modifications wiped out.
* Attempts to lock a file (or a database) when you already have a file
or database locked will normally fail with an FSERR 64. If you :PREP
your program with ;CAP= MR, the attempt will succeed, but you stand
the risk of causing a deadlock (which will still require a system
reboot to resolve).
* If you must use ;CAP = MR, make sure that all your multiple locks
are acquired in the same order -- if one program locks file A and
then file B, all programs must lock those files in that order;
otherwise, if any program locks file B and then file A, a deadlock
becomes quite possible.
PROTECTING YOUR FILES AGAINST SYSTEM FAILURES
MPE/XL relies very heavily (even more so than MPE/V) on disc caching
-- keeping as much disc data in memory as possible to speed up access
to it. Unlike MPE/V, which, by default, only used this cache for reads
and always did the writes to disc, MPE/XL caches writes, too; if you
write a record to a file, that record might not get written to disc
for an indefinite amount of time.
This has some substantial performance advantages (since a lot of disc
I/O is avoided this way), but obviously puts your files very much at
risk when the system crashes. KSAM files and IMAGE files seem to be
protected by MPE/XL against loss of data at system failure time;
unfortunately plain MPE files can very easily lose a lot of
recently-written data when the system crashes.
One of these forms of data loss could happen (and often did) on MPE/V
-- when you're appending to an MPE file, the EOF pointer does not get
updated on disc until the file is closed or a new extent is allocated.
Thus, the data that you append to an MPE file can get completely
destroyed by a system failure because the EOF pointer did not get
properly set.
The solution to this problem, just as in MPE/V, is to do FCONTROLs
mode 6, which post the EOF pointer to disc, as often as possible when
you're appending to an MPE file. You might, for instance, do an
FCONTROL mode 6 after every write, which will give you almost complete
safety but also slow things down substantially; or, you could keep a
counter, and do FCONTROLs mode 6 every, say, five or ten records, thus
minimizing your overhead while still protecting most of your data.
Unfortunately, on MPE/XL, there's more to it than this. Any data that
you write to a plain MPE file -- even if you're not appending to it --
might get lost in a system failure because it may not get posted to
disc until some time after you do the writes. On MPE/V, this
possibility was limited to the data that was in your memory buffers
(usually no more than about 2 blocks worth of data); on MPE/XL, any
data written since you opened the file could conceivably be lost.
For example, I ran a test with a file of 1000 records, each 256 bytes
wide; I overwrote all 1000 records, kept the file opened, and
re-booted the system. When the system came back up, only the first 768
of the new records were actually in the file; the remaining 232
records were still the old records from the time before I did my
writes. (Note that 768 = 3*256; I'm not sure if there's any
significance to this but I suspect that there is.)
What can you do about this? Well, the simplest solution seems to be
to call the FSETMODE intrinsic with the second parameter set to 2.
This means (according to the manual) "force your program to wait until
the physical write operation is completed (the record is posted)", and
this is what it seems to do. Of course, this causes each logical write
to generate a physical I/O -- a great deal of overhead -- but it
protects your data.
Alternatively, you can call FCONTROL mode 2 or FCONTROL mode 6 after
each write or once every several writes (FCONTROL mode 2 is faster and
may work well in cases where you're not appending and thus need not
post the EOF); this is more work for you as a programmer than just
calling FSETMODE, but it may be more efficient because you can do the
FCONTROLs once every several records, thus decreasing the overhead of
the extra disc I/O (but increasing the amount of data you may lose in
case of a system failure).
A FEW WORDS ABOUT PERFORMANCE TESTS
The performance guidelines I've talked about (such as "FREADs of files
that aren't in memory are faster than mapped file accesses" or
"FCONTROLs mode 2 are faster than FCONTROLs mode 6") are strictly
based on experience (my own or Stan Sieler's -- see his "MPE XL and
Performance: Not Incompatible" paper). This experience may be
inapplicable to your particular application, inapplicable to your
version of the operating system, or perhaps just plain mistaken; I
strongly encourage you to run your own performance tests to figure out
how fast various file access methods work for you.
Unfortunately, file system performance measurement on MPE/XL is
substantially more difficult than on MPE/V because of MPE/XL's immense
caching capabilities. It is almost guaranteed that, if you run a test
twice in a row, you will get completely different results -- the first
time your data was quite likely out on disc, but the second time it
had just been read into memory and was therefore quite probably still
in memory. Unlike MPE/V, there are no :STOPCACHE commands that you can
use to make sure that this doesn't happen.
There are two key things you can do to detect possible bias due to a
file's presence in memory and to avoid such bias:
* To find out how much of a file is in memory, do the following:
- Go into DEBUG.
- Enter "MAP filename" to open the file as mapped; this will output
a line such as:
1 MYFILE.MYGROUP.MYACCT 1234.0 Bytes = ...
- The 1234.0 in the above line is the virtual memory address of
the file -- type
=VAINFO(1234.0, "PAGES_IN_MEM")*#16,#
The value output will be the number of sectors of the file that
are currently in memory (the #16 is there because there are 16
sectors per 4,096-byte page).
- Close the file by saying "UNMAP n", where n is the first number
output by the MAP command (in this example, 1).
* Getting the file out of memory is a tougher proposition. My
experience has been that the only way of doing this is to cause
enough memory pressure to get the file's pages to be discarded
(after they have, of course, been flushed to disc).
One way of doing this is to read a very large file into memory;
SL.PUB.SYS (22 megabytes on my system), NL.PUB.SYS (15 megabytes),
and XL.PUB.SYS (6 megabytes) are good candidates. Just say:
FILE S=SL.PUB.SYS;LOCK COPY *S,TESTFILE;YES PURGE TESTFILE
This will read all of SL.PUB.SYS into memory, which on my 925/LX is
enough to flush any other files I may already have in memory. All
you rich people out there with 128 megabytes of unused memory may
need more than just this file, but you can always tell if the
flushing succeeded by using DEBUG's VAINFO function discussed above
-- if it tells you that your file has only 0 pages in memory, you
know that you've flushed it out.
Given these precautions, you should be able to do your own performance
tests (on an otherwise idle system, of course). Beware, though -- at
least one key test I know of yielded completely different results on
MPE/XL 1.1 and 1.2 -- much of the file system's performance
characteristics seem to be quite MPE-version-dependent.
ODDS AND ENDS
Finally a few miscellaneous features which couldn't fit in anywhere
else:
* DEBUG/XL's MAP command makes the debugger a powerful data file
editor even more convenient than the old DISKED on MPE/V. (Anything
would be more convenient than DISKED. Do you remember how, when you
asked for it to display octal and ASCII on the same line it would
display 8 words of the data in octal and then the first 8 bytes of
the data in ASCII, completely ignoring the last 8 bytes?)
In DEBUG/XL, you can say
MAP filename WRITEACCESS
DEBUG will output for you the "file index number" (used to close
the file with the UNMAP command), the filename, the file's visual
address, and the file size, e.g.
1 MYFILE.MYGROUP.MYACCT 1237.0 Bytes = 7560
(For this example, we're assuming that you're in CM debug, so the
numbers are output in octal; in NM debug, the output and default
input would be in hex.) You can then display and modify data with
addresses 1237.0 through 1237.7557 -- all the bytes in the file;
thus, you could say
DV 1237.200,10
to display the 10 (octal) 32-bit words starting with byte 200
(octal) of the file -- the MV command will let you modify data. Note
that DV and MV expect byte addresses, not record numbers and offsets
within records; you have to do the calculation yourself (for
instance, if each file's records are #256 [decimal] bytes long,
record #10 occupies bytes #2560 through #2815).
The MAP command also provides you with one of the few ways to easily
see (and edit) a file's user labels (:FCOPY, for instance, doesn't
let you display their contents, and neither does the :PRINT
command).
When MAP gives you an address whose second word is not 0 (1237.1400,
for instance, rather than 1237.0), this means that the file has user
labels (in this case, %1400/%400 = 3 labels). User label 0 starts at
1237.0, user label 1 starts at 1237.400, user label 2 starts at
1237.1000, and data record 0 starts at 1237.1400. (User labels are
always %400 = #256 bytes long.)
Thus, you can use DV and MV to modify the file's user labels; also,
remember that data record 0 now starts at a byte address other than
0 (in the example, %1400) -- keep this in mind when calculating the
byte address of a particular byte in a particular record. If your
file's records are #80 (= %120) bytes long, then, say, byte 6 of
record 4 will be at location 1094 (= %1400+4*%120+6).
* If you do a :DISCFREE A (which shows you how many bee space chunks
of each size there are on your disc), beware! You'll often see
several large free chunks on LDEV 1 even though you're running out
of disc space (or at least of contiguous disc space).
On MPE/XL (unlike MPE/V), transient space (analogous to MPE/V's
virtual memory) is treated as free disc space; however at least 17%
of the system disc (or more if you configure it that way) is
reserved for transient space. Thus, you could have a huge chunk of
free space on your system disc and still have it completely unusable
for new disc files because it's reserved for transient space.
:DISCFREE B tells you how much space is reserved for transient
space, so its output shouldn't be too confusing; however, :DISCFREE
A's output can be quite misleading if you don't keep the transient
space issue in mind.
This should probably not be overwhelmingly important, since
contiguous space is less important on MPE/XL than on MPE/V, and you
should therefore run :DISCFREE B more often than :DISCFREE A;
however, I got bit by this thing myself when I was doing research
for this paper, so I decided to mention it.
CONCLUSION
The MPE/XL file system is different from the MPE/V file system in many
respects but is also similar to it in many respects. This paper was
largely dedicated to the differences (since they're more interesting),
but there are very many similarities as well, largely dictated by the
requirement of complete (well, almost complete) compatibility -- a
requirement that HP rigidly enforced on itself, and, I must say ,very
much lived up to.
Many of the old and unpleasant limitations of the MPE/V file system
have been lifted; a few remain in place (such as the 3-level directory
structure and a few other, relatively minor, problems); a few new ones
have probably been added, but the user community hasn't discovered
them yet and probably won' t for some time. (Who would have thought,
in 1972, that people would be running into the 2,097,120-sector limit
on file size?) Performance and disc space are still potential
problems, and will be for a long time to come as long as CPU power and
disc storage cost money.
I would like to thank Jason Goertz, Bob Green, Randy Medd, David
Merit, Ross Scroggs, and, especially, Steve Cooper and Stan Sieler for
their reviewing the paper and for their many excellent comments and
suggestions. I would also like to refer the interested reader to Stan
Sieler's "MPE XL and Performance: Not Incompatible" paper, published
in the SCRUG '89 Proceedings, and, of course, to the Beyond RISC! book
from Software Research Northwest (by S.Cooper, J.Goertz, S.Levine,
J.Mosher, S.Sieler, and J.Van Damme edited by W.Holt).