THE TRUTH ABOUT DISC FILES
by Eugene Volokh, VESOFT
Presented at 1982 HPIUG Conference, San Antonio, TX, USA
Published by SUPERGROUP Magazine, Feb 1983.
Published in "Thoughts & Discourses on HP3000 Software", 1st-4th ed.
ABSTRACT
The disc file is probably the most important part of MPE; however, due
to the large number of different options and considerations inherent
in disc files, these objects are often 'under-understood' -- this
paper will try to present the truth and nothing but the truth (the
whole truth will not be printed owing to lack of paper) about disc
files, which will hopefully remedy this situation.
SECTION I -- FILE STRUCTURE
WHERE IT'S AT
Before discussing disc files themselves, we must take a moment to
point out some terms, probably already known to you, regarding the
physical medium on which disc files reside -- the disc. This disc
consists of a lot of 128-word SECTORS, and is assumed to be configured
on the system as one logical device.
Some considerations to be judged when referring to these discs are:
one, space -- each disc has an ever-so-finite amount of sectors on it,
the number of which varies from disc to disc, but is, by Murphy's Law,
never enough -- and two, speed of access, which is typically on the
order of 30 disc accesses per second.
The discs typically used with the HP are ones that constantly rotate
in order for all parts of the disc to be accessible by the unrotating
DISC HEAD. It is this rotation of the disc that is the culprit in the
slowness of disc accesses.
Similar considerations can be applied to the two other significant
types of hardware: memory (which is very, very fast yet lamentably
limited -- up to 4 Megabytes on a Series 44) and tapes (which are
virtually infinite yet quite slow).
The above hardware considerations, though elementary, will be of
paramount importance in further discussion.
THE EXTENT QUESTION
Let us start at the beginning -- the creation of the file. We will
examine what the MPE operating system has to do to create a file. For
example, let us say that you ask MPE to build you a data file, which
is to have room for at most 100,000 records of 128 words each (note
that 128 words is the size of a sector, and thus a good value for
simplicity). This would be done, perhaps, by an MPE command akin to
':BUILD ING;DISC=100000' (MPE will automatically assume 128 words as
the record size). Now, what does MPE do?
Well, of course, MPE must allocate some disc space for that file. In
this particular case, MPE must allocate a whopping 100,001 sectors
(the 1 extra sector is for the file label, a place where MPE holds
internal file information like the lockword, etc.) all at one time.
But, wait a minute! There may be 100,001 sectors out there on your
disc (or discs), but it's possible that there is no one single gap
that large out there. Moreover, maybe you don't really need all that
space -- quite probably, you'll never use more than 10% of it! So, we
are faced with a dilemma: if MPE were to allocate the space for that
file nicely and simply, in one big chunk, it might not have enough
space on disc; or, if it does, most of that space would probably be
wasted, as (for a time, at least) you would not use all of that space.
Let us look at the other 'extreme' solution. Why don't we, perhaps,
allocate only one sector of space at a time -- one in the beginning,
for the file label, and one every time the user needs one. That way,
even if the disc is hopelessly fragmented (i.e. there are very many
one-sector pieces of free space out there, but no large ones), we can
probably fit a sector -- if we can't, it is time to buy another disc;
moreover, we do not allocate any disc space until we really need it.
This was, perhaps, a decent solution in the 'good old days' when disc
space was very expensive. But, now, the operating system would have to
maintain 100,001 pointers to enable access to that file, which makes
the above method unworkable.
Enter the EXTENT! The extent is a reasonable compromise between the
two extreme methods outlined above. A file can consist of anywhere
from 1 to 32 extents (the default number is 8). Now, when we build
the above file (with 8 extents), we will have to allocate only about
12,500 sectors in the beginning (a savings of disc space) and allocate
new extents only every 12,500 records (a savings of disc accesses).
We could, however, allocate the file with only 1 extent, thus losing
out on disc space but gaining on disc accesses (but, of course, the
savings on disc accesses is rather small compared to the incredible
wastage of disc space), or with 32 extents, thus saving disc space at
the expense of a few extra disc accesses.
Two other considerations come into play, however: one is that
accessing files with a lot of extents FRAGMENTS THE DISC (i.e.
increases the number of small holes at the expense of large holes),
making new files harder to allocate in the future, and another is that
it is better to run out of disc space when building a disc file than
when allocating a new extent in the middle of the program (precious
time and internal data consistency may be lost this way). The former
can be handled best by decreasing the number of extents (at the
expense of, of course, disc space) and the latter by allocating at
:BUILD-time all of the specified extents (but only if you are sure you
will use all of the space). Note that the number of extents (maximum
and initially allocated) may be specified in the :BUILD command's DEV
keyword, whose format is 'DEV=device[,maxexts] [,initalloc]', where
maxexts defaults to 8 and initalloc to 1.
FOR THOSE WITH MULTIPLE DISCS
If you are the proud owner of several disc drives, another factor
comes into play. For example, let us say that you build a file with
the command ':BUILD ING;DISC=100000' (note that the maximum number of
extents defaults to 8, 1 initially allocated), and start to wonder
about which disc your file resides on. Well, MPE, has adopted the
so-called 'eeny, meeny, miney, moe' algorithm. That is, if you
succeed in filling all 8 extents of your file, you may well find that
that file does not reside on just one disc; rather, it resides on the
discs of the DEVICE CLASS 'DISC' (which are special sets of different
devices, not necessarily discs, configured at system set-up time).
Each extent, of course, resides wholly on one disc; but, the extents
may reside on different discs -- thus, a file with 8 extents may well
find itself with 4 extents on disc #1, 2 on disc #2, 1 on disc #3, 1
on disc #4, and 0 on disc #5. If you want that file to reside
exclusively on disc #4, however, 'no sweat' (as is said in the
vernacular)! Merely :BUILD the file with the 'DEV=4' parameter. Or,
if you set up another device class called PRODDISC which will contain
discs #3, #4, #5, building the file on DEV=PRODDISC will ensure that
all extents of that file will be located on one of those devices.
What, you may ask, is the importance of this? Well, the word that has
leaked down from HP is: SPREAD OUT YOUR FILES -- for instance, if you
have two heavily accessed files, it might be wise to put them on two
different discs.
This is done for the following reason: let us assume that you have two
disc drives, each one able to perform approximately 30 I/Os per
second, and you spread out your files in such a way that each disc
gets about 30 I/O requests per second -- those requests will be
executed within one second. But, if there are 20 I/O requests per
second to one disc and 40 to the second disc, the first disc will not
perform up to capacity, and 10 of the requests to the second disc will
have to wait for a second or more, thus degrading system performance.
(NOTE: The entire question of head locality and what it buys you is
the topic of much argument in the HP community, and is generally
unresolved. The above statements are an expression of the author's
understanding of this issue and may be fallacious in some respects.)
Another promising idea is to configure all of your devices except the
system disc as device class 'DISC', thus keeping files off the system
disc, and thus reducing the amount of access to the system disc, which
already has the operating system and the virtual memory on it.
However, with MPE IV, in which you are allowed to spread virtual
memory over several devices, this may not be as important.
Note that for easy file disc location handling, MPEX/3000's %LISTF ,4
and %ALTFILE commands and ADAGER's DBCREATE and SETMOVE functions
should be used.
THE LOGICAL FILE STRUCTURE
Besides the physical file structure described above -- extents,
sectors, etc. -- MPE files also have an internal logical structure,
not enforced in most ways by the actual file contents but rather by
certain logical file descriptors like the record size, the blocking
factor, the block size, the file type, and the like. First of all, we
will discuss the simplest sort of MPE file -- the fixed-record length
file.
THE FIXED RECORD LENGTH FILE
A file is more than just a collection of data placed out on disc. It
usually has certain logical relationships within it. One of the most
frequent and fundamental relationships is one in which data is
organized into chunks (called RECORDS) of a fixed length; for
instance, if you have a data file which contains [for each customer]
the customer code (6 characters), customer name (30 characters), and
the amount owed you by the customer (8 zoned decimal characters), you
have a 44-character entry for each customer. Therefore, it would be
logical, for the sake of ease of access, to build that file with
44-byte (or 22-word) records, having one record per customer. So, to
build that file, you would perform a BUILD command with the
REC=-44,,F,ASCII parameter ("-" stands for bytes and "F" for fixed
record length).
THE BLOCK
A familiar example of a fixed record length disc file is your usual
EDITOR /KEEP-NUMBERED file, a file with a record size of 80 bytes = 40
words. However, do you know that in your EDITOR /KEEP files more than
6% of all disc space they occupy is wasted? This may not sound like
much, but if you are running short on disc space, this can be a lot --
what's more, that disc space can be saved (for large files) by merely
specifying a certain :FILE equation for the file to be kept. What,
you may ask, is the reason for this wastage? Well, the answer lies in
the secrets of the BLOCK.
The fundamental unit of disc I/O (as far as MPE is concerned) is the
SECTOR (128 words). Practically all disc I/O ends up as multiples of
128 words. Forty, of course, is not a multiple of 128. So, if MPE
decided to place 40 words per sector, it would waste not 6%, but 69%
of each sector! So, you ask, why not pack three 40-word records into
one 128-word sector? Well, that's exactly what MPE does; but because
128 is not a multiple of 40 either, 6% of the file's disc space is
still wasted. (Although 6% may not sound like much, for some unlucky
files which have different record lengths, it can be worse, with up to
50% wasted space!) But, there is light at the end of the tunnel! We
can very snugly fit 16 40-word records into 5 128-word sectors -- a
perfect fit.
From the above labyrinth come the notions of the BLOCKING FACTOR and
the BLOCK. The BLOCKING FACTOR is, very simply, the number of records
that we choose to fit into a multiple of 128 words -- in the above
'snug fit' scenario, this is 16; in the 6% wastage method that MPE
uses, the blocking factor is 3 (3 records to 1 sector); in the (ugh!)
69% wastage at 1 record to 1 sector, the blocking factor is 1. The
BLOCK therefore, is BLOCKING FACTOR records -- i.e. when the blocking
factor is 16, the block is 16*40 = 640 words = 5 sectors.
In general, MPE chooses the blocking factor as follows: if the record
size of a file is less than one sector (128 words), the blocking
factor = 128/recordsize = the number of records that will fit into one
sector; if the record size of a file is greater than 128 words, the
blocking factor is always 1. A good example of the possible wastage
is when a record is 65 words long; then, 128/65 = blocking factor of
1, wasting 63 words for every 65 words used -- a wastage of 49%! If
that record was, however, 64 words long, then the blocking factor
would be 2, with NO wastage.
By the way, it happens that the blocking factor for a new file can be
defined in a :BUILD or :FILE command -- always as the second
subparameter (between the record size and the F, V, or U record
format) of the REC= keyword. Thus, if you want to eliminate the 6%
waste due to the blocking factor of 3 on EDITOR keep files, just
execute an equation of the form ':FILE filename;REC=,16' right before
keeping the file as 'filename', and presto! -- out comes a file with a
blocking factor of 16. For already existing files, some disgustingly
complicated tricks can be used -- or, if you are blessed with a copy
of MPEX/3000, just use the BLKFACT= keyword of the %ALTFILE command.
Now, you may wonder, what causes MPE to choose a default blocking
factor calculation system that leads to considerable wastage in
perhaps one of the most common form of files? Well, for one, it would
be unfair not to remark at this point that the 'NO wastage' schemes
described above really DO waste some space (although not a lot). The
reason for this is that a file (in fact, each extent of a file) must
be an integral number of blocks. If it isn't, a full block is
allocated for less than 'BLOCKING FACTOR' records. Thus, if you have
a file containing 50 80-byte records with a blocking factor of 16, it
would use up 4 blocks, the last one having only 2 actual records --
this file will thus use 21 sectors; however, if this file is built
with a blocking factor of 3, it would use up 17 blocks (the last one
also having only 2 records), and would thus use only 18 sectors of
disc space. However, this consideration is less important for larger
files. Another reason for MPE's default blocking factor strategy is
that the block and the blocking factor govern more than just disc
space usage -- they also control certain parameters of buffered file
access (see the chapter on FILE ACCESS). However, for most files
(especially large ones!) it is beneficial to select your own blocking
factors (with the use of the contributed BLOCK program, for instance).
THE VARIABLE RECORD LENGTH FILE
Let us take a hypothetical EDITOR COBOL-format file. At the beginning
of each line there is a 6-digit line number; the other 74 characters
contain the line, blank-padded. Now, those trailing blanks,
especially in large source files, convey absolutely no information to
anybody, and (since the average length of a line could be estimated at
half of 74 characters) will cause a wastage of APPROXIMATELY 50% OF
THE DISC SPACE USED BY THOSE FILES! But, you reply, if EDITOR built
the file with a record length of, say, 40 characters, all of my lines
that are longer than 40 characters will get truncated. Well, you're
right -- but that is not what is to be done! Wouldn't it be nice if
EDITOR and/or the file system allowed you to have files with not a
FIXED record length, but with a VARIABLE record length -- i.e. lines
that are 74 characters long will use 74 characters and lines that are
10 characters long will use 10 characters? Well, it does!
In fact, if you type in the little-known /SET VARIABLE command in
EDITOR, it will instruct EDITOR to keep the workfile as a variable
length record file (WARNING: USE THIS ONLY FOR COBOL AND DATA FILES,
NEVER FOR NUMBERED FORTRAN OR SPL SOURCES, OR THOSE SOURCES WON'T BE
COMPILER READABLE!!!), thus letting it ignore those trailing blanks,
but still keeping the file format transparent to other programs that
read these files -- for example, compilers. In your own programs (not
just in EDITOR), you can read variable record length files without
changing your programs at all -- COBOL's or FORTRAN's READ command can
read variable record length files. You can write them without any
changes either -- if you write a 10-character record to a fixed record
length file of 80 characters, the record will be padded with 70 blanks
or nulls; if you write that record to a variable record length file,
the record will not be padded by anything, thus saving the space
required for the padding. To build a variable record length file,
specify the third subparameter of the REC= parameter of the :FILE or
:BUILD command as 'V' (e.g. REC=-80,,V). The record size specified is
now no longer the actual record size of each record but rather the
maximum; whether the file is ASCII or BINARY now really doesn't
matter. Also, do not call in the National Guard (or PICS) when you
see on a :LISTF that the END OF FILE for that file is GREATER THAN ITS
FILE LIMIT -- it can happen with variable record length files.
Therefore, with COBOL source files (especially) and unnumbered data
files, variable-length records are usually the way to go; again,
however, we must warn you that numbered default-format (e.g. SPL or
FORTRAN source) files SHOULD NEVER BE KEPT WITH THE /SET VARIABLE
OPTION SET or else they will not be readable by the compiler.
However, as the old proverb says, 'EVERY SILVER LINING COMES WITH A
CLOUD ATTACHED TO IT.' Variable record length files have some
drawbacks; for one, they cannot be accessed directly (for instance,
with the FREADDIR, FWRITEDIR, or FPOINT intrinsics, or FORTRAN's
READ/WRITE (fnum @ record) construct); i.e. you can read their records
sequentially, but you cannot ask to get, for instance, the 17th record
of the file. Moreover, they cannot be accessed by many file copiers
using the fast MR NOBUF file access method (see under FILE ACCESS in
this paper), such as HP's own DSCOPY, MPEX's %FCOPY ,,FAST, MPEX's
%ALTFILE, SUPRTOOL/ROBELLE, etc. Also, before MPE IV, append access
to variable record length files was not supported; it is supported
starting with MPE IV.
Another important consideration to keep in mind when using variable
record length files is that when you build a new variable record
length file with record size RECSIZE and blocking factor BLKFACT, the
resultant block size of the file will not be RECSIZE*BLKFACT (as in
fixed record length files), but rather RECSIZE*BLKFACT +
(BLKFACT+1)*(2 bytes). Thus, if you build a variable record length
file of record size 80 bytes and blocking factor 3, the file will
actually have a block size of (80*3+4*2)=248 bytes. However, if the
same file is built with a blocking factor of 16, the block size will
end up being (80*16+17*2)=1314 bytes, not 1,280 bytes! The end result
is that AN OPTIMAL BLOCKING FACTOR FOR A FIXED RECORD LENGTH FILE MAY
BE FAR FROM OPTIMAL FOR VARIABLE RECORD LENGTH FILES!
Incidentally, MPE IV's new INTER-PROCESS COMMUNICATION features (i.e.
Message and Circular files) rely EXCLUSIVELY on variable record length
files (q.v. COMMUNICATOR issue 26 -- the C MIT).
UNDEFINED RECORD LENGTH FILES
There exists another type of disc file -- the undefined record length
file. These are rather bizarre specimens which are not intended to be
and should not be used as disc files, but rather are supposed to be
utilized as tape files and terminal files, which are beyond the scope
of this paper.
ASCII VS. BINARY FILES
When using fixed record length files, it often happens that you may
write a 30-character record into a file with a record length of 80.
Then, what happens to the other 50 characters of the record? Well,
for some files (for instance source files) that contain simple text
data, you would typically want to initialize them to spaces because of
the nature of the file. If that is what you want, you would build
that file (EDITOR will build it that way for you) as an ASCII file.
This parameter can be specified as the fourth subparameter of the REC=
parameter of the :FILE or :BUILD command, e.g. REC=,,,ASCII. However,
for some data files, you may want to pad the records with binary zeros
(nulls). Files built in such a way are called BINARY files, and can
be built by specifying the BINARY parameter as the fourth subparameter
of the REC= parameter of the :FILE or :BUILD command, for example
REC=,,,BINARY. Note that this is usually not necessary as BINARY is
the default file mode. Also note that since no record padding is done
in variable record length files, the ASCII vs. BINARY distinction is
usually irrelevant to them.
THE FILE CODE
If you do a :LISTF mode 1 or 2 on a group of files, you may notice
that some files have a file code of 0 (blanks), some of PROG, USL,
EDTCT, KSAM, PRIV, and assorted numeric codes. These filecodes, for
the most part, are merely for the sake of file identification -- they
have no physical influence on the actual contents of files. If you
change the filecode of a file (for example with MPEX/3000's %ALTFILE
filename;CODE= command), the contents of the file will not magically
change. However, the filecode is useful for identification purposes
-- for instance, the MPE loader knows that files of filecode PROG are
:RUNable program files, the EDITOR knows that files of code EDTCT are
/SET FORMAT=COBOL files, and QEDIT/ROBELLE knows that files of code
111 are its files. In fact, you can set up your own file
identification system for source or data files -- you can build files
with a certain file code (via the CODE= parameter of a :FILE or :BUILD
command), alter the file code (with MPEX/3000 or by copying the file),
and examine the file code (via the :LISTF command or,
programmatically, with the FGETINFO intrinsic). Certain tools like
MPEX also allow you to LISTF files by file code. An example of this
kind of file identification system (recently implemented by us) is to
set the file code to be the Julian date of the day on which it was
created, or some other important date.
Note that the file code of each file is in reality a number -- for
example, program files (PROG) have a file code of 1029, but they are
listed in a :LISTF output as PROG. Also, KSAM files do not actually
have a numeric file code that identifies them as such -- they can in
reality have any numeric file code. However, KSAM files which have a
file code of 0 (which usually shows up as blanks on a :LISTF listing)
will be printed as having code = KSAM. Files that are listed as
having file code = PRIV are in reality files that have NEGATIVE file
codes (like IMAGE files). Unlike usual files, they can only be
accessed by programs running in PRIVILEGED MODE. This is handy, for
instance, for IMAGE files, to ensure that an ordinary user cannot
physically change an IMAGE file without going through the existing
IMAGE utilities/intrinsics.
USER LABELS
It is often desirable or necessary to store information in a file in
such a way that it can later be retrieved, but is nonetheless
transparent when you read it in an ordinary fashion. The concept of
USER LABELS provides this capability. With it, you can write special
label records (the maximum number of which is specified at open time,
defaults to 0, and can be up to 254) with the FWRITELABEL intrinsic,
read them with the FREADLABEL intrinsic, but have them be transparent
to any user who reads or writes ordinary records to that file. These
labels are used by IMAGE, KSAM, and the message system file (e.g.
CATALOG and CICAT). Another advantage of user labels is that you can
write user labels when you open the file for read access, read user
labels when you open the file for write access, and open the file for
OUT access (see access modes below) which will erase all of the file's
records but not its user labels.
CARRIAGE CONTROL FILES, RELATIVE I/O FILES, MESSAGE FILES, CIRCULAR
FILES, KSAM FILES, IMAGE FILES, AND OTHER MONSTERS THAT INHABIT THE HP
3000
This paper will not talk about the above types of files (for want of
time, will, and disc space). However, maybe sooner or later you will
hear the truth about them, too!
SECTION II -- FILE ACCESS
Once a file is built, it really isn't much good if you can't access it
-- read it, write it, append to it, etc. In this chapter we will
discuss the different methods of accessing files that MPE provides for
you.
BUFFERED FILE ACCESS
A while back we referred to the concept of the BLOCK. Well, it turns
out that the block is more than a convenient way of storing records on
disc. In fact, it plays a very important role in the default mode of
file access called BUFFERED FILE ACCESS. Let us assume that you are
reading a 10,000-record disc file which has a record size of 40 words
(80 bytes), a blocking factor of 16, and thus a block size of 640
words. Let us assume that you had to do one disc I/O for each record
-- this would come up to a total of 10,000 disc I/Os, quite a lot!
So, MPE implemented a rather ingenious idea called file buffering.
Each file opened as a buffered file has allocated for it a certain
amount (default 2, changeable at open time with the FOPEN intrinsic or
the BUF= parameter of a :FILE equation) of buffers, each of length
equal to the file's block size (in this case 640 words). These buffers
are placed in an Extra Data Segment (because extra data segment access
is faster than disc access) and accessed there. They are read from or
written to disc only when a record that is not in the buffer is
requested. Thus, for the file described above, only 10,000/16 disc
I/O's = 625 disc I/O's are necessary -- a considerable savings! The
advantage of having more than 1 buffer is that then you can access,
for instance, records 17-32 (in one buffer) and 49-64 (in the other
buffer) without necessitating a disc I/O each time you switch from one
record range to the other. However, if you then read in record 100,
the contents of buffer 1 will be flushed out to disc and buffer 1 will
then contain records 97-112. In general, with buffering, one disc I/O
is required for every (BLOCKING FACTOR) records -- in this case, one
disc I/O is needed for each 16 records.
In the discussion above, we advised you to set up blocking factors so
that BLOCKING FACTOR * RECORD SIZE be an even multiple of 128;
therefore, for instance, 16 was chosen for files with records of 40
words in length. But, there is more than one way to skin a blocking
factor! In fact, since 16 * 40 is a multiple of 128, 32 * 40 certainly
is too! Very little disc wastage will result from changing the
blocking factor from 16 to 32, but each buffer will now be not 640
words long, but rather 1,280 words long, and now only 313 (=10,000/32)
disc I/Os will be necessary to read the file! A blocking factor of 64
will require less than 160 I/O's, and so on. This will not
necessarily halve the time used by the read, but it will surely
decrease it. Of course, the same thing can be said for writing to
files. We must, however, point out that memory space will be used much
more heavily by files that have large blocking factors. Also, the
total size of the buffers must be less than or equal to 8,192 words
(or 14,000 words starting with the D MIT version of MPE). Since the
default number of buffers is 2, this puts an upper limit of 4,096
words (or 7,000 words starting with the D MIT) on the block size of a
file. However, you can increase that maximum to 8,192 words (or
14,000 words starting with the D MIT) by opening the file with 1
buffer (by specifying BUF=1 on a file equation).
MULTIPLE RECORD NON-BUFFERED ACCESS (MR NOBUF)
The buffering method described above is rather good, but it is still
not optimal; first of all, access to the extra data segment in which
the buffers are located is faster than disc access, but nonetheless
not as fast as access to your own stack. Moreover, as was pointed out
above, certain memory usage considerations forbid the buffers from
being more than 5K to 10K words, which is also not optimal. Wouldn't
it be truly wonderful if one could read not just single records, not
just blocks of 16 or so records, but 4,000 words at one shot? Well,
one can, through the magic of MR NOBUF, probably the MOST POWERFUL AND
FASTEST FILE ACCESS METHOD NOW AVAILABLE! MR stands for Multiple
Record I/O (do not confuse this with the Multiple RIN capability, also
abbreviated MR), and NOBUF stands for No Buffering (this is a bit of a
misnomer -- it means that it is you, not MPE, who will provide the
buffer space needed). Note that MR must be used with NOBUF!
Certain factors to beware of when using MR NOBUF are, for one, that
this method is rather hard to use with variable record length files.
Also, the efficiency of this method is best with a buffer size of
4,096 words. Another factor is that when the block size (BLOCKING
FACTOR * RECORD SIZE) is not a multiple of 128 words, MR NOBUF is not
that much more efficient than ordinary file access, and simple NOBUF
access should be used instead.
By far, the best application of MR NOBUF is with file copying. FCOPY,
which uses ordinary buffering methods, is often 10 to 20 times slower
than MR NOBUF copiers like HP's own DSCOPY, MPEX/3000's %FCOPY
,,FAST/DSLINE or %ALTFILE commands, SUPRTOOL/ROBELLE, and numerous
other programs. However, you can do MR NOBUF reading and writing from
your own programs by specifying the MR NOBUF access options when
accessing the file (or specifying the MR or NOBUF parameters on the
:FILE equation -- however, a bug present in some versions of MPE
forces you to specify MR in the FOPEN because it ignores the MR :FILE
equation parameter; see 'ANOTHER MPE FEATURE (BUG)' in SCRUGletter,
Jan 1981 Vol 4 #1). This will allow you to read more than one record
(always at least one block, however) at a time, and also lets you do
direct I/O (e.g. with FREADDIR and/or FWRITEDIR) on a block number
rather than record number basis.
However, reading files MR NOBUF in your own programs is rather hard to
do because of many concerns that have to do with deblocking records.
Because of this, it is suggested that you either do most of your
record selection outside of your program (with SUPRTOOL/ ROBELLE, for
instance), or develop your own MR NOBUF I/O routines that can be
easily called from your applications programs.
MR NOBUF I/O can therefore really cut down the execution time and CPU
time demands of your disc I/O-heavy programs. The only problems with
MR NOBUF are that it is hard to apply to variable record length files
and KSAM files and that it may (because of the large in-stack buffers
necessary) use up a lot of stack space and much memory space.
THE ACCESS TYPES FOR DISC FILES
In the access options parameter of the FOPEN intrinsic or in the ACC=
parameter of the :FILE command you can specify the so-called access
type which defines whether a program will read the file, write to the
file, do both, or append records to the file. There are 7 legal
access types, which can be very useful if used properly.
The default access type is IN access. This is read-only access -- all
attempts at writing records to files opened with IN access will fail
with File System Error 40 -- OPERATION INCONSISTENT WITH ACCESS TYPE.
If you only want to read the file, you should open it with this access
type; this will prevent your program from accidentally writing over
the file; it will work even if somebody else has the file opened in
Share or Exclusive Allow Read mode (see the SHARING FILES chapter) and
it will also work if the file's security prevents you from doing
anything but reading that file.
Another type of access is OUT access. OPENING OLD FILES WITH THIS
ACCESS TYPE WILL ERASE THEM! If you do not want that to happen, you
should open the file with OUTKEEP access. However, if you want to
erase the file, or the file is new, or you do not care about its old
contents anyway, this is the access type that should be specified.
Note that you need WRITE access to the file to open it in this mode.
OUTKEEP access is useful for opening files to write to them, WITHOUT
DESTROYING THEIR OLD CONTENTS (as OUT access would do). You need
WRITE access to the file to open it with this access type.
Often you do not need to write over the old contents of a file -- you
merely need to add new records to it. In that case, APPEND access is
for you -- it forces the record pointer to be positioned at the end of
the file and permits you only to append records to the file. Another
advantage is that it requires that you have only APPEND access (not
WRITE access) to the file to open it thus. So, if you wish to permit
users to only append and not overwrite data in a given file, they
should be allowed only APPEND, and not WRITE, access to this file.
For instance, VESOFT's SECURITY/3000 permits APPEND access to its
security violation log file, but not WRITE access (so user's cannot
obliterate the record of their violations).
The above access types permit you either to READ ONLY or WRITE ONLY,
but never both. INOUT access lets you both READ and WRITE to the
file. All intrinsics (except FUPDATE) can be used against that file
in this mode. Note that you need READ and WRITE access to open a file
in this way.
There is also a special form of access called UPDATE access that is
PRECISELY the same as INOUT access except that it permits the usage of
the FUPDATE intrinsic. Since this is apparently no less expensive
than INOUT access, and requires no extra access to the file, it is
suggested that this option be used instead of the INOUT access type
because it is more powerful and no more dangerous.
Another access, permissible only to programs that run in Privileged
Mode (Ohmigod!), is EXECUTE access; its advantages are twofold: for
one, it requires only EXECUTE access, not READ access to a file;
moreover, it allows you to write to loaded program or SL files. This
is listed only for the sake of completeness, and all you nice non-
privileged users out there don't even need to know about it. For a
discussion of privileged mode, see PRIVILEGED MODE: USE VS. ABUSE,
SCRUGletter July 1981, Vol 4 #4.
POSTING END OF FILE TO DISC
As was mentioned before, each file has a special record called 'the
file label' (which contains all sorts of information about the file,
such as its type, name, and, among other things, its end of file,
which is the number of records which the file contains). Now, if every
time that you wrote a record to the file, MPE would have updated that
file's file label, your programs would run quite slow -- after all,
that would mean extra disc I/Os to handle. For this reason, MPE does
not post the end of file to disc until a record write would cause it
to allocate a new extent (in which case it would have to change the
file label anyway), thus saving the extra I/Os.
This is all fine and dandy, provided that MPE will actually get a
chance to post the end of file to disc sometime. But, what if the
system crashes after you wrote the record but before MPE posted the
end of file? Then, even though the record (or records, as the case
may be), are already out on disc, MPE does not know about it because
the end of file pointer does not reflect this. So, you've just lost
all those records that were written before the system crashed.
You can, however, minimize your losses through a little-known feature
of the file system by calling the FCONTROL intrinsic (see System
Intrinsics Manual) with a parameter 6 (WRITE END OF FILE) which lets
you post the end of file to disc. If you do this after you write each
record, the most records that you will ever lose due to a system
failure is one! Of course, this will triple the number of disc I/Os
that you'll have to do, so this is not advised for large batch runs;
however, if you are updating a disc file interactively, the time it
takes to input all of the data from the screen will dwarf the time it
will take to do the extra I/O to such a degree that that the posting
of the end of file will be virtually free in terms of time, and may
save you hours of re-entering vital data.
WHEN YOU ARE NOT ALONE
When you use any of the access types listed above except read only
(IN) access, the file specified will be opened EXCLUSIVELY; that is,
you cannot open it if anybody else has it opened, but, once you have
it opened, NOBODY ELSE CAN USE IT UNTIL YOU CLOSE IT. This is, of
course, somewhat of a problem if that file is intended to be read and
written by many different users. There are several ways to get around
this dilemma.
EXCLUSIVE ALLOW READ ACCESS
One of these ways is Exclusive Allow Read (EAR) access. This permits
you to forbid all other users from writing to a file, while letting
them read that file. Also, this access (unlike EXCLUSIVE) will be
granted to you even if the file is already being accessed for read
access (but not for write access) by someone else. This can be
specified by setting the appropriate bits in the access options of the
FOPEN call, or issuing a FILE equation with the EAR keyword.
TRUE SHARED ACCESS
But, sometimes you do not want to have only one writer to a file, or
one writer and several readers, but MORE THAN ONE PERSON WRITING TO A
GIVEN FILE. This can be accomplished with SHARED ACCESS (to use,
specify the appropriate bits in the FOPEN call or append the SHR
keyword to the file equation for that file), which is the default mode
for read only access, but has to be explicitly specified and handled
when writing to a file. Shared access is a very complicated form of
access, one at which we will look closer in the next chapter.
SECTION III -- SHARED FILES
SHARING FILES WITH INPUT/OUTPUT ACCESS
Merely specifying SHR access when opening the file will get you where
you are going -- it will allow you and anybody else who opens the file
with SHR access to read and write to this file. But, let us suppose
the following situation: two processes have opened one file for IN/OUT
access in SHR mode, and the following happens:
PROCESS A PROCESS B
Reads a record
Reads the same record
Changes the record
Changes the record
Writes the record back
Writes the record back
In the above scenario, process A reads the record before process B
reads the same record but writes it back out after process B reads it
in! That way, process A's changes WILL NOT BE REFLECTED IN THE FILE
because of the interference of process B. In fact, what is needed is
a method of 'LOCKING OUT' all other writers of the file while the file
is being updated! Well, MPE's FLOCK and FUNLOCK intrinsics provide
this method.
DYNAMIC LOCKING AND UNLOCKING FOR SHARED FILES
In order to use dynamic locking, the process that opens the file must
open it with dynamic locking enabled (the LOCK parameter on the :FILE
equation -- together with the IN/OUT SHR access, the file equation
would now look like :FILE file;LOCK;SHR;ACC=INOUT -- or the
appropriate bit in the access options of the FOPEN intrinsic call).
Then, before each 'logical transaction' (a period in time in which the
data in the file is not consistent -- in the above example, while the
record is being changed, the current state of the file does not
reflect the true intended state; therefore, the file must be locked
before the read and unlocked after the write) the file must be locked,
and then unlocked after the end of the transaction (note that opening
a file with dynamic locking enabled does not actually lock the file).
This will ensure that there will be no inconsistencies like the one
shown above. Note that THIS WILL WORK ONLY IF ALL WRITERS LOCK THE
FILE IN APPROPRIATE CASES -- this locking arrangement works only for
programs that honor it.
SHARING FILES WITH APPEND ACCESS
In some cases, however, locking does not really help. For example, if
two writers are just writing to a file (no reading, etc.), the
'logical transactions' like the ones described above are composed of
merely one write. For these transactions, it does no good to lock the
file. One of the most common examples of this type of file access is
shared append access to a file by two or more writers.
In fact, if the file has a blocking factor of 1, there is no need to
do anything but the write. However, look at what happens when the
file has a blocking factor other than 1, for example 3; consider
process A and process B, both writing to the same file:
PROCESS A PROCESS B
Record 1 written; kept in buffer
Record 1 written; kept in buffer
Record 2 written; kept in buffer
Record 2 written; kept in buffer
Record 3 written; buffer flushed
Record 3 written; buffer flushed
Note that, by the principles of buffering, the actual disc I/O is not
done until the third record is written and the buffer is flushed out
to disc. But because of that, when it is flushed out to disc, the
buffers from process A and process B interfere with each other, and
data can be lost.
Therefore, the rule for locking when appending (or performing any
other such operation in which each transaction contains only one
operation) is: LOCK WHEN THE BLOCKING FACTOR IS GREATER THAN 1; IF THE
BLOCKING FACTOR IS 1, LOCKING IS UNNECESSARY.
MULTIPLE FILE ACCESS
Another way to ensure that no data is lost while writing to a file is
with a useful tool (which is even more useful under MPE IV) called
MULTIPLE FILE ACCESS. With multiple file access in shared mode, the
internal file control information and the I/O buffers are shared, as
is the file itself, thus avoiding many problems of ordinary shared
access.
So, if process A and process B (IN THE SAME JOB/SESSION) access a file
SHARED, APPEND, and MULTI, then their internal end-of-file and buffer
pointers are shared; thus, the risk of one's file I/O interfering with
the other's is eliminated. To specify MULTI-access, set the
appropriate bit in the FOPEN parameters or specify the ;MULTI keyword
on a :FILE command for the file in question.
So, very many of the problems and complicated locking strategies
discussed above can be avoided if MULTI-access to that file is used.
However, there are two things that you must keep in mind when using
MULTI-access; for one, ordinary sequential reads and writes to that
file will not behave as expected. Why? Well, the current record
pointer is among those values that is shared with MULTI-access and
thus if process A reads a record sequentially and then process B
requests to read a record sequentially, process B will get the next
record because the record pointer was already incremented by A's read.
Thus, if the two processes read the file sequentially with
MULTI-access, each one will read approximately half the file instead
of the full file!
MULTI-ACCESS WAS PERMISSIBLE WITHIN ONE JOB/SESSION ONLY UNDER MPE
III. However, under MPE IV, you can use the GMULTI (Global MULTI
access), which can be specified in the FOPEN parameters or with the
GMULTI keyword of the :FILE equation, to have MULTI-ACCESS ACROSS
JOBS/SESSIONS, with which you can avoid most of the problems of shared
file access very easily.
MORE ABOUT LOCKING
There are two methods of locking files: UNCONDITIONAL, which means 'if
the file is already locked by somebody else, wait for them to unlock
it, and then establish the lock' and CONDITIONAL, which means 'if the
file is already locked by somebody else, return to me immediately with
an error condition'. The UNCONDITIONAL method is usually the most
useful, although the CONDITIONAL option is handy when you do not want
to take the risk of waiting a long time (if the program that has it
locked won't unlock it for a while). Needless to say, the file should
not be locked for a long time, and SHOULD NEVER BE LOCKED WHILE A
TERMINAL READ IS GOING ON unless you do not mind the fact that if the
terminal operator goes to lunch, everybody else who tries to
unconditionally lock the file will hang.
LOCKING MULTIPLE FILES, OR THE SECRETS OF MULTIPLE RINs (MR)
Let us consider another hypothetical circumstance: Process A locks
File 1; meanwhile Process B locks File 2. Then, Process A tries to
unconditionally lock File 2 and is then impeded until Process B
unlocks File 2. Meanwhile, Process B tries to unconditionally lock
File 1 and is then impeded until Process A unlocks File 1. Thus,
Process A is waiting for Process B and Process B is waiting for
Process A. Result: Deadlock. Both processes are hung until the
system is re-started.
The sages of Cupertino thought of this when designing the system; in
fact, their solution (which may not sound like much of a solution, but
which is better than nothing) is TO FORBID PROGRAMS TO LOCK MORE THA
ONE FILE AT A TIME. But, one may object, what if I have to lock more
than one file at a time? Well, the answer to that problem is that you
can get around (but at your own risk) that restriction if the program
that does the locking has Multiple RINs (MR -- not to be confused with
Multiple Record access) capability (i.e. was :PREPed with it). By the
way, RIN stands for Resource Identification Number. These programs
can, IF THEY REALLY HAVE TO, lock two or more files at a single time.
Needless to say, this capability should not be freely given to
everybody and his brother, but only to people who really need it and
are smart enough to use it without causing deadlocks.
That brings us to the question: How do you get around the deadlock
problem? Well, you may have already noted that the reason why the
programs got into a deadlock was that one locked File 1 before File 2
and the other locked File 2 before File 1. If they had only kept a
consistent locking arrangement (e.g. File 1 must ALWAYS be locked
before File 2), they would not have had the problem -- this is
probably the best way to avoid the deadlocks. Another way is to lock
the files CONDITIONALLY, and if the lock fails, do something else (or
even go into a loop, which can at least be broken out of by aborting
the job or doing a break/:ABORT, rather than re-starting the system).
SUMMARY OF LOCKING AND LOCKING STRATEGY
The following are the 10 Commandments of Locking:
1. THOU SHALT LOCK AROUND LOGICAL TRANSACTIONS WHICH INVOLVE TWO OR
MORE OPERATIONS. For example, this kind of a logical transaction
would be a read of a record followed by a modification of that
record followed by a write. If you do not lock around this, you
stand the risk of losing data consistency.
2. THOU SHALT ALSO LOCK AROUND ALL LOGICAL TRANSACTIONS THAT INVOLVE
A FILE WHICH YOU SHARE WITH SOMEBODY WHO HAS TRANSACTIONS WHICH
INVOLVE TWO OR MORE OPERATIONS. That means if process A's
transactions are just single writes and process B's transactions
are reads followed by writes, both process A AND process B must
lock around their transactions.
3. THOU NEED NOT LOCK A SHARED FILE IF ALL ITS WRITERS' TRANSACTIONS
INVOLVE JUST ONE OPERATION AND ITS BLOCKING FACTOR IS 1. Thus, if
process A and process B are writing to a shared file, and their
transactions are merely single writes (e.g. they are appending to
a file), neither one has to lock the file.
4. THOU SHALT USE GMULTI ACCESS UNDER MPE IV WHEN THOU ARE APPENDING
TO A SHARED FILE. This can save you time, worry, and your neck.
5. HONOR THY LOCKING ARRANGEMENTS. This means that if it has been
decided that a shared file is to be locked by its writers, all
writers must lock it. If so much as one writer fails to lock the
file, all of the locking arrangements will be useless.
6. THOU SHALT NOT KEEP A FILE LOCKED WHILE A TERMINAL READ IS IN
PROGRESS. If you do, the file will be locked down until something
is entered, which could mean an indefinite waiting period for any
other program that wants to lock the file.
7. THOU SHALT NOT LOCK MORE THAN ONE FILES AT THE SAME TIME WITHOUT
MR CAPABILITY. The second file lock will fail unless your program
was :PREPed with MR capability.
8. THOU SHALT PROTECT THYSELF FROM DEADLOCKS BY ESTABLISHING A FIXED
FILE LOCKING SEQUENCE IF THOU USES MR CAPABILITY. Thus, if
process A locks file 1 and then file 2, process B must lock in the
same order, (i.e. file 1 and then file 2, not file 2 and then file
1!).
9. THOU SHALT NOT GIVE MR CAPABILITY TO JUST ANYBODY. MR capability
can cause big trouble, and thus should be passed out sparingly.
10. THOU SHALT USE IMAGE/3000 IF THY FILE LOCKING ARRANGEMENTS GET TOO
COMPLICATED. IMAGE/3000 has file locking capabilities far
superior to MPE's file locking features. If you find that your
locking arrangements are getting too complicated or programs are
waiting for inordinate amounts of time to get at a shared file,
think about converting it to an IMAGE file -- it may be worth your
while.
SECTION IV -- FILE DOMAINS AND EQUATIONS
PERMANENT AND TEMPORARY FILES
Most of the files that we discussed in previous sections were usual
PERMANENT files -- files that, once built, exist until they are
:PURGEd or somehow deleted. There is, however, another type of file,
one that is also often quite useful. This is the JOB/SESSION
TEMPORARY FILE. These files, once built (by placing the ;TEMP keyword
in the :BUILD or :FILE command), exist until they are :PURGEd (by
performing a ':PURGE filename,TEMP') OR UNTIL THE JOB OR SESSION IN
WHICH THEY WERE CREATED LOGS OFF. Why are these files desirable?
Imagine, for instance, that you want to create a certain file that you
want to stream. After the file is streamed (in the same job or
session that it was built in), you no longer need it. If you were to
create that file as a permanent file and then purge it, it is quite
possible that somebody else may have built a file with the same name;
for instance, if the same program is being run on another terminal and
that file is created there.
However, if you create the file as a temporary file, you can be
certain that creating it will not interfere with anybody else; the
nature of job/session temporary files is such that two different jobs
or sessions can create within them temporary files with the same name
which do not interfere with each other.
Most MPE commands either attempt to open the file given to them as a
temporary file and then (if the temporary open fails) as a permanent
file (e.g. :STREAM,:COBOL,:RUN, etc.), thus being able to accept both
temporary and permanent files, or have special keywords that instruct
them to open the file as a temporary file (e.g. PURGE file,TEMP).
Programs that open files as permanent can be instructed to open the
file as job/session temporary by issuing a file equation of the form
':FILE filename,OLDTEMP'. Note that some commands and subsystems
(e.g. :BASICOMP, :PREP, :SEGMENTER's -BUILDUSL command) build files as
temporary files; others can be instructed to build files as temporary
by using a file equation like ':FILE filename;TEMP'.
If you need to keep a temporary file as a permanent file with the same
name, you can do a ':SAVE filename'; if you want to keep it as one
with a different name, do a ':RENAME oldfile,newfile,TEMP' and a
':SAVE newfile'. The names of your temporary files can be listed with
LISTEQ2 or (in a more complete, :LISTF-like format, with MPEX/3000's
%LISTF fileset: TEMP command).
$NEWPASS AND $OLDPASS
Two other useful critters are the system-defined files called $NEWPASS
and $OLDPASS. Consider, for instance, the :COBOL command. When the
USL file is omitted in this command, it is usually followed by a :PREP
command that is to prepare the resultant USL file into a program file.
But what intermediate USL file should be used? Well, if you use a
permanent or temporary file you run the risk of having a file with
that name already in existence. This is where $NEWPASS and $OLDPASS
come in. $NEWPASS is a peculiar file that, when closed, magically
turns into $OLDPASS. So, once you open $NEWPASS, write to it, and
close it, you can then open $OLDPASS, and read it.
So, in the case of the :COBOL and :PREP, the USL file parameter of the
:COBOL command defaults to $NEWPASS. The USL file is closed and
presto!, it becomes $OLDPASS. Now, you can execute a command of the
form ':PREP $OLDPASS,progfile', and that USL will be :PREPed into the
specified program file. If you really want to be fancy and you don't
need the program file to be a temporary or permanent file, you can do
a ':PREP $OLDPASS,$NEWPASS', and, after this is done, the program file
(which was specified as $NEWPASS) becomes $OLDPASS. Now, you can just
':RUN $OLDPASS.'. Note that $OLDPASS contains the USL file from
:COBOLPREP (or :FORTPREP, :SPLPREP, etc.) and the program file from
:COBOLGO (or :FORTGO, :SPLGO, etc.).
If you decide that you want to save the contents of $OLDPASS in a
permanent file, just do a ':SAVE $OLDPASS,filename'. A rather
bizarre, undocumented feature is that to save $OLDPASS as a TEMPORARY
file, you can do a ':RENAME $OLDPASS,filename'! Of course, $OLDPASS
vanishes as soon as you :BYE off.
THE CARE AND FEEDING OF :FILE EQUATIONS
Perhaps one of the single most important and least understood tools in
handling files is the :FILE equation. The file equation allows one to
re-define certain open parameters of old and new files. For example,
let us say that you are keeping a file with EDITOR, and you want to
keep it with a blocking factor of 16 and 32 extents. Then you would
issue the file equation ':FILE filename;REC=,16;DEV=,32'. Note that
THIS DOES NOT BUILD THE FILE! However, when you execute the /KEEP
command (and EDITOR therefore opens the file) or when you open it from
your own or any other program as a new file, it will be opened with a
blocking factor 16 and 32 extents.
If, however, the specified file already exists and has a blocking
factor of 3 and 8 extents and you issue the file equation in the hope
that the equation will magically transform it, you're in for a
letdown. This is because if that file already has a blocking factor
of 3, it will always have a blocking factor of 3 even if you say on
the :FILE equation or when opening the file that it has a blocking
factor of 16. Its blocking factor is 3, and merely opening it with
another blocking factor changes nothing. To truly change the blocking
factor, record size, number of extents, file limit, or any one of the
other file parameters, you need to either rebuild the file (remember,
these parameters can be redefined when you are building a new file)
and copy the old contents of the file into it, or use utilities such
as MPEX/3000.
However, some options can be redefined for OLD files. These are not
the file options (like CCTL or REC) but the access options (like ACC,
BUF, MR, etc.), which are not inherent parts of the file, but rather
attributes of access, which are defined when the file is opened. These
can therefore be redefined for OLD or NEW files.
Another class of :FILE equation parameters governs actions that are to
be performed not at OPEN time, but rather at CLOSE time. The only
parameters in this class are disposition parameters. The SAVE option
instructs the program to close the file as a permanent file; the TEMP
option tells it to close the file as a job/session temporary file
(q.v. TEMPORARY VS. PERMANENT FILES); and, the DEL option will delete
the file referenced when it is closed.
Note that although all :FILE equation parameters correspond to some
FOPEN or FCLOSE parameter, not all FOPEN and FCLOSE parameters can be
redefined with a :FILE equation; for instance, the number of user
labels (on open), or the flag that indicates whether space between end
of file and file limit is to be released (on close) cannot be
redefined with :FILE equations.
If you do not want the user to be able to redefine the open or close
parameters of a file, you should open the file with the Disallow File
Equations bit in the FOPTIONS parameter of the FOPEN intrinsic set.
A GLOSSARRY OF COMMON DISC FILE HANDLING TERMS
ACCESS-MODES The file's ACCESS MODE (IN, OUT, OUTKEEP, APPEND,
INOUT, UPDATE, or EXECUTE) which is defined at file open
time and which restricts the actions that can be
performed on the file. This can be redefined with the
ACC= parameter of the :FILE equation. See APPEND
ACCESS, IN ACCESS, INOUT ACCESS, OUT ACCESS, OUTKEEP
ACCESS, UPDATE ACCESS.
ACCESS-OPTION The ACCESS OPTIONS is a parameter to the FOPEN
intrinsic (q.v.) which defines the access mode, sharing
status, dynamic locking flags, etc. See FOPEN.
ASCII ASCII files are fixed/undefined-length files which are
padded or initialized to blanks instead of zeros. That
is, writing a record that is shorter than the record
size causes the result to be blank-padded. To create,
use ASCII as the 4th subparameter of the REC= parameter
of the :FILE/:BUILD command. See BINARY.
BINARY BINARY files are fixed/undefined-length files which are
padded or initialized to zeros (nulls). To create, use
BINARY as the 4th subparameter of the REC= parameter of
the :FILE/:BUILD command. See ASCII.
BLOCK A BLOCK is the unit in which data is transferred
between I/O devices and file buffers on disc. One
BLOCK = BLOCKING FACTOR records. Each block always
starts on a sector boundary, and thus, for disc space
usage efficiency should be equal to an integral number
of sectors whenever possible. See BLOCKING FACTOR,
BLOCK SIZE.
BLOCKING-FACT The BLOCKING FACTOR is the number of records per block.
To optimize disc space usage, set the blocking factor
such that BLOCKING FACTOR * RECORD SIZE is a multiple
of 128 words. To optimize file access speed, set the
blocking factor as large as possible. To minimize
memory usage, set the blocking factor as small as
possible. To set the BLOCKING FACTOR for new files,
specify it as the 2nd subparameter of the REC=
parameter of the :FILE or :BUILD command. See BLOCK,
BLOCK SIZE.
BLOCK-SIZE For fixed record length files, BLOCK SIZE = BLOCK
FACTOR * RECORD SIZE. For variable record length
files, BLOCK SIZE = BLOCK FACTOR * RECORD SIZE + (BLOCK
FACTOR + 1) * (2 bytes). The most efficient disc space
usage occurs when the block size of a file is equal to
an integral number of SECTORS. See BLOCK, BLOCK FACTOR.
BUFFERING The default mode of file access is BUFFERED FILE ACCESS
-- in this mode records are not immediately read from
or written to disc, but are rather kept in an extra
data segment which contains (BUFFERS) buffers of length
(BLOCK SIZE) words each. See BUFFERS, NOBUF ACCESS.
BUFFERS When a file is accessed in buffered mode, a certain
number of BUFFERS is allocated, each one of length
(BLOCK SIZE) words, in one extra data segment. The
default number of buffers is 2, and can be redefined
with the BUF= parameter of a file equation. See
BUFFERING.
DEADLOCK A situation in which two processes are hung, each one
waiting for the other to do something. This can happen
when several files are locked by processes with MR
capability. See LOCKING FILES, MR CAPABILITY.
DEVICE The DEVICE on which a disc file resides can be a single
disc (specified by placing its device number in the
FOPEN call or as the 1st subparameter of the DEV=
keyword of the :FILE equation) or a device class, a
collection of disc devices grouped under a generic name
(specified in the same place as the device number).
All of the extents of the file are placed on this
device or device class.
DOMAIN The DOMAIN of a file can be PERMANENT or TEMPORARY.
This can be specified in a :BUILD command (;TEMP
indicates TEMPORARY, omission of it means PERMANENT) or
a :FILE command (for old files, :FILE filename,OLD
means PERMANENT and :FILE filename,OLDTEMP means
TEMPORARY; for new files, :FILE filename;TEMP means
TEMPORARY and ;SAVE means permanent). See PERMANENT,
TEMPORARY.
EAR EAR (short for Exclusive Allow Read) is an access mode
which permits a user to open a file for write access,
while still allowing other users read access to the
file. See EXCLUSIVE ACCESS, SHARED ACCESS.
END-OF-FILE The END OF FILE is usually the number of records that
have been written to a given file. It is usually less
than the file limit (q.v.), which is the maximum number
of records in a file, but could be greater than it in
variable record length files (q.v.).
EXCLUSIVE EXCLUSIVE ACCESS to a file is an access mode in which
the accessor forbids everybody else to access that file
while he is accessing it. This mode is the default
mode for all non-read access. It can be specified in
the access options of an FOPEN call or in a :FILE
equation with the EXC parameter. See EAR ACCESS,
LOCKING, SHARED ACCESS.
EXTENT An EXTENT is a collection of blocks that occupies
contiguous space on a given disc. There can be up to
32 such extents in a file; the default is 8. See EXTENT
SIZE, MAXIMUM EXTENTS, NUMBER OF EXTENTS.
EXTENT-SIZE All extents of any file must be of equal length (in
sectors) except the last one, which may be of smaller
length. For formulae for these lengths, see APPENDIX B
-- DETERMINING DISC SPACE USAGE. See EXTENT.
FILE-CODE The FILE CODE of a file is an integer which describes
the type of this file; some of the more common codes
have special mnemonics corresponding to them (e.g. PROG
= 1029 = file code of program files). These mnemonics
show up on :LISTFs of that file, and can also be
specified on a :BUILD or :FILE command. The code,
whether mnemonic or numeric, can be placed on the CODE=
parameter of a :BUILD or :FILE command.
FILE-EQUATION A file equation is a useful tool that allows a user to
redefine certain open or close parameters of the file
(e.g. the file code (CODE), the access type (ACC), the
close disposition (SAVE/TEMP), etc.). It can be
specified through the MPE :FILE command.
FILE-LABEL The FILE LABEL of a file contains information about
that file (e.g. file name, creator id, file code,
record size, extent information, etc.) needed by MPE.
Ordinary users need not worry about this entity.
FILE-LIMIT The maximum number of records permitted in a file,
necessary for knowing how much disc space to allocate,
specified at file creation time. Note that the END OF
FILE can actually exceed the FILE LIMIT for variable
record length files. The file limit can be specified
in a :BUILD or :FILE command as the first subparameter
of the DISC= keyword.
FIXED-LENGTH FIXED RECORD LENGTH files are files whose records have
a fixed length -- if a record of smaller length is
written to the file, the record is padded on the right
with an appropriate number of blanks (ASCII files) or
nulls (BINARY files). An example of this kind of file
is the usual EDITOR file which has a fixed length of 80
bytes. To build fixed record length files, specify F
as the third subparameter of the REC= parameter on a
:BUILD or :FILE command (e.g. REC=-80,,F). See
VARIABLE RECORD LENGTH, UNDEFINED RECORD LENGTH.
FOPEN FOPEN is a system intrinsic which permits its caller to
open a file. BASIC, COBOL, FORTRAN, and RPG users need
not be concerned about this intrinsic because their
languages provide file access features already (this is
therefore mostly used by SPL programmers); however,
this intrinsic is alluded to often in this paper
because all file open commands in all languages
eventually translate out to this intrinsic.
GMULTI-ACCESS GMULTI access is an extended form of MULTI access
(q.v.) available only in MPE IV. Its usage (which can
be specified by appending the GMULTI keyword to the
:FILE equation) together with SHR and ACC=APPEND
provides a painless way of appending to shared files.
See MULTI, SHARED ACCESS.
LOCKING MPE's DYNAMIC FILE LOCKING mechanism (available through
the FLOCK and FUNLOCK intrinsics) gives users a way to
have more than one program write to a file without
jeopardizing data consistency. In order to call FLOCK
and FUNLOCK, the file must have been previously opened
with the dynamic locking access option set (which can
be done in the FOPEN call or using the LOCK parameter
of the :FILE command). See DEADLOCKS, MULTIPLE RINs,
SHARED ACCESS.
MAX-EXTENTS The MAXIMUM NUMBER OF EXTENTS defines the number of
extents (q.v.) into which a file is to be split. Note
that (usually) not all of these extents are allocated
at the time a file is built -- the default is 1
(although more can be allocated initially by specifying
their number as the 3rd subparameter of the DISC=
keyword of the :FILE command). The maximum number of
extents can be specified as the 2nd subparameter of the
DISC= keyword of the :FILE command, and defaults to 8.
See EXTENTS, NUM EXTENTS.
MULTI-ACCESS MULTI access is a form of access that is very useful
for sharing files. It permits you to share not just
the files but also internal file control information
and file buffers. It can be specified by placing the
MULTI keyword on a :FILE command. See GMULTI ACCESS,
SHARED ACCESS.
MULTIPLE-RINs The MR (MULTIPLE RINs) capability is an account, file,
group, and user capability that governs a program's
ability to have more than one file locked at at a time.
In order for a program to be permitted to do this, it
must have been :PREPed with MR capability by a user
with MR capability, and it must reside in a group that
has MR capability. See DEADLOCKS, LOCKING.
MULTI-RECORD MULTI-RECORD ACCESS (abbreviated MR) is a mode in which
a file accessor can read more than one record at a
time, thus greatly speeding up file access. This
option must be used together with the NOBUF option (see
NOBUF ACCESS). It can be specified on a :FILE equation
as the MR parameter. See NOBUF ACCESS.
$NEWPASS $NEWPASS is a special system-defined temporary file
that, when closed, turns into $OLDPASS (q.v.). This
file (and $OLDPASS) disappears (along with all job/
session temporary files) at logoff time. See $OLDPASS,
TEMPORARY FILES.
NUM-BUFFERS The NUMBER OF BUFFERS is the number of I/O buffers
allocated for buffered file access (q.v.). This number
can be specified with the BUF= parameter of a :FILE
equation. See BUFFERING.
NUM-EXTENTS The NUMBER OF EXTENTS is the number of extents that are
currently allocated in the file; this starts out as the
initially allocated number of extents (see EXTENTS) and
is increased by 1 whenever a record which will not fit
into the currently allocated number of extents is
written to the file. See EXTENTS, MAXIMUM EXTENTS.
$OLDPASS $OLDPASS is a special system-defined temporary file
that is the last $NEWPASS (q.v.) file closed. This
file disappears at logoff time, but can be saved with
the MPE :SAVE command. See $NEWPASS, TEMPORARY FILES.
PERMANENT-FILE A permanent file is a disc file which is accessible by
all users in the system (that have the proper access to
it, of course) and remains until it is :PURGEd, as
opposed to a temporary file (q.v.) which can be
accessed only by the session in which it was created
and is automatically deleted when that session logs
off. To specify that a file is to be accessed as an
OLD permanent file, execute file equation of the form
':FILE filename,OLD'; to specify that a file is to be
saved as a NEW permanent file, just place the SAVE
keyword in a :FILE equation for that file. See
TEMPORARY FILES.
RECORD-LENGTH The RECORD LENGTH of a file is the length of each
record in that file if it is a fixed or undefined
record length file, and the maximum length of each
record in that file if it is a variable record length
file. This parameter can be specified as the 1st
subparameter of the REC= parameter in a file equation.
See FIXED RECORD LENGTH, UNDEFINED RECORD LENGTH,
VARIABLE RECORD LENGTH.
SECTOR A SECTOR is 128 words of disc space.
SHARED-ACCESS Files open in SHARED ACCESS mode can be written by more
than one program at the same time. This option can be
specified in a :FILE equation with the SHR parameter.
It is imperative for data consistency that the dynamic
locking (q.v.) facility be used by all programs that
write to a file shared by two or more writing programs.
See EAR ACCESS, EXCLUSIVE ACCESS.
TEMPORARY-FILE A temporary file is a disc file that can be accessed
only in the session that created it, and is
automatically purged when that session terminates. It
can, however, be saved as a permanent file (q.v.) with
the MPE :SAVE command, and purged before the session
logs off with the PURGE filename,TEMP command. To
specify that a file is an OLD temporary file, use a
file equation like ':FILE filename, OLDTEMP; the fact
that it is to be saved as a NEW temporary file can be
specified by appending the TEMP keyword to a :FILE
equation for that file. See PERMANENT FILES.
UNDEFINED-LEN Undefined record length files are not intended to be
used as disc files. Instead, use fixed / variable
record length files. See FIXED RECORD LENGTH, VARIABLE
RECORD LENGTH.
USER-LABELS User labels are records which, although parts of the
file, are transparent to the normal reader of that
file, and which can only be accessed via via FREADLABEL
and FWRITELABEL intrinsics.
VARIABLE-LEN Variable record length files are files in which all
records are not required to have the same length. When
records of a length less than the record length (which,
incidentally, is the maximum length of any record in
that file) are written to the file, no padding is done
(which means that the ASCII / BINARY distinction has no
meaning here), but rather the size of the record to be
written becomes the record length of that particular
record. A result of this is that no space is wasted
due to padding, which makes these files much more
efficient users of disc space than fixed record length
files (q.v.). See FIXED RECORD LENGTH, UNDEFINED
RECORD LENGTH.
DETERMINING DISC SPACE USED BY FILES GIVEN FILE PARAMETERS
Perhaps because there are so many different file parameters (record
size, blocking factor, end of file, file limit, etc.) that are
involved in determining the disc space used up by a certain file, the
formula for this calculation is hard to come by and is quite
complicated. However, we will attempt to list it together with all
its interesting ramifications below. Note that this method will work
only for FIXED RECORD LENGTH FILES that are to be WRITTEN IN A
SEQUENTIAL FASHION (i.e. no directed writes). The parameters needed
for this algorithm are the RECORD SIZE (in words), BLOCKING FACTOR,
END OF FILE, FILE LIMIT, NUMBER OF USER LABELS, and MAXIMUM NUMBER OF
EXTENTS REQUESTED. This method will yield the NUMBER OF SECTORS USED
BY THE FILE,THE EXTENT SIZE OF ALL BUT THE LAST EXTENT, THE EXTENT
SIZE OF THE LAST EXTENT OF THE FILE, THE MAXIMUM NUMBER OF EXTENTS
GRANTED, THE NUMBER OF EXTENTS ACTUALLY USED, and THE BLOCK SIZE OF
THE FILE.
BLOCKING CONSIDERATIONS
The first parameter that must be determined for this calculation is
the BLOCK SIZE, in SECTORS, which we will denote by the 'variable'
name BLKSIZE. Using standard SPL notation, the names BLKFACT =
blocking factor and RECSIZE = record size, and keeping in mind that
ALL DIVIDES PERFORMED BY US FROM NOW ON WILL BE 'CEILING' DIVIDES,
I.E. DIVIDES IN WHICH THE RESULT IS THE SMALLEST INTEGER THAT IS
LARGER THAN OR EQUAL TO THE FRACTIONAL DIVIDE RESULT (E.G. 5/2=3,
20/4=5), we get the following formula:
BLKSIZE:=(RECSIZE*BLKFACT)/128; <<Record size IN WORDS>>
Next, we must find out the number of blocks (not records, but blocks)
that are used up by the data portion of the file and the label (user
label and file label) portion of the file. The formulae for this
(note FLIMIT = file limit, ULAB = number of user labels allocated)
are:
DATABLKS:=FLIMIT/BLKFACT;
LABBLKS:=(ULAB+1)/BLKSIZE;<<the 1 is for the file label>>
TOTALBLKS:=DATABLKS+LABBLKS; << blocks used by both >>
EXTENT CONSIDERATIONS
At this point, we can determine the extent size (in blocks or in
sectors) of each file extent. The formula is (given MAXEXTS is the
maximum number of extents requested by the file creator at creation
time) as follows:
EXTSIZE'BLOCKS:=TOTALBLKS/MAXEXTS; << in blocks >>, or
EXTSIZE'SECTORS:=EXTSIZE'BLOCKS*BLKSIZE; <<in sectors>>
For our purposes, we will use the EXTSIZE'BLOCKS-in-blocks formula.
Now, let us digress for a moment.
As we have said before, the maximum number of extents of a given file
can be specified in a :BUILD or :FILE command, and defaults to 8. But,
the maximum number of extents actually granted (this is NOT the number
of extents actually used!) may be smaller than the maximum number of
extents requested in this way! In order to explain the reason for
this, we must first recall a fact that will be of paramount importance
to us in this entire discussion: ALL EXTENTS OF A FILE MUST BE OF THE
SAME LENGTH, EXCEPT THE LAST ONE, WHICH MAY BE OF SMALLER LENGTH.
Let us suppose that you try to :BUILD a file of 100 blocks with 16 as
the maximum number of extents using, for example an MPE command like
:BUILD MYFILE;DISC=100,16). The file system must fit an integer
number of blocks into one extent. Now, how many blocks can fit into
one extent? Well, the number is 7 (the ceiling of 100/16). But, by
the rule stated above, all extents of a file except the last one must
be of the same length. Thus, each extent except the last one must be
7 blocks long. But, only 14 such extents can fit into 100 blocks,
leaving one 2-block extent! So, the file system cannot possibly grant
you a maximum number of extents larger than 15, even though you asked
for 16!
In short, the 'real' number of maximum extents granted turns out to
be:
REALMAXEXTS:=TOTALBLKS/EXTSIZE'BLOCKS;
where TOTALBLKS and EXTSIZE'BLOCKS were defined above.
Now, the above statements have yet to use the END OF FILE parameter.
Nevertheless, this parameter is a vital one to our calculation. It
permits us to determine another crucial factor, the number of extents
currently used (USEDEXTS), through the following formula:
USEDEXTS:=(LABBLKS+EOF/BLKFACT)/EXTSIZE'BLOCKS;
The above takes the number of blocks actually used by the file and
divides it by the number of blocks per extent, determining the number
of extents actually used.
Now, we have the answer: if the number of extents used is the real
maximum number of extents, i.e. all of the file's extents are
allocated, the number of sectors used can be found by the following:
SECTORS:=TOTALBLKS*BLKSIZE;
If, however, some of the extents of the file remain unallocated, we
can find the number of sectors used with this formula:
SECTORS:=IF USEDEXTS=REALMAXEXTS THEN
TOTALBLKS*BLKSIZE
<< if all extents are allocated >>
ELSE
USEDEXTS*EXTSIZE'BLOCKS*BLKSIZE;
THE FACTS IN A NUTSHELL
In short, the above rantings and ravings boil down to the following
algorithm:
(variables:
MAXEXTS = maximum number of extents requested
RECSIZE = record size (in words) of the file
BLKFACT = blocking factor of the file
FLIMIT = the file's file limit
EOF = the file's end of file
ULAB = the number of user labels allocated in file
BLKSIZE:=(RECSIZE*BLKFACT)/128;
DATABLKS:=FLIMIT/BLKFACT;
LABBLKS:=(ULAB+1)/BLKSIZE;
TOTALBLKS:=DATABLKS+LABBLKS;
EXTSIZE'BLOCKS:=TOTALBLKS/MAXEXTS;
REALMAXEXTS:=TOTALBLKS/EXTSIZE'BLOCKS;
USEDEXTS:=(LABBLKS+EOF/BLKFACT)/EXTSIZE'BLOCKS;
SECTORS:=IF USEDEXTS=REALMAXEXTS THEN
TOTALBLKS*BLKSIZE
ELSE
USEDEXTS*EXTSIZE'BLOCKS*BLKSIZE;
Let us analyze an example case (you can verify it yourself!):
MAXEXTS = 8 extents
RECSIZE = 40 words
BLKFACT = 3 records per block
FLIMIT = file limit of 10000
EOF = 4600 records
ULAB = 0 user labels
BLKSIZE := (40*3)/128 = 1 sector;
DATABLKS := 10000/3 = 3334 blocks;
LABBLKS := 1/1 = 1 block;
TOTALBLKS := 3334+1 = 3335 blocks;
EXTSIZE'BLOCKS := 3335/8 = 417 blocks;
REALMAXEXTS := 3335/417 = 8 extents;
USEDEXTS := (1+4600/3)/417 = 4 extents;
SECTORS := since USEDEXTS(4)<>REALMAXEXTS(8),
then 4*417*1 = 1668 sectors;
A SUMMARY OF METHODS TO SAVE DISC SPACE
The following is a summary of some of the possible methods to save
disc space without deleting files or making them unreadable (methods
are arranged in order of descending average percentage savings):
:% savings: method :
:---------:------------------------------------------------:
: 25%-75% : Convert source files to QEDIT format; this :
: : format is very efficient in terms of disc space:
: : usage yet is still readable by compilers. :
: 25%-50% : Convert data/COBOL files to variable record :
: : length. This can be accomplished with EDITOR's:
: : /SET VARIABLE command. :
: 0%-50% : Improve blocking factor of files; :
: : a file's block size should be a multiple of 128:
: : words or disc space will be wasted. :
: 0%-25% : Set file limit of files to end of file; :
: : if the file limit of a file is not its EOF :
: : disc space is probably being lost. Note that :
: : for data files, the file limit should be :
: : greater than the EOF to allow for expansion. :
: : If you don't want to do this, set the file's :
: : maximum number of extents to 32. :
:---------:------------------------------------------------:
These operations can be performed on files one by one, or en
masse using MPEX/3000.
A SUMMARY OF METHODS TO SPEED UP FILE ACCESS
The following is a summary of some possible methods of speeding up
disc file access, arranged in order of descending average percentage
savings of file access time:
:% savings: method :
:---------:------------------------------------------------:
: 50%-95% : Use MR NOBUF access for file reading/writing; :
: 5%-10% : Increase the block factor of files; :
: : this will increase the block size, and thus the:
: : buffer size of files accessed with buffering :
: : and thus decrease the number of disc I/Os :
: : needed to access them. Make the block size as :
: : large as possible but no more than 8,000. :
: : Set BUF=1 (only 1 buffer) to avoid getting :
: : file system error 57 (OUT OF VIRTUAL MEMORY). :
:---------:------------------------------------------------:
RELATED PAPERS / USEFUL PROGRAMS
As we could not (and never intended to) say everything there is to say
about disc files, we would like to refer you to the following useful
reference documents and utility programs:
PAPERS
ANOTHER MPE FEATURE (BUG)
A discussion of a bug in Multi-Record file access
by Vladimir Volokh
VESOFT, Inc.
SCRUGletter Volume 4 Issue 1 for January 1981
HOW TO AVOID PROBLEMS WITH MPE CARRIAGE CONTROL (CCTL)
All there is to know (well, almost) about Carriage Control
Robert M. Green
ROBELLE Consulting Ltd.
8648 Armstrong Road, R.R. #6
Langley, B.C. V3A 4P9 CANADA
(604) 888-3666
HP3000 Computer Systems MPE Commands Reference Manual
Section VI -- MANAGING FILES
HP3000 Computer Systems MPE Intrinsics Reference Manual
Section III -- ACCESSING AND ALTERING FILES
HP3000 Computer Systems MPE IV Intrinsics Reference Manual
Section III -- INTERPROCESS COMMUNICATION AND CIRCULAR FILES
Section X -- ACCESSING AND ALTERING FILES
PRIVILEGED MODE -- USE AND ABUSE
What is privileged mode and how to use it safely
by Eugene Volokh
VESOFT, Inc.
SCRUGletter Volume 4 Issue 4 for June 1981
SOFTWARE
MPEX/3000
Many useful extensions to MPE
available from VESOFT, Inc.
7174 Melrose Ave.
Los Angeles, CA 90046 USA
(213) 937-6620
QEDIT/ROBELLE
A superior editor, with disc space-saving features
available from ROBELLE Consulting Ltd.
8648 Armstrong Road, R.R. #6
Langley, B.C. V3A 4P9 CANADA
(604) 888-3666
CRYPTIC FILE SYSTEM ERROR MESSAGE DE-CRYPTED
In addition to its other failings, the System Intrinsics Manual does
not explain the exact reason for and/or workaround for most file
system errors. In fact, most file system error messages are very hard
to understand. The following is an attempt at an adequate explanation
of the causes, effects, and workarounds for different file system
errors that pertain to disc files:
0 END OF FILE (FSERR 0): This error is encountered when a program
attempts to read beyond the end of file or write beyond the file
limit. WORKAROUND: Change the program or the file.
1 ILLEGAL DB REGISTER SETTING (FSERR 1): Should never occur for
non-privileged mode programs. For privileged mode programs, this
means that the programmer attempted to do an FFILEINFO, FGETINFO,
FOPEN, or FRENAME in split-stack mode (i.e. after a call to the
EXCHANGEDB or SWITCHDB procedures). WORKAROUND: Do not perform
the function in split-stack mode.
2 ILLEGAL CAPABILITY (FSERR 2): A function that requires privileged
mode capability (e.g. open a file for NOWAIT I/O, open a file for
EXECUTE access, etc.) was attempted without privileged mode
capability. WORKAROUND: Enter privileged mode before executing
the function or do not attempt to execute it at all.
8 ILLEGAL PARAMETER VALUE (FSERR 8): Parameters specified in the
FOPEN call are mutually contradictory; for instance, an attempt
to open a file NOWAIT on a serial disc was detected, or the
program tried to open a new KSAM file without specifying the
FORMALDESIGNATOR or KSAMPARAM parameters on the FOPEN.
WORKAROUND: Correct the parameter.
9 INVALID FILE TYPE SPECIFIED IN FOPTIONS (FSERR 9): The file type
field of the FOPEN file options is not one of 0 (STD = standard
file), 1 (KSAM file), 2 (RIO file), 4 (CIR = circular file), or 6
(MSG = message file). WORKAROUND: Correct the file type field.
10 INVALID RECORD SIZE SPECIFICATION (FSERR 10): The record size
requested was more than 32,767 bytes. WORKAROUND: Specify a
smaller record size.
11 INVALID RESULTANT BLOCK SIZE (FSERR 11): If the user requests
were honored, the block size (BLOCK FACTOR * RECORD SIZE) of the
resultant file would be greater than 32,767 bytes. WORKAROUND:
Specify a smaller record size or blocking factor.
12 RECORD NUMBER OUT OF RANGE (FSERR 12): The user passed a negative
record number to the FPOINT, FREADDIR, or FWRITEDIR intrinsic --
this is illegal. WORKAROUND: Correct your program.
22 SOFTWARE TIME-OUT (FSERR 22): The user tried to read an empty
message file or write to a full message file, an action which
would cause the user to be impeded until the file was no longer
empty or full, respectively (see MPE IV INTRINSICS MANUAL).
However, a time out was set with the FCONTROL intrinsic (mode 4)
and the request timed out before it could be honored. WORKAROUND:
Do not set the time out, or ensure that the request can be
serviced before it times out.
26 TRANSMISSION ERROR (FSERR 26): Hardware failure. WORKAROUND:
Call your CE.
30 UNIT FAILURE (FSERR 30): Hardware failure. WORKAROUND: Call your
CE.
40 OPERATION INCONSISTENT WITH ACCESS TYPE (FSERR 40): The access
type specified at FOPEN time does not permit this operation; for
instance, an FWRITE is not permitted when a file is opened with
ACC=IN. WORKAROUND: Specify an access type at FOPEN time which
permits this operation, or do not perform the operation at all.
41 OPERATION INCONSISTENT WITH RECORD TYPE (FSERR 41): It seems that
this error should never show up and is merely a leftover from a
previous version of MPE.
42 OPERATION INCONSISTENT WITH DEVICE TYPE (FSERR 42): The program
tried to execute an operation that is incompatible with the
device that it is trying to perform it on; for instance, it is
trying to read the line printer or change the baud rate of a disc
drive. WORKAROUND: Do not execute the operation.
43 WRITE EXCEEDS RECORD SIZE (FSERR 43): An attempt was made to
write a record that would not fit in the destination file, e.g.
trying to write a 100-byte record into a file with a record
length of 80 bytes. WORKAROUND: Change the file's record size,
change the length of the record to be written, or open the file
with the Multi-Record (MR) access option.
44 UPDATE AT RECORD ZERO (FSERR 44): The FUPDATE intrinsic (which is
equivalent to the COBOL REWRITE statement) was called with the
record pointer at record 0, which indicates that no record has
been read and therefore no record can be updated. WORKAROUND:
Call FPOINT or FREAD before the FUPDATE call.
45 PRIVILEGED FILE VIOLATION (FSERR 45): A program attempted to open
a privileged file (one with a negative file code; e.g. an IMAGE
file) while specifying a filecode not equal to the file's
filecode or while not in privileged mode. WORKAROUND: Enter
privileged mode before the call or specify the correct filecode.
46 OUT OF DISC SPACE (FSERR 46): The device class on which this file
resides (if this error occurs at extent allocation time) or is
requested to reside (if this error occurs at file creation time)
does not have enough contiguous disc space to accommodate this
file; i.e. if NUMEXTS is the number of extents to be allocated
and EXTSIZE is the size (in sectors) of one extent, this device
class does not have NUMEXTS contiguous chunks of EXTSIZE sectors
each. WORKAROUND: Move the file to another, less full, device
class, decrease the requested file size, or decrease the extent
size by increasing the number of extents in the file.
47 I/O ERROR ON FILE LABEL (FSERR 47): The internal file label of
this file cannot be accessed. Most likely, the file is totally
clobbered and will return INVALID FILE LABEL (FSERR 108) when it
is subsequently accessed. WORKAROUND: None.
48 OPERATION INVALID DUE TO MULTIPLE FILE ACCESS (FSERR 48): One of
the following conditions is true: 1) The program is trying to
purge (i.e. close with disposition DEL) a file that is currently
loaded or being stored/restored, 2) The program is trying to
rename (with the FRENAME intrinsic) a file that it does not have
exclusive access to, or 3) The program is trying to open with
LOCK access a file that someone else has opened with NOLOCK
access or vice versa. WORKAROUND: 1) Don't purge the file or
wait for the file to become purgeable again, 2) Don't rename the
file or open the file with EXC access, or 3) Open the file with
LOCK or NOLOCK access (whichever the other program has the file
open with).
49 UNIMPLEMENTED FUNCTION (FSERR 49): The program specified an
invalid parameter value in a file system intrinsic call; e.g. a
disposition of 5, 6, or 7 at FCLOSE time or a file type of RIO on
pre-Athena systems (ones which do not support RIO files).
WORKAROUND: Correct your program.
50 NONEXISTENT ACCOUNT (FSERR 50): An attempt was made to open a
file in an account which was not configured in the system.
WORKAROUND: Correct the filename or build the account.
51 NONEXISTENT GROUP (FSERR 51): An attempt was made to open a file
in a group which was not configured in the system. WORKAROUND:
Correct the filename or build the group.
52 NONEXISTENT PERMANENT FILE (FSERR 52): An attempt was made to
open a file which does not exist. WORKAROUND: Correct the
program or build the file.
53 NONEXISTENT TEMPORARY FILE (FSERR 53): The program tried to open
a temporary file which does not exist. WORKAROUND: Correct the
program or build the file.
54 INVALID FILE REFERENCE (FSERR 54): The program tried to open a
file whose filename was invalid; for instance, the file, group,
or account name was longer than 8 characters long, an invalid
system-defined file was specified (e.g. $XYZZY), or no file
equation was found for a back-referenced file (e.g. *MANSION with
no file equation for file MANSION). WORKAROUND: Correct the
filename specified.
55 DEVICE UNAVAILABLE (FSERR 55): The program tried to open a
message file that was already opened for MULTI (not GMULTI)
access by another job/session. WORKAROUND: Wait for the other
job/session to finish, or rewrite the other program to open the
file NOMULTI or GMULTI.
56 INVALID DEVICE SPECIFICATION (FSERR 56): The device number or
device class on which the file was to be opened is not configured
on the system. WORKAROUND: Correct the program.
57 OUT OF VIRTUAL MEMORY (FSERR 57): The buffer size (NUMBER OF
BUFFERS * RECORD SIZE * BLOCKING FACTOR) of the file to be opened
exceeds 8,192 words (or 14,000 words starting with the D MIT
version of MPE). WORKAROUND: Decrease the number of buffers (by
specifying BUF=1 on a :FILE equation, for instance), decrease the
record size of the file, or decrease the blocking factor of the
file.
58 NO PASSED FILE (FSERR 58): The program attempted to open
$OLDPASS, but no $OLDPASS file exists. WORKAROUND: Correct the
program or build a $OLDPASS file.
60 GLOBAL RIN UNAVAILABLE (FSERR 60): The program requested dynamic
locking at file open time, but the RIN (Resource Identification
Number) necessary for dynamic locking could not be obtained.
WORKAROUND: Free some global RINs (with the :FREERIN command),
file RINs (by closing files opened with LOCK access), open the
file with NOLOCK access, or enlarge the RIN table.
61 OUT OF GROUP DISC SPACE (FSERR 61): The program tried to allocate
more disc space than is allowed for a given group; e.g. it tried
to build a 10,000-sector file in a group which already had 95,000
sectors and was limited to 100,000 sectors. WORKAROUND: Decrease
the amount of disc space used by files in that group (by purging
or squeezing files) or ask the account manager to increase the
group disc space limit.
62 OUT OF ACCOUNT DISC SPACE (FSERR 62): The program tried to
allocate more disc space than is permitted for the account in
which it tried to allocate it. WORKAROUND: Decrease the amount
of disc space used by files in that account (by purging or
squeezing files) or ask the system manager to increase the
account disc space limit.
64 USER LACKS MULTI-RIN CAPABILITY (FSERR 64): The program was not
:PREPed with MR (Multi-Rin) capability, yet tried to lock a file
when another file (or RIN) was already locked by that program.
WORKAROUND: :PREP the program with MR capability or do not try to
lock a file when you have already locked another one.
71 TOO MANY FILES OPEN (FSERR 71): The program attempted to open a
file, but there was not enough room in the system area (PCBX) of
the program's stack for the information for that file.
WORKAROUND: Close some files which are no longer necessary before
trying the open, or run the program with the ;NOCB keyword in the
:RUN command.
72 INVALID FILE NUMBER (FSERR 72): An attempt was made to access
(e.g. read or write) a file that had not been opened or that is a
privileged file; for instance, a read was requested against file
number 10, but no file file number 10 was open. WORKAROUND:
Correct your program or enter privileged mode before trying to
access the file (if the file is privileged).
73 BOUNDS VIOLATION (FSERR 73): You are attempting to read or write
more data than could fit into your I/O buffer (e.g. you are
trying to read 100 words into an 80-word array). WORKAROUND:
Decrease the length of the data you are trying to read or write
or enlarge your program's I/O buffer.
74 NO ROOM LEFT IN STACK SEGMENT FOR ANOTHER FILE ENTRY (FSERR 74):
When trying to do an FOPEN, the file system tried to expand your
stack data segment to have room for the information on the file
being opened; this caused an error because it would make the data
segment too large. WORKAROUND: Often, this does not mean that you
were using too much stack space AT THE TIME OF THE FOPEN. Rather,
you might have used a lot of stack space earlier, and that space
was not deallocated (since your stack data segment is never
automatically shrunk by the system). If you call "ZSIZE (0)" (to
shrink the stack data segment to only the amount you're currently
using) immediately before the FOPEN call, the problem may very
well go away. If it doesn't, you might consider using less stack
space or running your program with ;NOCB.
90 EXCLUSIVE VIOLATION: FILE BEING ACCESSED (FSERR 90): Exclusive
access was requested to a file which is already being accessed;
thus, exclusive access cannot be granted. WORKAROUND: Specify
SHR (shared) or EAR (exclusive - allow read) access when opening
the file or wait for the accessor to close the file.
91 EXCLUSIVE VIOLATION: FILE BEING ACCESSED EXCLUSIVELY (FSERR 91):
Access was requested to a file which is being accessed
exclusively by some other user. WORKAROUND: Wait for the
accessor to close the file.
92 LOCKWORD VIOLATION (FSERR 92): An invalid lockword was specified
at file open time or when the file system prompted the user for a
lockword. WORKAROUND: Specify a correct lockword or remove or
change the lockword on the disc file.
93 SECURITY VIOLATION (FSERR 93): Permitting the user to access this
file in the specified access mode would be a breach of file
security. WORKAROUND: Change the access mode specified in the
program to one which is permitted or ask the file's creator to
:RELEASE or :ALTSEC the file.
94 USER IS NOT CREATOR (FSERR 94): An attempt was made to :RENAME or
FRENAME a file by someone other than the file's creator.
WORKAROUND: Do not perform the :RENAME or FRENAME, ask the
creator of the file to do the :RENAME, or (if you have read and
write access to the file and are a user of MPEX/3000) use MPEX's
%RENAME command.
96 DISC I/O ERROR (FSERR 96): Hardware failure. Call your CE.
100 DUPLICATE PERMANENT FILE NAME (FSERR 100): The program tried to
save (close with SAVE disposition) a new or temporary file as a
permanent file, but a permanent file with that name already
exists. WORKAROUND: Purge the other file with that name.
101 DUPLICATE TEMPORARY FILE NAME (FSERR 101): The program tried to
save as a temporary file (close with TEMP disposition) a new
file, but a temporary file with that name already exists.
WORKAROUND: Purge the other temporary file with that name.
102 DIRECTORY I/O ERROR (FSERR 102): The directory (or part of it) is
clobbered. You're in big trouble. WORKAROUND: None.
103 PERMANENT FILE DIRECTORY OVERFLOW (FSERR 103): There is no more
room in the system file directory for this file (the system file
directory typically allows approximately 1,200 files per group).
WORKAROUND: Purge some of the files in the group in which you
wish to build the file.
104 TEMPORARY FILE DIRECTORY OVERFLOW (FSERR 104): There is no more
room in your job / session temporary file directory for this
file. WORKAROUND: Purge some temporary files or :RESET some
:FILE equations or :CRESET some :CLINE equations.
105 BAD VARIABLE BLOCK STRUCTURE (FSERR 105): The variable record
length file being accessed has an inconsistent structure, or
would have an inconsistent structure if this access were to go
through (if you are writing NOBUF). WORKAROUND: If you are
writing NOBUF, correct your program; otherwise, none.
106 EXTENT SIZE EXCEEDS MAXIMUM (FSERR 106): The program attempted to
build a file which would have extents larger than 65,534 sectors,
the maximum permitted. WORKAROUND: Increase the number of
extents in the file or decrease the extent size by decreasing the
record size or file limit of the file.
107 INSUFFICIENT SPACE FOR USER LABELS (FSERR 107): The maximum
number of user labels for a file is 254. WORKAROUND: Decrease
the number of user labels requested by the program.
108 INVALID FILE LABEL (FSERR 108): The file is inaccessible because
the file is invalid (probably destroyed - and unrecoverable).
WORKAROUND: None.
109 INVALID CARRIAGE CONTROL (FSERR 109): The program tried to do a
write with a CCTL code of 1 (embedded CCTL) but with a buffer
length of 0; or, the program attempted an FCONTROL mode 1
(transfer CCTL code) with a parameter of 1. WORKAROUND: Correct
the program.
110 ATTEMPT TO SAVE PERMANENT FILE AS TEMPORARY (FSERR 110): An
attempt was made to close a permanent file with temporary (TEMP)
disposition: this is illegal. WORKAROUND: Correct the program.
148 INACTIVE RIO RECORD (FSERR 148): An FPOINT, FREADDIR, or FSPACE
positioned the record pointer at an inactive record in an RIO
(Relative I/O) file. WORKAROUND: None necessary.
149 MISSING ITEM NUMBER OR RETURN-VARIABLE (FSERR 148): An item
number was specified without a corresponding variable or vice
versa in an FFILEINFO intrinsic call. WORKAROUND: Correct the
program.
150 INVALID ITEM NUMBER (FSERR 150): An item number specified in an
FFILEINFO intrinsic call is invalid. WORKAROUND: Correct the
program.
ACKNOWLEDGEMENTS
Praises and kudos GOTO:
Robert Saunders (of the HP lab) for much important information;
Vladimir Volokh (of VESOFT, Inc.), Robert Green (of ROBELLE
Consulting), Joe Berry (of the Los Angeles Airport Area SE Office),
and many others for comments, questions, criticisms, suggestions, and
overall moral support.