This
description is based mostly on Markus Fritze's article. He is floppy
expert and made some protections for Atari ST in past. I added couple
things to it + rearranged some parts . Made without format's author
contribution, this is certainly not complete. May contain some minor
errors, but intention is to help people who want to do something more
with thousands of STX images available - and not just playing under
emulators.
First must to say some relevant things: Pasti STX image format is not real floppy image. Of course, I talk about images available around. Is there some non/published variant with more details, more suitable for writing onto floppies - it is not on me to talk about. I examined over 1000 STX images already. There is 2 major type: made with stock Atari ST machine and public avail. SW. Another is made with Discovery Cartridge. Main difference between is way how track image is made. 'Public' images contain track images as WD1772 floppy controller reads them. Such images are good for emulators, but not (really) for writing onto floppy. Plus, track image can not hold sector data, and it must be stored separately, so such STX images are longer - typical size is 2.1x of same format unprotected floppy image. With DC made images can hold sector data inside track image, so size is usually 1.5x of same format unprotected floppy image. More later.
Other relevant informations regarding copy protections: there is fuzzy bit mask, timing info for every sector, FDC status flag etc. All it is mostly in form suitable for emulation, and not for floppy write. No wonder - Pasti is made in cowork with Steem authors, and for sure with heavy usage of Steem Debugger + Pasti Debugger in.
What are chances for having floppy write option from Pasti images one day ? How Pasti images can be used with HxC floppy emulator ?
I'm not much optimistic about writing onto floppies. Even if it will be made, it will be limited. So, don't dream that you will be able to write any STX onto floppy. I'm not say that it is entirely impossible. Just conversion from current format to writeable one is very complicated. Will someone do it ever ? With HxC it is better - some STX images already are usable. Improving support for STX is currently under work (Sept. 2011). But there is danger of protections with delayed effect. It will let you play, but on some later stage will crash machine or screw screen, or similar. So, real testing may be time consuming. Track read based protections are common, and HxC just started to emulate some of them ...
Update: HxC support is stopped at end of 2011. Someone just lost motivation .
The format self:
File header (16 bytes)
0x00 : 'R','S','Y',0 - File ID
0x04 : 0x0300.w - Probably the version number of the format
0x06 : 0x0100.w for public, or 0xCC00 for DC type
0x08 : 0.w *
0x0a : tracks.b (e.g. 80 or 160 for double sided)
0x0b : 1.b *
0x0c : 0.l *
(w-word, 16 bits, b-byte, l-long, 32 bits. Little endian)
I saw 3 STX file header types so far:
this is
by images made with public tool and stock Atari ST .
images
made with DC .
with DC too, I guess.
After the file header one block for each track is following. Each block also starts with a 16 byte header:
Track Header (16 bytes):
0x00.l : length of the total track data in bytes. This value plus the current position in the
file points to the following track header.
0x04.l : bytes of fuzzy sector mask in this track
0x08.w : number of sectors in the track - but some may be only sector address blocks, without sector data block - then loading time is 0 .
0x0A.w : bitmask with flags
0x0C.w : length of the track image in bytes (around 6kb is normal)
0x0E.b : track number - it is: 0 - last track on side A (usually 0-79 or 81), and 128 - last track on side B (128-207/9) . So, side is determined by bit 7 (clever) .
0x0F.b : Track image type flag: if 0 image is WDC track dump (public tool made). If 0xCC image is DC type .
The bitmask for the flags (word at 0x0A in track header)
7 - The track image data also contains a sync-offset to the beginning of the first sync marker. It is word at track image data start .
6 - If set, and track image type flag is 0 : the track is saved as an image with all gaps via the read track command. Sector contents are after track data (public t.).
The reason is that WD1772 track read command looses sync during track read, so sector data is not properly read (more later).
If set, and track image type flag is 0xCC: the track is saved as image with DC device. Sector data is usually embedded, but some may be after track image data.
In all cases, sector data offsets must be calculated from track image data beginning - more later.
If bit 6 clear, no track image data.
5 - with PASTI 0.4b always set, if Bit 0 is set (= track protected)
4 - unused
3 - unused
2 - unused
1 - unused
0 - If set the track is protected or with a non-standard sectorsize (!= 512 bytes). Well, it should be so. But in reality with public type images it is always set. So, by unprotected tracks too. Likely because there is always track image too.
Above description may little confuse. In practice, there are some typical values:
0xC1 - Usually by made with DC - track is not saved with WD1772 read track command, but with DC - but not always. See track image type flag for sure.. Typical track length is about 6KB. Sector datas are inside track data. But some may be after .
0x61 : made with public tool. There is track image, as WD1772 read track command readed it. There are all sector datas too. It means that total track data is about 11KB. But in practice, track image is needed only for couple tracks (protections) or for none. So, there is overhead, caused by reason that images are made usually by people not able to determine what tracks are protected.
0x21 : there are only sector datas. No track dump. Usually by DC type. By public not - all tracks are usually with flag 0x61 .
Empty tracks are with flag 0x21 and 0 sectors. Actually, then there is only track header present.
If Bit 0 is clear, the track is unprotected and the number of sectors as stored in the track block header are simply saved as 512 byte blocks directly after the header. The track is therefore 16 + 512 * sectors bytes long. Rare case to see, even if most of tracks on many floppies are actually unprotected.
If Bit 6 and 7 are clear, there is no track image and the sectors are stored just by order end length - just like the fuzzy sector mask, except this time for all sectors. Warning: sectors can be missing, if only a header was read, but the data was missing. The FDC status has Bit 4 (Record not found) set in this case.
If Bit 6 is set and 7 is clear, the track image is written with a track image header and the sectors follow the track image in the usually order and length. The track image header is just a word (little endian, as usual) that contains the length of the track image.
Bit Bit 6 and 7 is set, the track image is written with a track image header that has two words: the first one is a byte offset to the first 0xA1 address mark sync, the second word is the length of the track image in bytes. Usually DC type, but not always.
If Bit 7 is set, the sector data is usually merged inside the track image, the sector offsets then point inside the actual track image. However, it is possible for single sectors to have an offset _behind_ the track image, if PASTI couldn't map them correctly into the track image. A software should always read the sectors based on the sector offset in the sector header. See track image type flag ....
If Bit 0 is set, the track header is first followed with the sector headers. One for each sector in the track, again 16 bytes per header.
Sector Header (16 bytes):
0x00.l : sector offset relative to last sector header end , or to end of fuzzy mask if exists.
0x04.w : position of the sector header relative to the beginning of the track as a
time stamp. The position timing is depended on the RPM speed of the drive,
but for a ~6250 byte track the value for a full rotation is around
50000 (for 300 rpm). This allows software to simulate the time it takes
to find the sector and read it, something certain copy protection schemes
test.
0x06.w : read timing of the sector data, certain copy protections write sectors so, that
it takes longer to read them. The protection measures the time to read the
sector. 16384 is the default value, larger values mean that it takes longer
to read the sector than normal. However if the measured value by PASTI is
within 2% of this value (16384 +/- 320), the value written to the stx file
is 0 as a sign for a stable read speed.
0x08.b : Track number from the address block identifying the sector (typically 0..79)
0x09.b : Side from the address block identifying the sector (typically 0 or 1)
Note: above 2 may be false, as part of the protection (Gold of the Aztecs + others) .
0x0a.b : Sector from the address block identifying the sector (typically 1..9 or 10, 11)
0x0b.b : Size of the sector from the address block identifying the sector
(typically: 2=512 or 3=1024 bytes) - but may be more. Then only bits 0-1 matter.
0x0c.b : First byte of the CRC over the address block
0x0d.b : Second byte of the CRC over the address block
0x0e.b : FDC status register after reading the sector
0x0f.b : Sector flags, always 0x00
The CRC is a CCITT CRC16 calculated by the FDC. It is initialized with 0xFFFF and includes the sync marks, therefore typically 0xA1,0xA1,0xA1,0xFE plus the 4 bytes from the address block.
The FDC status is typically 0x00. Also common is 0x10 for a sector not found and 0x08 for a CRC error when reading. Bit 7 has a special meaning: it hints that the sector has fuzzy bits and the fuzzy sector mask should be used. If Bit 7 is set, Bit 3 is usually also set, because fuzzy bits will trigger a CRC error inside the sector.
After the sector headers the fuzzy sector mask is following, if the value is != 0 in the track header. The mask identifies the bits that are random in a given sector. The size of the fuzzy sector mask depends on the sector size (from the header) and the number of fuzzy sectors in the track itself. The fuzzy sector mask has bits set for every bit in in the sector, that is _not_ random. Therefore a byte generated with the mask could be calculated like this:
fdcSectorByte[byteOffset] = (sectorData[byteOffset] & fuzzyMask[byteOffset]) | (rand() & ~fuzzyMask[byteOffset])
Following the fuzzy sector mask is the track data area . This area can contain either the sectors by itself (when no track image), or the complete track image with the sectors embedded (DC type), or the complete track with the sectors separate (following the track image) (public type). How they are stored is determined by Bits 6 and 7 in the track flags.
If neither Bit 6 or 7 in the track flags is set, after the fuzzy sector mask, the sectors are written to the file. only the content of the sectors is written. Again with the sector size in the header taking into consideration. These tracks can be also easily recognized by sector offsets that are always a power of 2 (typically in 512 byte increments), starting at 0.
The track order:
Sometimes tracks are imaged in sidewise order: side A: track 0, 1, 2 ... 79 , then side B: track 0, 1, 2 ... 79 . Sometimes alternating: side A track 0, Side B track 0, side A track 1, side B track 1, etc. I don't see special flag for it, but it is easy to get order type by looking second track record - if there is 128 for track, then it is alternating type.
A typical protected file looks like this:
*
File Header
*
Track Header
o
Sector Header
o
Sector Header
o
Sector Header
o
…
o
Fuzzy sector mask - if exists
o
Track data - may be track image self (public) or image containing sector datas. May missing.
o
Sector data x sector count - if not in track image.
*
Track Header
o
Sector Header
o
Sector Header
o
Sector Header
o
…
o
Fuzzy sector mask - if exists
o
Track data - may be track image self (public) or image containing sector datas. May missing.
o
Sector data x sector count - if not in track image.
*
repeat ...
The track image:
It begins with track image header - 2 or 4 bytes depending on bit 7 of track flag. (see above) .
As said in beginning there is 2 type of track image. WD 1772 floppy controller, used in Atari ST serie has track read command. But it reads track content not exactly as may expect - the point is that WD 1772 does resync during whole track read, so while reading sector data too. It results in reading not exactly what is written. More precise, the bit seq. %000101001 at any place in track will init resync, what causes that following bits will be readen not correctly.
With DC made track images have no such problem, and I guess that can be used by floppy writing more-less direct.
But do we really need track image for floppy write, and how it is used by copy protections ? First to answer on second: generally, by copy protections there is 2 way of track image usage: first, more common is using track read and testing for some specific data pattern, what differs from standard floppy format. For instance Copylock from 1988 will seek 2 byte serial right before first sector address block sync mark, on track 0. Normally, there are zeros.
Second way is to use track image for data read. It allows lot of data - up to 6KB/track, but needs some special measures, to avoid false resync. By Maupiti Island all problematic sequences are prefixed with byte 7, by International Soccer Challenge with byte 0x0F (the effect is same). It avoids appearance of mentioned bit seq. so all bytes will be readen correct, after first and only sync. Loader rutine then must restoring correct data by removing prefixes and inserting correct values . Writing such track with regular Atari (or PC) is not possible . + 6KB/track is not possible anyway without special equipment.
Back to possibilities to write floppies basing on WD1772 track image read: I guess that it is possible, at least by most. But needs proper conversion, what may be not easy.
How Pasti works in practice ?
Works well with Steem emulator, and likely with Saint too. HxC support is improved . And we have thousands of images avaialable around.
But there is a lot of STX images for not copy-protected floppies. Because people can not judge what is really protected, and by some recommendations (do it with Pasti for sure) . Maybe the plan was that some team will verify Pasti images and make list which are really protected, which are OK ? Nothing from it is realised, for sure. Actually, author did nothing, really and literaly nothing in last 4 years. Or maybe 7 years, considering that public imaging tool (v0.4b) is made in 2005. Still beta ! In meantime count of images groved, and poeple want some details. I hope that this will at least partially satisfy needs.
I already did SW for converting STX to STT format (only with specific protections, but already have about 60 working STT images, usable with HxC and some even writeable onto floppies). SW for shrinking oversized STX files by dropping unnecessary track images (only for experts).
Maybe this can help to add STX support for some other emulator (Hatari) ?
Technically, it seems as great achievement. As flaw I can count lack of checksum for Pasti files. Actually, it would be best to have checksums for all track datas separately. Considering not too reliable ways how STX images are transferred to PC: with transfer floppies, in multiple chunks, or via some cable connection between ST and PC. Even hard disk transfer is not 100 % safe.
The best thing is that is possible to make images of copy protected floppies with stock Atari ST machines. And result of it is really lot of titles imaged. But looking about how author self supports it (or better said, how supports it not) I can only wish that someone other, with more sense for community and communication would do it.
Credits go to Markus Fritze, Klaus Brod and hunderts of people who made Pasti popular by imaging, sending, uploading, hosting image files.
Rough Pasti image overview - what is available around:
P - made with public tool, C - with Discovery Cartridge
P - contains WD track dump (track read command data) for all tracks
C - not always contains all DC track dumps - may have only sector datas
16 Bit Hit Machine (3 titles) , 27.1.2011 - P
1943 The Battle of Midway , 6.8.2005 - P
1st Division Manager , 15.11.2005 - C, all tracks
221B Baker Street, 30.7.2010 - P
3D Pool Microprose, 16.4.2009 - P
3D Pool Microplay , 16.4.2009 - P Seems as complete same as Microprose
version above - date, time and len is same !
3D Pool Mirror Image , 16.4.2009 - P. Funny - same date, just this is 2 sided.
Likely all 3 '3D Pool' images are from very same floppy .
3D Tennis , 17.3.2007 - P, unprot
4 Wheel Drive col. (2 flo), Date invalid, - P
5 Intelligent Strategy Games (2 flo) , 1.12.2005 - P, unprot.
above correct title is: Intelligent Strategy Games 5
520 STE Discovery Xtra (5 flo) , 5.7.2009 - P
5th Gear , 14.12.2008 - P
9 Lives , 19.6.2005 - P
9 Lives , 1.11.2005 - C, all tracks
A320 Airbus , 31.10.2006 - P, unprot
Aaargh! (2 flo), 32.1.2010 - P
ABZoo (2 flo) , invalid date - P
Action Service , 19.6.2005 - P
Action Pack (10 flo) , 8.11.2010 - P
Action ST Vol 1 (4 flo) , invalid date - P
Action Fighter , 30.11.2005 -C, no tracks
Addams Family , 13.8.2005 - P
Addictaball , 7.3.2010 - P
Bards Tale, The (2 flo) , 29.10.2004 & 3.12.2005 - C
James Pond , 13.6.2006 - P
Kristal, The (4 flo) , 25.6.2005 - P
Masterblazer , 20.12.2009 - P
Mickey Mouse (2 flo) , 19.6.2006 - C, no tr.
Spherical (2 flo) , 17.3.2010 - P
Trinity , 2.8.2008 - P
Wild Streets , 16.12.2004 - C, no tr.
Zombi (2 flo) , invalid date - P
Notes: started by Alphabet, then choosing randomly few from STX collection with over 1500 images - taken from GameBase ST, diverse WEB sites (currently Atarimania is most active and has most) . Obviously aprox. 90 % is made with public tool .
Why much more public type STX images ? For sure because most of people have no Discovery Cartridge :-) . But this initiates some thoughts about whole 'Pasti preservation project' :
I'm pretty sure that at start ideas, plans were much bigger than it outcame. Of course it is not the end, we may see new and new images yet for years. But I doubt that number of DC type Pasti images will be very big ever. And likely, only that type has real perspective to be writeable (because of track image type) onto floppies or working with HxC . So, the preservation is done (~70% of all Atari SW), but is it real preservation ? I mean beeing limited on SW emulators only. I leave it on reader to judge. What I can say is: 7-8 years after starting creation of Pasti format and SW we still missing informations, relevant and simple infos. Jorge Zwick is persona not talking much , at least not talking about something he should, and he was asked many times by many people. Pasti WEBpage contains very little information, not updated for years. SW is in beta stage for 8 years ... Worst is that beside lack of infos we have misleading informations: about publishing specs, sources, plans about floppy write support etc. Excuse of beeing busy just works not for me. Not giving any specs in 8 years ? Come on ... There must be some other reason, and I guess that it is related with lacking or 'fuzzy' replies to people in forums :-)
For the end, very important issue: reliability of STX images
Are all Pasti images 100% OK ? What are chances that there are errors in STX image ?
Usually people test STX images before make them public available. Test in Steem, of course. But how test ? I was not present at any of imaging made by someone other, but can say with confidence that at least 99% tests is made by checking only is game starts, until begin level, or couple levels more.
And here is possible PROBLEM : I start with recent case: I imaged game Voyager with public Pasti. Floppy is 22 years old. Only with fourth floppy drive I got working STX image. Running game directly from floppy confirmed that problem is not Pasti - with first drive game worked not on real Atari from original floppy. I examined images, and found that error was exactly in copy protection. It is Copylock, and sector 6 on track 0 was readen incorrect in first 3 imagings. It is slower reading sector with fake CRC error. But must be readen 100 % correct - if not, game will not start. Pasti did not notice error - normal, as there is CRC error reported by FDC. It just confirms that CRC error is put there with purpose, to confuse copy SW. This was straight case, and imaging error was obvious after short testing.
But we have a lot of copy protections with delayed effect. or performing check(s) in later game stage(s). It was btw., recommendetation of Rob Northen self too. So, it may happen that protection check fails, but game still will work well until some later stage. Or floppy read for protection check will happen on stage 11, for instance.
So, the only reasonable answer is: we can not be in all cases sure that some Pasti image is 100% OK, because there are situations when image may contain error(s), mostly caused by age of floppy disks imaged, + by floppy drive, head align and similar. Pasti HW and SW is not enough sofisticated to detect all possible errors during imaging.
So, playing games from STX image(s) until end is required test - about image correctness.
It still not proves that every bit is readen correct, but is good check for correct imaging of crucial part: copy protection.
Generally, copy protections are what makes floppy imaging very hard and troublesome. And obviously, the place where most likely it may fall. My opinion is that true preservation should include instead images of copy ptotected public releases : the sources of original, and executable SW in not copy protected form. But as things are in real World - it will be not much of such ever. Many source is lost forever, authors, copyright holders care not for preservation. They have no any copies, so for some SW only source is pirate crack. It is big shame for all involved, and maybe just indicates that all what interesting them is money.
And that above is not only some dark scenery, theory: many of Atari ST floppies with game originals after 1989, 1990 have some copy protection with delayed effect. It may be over 200 titles. So, it is almost sure that some STX images are bad, considering that imaging was made when disks were aprox. 18 years old in average.
Update: Expectable problems with Pasti images and what we have online
As said, Pasti imaging is not flawless. Since author does nothing to better inform people about it, not giving any clues about advanced usage, I feel need to write here my experiences and recommendations to make images with less possible errors.
If reading is unreliable some sectors may be readen bad. Then in normal case SW reports CRC errors. Pasti usually warns not, just writes readen, with CRC error flag on - assuming that it may be copy protection. The real bad thing is that quick checking of image will not show any errors, because need to play more-less to reach problematic part. It was case for instance with image of Kid Gloves posted at AF.
Average user can hardly determine such problems - but examining STX file may give some clue - if there are CRC errors in middle of disk, over more sectors, it is likely not copy protection.
What average user can, and actually need to do is: taking care about cleanliness of floppy drive. Floppies self is not recommended to clean, but methode of multiple reading will for sure make it cleaner.
There are cases when imaging just don't work, and program stops. I will not go much in that here - reasons may be some not supported protection, or just very bad condition of disk. If cleaning helps not, it is likely too damaged, and impossible to get correct image. Still, it may be worth to keep such floppies. Some parts must be still OK - may help in building image from multiple sources. This initiates thoughts about how to coordinate hunt for missing rare titles ..
Another case, where image can fail is when copy protection is not enough tolerant, and happens with some floppy drives used by imaging. It is certainly rare case, but again, quick test of image may be noot enough. So, it may happen that drive gives index pulse little earlier or later than coders expected, and then copy protection check will detect it as pirate copy. I saw such in case of games To Be On Top and Rubicon.
Btw. not enough tolerant code is not so rare case, and not only at copy protections. For instance famous FastCopy Pro 3 had problems with formatting floppies on several floppy drives - for me it never worked. It was corrected in later revisions.
Fortunately, such 'bad' STX images (with little shifted track image) can be used to get 'perfect' copy - by running in Steem Debugger can see fine how copy protection check fails, and correcting STX image by shifting little problematic track image.
Delayed protection problem is likely not only Pasti problem, but general problem with floppy images.
What is good in all this: in fact, testing of STX images is done in many cases. Not in some organised form, but just by playing games until end in emulators. Of course, mostly quality games were played. People could report problems in diverse forums. Normally and especially with Atari ST, you can not be always sure that the problem is bad STX (image), but after some time things getting clarified and some can finish game, and say: it is good :-)
Simple SW for giving infos about STX image:
DL PastiInf - usage help included.
P. Putnik , Latest update: Jan. 2013.
First must to say some relevant things: Pasti STX image format is not real floppy image. Of course, I talk about images available around. Is there some non/published variant with more details, more suitable for writing onto floppies - it is not on me to talk about. I examined over 1000 STX images already. There is 2 major type: made with stock Atari ST machine and public avail. SW. Another is made with Discovery Cartridge. Main difference between is way how track image is made. 'Public' images contain track images as WD1772 floppy controller reads them. Such images are good for emulators, but not (really) for writing onto floppy. Plus, track image can not hold sector data, and it must be stored separately, so such STX images are longer - typical size is 2.1x of same format unprotected floppy image. With DC made images can hold sector data inside track image, so size is usually 1.5x of same format unprotected floppy image. More later.
Other relevant informations regarding copy protections: there is fuzzy bit mask, timing info for every sector, FDC status flag etc. All it is mostly in form suitable for emulation, and not for floppy write. No wonder - Pasti is made in cowork with Steem authors, and for sure with heavy usage of Steem Debugger + Pasti Debugger in.
What are chances for having floppy write option from Pasti images one day ? How Pasti images can be used with HxC floppy emulator ?
I'm not much optimistic about writing onto floppies. Even if it will be made, it will be limited. So, don't dream that you will be able to write any STX onto floppy. I'm not say that it is entirely impossible. Just conversion from current format to writeable one is very complicated. Will someone do it ever ? With HxC it is better - some STX images already are usable. Improving support for STX is currently under work (Sept. 2011). But there is danger of protections with delayed effect. It will let you play, but on some later stage will crash machine or screw screen, or similar. So, real testing may be time consuming. Track read based protections are common, and HxC just started to emulate some of them ...
Update: HxC support is stopped at end of 2011. Someone just lost motivation .
The format self:
File header (16 bytes)
0x00 : 'R','S','Y',0 - File ID
0x04 : 0x0300.w - Probably the version number of the format
0x06 : 0x0100.w for public, or 0xCC00 for DC type
0x08 : 0.w *
0x0a : tracks.b (e.g. 80 or 160 for double sided)
0x0b : 1.b *
0x0c : 0.l *
(w-word, 16 bits, b-byte, l-long, 32 bits. Little endian)
I saw 3 STX file header types so far:
this is
by images made with public tool and stock Atari ST . images
made with DC .
with DC too, I guess.After the file header one block for each track is following. Each block also starts with a 16 byte header:
Track Header (16 bytes):
0x00.l : length of the total track data in bytes. This value plus the current position in the
file points to the following track header.
0x04.l : bytes of fuzzy sector mask in this track
0x08.w : number of sectors in the track - but some may be only sector address blocks, without sector data block - then loading time is 0 .
0x0A.w : bitmask with flags
0x0C.w : length of the track image in bytes (around 6kb is normal)
0x0E.b : track number - it is: 0 - last track on side A (usually 0-79 or 81), and 128 - last track on side B (128-207/9) . So, side is determined by bit 7 (clever) .
0x0F.b : Track image type flag: if 0 image is WDC track dump (public tool made). If 0xCC image is DC type .
The bitmask for the flags (word at 0x0A in track header)
7 - The track image data also contains a sync-offset to the beginning of the first sync marker. It is word at track image data start .
6 - If set, and track image type flag is 0 : the track is saved as an image with all gaps via the read track command. Sector contents are after track data (public t.).
The reason is that WD1772 track read command looses sync during track read, so sector data is not properly read (more later).
If set, and track image type flag is 0xCC: the track is saved as image with DC device. Sector data is usually embedded, but some may be after track image data.
In all cases, sector data offsets must be calculated from track image data beginning - more later.
If bit 6 clear, no track image data.
5 - with PASTI 0.4b always set, if Bit 0 is set (= track protected)
4 - unused
3 - unused
2 - unused
1 - unused
0 - If set the track is protected or with a non-standard sectorsize (!= 512 bytes). Well, it should be so. But in reality with public type images it is always set. So, by unprotected tracks too. Likely because there is always track image too.
Above description may little confuse. In practice, there are some typical values:
0xC1 - Usually by made with DC - track is not saved with WD1772 read track command, but with DC - but not always. See track image type flag for sure.. Typical track length is about 6KB. Sector datas are inside track data. But some may be after .
0x61 : made with public tool. There is track image, as WD1772 read track command readed it. There are all sector datas too. It means that total track data is about 11KB. But in practice, track image is needed only for couple tracks (protections) or for none. So, there is overhead, caused by reason that images are made usually by people not able to determine what tracks are protected.
0x21 : there are only sector datas. No track dump. Usually by DC type. By public not - all tracks are usually with flag 0x61 .
Empty tracks are with flag 0x21 and 0 sectors. Actually, then there is only track header present.
If Bit 0 is clear, the track is unprotected and the number of sectors as stored in the track block header are simply saved as 512 byte blocks directly after the header. The track is therefore 16 + 512 * sectors bytes long. Rare case to see, even if most of tracks on many floppies are actually unprotected.
If Bit 6 and 7 are clear, there is no track image and the sectors are stored just by order end length - just like the fuzzy sector mask, except this time for all sectors. Warning: sectors can be missing, if only a header was read, but the data was missing. The FDC status has Bit 4 (Record not found) set in this case.
If Bit 6 is set and 7 is clear, the track image is written with a track image header and the sectors follow the track image in the usually order and length. The track image header is just a word (little endian, as usual) that contains the length of the track image.
Bit Bit 6 and 7 is set, the track image is written with a track image header that has two words: the first one is a byte offset to the first 0xA1 address mark sync, the second word is the length of the track image in bytes. Usually DC type, but not always.
If Bit 7 is set, the sector data is usually merged inside the track image, the sector offsets then point inside the actual track image. However, it is possible for single sectors to have an offset _behind_ the track image, if PASTI couldn't map them correctly into the track image. A software should always read the sectors based on the sector offset in the sector header. See track image type flag ....
If Bit 0 is set, the track header is first followed with the sector headers. One for each sector in the track, again 16 bytes per header.
Sector Header (16 bytes):
0x00.l : sector offset relative to last sector header end , or to end of fuzzy mask if exists.
0x04.w : position of the sector header relative to the beginning of the track as a
time stamp. The position timing is depended on the RPM speed of the drive,
but for a ~6250 byte track the value for a full rotation is around
50000 (for 300 rpm). This allows software to simulate the time it takes
to find the sector and read it, something certain copy protection schemes
test.
0x06.w : read timing of the sector data, certain copy protections write sectors so, that
it takes longer to read them. The protection measures the time to read the
sector. 16384 is the default value, larger values mean that it takes longer
to read the sector than normal. However if the measured value by PASTI is
within 2% of this value (16384 +/- 320), the value written to the stx file
is 0 as a sign for a stable read speed.
0x08.b : Track number from the address block identifying the sector (typically 0..79)
0x09.b : Side from the address block identifying the sector (typically 0 or 1)
Note: above 2 may be false, as part of the protection (Gold of the Aztecs + others) .
0x0a.b : Sector from the address block identifying the sector (typically 1..9 or 10, 11)
0x0b.b : Size of the sector from the address block identifying the sector
(typically: 2=512 or 3=1024 bytes) - but may be more. Then only bits 0-1 matter.
0x0c.b : First byte of the CRC over the address block
0x0d.b : Second byte of the CRC over the address block
0x0e.b : FDC status register after reading the sector
0x0f.b : Sector flags, always 0x00
The CRC is a CCITT CRC16 calculated by the FDC. It is initialized with 0xFFFF and includes the sync marks, therefore typically 0xA1,0xA1,0xA1,0xFE plus the 4 bytes from the address block.
The FDC status is typically 0x00. Also common is 0x10 for a sector not found and 0x08 for a CRC error when reading. Bit 7 has a special meaning: it hints that the sector has fuzzy bits and the fuzzy sector mask should be used. If Bit 7 is set, Bit 3 is usually also set, because fuzzy bits will trigger a CRC error inside the sector.
After the sector headers the fuzzy sector mask is following, if the value is != 0 in the track header. The mask identifies the bits that are random in a given sector. The size of the fuzzy sector mask depends on the sector size (from the header) and the number of fuzzy sectors in the track itself. The fuzzy sector mask has bits set for every bit in in the sector, that is _not_ random. Therefore a byte generated with the mask could be calculated like this:
fdcSectorByte[byteOffset] = (sectorData[byteOffset] & fuzzyMask[byteOffset]) | (rand() & ~fuzzyMask[byteOffset])
Following the fuzzy sector mask is the track data area . This area can contain either the sectors by itself (when no track image), or the complete track image with the sectors embedded (DC type), or the complete track with the sectors separate (following the track image) (public type). How they are stored is determined by Bits 6 and 7 in the track flags.
If neither Bit 6 or 7 in the track flags is set, after the fuzzy sector mask, the sectors are written to the file. only the content of the sectors is written. Again with the sector size in the header taking into consideration. These tracks can be also easily recognized by sector offsets that are always a power of 2 (typically in 512 byte increments), starting at 0.
The track order:
Sometimes tracks are imaged in sidewise order: side A: track 0, 1, 2 ... 79 , then side B: track 0, 1, 2 ... 79 . Sometimes alternating: side A track 0, Side B track 0, side A track 1, side B track 1, etc. I don't see special flag for it, but it is easy to get order type by looking second track record - if there is 128 for track, then it is alternating type.
A typical protected file looks like this:
*
File Header
*
Track Header
o
Sector Header
o
Sector Header
o
Sector Header
o
…
o
Fuzzy sector mask - if exists
o
Track data - may be track image self (public) or image containing sector datas. May missing.
o
Sector data x sector count - if not in track image.
*
Track Header
o
Sector Header
o
Sector Header
o
Sector Header
o
…
o
Fuzzy sector mask - if exists
o
Track data - may be track image self (public) or image containing sector datas. May missing.
o
Sector data x sector count - if not in track image.
*
repeat ...
The track image:
It begins with track image header - 2 or 4 bytes depending on bit 7 of track flag. (see above) .
As said in beginning there is 2 type of track image. WD 1772 floppy controller, used in Atari ST serie has track read command. But it reads track content not exactly as may expect - the point is that WD 1772 does resync during whole track read, so while reading sector data too. It results in reading not exactly what is written. More precise, the bit seq. %000101001 at any place in track will init resync, what causes that following bits will be readen not correctly.
With DC made track images have no such problem, and I guess that can be used by floppy writing more-less direct.
But do we really need track image for floppy write, and how it is used by copy protections ? First to answer on second: generally, by copy protections there is 2 way of track image usage: first, more common is using track read and testing for some specific data pattern, what differs from standard floppy format. For instance Copylock from 1988 will seek 2 byte serial right before first sector address block sync mark, on track 0. Normally, there are zeros.
Second way is to use track image for data read. It allows lot of data - up to 6KB/track, but needs some special measures, to avoid false resync. By Maupiti Island all problematic sequences are prefixed with byte 7, by International Soccer Challenge with byte 0x0F (the effect is same). It avoids appearance of mentioned bit seq. so all bytes will be readen correct, after first and only sync. Loader rutine then must restoring correct data by removing prefixes and inserting correct values . Writing such track with regular Atari (or PC) is not possible . + 6KB/track is not possible anyway without special equipment.
Back to possibilities to write floppies basing on WD1772 track image read: I guess that it is possible, at least by most. But needs proper conversion, what may be not easy.
How Pasti works in practice ?
Works well with Steem emulator, and likely with Saint too. HxC support is improved . And we have thousands of images avaialable around.
But there is a lot of STX images for not copy-protected floppies. Because people can not judge what is really protected, and by some recommendations (do it with Pasti for sure) . Maybe the plan was that some team will verify Pasti images and make list which are really protected, which are OK ? Nothing from it is realised, for sure. Actually, author did nothing, really and literaly nothing in last 4 years. Or maybe 7 years, considering that public imaging tool (v0.4b) is made in 2005. Still beta ! In meantime count of images groved, and poeple want some details. I hope that this will at least partially satisfy needs.
I already did SW for converting STX to STT format (only with specific protections, but already have about 60 working STT images, usable with HxC and some even writeable onto floppies). SW for shrinking oversized STX files by dropping unnecessary track images (only for experts).
Maybe this can help to add STX support for some other emulator (Hatari) ?
Technically, it seems as great achievement. As flaw I can count lack of checksum for Pasti files. Actually, it would be best to have checksums for all track datas separately. Considering not too reliable ways how STX images are transferred to PC: with transfer floppies, in multiple chunks, or via some cable connection between ST and PC. Even hard disk transfer is not 100 % safe.
The best thing is that is possible to make images of copy protected floppies with stock Atari ST machines. And result of it is really lot of titles imaged. But looking about how author self supports it (or better said, how supports it not) I can only wish that someone other, with more sense for community and communication would do it.
Credits go to Markus Fritze, Klaus Brod and hunderts of people who made Pasti popular by imaging, sending, uploading, hosting image files.
Rough Pasti image overview - what is available around:
P - made with public tool, C - with Discovery Cartridge
P - contains WD track dump (track read command data) for all tracks
C - not always contains all DC track dumps - may have only sector datas
16 Bit Hit Machine (3 titles) , 27.1.2011 - P
1943 The Battle of Midway , 6.8.2005 - P
1st Division Manager , 15.11.2005 - C, all tracks
221B Baker Street, 30.7.2010 - P
3D Pool Microprose, 16.4.2009 - P
3D Pool Microplay , 16.4.2009 - P Seems as complete same as Microprose
version above - date, time and len is same !
3D Pool Mirror Image , 16.4.2009 - P. Funny - same date, just this is 2 sided.
Likely all 3 '3D Pool' images are from very same floppy .
3D Tennis , 17.3.2007 - P, unprot
4 Wheel Drive col. (2 flo), Date invalid, - P
5 Intelligent Strategy Games (2 flo) , 1.12.2005 - P, unprot.
above correct title is: Intelligent Strategy Games 5
520 STE Discovery Xtra (5 flo) , 5.7.2009 - P
5th Gear , 14.12.2008 - P
9 Lives , 19.6.2005 - P
9 Lives , 1.11.2005 - C, all tracks
A320 Airbus , 31.10.2006 - P, unprot
Aaargh! (2 flo), 32.1.2010 - P
ABZoo (2 flo) , invalid date - P
Action Service , 19.6.2005 - P
Action Pack (10 flo) , 8.11.2010 - P
Action ST Vol 1 (4 flo) , invalid date - P
Action Fighter , 30.11.2005 -C, no tracks
Addams Family , 13.8.2005 - P
Addictaball , 7.3.2010 - P
Bards Tale, The (2 flo) , 29.10.2004 & 3.12.2005 - C
James Pond , 13.6.2006 - P
Kristal, The (4 flo) , 25.6.2005 - P
Masterblazer , 20.12.2009 - P
Mickey Mouse (2 flo) , 19.6.2006 - C, no tr.
Spherical (2 flo) , 17.3.2010 - P
Trinity , 2.8.2008 - P
Wild Streets , 16.12.2004 - C, no tr.
Zombi (2 flo) , invalid date - P
Notes: started by Alphabet, then choosing randomly few from STX collection with over 1500 images - taken from GameBase ST, diverse WEB sites (currently Atarimania is most active and has most) . Obviously aprox. 90 % is made with public tool .
Why much more public type STX images ? For sure because most of people have no Discovery Cartridge :-) . But this initiates some thoughts about whole 'Pasti preservation project' :
I'm pretty sure that at start ideas, plans were much bigger than it outcame. Of course it is not the end, we may see new and new images yet for years. But I doubt that number of DC type Pasti images will be very big ever. And likely, only that type has real perspective to be writeable (because of track image type) onto floppies or working with HxC . So, the preservation is done (~70% of all Atari SW), but is it real preservation ? I mean beeing limited on SW emulators only. I leave it on reader to judge. What I can say is: 7-8 years after starting creation of Pasti format and SW we still missing informations, relevant and simple infos. Jorge Zwick is persona not talking much , at least not talking about something he should, and he was asked many times by many people. Pasti WEBpage contains very little information, not updated for years. SW is in beta stage for 8 years ... Worst is that beside lack of infos we have misleading informations: about publishing specs, sources, plans about floppy write support etc. Excuse of beeing busy just works not for me. Not giving any specs in 8 years ? Come on ... There must be some other reason, and I guess that it is related with lacking or 'fuzzy' replies to people in forums :-)
For the end, very important issue: reliability of STX images
Are all Pasti images 100% OK ? What are chances that there are errors in STX image ?
Usually people test STX images before make them public available. Test in Steem, of course. But how test ? I was not present at any of imaging made by someone other, but can say with confidence that at least 99% tests is made by checking only is game starts, until begin level, or couple levels more.
And here is possible PROBLEM : I start with recent case: I imaged game Voyager with public Pasti. Floppy is 22 years old. Only with fourth floppy drive I got working STX image. Running game directly from floppy confirmed that problem is not Pasti - with first drive game worked not on real Atari from original floppy. I examined images, and found that error was exactly in copy protection. It is Copylock, and sector 6 on track 0 was readen incorrect in first 3 imagings. It is slower reading sector with fake CRC error. But must be readen 100 % correct - if not, game will not start. Pasti did not notice error - normal, as there is CRC error reported by FDC. It just confirms that CRC error is put there with purpose, to confuse copy SW. This was straight case, and imaging error was obvious after short testing.
But we have a lot of copy protections with delayed effect. or performing check(s) in later game stage(s). It was btw., recommendetation of Rob Northen self too. So, it may happen that protection check fails, but game still will work well until some later stage. Or floppy read for protection check will happen on stage 11, for instance.
So, the only reasonable answer is: we can not be in all cases sure that some Pasti image is 100% OK, because there are situations when image may contain error(s), mostly caused by age of floppy disks imaged, + by floppy drive, head align and similar. Pasti HW and SW is not enough sofisticated to detect all possible errors during imaging.
So, playing games from STX image(s) until end is required test - about image correctness.
It still not proves that every bit is readen correct, but is good check for correct imaging of crucial part: copy protection.
Generally, copy protections are what makes floppy imaging very hard and troublesome. And obviously, the place where most likely it may fall. My opinion is that true preservation should include instead images of copy ptotected public releases : the sources of original, and executable SW in not copy protected form. But as things are in real World - it will be not much of such ever. Many source is lost forever, authors, copyright holders care not for preservation. They have no any copies, so for some SW only source is pirate crack. It is big shame for all involved, and maybe just indicates that all what interesting them is money.
And that above is not only some dark scenery, theory: many of Atari ST floppies with game originals after 1989, 1990 have some copy protection with delayed effect. It may be over 200 titles. So, it is almost sure that some STX images are bad, considering that imaging was made when disks were aprox. 18 years old in average.
Update: Expectable problems with Pasti images and what we have online
As said, Pasti imaging is not flawless. Since author does nothing to better inform people about it, not giving any clues about advanced usage, I feel need to write here my experiences and recommendations to make images with less possible errors.
If reading is unreliable some sectors may be readen bad. Then in normal case SW reports CRC errors. Pasti usually warns not, just writes readen, with CRC error flag on - assuming that it may be copy protection. The real bad thing is that quick checking of image will not show any errors, because need to play more-less to reach problematic part. It was case for instance with image of Kid Gloves posted at AF.
Average user can hardly determine such problems - but examining STX file may give some clue - if there are CRC errors in middle of disk, over more sectors, it is likely not copy protection.
What average user can, and actually need to do is: taking care about cleanliness of floppy drive. Floppies self is not recommended to clean, but methode of multiple reading will for sure make it cleaner.
There are cases when imaging just don't work, and program stops. I will not go much in that here - reasons may be some not supported protection, or just very bad condition of disk. If cleaning helps not, it is likely too damaged, and impossible to get correct image. Still, it may be worth to keep such floppies. Some parts must be still OK - may help in building image from multiple sources. This initiates thoughts about how to coordinate hunt for missing rare titles ..
Another case, where image can fail is when copy protection is not enough tolerant, and happens with some floppy drives used by imaging. It is certainly rare case, but again, quick test of image may be noot enough. So, it may happen that drive gives index pulse little earlier or later than coders expected, and then copy protection check will detect it as pirate copy. I saw such in case of games To Be On Top and Rubicon.
Btw. not enough tolerant code is not so rare case, and not only at copy protections. For instance famous FastCopy Pro 3 had problems with formatting floppies on several floppy drives - for me it never worked. It was corrected in later revisions.
Fortunately, such 'bad' STX images (with little shifted track image) can be used to get 'perfect' copy - by running in Steem Debugger can see fine how copy protection check fails, and correcting STX image by shifting little problematic track image.
Delayed protection problem is likely not only Pasti problem, but general problem with floppy images.
What is good in all this: in fact, testing of STX images is done in many cases. Not in some organised form, but just by playing games until end in emulators. Of course, mostly quality games were played. People could report problems in diverse forums. Normally and especially with Atari ST, you can not be always sure that the problem is bad STX (image), but after some time things getting clarified and some can finish game, and say: it is good :-)
Simple SW for giving infos about STX image:
DL PastiInf - usage help included.
P. Putnik , Latest update: Jan. 2013.