Limitations of Atari ST TOS in working with hard disks are pretty well known: despite it is intended to be compatible with widely used DOS FAT16 disks, partitions - so with all swapped bytes in FAT, MBR , it is compatible only in case of partitions under 32 MB size. There is partition size limit of 512 MB in case of TOS 1.04 and higher. TOS uses so called large sectors in case of partitions over 32 MB size, and it is not efficient way. It means for instance that in case of some partition in range of 257-511 MB, when need to read only 512 bytes from drive, it must read 8 KB, because it is logical sector size. And today, with modern storage, most of people create larger partitions.
I was thinking about improving this already some 10 years ago. I seen approx. what is the problem in TOS 1.04 with FAT16 filesystem. Main problem is that sector addressing is with 16-bit variable instead 32-bit one. Other problem is using signed variable for cluster number, so max cluster count is not 2 pow 16, but just 2 pow 15 - what means max some 32500 clusters - and worse, it means smaller max partition size.
Finally, I did this patching - it was extremely hard. Code is done with some ancient C-compiler in 1989, and that was pretty hard to follow. Certainly, things would be easier with complete C source of TOS 1.04, but there is no such. There are some sources for TOS filesystem, but here, we need all variables of TOS 1.04, otherwise whole RAM usage will be messed up. So, I decided to perform all changes in disassembled TOS 1.04 S file, keeping all variables intact. There was place for expanding 16-bit variable to 32-bit one in BCB and OFD structures. Most of it happens on stack, and a6 link space anyway in C code. I done changes at about 240 places, and resulting code would be longer, so would not fit before AES, but luckily there was lot of addressing what can be converted to short (TOS variables are mostly in lowest 32KB), so I shortened it some 4 KBytes just by it, and that was only for part of TOS. Harder part was making real 16-bit FAT to work, but it is OK now, and with this:

Max partition size is now 1GB. It is better than Falcon's 1 GB, since clusters are shorter (16 KB vs. 32 KB)
It is DOS FAT 16 compatible, so no need for BigDOS or DOS/TOS compatible partitioning - you may access DOS partitions (created by Windows or some PC partitioning SW) up to 1GB. Of course, this can work in same mode as original TOS 1.04 filesystem, so access to TOS(AHDI) type partitions works still - and it can be better with proper driver - then will access them in short sector mode.
No need for extra buffers, because no large sectors. That means that RAM usage of hard disk driver may be only some 2-6 KB totally.
Of course, some extra buffers can make it faster, but now, it will be optional and not mandatory.
Disk access is more efficient because no large sectors. Max partition count remains same: 14 .

Updates in March 2018:
I did same with TOS 1.62, which is pretty much same as 1.04 (diff. is in handling extra STE HW) .
I plan to make 4 versions with this updated FAT16 filesystem:

1.05 - updated TOS 1.04, for ST, Mega ST machines. ROM start address is $FC0000 .
More place for diverse TOS mods - some 5KB free.

1.63 - updated TOS 1.62, for STE, Mega STE (added extra code for MSTE) .

1.56 - combination of updated 1.04 and TOS 2.06 Desktop, AES .  For STE, Mega STE . May work on ST too (recommended), but needs little extra logic because ROM start must be at $E00000 .

1.66 - combination of updated 1.62 and TOS 2.06 Desktop, AES .  For STE, Mega STE .

Video of booting with TOS 1.66 :


Note that UltraSatan displays not properly capacity of 16 GB card. I created only 3 partitions in Win 7, because from some reason it could not create extended partition. But that's not relevant, there will be partitioner SW what will do it well on Atari.
The benefits: more efficient and faster work, because smaller logical sector size. Larger partitition sizes are possible. Less RAM usage, because no need for extra buffers. RAM usage of driver can be very low - only 2KB !
This needed really thorough testings, so I did not rush with it. Now seems pretty much reliable and stable.



View of 1GB partition content:

     
And here is first extra problem - above pic is done with TOS 2.06 combined with this TOS 1.05 filesystem. In TOS 1.04 displaying of larger file sizes in Text view mode works not - it crashes when is above 100 MB. So, it will need some patch of Desktop part of 1.04 too ...
Luckily, no problems in Icon view mode.

And there is it in patched 1.04, with now added large file displaying fix:


Some people certainly noticed those crashes when used TOS 1.xx in emulators, with GEMDOS hard disk emulation (so some DIR on PC assigned for Atari logical drive, like C: ) - it crashes in Text view mode if there are some very large files (above 100MB) . So, this fix is good for it too.

I made simple test program which measures disk access data flow. Tested with pretty much average SW - starting some medium long program what loads couple shorter and longer files:  with usual large logical sectors of AHDI/TOS partition it was 294912 bytes loaded from disk. Running same SW, from same place, but with 512 byte sectors resulted in 225792 bytes loaded from disk. Or 30% less . There is just too much overhead with large sectors, especially in case of many shorter files, or many short data chunk accessing.

Future plans: Making special hard disk driver, what will utilize this real FAT16 capability of modded TOS. With some extra buffer settings as option. RAM usage with regular buffers (which are always there in TOS) can be some 2 KB only. Extra buffers can improve speed, so it's on user when will activate them.
I may selling programmed chips with this modded TOS. There may be some other patches too - like IDE autoboot support, special TOS AUTO folder run patch for max RAM for user SW ...



Virtual floppy feature

Adding currently support for it in TOS ROM - what is simply best place for that. See demo video:



Unlike in Floppy Image Runner, images are stored on hard disk/Flash card, so RAM usage is minimal, actually it is 0 practically - all code is in ROM. Can change images 'on fly'. But better is to make one larger image with all needed files. For instance something on 5 floppies, all in 1 image of 4-5 MB. Some may say that you can copy all it on hard disk, and solved. Well, not always - there is plenty of SW made to run only from A:

It will be best usable for people with bad or missing floppy drives - and today, floppy drives and disks are really problem. It is possible to disable or allow hard disk driver boot, so can do some file transfers.
Plan is to add 'default image' feature - at boot may select it, then image will boot as A: - good for usual ACCessories and like.

Technical background of Virtual floppy :
Very basic is same as in Floppy Image Runner. There are 3 ways of accessing some data/file on floppy in case of Atari ST, TOS :
1: Via filesystem, so accessing file by it's name, path .
2: Via file position on disk - then can use XBIOS 8/9 calls  or RWABS . First is used in number of games (Sundog), second in few.
3: Custom floppy access code in SW (games).
Most important is that first 2 ways can be redirected to hard disk/Flash card, while third not. So, Virtual floppy redirects all calls via TOS floppy access to hard disk.
About 60% of Atari ST games using TOS calls for disk access.  And about 100% of user disks :-)  So, this will work with lot of it. Of course, copy protections will not pass, except very rare cases.

When installing active images for usage, program for that will set some special variables, and it will activate with warm reset. Will boot much faster, because no floppy drive test, and of course overall speed is faster.  Another warm reset will boot again with active Virtual floppy - then what is set at that moment for A will be boot disk. AUTO run possible.   To go back in normal mode just perform cold reset or switch off computer for short.
Since code is in ROM, it most be for user's mass storage adapter. 3 types will be available:  basic ACSI - up to 1GB accessible (Mega STE internal adapter for instance),  ICD extended ACSI (UltraSatan), and IDE .
I call that driver in ROM  Raw Image Access (RIA) . That can be used for gaming from hard disk by machines with only 512KB RAM - for games which work not via Virtual Floppy - hard disk adaptations with low RAM requirement - needs some changes in disk access code.



Did some new icon designs for Desktop:








I think that they are much better indicative than what I seen in different TOS updates.
Why Flash card as drive icon ? Because it is year 2018 :-) And with virtual floppy feature, that is what will be real storage media. Not floppy, not even classic hard disks are now used as much as Flash cards.


Latest progress: this week, at May 20, I dealt with something what was not planned. While worked on new icons for Desktop, I noticed something interesting: TOS copies 2 RSCs and DESKTOP.INF template straight from ROM to RAM, and it means that it is present in 2 copies when Desktop starts. Size is about 16 KB. Yes, some parts must be in RAM, since there are changeable fields, values, flags, coordinates .... But lot of it is static - images, icons in first place, then lot of txt. So, came idea to move in RAM only changeable parts of it, and keep static ones in ROM, and point there from part what is in RAM. That needed serious rework of RSC, disassembling AES parts of TOS 1.04 and 1.62 etc. Took some 5 days, and because I used packing for part going in RAM, the result is:  6112 bytes more free RAM, and 6.3 KB more free space in TOS ROM (only with this 1 mod).
Some could say that not much, and that's true, but this is area where can not gain much bytes. However, they worth gold.
Extra free space in TOS 1.04 - and now I have some 13 KB instead original 1 KB can be used for diverse new things - like Virtual Floppy, better hard disk support, etc. And even can make TOS 1.04 what can work with 68020-30 CPUs, so must not use TOS 2.06 . Latest needs actually some + 9 KB space.
To add, that first STs (1985), when loaded TOS and DR Basic from floppy had only 5 KB free RAM left. And they had 512 KB RAM. So, this 6 KB more free RAM is not that bad at all :-)

With org 1.04:


With mod:





    PP,  Jan.- May 2018.