Data storage

Here I will talk about mass storage on Atari ST computer family (so STE, TT and Falcon too) . It means hard disks in first place. CD ROMs were used much less with Ataris, and they are now pretty much obsolete. And instead hard disks now (2021) we have cheap and high capacity Flash cards - lot of advantages - low power consumption, small size, and ideal for data exchange with some modern compter via USB reader..

_{Basic facts}: work with hard disks is solved in 2 layers (that stays for most of OS by diverse computers) :
Filesystem - deals with files, directories .
Hard disk driver - it is what accessing HW (hard disk/Flash card adapter) - and it gets requests from OS - like autoboot, sector transfers .
This concept allows work with wide range of adapters, and only driver part is what needs to be written for new type of adapter.
In TOS 1.xx there is only ACSI autoboot support. At 2.06 IDE autoboot is added , as IDE drives started to take over market - lower prices for drives and adapters too.
Falcon and TT have SCSI ports. Earlier TT has ACSI port too, later Falcon has instead it IDE port and place for internal 2.5 inch drive (was shipped optionally with 65, 85 MB drives).

_{The limitations}: most of limitations is in filesystem. Which is FAT16, and partially DOS FAT16 compatible. And because that it needs byte-swapped (low Endian) parameters in boot sectors, FAT (file allocation table). Sadly, compatibility goes only for partitions up to 32 MB size, above it it is not compatible, and the reason is 16-bit sector addressing used in TOS filesystem (16 bit addressing means exactly 32 MB with regular sector size of 512 bytes). So, above 32 MB TOS uses so called large logical sectors like 1 KB, 2 KB ... 8 KB (TOS 1.00-3.06, 4.02) or even 16 KB (TOS 4.04) . And that's not good. All what I saw in last 10 years indicate that the culprit is DRI's Alcyon C compiler - was not good with unsigned integers.

Partition size and count limitations:
TOS 1.00-1.02 : max size is 256 MB. Max 14 partitions, C-P .
TOS 1.04-3.06, 4.02: max size is 512 MB. Max 14 partitions, C-P .
TOS 4.04: max size is 1024 MB. Max 14 partitions, C-P .
Max count stays for all partitions on all attached storage - so if there are for instance 11 partitions on first one, only 3 from second can be handled.

_{Adapter types}: first one, with built in connector, DMA chip is ACSI . It is actually special revision of old SCSI specification (called SASI). Most known limit of it is 1 GB max accessible. That was for sure not some flaw in 1985 and usual sizes of hard disks (like 20 MB). Bigger problem was that there were no hard disks with ACSI support, so conversion was necessary - Atari self made Megafile, what has RLL drive inside. Independent manufacturers went on SCSI way in most cases. And most succesful was ICD. And they made great thing: overriding 1GB limit with their later controllers, like ICD Pro models, + speeds were really good for those years.
And here we are at something what needs to be mentioned, especially because is not unique: wrong data (claim) in DOCs . That's especially bad when it is Atari's official DOC. And they talk there about max 1 MB/sec transfer speed possible with ACSI port. Or Atari ProfiBuch says 10 Mbit/sec (about 1.25 MB/sec). However, there were diverse speed tests with ICD Pro adapters and results around 1.6 MB/sec !? That made me to build special test circuit for measuring real top speed of ACSI port (what is actually DMA chip speed test). And yes, it is more than DOCs say: 2 MBytes/sec in peak ! And that stays for all DMA chips in diverse ST(E) machines, including TT. That made me to design special ACSI adapter for work with CF cards. It works in DMA mode of CF cards, and since ACSI port is 8-bit needs 8-bit DMA mode. And only Sandisk cards support it. Max real speed is about 1.8 Mbytes/sec - surely best among ACSI port adapters. ACSI-CF adapter

IDE adapters: they became popular around 1990, and are for sure cheapest, simplest mass storage adapters for old computers. Plus IDE hard disks were what took over market, because low prices and good performances, added new features like LBA, DMA ....
No wonder that Falcon was equipped with IDE adapter too (1992). And at TOS 2.06 there is support for it in TOS.
There are unfortunately some timing problems, especially with newer CF cards: IDE timing problems
Experience says: Sandisk is most reliable choice for CF card with Atari ST adapters. Just be aware of lot of fake cards on market (especially online).

At about 2004 Flash cards arrived on market. Compact Flash (CF) was one of firsts - it is IDE compatible.
But most popular became SD cards - cheaper, smaller than CF cards, no fragile connector. And now we have diverse SD card adapters for Atari ST family.
First was Satandisk - not good speed - only some 150 KB/sec. Then UltraSatan - much faster, around 950 KB/sec (depends from SD card too). 2 slots, RTC built in. Indeed it is most popular one in last 7 years.
And new ones appear, some good for DIY . They support ICD protocol, so more than 1 GB accessible.
Special case is built in adapter in Mega STE - it is Basic ACSI to SCSI adapter, so max 1 GB accessible. Speed max about 1 MB/sec.
I made simple mod for it, to override 1 GB limit: MSTE internal adapter mod
It needs special driver, since is not ICD compatible (that would need moch more modifications and adding new chips).

_Compatibility: One of main arguments against non-ACSI hard disk interfaces on Ataris is that they are "not compatible" . This is pretty pathetic, together with claims that only good things are what Atari made, recommended. We know that best hard disk IFs are made by 3-rd party as ICD for instance. And they used some new solutions, not following Atari. Actually, biggest problem with Atari ST machines is lack of expansion bus, connector. It means that adding some IDE IF requires opening machine and soldering a lot. Except if you have Mega ST or Paskud. It is the reason why diverse ACSI-SCSI and now Satandisks are so popular - just connect adapter and use it... Just to mention that there was even IDE CD ROM IF for cartridge port available from ASH.
In reality all possible incompatibility is SW nature: starting with inability to boot from hard disk. What is case if you have TOS older than 2.06 and IDE hard disk. But people made so much diverse TOS patches, so adding IDE boot to some older TOS is made too. Paskud is special case: it boots with SW in cartridge EPROMs, as that IF is on cartridge for ST cartridge port.

Everything considering disk access after booting from ACSI port (except FAT16 filesys. handling) is not handled with TOS in fact.

Hard disk driver is what does disk access (as is told above), but actually it needs to do some other things - like RAM allocation for buffers, dealing with CPU caches (MSTE, TT, Falcon), partition detection, mounting, etc ...
And we are now at:
Compatibility with modern computers, OS:
As is said above, TOS FAT16 is compatible with widely used DOS FAT16 only up to 32 MB size partitions. But there is a way to make TOS/DOS compatible partitions - that needs special sized reserved sectors, FAT, etc. Only way to create them is special partitioner program for that, and it exists only for Atari - so don't think about doing it on PC (well, except with emulator) . And of course, special driver for it is needed. Then data transfer with some modern computer is really peace of cake. But as usual, there might be some problems - in this case it is Windows long file names (LFN) - such files will confuse TOS, which is not ready for something created around 1995, so must avoid them. I use Total Commander in Windows, where can set it to not use LFN . Unfortunately newer Win versions tend to write automatically some system DIRs and files on every attached Flash card and partitions on it.
Even if delete it, will write again in short time. So, since 2014 I added LFN filtering in my Atari hard disk driver SW - that's best defence.
And it is included in improved TOS versions, of course.

_SCSI vs IDE : SCSI was for sure more advanced than IDE. Until about 1992 .
In case of Atari ST machines simple IDE IF attached directly to bus is good enough because we can not achieve some bigger speeds due to machine's slow RAM. Only drawback is timing problem with some CF cards.
Usual complain against IDE is: it occupies CPU 100% during disk access, so all SW is stopped. That's only partially true. ACSI and it's DMA will in fact stop all other SW activity too during disk access - because then all what driver does after giving transfer start command is waiting. So, SW must wait until disk access is completely finished (what is usually end of file transfer) . And since IDE is usually faster - it is what will slow down SW less in fact.
Today, the real drawback of IDE is higher price of CF cards than SD cards, hard to find them, and mentioned timing problems. But if speed is priority, it is it .

_{Diverse misinformations, errors in DOCs}:
I wrote this based on own experience (what started in 1991, so it is 30 years now) - includes designing and making IDE adapter in 1992, diverse research, experiments, work on driver SW since 2006, more HW designs, etc . So, not simple copying from some other sources.
And yes, there are errors present in even established sources. Well, everyone makes mistakes, that's part of doing something. The real error is to blindly believe everything written, just because it is from some big name or whatever. The real error is to not correct mistakes when you are aware of them - and it happens that some just can not accept that someone independent - in past hobbist, now pro in it can know some things better. Well, I found even some HW design errors in STE (bad audio mixer ratio), made simple solution against it, and still, some people came with ridiculous explanation why it was so done by Atari - total nonsense btw.
So, let see some related ones:

As mentioned above ACSI/DMA peek speed is actually 2 MBytes/sec, and not 1 or 1.25 as some established sources claim (and were copied zillion times).

In reality ACSI/DMA does not let SW to work at full speed when there is disk access - then it must wait transfer finish, and CPU actually does waiting during it. Plus, there is some slowdown too, depending on transfer speed - when DMA accessing RAM, CPU must wait.
My speed test program can indicate that slowdown (and as I know only one among Atari speed test SW). Speed test PRG

Some sources claim that first partition (C:) can be max 16 or 32 MB, and that it is TOS limit. Completely wrong. If such limit exist, it is driver SW limit. For TOS it is same for all partitions.

And some claim that RAM usage of hard disk driver and TOS filesystem handle is bigger when there are more partitions - saw it in forums (what to expect from forums lead by arrogants ?) , and on some Atari ST WEB pages. And again, completely wrong. Actually RAM usage is set by hard disk driver, and consists from 2 parts: driver code self (it's resident part, it may use more during detections, partition mounts, but that's freed by well done drivers. And buffers - TOS self allocates by start 2 KB for disk buffers - that's good for floppies and for hard disk partitions of max 32 MB size - and it is actually 4x standard disk sector size (512 bytes) . Above it TOS/AHDI needs bigger buffers, so min 32 KB is needed for partitions between 256 and 512 MB (TOS 1.04 and above) . It can be more, but with today disk/Flash card/adapter speeds no real benefit. And again, count of partitions affects not RAM usage of driver SW and TOS disk handling.

Disk space usage efficiency :
Used space on disks/Flash cards is always bigger than sum of file lenghts. The reason is that there is minimal space what one file can use on disk - and it is called cluster. FAT is where is data (records) for all clusters in 1 partition. If it is 0 it means that it is free. If is $FFFF it means last cluster of file. Some other value there means number of next cluster of file - that is normally next cluster, but if it is not free when file is written to disk next cluster of file will be first free one in partition. This allows that file can be written to partition even when no so long free segment as file length . And then file is fragmented, what means slower work. But let's stay at space usage: so, if 1 cluster is for instance 16 KB, and that's case with partitions in range 257-512 MB (TOS 1.04 and higher) it means that some very short file will take 16 KB on disk. Or if file is 16 KB +1 byte, it will take 2 clusters, so 32 KB.
So, space usage efficiency is worse with many short files and longer clusters.
FAT16 actually means 16 bit cluster records, so should have up to some 65530 clusters. But it is not case with TOS, it can have max half of that, so 32765 records - and that's not good for space usage eff. It is even worse with TOS 1.00 and 1.02 - they can have max 16382 clusters - and that's the reason for smaller max partition sizes too.
What can user do if space is tight, and there are many short files to save/copy ? Using more smaller partitions instead one big.
Well, with today large capacity and cheap Flash cards, this is not so big problem. More chances are that problem will be how to utilize some larger capacity card. Because 14x512 MB is 7 GB - and that means that no sense to get some larger than 8 GB capacity card for Atari ST(E) with regular TOS.
But soon may happen that will be no able to find such cards in shops. Solution can be my improved TOS iTOS
on what I worked in last 3 years. It has fully DOS compatible FAT16 filesystem, while is compatible with TOS type partitions too.
Limits are: max 30 partitions of max 1 GB size - ergo can utilize 32 GB Flash card well (and they are never full 32 GB capacity, usually about 10% less - stays for other size cards too). Disk space usage is more efficient, because max cluster count is 65530. And all it with even less RAM allocated by driver.

P. Putnik, latest update: April 2023.