As we know, Atari ST design started somewhere in 1984, and main HW designer was Shiraz Shivji, man who designed most sold home computer C-64. In both cases, boss was same man - Jack Tramiel.  But one important thing, beside time period change (progress of time, to say so) whas change of company too. As I know, Jack Tramiel, founder of Commodore was pushed out from company by share holders. Not really interested in details, what matters here is that he continued his computer manufacturing activity in Atari, firm what was practically abandoned by Warner, and which could not survive anymore with making only game consoles and arcades, and 8-bit, gaming aimed home computer.
Ambitions were very high: making competition to IBM PC, Apple. Making universal 16-bit (so, second generation) home/business computer, and selling a lot of it. In that time GUI OS jumped in all it - graphical user interface, with mouse control, icons, menus, etc. So, normally that was targetted. And one other thing:  Motorola 68000 CPU. It was on market some years already, but was pretty much expensive, so used mostly by military, workstations. But prices went down, and Atari choose, as many others to go on with it.
HW design was easier and faster than making OS for it - as it is usual. Basically, only HW thing what came later for ST was blitter chip - it's complex as some CPU, so arrived with Mega ST in 1987.
With OS, there were for sure many dilemmas. Tramiel wanted GUI, Basic, and all it working with not too much RAM, ROM.
They engaged firm Digital Research (DR) for that. As established one in OS creation. However, that firm had 0 practice with GUI OS, was not really for Basic, and I'm sure they had 0 practice with Motorola 68000 .
First ST was planned with 256KB RAM and 128KB ROM - containing Bios, DOS and GEM, Desktop. That sounded in that time as pretty much, considering what 8-bit machines had. Now is clear that it was serious misjudgement. It's not only that GUI needs much more space, there was disk based OS, with hard disk support too, so FAT12, FAT16 filesystem, RS 232 serial communication, etc.
Code with 68000 CPU is usually longer than code with some 8-bit CPU. And when you do most of it with C-compiler of that time (inefficient), you see the following: TOS can not fit even in 192 KB space, and talking about v. without Basic.
I think that it was critical time, and they made some bad decisions then. Like choice of TOS ROM location in address space.
That resulted in need for changing it when STE was designed.
Machine launch was planned for early period of 1985, and TOS was far from being finished. That resulted in shipping first machines without complete TOS in ROM. ROM was only 16 KB, containing only very basic BIOS code needed to wake up HW, and to load OS from floppy.  And if we look it better: TOS.IMG size is 206.554 bytes. Much more than 192KB.

No, it's not GEM, Desktop, it is just picture shown while loading TOS from floppy


TOS from floppy under work. But not much free RAM is left.

It is clear, that they needed some extra work beside normal difficulties when making new OS on new CPU (for them), HW, in limited time - to make all it fit in 192KB.  Idea of expanding ROM space was probably rejected because costs, they went rather on expaning RAM, what was good - RAM is flexible.  Still, even 512KB, what was a lot in that time was tight for TOS and Basic in RAM. I remember some very bad reviews in magazines - There was some 5KB RAM only for user's Basic PRG free. That's really disappointing when there is 100x more RAM. But Basic was really not target then. DR used C, and wanted that users use it too. No wonder that DR Basic was crap, and some independent firms were who made good Basic versions for ST.
So, I guess that one of hardest times for TOS programmers was period when they needed to make ROM version of full TOS.
Squize all it in 192 KB, not dropping any function out, with inefficient compiler, not well known CPU. Actually, they needed to add hard disk support too, since I don't see that it was present in disk TOS 1.00 - at least it does not boot hard disk driver.
The problem was solved until end of year 1985, but some tricky way, called Line-F was used. Basically, it is shortest possible subrutine call, for most called functions. 68000 generates exception when opcode starts with $F. That exception jumps to executing code, remaining 12 bits are function code + possible some parameter.  As I calculated, it self saved some 6-7 KB space in TOS ROM, at price of little slower work.  Just to add here, that it was used up to TOS 1.04. TOS 1.06, what is practically same, from same time, just for STE, and has 256KB ROM space uses normal subrutine calls instead Line-F.

So, until end of 1985 Atari ST was completed, as was imagined. And yes, it sold well, especially in EU, Not that good in USA. The reasons for it deserve separated large space to elaborate. I will not go in it here.

It is normal, that TOS needed updates by time. Design of it & CPU were ready for. But not all SW done. And that leads us to painful part, at least I think that it is it.  Poor documentation of TOS. Really hard to understand why it was neglected to make some really detailed doc(s) for those who would help selling of machine by making quality SW in short time.
Because docs were not detailed, and I dare to say that there were some really not serious comments of authors self in.
I can give here couple examples from own experience:
Reading input devices by TOS functions:  reading of keyboard was relative good, but mouse and especially joystick was solved in by me overcomplicated way, what was on top of it not documented enough detailed. The result: lot of games with some hacks to read input, and working only on early TOS versions. Examples: Tracker - only TOS 1.00 (mouse controlled). Arkanoid - only TOS 1.00 - joystick read fails . And there are hundreds of similar. What hurts most is probably STOS, done by good programmer, but even he was not aware how to do it right - and he is not to blame for that, but Atari.
STOS using some TOS address tables for input read. That fails with later TOS versions. Then STOS fixers were made, which basically add tables for newer TOS versions (and remove some, because space is limited). I did STOS fixer what puts there proper code, so works on all TOS versions without tables. And all it fits in space - here talking of course about executables, done with STOS compiler. But it was thanks to some people who discovered and published right way how to do it (instead Atari DOC creators). Hmmm... here I can not resist to mention Pasti  similarity - all Pasti documentation is done not by person who should do it, but some others, who spent a lot of time discovering all it.
Another example of poor doc. is ACSI port programming code example, published in Hitchiker Guide to TOS, somewhere in 1986 first, I guess. That was present in Atari ST ProfiBuch, 1987 edition too (what was my main Atari literature) . And that code has error - what causes bad work, when more than 1 device is attached to ACSI port. In later editions of ProfiBuch, ACSI code example is not present.

And one special example, just recalled it (June 3 2018) : in Atari docs stay that max transfer speed of ACSI port, so DMA chip is 1 MB/sec, in some stays 10 Mbit/sec. what would be 1.25 MB/sec. Well, just another shallowness. I did lot of tests, starting from early 520 STs, up to TT (latest model with DMA chip). In all them, with any DMA chip version, max speed is 2 MB/sec . Note, I talk max speed - and speed depends from attached adapter, interface. So, speed with fastest interfaces.

I know that making DOCs for SW is not fun part, myself don't like that step. But it is something elementary for serious SW. Even for games in many cases. Too bad I was not good with Jack Tramiel in that time to warn him :-)

Back to HW:  in 1987 Mega ST is launched. With blitter chip, some expansion slot, profi case, separated keyboard. No TV out. So, clearly business user orientation. New TOS v. 1.02.  What was not really much better than 1.00 , must say - TOS needed more RAM for instance. Blitter is nice thing, but SW support failed, even much later. I really don't know what docs Atari published for it's programming. I used some other sources for sure.

Much better was with TOS 1.04, what was in machines at 1989, same time when Atari STE with TOS 1.06 was launched. Basically 2 TOS-es are same. 1.06 has support for STE extra HW, and no Line-F, as explained above.
It was really usable with hard disks, unlike 1.00-1.02 . GEM look is better, etc. And TOS 1.04 even needs less RAM than 1.02.
So, they optimised it better, for sure. Still, used compiler was not that good.


TOS 1.04, aka Rainbow TOS

Following is hard disk related, and little more technical:
In that time, capacities of hard disks were such, that 32 MB partitions were not sufficient anymore. Therefore so called BigDOS partitions were introduced. FAT16 can address 2 POW 16 clusters, so 65536 . What is 32MB if cluster size is 1 sector. But it is usually 2 sectors (1KB). And then there is more than 2 POW 16 sector in partition, what needs 32-bit sector addressing instead 16-bit one. And here is for me probably biggest mistery of all bad decisions/solutions by Atari ST:  why they kept 16-bit sector (recno) addressing even in 1989 ? In some TOS sources there stays that recno variable should be 32-bit. But it remained 16-bit even in TT. On addition of it cluster # variable is signed, what limits cluster count to some 32000 .
Then how they solved that partitions can be up to 512MB ?  By using so called large logical sectors. Cluster size remained 2 sectors, but since logical sectors can be up to 8KB (16x normal sector of 512 bytes) it gives 2x8x32576 = 512MB max partition size.
What was fine for that time, and actually is not bad even today for Atari ST SW needs. But that way is just not efficient, and needs extra disk buffers, what on top of it must be created by hard disk driver SW, since TOS does nothing about ...
Inefficency is that when you need to access only some 100 bytes on disk, you still need to load whole 8KB, because protocol, better said 16-bit recno makes not possible less. I needed to design video playback SW to take care to load everything in 16KB parts, otherwise loading would not be enough fast.
Well, this flaw is now fixed by me, just for case you asking why I'm so obsessed with it :-)

-------------
Adding new 'discovery' at May 28 2018:  While I dealt with replacing Desktop icons of TOS 1.04, found something interesting.
It just copies directly from ROM to RAM some 16 KB, where are 2 RSCs - for AES and Desktop + DESKTOP.INF template.
Indeed, it must be in RAM (at least bigger part of it), but why they did not use packing then ? Even with some primitive packer could save 8 KB ROM space, and then would be no need for line-F. Really sad to see that it was not done better in 1989. Actually, it is even worse with 2.06 - there it is about 32 KB, although, ROM space was not problem then (enlarged to 256KB).
So yes, TOS 1.04 is nice toy for diverse improvements :-)
----------------

STE: compatibility on HW level is great. Basically flawless. On SW level it is not so great, mostly TOS is what makes problems.
Actually poor TOS docs, what resulted in SW being not coded properly. And some bugs in some SW what fails because silly errors in code (Grand Monster Slam), not making problem on ST.

Unfortunately, golden era of Atari ST was in decline in 1989. Amiga attracted more buyers. SW houses started to abandon ST too. And doing SW for STE, or at least SW what used STE enhanced capabilites was not spread.  We can say that there were made about some 100 games in that time (until 1993), which used them. I think that in later years, until now, count is bigger. But that did not help Atari in those years, and it just stopped to exist.

Unfortunately, there were some mistakes in HW design of STE too. Worst by me is sound mixer - despite there is a divider in factor of 6 for Yamaha (PSG) sound, it is not usable, so you can not make balanced audio of simultaneous PSG and DMA audio.
That bug was corrected only in later Falcon.

And best things were done when was practically too late:  Atari Mega STE and TOS 2.06 (first was 2.05, then bugfix 2.06) .
Mega STE is in same case as TT (here I will not go in TT and Falcon story) - has built in hard disk adapter for SCSI drives, 16 MHz CPU with 16 KB cache, what initially works at ST compatible 8 MHz. Then, there is VME expansion port. That was not best idea by me, since cards for it were expensive and not spread. 
Mega STE was well built, and most reliable of all them. Target users professionals. I really don't know what was the price when it was launched.  But all it was too late. PCs took over serious market, Amiga already took over gamer market. TOS 2.06 had much better Desktop, some later CPU as 68010 support - and lower old SW compatibility.


Now, probably most interesting question is: what are the reasons for later failure, for dropping out from business, after some 8 years ?
My opinions is that there were mistakes from very beginning. Here I mentioned, described some of. Development should be faster, STE came too late, and worse, extra capabilities were not used well, mostly because SW authors ignored them - probably because lack of motivation (expected sales), bad documentation.
Then, part of success was in using custom high integrated chips, manufacturing machines in Taiwan. After 1991, exactly the Taiwan firms went in developing custom chips for PCs - what dropped prices, made motherboard progress much faster.
Jack Tramiel became too old, and giving leadership to his sons made not it better.
And one important thing:  lack of universal expansion port, what would contain CPU bus, + diverse control signals. Where can attach for instance ROMs, with new TOS version, mass storage adapters, CPU accelerators, and maybe even RAM expansion. Expansion port appeared first in Mega ST, and was not god for all types of expansion. Even worse was that it was not designed future compatible, so TT, Mega STE have different one (VME), and Falcon again different one.
Sales in USA were never much good. That is complex question, but image of Atari, as gaming oriented company helped not for sure.
Then, as we know, Amiga was designed by former Atari HW designers, and they did it with Atari as customer in sight. Then some strange things happened, and project went to Commodore, without Jack Tramiel then. What could be done from Amiga project under Atari (certainly with different name) is interesting question. If Jack Tramiel was interested at all for such machine .

Here need to say some words about Atari ST SW, mostly TOS related things: compatibility problems with diverse TOS versions of Atari ST SW are known for users, and it is one of areas where it is bad without really good reason. Basic concept of CPU MC68000 and TOS is such, that ensures that SW will work on later TOS versions - of course when it follows some rules. So, some could say that we need to blame only SW authors, which were deaf for rules. Is it really so ? No, big part of it is on Atari. Poor documentation, with missing necessary details made real headache for programmers, and forced them to use some so called 'not clean' solutions. For instance, input read of keyboard, mouse and joystick is solved in overcomplicated way, what has it benefits (in rare cases), but most of SW just wants simple way to read state of those inputs. But it is not just it needs little more code to read, sadly, it is so poorly documented, that plenty of SW just using wether own code for input read, or some 'hacky' way to read input partially via TOS. Worst is joystick read.
TOS 2.06 made some larger changes, what makes it known for not game friendly, but many serious SW works not (well) too. Was it even mentioned that Timer C is reserved ?
Things are that there is simply no TOS version what is best considering SW compatibility. Earlier ones are condidered as better for games, and it is true, but not always. And if you are hard disk user, then 1.04 is absolute minimum.
Some SW using really bad solutions, ideas. For instance great game Stardust has really stupid code for detecting HW on which it runs, what fails if there is not specific TOS version (relays on content of specific address in ROM, but that is different in any TOS v.), or when runs in emulator.
Best is to have multiple TOS versions in machine, switchable. Not something new. I had it in my ST from 1992. 1.04 or 2.06 was it.
Today is better having 4 TOS versions (all it fits in 2 EPROM chips).


Atari ST today, in year 2018.  Of course, it is now retro machine, but some still use it for doing some work. Most for playing. And even more via emulators. Which appeared for Atari ST after year 1998, when PCs became enough fast.
There are diverse add-ons available. Probably most popular are Flash card based mass storage solutions. Then HW floppy emulators, since real drives and disks became really problematic and unreliable. There are accelerators, some graphic improvements available too, and of course FPGA clones - although now some developers pulled out.
So, yes, Atari ST is still alive in some way. For not big number of people. I guess that is is under 10000 on whole Planet, counting all, even occasional users. And even some new SW, new games appear from time to time. There are plans for some clones too.
I guess that's normal after millions were sold.
As bigger quality I consider it's power/price ratio. Universal usability for serious and fun SW. That still matters a lot. Some could say, that Atari shown the way to PC. I think that it was just natural way, and Atari ST was first which realized it well.

Some interesting facts about TOS


    PP,  May  2018.