f



format of floppy disk image file

Regarding floppy disk image creation tools (fdutils, 22DISK, ImageDisk, Tel=
eDisk, cpmtools, etc.) and how they write their resultant image file ...

As far as I have learned, there are three (3) primary ways data is physical=
ly written to a double-sided floppy disk: flip, flat, and up-and-over.

For "flip," logical tracks 00 .. 05 are written like this:
00 02 04 <-- head 0
---------- <-- disk medium
01 03 05 <-- head 1

For "flat," logical tracks 00 .. 05 are written like this:
00 01 02
----------
03 04 05

For "up-and-over," logical tracks 00 .. 05 are written like this:
00 01 02
----------
05 04 03

My question is this: is there a standard - de facto or otherwise - that say=
s how a floppy imaging tool will read the physical disk and produce its ima=
ge file? Will it simply "scan the disk" or does it use knowledge of how the=
 disk was physically written to produce a "sequential track" file?

Using the "flip" example above, when such a physical disk is read and an im=
age file is created, would the imaging tool write the file as "00 02 04 01 =
03 05" or would it write the file as "00 01 02 03 04 05" ... ?

Thanks!
0
LiamF
11/18/2016 12:20:21 PM
comp.os.cpm 3422 articles. 0 followers. Post Follow

31 Replies
501 Views

Similar Articles

[PageSpeed] 15

On Friday, November 18, 2016 at 1:20:22 PM UTC+1, LiamF wrote:

> As far as I have learned, there are three (3) primary ways data is
> physically written to a double-sided floppy disk: flip, flat, and
> up-and-over.
> 
> For "flip," logical tracks 00 .. 05 are written like this:
> 00 02 04 <-- head 0
> ---------- <-- disk medium
> 01 03 05 <-- head 1

Also the opposite was in use:

01 03 05
--------
00 02 04

> My question is this: is there a standard - de facto or otherwise
> - that says how a floppy imaging tool will read the physical disk and
> produce its image file? Will it simply "scan the disk" or does it use
> knowledge of how the disk was physically written to produce a "sequential
> track" file?

All tools use some sort of database with descriptions for the hundreds
of formats that were used. Just by scanning the disk you won't find
out it which order sectors are written.

> Using the "flip" example above, when such a physical disk is read and an
> image file is created, would the imaging tool write the file as "00 02 04
> 01 03 05" or would it write the file as "00 01 02 03 04 05" ... ?

For the Cromemco disk images I use 00 01 02 ... because that makes the
computation somewhat easier, of course others might prefer another scheme.
0
Udo
11/18/2016 1:13:33 PM
Liam wrote:
> "...is there a standard... that says how a floppy
> imaging tool will read the physical disk and
> produce its image file?

You'll likely find that everyone that wrote a disk imaging format and application program, did it uniquely or based their divergent changes upon what they didn't like about a prior format/program.

So while there was no standard, there will be recurring patterns in the various solutions created, due to constraints in the floppy disk controller used, the feasible floppy formats, the host computer circuit design and operation system software.

Some have documented their formats and some have been analyzed by others and documented. Others require digging into the firmware and circuit board schematics to find out how a particular vintage computer operated with its floppy disk drives.

Its an interesting but challenging subject. If you have a specific computer system, or image format/program in mind, post that information. You may find that someone has already done a lot of the work you're doing some time in the past 40 years, and can answer your questions more directly.

JDallas
0
11/18/2016 6:32:25 PM
> Also the opposite was in use:
> 
> 01 03 05
> --------
> 00 02 04
> 

((sigh)) I never considered that one.
It seems rather counterintuitive and rages against my sensibilities.
I suppose I ignore it at my own peril, read: incompatibility.
0
LiamF
11/19/2016 11:30:56 AM
On Saturday, November 19, 2016 at 12:30:57 PM UTC+1, LiamF wrote:

> > Also the opposite was in use:
> > 
> > 01 03 05
> > --------
> > 00 02 04
> > 
> 
> ((sigh)) I never considered that one.
> It seems rather counterintuitive and rages against my sensibilities.
> I suppose I ignore it at my own peril, read: incompatibility.

If it was technically doable it was done, everyone wanted their own
unique disk format. Was kind of a hobby to create another one ;-)
This resulted in hundreds of disk formats, the only one that was
kinda standard: IBM single side, single density, 26 sectors 77 tracks.
 
0
Udo
11/19/2016 11:51:15 AM
> You'll likely find that everyone that wrote a disk imaging format and app=
lication program, did it uniquely or based their divergent changes upon wha=
t they didn't like about a prior format/program.
I was afraid of that but I am unsurprised. I suppose what's easiest for me =
is to go with my assumptions/preferences and be prepared to mitigate my com=
patibility woes with one-off "format translators."

> Its an interesting but challenging subject. If you have a specific comput=
er system, or image format/program in mind, post that information. You may =
find that someone has already done a lot of the work you're doing some time=
 in the past 40 years, and can answer your questions more directly.
I don't have a specific image format/program in mind at the moment. (I also=
 don't have at this time the requisite hardware for creating FDD images.) I=
've written a small and "simple" 8080 CPU & system emulator. Currently I ha=
ve the joy of running CP/M 2.2 and some other OS's on the system. The emula=
tor supports multi-head disk formats but I've never tested it beyond a sing=
le head. I downloaded several double-sided floppy image files from various =
places and quickly discovered great differences, even when "identical geome=
try" was indicated. (I found that some imagers go as far as to compress the=
 resultant file!) Instead of trying to build some grand "image format compa=
tibility" mechanism into my emulator, methinks my time would be better spen=
t sketching out a "translation framework" or other such reinvented wheel th=
at would allow me to quickly do ad hoc import/export of the foreign formats=
 as they arise.
0
LiamF
11/19/2016 12:00:17 PM
> If it was technically doable it was done, everyone wanted their own
> unique disk format. Was kind of a hobby to create another one ;-)
> This resulted in hundreds of disk formats, the only one that was
> kinda standard: IBM single side, single density, 26 sectors 77 tracks.
IBM3740 is my most best friend ever.
0
LiamF
11/19/2016 12:03:48 PM
On Saturday, November 19, 2016 at 1:00:17 PM UTC+1, LiamF wrote:

 I've written a small and "simple" 8080 CPU & system emulator. Currently I =
have the joy of running CP/M 2.2 and some other OS's on the system. The emu=
lator supports multi-head disk formats but I've never tested it beyond a si=
ngle head. I downloaded several double-sided floppy image files from variou=
s places and quickly discovered great differences, even when "identical geo=
metry" was indicated. (I found that some imagers go as far as to compress t=
he resultant file!) Instead of trying to build some grand "image format com=
patibility" mechanism into my emulator, methinks my time would be better sp=
ent sketching out a "translation framework" or other such reinvented wheel =
that would allow me to quickly do ad hoc import/export of the foreign forma=
ts as they arise.

Same as with hundreds disk formats OF COURSE we also need hundreds of image=
 formats ;-)
I understand why image formats exists, that perverse every detail for a flo=
ppy disk. These
can be used to write the image back to a disk and have a working one. Of co=
urse it is superfluous,
simply because in a foreseeable future you won't have any working floppy di=
sk anymore.

That is why I use raw images in z80pack only, trivial to handle with tools,=
 even any hex editor
will do. I don't want to write the images back to floppy disk anyway.

0
Udo
11/19/2016 12:22:15 PM
> That is why I use raw images in z80pack only, trivial to handle with tool=
s, even any hex editor
> will do. I don't want to write the images back to floppy disk anyway.

Honestly, it was this idea of "raw images" that prompted my original post: =
since there are several - if not may - ways to physically lay tracks on a f=
loppy disk, how does one define "raw image." It would seem that this is act=
ually defined by the imaging tool and could be as varied as the number of d=
ifferent physical layouts used over the years.
0
LiamF
11/19/2016 12:53:23 PM
On Saturday, November 19, 2016 at 1:53:24 PM UTC+1, LiamF wrote:

> Honestly, it was this idea of "raw images" that prompted my original post: since there are several
> - if not may - ways to physically lay tracks on a floppy disk, how does one define "raw image."
> It would seem that this is actually defined by the imaging tool and could be as varied as the number
> of different physical layouts used over the years.

Raw image means, that it only contains the user data for every sector on every track, no gap bytes,
no ID fields with position, checksum, whatever. Only raw data, not compressed.

The tool you use for reading a disk into an image file defines the image format. If it reads disk sides
alternating and not one whole side after another, well, then that is so. You use this tool and have
the image it produces, or you don't use it and won't have an image, or you write a new one.

And of course you can write tools to convert image formats into others. So I just convert other images
into a simple raw format and use that if possible.

0
Udo
11/19/2016 1:27:19 PM
It's been very helpful to talk this out. Luckily, all the images I've been =
working with (downloaded and created by me) are "raw." Since I've been work=
ing (so far) with only singled-sided images, the physical track order match=
es the logical track order. I now have a far better idea of what to expect =
as I start working with double-sided images. (And I have a better idea of w=
hat I need to do if I produce a double-sided image.)
Thank you!
0
LiamF
11/19/2016 1:48:46 PM
On Saturday, November 19, 2016 at 2:48:47 PM UTC+1, LiamF wrote:
> It's been very helpful to talk this out. Luckily, all the images I've been working with (downloaded
> and created by me) are "raw." Since I've been working (so far) with only singled-sided images, the
> physical track order matches the logical track order. I now have a far better idea of what to expect
> as I start working with double-sided images. (And I have a better idea of what I need to do if I
> produce a double-sided image.)
> Thank you!

You are welcome. Have a look the the z80pack Cromemco FDC emulation sources. This one
handles 5.25" and 8" images, single sided, double sided, single density, double density, every
possible combination of that. There is a function that calculates the offset for an lseek() on
an image file, dependent on the disk format. Should give you an idea how to handle double
sided disks.
0
Udo
11/19/2016 2:03:29 PM
In article <ee087cf7-3184-4a0d-9f0d-98fc15de1df4@googlegroups.com>,
LiamF  <william.t.fry@gmail.com> wrote:
>It's been very helpful to talk this out. Luckily, all the images I've been =
>working with (downloaded and created by me) are "raw."

I've been following this thread with interest
because I too may someday be tasked with archiving disks-of-many-formats.
I'm unsure if anyone clarified the terms
"disk image" vs. "disk contents".

As an analogy, consider a facsimile reproduction of an author's manuscript
vs. a typeset copy of the book.
It is all the same words but the manuscript is often in the author's own handwriting
and shows all sorts of edits and changes vs. the polished finished product.
With PDF/e-book formats, you can even reformat the book
with larger fonts, or different fonts. But it's all the same words.

I'd consider a "disk image" an exact copy of the raw bits,
meaning it preserves the sector skew and order of the original.
Perhaps even the entire raw track, preserving the formatting
(preamble, postamble, space between sectors).
That's what folks need when they ask for a 'boot floppy'.
But it's almost unuseable to any other machine.

When I was decommissioning my CP/M system,
I copied the files to my DOS PC via Ymodem.
I saved the CONTENTS of the disk files albeit in a new format (DOS/FAT file system).
Then they were copied to Linux systems.
That's how we shared files, whether paper tape, data cassette, or via BBS.
The Walnut Creek CD is file contents, not raw disk images.
For most purposes, that's what we need: the file contents with the original file names.

I see merit to archiving things BOTH WAYS.

-- jeff jonas

0
jeffj
11/19/2016 10:39:50 PM
Udo Munk <udo.munk@freenet.de> writes:
> On Saturday, November 19, 2016 at 1:53:24 PM UTC+1, LiamF wrote:
> 
> Raw image means, that it only contains the user data for every sector on every track, no gap bytes,
> no ID fields with position, checksum, whatever. Only raw data, not compressed.

That's an interesting take. I tend to call that 'cooked'. What I call 'raw'
is the byte output of a FDC chip like a 1793 with inter-sector gap bytes,
sync bytes, density bytes, CRC bytes, sector ID bytes, etc, etc.

But labels are just labels, as long as the software understands what the
data means, who cares what *we* call it. <grin>

Jack
-- 
"It's rather cold." she said bitchily.

0
Jack
11/20/2016 12:08:02 AM
Jeff wrote:
"...the terms "disk image" vs. "disk contents...
...a "disk image" an exact copy of the raw bits, 
meaning it preserves the sector skew and
order of the original..."

I think a more useful method is in the middle.

I agree with your definition of disk-image, in so far as the data is attainable by a floppy disk controller. That's not always the case. A track-read command that was available in later floppy disk controller chips (FDC) comes closest to a disk-image. Its overkill in most cases and creates needlessly bloated image files.

A complete disk-image is most necessary when some form of disk verification is used, such as a game/application that checks a non-standard sector or manipulated header to verify that it is a disk bought from the game/application provider. Its a record of the best image of the disk, warts and all. But that level of detail is seldom really required. If every disk-image has that level of detail, it has a lot of wasteful bulk and the useful file data is cryptically woven into the mess.

If only it was as easy as using a track-read. The early floppy disk controller boards didn't have LSI chips to offer track-reads. With hard and soft sectored formats done in MSI logic, there are a multitude of ways to format tracks and the raw format is usually not recoverable to the host system. For example the sector gaps could be blank areas of the disk in some cases, or written patterns in others. If you only have a sector read available, you can only map a physical sector to its data contents and use the directory structure to thread it together. Of course the logical sector table needs to be known to thread the sectors correctly. But all the information about the track layout and spacing is largely unavailable. That leaves the image-disk program to layout the track, rightly or wrongly.

When I was in grad school ('90-'94), I was corresponding on usenet(?) with someone that was working on programming a new disk imaging format and program. In the early 80's, I had designed hard drive and floppy disk controller boards and so was helping with the raw format of floppies, and helped a little with ways to define the file structure; like removing parts that were already ruled out by some of the leading data.

I said at the time that I had a document from IBM and another from Shugart that would be useful for deriving the correct gap lengths, which would have allowed omitting the raw gap images. Technically it would have likely improved the gap and sector placements on the track, as some systems appeared to have implemented them non-optimally. The catch was that all my documents were in storage because I was in grad school. I couldn't even be sure if I still possessed those documents. So I had to punt on that idea to improve gap and sector placement and removing the raw stream from the image file.

Years later, in 2014, I found the Shugart document when I started digging into my stored documents and computers from the 70s onward. I mentioned its name and document number in comp.os.cpm back in 2014+ when I was helping diagnose a problem with a Zorba disk made from an image-file. The Shugart document is highly flawed with gross errors and miscalculations, poorly based upon the 18 page IBM report that I read and trashed in 1983 while at SD Systems. The intent of the Shugart document was to take the crux of the IBM research study on how to make a reliable format, and just include the calculations and data, no statistical analysis and no graphs.

The document is I have is: "Shugart, Format Manual, 8 inch Floppy, December, 1981"

It was supplied by a disk vendor when I was working on a hard drive plus floppy drive controller design for SD Systems. My boss (Gary Keto) and I both read and discussed the two documents to assure we understood it correctly. The Shugart document could be applied to 5 1/4inch mini drives by changing some of the figures accordingly. Apparently the authors mixed up the two sets of figures repeatedly in their calculations and created a lot of unusable nonsense. In 1983, spreadsheets were not common, BASIC programs were typically used instead.

However, in 2014 better spreadsheets are common and untangling the mess was finally necessary. As I had read the original IBM research paper in 1983, I knew what the figures were trying to do, and only had to sort out the mistakes. To figure out the Zorba format problem, I had to correct the Shugart document and apply the calculations to various formatting patterns to uncover the overwrite scenario.

So now in 2016, I think it is possible to do something with the idea I suggested when I was in grad school. Take a disk image and turn it into a form that puts the program/data files into binary form without sector patterns, and then in a separate file, document the format of the disk in a much more condensed form with no need of raw streams. This would allow the program/data files to be paired with a different format file to write the same disk formatted for a different computer system.

That's the theory anyhow.

Note that this approach only encapsulates normal file formatting, and would not replicate the verification disk forced-anomalies typical of some game or application programs mentioned above. But it would make image files a source to reference when trying to rebuild a program with a sector error. Floppy recovery pushed way beyond the mere CRC checking.

This separation of the program/data files from the formatting descriptors, leads to an application program that could compare a program that appeared on disks for different systems to identify the base program that should be the same, bleaching out system re-locations, to find the core commonality. That could lead to filling in a missing sector from a bad floppy. For example, if an unreadable floppy sector is in the base region of the know program/data, than that sector could be recovered from another system. The base regions of the code are the same, though some re-locations may apply; something software can handle.

It would also identify the blocks of code that are completely divergent between the two systems; that would typically be special drivers unique to each. Unreadable sectors in these divergent areas would require a source for the same host system, if available, otherwise no restoration would be available beyond hacking the missing data from examples in other system versions.

That's a new level of system recovery that would be interesting to investigate. Ironically, the disks of that era may not be readable much longer, so while this seems to open new methods of recovery, the need is quickly disappearing.

But its an interesting problem to solve. :)

JDallas
0
11/20/2016 4:10:48 AM
Jeff Jonas wrote:
> I'd consider a "disk image" an exact copy of the raw bits,
> meaning it preserves the sector skew and order of the original.
> Perhaps even the entire raw track, preserving the formatting
> (preamble, postamble, space between sectors).
> That's what folks need when they ask for a 'boot floppy'.
> But it's almost unuseable to any other machine.

I disagree. An image is the sequence of sectors, one after the other
from 1 to n, in the order the higher levels of the OS expect and address
them. If you need and want to read or write from and to a physical
floppy you will need the relevant hardware and routines anyway. The
format you propose is of no extra help here and most setups can't read
it from the media in the first place.

> I see merit to archiving things BOTH WAYS.

Exactly. There are quite a few things besides boot sectors that a file
copy won't get.

Axel

-- =

/=AF\   No  |    Dipl.-Ing. F. Axel Berger    Tel: +49/ 221/ 7771 8067
\ /  HTML |    Roald-Amundsen-Stra=DFe 2a     Fax: +49/ 221/ 7771 8069
=A0X    in  |    D-50829 K=F6ln-Ossendorf      http://berger-odenthal.de
/ \  Mail | -- No unannounced, large, binary attachments, please! --
0
Axel
11/20/2016 5:34:11 AM
On Sun, 20 Nov 2016, Axel Berger wrote:

> Jeff Jonas wrote:
>> I'd consider a "disk image" an exact copy of the raw bits,
>> meaning it preserves the sector skew and order of the original.
>> Perhaps even the entire raw track, preserving the formatting
>> (preamble, postamble, space between sectors).
>> That's what folks need when they ask for a 'boot floppy'.
>> But it's almost unuseable to any other machine.
>
> I disagree. An image is the sequence of sectors, one after the other
> from 1 to n, in the order the higher levels of the OS expect and address
> them. If you need and want to read or write from and to a physical
> floppy you will need the relevant hardware and routines anyway. The
> format you propose is of no extra help here and most setups can't read
> it from the media in the first place.

In the Apple ][ world we have 3 major types of floppy disk images.

There is the "Nibble Image", which replicates the track structure as 
closely as the drive is capable of reading it under normal conditions.

Then there are "DOS Order" and "ProDOS Order", which are the data after 
decoding, skewed according to the differing preferences of two different 
disk operating systems which use the same low-level disk format.

Because low level formats can vary dramatically there for programs which 
use their own custom DOSes, even the "Nibble Image" is incapable of 
compensating for everything; for one, it can't handle cases where the 
software might deliberately throw the drive out of synch or run with the 
motor stepped between tracks - techniques that were incredibly common for 
detecting copied disks.  Formats are still being worked on to preserve 
these disks without having to resort to cracking.

-uso.
0
Steve
11/20/2016 6:49:32 AM
On Sunday, November 20, 2016 at 7:49:35 AM UTC+1, Steve Nickolas wrote:
> On Sun, 20 Nov 2016, Axel Berger wrote:
> 
> > Jeff Jonas wrote:
> >> I'd consider a "disk image" an exact copy of the raw bits,
> >> meaning it preserves the sector skew and order of the original.
> >> Perhaps even the entire raw track, preserving the formatting
> >> (preamble, postamble, space between sectors).
> >> That's what folks need when they ask for a 'boot floppy'.
> >> But it's almost unuseable to any other machine.
> >
> > I disagree. An image is the sequence of sectors, one after the other
> > from 1 to n, in the order the higher levels of the OS expect and address
> > them. If you need and want to read or write from and to a physical
> > floppy you will need the relevant hardware and routines anyway. The
> > format you propose is of no extra help here and most setups can't read
> > it from the media in the first place.

Worse, the extra bytes use up space and never ever are used for anything.
For using the image in an emulation more complex algorithms are needed
for calculating the offset to a data sector somewhere in the image.

> In the Apple ][ world we have 3 major types of floppy disk images.
> 
> There is the "Nibble Image", which replicates the track structure as 
> closely as the drive is capable of reading it under normal conditions.
> 
> Then there are "DOS Order" and "ProDOS Order", which are the data after 
> decoding, skewed according to the differing preferences of two different 
> disk operating systems which use the same low-level disk format.
> 
> Because low level formats can vary dramatically there for programs which 
> use their own custom DOSes, even the "Nibble Image" is incapable of 
> compensating for everything; for one, it can't handle cases where the 
> software might deliberately throw the drive out of synch or run with the 
> motor stepped between tracks - techniques that were incredibly common for 
> detecting copied disks.  Formats are still being worked on to preserve 
> these disks without having to resort to cracking.

For stuff like this one needs images with all the details, or one would have
to shortcut code that tests for sectors with wrong checksums and what all.
Fortunately the nonsense wasn't done to CP/M disks for the Apple ][, so for
that raw images with sector data only work. And cpmtools will work with the
images too.
0
Udo
11/20/2016 11:41:56 AM
On 11/19/16 8:10 PM, Jay_in_Dallas wrote:

> Note that this approach only encapsulates normal file formatting, and would not replicate the verification disk forced-anomalies typical of some game or application programs mentioned above. But it would make image files a source to reference when trying to rebuild a program with a sector error. Floppy recovery pushed way beyond the mere CRC checking.
> 
> This separation of the program/data files from the formatting descriptors, leads to an application program that could compare a program that appeared on disks for different systems to identify the base program that should be the same, bleaching out system re-locations, to find the core commonality. That could lead to filling in a missing sector from a bad floppy. For example, if an unreadable floppy sector is in the base region of the know program/data, than that sector could be recovered from another system. The base regions of the code are the same, though some re-locations may apply; something software can handle.
> 
> It would also identify the blocks of code that are completely divergent between the two systems; that would typically be special drivers unique to each. Unreadable sectors in these divergent areas would require a source for the same host system, if available, otherwise no restoration would be available beyond hacking the missing data from examples in other system versions.
> 
> That's a new level of system recovery that would be interesting to investigate. Ironically, the disks of that era may not be readable much longer, so while this seems to open new methods of recovery, the need is quickly disappearing.
> 
> But its an interesting problem to solve. :)
> 
> JDallas
> 
> 
> 

Actually, the need for format documentation is increasing for the museum/preservation community.
At the Computer History Museum where I work, we have tens of thousands of floppies that have been
collected and I spend a great deal of time trying to recover undocumented hard and soft sectored
floppies and determine what is a candidate for long-term archival preservation.

The first problem is accurately imaging the original media, which is a challenge in first determining
what system it came from, reading the sectors, verifying the sectors with provisions for dealing with
soft errors and copy-protection, attempting to determine what the file system was, then extracting the
files or at least creating a table of contents.

This is on media that is deteriorating, and in some cases for the really bad brands, is a destructive
process so I only get one shot at recovery.

One of the things I've been arguing for is a list of known copy-protected software so I can tell up front
if there are going to be expected media errors or not.

0
Al
11/20/2016 3:21:02 PM
On Friday, November 18, 2016 at 12:20:22 PM UTC, LiamF wrote:
> Using the "flip" example above, when such a physical disk is read and an =
image file is created, would the imaging tool write the file as "00 02 04 0=
1 03 05" or would it write the file as "00 01 02 03 04 05" ... ?

In general, an imaging tool for a system will use the order that was most c=
ommon on that system. MS-DOS and Linux on the PC use the 'flip' order, for =
example, so disc images created by them will write the file 00 01 02 03 04.=
...

This does mean that if you have an MS-DOS or Linux system image a disc in t=
he 'up-and-over' format, the tracks in the file will be in the order 00 05 =
01 04 02 03, and you'll have to be aware of that when processing the file.

There are some systems (like the +D interface for the ZX Spectrum) where di=
sc images can come in one of two formats: native filesystem order or PC 'fl=
ip' order, and because they don't contain any metadata you have to look at =
the file extension to determine which one is in use. That's one reason why =
I prefer archiving in formats like ImageDisk or CPCEMU .DSK where the image=
 file contains metadata giving the physical cylinder / head / sector, recor=
ding mode and so on.

--=20
John Elliott
0
John
11/21/2016 10:07:28 AM
On Monday, November 21, 2016 at 11:07:29 AM UTC+1, John Elliott wrote:

> That's one reason why I prefer archiving in formats like ImageDisk or
> CPCEMU .DSK where the image file contains metadata giving the physical
> cylinder / head / sector, recording mode and so on.

The metadata might not be obvious to guess, so someone needs to write
documentation for that. But then again one could leave the metadata
out and just write the documentation for the image itself, just saying.

While the idea with the metadata is good, the problem is that of course
everyone and their dog needs to invent some other metadata format :(
If there would be some sort of standard everyone could agree on and
use it, then this could be used to archive disk images and use them
right away in any emulations.

Isn't the case, I'm not interested in supporting dozends of image
formats and probably others aren't eager to do that.

If anyone is willing to write some standard I can provide feedback
from the sight of using such images from emulations. E.g. compression
is no good, try to write a record somewhere in the middle of a
compressed file.
0
Udo
11/21/2016 12:09:10 PM
On Monday, November 21, 2016 at 12:09:11 PM UTC, Udo Munk wrote:
> If anyone is willing to write some standard I can provide feedback
> from the sight of using such images from emulations. E.g. compression
> is no good, try to write a record somewhere in the middle of a
> compressed file.

My own attempt at this is LDBS: http://www.seasip.info/Unix/LibDsk/ldbs.htm=
l

I wrote it for internal use in LibDsk -- a lot of archival disk image forma=
ts have the problem that they can't be rewritten in-place, as you point out=
, or can only locate track <n> by first processing tracks 0 to <n-1>. So I =
found I was writing lots of variants on 'parse the file into an intermediat=
e format, write a driver that operates on that intermediate format, write o=
ut the archival image at file close'. By using a common file format (LDBS) =
as the intermediate format, I wouldn't have to keep doing the 'write a driv=
er' stage.

CPCEMU .DSK was written for an emulator and so has some level of in-place r=
ewrite capability, but there are some things (like reformatting a track fro=
m 8 sectors to 9) that it can't handle.

--=20
John Elliott
0
John
11/21/2016 3:49:36 PM
Al wrote:
> "...the need for format documentation is increasing
> for the museum/preservation community...the first
> problem is accurately imaging the original media,
> which is a challenge in first determining what
> system it came from...this is on media that is
> deteriorating, and in some cases for the really bad
> brands, is a destructive process..."

Assuming for a moment, a non-destructive floppy recovery tool (see below) instead of a FDD, the problem of identification may be enhanced by recovery of the raw format information on the tracks, to supplement the diverse cylinder pattern *signature* of possible host system sets.

While the likely intent was that every new floppy format would adjust the sector placement and gap/sync settings, either per IBM's study leading to their standard formats, or by equally clever study within each computer system company, it appeared to me at least in 1983, that the drive manufacturers and the FDC chip manufacturers didn't really want to take ownership or provide much guidance in formatting issues. They appeared to stick to what they knew about their drives or their chips and let the system companies take any risks beyond the small set of small sector IBM floppy format standards.

What I heard at the time was that most simply used the same gap and sync lengths from the nearest IBM floppy format standard and if it worked, no further study was done. If it didn't work, tinkering was the next economical process and a fair heuristic could be derived to do that reasonably well without looking at the problem too deeply. This might be confirmed by studying some of the collected image files that characterize the gap and sync blocks, but I haven't been motivated to do this myself.

The point of this is that their may be more *signature* information in the track format than is currently being used. The counter argument is the blind adoption of the nearest IBM floppy standard's gap and sync choices. That suggests that unless some company pushed the data near the breaking point on a track, they'd just use what everyone else used - which would *not* offer additional signature to identify what host system, or set of possible host systems, a floppy was created for. But any floppy format that diverged from the "This Will Work" settings shared between companies, may be unique enough to enhance identification of the floppy's host.

Now reading that information is not immediately accessible in every floppy disk controller. But its relatively easy to bypass all the FDC functions with Today's very fast micros and create a track imaging board that basically provides the location of every magnetic flux marking on a floppy track. This copious data would then be broken down on a PC. Note that once you bypass the need of the data-separator you can spin the floppy at any speed, slow or faster, as long as you new imaging board can store the information fast enough. If the floppy turn slower, there are more micros that can manage the task. A CPLD or FPGA could do it too, but personally I prefer maximum flexibility that firmware offers.

Not too that the speed flexibility means that the use of the typical floppy disk drive motors could be abandoned for one that favors a more accurate *constant* rotational speed. A true constant speed of rotation will improve the flux marking recovered by this type of track imaging. The speed isn't important during this imaging process because no matter what it was, as long as the resolution was good enough to map the flux markings the PC computer when crunching the imaged data, can recover the clock from the flux pattern, even as it changes mid track due to the way floppy disk drives controlled their instantaneous speed.

When the PC crunches the data, it can look for signatures to reduce the set of potential host systems for that floppy. With 10,000 floppies and many active systems in a museum, combined with the sectoring pattern compiled by years of prior imaging and recovery work of many, such signatures could be compiled.

Now the elephant in the room is the media-destructive floppy reader.

Some notes on an idealized tool, a subset of which might be practical at some time:

At some point in the future or perhaps already in the past, a restored floppy disk drive will not be the correct solution for reading floppies that are *fragile* in some sense of its condition. I've pondered this a little and have concluded that the next step in floppy recovery will be a less destructive floppy operation that requires removing the media from the jacket so it can be placed in a new reader where the media will be fixed (i.e. not spinning) and the read/write heads spin above it. To hold the media in this fixed position and to prevent the read/write heads from damaging the media, a thin protective plate needs to placed atop the media. Of course that material, with its various requirements for (1) resisting read/write head wear, (2) being very thin thickness to allow the read/write head to be extremely close to the media surface, (3) having strength so that the cover plate doesn't bind up and destroy the media and (4) having magnetically inert properties (or slightly positive properties to extend the flux field upward) so that it doesn't lessen the magnetic flux markings that need to be read. These are the unknowns for now.

While this may be so far of a departure from the floppy disk drive that some react by screaming "witchcraft!", I do think its possible, for what that's worth.

As to the spinning read/write heads, with complete access and with no locked in step distances, it could adjust itself to and track pattern and self-center upon skewed floppies where the tracks are not correctly aligned. As the read/write heads have been removed from the floppy disk drive, there is no need to carry forward its READDATE set pulse width. That would allow it to even read the Apple and Commodore formats that put more sectors in the outer tracks, without changing the speed of the read/write traversals.

To avoid making every floppy recovery a long duration task, the tool needs to be micro or PC controlled to quickly configure and change course as necessary so that mountains of floppies can be processed in the quickest time possible.

Sure this is kind of an Ideal floppy reader description, but some aspects might be an interim tool that would make sense to try as the floppy disk drives become more of a floppy shredder solution instead.

JDallas
0
11/21/2016 5:20:55 PM
Two additional thoughts on the ideal floppy recovery machine:

(1) The "material" could feasibly be a roll of one-sided, wax-coating paper as used in the food packaging industry. Advantages are that: * its thin, * its likely magnetically inert, * wax-side down wouldn't reposition any loose magnetic dust on the floppy media surface even if some sticks to it during a read operation due to pressure of the r/w head, * its cheap, *its disposable - after each floppy surface recovery; just lift off the media (even if some sticks destructively) and advance the sheet from the roll for the next clean wax-paper measure to contact the next floppy surface processed, * it limits the head cleaning to paper dust which can be cleared with a blast of compressed air, * the paper can be seated taut in a floppy cutout frame so that it stays flat and doesn't move during r/w head contact. In this scenario, the protective "plate" material is paper so I'll call it the paper "plate" scenario, not to be confused with something used at a picnic.

(2) The mechanical contact of the read/write head (r/w) spinning during contact to the paper side of the wax-paper protective "plate" could be balanced by distributing a few working r/w heads around the circle of rotation, such that each r/w head feeds into a different track read recovery board. While the primary reason is to balance the contact force around a group of close tracks to avoid paper distortion, it has an additional advantage of picking up several tracks per revolution, thereby reducing the total time-to-read one floppy media side. And if several revolutions are read per track for statistical sampling to improve reading of marginal flux patterns of a poor condition floppy, the read time is reduced by the number of r/w heads used. For example, if you used four r/w head and staggered them 90 degrees from each other, they'd distribute the contact load/distortion on the protective paper "plate" to be more similar to a ring-contact. Four heads mean the floppy side can be read in one fourth of the time as a single head. Fewer heads make a lighter contact with less friction, but too many increase the forces on the paper "plate" and make matters worse.

Note that under (2) there is no reason to wait for index/sector pulses before reading as a normal floppy disk controller board would do; those locations can be mapped to the radial position in post-processing on the PC. As long as one revolution plus a 10% to 20% overlap of each track is collected for processing and as long as the index/sector locations can be applied to each track data collection, the post-processing on a PC can create the virtual floppy disk drive equivalence, to recovery the files from the floppy.

JDallas
0
11/25/2016 4:38:53 PM
On Fri, 25 Nov 2016 10:39:00 -0600, Jay_in_Dallas
<jay.c.box@earthlink.net> wrote:

>Two additional thoughts on the ideal floppy recovery machine:
>
>(1) The "material" could feasibly be a roll of one-sided, wax-coating pape

Really, Jay, way too complicated.

Why subject anything rare/valuable to damage via physical contact?

Many years ago, and almost certainly today, 3M made a developing fluid
for magnetic material. Using that, you can even recover data from
disks (soft OR hard) that have suffered bullet holes.

You ''develop'' the material, then read the light/dark transitions
optically, no need to touch the surface at all.

Something similar in the gadget you lay on for example a piece of
recording tape. There's liquid magnetic fluid trapped inside it, and
the weak field of the tape will ''show'' by aligning with that fluid,
so you can see how many tracks, sometimes even the direction they were
recorded. 

You can check it out.

Bill
0
Bill
11/26/2016 4:15:10 AM
I'm familiar with its use in hard drive recovery, and heard of it as a destructive coating; maybe when pushed that that flux density. I mentioned it in my original message but removed it as dead-wood, less applicable to floppies. The chemical coating has been mentioned in previous messages I've read, probably in comp.os.cpm and in other vintage computer blogs.

The point of the ideal recovery reader is that floppies degrade to the point where a floppy disk drive is no longer a good bet. Chemical coating are going to be more expensive per floppy, and that expense means its used for important data, like bullet-riddled evidence like Bonnie and Clyde's AOL account disk.

A mechanical reader that disposes a roll of paper is cheaper per unit.

I don't think anything like that will ever be built, but its interesting to ponder.

JDallas
0
11/26/2016 5:09:46 AM
Den 18-11-2016 kl. 13:20 skrev LiamF:
> Regarding floppy disk image creation tools (fdutils, 22DISK, ImageDisk, TeleDisk, cpmtools, etc.) and how they write their resultant image file ...
>
> As far as I have learned, there are three (3) primary ways data is physically written to a double-sided floppy disk: flip, flat, and up-and-over.
>
> For "flip," logical tracks 00 .. 05 are written like this:
> 00 02 04 <-- head 0
> ---------- <-- disk medium
> 01 03 05 <-- head 1
>
> For "flat," logical tracks 00 .. 05 are written like this:
> 00 01 02
> ----------
> 03 04 05
>
> For "up-and-over," logical tracks 00 .. 05 are written like this:
> 00 01 02
> ----------
> 05 04 03
>
> My question is this: is there a standard - de facto or otherwise - that says how a floppy imaging tool will read the physical disk and produce its image file? Will it simply "scan the disk" or does it use knowledge of how the disk was physically written to produce a "sequential track" file?
>
> Using the "flip" example above, when such a physical disk is read and an image file is created, would the imaging tool write the file as "00 02 04 01 03 05" or would it write the file as "00 01 02 03 04 05" ... ?
>
> Thanks!
>

The imagefiles contain information about sector-skew and "flip" (as you 
described) and so, this information is placed on the physical disk in 
whatever order the utility program does it.
There is AFAIR no defacto standard for that.
After the information is written to the floppy, the disk should perform 
and behave exactly like any disk written by the original system.
For example a Newbrain system disk should be identical in physical 
format to a copy made by a utility using a imagefile that knows the 
Newbrain system format.

There never was a standard to skew/sector-interleave or flipsides until 
the IBM PC as those parameters were determined by the hardware of the 
individual computer (floppy disc controller brand, DMA, CPU clock etc), 
and by the design of the manufactorer.

Regards, cep


0
snurrberget
11/27/2016 11:34:32 PM
Reply: