f



raw vs. cooked files under linux

Hi all,

i am an linux newbie.
I must shift our Databases from SCO Unix  (IDS 7.31 UD4) to SuSE Linux
Enterprise Edition 8.1 (2.4.21-190-smp glibc 2.2.5) or (8.2) or 9 and SuSE
Linux Professional 9.3 (2.6.11.4 glibc 2.3.4) (IDS 7.31 UD8).

Later on i want to upgrade (in place) to IDS 10.0.
Which Version of SuSE Linux Enterprise Edition must I use (8.? , 9 ..)?

Under Sco Unix we use raw-devices.
Is it better to use cooked files or raw files under these Linux-Versions?
A colleague mentioned, it can be, that later releases of Linux don't support
raw-devices. Is that correct?

Experiences or comments are welcome.
Thanks in advance.
(Excuse my poor school-english)

Heinz

sending to informix-list
0
Heinz
5/11/2005 8:48:28 AM
comp.databases.informix 16083 articles. 0 followers. Post Follow

13 Replies
707 Views

Similar Articles

[PageSpeed] 4

If you are in a slow-change environment raw-files are not a bad thing.
They create extra problems on the management side that cooked files
do not.  Fast-changing environments create more demand on your time
so why put yourself into extra work with raw files?  They don't give
you enough of a difference in performance to really warrant them unless
you are really demanding that extra 10%.  Cooked files are also also
more flexible on backups, you can back the dbspaces up with a regular
backup tool or use the Informix backup tools, but raw files will have
to be backed up with Informix tools only.

SLES9 is quite a sea change from SLES8, and you could probably get away
without SLES and go with the workstation Pro 9.3 if you are on a budget.
SLES has better support for SCSI than the workstation versions and you
must buy support or the online update doesn't work.  It's an enterprise
lock-in you might not need so try before you buy.  SLES9 can be downloaded
for trial, so give it a spin.  You should be able to download IDS 10 for
trial as well, thus no out of pocket to try it before paying for it.


Heinz Weitkamp wrote:
> Hi all,
> 
> i am an linux newbie.
> I must shift our Databases from SCO Unix  (IDS 7.31 UD4) to SuSE Linux
> Enterprise Edition 8.1 (2.4.21-190-smp glibc 2.2.5) or (8.2) or 9 and SuSE
> Linux Professional 9.3 (2.6.11.4 glibc 2.3.4) (IDS 7.31 UD8).
> 
> Later on i want to upgrade (in place) to IDS 10.0.
> Which Version of SuSE Linux Enterprise Edition must I use (8.? , 9 ..)?
> 
> Under Sco Unix we use raw-devices.
> Is it better to use cooked files or raw files under these Linux-Versions?
> A colleague mentioned, it can be, that later releases of Linux don't support
> raw-devices. Is that correct?
> 
> Experiences or comments are welcome.
> Thanks in advance.
> (Excuse my poor school-english)
> 
> Heinz
> 
> sending to informix-list

0
Data
5/11/2005 1:07:39 PM
Heinz Weitkamp wrote:
> Hi all,

Go RAW all the way.  Contrary to the Goob's understanding my own testing 
shows RAW devices to be 15-25% faster than COOKED devices which are in turn 
10-15% faster than filesystem files, even on the best filesystem 
implementations that adds up to a 25% improvement.  Well worth any minor 
inconveniences in my book.  I have not yet tested Linux RAW devices versus 
the various filesystems that Linux directly supports, but early reports when 
Linux RAW devices were first rolled out showed similar results.  The 
performance gain was one reason why Linus Torvalds was convinced finally to 
include RAW devices in Linux, a development he had resisted for years.  The 
other major reason being the need to provide high speed low level IO 
capability to software like IDS that manages its own disk space.

I also don't know why Data Goob thinks that you can backup cooked files or 
devices for IDS any differently than you can raw devices.  An archive taken 
  without using ontape or onbar is known as an 'external' archive.  In all 
three cases you can use ontape or onbar to perform online backups.  In all 
three cases if you want to produce an external archive successfully (read 
"be able to restore the data later") you have to perform the archive while 
the engine is not modifying the disk contents.  That means one of three 
scenarios 1, 2a, or 2b:

1) Shutdown the engine and use dd, tar, Legato Networker, or whatever to 
copy the chunk files to tape/CD/DVD/whatever.
2) Run onmode -c block to force a hard checkpoint and block the server from 
modifying anything until you're backup is complete.  Then you either:
    a) Back up the chunks as in 1) or
    b) Break mirrors, onmode -c unblock, backup the mirrors, recreate the 
mirrors.

UNDER NO CIRCUMSTANCES can you successfully archive the underlying chunk 
files - whether they are RAW, COOKED, or files - while the IDS engine is 
actively processing transactions!  For an OLTP or OLTP-like server, you 
could not reliably successfully restore the resulting archive.  AN 
unreliable archive is no archive at all!

Sure if you're running a DW server and nothing's been modified in a week and 
there was a recent hard checkpoint (or you forced one) you'll probably luck 
out and get a usable external backup without blocking or shutting down the 
engine.

Art S. Kagel

> i am an linux newbie.
> I must shift our Databases from SCO Unix  (IDS 7.31 UD4) to SuSE Linux
> Enterprise Edition 8.1 (2.4.21-190-smp glibc 2.2.5) or (8.2) or 9 and SuSE
> Linux Professional 9.3 (2.6.11.4 glibc 2.3.4) (IDS 7.31 UD8).
> 
> Later on i want to upgrade (in place) to IDS 10.0.
> Which Version of SuSE Linux Enterprise Edition must I use (8.? , 9 ..)?
> 
> Under Sco Unix we use raw-devices.
> Is it better to use cooked files or raw files under these Linux-Versions?
> A colleague mentioned, it can be, that later releases of Linux don't support
> raw-devices. Is that correct?
> 
> Experiences or comments are welcome.
> Thanks in advance.
> (Excuse my poor school-english)
> 
> Heinz
> 
> sending to informix-list
0
Art
5/11/2005 7:30:22 PM
Art S. Kagel wrote:
> Heinz Weitkamp wrote:
> 
>> Hi all,
> 
> 
> Go RAW all the way.  Contrary to the Goob's understanding my own testing 
> shows RAW devices to be 15-25% faster than COOKED devices which are in 
> turn 10-15% faster than filesystem files, even on the best filesystem 
> implementations that adds up to a 25% improvement.  Well worth any minor 
> inconveniences in my book. 

OK so you get some speed increases, but also the additional management
of linking raw files.   The original post indicated he was a newbie.
Raw files are not necessarily difficult, but they ARE optional and not
something I would entrust to a newbie.  It's a great learning experience,
but who's paying for it??

But another point to make is that in many SAN environments, raw files are
a complete waste of time, with the data already being spread out over many
drives, and, with speeds typically in the 10-15K rpm range, or even faster,
you get speeds today on cooked files that are faster than raw was even a
couple of years ago.

Another point, if **YOU** are the one managing the system, do you
want to create extra work for yourself, and extra risk if you don't
have to or don't have the experience to work with them?  In a
lot of shops where ONE DBA is the rule, dear god, make sure you know
what to do with raw systems before taking on the additional management.
( Your tips below illustrate the point )

Do you know what to do to restore?  This will typically happen right
as you were about to go on your vacation, so be prepared.

Lastly, many environments change so rapidly that implementing raw
disks can actually impede repurposing or modifying architectures
at the pace necessary for the environment/business.  To be sure you
can argue all my points to the opposing view, but the bottom line is to
do what is right for your environment and the best overall risk for you
and your career.  I would argue RAW is great if you can, but I'd argue
more that it isn't worth the risk in most circumstances, even if you
can.

If you want to assume the risks, go for it, making doggone sure you
do the backups, and perform an actual restore test to your satisfaction,
something very few DBAs are willing to do.  You will be quite surprised
what your options are at restore-time, make sure you communicate them
to management ahead of a disaster--you **will** be surprised what you can
and cannot do if you never do a restore test--reading documentation and
never restoring at least once is begging the boss to fire you for
incompetence.  Of course with the almost zero number of Informix people
available to replace you, your job is probably pretty secure even if you
screw it up.    :-)



  I have not yet tested Linux RAW devices
> versus the various filesystems that Linux directly supports, but early 
> reports when Linux RAW devices were first rolled out showed similar 
> results.  The performance gain was one reason why Linus Torvalds was 
> convinced finally to include RAW devices in Linux, a development he had 
> resisted for years.  The other major reason being the need to provide 
> high speed low level IO capability to software like IDS that manages its 
> own disk space.
> 
> I also don't know why Data Goob thinks that you can backup cooked files 
> or devices for IDS any differently than you can raw devices.  An archive 
> taken  without using ontape or onbar is known as an 'external' archive.  
> In all three cases you can use ontape or onbar to perform online 
> backups.  In all three cases if you want to produce an external archive 
> successfully (read "be able to restore the data later") you have to 
> perform the archive while the engine is not modifying the disk 
> contents.  That means one of three scenarios 1, 2a, or 2b:
> 
> 1) Shutdown the engine and use dd, tar, Legato Networker, or whatever to 
> copy the chunk files to tape/CD/DVD/whatever.
> 2) Run onmode -c block to force a hard checkpoint and block the server 
> from modifying anything until you're backup is complete.  Then you either:
>    a) Back up the chunks as in 1) or
>    b) Break mirrors, onmode -c unblock, backup the mirrors, recreate the 
> mirrors.

Certainly doable for a noob.

:-)

I learn something new every day, but this is not for newbies.  Even experienced
people will be challenged, especially if they don't do this every day.  Knowing
the current IT market and the talent pool you will not find many people who can
do this, or would want to, thusly, they will opt for cooked--it's easier on the
blood pressure and the family.

> 
> UNDER NO CIRCUMSTANCES can you successfully archive the underlying chunk 
> files - whether they are RAW, COOKED, or files - while the IDS engine is 
> actively processing transactions!  For an OLTP or OLTP-like server, you 
> could not reliably successfully restore the resulting archive.  AN 
> unreliable archive is no archive at all!
> 
> Sure if you're running a DW server and nothing's been modified in a week 
> and there was a recent hard checkpoint (or you forced one) you'll 
> probably luck out and get a usable external backup without blocking or 
> shutting down the engine.
> 
> Art S. Kagel
> 
>> i am an linux newbie.
>> I must shift our Databases from SCO Unix  (IDS 7.31 UD4) to SuSE Linux
>> Enterprise Edition 8.1 (2.4.21-190-smp glibc 2.2.5) or (8.2) or 9 and 
>> SuSE
>> Linux Professional 9.3 (2.6.11.4 glibc 2.3.4) (IDS 7.31 UD8).
>>
>> Later on i want to upgrade (in place) to IDS 10.0.
>> Which Version of SuSE Linux Enterprise Edition must I use (8.? , 9 ..)?
>>
>> Under Sco Unix we use raw-devices.
>> Is it better to use cooked files or raw files under these Linux-Versions?
>> A colleague mentioned, it can be, that later releases of Linux don't 
>> support
>> raw-devices. Is that correct?
>>
>> Experiences or comments are welcome.
>> Thanks in advance.
>> (Excuse my poor school-english)
>>
>> Heinz
>>
>> sending to informix-list

0
Data
5/12/2005 1:08:53 AM
On Wed, 11 May 2005 21:08:53 -0400, Data Goob <datagoob@netscape.net>
wrote:

>Art S. Kagel wrote:
>> Heinz Weitkamp wrote:
>> 
>>> Hi all,
>> 
>> 
>> Go RAW all the way.  Contrary to the Goob's understanding my own testing 
>> shows RAW devices to be 15-25% faster than COOKED devices which are in 
>> turn 10-15% faster than filesystem files, even on the best filesystem 
>> implementations that adds up to a 25% improvement.  Well worth any minor 
>> inconveniences in my book. 
>
>OK so you get some speed increases, but also the additional management
>of linking raw files.   The original post indicated he was a newbie.
>Raw files are not necessarily difficult, but they ARE optional and not
>something I would entrust to a newbie.  It's a great learning experience,
>but who's paying for it??
>

All it takes is for someone with the correct access to delete those
"large" files out there to save space.  

>But another point to make is that in many SAN environments, raw files are
>a complete waste of time, with the data already being spread out over many
>drives, and, with speeds typically in the 10-15K rpm range, or even faster,
>you get speeds today on cooked files that are faster than raw was even a
>couple of years ago.
>
>Another point, if **YOU** are the one managing the system, do you
>want to create extra work for yourself, and extra risk if you don't
>have to or don't have the experience to work with them?  In a
>lot of shops where ONE DBA is the rule, dear god, make sure you know
>what to do with raw systems before taking on the additional management.
>( Your tips below illustrate the point )
>

Extra work?  Not too much, that I know of . . . 

>Do you know what to do to restore?  This will typically happen right
>as you were about to go on your vacation, so be prepared.
>
>Lastly, many environments change so rapidly that implementing raw
>disks can actually impede repurposing or modifying architectures
>at the pace necessary for the environment/business.  To be sure you
>can argue all my points to the opposing view, but the bottom line is to
>do what is right for your environment and the best overall risk for you
>and your career.  I would argue RAW is great if you can, but I'd argue
>more that it isn't worth the risk in most circumstances, even if you
>can.
>

Sure, just backup the cooked files, right, Goob?  Bad advice out of
the box, dude.  You'd lose all transactional integrity . . . .

>If you want to assume the risks, go for it, making doggone sure you
>do the backups, and perform an actual restore test to your satisfaction,
>something very few DBAs are willing to do.  You will be quite surprised
>what your options are at restore-time, make sure you communicate them
>to management ahead of a disaster--you **will** be surprised what you can
>and cannot do if you never do a restore test--reading documentation and
>never restoring at least once is begging the boss to fire you for
>incompetence.  Of course with the almost zero number of Informix people
>available to replace you, your job is probably pretty secure even if you
>screw it up.    :-)
>

Takes fewer Informix DBA to do the job.    8-)


0
John
5/12/2005 2:59:15 AM
Data Goob wrote:
> Art S. Kagel wrote:
> 
>> Heinz Weitkamp wrote:
>>
>>> Hi all,
>>
>>
>>
>> Go RAW all the way.  Contrary to the Goob's understanding my own 
>> testing shows RAW devices to be 15-25% faster than COOKED devices 
>> which are in turn 10-15% faster than filesystem files, even on the 
>> best filesystem implementations that adds up to a 25% improvement.  
>> Well worth any minor inconveniences in my book. 
> 
> 
> OK so you get some speed increases, but also the additional management

SOME speed increase?  You call 25-40% improvement SOME increase?  Ask your 
users if they'd like their apps to run 25-40% faster and see if they think 
it's insignificant!

> of linking raw files.   The original post indicated he was a newbie.
> Raw files are not necessarily difficult, but they ARE optional and not
> something I would entrust to a newbie.  It's a great learning experience,
> but who's paying for it??

This is NOT rocket science.  The RAW device files are created at the time 
the disk partitions are created.  All any DBA has to do is create a symbolic 
link to the RAW device (and he'd want to do that anyway so he can 
move/rename the actual file/device even if it's COOKED or a FS file) and ask 
the SA's (assuming he's not doubling as SA in which case the newbie moniker 
is probably misplaced) to set the permissions on the device and make sure 
they stay set.  POOF!  RAW chunks!  And as John points out there is FAR more 
risk that some well intentioned SA will delete those HUGE chunk files.

> But another point to make is that in many SAN environments, raw files are
> a complete waste of time, with the data already being spread out over many
> drives, and, with speeds typically in the 10-15K rpm range, or even faster,
> you get speeds today on cooked files that are faster than raw was even a
> couple of years ago.

You are completely misinformed.  We got SANs, hundreds of SANs, the very 
best SANS in existence.  We also have dozens of IDS server machines.  All 
using RAW disk because they still test out to be MUCH faster than COOKED 
devices or COOKED filesystem files, even on SANs.

It sounds like you think I'm proposing using singleton disk drives when I 
say RAW.  NOOO.  As always I promote using RAID10 everywhere for anything 
that's not absolutely trivial.  Not only that, we tend to plaid those RAID10 
arrays into massive logical devices covering many dozens of mirrored pairs 
across multiple SANs.  Either way you have to carve up those massive logical 
devices into smaller logical devices (still spread over many many spindles). 
  For servers earlier than 9.40 I have the SAs carve out 2GB chunks, for 
9.40+ they are carving several different sizes so that the DBAs have the 
flexibility to add a 2, 4, 8, or 16GB chunk where needed.  This is easy stuff.

Yes, the intricacies of EMC's SANs and VMs are a bit esoteric, and I leave 
that for the SAs, but you cannot say, "Ahh but what if the DBA is the SA?", 
because if he is, then he should know his jobs, both of them, and that part 
becomes routine very quickly.

You assert that cooked files are as fast today as RAW on SANs.  Have YOU 
tested it?  Are you comparing RAW to COOKED with O_SYNC?  Or are you 
depending on the vendor stats?  Not me, not with my reputation, user 
satisfaction, and my job at stake.

> Another point, if **YOU** are the one managing the system, do you
> want to create extra work for yourself, and extra risk if you don't
> have to or don't have the experience to work with them?  In a
> lot of shops where ONE DBA is the rule, dear god, make sure you know
> what to do with raw systems before taking on the additional management.
> ( Your tips below illustrate the point )

There's NOTHING to do with RAW devices that's different or harder than using 
COOKED devices and you save having to create filesystems if you are 
comparing to COOKED FS files instead of devices.  As to my 'tips' those are 
not tips.  That's how one MUST perform an external archive regardless of 
whether the chunks are RAW device, COOKED device, or FS files.  That's the 
same anyway.  It's your suggestion to archive externally rather than use the 
simple ontape or to integrate the Informix archive with the other system 
archives using onbar that adds more work for the DBA, not any of mine.  Did 
you READ what I wrote?

> Do you know what to do to restore?  This will typically happen right
> as you were about to go on your vacation, so be prepared.

Most likely as I sitting down to dinner at a REALLY nice place.  But, of 
course I know how to restore, so do the system admins, the operators, and 
all of the other DBAs so I'd better not get a call for something as trivial 
and routine as a restore.

> Lastly, many environments change so rapidly that implementing raw
> disks can actually impede repurposing or modifying architectures
> at the pace necessary for the environment/business.  To be sure you
> can argue all my points to the opposing view, but the bottom line is to
> do what is right for your environment and the best overall risk for you
> and your career.  I would argue RAW is great if you can, but I'd argue
> more that it isn't worth the risk in most circumstances, even if you
> can.

You're kidding right?  I discovered a month or so ago that some dense junior 
SA had set up a bunch of new IDS server machines with JBOD instead of a 
RAID10 array (Boy was I pissed) for servers that have been in production for 
almost a year (a YEAR with no mirrors - AAAAHHH).  I discovered it because I 
was seeing slow response times for writes on IDS servers that seemed to be 
tuned fine.  Do you know what it took to get all that data (many GBs) moved 
onto RAID10 arrays?  Do you know how much pain it caused?  None!  It took 
the SAs about 6 hours to copy the JBOD disk contents to a new RAID10 array 
while the server was offline, and repoint the chunk links to the new RAW 
devices and we were back online.  Could it have been done with no downtime? 
  Sure, well with only 5 minutes downtime anyway, just add the new chunks as 
IDS level mirrors, let them catch up, shutdown, swap the links between 
primary and mirror chunks, restart, and drop the mirrors.  Trivial really, 
and I suggested doing just that but we had the downtime available, so why 
not let the SAs do all the work.

> If you want to assume the risks, go for it, making doggone sure you
> do the backups, and perform an actual restore test to your satisfaction,
> something very few DBAs are willing to do.  You will be quite surprised

If you're not doing the periodic restore tests and running periodic archive 
verifies using archecker then you deserve to be fired for incompetence. 
What does that have to do with using RAW devices?  Or with how using RAW 
devices affects how one performs the archives in the first place?  And BTW, 
while I was running the overall DBA group here, for almost six years, our 
restore tests never failed and neither did any live restore.  Since OL5.08 
Informix archives have been utterly reliable.  I trust ontape archives with 
my company's data, with its business, with my own job.

> what your options are at restore-time, make sure you communicate them
> to management ahead of a disaster--you **will** be surprised what you can
> and cannot do if you never do a restore test--reading documentation and
> never restoring at least once is begging the boss to fire you for
> incompetence.  Of course with the almost zero number of Informix people
> available to replace you, your job is probably pretty secure even if you
> screw it up.    :-)

Almost zero?  Will you be in Denver?  Come see how many Informix DBAs attend 
my own sessions or Jonathan's or Lester's.  There are a lot more of us out 
there then you imagine.

>  I have not yet tested Linux RAW devices
> 
<SNIP>
>> 1) Shutdown the engine and use dd, tar, Legato Networker, or whatever 
>> to copy the chunk files to tape/CD/DVD/whatever.
>> 2) Run onmode -c block to force a hard checkpoint and block the server 
>> from modifying anything until you're backup is complete.  Then you 
>> either:
>>    a) Back up the chunks as in 1) or
>>    b) Break mirrors, onmode -c unblock, backup the mirrors, recreate 
>> the mirrors.
> 
> 
> Certainly doable for a noob.

That was my point, but you missed it.  You're the one who suggested making 
archives of the disks without using ontape or onbar, not me.  But you 
suggested that it's simpler and more reliable than ontape or onbar.  I'm, 
sorry, but those three admittedly complex options are the ONLY ways to 
reliably archive the disks underlying an IDS instance and be able to restore 
later to create a working instance.  It's not simpler than just typing 
"ontape -s -L 0" and responding 'y' when the tape's been mounted.  Are there 
shops out there that perform their archives externally?  Yes, some of the 
largest shops do that, usually by using three way mirrors and breaking off 
one of the triplets under onmode -c block.  This takes a minute or five then 
they're back online with only two way mirrors.  They can then archive 
massive amounts of disk space or just swap out disconnected drives for new 
ones and warehouse them.  Once the swapping or archiving is finished they 
simply add all the triplet drives back into the mirrors and wait for them to 
catch up so they can archive again.

Remember, I'm not recommending anyone do that, but at a massive site with 
many TB of disk there's just no other way to archive the server in any 
reasonable way.  We have servers that before they were upgraded to faster 
hardware required 22 hours to archive which means that the two tape drives 
were pretty much going 24x7 between archiving, changing tapes, and verifying 
tapes.

> :-)
> 
> I learn something new every day, but this is not for newbies.  Even 
> experienced

If you just learned about this then HOW are you archiving your drives 
without using ontape or onbar?  Let me re-re-repeat, you have to go through 
this no matter HOW you configure your chunks if you are not using ontape or 
onbar.  How do you expect to be able to restore?  Have YOU ever tested it? 
On an actively updated server?  I think not.

> people will be challenged, especially if they don't do this every day.  
> Knowing
> the current IT market and the talent pool you will not find many people 
> who can
> do this, or would want to, thusly, they will opt for cooked--it's easier 
> on the
> blood pressure and the family.
> 
>>
>> UNDER NO CIRCUMSTANCES can you successfully archive the underlying 
>> chunk files - whether they are RAW, COOKED, or files - while the IDS 
>> engine is actively processing transactions!  For an OLTP or OLTP-like 
>> server, you could not reliably successfully restore the resulting 
>> archive.  AN unreliable archive is no archive at all!
<SNIP>

I see you ignored these last two paragraphs where I explicitely stated that 
you cannot do what you propose this newbie do.  I'm sorry if I'm flaming 
you.  I only flame about once a year, but you're spreading false information 
to a newbie and asserting it's the truth.  I can't let it go unchallenged.

Art S. Kagel
0
Art
5/12/2005 6:33:52 PM
On Thu, 12 May 2005 14:33:52 -0400, "Art S. Kagel"
<kagel@bloomberg.net> wrote:

>Data Goob wrote:
>> Art S. Kagel wrote:
>> 
>
>> what your options are at restore-time, make sure you communicate them
>> to management ahead of a disaster--you **will** be surprised what you can
>> and cannot do if you never do a restore test--reading documentation and
>> never restoring at least once is begging the boss to fire you for
>> incompetence.  Of course with the almost zero number of Informix people
>> available to replace you, your job is probably pretty secure even if you
>> screw it up.    :-)
>
>Almost zero?  Will you be in Denver?  Come see how many Informix DBAs attend 
>my own sessions or Jonathan's or Lester's.  There are a lot more of us out 
>there then you imagine.
>

Well said, Art.  

As for the job security comments . . . just remember that there are
still a goodnumber of DBAs who will not be able to go to Denver.

>
>I see you ignored these last two paragraphs where I explicitely stated that 
>you cannot do what you propose this newbie do.  I'm sorry if I'm flaming 
>you.  I only flame about once a year, but you're spreading false information 
>to a newbie and asserting it's the truth.  I can't let it go unchallenged.
>
>Art S. Kagel


0
John
5/13/2005 1:48:27 AM
"Art S. Kagel" <kagel@bloomberg.net> wrote in message 
news:4283A190.9060802@bloomberg.net...

> SOME speed increase?  You call 25-40% improvement SOME increase?  Ask your 
> users if they'd like their apps to run 25-40% faster and see if they think 
> it's insignificant!

Of course, i/o rates are just one of many factors contributing to overall 
user response time, and it is unlikely in the extreme that x% improvement in 
i/o times will translate to x% improvement in response times.


0
Neil
5/13/2005 6:52:26 AM
Art,

Your points are well made, and I'm willing to admit I could be
misinformed, but quite honestly you are working in an exceptional
environment compared to a lot of environments out there.   You
make the assertion that everyone should simply get it when it
comes to disk drives, and quite honestly very few of us can keep
up with this. You also make the assumption that it is easy when
you should see what I have to work with when using sys admins--you
even allude to the jr DBA who got it wrong.  Imagine that this is
95% of the environments out there.  And as far as misinforming
people bang on EMC, they even preach their RAID5 is 'better',
so we even have to deal with that as well as a lot of other
vendor misinformation.

Also imagine how Informix is less than 3% of the market
so those of you out there that are quite comfortable with raw
files and Informix are in a very small minority.  It just ain't
like Informix-on-raw is everywhere, it isn't!  I used to preach
raw files everywhere but became weary dealing with people who
don't know the difference, or aren't knowledgable, can't do it,
etc., want to just get the system up, want to use MSSQL, etc.

Put your notes in the FAQ, I'm willing to refer people to the
FAQ not only for Informix but any other DB out there and your
time spent on this is even more valuable to the community at
large.  I already refer to your RAID5 comments, you can bet
that I'll add your notes on raw-vs-cooked to my knowledge base.
Not all of us get to work in the same kind of environment you do,
or with Informix for that matter.   You just have no idea what
some of the environments are like out there, just unbelievable.


Art S. Kagel wrote:
> Data Goob wrote:
> 
>> Art S. Kagel wrote:
>>
>>> Heinz Weitkamp wrote:
>>>
>>>> Hi all,
>>>
>>>
>>>
>>>
>>> Go RAW all the way.  Contrary to the Goob's understanding my own 
>>> testing shows RAW devices to be 15-25% faster than COOKED devices 
>>> which are in turn 10-15% faster than filesystem files, even on the 
>>> best filesystem implementations that adds up to a 25% improvement.  
>>> Well worth any minor inconveniences in my book. 
>>
>>
>>
>> OK so you get some speed increases, but also the additional management
> 
> 
> SOME speed increase?  You call 25-40% improvement SOME increase?  Ask 
> your users if they'd like their apps to run 25-40% faster and see if 
> they think it's insignificant!
> 
>> of linking raw files.   The original post indicated he was a newbie.
>> Raw files are not necessarily difficult, but they ARE optional and not
>> something I would entrust to a newbie.  It's a great learning experience,
>> but who's paying for it??
> 
> 
> This is NOT rocket science.  The RAW device files are created at the 
> time the disk partitions are created.  All any DBA has to do is create a 
> symbolic link to the RAW device (and he'd want to do that anyway so he 
> can move/rename the actual file/device even if it's COOKED or a FS file) 
> and ask the SA's (assuming he's not doubling as SA in which case the 
> newbie moniker is probably misplaced) to set the permissions on the 
> device and make sure they stay set.  POOF!  RAW chunks!  And as John 
> points out there is FAR more risk that some well intentioned SA will 
> delete those HUGE chunk files.
> 
>> But another point to make is that in many SAN environments, raw files are
>> a complete waste of time, with the data already being spread out over 
>> many
>> drives, and, with speeds typically in the 10-15K rpm range, or even 
>> faster,
>> you get speeds today on cooked files that are faster than raw was even a
>> couple of years ago.
> 
> 
> You are completely misinformed.  We got SANs, hundreds of SANs, the very 
> best SANS in existence.  We also have dozens of IDS server machines.  
> All using RAW disk because they still test out to be MUCH faster than 
> COOKED devices or COOKED filesystem files, even on SANs.
> 
> It sounds like you think I'm proposing using singleton disk drives when 
> I say RAW.  NOOO.  As always I promote using RAID10 everywhere for 
> anything that's not absolutely trivial.  Not only that, we tend to plaid 
> those RAID10 arrays into massive logical devices covering many dozens of 
> mirrored pairs across multiple SANs.  Either way you have to carve up 
> those massive logical devices into smaller logical devices (still spread 
> over many many spindles).  For servers earlier than 9.40 I have the SAs 
> carve out 2GB chunks, for 9.40+ they are carving several different sizes 
> so that the DBAs have the flexibility to add a 2, 4, 8, or 16GB chunk 
> where needed.  This is easy stuff.
> 
> Yes, the intricacies of EMC's SANs and VMs are a bit esoteric, and I 
> leave that for the SAs, but you cannot say, "Ahh but what if the DBA is 
> the SA?", because if he is, then he should know his jobs, both of them, 
> and that part becomes routine very quickly.
> 
> You assert that cooked files are as fast today as RAW on SANs.  Have YOU 
> tested it?  Are you comparing RAW to COOKED with O_SYNC?  Or are you 
> depending on the vendor stats?  Not me, not with my reputation, user 
> satisfaction, and my job at stake.
> 
>> Another point, if **YOU** are the one managing the system, do you
>> want to create extra work for yourself, and extra risk if you don't
>> have to or don't have the experience to work with them?  In a
>> lot of shops where ONE DBA is the rule, dear god, make sure you know
>> what to do with raw systems before taking on the additional management.
>> ( Your tips below illustrate the point )
> 
> 
> There's NOTHING to do with RAW devices that's different or harder than 
> using COOKED devices and you save having to create filesystems if you 
> are comparing to COOKED FS files instead of devices.  As to my 'tips' 
> those are not tips.  That's how one MUST perform an external archive 
> regardless of whether the chunks are RAW device, COOKED device, or FS 
> files.  That's the same anyway.  It's your suggestion to archive 
> externally rather than use the simple ontape or to integrate the 
> Informix archive with the other system archives using onbar that adds 
> more work for the DBA, not any of mine.  Did you READ what I wrote?
> 
>> Do you know what to do to restore?  This will typically happen right
>> as you were about to go on your vacation, so be prepared.
> 
> 
> Most likely as I sitting down to dinner at a REALLY nice place.  But, of 
> course I know how to restore, so do the system admins, the operators, 
> and all of the other DBAs so I'd better not get a call for something as 
> trivial and routine as a restore.
> 
>> Lastly, many environments change so rapidly that implementing raw
>> disks can actually impede repurposing or modifying architectures
>> at the pace necessary for the environment/business.  To be sure you
>> can argue all my points to the opposing view, but the bottom line is to
>> do what is right for your environment and the best overall risk for you
>> and your career.  I would argue RAW is great if you can, but I'd argue
>> more that it isn't worth the risk in most circumstances, even if you
>> can.
> 
> 
> You're kidding right?  I discovered a month or so ago that some dense 
> junior SA had set up a bunch of new IDS server machines with JBOD 
> instead of a RAID10 array (Boy was I pissed) for servers that have been 
> in production for almost a year (a YEAR with no mirrors - AAAAHHH).  I 
> discovered it because I was seeing slow response times for writes on IDS 
> servers that seemed to be tuned fine.  Do you know what it took to get 
> all that data (many GBs) moved onto RAID10 arrays?  Do you know how much 
> pain it caused?  None!  It took the SAs about 6 hours to copy the JBOD 
> disk contents to a new RAID10 array while the server was offline, and 
> repoint the chunk links to the new RAW devices and we were back online.  
> Could it have been done with no downtime?  Sure, well with only 5 
> minutes downtime anyway, just add the new chunks as IDS level mirrors, 
> let them catch up, shutdown, swap the links between primary and mirror 
> chunks, restart, and drop the mirrors.  Trivial really, and I suggested 
> doing just that but we had the downtime available, so why not let the 
> SAs do all the work.
> 
>> If you want to assume the risks, go for it, making doggone sure you
>> do the backups, and perform an actual restore test to your satisfaction,
>> something very few DBAs are willing to do.  You will be quite surprised
> 
> 
> If you're not doing the periodic restore tests and running periodic 
> archive verifies using archecker then you deserve to be fired for 
> incompetence. What does that have to do with using RAW devices?  Or with 
> how using RAW devices affects how one performs the archives in the first 
> place?  And BTW, while I was running the overall DBA group here, for 
> almost six years, our restore tests never failed and neither did any 
> live restore.  Since OL5.08 Informix archives have been utterly 
> reliable.  I trust ontape archives with my company's data, with its 
> business, with my own job.
> 
>> what your options are at restore-time, make sure you communicate them
>> to management ahead of a disaster--you **will** be surprised what you can
>> and cannot do if you never do a restore test--reading documentation and
>> never restoring at least once is begging the boss to fire you for
>> incompetence.  Of course with the almost zero number of Informix people
>> available to replace you, your job is probably pretty secure even if you
>> screw it up.    :-)
> 
> 
> Almost zero?  Will you be in Denver?  Come see how many Informix DBAs 
> attend my own sessions or Jonathan's or Lester's.  There are a lot more 
> of us out there then you imagine.
> 
>>  I have not yet tested Linux RAW devices
>>
> <SNIP>
> 
>>> 1) Shutdown the engine and use dd, tar, Legato Networker, or whatever 
>>> to copy the chunk files to tape/CD/DVD/whatever.
>>> 2) Run onmode -c block to force a hard checkpoint and block the 
>>> server from modifying anything until you're backup is complete.  Then 
>>> you either:
>>>    a) Back up the chunks as in 1) or
>>>    b) Break mirrors, onmode -c unblock, backup the mirrors, recreate 
>>> the mirrors.
>>
>>
>>
>> Certainly doable for a noob.
> 
> 
> That was my point, but you missed it.  You're the one who suggested 
> making archives of the disks without using ontape or onbar, not me.  But 
> you suggested that it's simpler and more reliable than ontape or onbar.  
> I'm, sorry, but those three admittedly complex options are the ONLY ways 
> to reliably archive the disks underlying an IDS instance and be able to 
> restore later to create a working instance.  It's not simpler than just 
> typing "ontape -s -L 0" and responding 'y' when the tape's been 
> mounted.  Are there shops out there that perform their archives 
> externally?  Yes, some of the largest shops do that, usually by using 
> three way mirrors and breaking off one of the triplets under onmode -c 
> block.  This takes a minute or five then they're back online with only 
> two way mirrors.  They can then archive massive amounts of disk space or 
> just swap out disconnected drives for new ones and warehouse them.  Once 
> the swapping or archiving is finished they simply add all the triplet 
> drives back into the mirrors and wait for them to catch up so they can 
> archive again.
> 
> Remember, I'm not recommending anyone do that, but at a massive site 
> with many TB of disk there's just no other way to archive the server in 
> any reasonable way.  We have servers that before they were upgraded to 
> faster hardware required 22 hours to archive which means that the two 
> tape drives were pretty much going 24x7 between archiving, changing 
> tapes, and verifying tapes.
> 
>> :-)
>>
>> I learn something new every day, but this is not for newbies.  Even 
>> experienced
> 
> 
> If you just learned about this then HOW are you archiving your drives 
> without using ontape or onbar?  Let me re-re-repeat, you have to go 
> through this no matter HOW you configure your chunks if you are not 
> using ontape or onbar.  How do you expect to be able to restore?  Have 
> YOU ever tested it? On an actively updated server?  I think not.
> 
>> people will be challenged, especially if they don't do this every 
>> day.  Knowing
>> the current IT market and the talent pool you will not find many 
>> people who can
>> do this, or would want to, thusly, they will opt for cooked--it's 
>> easier on the
>> blood pressure and the family.
>>
>>>
>>> UNDER NO CIRCUMSTANCES can you successfully archive the underlying 
>>> chunk files - whether they are RAW, COOKED, or files - while the IDS 
>>> engine is actively processing transactions!  For an OLTP or OLTP-like 
>>> server, you could not reliably successfully restore the resulting 
>>> archive.  AN unreliable archive is no archive at all!
> 
> <SNIP>
> 
> I see you ignored these last two paragraphs where I explicitely stated 
> that you cannot do what you propose this newbie do.  I'm sorry if I'm 
> flaming you.  I only flame about once a year, but you're spreading false 
> information to a newbie and asserting it's the truth.  I can't let it go 
> unchallenged.
> 
> Art S. Kagel

0
Data
5/13/2005 6:58:46 AM
John Carlson wrote:
> On Thu, 12 May 2005 14:33:52 -0400, "Art S. Kagel"
> <kagel@bloomberg.net> wrote:
> 
> 
>>Data Goob wrote:
>>
>>>Art S. Kagel wrote:
<SNIP>>
> As for the job security comments . . . just remember that there are
> still a goodnumber of DBAs who will not be able to go to Denver.

Truth.  This is the first time Bloomberg is allowing me to attend, and I'm 
here twelve years.

And your main point is a good one - that as many as attend the conference 
there are probably a few times that number who can not, will not, or simply 
do not attend.  Heck, I'm the only one attending from my company, none of 
the Informix or DB2 DBAs will be joining me.

As to being able to replace us ... when I asked to get back to coding most 
of the time Bloomberg was able to find, train, and hire nine DBAs and a 
manager to replace me.  That's more a statement of how easy it was to find 
qualified people [and to how dumb I was for working so hard and so quietly 
that no one realized that I could have used eight more people on my team] 
than it is a statement as to whether it really takes nine people to replace me.

Art S. Kagel
0
Art
5/13/2005 2:19:01 PM
Data Goob wrote:
> Art,
> 
> Your points are well made, and I'm willing to admit I could be
> misinformed, but quite honestly you are working in an exceptional
> environment compared to a lot of environments out there.   You
> make the assertion that everyone should simply get it when it
> comes to disk drives, and quite honestly very few of us can keep
> up with this. You also make the assumption that it is easy when
> you should see what I have to work with when using sys admins--you
> even allude to the jr DBA who got it wrong.  Imagine that this is
> 95% of the environments out there.  And as far as misinforming
> people bang on EMC, they even preach their RAID5 is 'better',
> so we even have to deal with that as well as a lot of other
> vendor misinformation.

There actually is one implementation of RAID5 that stands up to testing
as far superior to others and that is Apple Computer's XServe RAID.
These units have two XOR engines and come remarkably close in
performance to 0+1 (within a few percentage points) and also have the
advantage of having minimal loss of performance when running in degraded
mode. So far this year most of the storage I have worked with has been
Apple's and it is a poorly kept secret that Oracle Corp. is now using
them in their data center in Redwood Shores.
-- 
Daniel A. Morgan
University of Washington
damorgan@x.washington.edu
(replace 'x' with 'u' to respond)
0
DA
5/13/2005 2:26:39 PM
Data Goob wrote:
> Art,
> 
> Your points are well made, and I'm willing to admit I could be
> misinformed, but quite honestly you are working in an exceptional
> environment compared to a lot of environments out there.   You

No argument.  I am blessed.

> make the assertion that everyone should simply get it when it
> comes to disk drives, and quite honestly very few of us can keep

Actually, while I STRONGLY recommend RAW disk and RAID10, I do not expect 
anyone to 'get it'.  Indeed, every single time I post on either the subject 
of RAW-vs-COOKED or RAID5-vs-RAID10 I ALWAYS recommend MOST strongly that 
everyone test it for themselves, because, as you correctly point out, every 
environment is different.  Even if a particular reader's RAID5 or COOKED 
FILE implementation is no better than the ones I've tested, their RAID10 or 
RAW implementation could be FAR WORSE.  Nope, don't expect anyone to get 
anything except my own credo: Test it for yourself!  Don't believe what's 
touted by the vendors, the Data Goob ;-), nor by me.

> up with this. You also make the assumption that it is easy when
> you should see what I have to work with when using sys admins--you

It IS easy.  But as I am fond of saying (until everyone around me is staring 
at the ceiling and shuffling their feet) "The problem with making things 
foolproof is that fools are so devious!".  The reason that the newbie SA 
(not DBA) didn't allocate a RAID10 array for the new machines is because 
noone told him he should and the senior SAs didn't realize that they hadn't 
clued him in.

> even allude to the jr DBA who got it wrong.  Imagine that this is
> 95% of the environments out there.  And as far as misinforming
> people bang on EMC, they even preach their RAID5 is 'better',
> so we even have to deal with that as well as a lot of other
> vendor misinformation.

Ya, I know.  But their 'better' RAID5 came from the DG Clariion purchase and 
it was those arrays that we were comparing RAID5 to RAID10 on originally!

> Also imagine how Informix is less than 3% of the market
> so those of you out there that are quite comfortable with raw
> files and Informix are in a very small minority.  It just ain't
> like Informix-on-raw is everywhere, it isn't!  I used to preach
> raw files everywhere but became weary dealing with people who
> don't know the difference, or aren't knowledgable, can't do it,
> etc., want to just get the system up, want to use MSSQL, etc.

Right, Oracle for instance is less affected

> Put your notes in the FAQ, I'm willing to refer people to the
> FAQ not only for Informix but any other DB out there and your
> time spent on this is even more valuable to the community at
> large.  I already refer to your RAID5 comments, you can bet
> that I'll add your notes on raw-vs-cooked to my knowledge base.
> Not all of us get to work in the same kind of environment you do,
> or with Informix for that matter.   You just have no idea what
> some of the environments are like out there, just unbelievable.
> 
<SNIP>
This is in the FAQ.  Here's an extract:

....the safety issue of cooked files is no longer a problem. The big problem 
with cooked files still is performance. All writes and reads to/from cooked 
files MUST go through the UNIX buffer cache. This means an additional copy 
from the server's output buffer to the UNIX buffer page then a synchronous 
write to disk. This is opposed to a write to a way file where the server's 
output buffer is written directly to disk without the intervening copy. This 
is just faster. Anyone who has written anything that can test this can 
attest to the difference in speed. Here are my test results:

FileType	Sync?	Times (real/user/system) 2run avg
---------------	-----	-----------------------------------------
Filesystem file	  N     14.40/3.70/2.52
		  Y	15.02/3.61/2.63
Cooked disk	  N	12.81/3.74/2.24
		  Y	13.42/3.84/2.43
Raw disk	  N	 9.32/3.67/1.52
		  Y	 9.40/3.66/1.44

 From this you can clearly see the cost of Cooked files and of synced cooked 
files. The tests were done with a version of my ul.ec utility modified to 
optionally open the output file O_SYNC. Cooked disk partition is almost 50% 
slower than raw disk partition and cooked filesystem files are almost 60% 
slower. The penalty for O_SYNC is an additional 5% for cooked files and 
negligible for RAW files (as expected). The test file was 2.85MB written 
using 4K I/O pages (the default fopen application buffer size) which should 
simulate Informix performance. The Cooked and Raw disk partition tests were 
conducted to a singleton 9GB Fast Wide SCSI II drive using the raw and 
cooked device files corresponding to the same drive.


0
Art
5/13/2005 3:44:28 PM
Neil Truby wrote:
> "Art S. Kagel" <kagel@bloomberg.net> wrote in message 
> news:4283A190.9060802@bloomberg.net...
> 
> 
>>SOME speed increase?  You call 25-40% improvement SOME increase?  Ask your 
>>users if they'd like their apps to run 25-40% faster and see if they think 
>>it's insignificant!
> 
> 
> Of course, i/o rates are just one of many factors contributing to overall 
> user response time, and it is unlikely in the extreme that x% improvement in 
> i/o times will translate to x% improvement in response times.
> 
> 
Granted Neil.  Perhaps my point was too broad to penetrate to the heart of 
what I wanted to say - so no point at all ;-(

I did not mean to imply that a 25% increase in IO speed would translate into 
a 25% increase in application throughput or response times.  My point was 
just that - just as our users would recognize a 25-40% improvement as 
something significant, recognizable, and valuable - so should we not devalue 
the impact that a 25-40% increase would have on overall server performance.

Art S. Kagel

0
Art
5/13/2005 3:45:22 PM
DA Morgan wrote:
> Data Goob wrote:
> 
>> Art,
>>
>> Your points are well made, and I'm willing to admit I could be
>> misinformed, but quite honestly you are working in an exceptional
>> environment compared to a lot of environments out there.   You
>> make the assertion that everyone should simply get it when it
>> comes to disk drives, and quite honestly very few of us can keep
>> up with this. You also make the assumption that it is easy when
>> you should see what I have to work with when using sys admins--you
>> even allude to the jr DBA who got it wrong.  Imagine that this is
>> 95% of the environments out there.  And as far as misinforming
>> people bang on EMC, they even preach their RAID5 is 'better',
>> so we even have to deal with that as well as a lot of other
>> vendor misinformation.
> 
> 
> There actually is one implementation of RAID5 that stands up to testing
> as far superior to others and that is Apple Computer's XServe RAID.
> These units have two XOR engines and come remarkably close in
> performance to 0+1 (within a few percentage points) and also have the
> advantage of having minimal loss of performance when running in degraded
> mode. So far this year most of the storage I have worked with has been
> Apple's and it is a poorly kept secret that Oracle Corp. is now using
> them in their data center in Redwood Shores.

Interesting.  Got to keep that in mind.  But my big question, which you are 
unlikely to have an answer to, is did Apple solve the partial media failure 
problem in RAID5?

Art S. Kagel
0
Art
5/13/2005 5:24:41 PM
Reply:

Similar Artilces:

IDS 10 on RHEL4 Linux: I/O benchmark of RAW DEVICES vs BLOCK DEVICES vs COOKED FILES
Hello there, this is a long message (sorry in advance) which contains some hopefully interesting information. There are also some questions at the end. I have this nice new Red Hat Enterprise Linux 4 machine, it's an HP DL 385 with 8 internal 10K RPM SAS disks, 2 dual-core AMD Opteron CPUs and 4 GB RAM. Machine features 256 Mb of cache on the disks controller. Here it is: http://tinyurl.com/p6k6y I have installed Informix Dynamic Server 10.00.FC4 (64 bit). I bumped into this IBM article: http://tinyurl.com/gfovb, where it says that Informix IDS 10 on RHEL4 Linux is able to access block devices in unbuffered fashion -like it does for the raw devices- using KAIO (Kernel Asynchronous I/O). The server is brand new, and I can play with it a bit before it goes into production, so I decided to do some performance testing of RAW vs BLOCK vs COOKED, to see which one was better from the I/O performance point of view. Some definitions, first: * RAW DEVICE or CHARACTER DEVICE = devices through which data is transmitted one character at a time, using unbuffered input and output routines; each character is read from, or written to, the device immediately; such devices are disk partitions without a file system on them, and are therefore managed from outside the operating system using direct access and bypassing the OS layer and its cache * BLOCK DEVICE = devices through which data is transmitted in the form of blocks, the most sign...

AW: raw vs. cooked files under linux
Thank you very much, Jean and Sandor, for your quick answers. One question more: I saw in the List on link: http://www-306.ibm.com/software/data/informix/linux/ids.html that IDS 7.31.UD8 don't support SUSE SLES9. Do you know, whether IDS 7.31.UD8 works under SUSE SLES9, because first i want to install IDS 7.31.UD8 on Linux and later IDS 10.0 without (if possible) changing the Linux Version? Thanks. Heinz -----Urspr�ngliche Nachricht----- Von: Sandor Szabo [mailto:sandor.szabo@de.ibm.com] Gesendet: Mittwoch, 11. Mai 2005 13:31 An: Heinz Weitkamp Cc: user-group Informix (E-Mail); owner-informix-list@iiug.org Betreff: Re: raw vs. cooked files under linux Dear Heinz, I would suggest that you pick SUSE SLES 9 This SUSE version works perfectly with IDS 10. Please check the following link: http://www-306.ibm.com/software/data/informix/linux/ids.html With IDS 10 you have the freedom of choice to use cooked or raw devices with KAIO This link should answer your raw device support question http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0503szabo/? ca=dgr-lnxw41IDSTen Let me know if you have any additional questions. bye Sandor IBM Informix Development Munich Information Management Hollerithstra�e 1 81829 Munich Germany Phone: +49 89 4504 1429 "Heinz Weitkamp" <heinz.weitkamp@w estfleisch.de> ...

Re: raw vs. cooked files under linux
SCO -> Linux ... good ;) If you want to upgrade to IDS 10.0 then better use Suse 9 'cause IDS use Kernel 2.6 specific features like KAIO. There is a nice document on IDS 10 and Linux on IBM site (can't remember link...). Regarding cooked vs. raw devices O bet for raw devices, they are very easy to configure in linux and give you some benefits like KAIO for example. J. Heinz Weitkamp escribi�: > Hi all, > > i am an linux newbie. > I must shift our Databases from SCO Unix (IDS 7.31 UD4) to SuSE Linux > Enterprise Edition 8.1 (2.4.21-190-smp glibc 2.2.5) or (8.2) or 9 and SuSE > Linux Professional 9.3 (2.6.11.4 glibc 2.3.4) (IDS 7.31 UD8). > > Later on i want to upgrade (in place) to IDS 10.0. > Which Version of SuSE Linux Enterprise Edition must I use (8.? , 9 ..)? > > Under Sco Unix we use raw-devices. > Is it better to use cooked files or raw files under these Linux-Versions? > A colleague mentioned, it can be, that later releases of Linux don't support > raw-devices. Is that correct? > > Experiences or comments are welcome. > Thanks in advance. > (Excuse my poor school-english) > > Heinz > > sending to informix-list > sending to informix-list ...

Re: raw vs. cooked files under linux #2
Dear Heinz, I would suggest that you pick SUSE SLES 9 This SUSE version works perfectly with IDS 10. Please check the following link: http://www-306.ibm.com/software/data/informix/linux/ids.html With IDS 10 you have the freedom of choice to use cooked or raw devices with KAIO This link should answer your raw device support question http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0503szabo/?ca=dgr-lnxw41IDSTen Let me know if you have any additional questions. bye Sandor IBM Informix Development Munich Information Management Hollerithstra�e 1 81829 Munich Germany Phone: +49 89 4504 1429 "Heinz Weitkamp" <heinz.weitkamp@w estfleisch.de> To Sent by: "user-group Informix \(E-Mail\)" owner-informix-li <informix-list@iiug.org> st@iiug.org cc Subject 11.05.2005 10:48 raw vs. cooked files under linux ...

RE: raw vs. cooked files under linux #5
We use raw chunks on all our unix versions. Most are Linux. We find it easier to set up and manage raw devices, including under Linux. We also have a SAN environment running RAW devices under Linux - no problems at all. We have not had any issues and that includes recovering systems. Its all very simple when using RAW devices. MW > -----Original Message----- > From: owner-informix-list@iiug.org > [mailto:owner-informix-list@iiug.org] On Behalf Of Data Goob > Sent: Thursday, 12 May 2005 1:09 p.m. > To: informix-list@iiug.org > Subject: Re: raw vs. cooked files under linux > > Art S. Kagel wrote: > > Heinz Weitkamp wrote: > > > >> Hi all, > > > > > > Go RAW all the way. Contrary to the Goob's understanding my own > > testing shows RAW devices to be 15-25% faster than COOKED devices > > which are in turn 10-15% faster than filesystem files, even on the > > best filesystem implementations that adds up to a 25% improvement. > > Well worth any minor inconveniences in my book. > > OK so you get some speed increases, but also the additional management > of linking raw files. The original post indicated he was a newbie. > Raw files are not necessarily difficult, but they ARE optional and not > something I would entrust to a newbie. It's a great learning > experience, but who's paying for it?? > > But anot...

RE: raw vs. cooked files under linux #3
We're about to setup a new client on IDS 10 on Linux and I've now seen two recommendations for SUSE in relation to IDS 10. Does anyone have experience with Redhat Enterprise Edition and IDS 10? Any issues there that we should be aware of? Bill Weaver Director of Engineering Amicus, Inc. 512-531-3463 (office) 512-531-3401 (fax) -----Original Message----- From: owner-informix-list@iiug.org [mailto:owner-informix-list@iiug.org] On Behalf Of Sandor Szabo Sent: Wednesday, May 11, 2005 6:31 AM To: Heinz Weitkamp Cc: user-group Informix (E-Mail); owner-informix-list@iiug.org Subject: Re: raw vs. cooked files under linux Dear Heinz, I would suggest that you pick SUSE SLES 9 This SUSE version works perfectly with IDS 10. Please check the following link: http://www-306.ibm.com/software/data/informix/linux/ids.html With IDS 10 you have the freedom of choice to use cooked or raw devices with KAIO This link should answer your raw device support question http://www-128.ibm.com/developerworks/db2/library/techarticle/dm-0503szabo/? ca=dgr-lnxw41IDSTen Let me know if you have any additional questions. bye Sandor IBM Informix Development Munich Information Management Hollerithstra�e 1 81829 Munich Germany Phone: +49 89 4504 1429 "Heinz Weitkamp" <heinz.wei...

Re: raw vs. cooked files under linux #10
DA Morgan said: > Data Goob wrote: >> Art, >> >> Your points are well made, and I'm willing to admit I could be >> misinformed, but quite honestly you are working in an exceptional >> environment compared to a lot of environments out there. You >> make the assertion that everyone should simply get it when it >> comes to disk drives, and quite honestly very few of us can keep >> up with this. You also make the assumption that it is easy when >> you should see what I have to work with when using sys admins--you >> even allude to the jr DBA who got it wrong. Imagine that this is >> 95% of the environments out there. And as far as misinforming >> people bang on EMC, they even preach their RAID5 is 'better', >> so we even have to deal with that as well as a lot of other >> vendor misinformation. > > There actually is one implementation of RAID5 that stands up to testing > as far superior to others and that is Apple Computer's XServe RAID. > These units have two XOR engines and come remarkably close in > performance to 0+1 (within a few percentage points) and also have the > advantage of having minimal loss of performance when running in degraded > mode. So far this year most of the storage I have worked with has been > Apple's and it is a poorly kept secret that Oracle Corp. is now using > them in their data center in...

Re: raw vs. cooked files under linux #4
I guess almost all of us are convinced at this point of the performance improvement from using raw devices. But it would be great if we could get a benchmark report with a couple of Filesystems, specially on Linux, just in case we really need to use Filesystems. (JFS, Ext3, XFS , ReiserFS, ... ) I am a fan of XFS on Linux, because of its speed/stability/reliability against the other filesystems. I would be glad to see if XFS is better for database-OLTP performance or not. Could someone point us to a benchmark report of filesystems/raw devices ? Best Regards ----- Original Message ----- From: "Art S. Kagel" <kagel@bloomberg.net> To: <informix-list@iiug.org> Sent: Wednesday, May 11, 2005 1:30 PM Subject: Re: raw vs. cooked files under linux > Heinz Weitkamp wrote: > > Hi all, > > Go RAW all the way. Contrary to the Goob's understanding my own testing > shows RAW devices to be 15-25% faster than COOKED devices which are in turn > 10-15% faster than filesystem files, even on the best filesystem > implementations that adds up to a 25% improvement. Well worth any minor > inconveniences in my book. I have not yet tested Linux RAW devices versus > the various filesystems that Linux directly supports, but early reports when > Linux RAW devices were first rolled out showed similar results. The > performance gain was one reason why Linus Torvalds was convinced finally to > ...

RE: raw vs. cooked files under linux #9
-----Original Message----- From: owner-informix-list@iiug.org [mailto:owner-informix-list@iiug.org] On Behalf Of John Carlson Sent: 13 May 2005 03:48 AM > >Almost zero? Will you be in Denver? Come see how many Informix DBAs attend >my own sessions or Jonathan's or Lester's. There are a lot more of us out >there then you imagine. > [snip] >As for the job security comments . . . just remember that there are >still a goodnumber of DBAs who will not be able to go to Denver. True, my company will only allow me to go places I can drive to (short distance) - very sad ........ sending to informix-list ...

Re: raw vs. cooked files under linux #8
How does one use O_SYNC with and ext3 filesystem on Linux? I don't see it mentioned anywhere. Normally I use raw but SA and management would rather use cooked. They would also rather use Postgres but that another story. Thanks >You assert that cooked files are as fast today as RAW on SANs. Have YOU >tested it? Are you comparing RAW to COOKED with O_SYNC? Or are you >depending on the vendor stats? Not me, not with my reputation, user >satisfaction, and my job at stake. > > sending to informix-list Peter J Diaz de Leon wrote: > How does one use O_SYNC with and ext3 filesystem on Linux? I don't see > it mentioned anywhere. Normally I use raw but SA and management > would rather use cooked. They would also rather use Postgres but that > another story. > Probably the same way one does for any filesystem, by adding the O_SUNC flag to the oflag argument to the open() system call. So: out_handle = open( "/myfs/mychunkfile", O_WRONLY | O_SYNC ); But isn't ext3 a journaled filesystem? One would first off, not want to use a journaled filesystem for IDS chunks for several reasons: 1- Performance, the overhead of writing the journal records 2- No gain for the pain. You can NEVER use the journal to reproduce the file because it's state would always be to the moment. If the disk crashes you'll have to restore from the IDS archives anyway. Art S. Kag...

RE: raw vs. cooked files under linux #6
When you compare cooked versus raw here, are you focusing on Linux only, or talking about this in general - other Unix platforms as well ? -----Original Message----- From: owner-informix-list@iiug.org [mailto:owner-informix-list@iiug.org] On Behalf Of Data Goob Sent: 11 May 2005 03:08 PM To: informix-list@iiug.org Subject: Re: raw vs. cooked files under linux If you are in a slow-change environment raw-files are not a bad thing. They create extra problems on the management side that cooked files do not. Fast-changing environments create more demand on your time so why put yourself into extra work with raw files? They don't give you enough of a difference in performance to really warrant them unless you are really demanding that extra 10%. Cooked files are also also more flexible on backups, you can back the dbspaces up with a regular backup tool or use the Informix backup tools, but raw files will have to be backed up with Informix tools only. SLES9 is quite a sea change from SLES8, and you could probably get away without SLES and go with the workstation Pro 9.3 if you are on a budget. SLES has better support for SCSI than the workstation versions and you must buy support or the online update doesn't work. It's an enterprise lock-in you might not need so try before you buy. SLES9 can be downloaded for trial, so give it a spin. You should be able to download IDS 10 for trial as well, thus no out of pocket to try it before pay...

Re: raw vs. cooked files under linux #7
We are running 24 instances of IDS 9.4 on 24 machines - about 95% raw chunks - no problems so far -Can you explain some of your reasoning below? What specifically are the risks and additional management? ----- Original Message ----- From: "Data Goob" <datagoob@netscape.net> To: <informix-list@iiug.org> Sent: Wednesday, May 11, 2005 9:08 PM Subject: Re: raw vs. cooked files under linux > Art S. Kagel wrote: > > Heinz Weitkamp wrote: > > > >> Hi all, > > > > > > Go RAW all the way. Contrary to the Goob's understanding my own testing > > shows RAW devices to be 15-25% faster than COOKED devices which are in > > turn 10-15% faster than filesystem files, even on the best filesystem > > implementations that adds up to a 25% improvement. Well worth any minor > > inconveniences in my book. > > OK so you get some speed increases, but also the additional management > of linking raw files. The original post indicated he was a newbie. > Raw files are not necessarily difficult, but they ARE optional and not > something I would entrust to a newbie. It's a great learning experience, > but who's paying for it?? > > But another point to make is that in many SAN environments, raw files are > a complete waste of time, with the data already being spread out over many > drives, and, with speeds typically in the 10-15K rpm ran...

Raw vs. Cooked files
Hello, I am installing IDS 10 on RHEL3 on an intel x86 server. I need to make the choice between raw devices and cooked files. I found the following information in the FAQ's. Is this information still accurate? Do cooked files continue to be significantly slower than the raw files? I have heard that UNIX file systems have improved over the years and the performance divide between raw and cooked files has narrowed considerabley. Is it true? Thanks! - Rajesh -------------------------------------------------------------------------------------- 6.38 Is raw disk faster than cooked files? On 22nd Jun 1998 kagel@bloomberg.net (Art S. Kagel) wrote:- .....................the safety issue of cooked files is no longer a problem. The big problem with cooked files still is performance. All writes and reads to/from cooked files MUST go through the UNIX buffer cache. This means an additional copy from the server's output buffer to the UNIX buffer page then a synchronous write to disk. This is opposed to a write to a way file where the server's output buffer is written directly to disk without the intervening copy. This is just faster. Anyone who has written anything that can test this can attest to the difference in speed. Here are my test results: FileType Sync? Times (real/user/system) 2run avg --------------- ----- ----------------------------------------- Filesystem file N 14.40/3.70/2.52 Y 15.02/3.61/2.63 Cooked ...

Re: Raw vs. Cooked files
Yes even with DB2 UDB you can get 20-25% improvement going with RAW over Cooked. If you use cooked the IO has to be written by the Unix Operating system - another set of commands to execute. Raw is controlled by the database. Why o Why Are we still having this discussion? Rob Vorbroker --- Rajesh Kapur <rkapur@mpr.org> wrote: > Hello, > > I am installing IDS 10 on RHEL3 on an intel x86 > server. I need to make the > choice between raw devices and cooked files. > I found the following information in the FAQ's. Is > this information still > accurate? Do cooked files continue to be > significantly slower than the raw > files? I have heard that UNIX file systems have > improved over the years and > the performance divide between raw and cooked files > has narrowed > considerabley. Is it true? > > Thanks! > - Rajesh > -------------------------------------------------------------------------------------- > > 6.38 Is raw disk faster than cooked files? > On 22nd Jun 1998 kagel@bloomberg.net (Art S. Kagel) > wrote:- > > ....................the safety issue of cooked files > is no longer a problem. > The big problem with cooked files still is > performance. All writes and reads > to/from cooked files MUST go through the UNIX buffer > cache. This means an > additional copy from the server's output buffer to > the UNIX buffe...

Measurements: (was: Raw vs. Cooked files)
Hi, if you are interested : http://www.ibm.com/servers/eserver/zseries/library/techpapers/pdf/gm130667.pdf (There's a part where they tested raw devices vs. file system files as well. These measurements were all done on zSeries, not by myself as I did not get to do such extensive work. But this document is really extensive ...) Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management owner-informix-list@iiug.org wrote on 19.05.2005 16:43:23: > Martin Fuerderer wrote: > > Hi, > > I emailed a simple response yesterday, but the link seems to be lost. > Here's another more detailed one: > > It does not matter HOW much memory the OS has for a buffer cache. There are > still several reasons why RAW will ALWAYS be faster than COOKED devices or > OS filesystem files: > > 1) If the OS cache is too small bulk server IO like checkpoints and peak > load LRU flushing will thrash the cache killing OS and server performance. > - Thanks for pointing that out Jake. > > 2) All IO to a COOKED device or FS file MUST first be copied from the IDS > buffers to the OS buffers before the actual IO operation. This MUST be > extra overhead not needed on a RAW device - by definition! > > 3) If you are writing to an FS file chunk instead of a COOKED device you > have the additional overhead of the filesystem and direc...

Re: Raw vs. Cooked files #2
Hi, I'd say the correct answer to the question is: "It depends." -------------------------------------------------------------------- The implementation of file system I/O buffering by the OS can be very efficient on Linux. But for that the OS needs sufficient memory for the buffering. And this is what makes the "It depends". Therefore there are 2 extremes: a) almost all physical memory is used by IDS in the form of shared memory (allocated SHM segments listed by "onstat -g seg"). A good indication for this scenario is a machine that uses little swap space, because it already has swapped out some minor "applications". The system is at the (sensible) limit of its memory usage. In this scenario, raw devices and the utilization of KAIO (available with IDS 10.00.UC1 on Linux) will in general perform better than cooked files. b) there's a lot of unused physical memory left over, even though IDS is configured optimally (e.g. BUFFERS is big enough, etc.). A recent example of such a scenario was 32-bit IDS installed on a 64-bit Linux, where IDS can use only ~4 GB of the available 24 GB of physical memory. In such a scenario, the OS can allocate a lot of memory for file system I/O buffering and this will make file system I/O (i.e. for chunks on cooked files) very efficient. In this scenario it will normally perform better than raw devices and KAIO. This i...

Re: Raw vs. Cooked files #3
HOWEVER, Informix IDS does not user the OS buffering. It opens all COOKED chunks with O_SYNC which bypasses most of the benefits of the buffering in favor of added data integrity. Art S. Kagel ----- Original Message ----- From: Martin Fuerderer <MARTINFU@de.ibm.com> At: 5/18 4:27 Hi, I'd say the correct answer to the question is: "It depends." -------------------------------------------------------------------- The implementation of file system I/O buffering by the OS can be very efficient on Linux. But for that the OS needs sufficient memory for the buffering. And this is what makes the "It depends". Therefore there are 2 extremes: a) almost all physical memory is used by IDS in the form of shared memory (allocated SHM segments listed by "onstat -g seg"). A good indication for this scenario is a machine that uses little swap space, because it already has swapped out some minor "applications". The system is at the (sensible) limit of its memory usage. In this scenario, raw devices and the utilization of KAIO (available with IDS 10.00.UC1 on Linux) will in general perform better than cooked files. b) there's a lot of unused physical memory left over, even though IDS is configured optimally (e.g. BUFFERS is big enough, etc.). A recent example of such a scenario was 32-bit IDS installed on a 64-bit Linux, where IDS can use only ~4 GB of the ava...

Informix 12.10, Linux, cooked files
Hello, I'm going to install IFX 12.10.FC5 on Linux RH. I would use local disks with cooked files, what FS is better to use? Ext3? Ext4? Tx, Francesco Em segunda-feira, 9 de novembro de 2015 09:47:54 UTC-2, franc....@gmail.com escreveu: > Hello, > I'm going to install IFX 12.10.FC5 on Linux RH. > I would use local disks with cooked files, what FS is better to use? > Ext3? Ext4? > > Tx, > Francesco ext4 is a good choice. Em segunda-feira, 9 de novembro de 2015 09:47:54 UTC-2, franc....@gmail.com escreveu: > Hello, > I'm going to in...

Re: Measurements: (was: Raw vs. Cooked files)
Hi, maybe. :) This work was done by zSeries people (which are in a different department/organization from us in IBM Informix Data Management ...). They used IDS mainly for their purpose (to show zLinux capabilities like scaling with a real, commercially available application). Therefore we cannot directly demand from them to do this again with IDS 10.00. However, there is a possibility, that for their next round of measurements they will use IDS 10.00 (as they made good experiences with IDS 9.4). Though in that case I would not expect results before sometime next year .... Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management owner-informix-list@iiug.org wrote on 29.06.2005 02:31:50: > > Can we get this done with IDS 10? > > Martin Fuerderer wrote: > > Hi, > > > > if you are interested : > > > > http://www.ibm.com/servers/eserver/zseries/library/techpapers/pdf/gm130667.pdf > > > > (There's a part where they tested raw devices vs. file system files as > > well. > > These measurements were all done on zSeries, not by myself as I did not > > get to do such extensive work. But this document is really extensive ....) > > > > Regards, > > Martin > > -- > > Martin Fuerderer > > IBM Informix Development Munich, Germany > > Information Management sending to i...

Re: Measurements: (was: Raw vs. Cooked files) #2
Hi, > Why is NOAGE set to 0? Surely it should be set to 1? For this work the machine was dedicated solely to IDS server. No other applcations were running on the machine. Therefore NOAGE 1 vs. 0 would not have made much of a difference as there were no other processes to compete with IDS for CPU-time. Regards, Martin -- Martin Fuerderer IBM Informix Development Munich, Germany Information Management owner-informix-list@iiug.org wrote on 29.06.2005 02:32:39: > > Nice but.... > > Can we get this done with IDS 10? > > Set RESIDENT to -1 to make everything resident. > > Why is NOAGE set to 0? Surely it should be set to 1? > > > Martin Fuerderer wrote: > > Hi, > > > > if you are interested : > > > > http://www.ibm.com/servers/eserver/zseries/library/techpapers/pdf/gm130667.pdf > > > > (There's a part where they tested raw devices vs. file system files as > > well. > > These measurements were all done on zSeries, not by myself as I did not > > get to do such extensive work. But this document is really extensive ....) > > > > Regards, > > Martin > > -- > > Martin Fuerderer > > IBM Informix Development Munich, Germany > > Information Management > > > sending to informix-list Martin Fuerderer schrieb: > Hi, > > >>Why is NOAGE set to 0? Surely i...

Raw device VS cooked file
Hi everybody, a question about raw VS cooked once again (sorry about that), hopefully this time a bit different. I am using IDS 10.0 on Red Hat Enterprise Linux version 4, as well as on Solaris 8 and 9. I use Informix since years, and always have installed it making use of raw devices, since there always have been a clear statement in the IBM documentation telling that on UNIX raw devices are the suggested choice for the best I/O performance. The IBM IDS 10 Server Administrator Guide, http://publib.boulder.ibm.com/epubs/pdf/25122672.pdf, at page 241 says infact: --------------------------------------------------------------------------- <<On UNIX, you should use raw disk devices to store data whenever performance is important.>> --------------------------------------------------------------------------- However, looking further in the same document, at page 242, I have noticed that for the first time there is also this new and interesting statement: --------------------------------------------------------------------------- <<Important: While you should use raw disk devices on UNIX to achieve better performance, recent advances in I/O caching for cooked writes can provide similar if not better performance. To determine the best device performance, perform benchmark testing on the system with both types of devices for the dbspace and table layout.>> --------------------------------------------------------------------------- ...

RE: Raw device VS cooked file
In my experience with AIX. The new enhanced journal file system (JFS2), there are file system option to use SIO or DIO. Using DIO (direct io) gives pretty close results to raw partition with Oracle and Informix. I am not sure if Solaris, HPUX or Linux has any option similar to AIX Basim Chafik Senior Systems Analyst IBM Certified Advanced Technical Expert (CATE) 1.800.688.4895 basim.chafik@plx.com plexus (Division of BancTec) -----Original Message----- From: Rupan3rd [mailto:rupan.nospam.3rd@hotmail.com] Sent: Thursday, January 12, 2006 11:18 AM To: informix-list@iiug.org Subject: Re: Raw device VS cooked file - any benchmark recently done? I know what pros and cons of using raw might be. My need is to have the best performance for my OLTP system, therefore I was wordering if and in what cases it might be actually true that there are disk devices able to achieve "I/O caching for cooked writes" that "can provide similar if not better performance" than raw areas. Still the documentation does not explicitly say if the cooked READS would be that good as well nor how to benchmark nor the effect of the fragmentation. Regards Rupan3rd malcolm weallans wrote: > I would really like to see any evidence that people have on this issue. I > have a number of systems that are running with acceptable performance on raw > disks. But could we be doing better and at what cost? > > But from almost ...

How to copy a sun-database in one informix database to another informix database?
Hi, Sorry I am not an informix dba, and I was faced to copy a sub-database(sorry I don't know the clear name of such concept, as informix is like sql-server, which used multiple database) from one informix database system to another informix database. I know in oracle, we can use transportable tablespace, copy the datafile and transport the metadata, and do something else. How can I do it in informix then? Is there document which talk about the detail step? Thanks It would really help if you knew the versions of the Informix databases. Assuming Informix IDS 7.x or IDS...

Raw vs. Cooked
I am the dba for our shop's Informix based app. We use raw devices on our Sun Solaris boxes. We are about ready to start the process of redeveloping our agency's main app using Oracle. I am starting to familiarize myself with Oracle. I'm wondering about the ole cooked vs. raw space issue. In my Informix experience, I am quite comfortable using raw space. However,I have gotten the impression from Oracle folks (for example, an Oracle instructor) that the customary practice with Oracle is to use cooked space. In fact, the instructor said there wouldn't be any difference, so ...

Web resources about - raw vs. cooked files under linux - comp.databases.informix

Punky Brüster – Cooked on Phonics - Wikipedia, the free encyclopedia
Punky Brüster – Cooked on Phonics (a pun of the literacy program Hooked on Phonics ) is the debut studio album by Canadian musician Devin Townsend ...

The cooked veggieducken - Flickr - Photo Sharing!
Explore The Sporkful's photos on Flickr. The Sporkful has uploaded 46 photos to Flickr.

Chef dismembered, cooked girlfriend in Brisbane apartment
A young chef had been boiling his girlfriend's dismembered body parts in his Brisbane apartment when he was discovered by police on Saturday ...


Carbon tax not all it was cooked up to be
Carbon tax not all it was cooked up to be

HomeCooked app: Buy your neighbour’s home-cooked food instead of takeaways - The Courier-Mail Search ...
WE COULD soon be bidding farewell to the fish and chip shop and saying ta-ta to the takeaway Thai if a plan to transform the way we eat, in the ...


Tanzanian man 'beheaded and cooked'
A Tanzanian man has been beheaded and cooked following accusations of adultery.

They came, they cooked, they conquered: it's all about being Italian
They came, they cooked, they conquered: it's all about being Italian

The books are being cooked on 457 visas
The foreign worker program is being rorted just as the student visas were.

Resources last updated: 3/1/2016 5:48:09 AM