I have a (virtual) machine which panics with a checksum error on
importing a ZFS pool (consisting of a single device).  The machine's
running 10u4, currently with no patches.  The pool was last accessed
by a fully-patched 10u4 VM and was not exported cleanly. (The pool
actually came from a later version of the same imstance.)

My guess is that this (panic on import) should never happen: I
certainly can't see why it should - if the pool is knackered it should
say that and refuse to import it, not blow up. So I probably need to
put back in the patches that the later version had.  Of course those
patches live on the ZFS pool I want to import so I'll have to download
them all again, but I can do that.

But it might be that the pool is really broken somehow (I'm not quite
sure what happens when you roll back the VM).

So, does anyone know if there are known cases where ZFS "should" panic
on import?

thanks

In article <1d06177e-a5e6-44be-8e2a-d7ec603a5028@s13g2000prd.googlegroups.com>,
	Tim Bradshaw <tfb+google@tfeb.org> writes:
> I have a (virtual) machine which panics with a checksum error on
> importing a ZFS pool (consisting of a single device).  The machine's
> running 10u4, currently with no patches.  The pool was last accessed
> by a fully-patched 10u4 VM and was not exported cleanly. (The pool
> actually came from a later version of the same imstance.)
> 
> My guess is that this (panic on import) should never happen: I
> certainly can't see why it should - if the pool is knackered it should
> say that and refuse to import it, not blow up. So I probably need to
> put back in the patches that the later version had.  Of course those
> patches live on the ZFS pool I want to import so I'll have to download
> them all again, but I can do that.
> 
> But it might be that the pool is really broken somehow (I'm not quite
> sure what happens when you roll back the VM).
> 
> So, does anyone know if there are known cases where ZFS "should" panic
> on import?

I think most filesystems do that by default on unrecoverable
corruption. Of course, most filesystems won't actually spot
many corruptions, and will just silently return you the
corrupt data, but zfs should spot all such corruptions.

You could try issuing the import command with:
-o failmode=continue

The default is:
-o failmode=panic

-- 
Andrew Gabriel
[email address is not usable -- followup in the newsgroup]

On Feb 2, 1:26 am, and...@cucumber.demon.co.uk (Andrew Gabriel) wrote:

>
> I think most filesystems do that by default on unrecoverable
> corruption. Of course, most filesystems won't actually spot
> many corruptions, and will just silently return you the
> corrupt data, but zfs should spot all such corruptions.

Well, to modify my question: is there a way of looking at an
(unimported, since import panics the box) pool and reporting any
errors with it. I may be missing something but I could not see a way
of doing that.  That's why I assumed that import should not panic: if
the only way of checking a pool is to import it, and if import on a
corrupt pool will panic, there is a problem.

On 2008-02-02, Andrew Gabriel <andrew@cucumber.demon.co.uk> wrote:
> I think most filesystems do that by default on unrecoverable
> corruption. Of course, most filesystems won't actually spot
> many corruptions, and will just silently return you the
> corrupt data, but zfs should spot all such corruptions.

ACK, but IMHO ZFS isn't handling the "telling about corruption"-part
very well, ATM. I recently ran into an ugly situation, because of cheap
and crappy hardware (don't ask) and a "problematic" ZFS-configuration.
We had a zpool consisting of only one device on a hardware RAID-5. Some-
where during one night, ZFS decided that the RAID had gone bad and since
it was the only device in the zpool, it immediately panic'ed the system.
Upon reboot the 10u4 system tried to import the faulted zpool, only to
immediatley panic again and repeating the cycle over and over ...
The system only came up after manual intervention - single user and
removal of /etc/zfs/zpool.cache. I opened a support case (ID 38024509)
but it turned out that the "endless reboot cycle on ZFS import failure"-
problem will be addressed in 10u6 at best. Also the general opinion of
the Sun Techs about ZFS seems to be, that if you're screwed, you're
screwed really big time. With UFS one could at least recover the data
partially or use recovery tools to try and fix the FS metadata.
If you look at the ZFS-related bug reports on SunSolve the warm and
fuzzy feeling that ZFS-marketing is trying to push fades away pretty
fast. Don't get me wrong, i still like using Suns products, but ZFS is
far away from being the best thing since sliced bread. This has already
been discussed before and i also maintain the point, that ZFS was re-
leased _far_ to early. It has to be argued if the benefit of "in the
field tests" weights over "people being driven away because they had
bad experiences with the product".
To make this post a complete rant ;-) - lately if been having so much
trouble with Sun products, that i can't help but wonder if there's any
quality control at Sun at all. In the last month i've been bitten by
four seperate other problems (6604195, 6609869, 6562648, 6523130/with
Sun Cluster) that were introduced by patches. And that wasn't patch-
clusters or manually patching either, but with Sun UCE and a rather
conservative choice of baseline sets.
I don't know what is going on at Sun ATM, but i can definately say,
that i don't like where this is heading ...

Best regards,

	Frank

On Feb 5, 4:36=A0am, Frank Fegert <fra.nospam...@gmx.de> wrote:
> On 2008-02-02, Andrew Gabriel <and...@cucumber.demon.co.uk> wrote:
>
> > I think most filesystems do that by default on unrecoverable
> > corruption. Of course, most filesystems won't actually spot
> > many corruptions, and will just silently return you the
> > corrupt data, but zfs should spot all such corruptions.
>
> ACK, but IMHO ZFS isn't handling the "telling about corruption"-part
> very well, ATM. I recently ran into an ugly situation, because of cheap
> and crappy hardware (don't ask) and a "problematic" ZFS-configuration.
> We had a zpool consisting of only one device on a hardware RAID-5. Some-
> where during one night, ZFS decided that the RAID had gone bad and since
> it was the only device in the zpool, it immediately panic'ed the system.
> Upon reboot the 10u4 system tried to import the faulted zpool, only to
> immediatley panic again and repeating the cycle over and over ...
> The system only came up after manual intervention - single user and
> removal of /etc/zfs/zpool.cache. I opened a support case (ID 38024509)
> but it turned out that the "endless reboot cycle on ZFS import failure"-
> problem will be addressed in 10u6 at best. Also the general opinion of
> the Sun Techs about ZFS seems to be, that if you're screwed, you're
> screwed really big time. With UFS one could at least recover the data
> partially or use recovery tools to try and fix the FS metadata.
> If you look at the ZFS-related bug reports on SunSolve the warm and
> fuzzy feeling that ZFS-marketing is trying to push fades away pretty
> fast. Don't get me wrong, i still like using Suns products, but ZFS is
> far away from being the best thing since sliced bread. This has already
> been discussed before and i also maintain the point, that ZFS was re-
> leased _far_ to early. It has to be argued if the benefit of "in the
> field tests" weights over "people being driven away because they had
> bad experiences with the product".
> To make this post a complete rant ;-) - lately if been having so much
> trouble with Sun products, that i can't help but wonder if there's any
> quality control at Sun at all. In the last month i've been bitten by
> four seperate other problems (6604195, 6609869, 6562648, 6523130/with
> Sun Cluster) that were introduced by patches. And that wasn't patch-
> clusters or manually patching either, but with Sun UCE and a rather
> conservative choice of baseline sets.
> I don't know what is going on at Sun ATM, but i can definately say,
> that i don't like where this is heading ...
>
> Best regards,
>
> =A0 =A0 =A0 =A0 Frank

You're not alone in your observation - our company is migrating Sun
out of our datacenter because of these issues.  For one, we found a
bug that
reliably causes a kernel panic in Solaris 10 and were told it won't be
fixed until 10 u5 - and that was 2-3 months ago.  Seems almost
arrogant.

On Feb 4, 5:25 pm, Tim Bradshaw <tfb+goo...@tfeb.org> wrote:
>
> Well, to modify my question: is there a way of looking at an
> (unimported, since import panics the box) pool and reporting any
> errors with it. I may be missing something but I could not see a way
> of doing that.  That's why I assumed that import should not panic: if
> the only way of checking a pool is to import it, and if import on a
> corrupt pool will panic, there is a problem.

To reply to my own article: I've read the manual fairly carefully, and
it appears that yes, there is a problem.

Tim Bradshaw <tfb+google@tfeb.org> writes:

>On Feb 4, 5:25 pm, Tim Bradshaw <tfb+goo...@tfeb.org> wrote:
>>
>> Well, to modify my question: is there a way of looking at an
>> (unimported, since import panics the box) pool and reporting any
>> errors with it. I may be missing something but I could not see a way
>> of doing that.  That's why I assumed that import should not panic: if
>> the only way of checking a pool is to import it, and if import on a
>> corrupt pool will panic, there is a problem.

>To reply to my own article: I've read the manual fairly carefully, and
>it appears that yes, there is a problem.

Quite, import should not panic and if it does it's a bug.

(Best direct access to ZFS developers is through opensolaris.org)

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

On Feb 5, 10:47 am, lahuman9 <lahum...@gmail.com> wrote:
> On Feb 5, 4:36 am, Frank Fegert <fra.nospam...@gmx.de> wrote:
>
>
>
> > On 2008-02-02, Andrew Gabriel <and...@cucumber.demon.co.uk> wrote:
>
> > > I think most filesystems do that by default on unrecoverable
> > > corruption. Of course, most filesystems won't actually spot
> > > many corruptions, and will just silently return you the
> > > corrupt data, but zfs should spot all such corruptions.
>
> > ACK, but IMHO ZFS isn't handling the "telling about corruption"-part
> > very well, ATM. I recently ran into an ugly situation, because of cheap
> > and crappy hardware (don't ask) and a "problematic" ZFS-configuration.
> > We had a zpool consisting of only one device on a hardware RAID-5. Some-
> > where during one night, ZFS decided that the RAID had gone bad and since
> > it was the only device in the zpool, it immediately panic'ed the system.
> > Upon reboot the 10u4 system tried to import the faulted zpool, only to
> > immediatley panic again and repeating the cycle over and over ...
> > The system only came up after manual intervention - single user and
> > removal of /etc/zfs/zpool.cache. I opened a support case (ID 38024509)
> > but it turned out that the "endless reboot cycle on ZFS import failure"-
> > problem will be addressed in 10u6 at best. Also the general opinion of
> > the Sun Techs about ZFS seems to be, that if you're screwed, you're
> > screwed really big time. With UFS one could at least recover the data
> > partially or use recovery tools to try and fix the FS metadata.
> > If you look at the ZFS-related bug reports on SunSolve the warm and
> > fuzzy feeling that ZFS-marketing is trying to push fades away pretty
> > fast. Don't get me wrong, i still like using Suns products, but ZFS is
> > far away from being the best thing since sliced bread. This has already
> > been discussed before and i also maintain the point, that ZFS was re-
> > leased _far_ to early. It has to be argued if the benefit of "in the
> > field tests" weights over "people being driven away because they had
> > bad experiences with the product".
> > To make this post a complete rant ;-) - lately if been having so much
> > trouble with Sun products, that i can't help but wonder if there's any
> > quality control at Sun at all. In the last month i've been bitten by
> > four seperate other problems (6604195, 6609869, 6562648, 6523130/with
> > Sun Cluster) that were introduced by patches. And that wasn't patch-
> > clusters or manually patching either, but with Sun UCE and a rather
> > conservative choice of baseline sets.
> > I don't know what is going on at Sun ATM, but i can definately say,
> > that i don't like where this is heading ...
>
> > Best regards,
>
> >         Frank
>
> You're not alone in your observation - our company is migrating Sun
> out of our datacenter because of these issues.  For one, we found a
> bug that
> reliably causes a kernel panic in Solaris 10 and were told it won't be
> fixed until 10 u5 - and that was 2-3 months ago.  Seems almost
> arrogant.

can you be please more specific.. we are also using zfs a lot in
datacenters in present days..

Daniel

On Feb 5, 8:30=A0am, Daniel Brnak <Daniel.Br...@gmail.com> wrote:
> On Feb 5, 10:47 am, lahuman9 <lahum...@gmail.com> wrote:
>
>
>
>
>
> > On Feb 5, 4:36 am, Frank Fegert <fra.nospam...@gmx.de> wrote:
>
> > > On 2008-02-02, Andrew Gabriel <and...@cucumber.demon.co.uk> wrote:
>
> > > > I think most filesystems do that by default on unrecoverable
> > > > corruption. Of course, most filesystems won't actually spot
> > > > many corruptions, and will just silently return you the
> > > > corrupt data, but zfs should spot all such corruptions.
>
> > > ACK, but IMHO ZFS isn't handling the "telling about corruption"-part
> > > very well, ATM. I recently ran into an ugly situation, because of chea=
p
> > > and crappy hardware (don't ask) and a "problematic" ZFS-configuration.=

> > > We had a zpool consisting of only one device on a hardware RAID-5. Som=
e-
> > > where during one night, ZFS decided that the RAID had gone bad and sin=
ce
> > > it was the only device in the zpool, it immediately panic'ed the syste=
m.
> > > Upon reboot the 10u4 system tried to import the faulted zpool, only to=

> > > immediatley panic again and repeating the cycle over and over ...
> > > The system only came up after manual intervention - single user and
> > > removal of /etc/zfs/zpool.cache. I opened a support case (ID 38024509)=

> > > but it turned out that the "endless reboot cycle on ZFS import failure=
"-
> > > problem will be addressed in 10u6 at best. Also the general opinion of=

> > > the Sun Techs about ZFS seems to be, that if you're screwed, you're
> > > screwed really big time. With UFS one could at least recover the data
> > > partially or use recovery tools to try and fix the FS metadata.
> > > If you look at the ZFS-related bug reports on SunSolve the warm and
> > > fuzzy feeling that ZFS-marketing is trying to push fades away pretty
> > > fast. Don't get me wrong, i still like using Suns products, but ZFS is=

> > > far away from being the best thing since sliced bread. This has alread=
y
> > > been discussed before and i also maintain the point, that ZFS was re-
> > > leased _far_ to early. It has to be argued if the benefit of "in the
> > > field tests" weights over "people being driven away because they had
> > > bad experiences with the product".
> > > To make this post a complete rant ;-) - lately if been having so much
> > > trouble with Sun products, that i can't help but wonder if there's any=

> > > quality control at Sun at all. In the last month i've been bitten by
> > > four seperate other problems (6604195, 6609869, 6562648, 6523130/with
> > > Sun Cluster) that were introduced by patches. And that wasn't patch-
> > > clusters or manually patching either, but with Sun UCE and a rather
> > > conservative choice of baseline sets.
> > > I don't know what is going on at Sun ATM, but i can definately say,
> > > that i don't like where this is heading ...
>
> > > Best regards,
>
> > > =A0 =A0 =A0 =A0 Frank
>
> > You're not alone in your observation - our company is migrating Sun
> > out of our datacenter because of these issues. =A0For one, we found a
> > bug that
> > reliably causes a kernel panic in Solaris 10 and were told it won't be
> > fixed until 10 u5 - and that was 2-3 months ago. =A0Seems almost
> > arrogant.
>
> can you be please more specific.. we are also using zfs a lot in
> datacenters in present days..
>
> Daniel- Hide quoted text -
>
> - Show quoted text -

The issue we have is not related to ZFS specifically.

If you are running ZFS in production you should just go ahead and
fire yourself.

lahuman9 wrote:
> On Feb 5, 8:30 am, Daniel Brnak <Daniel.Br...@gmail.com> wrote:
>> can you be please more specific.. we are also using zfs a lot in
>> datacenters in present days..
>>
>> Daniel- Hide quoted text -
>>
>> - Show quoted text -
> 
> The issue we have is not related to ZFS specifically.
> 
> If you are running ZFS in production you should just go ahead and
> fire yourself.

Maybe you should learn how to trim and quote correctly before making
such ridiculous statements.

-- 
Ian Collins.

On Feb 5, 9:13=A0pm, Ian Collins <ian-n...@hotmail.com> wrote:
> lahuman9 wrote:
> > On Feb 5, 8:30 am, Daniel Brnak <Daniel.Br...@gmail.com> wrote:
> >> can you be please more specific.. we are also using zfs a lot in
> >> datacenters in present days..
>
> >> Daniel- Hide quoted text -
>
> >> - Show quoted text -
>
> > The issue we have is not related to ZFS specifically.
>
> > If you are running ZFS in production you should just go ahead and
> > fire yourself.
>
> Maybe you should learn how to trim and quote correctly before making
> such ridiculous statements.
>
> --
> Ian Collins.

People got the point.  If you are running a 1.0 file system in
production you
should be fired.  People don't run 1.0 apps in production, much less a
filesystem.

Only very recently was Netbackup able to even backup ZFS.

On Feb 5, 8:27 pm, lahuman9 <lahum...@gmail.com> wrote:
> On Feb 5, 9:13 pm, Ian Collins <ian-n...@hotmail.com> wrote:
>
>
>
> > lahuman9 wrote:
> > > On Feb 5, 8:30 am, Daniel Brnak <Daniel.Br...@gmail.com> wrote:
> > >> can you be please more specific.. we are also using zfs a lot in
> > >> datacenters in present days..
>
> > >> Daniel- Hide quoted text -
>
> > >> - Show quoted text -
>
> > > The issue we have is not related to ZFS specifically.
>
> > > If you are running ZFS in production you should just go ahead and
> > > fire yourself.
>
> > Maybe you should learn how to trim and quote correctly before making
> > such ridiculous statements.
>
> > --
> > Ian Collins.
>
> People got the point.  If you are running a 1.0 file system in
> production you
> should be fired.  People don't run 1.0 apps in production, much less a
> filesystem.
>
> Only very recently was Netbackup able to even backup ZFS.

I'm amused at your pontification.  I've deployed ZFS in production in
several cases with excellent results.  I've had nothing but positive
experiences; good performance (not for DBs yet), features not
available in other file systems, no compatibility issues, and I'm
still employed!

On Feb 5, 11:02=A0pm, ITguy <southa...@gmail.com> wrote:
> On Feb 5, 8:27 pm, lahuman9 <lahum...@gmail.com> wrote:
>
>
>
>
>
> > On Feb 5, 9:13 pm, Ian Collins <ian-n...@hotmail.com> wrote:
>
> > > lahuman9 wrote:
> > > > On Feb 5, 8:30 am, Daniel Brnak <Daniel.Br...@gmail.com> wrote:
> > > >> can you be please more specific.. we are also using zfs a lot in
> > > >> datacenters in present days..
>
> > > >> Daniel- Hide quoted text -
>
> > > >> - Show quoted text -
>
> > > > The issue we have is not related to ZFS specifically.
>
> > > > If you are running ZFS in production you should just go ahead and
> > > > fire yourself.
>
> > > Maybe you should learn how to trim and quote correctly before making
> > > such ridiculous statements.
>
> > > --
> > > Ian Collins.
>
> > People got the point. =A0If you are running a 1.0 file system in
> > production you
> > should be fired. =A0People don't run 1.0 apps in production, much less a=

> > filesystem.
>
> > Only very recently was Netbackup able to even backup ZFS.
>
> I'm amused at your pontification. =A0I've deployed ZFS in production in
> several cases with excellent results. =A0I've had nothing but positive
> experiences; good performance (not for DBs yet), features not
> available in other file systems, no compatibility issues, and I'm
> still employed!- Hide quoted text -
>
> - Show quoted text -

Good luck with that.  In 5-10 years we might consider it.  I'm not
going to
risk being the one to explain why there is something wrong with the
filesystem
and we can't read/write/export/import/mount/backup anything due to a
1.0 bug
because I thought ZFS was neat-o.  I can live without the features for
that.

Here's what a google search for "zfs bug" turned up:

http://www.techcrunch.com/2008/01/15/joyent-suffers-major-downtime-due-to-zf=
s-bug/
http://prefetch.net/blog/index.php/2007/11/28/is-zfs-ready-for-primetime/

^^ Read the comments, people are using hardware snapshots to secure
against *ZFS* data corruption.

lahuman9 <lahuman9@gmail.com> writes:

>Here's what a google search for "zfs bug" turned up:

>http://www.techcrunch.com/2008/01/15/joyent-suffers-major-downtime-due-to-zf=
>s-bug/

In this particular case, Joyent was running "snv_43"; NOT a
production version of the OS.  It was hit by the bug
on Jan 12, this year when we're at snv_8x.  A Solaris Nevada
build is released every two weeks so you can compute that the
age of that pre-release software was around 80 weeks (or 1.5 years)
The bug was fixed in snv_60, or around a year ago.

IMHO, This changes the picture somewhat.

>http://prefetch.net/blog/index.php/2007/11/28/is-zfs-ready-for-primetime/

>^^ Read the comments, people are using hardware snapshots to secure
>against *ZFS* data corruption.

The blog itself is scant on details; the only corruption I have witnessed
thus far was a buffer corrupted by another driver and then written
to disk (but checksum'ed after the corruption)

Software bugs will have such an effect and filessytems cannot protect
your data agains those; having said that, I do believe that ZFS needs
hardening against properly checksummed bogus metadata.

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

On Feb 6, 8:14=A0am, Casper H.S. Dik <Casper....@Sun.COM> wrote:
> lahuman9 <lahum...@gmail.com> writes:
> >Here's what a google search for "zfs bug" turned up:
> >http://www.techcrunch.com/2008/01/15/joyent-suffers-major-downtime-du...
> >s-bug/
>
> In this particular case, Joyent was running "snv_43"; NOT a
> production version of the OS. =A0It was hit by the bug
> on Jan 12, this year when we're at snv_8x. =A0A Solaris Nevada
> build is released every two weeks so you can compute that the
> age of that pre-release software was around 80 weeks (or 1.5 years)
> The bug was fixed in snv_60, or around a year ago.
>
> IMHO, This changes the picture somewhat.
>

Agreed. It's hard to believe someone let thought that would be a good
idea.

> >http://prefetch.net/blog/index.php/2007/11/28/is-zfs-ready-for-primet...
> >^^ Read the comments, people are using hardware snapshots to secure
> >against *ZFS* data corruption.
>
> The blog itself is scant on details; the only corruption I have witnessed
> thus far was a buffer corrupted by another driver and then written
> to disk (but checksum'ed after the corruption)
>
> Software bugs will have such an effect and filessytems cannot protect
> your data agains those; having said that, I do believe that ZFS needs
> hardening against properly checksummed bogus metadata.
>

=46rom an admin perspective, I don't understand the nitty gritty details
but I
do know it takes a long time for all the bugs to shake out of a new
piece
of software.  The more things that depend on that software the more
cautious one must be.  Don't get me wrong - I think ZFS has some great
features but I'm not going to risk my job over it at this point.  If
VxFS
corrupts itself it'll be Symantec's ass not mine :)

> Casper
> --
> Expressed in this posting are my opinions. =A0They are in no way related
> to opinions held by my employer, Sun Microsystems.
> Statements on Sun products included here are not gospel and may
> be fiction rather than truth.

On Feb 6, 1:14 pm, Casper H.S. Dik <Casper....@Sun.COM> wrote:
>
> Software bugs will have such an effect and filessytems cannot protect
> your data agains those; having said that, I do believe that ZFS needs
> hardening against properly checksummed bogus metadata.

I think this thread demonstrates there are some problems, since I was
running a production Solaris 10 release with current patches (PCA's
"missingrs" set). Although I don't share the very negative impression
of ZFS that some people in this thread clearly have, I would be
hesitant before using it in production applications unless the data
was transient.  The problems, as I see them, are:

1. Technical. I think we can agree that zpool import should not panic
a system, however badly mangled the pool is (I suppose there are cases
were it might induce a panic because it tickles some bug in a driver
or something, but I don't think that was the case here since the
"disk" was part of a virtual machine).  So there is at least one
serious bug in production.

2. Design.  The complete absence of offline diagnosis/debugging tools,
while it seems like a brave design decision, is actually, I think, not
a good thing.  Especially given (1) above: I want a way of being able
to say "yes, this pool is OK" without actually making the filesystems
on it available.

3. Approach.  There have been *long* periods where known problems went
unfixed, and this does not fill me with enthusiasm.

4. I still can't work out where ZFS fits.  There seems to be some
notion that one should use it with JBODs, but I think we all know that
big storage environments are just not like that, and probably will
never be like that.  I'm not convinced that ZFS's designers really
understood what the typical storage environment looks like. (This is
not to say that such environments are optimal - it's pretty clear that
FC SANs are on the way out for instance, since I can't see anything
they usefully provide that iSCSI/sufficiently-fast-ethernet does not
do, other than arcane complexity.)

Some of these problems (all but the first) are common to other Sun
products, but I think are mostly symptomatic of a company trying to
find its way in a world which is rather new to it.  I'm hoping they
will get resolved sooner rather than later.  Meantime ZFS is clearly
an incredibly cool system, if not quite battle-tested yet,

Tim Bradshaw <tfb+google@tfeb.org> writes:

>2. Design.  The complete absence of offline diagnosis/debugging tools,
>while it seems like a brave design decision, is actually, I think, not
>a good thing.  Especially given (1) above: I want a way of being able
>to say "yes, this pool is OK" without actually making the filesystems
>on it available.

"zpool scrub" for pool swhich aren't imported.

(I think the best "fsck" you can do for zfs is "fsck -n"; a tool
which detects errors but does not repair them.  The reason is that
fsck like tools detect and repair common inconsistencies which are
known artefacts of the filesystem's design.  As zfs is "always consistent
on disk", there are no such things to repair.

>3. Approach.  There have been *long* periods where known problems went
>unfixed, and this does not fill me with enthusiasm.

Unfixed in S10 or unfixed int he development release or also
undiagnosed?

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

On Feb 8, 9:45 am, Casper H.S. Dik <Casper....@Sun.COM> wrote:

>
> "zpool scrub" for pool swhich aren't imported.

Yes, something like that (I am going to look an idiot if it turns out
you can do this, but I don't think you can!)

>
> (I think the best "fsck" you can do for zfs is "fsck -n"; a tool
> which detects errors but does not repair them.  The reason is that
> fsck like tools detect and repair common inconsistencies which are
> known artefacts of the filesystem's design.  As zfs is "always consistent
> on disk", there are no such things to repair.

Well, what I meant was that, if we assume ZFS doesn't have a perfect
implementation (has not been proved correct, say...), then it might be
the case that actually there *are* bad things on the disk.  So I'd
like some tool which will look at a ZFS filesystem (or, probably a
pool) and look for such things (so far like fsck -n) and perhaps make
some attempt to fix them.

A related case is what I presume will be a common deployment of ZFS
where it is using a pool which is not redundant at the ZFS level but
is redundant at the underlying storage level (say a LUN exposed by a
SAN).  In that case there is always the possibility of silent
corruption from the stoage. I *think* this will go undetected until
the data is next read, when the checksums will pick it up.  It would
be nice to have a tool which will take such a damaged pool and
obliterate the damaged parts, resulting in data loss but perhaps not
the loss of the whole pool.  It may actually be the case that ZFS can
do this already.  (And, further, I may be wrong about the time the
error is detected.)

I guess what I'm after is a ZFS disk debugger...

>
> Unfixed in S10 or unfixed int he development release or also
> undiagnosed?

Unfixed in S10.  The particular case I'm thinking off is the ZFS-on-
old-Sun-IDE-controllers performance problem.  Granted this was not a
data-loss bug, and granted it probably did not affect much or any use
of ZFS in anger, but it *did* affect people who wanted to look at ZFS
by trying it on random old kit and it took months to fix (fixed in u3
I think, and ZFS came in in u1?).

Tim Bradshaw <tfb+google@tfeb.org> writes:

>Well, what I meant was that, if we assume ZFS doesn't have a perfect
>implementation (has not been proved correct, say...), then it might be
>the case that actually there *are* bad things on the disk.  So I'd
>like some tool which will look at a ZFS filesystem (or, probably a
>pool) and look for such things (so far like fsck -n) and perhaps make
>some attempt to fix them.

Right; but in part this runs counter to ZFS's design goals.
Unfortunately, one of ZFS design goals is confronting you with
the stark reality of dataloss rather than tapering over it by
making the filesystem "consistent.

E.g., "fsck" in ufs makes sure that the meta data is "sane".

It makes no such guarantees for the data in a file.  So typical
UFS data loss looks like "file recnetly modified with stale content
from other files".  People with flaky hardware or a flaky software
installation may have seen this when /etc/path_to_inst or other
crucial files were updated just prior to a panic and not properly
flushed to disk: the files were there and had the proper size;
but the content was bogus stale content from other files.

With that in mind, a ZFS "repair" utility could on a quiescent
and unmounted zpool do the following things:

	- delete all pointers to corrupt data.
	  (data with failed checksum or which fails consistency
	  checks)
	- garbage collect lost blocks.
	  (an analysis could be done to see if those
	  blocks are partial valid trees of sorts)

But I think it's clear that a bad zpool should either fail import
or make those bits which cannot be reached fail with "EIO".

>A related case is what I presume will be a common deployment of ZFS
>where it is using a pool which is not redundant at the ZFS level but
>is redundant at the underlying storage level (say a LUN exposed by a
>SAN).  In that case there is always the possibility of silent
>corruption from the stoage. I *think* this will go undetected until
>the data is next read, when the checksums will pick it up.  It would
>be nice to have a tool which will take such a damaged pool and
>obliterate the damaged parts, resulting in data loss but perhaps not
>the loss of the whole pool.  It may actually be the case that ZFS can
>do this already.  (And, further, I may be wrong about the time the
>error is detected.)

The problem with replication below the ZFS layer is that there is
no way for ZFS to ask the array for "the other copy".

With storage replication at the ZFS level this is easy: read good
copy, rewrite bad copy.

>Unfixed in S10.  The particular case I'm thinking off is the ZFS-on-
>old-Sun-IDE-controllers performance problem.  Granted this was not a
>data-loss bug, and granted it probably did not affect much or any use
>of ZFS in anger, but it *did* affect people who wanted to look at ZFS
>by trying it on random old kit and it took months to fix (fixed in u3
>I think, and ZFS came in in u1?).

Ah, yes, that one.

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Casper H.S. Dik <Casper.Dik@sun.com> wrote:
> (I think the best "fsck" you can do for zfs is "fsck -n"; a tool
> which detects errors but does not repair them.  The reason is that
> fsck like tools detect and repair common inconsistencies which are
> known artefacts of the filesystem's design.  As zfs is "always consistent
> on disk", there are no such things to repair.

Lets imagine a memory problem causes some bit-twiddling so that during
an update, garbage is written to storage.  The same problem causes some
other errors that end up panicing the system before much time passes.
Investigation finds the problem and it is repaired, but there's already
damage on the disk.

It might be handy to be able to access older uberblock trees, even if
the most recent one is valid at the top.  Of course you'd want to do
this without requiring a default (read/write) import.  And if you did
find an older one was more consistent, the ability to link it back in
(destroying the later trees) would be useful.

-- 
Darren Dunham                                           ddunham@taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

> >Unfixed in S10.  The particular case I'm thinking off is the ZFS-on-
> >old-Sun-IDE-controllers performance problem.  Granted this was not a
> >data-loss bug, and granted it probably did not affect much or any use
> >of ZFS in anger, but it *did* affect people who wanted to look at ZFS
> >by trying it on random old kit and it took months to fix (fixed in u3
> >I think, and ZFS came in in u1?).
> 
> Ah, yes, that one.

Plus two more issues I encountered:
1. Broken half of a zfs mirror due to bad disk does not take that half
offline.
2. zfs destroy can take a ridiculously long time (up to 90 minutes for a
zfs of a couple of dozen gigabytes).
Workaround given by Sun: cd <mountpoint>; umount <mountpoint>; zfs
destroy.
But that unshares an NFS share that is exported by zfs.

Supposed to be fixed for S10u5 (summer?).

Ufo

On Feb 6, 10:16 am, lahuman9 <lahum...@gmail.com> wrote:
> Here's what a google search for "zfs bug" turned up:
>
> http://www.techcrunch.com/2008/01/15/joyent-suffers-major-downtime-du...http://prefetch.net/blog/index.php/2007/11/28/is-zfs-ready-for-primet...
>
> ^^ Read the comments, people are using hardware snapshots to secure
> against *ZFS* data corruption.

I think you are experiencing the same problem as Joyent. I don't know
which build of Nevada the Solaris Marketing Releases are based.
http://www.joyeur.com/2008/01/24/new-podcast-quad-core-episode-2
http://www.joyeur.com/2008/01/22/bingodisk-and-strongspace-what-happened

Does anybody knows how to map a Nevada build to a equivalent Solaris
Marketing Release?
E.g. Build 75 -> Update X
--
Emerson Seiti Takahashi

Emerson Seiti Takahashi wrote:
> On Feb 6, 10:16 am, lahuman9 <lahum...@gmail.com> wrote:
>> Here's what a google search for "zfs bug" turned up:
>>
>> http://www.techcrunch.com/2008/01/15/joyent-suffers-major-downtime-du...http://prefetch.net/blog/index.php/2007/11/28/is-zfs-ready-for-primet...
>>
>> ^^ Read the comments, people are using hardware snapshots to secure
>> against *ZFS* data corruption.
> 
> I think you are experiencing the same problem as Joyent. I don't know
> which build of Nevada the Solaris Marketing Releases are based.
> http://www.joyeur.com/2008/01/24/new-podcast-quad-core-episode-2
> http://www.joyeur.com/2008/01/22/bingodisk-and-strongspace-what-happened
> 
> Does anybody knows how to map a Nevada build to a equivalent Solaris
> Marketing Release?
> E.g. Build 75 -> Update X

sorry but running solaris *nevada* (which is use-at-your-own-risk 
without any support from sun) on production machines gives me the creeps...

now to your question, I don't think you can map solaris update releases 
to a Nevada build, IMHO sun is just picking stuff that's tested and 
stable from nevada (+ the usual bugfixes) and includes it in their 
solaris updates

"Solaris 10 What's New" is worth reading :)

On Feb 8, 1:28 pm, Casper H.S. Dik <Casper....@Sun.COM> wrote:

>
> With that in mind, a ZFS "repair" utility could on a quiescent
> and unmounted zpool do the following things:
>
>         - delete all pointers to corrupt data.
>           (data with failed checksum or which fails consistency
>           checks)
>         - garbage collect lost blocks.
>           (an analysis could be done to see if those
>           blocks are partial valid trees of sorts)
>

Yes, this is the sort of thing I'd like.

> But I think it's clear that a bad zpool should either fail import
> or make those bits which cannot be reached fail with "EIO".

The trouble is that this (EIO) is way too late, because approximately
no applications will deal with EIO elegantly.  So doing this is likely
to result in some abrupt application failure at an inconvenient
point.  I'd much, much rather have a tool that could do this before
the application runs.  (You are going to say that some combination of
find & dd is such a tool and you probably are technically correct.)