Hey all;

An actual Solaris-related question here! Hold onto your hats folks!

We've implemented DSEE 6.3.1 on a few servers for a fairly modest
environment (~500k small entries, or less), and one of our admins has
done some ZFS tuning. Specifically, we have the following:
	set autoup=300
	set zfs:zfs_arc_max=0x800000000
	set zfs:zfs_vdev_cache_size=0
	set zfs:zfs_prefetch_disable=1
	set zfs:zfs_vdev_cache_bshift=13
	set zfs:zfs_nocacheflush=1

This on a T5240 with 64GB of RAM, and the zpool resides on six of the
eight internal disks. (mirrored UFS root is on the other two.) OS is
Solaris 10u8 (10/09).

Any thoughts? The zfs_nocacheflush=1 strikes me as downright dangerous,
but the rest seem to be relatively pointless at best to my untrained
eye.

Cheers,
Colin

On 12/31/10 11:37 AM, Colin B. wrote:
> Hey all;
>
> An actual Solaris-related question here! Hold onto your hats folks!
>
> We've implemented DSEE 6.3.1 on a few servers for a fairly modest
> environment (~500k small entries, or less), and one of our admins has
> done some ZFS tuning. Specifically, we have the following:

I guess the obvious question is way?  I've never had to do any manual 
tuning on LDAP servers.

> 	set autoup=300
> 	set zfs:zfs_arc_max=0x800000000
> 	set zfs:zfs_vdev_cache_size=0
> 	set zfs:zfs_prefetch_disable=1
> 	set zfs:zfs_vdev_cache_bshift=13
> 	set zfs:zfs_nocacheflush=1
>
> This on a T5240 with 64GB of RAM, and the zpool resides on six of the
> eight internal disks. (mirrored UFS root is on the other two.) OS is
> Solaris 10u8 (10/09).

Odd choice (the mirrored UFS root).

-- 
Ian Collins

In article <ifj1j1$ha7$1@speranza.aioe.org>,
Colin B. <cbigam@somewhereelse.shaw.ca> wrote:
>We've implemented DSEE 6.3.1 on a few servers for a fairly modest
>environment (~500k small entries, or less), and one of our admins has
>done some ZFS tuning. Specifically, we have the following:
>	set autoup=300
>	set zfs:zfs_arc_max=0x800000000
>	set zfs:zfs_vdev_cache_size=0
>	set zfs:zfs_prefetch_disable=1
>	set zfs:zfs_vdev_cache_bshift=13
>	set zfs:zfs_nocacheflush=1

>This on a T5240 with 64GB of RAM, and the zpool resides on six of the
>eight internal disks. (mirrored UFS root is on the other two.) OS is
>Solaris 10u8 (10/09).

>The zfs_nocacheflush=1 strikes me as downright dangerous,

Assuming (based on your description) that your T5240 does not have the 
optional internal RAID card, then this depends on the state of the disk 
write cache.  If it's disabled, zfs_nocacheflush will do no harm, but 
it's also pointless.  If the cache is enabled, it will improve 
performance somewhat at the expensive of losing any guarantee of data 
integrity on power failure.

You can set or query the write cache state with format(1m).

>but the rest seem to be relatively pointless at best to my untrained
>eye.

This looks like copy+paste tuning of ZFS for an OLTP workload, probably 
taken from [0].  It may or may not help, depending on workload and ZFS 
release, but to know for sure you'd need to benchmark it with and 
without the tuning in place.  (Sorry, I realise that's not particularly 
helpful.)

	- river.

[0] http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

Ian Collins <ian-news@hotmail.com> wrote:
> On 12/31/10 11:37 AM, Colin B. wrote:
>> Hey all;
>>
>> An actual Solaris-related question here! Hold onto your hats folks!
>>
>> We've implemented DSEE 6.3.1 on a few servers for a fairly modest
>> environment (~500k small entries, or less), and one of our admins has
>> done some ZFS tuning. Specifically, we have the following:
> 
> I guess the obvious question is way?  I've never had to do any manual 
> tuning on LDAP servers.

That is an excellent question, and one I asked as well. The answer was
"These are recommendations for zfs with directory server." Not exactly
what I'd call an illuminating answer.

> Odd choice (the mirrored UFS root).

As opposed to root ZFS? We're fairly conservative that way - root ZFS is
being used in the lab here and there, but not in production yet. In fact,
I don't know that it will be, since Oracle is working so hard to push us
away from SPARC, Solaris, and Oracle/Sun software packages.

Cheers,
Colin

River Tarnell <r.tarnell@ieee.org> wrote:
> In article <ifj1j1$ha7$1@speranza.aioe.org>,
> Colin B. <cbigam@somewhereelse.shaw.ca> wrote:
>>We've implemented DSEE 6.3.1 on a few servers for a fairly modest
>>environment (~500k small entries, or less), and one of our admins has
>>done some ZFS tuning. Specifically, we have the following:
>>       set autoup=300
>>       set zfs:zfs_arc_max=0x800000000
>>       set zfs:zfs_vdev_cache_size=0
>>       set zfs:zfs_prefetch_disable=1
>>       set zfs:zfs_vdev_cache_bshift=13
>>       set zfs:zfs_nocacheflush=1
> 
>>This on a T5240 with 64GB of RAM, and the zpool resides on six of the
>>eight internal disks. (mirrored UFS root is on the other two.) OS is
>>Solaris 10u8 (10/09).
> 
>>The zfs_nocacheflush=1 strikes me as downright dangerous,
> 
> Assuming (based on your description) that your T5240 does not have the 
> optional internal RAID card, then this depends on the state of the disk 
> write cache.  If it's disabled, zfs_nocacheflush will do no harm, but 
> it's also pointless.  If the cache is enabled, it will improve 
> performance somewhat at the expensive of losing any guarantee of data 
> integrity on power failure.

Pretty much what I was thinking.

> You can set or query the write cache state with format(1m).

Thanks. I do that so rarely that I always have to look it up.

I thought that the write cache on Sun devices used to be disabled by
default, but I see that on these disks it's enabled. Interesting. And
by disabling cache flushing, harmful.

> This looks like copy+paste tuning of ZFS for an OLTP workload, probably
> taken from [0].  It may or may not help, depending on workload and ZFS
> release, but to know for sure you'd need to benchmark it with and
> without the tuning in place.  (Sorry, I realise that's not particularly
> helpful.)
>
> [0] http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

Thanks. I've read that, as well as this blog[1], but can't really see that
most of these settings are going to help us. I suspect (STRONGLY) that cut-and-paste-without-reading was the source of the tuning.

Cheers,
Colin

[1] http://www.thezonemanager.com/2009_03_01_archive.html