I have a Sun E6500 that I think needs more memory and while I have
some knowledge of Solaris performance
monitoring It would be great if someone could confirm or deny my
diagnosis.

The system is running Solaris 2.6, has 10gb of RAM and is running 4
Ingres databases used by 500 users concurrently.

I ran "vmstat 1" and am reading it as follows.

Note, I've ignored the first line as it contains rubbish, and also the
free memory on my system
because it always shows around 160mb which as I understand is due to
file cache utilising any free memory.

The sr column is consistently over 5000 which is as I understand it is
pages scanned and tells me that the system
is searching for free memory regularly. The fr column I believe is
showing me that the page scanner is freeing up
about 400+ pages per second which I think is quite high.

There are also a large amount of page ins and page outs.

# vmstat 1
 procs     memory            page            disk          faults     
cpu
 r b w   swap  free  re  mf pi po fr de sr s6 s9 s1 s1   in   sy   cs
us sy id
 0 0 0   7696  7384   0   0  0  0  0 7056 0 0  6  0  6 4294967196 0 0
-22 -9 -130
 0 2 0 5411264 157072 7 524 8336 400 5088 6352 691 0 66 0 67 4803 7728
2953 18 9 73
 0 2 0 5409928 157632 14 641 7960 440 5048 8312 688 0 33 0 20 4623
19190 5615 23 11 67
 0 2 0 5409640 157176 28 1814 8984 592 5400 9552 726 0 46 0 38 4532
27164 7346 37 16 47
 0 2 0 5410360 158536 30 859 10896 480 5504 9896 720 0 51 0 54 5295
27847 8093 37 18 45
 1 4 0 5409504 158440 28 440 9688 728 5400 8912 694 0 44 0 45 5675
25475 8932 42 21 36
 1 5 0 5410032 160296 20 876 6984 544 3984 9176 505 0 123 0 100 5252
20088 6893 28 27 44
 1 3 0 5400560 160392 24 2007 5856 744 1800 26200 164 0 40 0 62 4711
26187 7589 39 24 37
 1 2 0 5399032 159344 161 3424 6760 400 1416 23584 163 0 19 0 21 4348
17469 4017 34 14 52

On average there are 3-4 jobs waiting in the run queue which I am
thinking is rather high.
# w -u
  2:20pm  up 30 day(s), 17:45,  466 users,  load average: 3.91, 4.53,
3.91

My swap volume is doing performing as follows :
# vxstat -i1 -g rootdg -v swapvol
                        OPERATIONS           BLOCKS        AVG
TIME(ms)
TYP NAME              READ     WRITE      READ     WRITE   READ  WRITE

Wed Oct 08 14:45:25 2003
vol swapvol        5370906    950719 104244184 107605008   10.5   30.7

Wed Oct 08 14:45:26 2003
vol swapvol             19         0       304         0    8.4    0.0

Wed Oct 08 14:45:27 2003
vol swapvol              5         0        80         0   10.0    0.0

Wed Oct 08 14:45:28 2003
vol swapvol              5         0        80         0   94.0    0.0

Wed Oct 08 14:45:29 2003
vol swapvol              8         0       128         0   10.0    0.0

Wed Oct 08 14:45:30 2003
vol swapvol              4         0        64         0    7.5    0.0

Ben Jackson wrote:
> I have a Sun E6500 that I think needs more memory and while I have
> some knowledge of Solaris performance
> monitoring It would be great if someone could confirm or deny my
> diagnosis.
> 
> The system is running Solaris 2.6, has 10gb of RAM and is running 4
> Ingres databases used by 500 users concurrently.

If one day you can aim for Solaris 8 or 9 then it has a much improved
algorithm (the "cyclic page cache") which manages filesystem cache and
avoids routinely having to scan for pages to free.  In 2.6 you can
enable priority_paging (it may even be on by default these days?)
but it's no substitute.

> I ran "vmstat 1" and am reading it as follows.
> 
> Note, I've ignored the first line as it contains rubbish, 

These are the stats since boot and aren't terribly useful usually.

> and also the
> free memory on my system
> because it always shows around 160mb which as I understand is due to
> file cache utilising any free memory.

Yes, by default we work to keep a certain number of pages free (some
fraction of physical memory).  The rest can be used by filesystem
cache etc and it's freed when necessary by a scanner.  It's this
algorithm that is avoided in 8 and 9.

> The sr column is consistently over 5000 which is as I understand it is
> pages scanned and tells me that the system
> is searching for free memory regularly.

Yes, sounds like it.  That's a high scan rate.

> The fr column I believe is
> showing me that the page scanner is freeing up
> about 400+ pages per second which I think is quite high.

Yes, and some of them may be immediately missed and faulted
back in.  Priority paging improved this.  The trouble with
sourcing 400+ pages/s via the scanner is that the scanner then
limits filesystem etc throughput.

> 
> There are also a large amount of page ins and page outs.

Those counters include page io from regular filesystem IO.
I guess Ingres is using datafiles in filesystem?  You
*may* get some mileage out of playing with the forcedirectio
mount_ufs(1M) mount option.

> # vmstat 1
>  procs     memory            page            disk          faults     
> cpu
>  r b w   swap  free  re  mf pi po fr de sr s6 s9 s1 s1   in   sy   cs
> us sy id
>  0 0 0   7696  7384   0   0  0  0  0 7056 0 0  6  0  6 4294967196 0 0
> -22 -9 -130
>  0 2 0 5411264 157072 7 524 8336 400 5088 6352 691 0 66 0 67 4803 7728
> 2953 18 9 73
>  0 2 0 5409928 157632 14 641 7960 440 5048 8312 688 0 33 0 20 4623
> 19190 5615 23 11 67
>  0 2 0 5409640 157176 28 1814 8984 592 5400 9552 726 0 46 0 38 4532
> 27164 7346 37 16 47
>  0 2 0 5410360 158536 30 859 10896 480 5504 9896 720 0 51 0 54 5295
> 27847 8093 37 18 45
>  1 4 0 5409504 158440 28 440 9688 728 5400 8912 694 0 44 0 45 5675
> 25475 8932 42 21 36
>  1 5 0 5410032 160296 20 876 6984 544 3984 9176 505 0 123 0 100 5252
> 20088 6893 28 27 44
>  1 3 0 5400560 160392 24 2007 5856 744 1800 26200 164 0 40 0 62 4711
> 26187 7589 39 24 37
>  1 2 0 5399032 159344 161 3424 6760 400 1416 23584 163 0 19 0 21 4348
> 17469 4017 34 14 52
> 
> On average there are 3-4 jobs waiting in the run queue which I am
> thinking is rather high.
> # w -u
>   2:20pm  up 30 day(s), 17:45,  466 users,  load average: 3.91, 4.53,
> 3.91

That's nothing.  A load average of that per cpu would be nothing to
get concerned about, either (you don't say how many cpus you have,
but even 1 would happily cope with a load average of 4 or 5 as above).

> My swap volume is doing performing as follows :
> # vxstat -i1 -g rootdg -v swapvol
>                         OPERATIONS           BLOCKS        AVG
> TIME(ms)
> TYP NAME              READ     WRITE      READ     WRITE   READ  WRITE
> 
> Wed Oct 08 14:45:25 2003
> vol swapvol        5370906    950719 104244184 107605008   10.5   30.7
> 
> Wed Oct 08 14:45:26 2003
> vol swapvol             19         0       304         0    8.4    0.0
> 
> Wed Oct 08 14:45:27 2003
> vol swapvol              5         0        80         0   10.0    0.0
> 
> Wed Oct 08 14:45:28 2003
> vol swapvol              5         0        80         0   94.0    0.0
> 
> Wed Oct 08 14:45:29 2003
> vol swapvol              8         0       128         0   10.0    0.0
> 
> Wed Oct 08 14:45:30 2003
> vol swapvol              4         0        64         0    7.5    0.0

So it looks active.  Besides avoiding paging to swap by adding memory,
you may be able to rebalance existing usage.  For example, if Ingres
allocates a lrage shared memory area or similar then try making that
a smaller fraction of the 10GB available.

Gavin