Hi All

What is different between page ins/outs and paging space page ins/
out ?

vmstat -s
           1889201253 total address trans. faults
            716070874 page ins
            268266358 page outs
               702336 paging space page ins
              1300804 paging space page outs
                    0 total reclaims

Below is our machine output, is page ins/outs normal ?

Date/Time           Page in     Page Out
=================== =========== ===========
2007/08/20_17:27:57   716251158   268334210
2007/08/20_17:28:07   716262327   268338566
2007/08/20_17:28:17   716274397   268343052
2007/08/20_17:28:27   716285384   268347983
2007/08/20_17:28:37   716299228   268353886
2007/08/20_17:28:47   716311282   268360359
2007/08/20_17:28:57   716324433   268365294
2007/08/20_17:29:07   716335859   268370183

ok when a page out wether its a pageout or a paging space page pout
occurs the LRUD daemon finds
a page in memory not least recently use so it frees it!! what do you
mean 'frees it' ? Well, if the page was a computational page
then LRUD forces it to be written to paging space so this is a paging
space pageout. If the page is a file that has been read into memory
then it writes it to disk IF the file changed otherwise it just gives
the page to another process to overwrite.
A paging space page in  indicates that a computational page (part of a
process) has been requested by another process and is read off of
paging space into memory. The other page in  indicates a file was
requested from a process so it was read from a filesystem into memory.
That vmstat -s data might be 'normal' but only you as a system admin
of that particular system can answer that question.
On AIX systems that run databases you do wantto do virtual memory
tuning so that very little paging space is used.
But on the same well-tuned Database system ytou might see many page in
and page outs because different files are being  accessed all the
time.
hope that this helps a little .....

On 8 20 ,   10 56 , LarryBoy <larry.tay...@co.travis.tx.us> wrote:
> ok when a page out wether its a pageout or a paging space page pout
> occurs the LRUD daemon finds
> a page in memory not least recently use so it frees it!! what do you
> mean 'frees it' ? Well, if the page was a computational page
> then LRUD forces it to be written to paging space so this is a paging
> space pageout. If the page is a file that has been read into memory
> then it writes it to disk IF the file changed otherwise it just gives
> the page to another process to overwrite.
> A paging space page in  indicates that a computational page (part of a
> process) has been requested by another process and is read off of
> paging space into memory. The other page in  indicates a file was
> requested from a process so it was read from a filesystem into memory.
> That vmstat -s data might be 'normal' but only you as a system admin
> of that particular system can answer that question.
> On AIX systems that run databases you do wantto do virtual memory
> tuning so that very little paging space is used.
> But on the same well-tuned Database system ytou might see many page in
> and page outs because different files are being  accessed all the
> time.
> hope that this helps a little .....

Thank.
Base on below information, why tends to set pi and po to zero value ?
http://www-03.ibm.com/servers/enable/site/peducation/wp/a266/page_4.html

It is seem that our machine is not memory-bound, It seem that is cpu-
bound(cpu id time is small)
 vmstat -t 1 10
System Configuration: lcpu=3 mem=6689MB
kthr     memory             page              faults        cpu
time
----- ----------- ------------------------ ------------ -----------
--------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa hr
mi se
 3  3 1032198   371   0   0   0 628  569   0 1216 13997 1447 45  7 38
10 09:00:46
14  8 1032255   370   0   0   0 1592 4349   0 2803 25605 4391 65 17  1
17 09:00:47
13  5 1032380   381   0   0   0 1657 4453   0 2751 26284 4350 49 20  1
30 09:00:48
 0 12 1032562   486   0   0   0 1915 5115   0 2975 25642 4895 56 19  0
25 09:00:49
12  5 1032585   366   0   0   0 1416 3799   0 2731 22188 4359 51 18  2
29 09:00:50
 6 12 1032576   376   0   0   0 1607 5128   0 3049 20635 4449 61 15  0
24 09:00:51
11 12 1033033   385   0   0   0 1926 25340   1 2760 55272 4007 59 23
4 14 09:00:52
 8 13 1032485   554   0   0   0 1045 8906   0 2766 24047 3997 68 19  0
13 09:00:53
10  6 1032544   366   0   0   0 1398 115324   0 2759 23395 4178 71 15
0 14 09:00:54
 4 15 1032518   450   0   0   0 1667 7211   0 2821 22323 4386 68 19  1
11 09:00:55

Check by "ps -e -o pid,ppid,pcpu,cpu,etime,time,vsz,user,rssize,comm".
One of process take about 22% CPU.

     PID     PPID  %CPU  CP    ELAPSED       TIME      VSZ
USER      RSS    COMMAND
 1753166        1  22.0  78   05:22:40   03:33:04    24280
root    24412   _sqlsrv2
 2785318        1  12.6  30      32:46   00:12:22     1368    phx
6300   _progres
 3121200        1   7.1  19   02:27:45   00:31:28     2232    phx
7164   _progres

> Thank.
> Base on below information, why tends to set pi and po to zero
> value ?
> http://www-03.ibm.com/servers/enable/site/peducation/wp/a266/page_4.html

As stated: Reading from disk is much slower then from ram.
>
> It is seem that our machine is not memory-bound, It seem that is
> cpu-bound(cpu id time is small)

If you have idle>0 i would say that the system has free computing
resources so its not cpu bound.

According to your posts its looks for me that you have trashing of
file cache - Wait i/o in conjunction with a high     sr/fr rate where
cy was even above 0 so your system is memory bound.
See man vmstat for details regarding sr/fr/cy

Your ps says that you are running a database ( progress ? ) which lead
to the conclusion that your system is not tuned for database usages
( usage of the CIO mount option  and DB related tuning settings ).

So read the documentation which comes for your database and configure
the database and os accordingly or in case you have more memory - put
it in.

hth
Hajo

On 8 21 ,   4 32 , Hajo Ehlers <serv...@metamodul.com> wrote:
> > Thank.
> > Base on below information, why tends to set pi and po to zero
> > value ?
> >http://www-03.ibm.com/servers/enable/site/peducation/wp/a266/page_4.html
>
> As stated: Reading from disk is much slower then from ram.
>
>
>
> > It is seem that our machine is not memory-bound, It seem that is
> > cpu-bound(cpu id time is small)
>
> If you have idle>0 i would say that the system has free computing
> resources so its not cpu bound.
>
> According to your posts its looks for me that you have trashing of
> file cache - Wait i/o in conjunction with a high     sr/fr rate where
> cy was even above 0 so your system is memory bound.
> See man vmstat for details regarding sr/fr/cy
>
> Your ps says that you are running a database ( progress ? ) which lead
> to the conclusion that your system is not tuned for database usages
> ( usage of the CIO mount option  and DB related tuning settings ).
>
> So read the documentation which comes for your database and configure
> the database and os accordingly or in case you have more memory - put
> it in.
>
> hth
> Hajo

Hi Hajo
But are paging space just 24% why Memory bound ?

<b>
System Configuration: lcpu=3 mem=6689MB
memory         page                                  faults
cpu
-------------- ------------------------------------- -----------------
------------
 r  b   avm    fre  re  pi   po   fr    sr     cy    in    sy    cs
us sy id wa
 2  3  1354885  372   0    0    1  1812  13693     0  2038 13319  2769
65  9  8 19
 4  7  1354901  367   0    4    1  2123  21329     0  2103 11001  2946
53  9 10 28
 6  7  1354900  362   0    2    1  1936  16109     0  2214 15161  3058
59 10  6 26
 9  3  1355214  492   0    2    0  2282  25674     0  2330 20846  3322
60 15  5 20
 8  8  1355123  376   0    2    0  1885  41644     1  2711 15529  3910
51 16  7 25
 7  6  1354998  360   0    0    0  1588  26703     0  2867 13146  4247
69 16  0 14
 4  7  1354958  383   0    1    0  1717  12439     0  2733 14364  3993
66 19  6  9
 4 10  1354958  364   0    1    0  1625  67156     0  2415 12604  3361
73 11  1 15
 8  4  1354971  361   0    0    0  1702   9267     0  2205 15116  3086
75  9  2 13
13  7  1354979  358   0   12    0  1781   7483     0  2183 13510  3045
75  8  0 17
 8  6  1354979  365   0    6    2  1686   6266     0  2043 14048  2852
74 10  4 13
 3  8  1354979  382   0    2    0  1720   7050     0  1919 11524  2702
81  8  0 11
 4  8  1354979  373   0    1    0  1596   5723     0  1986 13677  3011
73  7  1 19
 3 11  1354979  364   0    0    0  1604  10152     0  1987 13156  2951
74  6  0 19
16  4  1355213  362   0    3    1  2103   8158     0  1900 18582  2773
66 13  1 20
 8  7  1354979  375   0    0    1  1771   8105     0  1872 11950  2701
73  6  1 20
 4 11  1354979  361   0    0    1  1631   6302     0  1877 15135  2787
76  8  4 11
 2 10  1354979  365   0    0    1  1865   5817     0  2187 12215  3177
59 10  0 31
 3  9  1354985  370   0    0    3  1981   9215     0  2496 11559  3740
56 13  5 26
 4  6  1355036  452   0    0    1  2133   9397     0  2601 13492  4229
57 17  0 26
The average usage of cpu is : 96.95 Included wait
The average id    of cpu is : 3.05

2007/08/22,12:20:39,Users,690,paging space,3072MB,17%
2007/08/22,12:30:40,Users,692,paging space,3072MB,18%
2007/08/22,12:40:41,Users,687,paging space,3072MB,18%
2007/08/22,12:50:43,Users,682,paging space,3072MB,18%
2007/08/22,13:00:45,Users,681,paging space,3072MB,18%
2007/08/22,13:10:47,Users,677,paging space,3072MB,18%
2007/08/22,13:20:49,Users,678,paging space,3072MB,18%
2007/08/22,13:30:51,Users,672,paging space,3072MB,18%
2007/08/22,13:40:53,Users,684,paging space,3072MB,18%
2007/08/22,13:50:55,Users,693,paging space,3072MB,19%
2007/08/22,14:00:57,Users,694,paging space,3072MB,19%
2007/08/22,14:10:59,Users,707,paging space,3072MB,20%
2007/08/22,14:21:01,Users,712,paging space,3072MB,21%
2007/08/22,14:31:03,Users,702,paging space,3072MB,21%
2007/08/22,14:41:08,Users,715,paging space,3072MB,23%
2007/08/22,14:51:22,Users,697,paging space,3072MB,23%
2007/08/22,15:01:25,Users,710,paging space,3072MB,23%
2007/08/22,15:11:33,Users,716,paging space,3072MB,24%
2007/08/22,15:21:40,Users,712,paging space,3072MB,24%
2007/08/22,15:31:42,Users,716,paging space,3072MB,24%
2007/08/22,15:41:46,Users,716,paging space,3072MB,24%
</b>

> But are paging space just 24% why Memory bound ?

Because its looks for me that you have >>>>>file cache trashing <<<<!
which cause even some paging space in and outs.

Which means that either:
 - The system has not enough ram at all for its load.
 - The system and/or application is not properly configured for its
main task.

Since you provide no information about OS, Application and system
usage i can not give any tuning tips as already done.

regards
Hajo

BTW: get nmon to monitor your system and get familiar with the vmm of
AIX

Your vmstat's sr (pages of memory searched) is very high compared to
the fr (pages freed during LRUD's memory search)
that is what Hajo means by 'thrashing'
I have seen this type of behaviour when the VM is not tuned for the
database and when the DBA has kept making the datafile cache
(a generic DB term -- Oracle's specific term is SGA) larger and larger
until this type of condition occurred. When DBA's use a data file
cache
they pin (reserve only for application use) memory and thus decrease
the amount of memory other processes can use.

Remember if a file in memory is paged out then that action will not
increase paging space use --- so you can have paging out occurring
without an increase in the use of paging space.

To answer your question  "why tends to set pi and po to zero value ?"
pi means page in and po means page out  -- if fr (pages freed during
an LRUD search) is 0 then po will be 0   -- pi will be 0 if a file is
not  or 4K pages of a file are not read into memory  --- if no pages
are being freed (what you are seeing ) then no pages can be read in so
pi must be 0

hope that this adds clarity

On 8 22 ,   11 08 , LarryBoy <larry.tay...@co.travis.tx.us> wrote:
> Your vmstat's sr (pages of memory searched) is very high compared to
> the fr (pages freed during LRUD's memory search)
> that is what Hajo means by 'thrashing'
> I have seen this type of behaviour when the VM is not tuned for the
> database and when the DBA has kept making the datafile cache
> (a generic DB term -- Oracle's specific term is SGA) larger and larger
> until this type of condition occurred. When DBA's use a data file
> cache
> they pin (reserve only for application use) memory and thus decrease
> the amount of memory other processes can use.
>
> Remember if a file in memory is paged out then that action will not
> increase paging space use --- so you can have paging out occurring
> without an increase in the use of paging space.
>
> To answer your question  "why tends to set pi and po to zero value ?"
> pi means page in and po means page out  -- if fr (pages freed during
> an LRUD search) is 0 then po will be 0   -- pi will be 0 if a file is
> not  or 4K pages of a file are not read into memory  --- if no pages
> are being freed (what you are seeing ) then no pages can be read in so
> pi must be 0
>
> hope that this adds clarity

Thank. Yes, Our system may not turnning well.
I am not a System Admin. How to check physical memory and Virtual
memory ?
How to check pin memory ?
How to set pi/po by vmo ?
Thank for provide about what is "file cache trashing".

About OS
> prtconf
Processor Type: PowerPC_POWER5
Number Of Processors: 3
Processor Clock Speed: 1656 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
Memory Size: 7968 MB
Good Memory Size: 7968 MB
Firmware Version: IBM,SF230_120
Console Login: enable
Auto Restart: true
Full Core: false

Application
=========
Progress database
19 GB Production database ( Number of records cut-off at 11:17)
       Read      create      Update
----------- ----------- -----------
  739576225       92170      336659

11 GB History Database
2.5 GB transaction Database ( Number of records cut-off at 11:17)
       Read      create      Update
----------- ----------- -----------
  834936054        1843       17883

( Number of records read 1,500,000,000 to 1,800,000,000 daily ) scan
for updated records in order to update Oracle Database.

700+ users using Production database and History Database
10+ remote users using transaction Database and Production database by
Java connection


>iostat 5 5

System configuration: lcpu=3 disk=5

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    %
iowait
         48.3        12118.7              47.3      7.4
35.4       9.8

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           9.4      92.3      21.0    4981121  135425925
hdisk1           9.6      89.8      20.6    1285025  135425921
hdisk3           0.6     314.4       8.2   253228228  225212396
hdisk2          22.0     1457.9     180.6   1660079884  558609944
cd0              0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    %
iowait
         87.6        18976.8              64.3     10.8
10.5      14.3

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          37.7     455.9      63.7         20      2264
hdisk1          35.9     454.3      62.9          8      2268
hdisk3           0.0       0.0       0.0          0         0
hdisk2          80.2     2078.2     343.5       8716      1696
cd0              0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    %
iowait
        127.0        25815.2              73.9      4.2
6.8      15.1

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0           3.6      24.8       6.4         16       108
hdisk1           2.2      21.6       5.6          0       108
hdisk3           0.0       0.0       0.0          0         0
hdisk2          88.8     2956.0     501.4      13304      1476
cd0              0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    %
iowait
        178.4        46733.2              78.5     11.9
2.9       6.7

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          29.8     284.8      62.4          8      1416
hdisk1          30.2     284.8      62.6          8      1416
hdisk3           0.0       0.0       0.0          0         0
hdisk2          89.2     2448.0     427.4      10748      1492
cd0              0.0       0.0       0.0          0         0

tty:      tin         tout   avg-cpu:  % user    % sys     % idle    %
iowait
        123.2        31273.2              75.9      9.2
5.9       9.0

Disks:        % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk0          26.2     245.0      61.2         28      1197
hdisk1          27.8     240.2      60.0          4      1197
hdisk3           0.0       0.0       0.0          0         0
hdisk2          96.4     4471.2     533.4      20256      2100
cd0              0.0       0.0       0.0          0         0

...
> Thank. Yes, Our system may not turnning well.
> I am not a System Admin.
But the task is has to be done by a System Administrator and DB
Administrator


> How to check physical memory and Virtual memory ?
> How to check pin memory ?
> How to set pi/po by vmo ?
Determine and name the problem of your user
Tune and configure the applications according the provided guides.

> About OS> prtconf
....
I assume from the output that you are running AIX 5.3

$ man oslevel

So in case its AIX 5.3
and your not using raw devices
and your DB files are on their OWN filesystem with is JFS2

mount the DB fs with the CIO option and you might get some better
performace


A good starting point is
Oracle DB & RAC 10gR2 on IBM AIX 5L: Tips and Considerations
http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100556

Its for oracle but give a clue for other databases as well.

hth
Hajo