Hi,

This is w.r.t Process RSS on Solaris 8.
We are running a regression test for one of our processes, 
i.e. sending the same set of requests, in a mix, continuously.

What we are observing is that the RSS of the process is growing
continuously whereas the virtual size does not. This we are
monitoring with the ps -efly command. 
What could be the reason for this growth? 
When/how will it decrease?
Is this an ok behaviour?
If so, till how long will the RSS keep growing - till it
equals the virtual size ??
This also means that the system memory usage also keeps
increasing continuously ? 

I expected that since the same requests are being sent, there is
no need for this growth to happen - is this assumption correct ?

Can anyone help me in this regard.

Thanks & Regards,
Anil

Anil wrote:
> Hi,
> 
> This is w.r.t Process RSS on Solaris 8.
> We are running a regression test for one of our processes, 
> i.e. sending the same set of requests, in a mix, continuously.
> 
> What we are observing is that the RSS of the process is growing
> continuously whereas the virtual size does not. This we are
> monitoring with the ps -efly command. 
> What could be the reason for this growth? 

Try using pmap -x to watch the address space of your process
during the run.

The SZ is, I believe, just the total size of the virtual
memory segments taking no account of whether this memory
is backed by real physical memory or not.

If/when we access a page that is not mapped we will
pagefault to bring that page into physical memory
(from whereever it lives, or if this is anonymous memory
we'll arrange an anonymous page).  The RSS will increment
whenever a new physical page is assigned to this process
address space in this sort of way.

> When/how will it decrease?

If the process does not undo the mapping/allocation etc
then the RSS will not shrink unless memory pressure
(eg, from other applictions) requires that we free
some pages.  If the process releases the pages they'll
be detached from it's address space and no longer reflect
in it's RSS.

> Is this an ok behaviour?

I can't tell. Probably.  But it's possible that the
appliction is mismanaging it's allocations, and (for
example) keeps using new buffers instead of reusing
old ones.

> If so, till how long will the RSS keep growing - till it
> equals the virtual size ??

RSS can never exceed the virtual size.  It can equal it if we
touch every page in every segment and nobody forces the
release of any of these mappings (eg, plenty memory).
It makes more sense to watch the RSS of each segment
in the pmap -x output.

> This also means that the system memory usage also keeps
> increasing continuously ? 

RSS on one process could increase while another decreases,
and system usage would stay the same.  Watch the 'free'
column in vmstat output to see free physical memory.

> I expected that since the same requests are being sent, there is
> no need for this growth to happen - is this assumption correct ?

You'd expect the process to grow to much the same pmap -x
output (including RSS) if asked to do the same thing
over and over (assuming no other memory pressure) and
restarting the process each time.

Gavin

Anil wrote:
> What we are observing is that the RSS of the process is growing
> continuously whereas the virtual size does not. This we are
> monitoring with the ps -efly command. 
> What could be the reason for this growth? 
> When/how will it decrease?
> Is this an ok behaviour?

I can think of two obvious reasons why this would happen.

#1:  Your process is using so much memory that the system is
      paging heavily.  The system will tend to take lots of pages
      away from the process in a big batch, dropping the resident
      size down because pages have been swapped out.  Then, as
      your process continues to run, it will cause pages to be
      faulted in, and the resident size will increase gradually.

      If this is the case, it is a problem, and you need to figure
      out a way to make your software use less memory (or buy
      more memory for the systems it runs on).  You should be able
      to detect this problem by running "vmstat 5" in a window,
      watching the "sr" column, and then starting the process.
      If the "sr" column jumps up to a larger number, this means
      the system is running low on RAM and is having to search
      for pages it can free.

#2:  Your process has memory mapped a file and is accessing it.
      If you are doing this, then at first the pages from the file
      will just be mapped into the virtual address space but won't
      be using RAM.  As you access them, the pages will be brought
      into RAM and your resident size will increase.  This is
      not a problem; it's an expected part of how memory-mapped
      files work.  The only thing to be aware of is that if you
      access a memory-mapped file that's too big, you could cause
      the system to have to do some work to make sure the right
      pages are in RAM at the right time.  (But this still might
      be the best solution.)

Hope that helps.

   - Logan

Hello,

Thanks for the responses.

I am looking in the application to see if any changes 
can be made.
Any tips in this regard?

Thanks & Regards,
Anil

"Anil" <anilvadrev@yahoo.com> wrote in message
news:b00ecf0d.0309250643.3dad562a@posting.google.com...
> Hello,
>
> Thanks for the responses.
>
> I am looking in the application to see if any changes
> can be made.
> Any tips in this regard?

My two cents:

* If possible, munmap(2) large, memory mapped files
   when the program is done with them.

* Try to avoid passing huge data structures to functions
   within the code. Pass pointers instead.

* Always free(3C) any malloc(3C)-ed buffers as you are
   finished with them. Re-use said buffers if feasable.

* Cast a beady eye over the code for memory leaks.

> Thanks & Regards,
> Anil


--
Noel R. Nihill
UNIX� platform development
Motorola NSS
I *could* be arguing in my spare time.

"Noel R. Nihill" <nnihil01@Motorola.BLAH.com> wrote in message
news:bkv3nd$op9$1@avnika.corp.mot.com...
>
> "Anil" <anilvadrev@yahoo.com> wrote in message
> news:b00ecf0d.0309250643.3dad562a@posting.google.com...
> > Hello,
> >
> > Thanks for the responses.
> >
> > I am looking in the application to see if any changes
> > can be made.
> > Any tips in this regard?
>
> My two cents:
>
> * If possible, munmap(2) large, memory mapped files
>    when the program is done with them.
>
> * Try to avoid passing huge data structures to functions
>    within the code. Pass pointers instead.
>
> * Always free(3C) any malloc(3C)-ed buffers as you are
>    finished with them. Re-use said buffers if feasable.
>
> * Cast a beady eye over the code for memory leaks.

Oh. I'll just add that if your app is free of memory leaks
and you are not mapping enormous files, you just might
need to get more RAM! Sometimes a cigar is just a cigar...

> > Thanks & Regards,
> > Anil

--
Noel R. Nihill
UNIX� platform development
Motorola NSS
I *could* be arguing in my spare time.

In article <bkv457$oqs$1@avnika.corp.mot.com>,
Noel R. Nihill <nnihil01@Motorola.BLAH.com> wrote:
>Oh. I'll just add that if your app is free of memory leaks
>and you are not mapping enormous files, you just might
>need to get more RAM! Sometimes a cigar is just a cigar...

If the resident size is getting huge, it means you already have enough
RAM.  If you didn't, other processes would be taking RAM away from this
process and its RSS wouldn't be able to grow so large.

A large RSS is likely to be an indicator that the application's memory
access patterns don't exhibit good locality of reference.  An example of a
program like this is "named" (at least through BIND 8, they might have
improved it in BIND 9); each query that it processes is likely to touch
many pages in the DNS cache memory that haven't recently been used, and if
its RSS is significantly lower than its SZ it's probably thrashing and
performing abysmally.

If this is the primary application that the system is used for, and it's
not causing performance problems for other processes on the system, you
might want to leave well enough alone.  But if it looks like the
application's memory needs are going to grow significantly (which probably
means that it will need to have more resident memory to maintain the same
performance), you may want to examine the algorithm to see whether you can
improve the way it references memory.

If you process huge arrays or memory-mapped files, try not to loop through
the whole thing repeatedly; e.g. if the structure of the code is something
like:

for each element in array
  do_something(array[i])
for each element in array
  do_something_else(array[i])

try changing it to:

for each element in array
  do_something(array[i])
  do_something_else(array[i])

-- 
Barry Margolin, barry.margolin@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

Noel R. Nihill wrote:

> "Anil" <anilvadrev@yahoo.com> wrote in message
> news:b00ecf0d.0309250643.3dad562a@posting.google.com...

>>Thanks for the responses.
>>
>>I am looking in the application to see if any changes
>>can be made.

> * Cast a beady eye over the code for memory leaks.

It seems unlikely that it's a memory leak.  I believe the
original poster said that the virtual size remains constant,
but the resident size is growing.  A memory leak would
cause the virtual size to grow.

   -Logan

"Logan Shaw" <lshaw-usenet@austin.rr.com> wrote in message
news:vHGcb.112741$KW1.17609@twister.austin.rr.com...
> Noel R. Nihill wrote:
>
> > "Anil" <anilvadrev@yahoo.com> wrote in message
> > news:b00ecf0d.0309250643.3dad562a@posting.google.com...
>
> >>Thanks for the responses.
> >>
> >>I am looking in the application to see if any changes
> >>can be made.
>
> > * Cast a beady eye over the code for memory leaks.
>
> It seems unlikely that it's a memory leak.  I believe the
> original poster said that the virtual size remains constant,
> but the resident size is growing.  A memory leak would
> cause the virtual size to grow.

Excellent point.

>    -Logan

--
Noel R. Nihill
UNIX� platform development
Motorola NSS
I *could* be arguing in my spare time.

"Noel R. Nihill" <nnihil01@Motorola.BLAH.com> writes in comp.unix.solaris:
|* If possible, munmap(2) large, memory mapped files
|   when the program is done with them.

Also look into madvise() if you finish with some regions but not others
in the mmap()'ed files.

-- 
________________________________________________________________________
Alan Coopersmith                              alanc@alum.calberkeley.org
http://www.CSUA.Berkeley.EDU/~alanc/       aka: Alan.Coopersmith@Sun.COM
  Working for, but definitely not speaking for, Sun Microsystems, Inc.