Hello,
         My objective here is that i have a cpu intensive process and I
dont want my system's idle time to go below 20%.

what i do is that if the idle time (from sar command) is lower than 40%
my process would be stopped and would be started when the idle time is
greater than 40 %.

        For this, i am using the output of sar command to find out the
idle time of the machine. Everytime i find that the idle time of the
system goes below 40%, i send a SIGSTOP signal to my process and when
the idle time is greater than 40% i send a SIGCONT signal to my
process.

        I am so skeptical about the method i follow to control the CPU
utilization of a process. Is there anyother way that is more effective
and is not ad-hoc ?

         I am using solaris 9.

         I have hit a brick wall and any help would be greatly
appreciated.

Cheers
Manik

rsmanik@gmail.com wrote:
>         My objective here is that i have a cpu intensive process and I
> dont want my system's idle time to go below 20%.

So you've got some real-time processing needs that could arise at any
moment? For all other practical purposes idle time isn't really relevant.

>        I am so skeptical about the method i follow to control the CPU
> utilization of a process. Is there anyother way that is more effective
> and is not ad-hoc ?

Take a look at renice and related commands.

Jens
-- 
Jens Goerke, mobilcom Server Systeme
mobilcom Communicationstechnik GmbH, Hollerstr. 126, 24782 Buedelsdorf
Tel: +49-(0)4331-69-5505 Fax: +49-(0)4331-69-5599

j.goerke@mobilcom.de wrote:
> rsmanik@gmail.com wrote:
> >         My objective here is that i have a cpu intensive process and I
> > dont want my system's idle time to go below 20%.
>
> So you've got some real-time processing needs that could arise at any
> moment? For all other practical purposes idle time isn't really relevant.
>
> >        I am so skeptical about the method i follow to control the CPU
> > utilization of a process. Is there anyother way that is more effective
> > and is not ad-hoc ?
>
> Take a look at renice and related commands.
>
Thanks for the quick response.

Using renice i can only lower the priority. Which means, when there are
no other processes competing for CPU my process can take upto 100% CPU,
which is exactly what i am trying to avoid.

I should be able to make my process use only a defined % of CPU and not
more than that. 

- Manik

rsmanik@gmail.com writes:

>         My objective here is that i have a cpu intensive process and I
>dont want my system's idle time to go below 20%.

Why?  This seems like something you should the scheduler for;
give the process a very low priority so it'll only run when
there's no other work to be done.

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Casper H.S. Dik wrote:
> rsmanik@gmail.com writes:
>
> >         My objective here is that i have a cpu intensive process and I
> >dont want my system's idle time to go below 20%.
>
> Why?  This seems like something you should the scheduler for;
> give the process a very low priority so it'll only run when
> there's no other work to be done.

    As i told in my previous post, i have an overload protection
mechanism that will be triggered if the idle time is lower than 10 %.
And my process is a low priority one that uses all the CPU time that is
avaliable, hence the idle time plummets to 0. This creates problems as
the overload protection is triggered.

   Again, my objective is to set an upper limit for the amount of CPU
my process can use. 


- Manik

rsmanik@gmail.com writes:

>    As i told in my previous post, i have an overload protection
>mechanism that will be triggered if the idle time is lower than 10 %.
>And my process is a low priority one that uses all the CPU time that is
>avaliable, hence the idle time plummets to 0. This creates problems as
>the overload protection is triggered.

So it's the overload protection which is broken; unfortunately, Solaris
does not at this time give an easy CPU measurement of "niced" programs
(SunOS 4.x would account low priority processes differently)

>   Again, my objective is to set an upper limit for the amount of CPU
>my process can use. 

It seems to me that your problem is *not* the amount of CPU used
by your process, but rather the way in which the system looks at
CPU use.  Can the overload protection be rewritten to ignore processes
with a low priority?

Is it really a good idea to let 10% of your CPU cycles (and thus your
compute power/electricity used) go to waste?

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Casper H.S. Dik wrote:
> rsmanik@gmail.com writes:
>
> >    As i told in my previous post, i have an overload protection
> >mechanism that will be triggered if the idle time is lower than 10 %.
> >And my process is a low priority one that uses all the CPU time that is
> >avaliable, hence the idle time plummets to 0. This creates problems as
> >the overload protection is triggered.
>
> So it's the overload protection which is broken; unfortunately, Solaris
> does not at this time give an easy CPU measurement of "niced" programs
> (SunOS 4.x would account low priority processes differently)

No, actually it cant be. There has to be a buffer of 10% cpu always
avaliable for incoming calls or higher priority processes. I also
understand that the overload protection is not doing what is actually
expected. But i cant fix it.

> >   Again, my objective is to set an upper limit for the amount of CPU
> >my process can use.
>
> It seems to me that your problem is *not* the amount of CPU used
> by your process, but rather the way in which the system looks at
> CPU use.  Can the overload protection be rewritten to ignore processes
> with a low priority?

The overload protection is out of my scope and has already been
deployed.

> Is it really a good idea to let 10% of your CPU cycles (and thus your
> compute power/electricity used) go to waste?

I perfectly understand that but cant do anything about it. But
currently it seams that i need to have a work around for that. And that
work around is trying to give a upper limit for cpu usage of my
process.

I know it feels that the overload protection is overdoing its job, but
sadly i cant do anything about it. It is my process that i can do
something about.

- Manik

rsmanik@gmail.com wrote:
>          My objective here is that i have a cpu intensive process and I
> dont want my system's idle time to go below 20%.

put your program to sleep(3C) 20% of its time?

rsmanik@gmail.com wrote:
>         I am so skeptical about the method i follow to control the CPU
> utilization of a process. Is there anyother way that is more effective
> and is not ad-hoc ?
>
>          I am using solaris 9.

FWIW, there is an OpenSolaris project called CPU Caps:

http://www.opensolaris.org/os/project/rm/rctls/cpu-caps/

And you may also want to join the rm-discuss mailing list...

Rayson


>
>          I have hit a brick wall and any help would be greatly
> appreciated.
> 
> Cheers
> Manik

Oscar del Rio wrote:
> rsmanik@gmail.com wrote:
> >          My objective here is that i have a cpu intensive process and I
> > dont want my system's idle time to go below 20%.
>
> put your program to sleep(3C) 20% of its time?

    But, that doesnt solve my problem. If i put it to sleep for 2 in 10
seconds then for the 8 seconds it would be taking 100% CPU.

rsmanik@gmail.com wrote:
> No, actually it cant be. There has to be a buffer of 10% cpu always
> avaliable for incoming calls or higher priority processes. I also
> understand that the overload protection is not doing what is actually
> expected. But i cant fix it.

What does the overloead protection do when idle goes below 10%?
Kill processes? Reboot the server? Send a report?

> The overload protection is out of my scope and has already been
> deployed.

So somebody else took full responsibility of process management.
Let him/her deal with the results >;->

> I know it feels that the overload protection is overdoing its job, but
> sadly i cant do anything about it. It is my process that i can do
> something about.

One possible solution would be to assign proper priorities to the
different process groups, with the despatcher for urgent tasks on
top. That way everything would work as designed and the scheduler
would be the controlling instance again.

Jens
-- 
Jens Goerke, mobilcom Server Systeme
mobilcom Communicationstechnik GmbH, Hollerstr. 126, 24782 Buedelsdorf
Tel: +49-(0)4331-69-5505 Fax: +49-(0)4331-69-5599

rsmanik@gmail.com writes:


>Oscar del Rio wrote:
>> rsmanik@gmail.com wrote:
>> >          My objective here is that i have a cpu intensive process and I
>> > dont want my system's idle time to go below 20%.
>>
>> put your program to sleep(3C) 20% of its time?

>    But, that doesnt solve my problem. If i put it to sleep for 2 in 10
>seconds then for the 8 seconds it would be taking 100% CPU.

Any program uses 100% of the CPU some percentage of the time; you
could make something with nanosleep() and setitimer().

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

rsmanik@gmail.com wrote:

> j.goerke@mobilcom.de wrote:
> 
>>rsmanik@gmail.com wrote:
>>
>>>        My objective here is that i have a cpu intensive process and I
>>>dont want my system's idle time to go below 20%.
>>
>>So you've got some real-time processing needs that could arise at any
>>moment? For all other practical purposes idle time isn't really relevant.
>>
>>
>>>       I am so skeptical about the method i follow to control the CPU
>>>utilization of a process. Is there anyother way that is more effective
>>>and is not ad-hoc ?
>>
>>Take a look at renice and related commands.
>>
> 
> Thanks for the quick response.
> 
> Using renice i can only lower the priority. Which means, when there are
> no other processes competing for CPU my process can take upto 100% CPU,
> which is exactly what i am trying to avoid.
> 
> I should be able to make my process use only a defined % of CPU and not
> more than that. 
> 
> - Manik
> 

What is the point of using only 60% of an otherwise idle CPU????

If you use renice to lower your priority you process will not get in the 
way of any process with a higher priority.

I have run compute intensive processes at very low priority and had them 
using 99% plus of the CPU without interfering at all with the 
interactive users; when they needed CPU time, they got it.

"Richard B. Gilbert"  wrote:

> I have run compute intensive processes at very low priority and had them 
> using 99% plus of the CPU without interfering at all with the 
> interactive users; when they needed CPU time, they got it.

Sadly, the same is not true of memory. The low priority process still
causes a lot of swapping in the interactive user's programs when the
computer has a limited amount of memory. I don't know about other
resources, but I guess cpu is the only one where priority has an effect.

j.goerke@mobilcom.de wrote:
> rsmanik@gmail.com wrote:
> > No, actually it cant be. There has to be a buffer of 10% cpu always
> > avaliable for incoming calls or higher priority processes. I also
> > understand that the overload protection is not doing what is actually
> > expected. But i cant fix it.
>
> What does the overloead protection do when idle goes below 10%?
> Kill processes? Reboot the server? Send a report?

   It stops incoming requests. You can compare it to a webserver, not
handling any more requests as the cpu load is already high.

> > The overload protection is out of my scope and has already been
> > deployed.
>
> So somebody else took full responsibility of process management.
> Let him/her deal with the results >;->

   ge :-) I guess it would be better for them to banish my process as
it is a lower priority one. 


- Manik

rsmanik@gmail.com wrote:
> j.goerke@mobilcom.de wrote:
>> What does the overloead protection do when idle goes below 10%?
>> Kill processes? Reboot the server? Send a report?
>   It stops incoming requests. You can compare it to a webserver, not
> handling any more requests as the cpu load is already high.

So the idea of having the request handler run at a really high
priority would be defeated. At least with an Apache webserver the
requests get queued instead of rejected, for some time.

If the requests get denied, is the other station able to retry
them at a later time, preferably automatically?

>> So somebody else took full responsibility of process management.
>> Let him/her deal with the results >;->
>   ge :-) I guess it would be better for them to banish my process as
> it is a lower priority one. 

You could try and get approval for another server, to combine both
into a cluster with applications distributed between them, and the
bonus of being able to run all of them a bit slower on one server in
case the other one's not available.

Jens
-- 
Jens Goerke, mobilcom Server Systeme
mobilcom Communicationstechnik GmbH, Hollerstr. 126, 24782 Buedelsdorf
Tel: +49-(0)4331-69-5505 Fax: +49-(0)4331-69-5599

Marc <marc.glisse@gmail.com> writes:

>"Richard B. Gilbert"  wrote:

>> I have run compute intensive processes at very low priority and had them 
>> using 99% plus of the CPU without interfering at all with the 
>> interactive users; when they needed CPU time, they got it.

>Sadly, the same is not true of memory. The low priority process still
>causes a lot of swapping in the interactive user's programs when the
>computer has a limited amount of memory. I don't know about other
>resources, but I guess cpu is the only one where priority has an effect.

Limiting the CPU to < 90% doesn't help in that case.  (In fact, if it
runs the system out of memory, it will likely run with < 80% CPU anyway)

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

rsmanik@gmail.com writes:

>   It stops incoming requests. You can compare it to a webserver, not
>handling any more requests as the cpu load is already high.

So it measures the wrong things; it should measure request latency
and number of outstanding requests; not CPU use.

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

On Wed, 6 Dec 2006, rsmanik@gmail.com wrote:

>     As i told in my previous post, i have an overload protection
> mechanism that will be triggered if the idle time is lower than 10 %.
> And my process is a low priority one that uses all the CPU time that is
> avaliable, hence the idle time plummets to 0. This creates problems as
> the overload protection is triggered.

If I were you, I'd rethink this "protection mechanism".  Idle time is
a poor indicator of how overstressed the machine is.  The load
average would be a better indicator.

-- 
Rich Teer, SCSA, SCNA, SCSECA, OpenSolaris CAB member

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-group.com/rich

Casper H.S. Dik wrote:
> rsmanik@gmail.com writes:
>
> >   It stops incoming requests. You can compare it to a webserver, not
> >handling any more requests as the cpu load is already high.
>
> So it measures the wrong things; it should measure request latency
> and number of outstanding requests; not CPU use.
>

Frankly, I understand and feel it myself that overload protection isnt
actually doing overload protection. But it runs in my client's machine.
And i just cant do anything about it.

But what i can do, it to control the CPU limitation of my low priority
process. But i dont have a working solution. I have tried putting the
process to sleep, sending SIGSTOP /SIGCONT to the process when it
consumes more CPU resources.

>From what i have read and heard, it isn't possible in solaris 9.  And
from all the replies i have got here, it only increases my worst fears
that it is not possible in solaris 9. I guess if i understand right,
everyone wants me to work the overload protection out, as i cant do
anything to control the amount of CPU my process uses.

Thanks a lot for you help though. 

- Manik

rsmanik@gmail.com wrote:

> Casper H.S. Dik wrote:
> 
>>rsmanik@gmail.com writes:
>>
>>
>>>  It stops incoming requests. You can compare it to a webserver, not
>>>handling any more requests as the cpu load is already high.
>>
>>So it measures the wrong things; it should measure request latency
>>and number of outstanding requests; not CPU use.
>>
> 
> 
> Frankly, I understand and feel it myself that overload protection isnt
> actually doing overload protection. But it runs in my client's machine.
> And i just cant do anything about it.
> 
> But what i can do, it to control the CPU limitation of my low priority
> process. But i dont have a working solution. I have tried putting the
> process to sleep, sending SIGSTOP /SIGCONT to the process when it
> consumes more CPU resources.
> 
>>From what i have read and heard, it isn't possible in solaris 9.  And
> from all the replies i have got here, it only increases my worst fears
> that it is not possible in solaris 9. I guess if i understand right,
> everyone wants me to work the overload protection out, as i cant do
> anything to control the amount of CPU my process uses.
> 
> Thanks a lot for you help though. 
> 
> - Manik
> 

I've been in the computer business for a little more than 40 years and 
have worked with a number of different operating systems and I don't 
recall ever encountering one that will do what you want!

There are priority schedulers that allocate the CPU based on program 
priority and class schedulers that allocate resources based on a 
programs CPU vs. I/O balance.  Someone may have invented some other type 
of scheduler but, if so, it is not at all common.

Nothing you have said really explains why you cannot use priority 
scheduling with your process getting a low priority and "requests" 
getting a high priority.  Most multi-tasking/multi-processsing O/Ss seem 
to handle this sort of thing to everyone's satisfaction.

On 2006-12-06, rsmanik@gmail.com <rsmanik@gmail.com> wrote:
>
> Casper H.S. Dik wrote:
>> rsmanik@gmail.com writes:
>>
>> >   It stops incoming requests. You can compare it to a webserver, not
>> >handling any more requests as the cpu load is already high.
>>
>> So it measures the wrong things; it should measure request latency
>> and number of outstanding requests; not CPU use.
>
> Frankly, I understand and feel it myself that overload protection isnt
> actually doing overload protection. But it runs in my client's machine.
> And i just cant do anything about it.
>
> But what i can do, it to control the CPU limitation of my low priority
> process.

Except, as you've discovered: you can't.

If you have more than one CPU, then you could create a processor set
that excludes 20% of the CPUs and then run your process within that set.

Ceri
-- 
That must be wonderful!  I don't understand it at all.
                                                  -- Moliere

Hi,
    That is exactly what i was trying. Our clusters have a minimum of 2
processors. In most of the cases 8/16. All i did was use pbind to bind
my process to one process
and
"viola"

It uses no more than 100/no of processors %.

I saw that this feature is available in solaris 10 while i followed the
link provide by Rayson. I guess i overlooked that. pbind is available
in solaris 9.

And it was able to provide me trying to provide an upper limit to my
process. Though it cannot be configured to any value, i can configure
it to 100/no processors %. I guess that solves my issue.

Thanks everyone.

Ceri Davies wrote:
> On 2006-12-06, rsmanik@gmail.com <rsmanik@gmail.com> wrote:
> >
> > Casper H.S. Dik wrote:
> >> rsmanik@gmail.com writes:
> >>
> >> >   It stops incoming requests. You can compare it to a webserver, not
> >> >handling any more requests as the cpu load is already high.
> >>
> >> So it measures the wrong things; it should measure request latency
> >> and number of outstanding requests; not CPU use.
> >
> > Frankly, I understand and feel it myself that overload protection isnt
> > actually doing overload protection. But it runs in my client's machine.
> > And i just cant do anything about it.
> >
> > But what i can do, it to control the CPU limitation of my low priority
> > process.
>
> Except, as you've discovered: you can't.
>
> If you have more than one CPU, then you could create a processor set
> that excludes 20% of the CPUs and then run your process within that set.
>
> Ceri
> --
> That must be wonderful!  I don't understand it at all.
>                                                   -- Moliere

rsmanik@gmail.com wrote:
> 
> Casper H.S. Dik wrote:
>> rsmanik@gmail.com writes:
>>
>> >         My objective here is that i have a cpu intensive process and I
>> >dont want my system's idle time to go below 20%.
>>
>> Why?  This seems like something you should the scheduler for;
>> give the process a very low priority so it'll only run when
>> there's no other work to be done.
> 
>    As i told in my previous post, i have an overload protection
> mechanism that will be triggered if the idle time is lower than 10 %.
> And my process is a low priority one that uses all the CPU time that is
> avaliable, hence the idle time plummets to 0. This creates problems as
> the overload protection is triggered.
> 
>   Again, my objective is to set an upper limit for the amount of CPU
> my process can use. 

Fundamentally, this seems like a flawed objective. You've built a tool to
monitor this flawed objective, and you're looking for a way to implement
avoid triggering the monitor.

It would be much easier to let the process run at a low priority and if it
takes 100% of the CPU because there's nothing else to run, then that's just
what you WANT it to do! I can't imagine why you'd want to guarantee that
you're never going to use all of your computer.

Colin