I have a couple of instances of performance problems with web
applications.  I'm looking for general suggestions on how to
determine the source of the problems.  Both are with apache web
web servers running on SPARC hardware under Solaris 9.

The first is an e-mail application using mod_php.  People complain
that pages are slow to load and refresh.  I'd like to determine
where most of the delay for HTTP transactions is taking place.

The second uses perl CGI scripts that do Sybase transactions.
During intervals of high activity, the load average on the machine
becomes excessive.  In this case, I'd like to determine what is
causing the high load average.  I'm assuming that there is a
bottle neck of some sort.  These intervals of high demand only
occur a few times a year, and last for a day or two.


-- 
-Gary Mills-    -Unix Support-    -U of M Academic Computing and Networking-

In article <cvfn8p$9o1$1@canopus.cc.umanitoba.ca>,
 mills@cc.umanitoba.ca (Gary Mills) wrote:

> I have a couple of instances of performance problems with web
> applications.  I'm looking for general suggestions on how to
> determine the source of the problems.  Both are with apache web
> web servers running on SPARC hardware under Solaris 9.
> 
> The first is an e-mail application using mod_php.  People complain
> that pages are slow to load and refresh.  I'd like to determine
> where most of the delay for HTTP transactions is taking place.
> 
> The second uses perl CGI scripts that do Sybase transactions.
> During intervals of high activity, the load average on the machine
> becomes excessive.  In this case, I'd like to determine what is
> causing the high load average.  I'm assuming that there is a
> bottle neck of some sort.  These intervals of high demand only
> occur a few times a year, and last for a day or two.

Do you have any experience in system performance tuning and capacity 
planning?  How about database tuning?  

Do you have any sort of performance monitoring software running on the 
system to collect utilization statistics for trend analysis, like for 
example, sar.  Without information on what your system is doing when 
people complain "it's slow", you really can't do anything useful other 
than say "try again later".

You might start with the basic.  Adrian Cockroft's book, although dated 
and not directly relevant to web applications, still will give you 
basics of how Solaris does what it does.  You can apply this, along with 
performance monitoring information from tools, to determine where the 
bottleneck is on your system.

Some links:

http://www.sun.com/980922/tuning2/about.html
http://www.sun.com/sun-on-net/itworld/UIR010329cockcroftletters.html
http://www.sun.com/sun-on-net/performance/se3/
http://www.sun.com/951001/columns/adrian/column2.html

One thing to keep in mind, many complaints about performance are 
frequently due to the application either not being designed to scale or 
being poorly designed from the start.  Until you have data that points 
to Solaris being the problem, I'd focus on what you're doing in the php 
code and the CGI/Sybase code.  Chances are, it's there.

-- 
DeeDee, don't press that button!  DeeDee!  NO!  Dee...

In <vilain-A32073.11314922022005@comcast.dca.giganews.com> Michael Vilain <vilain@spamcop.net> writes:

>In article <cvfn8p$9o1$1@canopus.cc.umanitoba.ca>,
> mills@cc.umanitoba.ca (Gary Mills) wrote:

>> I have a couple of instances of performance problems with web
>> applications.  I'm looking for general suggestions on how to
>> determine the source of the problems.  Both are with apache web
>> web servers running on SPARC hardware under Solaris 9.
>> 
>> The first is an e-mail application using mod_php.  People complain
>> that pages are slow to load and refresh.  I'd like to determine
>> where most of the delay for HTTP transactions is taking place.
>> 
>> The second uses perl CGI scripts that do Sybase transactions.
>> During intervals of high activity, the load average on the machine
>> becomes excessive.  In this case, I'd like to determine what is
>> causing the high load average.  I'm assuming that there is a
>> bottle neck of some sort.  These intervals of high demand only
>> occur a few times a year, and last for a day or two.

>Do you have any experience in system performance tuning and capacity 
>planning?  How about database tuning?  

Some, and our DBMs do as well.

>Do you have any sort of performance monitoring software running on the 
>system to collect utilization statistics for trend analysis, like for 
>example, sar.  Without information on what your system is doing when 
>people complain "it's slow", you really can't do anything useful other 
>than say "try again later".

In the case of the e-mail application, I don't think that statistics
will help.  Response to HTTP transactions is always slow, even when
the machine is idle.  I'd just like to determine what portion of the
server-side processing is occupying most of the time.

In the case of the Sybase transactions, I agree that collecting
statistics at the time of the high demand would be a useful approach.
I'm going to suggest that.  I don't know if much can be done in
between these occasions.

>You might start with the basic.  Adrian Cockroft's book, although dated 
>and not directly relevant to web applications, still will give you 
>basics of how Solaris does what it does.  You can apply this, along with 
>performance monitoring information from tools, to determine where the 
>bottleneck is on your system.

Yes, I've read that book.  What I'm really looking for is process
instrumentation.  Last time we had a similar problem, I just used
`truss' with timestamps, and discovered that the e-mail application
was attempting to authenticate to the IMAP server with an incorrect
administrator password on every transaction.  The server imposes a
time delay after authentication failures.  This caused very slow
response to transactions.

>Some links:

>http://www.sun.com/980922/tuning2/about.html
>http://www.sun.com/sun-on-net/itworld/UIR010329cockcroftletters.html
>http://www.sun.com/sun-on-net/performance/se3/
>http://www.sun.com/951001/columns/adrian/column2.html

Thanks.

>One thing to keep in mind, many complaints about performance are 
>frequently due to the application either not being designed to scale or 
>being poorly designed from the start.  Until you have data that points 
>to Solaris being the problem, I'd focus on what you're doing in the php 
>code and the CGI/Sybase code.  Chances are, it's there.

The people who designed the application want to solve the problem
by throwing away Solaris and the SPARC hardware, replacing it with
a Linux cluster on Intel hardware.  I'd prefer to identify the real
problem, which may indeed turn out to be poor design.  That's why
I'm looking for diagnostic techniques.

-- 
-Gary Mills-    -Unix Support-    -U of M Academic Computing and Networking-

"Gary Mills" <mills@cc.umanitoba.ca> wrote in message
news:cvg811$1cg$1@canopus.cc.umanitoba.ca...
> >One thing to keep in mind, many complaints about performance are
> >frequently due to the application either not being designed to scale or
> >being poorly designed from the start.  Until you have data that points
> >to Solaris being the problem, I'd focus on what you're doing in the php
> >code and the CGI/Sybase code.  Chances are, it's there.
>
> The people who designed the application want to solve the problem
> by throwing away Solaris and the SPARC hardware, replacing it with
> a Linux cluster on Intel hardware.  I'd prefer to identify the real
> problem, which may indeed turn out to be poor design.  That's why
> I'm looking for diagnostic techniques.

If the system is slow when load is low, it's probably not the system, rather
it's the applications themselves, and a new OS and hardware aren't going to
solve the problem.

If you're seeing slow response on an idle machine, then the application is
basically either timing out against something (typically a network service,
much like you noted with IMAP. Has DNS changed recently at all?), or you
have a nasty contention issue with one process locking the other out. I
wouldn't so much as suspect the latter simply because most processes aren't
that time consuming, and you'd run into the contention problem more at a
higher load than on an idle machine, unless, of course, the machine is idle
even with lots of active connections, then it may truly be a problem that
you're simply serializing all of your users through something.

Otherwise you'd think you'd see a lot of I/O if it was a badly configured
data intensive task.

Since you're running PHP and Apache (I'm assuming 1.3, but who knows),
you're not bouncing in to any of the different threading implementations
that Solaris supports because Apache isn't threaded, and neither is PHP (I
don't think -- maybe it is today).

The hard part is somehow instrumenting your applications. I'd focus on the
PHP part and perhaps throw the DBA's at the Sybase part. Most DBMS
applications bog at the DBMS (even the badly written ones), but if the DBA's
can't find anything glaring, then you'll need to tear the CGI's open and
looking at them.

With Apache and PHP you should have a reasonable chance of using Truss again
if you like, since most of the Apache children are fairly static (and if you
want you can configure Apache to simply pre-fork all it's every going to
pre-fork and truss the children, assuming it's running PHP internally as
mod_php of course).

Otherwise, you'll need help from the PHP developers to help instrument the
application.

Regards,

Will Hartung
(willh@msoft.com)