f



Compiler farm with remote compile output?

I am compiling my application for a number of unix platforms. So far it's 
been going quite well doing this manually but with time the number of 
different builds can increase to more then 10 distributions. Naturally it is 
a hassle to monitor each screen to see if the compile suceeded and with no 
warnings or errors.

Thus I'm interested in automating this to some extent. E.g. it's common that 
some warning comes up on a specific platform. I'd like to capture this text 
(which is usually sent to stderr by gcc) and send it to a common machine for 
analyzing. This would make it easier for me to just start a compile and 
forget about it.
How would I be able to do such redirection? I don't nessecarily need a fully 
administration studio but it would be nice to start a compile manually on 
e.g. 5 machines and just monitor any bad compiles at one common place.
Any tools for maintaining multiple machines would naturally be interesting. 
E.g. I'm sure that Microsoft, Redhat or other large corporations don't have 
a person monitoring their actual compile machines. Instead there must be 
some central administration.

Any recommendations on how this could be done?

Thanks in advance.
-- Henrik 


0
Henrik
7/24/2006 4:39:18 PM
comp.unix.programmer 10848 articles. 0 followers. kokososo56 (350) is leader. Post Follow

4 Replies
671 Views

Similar Articles

[PageSpeed] 22

Henrik Goldman wrote:
> I am compiling my application for a number of unix platforms. So far it's
> been going quite well doing this manually but with time the number of
> different builds can increase to more then 10 distributions. Naturally it is
> a hassle to monitor each screen to see if the compile suceeded and with no
> warnings or errors.
>
> Thus I'm interested in automating this to some extent. E.g. it's common that
> some warning comes up on a specific platform. I'd like to capture this text
> (which is usually sent to stderr by gcc) and send it to a common machine for
> analyzing. This would make it easier for me to just start a compile and
> forget about it.
> How would I be able to do such redirection? I don't nessecarily need a fully
> administration studio but it would be nice to start a compile manually on
> e.g. 5 machines and just monitor any bad compiles at one common place.

The simplest way is surely just to ssh from one machine, kick your
compiles off in separate windows.

Or this can also be automated easily, e.g.
    for HOST in host1 host2 ; do
        ssh batchuser@$HOST 'cd builddir; make' 2&> $HOST.log &
    done

> Any tools for maintaining multiple machines would naturally be interesting.
> E.g. I'm sure that Microsoft, Redhat or other large corporations don't have
> a person monitoring their actual compile machines. Instead there must be
> some central administration.
>
> Any recommendations on how this could be done?

There are high tech solutions, look into cluster management tools like
OSCAR. The suggestions above are just a quick and dirty way to get
started.

> 
> Thanks in advance.
> -- Henrik

0
toby
7/24/2006 7:03:25 PM
"> The simplest way is surely just to ssh from one machine, kick your
> compiles off in separate windows.
>
> Or this can also be automated easily, e.g.
>    for HOST in host1 host2 ; do
>        ssh batchuser@$HOST 'cd builddir; make' 2&> $HOST.log &
>    done
>

One problem against that is that the build can sometimes span between alot 
of machines that aren't nessecarily turned on at the same time or located in 
same place. Also some of the machines are as old as can be and only 
understands telnet remote connections. Compiling software like ssh is 
mostlikely doable but can cost alot of time since it's not guarantied to 
compile that easily.

Would it be possible to redirect stderr to a file for the shell (by 
modifying the makefile) and then send it out from the machine itself? I 
don't nessecarily need that builds are centrally initiated. Instead it's 
more important that results are collected.

> There are high tech solutions, look into cluster management tools like
> OSCAR. The suggestions above are just a quick and dirty way to get
> started.
>

Besides Oscar doesn't seem to support all the platforms I use.

Thanks in advance.

-- Henrik 


0
Henrik
7/26/2006 4:45:54 AM
I just walk by and want to say something  , wish can help you !
I think you can just implement a pair of program , one act as a server
in the collecting data machine and another act as a client in the
compling machine .

When the client haved run , it create a named pipe and redirect his
stdin as the pipe and your makefile redirect all your compile mesages
to the named pipe and then be forward to the machine collecting data .

Or is there something like socket file , in which there is some similar
mechanism ?

0
struggle
7/26/2006 10:39:45 AM
Henrik Goldman wrote:
> "> The simplest way is surely just to ssh from one machine, kick your
> > compiles off in separate windows.
> >
> > Or this can also be automated easily, e.g.
> >    for HOST in host1 host2 ; do
> >        ssh batchuser@$HOST 'cd builddir; make' 2&> $HOST.log &
> >    done
> >
>
> One problem against that is that the build can sometimes span between alot
> of machines that aren't nessecarily turned on at the same time or located in
> same place. Also some of the machines are as old as can be and only
> understands telnet remote connections.

rsh/rexec would work equally well.

> Compiling software like ssh is
> mostlikely doable but can cost alot of time since it's not guarantied to
> compile that easily.
>
> Would it be possible to redirect stderr to a file for the shell (by
> modifying the makefile) and then send it out from the machine itself? I
> don't nessecarily need that builds are centrally initiated. Instead it's
> more important that results are collected.

You could put build logs on an nfs mount. Or the method I suggested --
remote shell and redirect to a local file on the initiating server.

>
> > There are high tech solutions, look into cluster management tools like
> > OSCAR. The suggestions above are just a quick and dirty way to get
> > started.
> >
>
> Besides Oscar doesn't seem to support all the platforms I use.
> 
> Thanks in advance.
> 
> -- Henrik

0
toby
7/26/2006 7:14:24 PM
Reply: