Making existing process efficient

  • Permalink
  • submit to reddit
  • Email
  • Follow


Problem
--------
I have a file in which there are millions of records. I need to read
those records and output records to different
files depending on the value of fields in the record. There would be
several records that would end up getting
inserted into the same file due to similar value of fields in the
records.

Current scenario
----------------
Our process runs on a mutiproc machine. Currently it does things in
the simplest possible way. It opens several
output streams (as and when required) and keeps writing records to the
streams one by one. It appends records when the corresponding stream
is already open.

I am _trying_ to do things efficiently here. It's important to
preserve the ordering of records in the generated
files i.e. if record A1 is prior to record A2 in the input file, then
A1 should still be prior to A2 in the result
file (if they land up in the same file).

I have thought of some ways -

a) Keep writing N (sufficiently large) number of records to an
internal buffer (B1). Fork a child process and open a
pipe with the child. The child would inherit the buffer (B1) from
parent. Child would sort the records by the stream where they would be
written. It would then write the records to the destination file. It
would keep writing to pipe the last record it could write
successfully. It would exit after writing the last record. This would
give the advantage of writing records belonging to the same file in
one go. This would reduce the number of seeks that would have
otherwise been caused by random access of files.

Parent would continue writing the next set of N records to another
internal buffer (B2). It would wait for the child
process to complete before forking again. When the child has finished
it would rename B2 as B1 and fork another
process. In case the child crashes abruptly, parent would write the
unsucessful records from buffer B1 to output
files.

b) (Complex approach) Let the parent write N records to an internal
buffer (B1) and fork a child process C1. It
would continue writing N more records to buffer B2 and fork another
child process. It would open a pipe with both
the child processes.

The child process C1 would sort the internal buffer B1 by the output
stream. It would then lock all the output files
before writing. It would communciate to parent that it has locked all
the files. It would then write and relinquish
locks one by one once it's done.

The child process C2 would sort the internal buffer B2 by the output
stream. It would wait for parent to
communicater that C1 has locked all the files. C2 would then  attempt
to grab locks on files and would write to them once it gets it.

I think approach (b) is unduly complex (that too without mentioning
crash recovery of child) and may or may not get any additional
performance benefits. Measurements would only tell me how much
performance gain I get with the approaches. (I feel that probably I/O
is the bottleneck).

Is there any other approach that could help me here?

Thanks..
P.S. - I am on FreeBSD 4.10
0
Reply km_jr_usenet (66) 12/5/2007 7:38:31 PM

See related articles to this posting

On Wed, 05 Dec 2007 11:38:31 -0800, Kelvin Moss wrote:

> Problem
> --------
> I have a file in which there are millions of records. I need to read
> those records and output records to different files depending on the
> value of fields in the record. There would be several records that would
> end up getting inserted into the same file due to similar value of
> fields in the records.
> 
> Current scenario
> ----------------
> Our process runs on a mutiproc machine. Currently it does things in the
> simplest possible way. It opens several output streams (as and when
> required) and keeps writing records to the streams one by one. It
> appends records when the corresponding stream is already open.
> 
> I am _trying_ to do things efficiently here. It's important to preserve
> the ordering of records in the generated files i.e. if record A1 is
> prior to record A2 in the input file, then A1 should still be prior to
> A2 in the result file (if they land up in the same file).

.. Create X children
.. Try read Y records, until EOF
..  Find next free child[1]
..  Give read records to child, along with start/end record number
..   Child writes data into child specific files, along with record number.
.. Mergesort child specific files into real ones, via. record number.

 Then you just need to find your optimal values of X and Y.

 If the merge sort is a problem as a separate step, with a bit more work
you can have each child output to a proc. per. output file and do the
merge sort in real time[1].

 For bonus points you can try creating children on other machines.


[1] Simple loop should work if the processing per. record is roughly the
same.

[2] Need some way for the child to say "have nothing for you, processing
record Y now ... so it doesn't deadlock if one child doesn't have any
records of type Z).

-- 
James Antill -- james@and.org
C String APIs use too much memory? ustr: length, ref count, size and
read-only/fixed. Ave. 44% overhead over strdup(), for 0-20B strings
http://www.and.org/ustr/
0
Reply james-netnews (137) 12/5/2007 8:06:41 PM

Kelvin Moss wrote:
> Problem
> --------
> I have a file in which there are millions of records. I need to read
> those records and output records to different
> files depending on the value of fields in the record. There would be
> several records that would end up getting
> inserted into the same file due to similar value of fields in the
> records.
> 
> Current scenario
> ----------------
> Our process runs on a mutiproc machine. Currently it does things in
> the simplest possible way. It opens several
> output streams (as and when required) and keeps writing records to the
> streams one by one. It appends records when the corresponding stream
> is already open.

     Unless the decision of which output file (files?)
should receive a record is incredibly complicated, the
task will be completely I/O-bound.  Implication: Fancy
schemes to use the CPU more efficiently or to apply the
power of additional CPU's/cores will help very little.

     That said, for a really large amount of data it may
be worth while avoiding the need to copy everything from
kernel space out to user space and back to kernel space
again, as in a simple read()/write() scheme.  Consider
using mmap() instead, with madvise() to tell the virtual
memory system that your accesses will be sequential.  At
the very least, use mmap() for the input file even if
you continue to use write() for the outputs.

> I have thought of some ways -
> 
> a) Keep writing N (sufficiently large) number of records to an
> internal buffer (B1). Fork a child process and open a
> pipe with the child. The child would inherit the buffer (B1) from
> parent. Child would sort the records by the stream where they would be
> written. It would then write the records to the destination file. It
> would keep writing to pipe the last record it could write
> successfully. It would exit after writing the last record. This would
> give the advantage of writing records belonging to the same file in
> one go. This would reduce the number of seeks that would have
> otherwise been caused by random access of files.
> 
> Parent would continue writing the next set of N records to another
> internal buffer (B2). It would wait for the child
> process to complete before forking again. When the child has finished
> it would rename B2 as B1 and fork another
> process. In case the child crashes abruptly, parent would write the
> unsucessful records from buffer B1 to output
> files.

     "It is a capital mistake to theorize before one has
data," and I lack data.  Still, it seems to me that this
is much too involved for any benefit it might bring --
indeed, the overhead of all those fork() calls might more
than offset any gain.

     If you're using mmap() for output, there's no need
for any of this.  Just mmap() the buffers to their output
files, copy records into them, and let the virtual memory
system take care of flushing the pages to disk.  As I
mentioned before, madvise() may help the VM system make
good choices.

     If you're using write() for output, the only delay
you incur in the write() call itself is the time to copy
the data from user space to kernel space; unless you've
explicitly asked for data synchronization, the file system
will perform the physical writes at some later time.  If
delays on the output side really are a concern, you could
use multiple writer threads to gain parallelism and keep
the input side running more or less unimpeded.  This is
likely to be a good deal more efficient than using
multiple processes, especially a process-per-write()!

     If you're using write(), changing to writev() could
save some time.  With write() you'd copy data from the
input buffer to an output buffer and thence to the kernel,
while with writev() you could copy a whole bunch of
discontiguous records straight from the input buffer to
the kernel, thus saving one copy step.  For "millions of
records," the savings might be worth while.  Of course,
it requires that you not re-fill the input buffer until
all the output streams are no longer interested in any
of its content, which may complicate things a little if
you're using multiple writer threads.  "It stands to
reason" (that is, "trust, but verify") that the performance
of mmap()/writev() should be better than mmap()/copy/write(),
but probably not quite as good as mmap()/mmap()/copy.

     The big take-aways:

     - You're almost certainly I/O bound, so schemes to
       optimize the CPU are optimizing the wrong thing.

     - And yet, with "millions of records" the copy steps
       may be worth a look.  Plain read()/copy/write()
       moves each byte three times; using mmap() on the
       input eliminates one move; using mmap() or writev()
       on the output eliminates another.

     - If more parallelism is needed, use multiple threads
       rather than multiple processes.  The synchronization
       issues are the same, but the overheads are lower.

     - Along with mmap(), use madvise() to inform the
       system of the access patterns for your buffers.

     - And, as always: Measure, measure, measure!

-- 
Eric.Sosman@sun.com
0
Reply Eric.Sosman (4552) 12/5/2007 8:32:13 PM

In article <pan.2007.12.05.20.06.41@and.org>,
 James Antill <james-netnews@and.org> wrote:

> On Wed, 05 Dec 2007 11:38:31 -0800, Kelvin Moss wrote:
> 
> > Problem
> > --------
> > I have a file in which there are millions of records. I need to read
> > those records and output records to different files depending on the
> > value of fields in the record. There would be several records that would
> > end up getting inserted into the same file due to similar value of
> > fields in the records.
> > 
> > Current scenario
> > ----------------
> > Our process runs on a mutiproc machine. Currently it does things in the
> > simplest possible way. It opens several output streams (as and when
> > required) and keeps writing records to the streams one by one. It
> > appends records when the corresponding stream is already open.
> > 
> > I am _trying_ to do things efficiently here. It's important to preserve
> > the ordering of records in the generated files i.e. if record A1 is
> > prior to record A2 in the input file, then A1 should still be prior to
> > A2 in the result file (if they land up in the same file).
> 
> . Create X children
> . Try read Y records, until EOF
> .  Find next free child[1]
> .  Give read records to child, along with start/end record number
> .   Child writes data into child specific files, along with record number.
> . Mergesort child specific files into real ones, via. record number.

Sort?  The data is already in order, it seems unlikely that using a 
sorting algorithm could possibly be more efficient than what he's 
already doing.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
0
Reply barmar (6124) 12/6/2007 4:32:43 AM

On Dec 5, 12:32 pm, Eric Sosman <Eric.Sos...@sun.com> wrote:
> Kelvin Moss wrote:
>      The big take-aways:
>
>      - You're almost certainly I/O bound, so schemes to
>        optimize the CPU are optimizing the wrong thing.
>
>      - And yet, with "millions of records" the copy steps
>        may be worth a look.  Plain read()/copy/write()
>        moves each byte three times; using mmap() on the
>        input eliminates one move; using mmap() or writev()
>        on the output eliminates another.
>
>      - If more parallelism is needed, use multiple threads
>        rather than multiple processes.  The synchronization
>        issues are the same, but the overheads are lower.
>
>      - Along with mmap(), use madvise() to inform the
>        system of the access patterns for your buffers.
>
>      - And, as always: Measure, measure, measure!

Thanks Eric. This all makes a lot of sense. Here are a few questions -

1. I had thought about mmaping the input file. Unfortunately the file
in question is very big (> 4 GB). I believe it won't help much to mmap
the file on a 32 bit system.
2. I don't use writev but as I understand writev probably makes
writing more efficient if the records to be written are scattered. If
I sort my data then I won't have the problem of scattered inputs.
Would writev still bring me some adavntages?
3. I am thinking that probably opening too many files in one single
process is also slowing down the I/O. So how about this approach -
Let the parent hash the output file descriptors into M buckets and
accordingly write to M buffers. Once a buffer is full, it would fork a
child process that would sort the records based on fd and write to
files. It would effectively cut down the number of open file
descriptors per process and disk seeks too (by sorting on fd).

Your comments would definitely help me.

Thanks ..






0
Reply km_jr_usenet (66) 12/7/2007 10:20:19 PM

Kelvin Moss wrote:
> On Dec 5, 12:32 pm, Eric Sosman <Eric.Sos...@sun.com> wrote:
>> Kelvin Moss wrote:
>>      The big take-aways:
>>
>>      - You're almost certainly I/O bound, so schemes to
>>        optimize the CPU are optimizing the wrong thing.
>>
>>      - And yet, with "millions of records" the copy steps
>>        may be worth a look.  Plain read()/copy/write()
>>        moves each byte three times; using mmap() on the
>>        input eliminates one move; using mmap() or writev()
>>        on the output eliminates another.
>>
>>      - If more parallelism is needed, use multiple threads
>>        rather than multiple processes.  The synchronization
>>        issues are the same, but the overheads are lower.
>>
>>      - Along with mmap(), use madvise() to inform the
>>        system of the access patterns for your buffers.
>>
>>      - And, as always: Measure, measure, measure!
> 
> Thanks Eric. This all makes a lot of sense. Here are a few questions -
> 
> 1. I had thought about mmaping the input file. Unfortunately the file
> in question is very big (> 4 GB). I believe it won't help much to mmap
> the file on a 32 bit system.

     Map the first chunk of the file, process it, unmap it,
map the next chunk, process it, unmap it, lather, rinse,
repeat.  Choice of chunk size is up to you.

> 2. I don't use writev but as I understand writev probably makes
> writing more efficient if the records to be written are scattered. If
> I sort my data then I won't have the problem of scattered inputs.
> Would writev still bring me some adavntages?

     You keep talking about sorting your data, but in the
original post you said

 >>> It's important to
 >>> preserve the ordering of records in the generated
 >>> files i.e. if record A1 is prior to record A2 in the
 >>> input file, then A1 should still be prior to A2 in the
 >>> result file (if they land up in the same file).

.... so I don't understand why any sorting is necessary or
desirable.  Unless I've misunderstood, you're taking in one
long stream of records and splitting them between multiple
output files, preserving their original order.  Maybe you
have an input file listing every United States citizen,
sorted by name, and you want to produce fifty-something
output files also sorted by name, one for each State or
territory.  If that's not close to what you're attempting,
describe your intentions more fully.

     Anyhow, the important thing about this framework is
that you don't need to rearrange the records, just route
each to the proper output (or outputs).  You could do this
by visiting each input record in turn, still in the input
buffer, and doing a write() to the chosen file (or files).
But with a large number of records that will be a lot of
trips in and out of the kernel, and it's probably a good
idea to try to write more than one record per trip.

     First idea: Set aside a buffer area for each output
file, copy each record to its proper buffer(s) as you
encounter it, and do a write() whenever a buffer fills up.
Advantages: Fewer round-trips to the kernel, simple to code.
Drawbacks: Every record gets copied at least twice, once
to "marshal" into its buffer, and a second time to send
the data to the kernel's file system cache.

     Second idea: Use writev() instead of write(), so you
can send all the Alaska records to the Alaska output
directly from the input buffer, without copying.  Advantage:
Fewer round-trips to the kernel, simple to code, only one
copy.  Drawbacks: Must remember to *do* the pending writev()
calls before changing the mapping of (or reading more data
into) the input buffer.

     Third idea: Like the first, but with the output buffers
mmap()'ed to their output files so all you need to do is
copy the record to the buffer and let the virtual memory
system do its magic; no write() or writev().  Advantages:
Fewest round-trips to the kernel, only one copy.  Drawbacks:
Using mmap() while a file changes size is a little more work.

> 3. I am thinking that probably opening too many files in one single
> process is also slowing down the I/O.

     How many is "too many?"  I'm not familiar with the limits
of FreeBSD, but Solaris and Linux can handle tens of thousands
of open files simultaneously without trouble.  How many do
you need?

> So how about this approach -
> Let the parent hash the output file descriptors into M buckets and
> accordingly write to M buffers. Once a buffer is full, it would fork a
> child process that would sort the records based on fd and write to
> files. It would effectively cut down the number of open file
> descriptors per process and disk seeks too (by sorting on fd).

     This idea of forking seems to fascinate you in a way I
can only think of as unhealthy.  Besides, it now sounds like
you need to copy every record at least twice: Once to dump
it into one of the M buffers, and again when the child "sorts"
(that word, again) the records.

     As for avoiding disk seeks -- Well, I've been assuming
that the input and output are files, not raw devices.  If
so, the file system takes care of managing the physical I/O,
combining multiple writes to adjacent areas into single
physical writes, optimizing the seek patterns, and so on.
You can exercise some control over the FS behavior from the
application level, but it's a weakish sort of control, kind
of like pushing on a rope.

> Your comments would definitely help me.

     More background on what you're trying to do would help me.
What are these millions of records, how big are they, how
many output files are there, why all this talk about sorting?
(And why do you think fork() is free?)

-- 
Eric.Sosman@sun.com
0
Reply Eric.Sosman (4552) 12/7/2007 11:14:03 PM

On Dec 7, 3:14 pm, Eric Sosman <Eric.Sos...@sun.com> wrote:
> > 1. I had thought about mmaping the input file. Unfortunately the file
> > in question is very big (> 4 GB). I believe it won't help much to mmap
> > the file on a 32 bit system.
>
>      Map the first chunk of the file, process it, unmap it,
> map the next chunk, process it, unmap it, lather, rinse,
> repeat.  Choice of chunk size is up to you.

OK..so I would need to keep track of bytes read through mapped region
since I would not get any EOF like error, right?


> > 2. I don't use writev but as I understand writev probably makes
> > writing more efficient if the records to be written are scattered. If
> > I sort my data then I won't have the problem of scattered inputs.
> > Would writev still bring me some adavntages?

>      You keep talking about sorting your data, but in the
> original post you said
>
>  >>> It's important to
>  >>> preserve the ordering of records in the generated
>  >>> files i.e. if record A1 is prior to record A2 in the
>  >>> input file, then A1 should still be prior to A2 in the
>  >>> result file (if they land up in the same file).
>
> ... so I don't understand why any sorting is necessary or
> desirable.  Unless I've misunderstood, you're taking in one
> long stream of records and splitting them between multiple
> output files, preserving their original order.  Maybe you
> have an input file listing every United States citizen,
> sorted by name, and you want to produce fifty-something
> output files also sorted by name, one for each State or
> territory.  If that's not close to what you're attempting,
> describe your intentions more fully.

That's more or less correct. Let me give a simplified example (using
first two fields as criterion for output file) - say the input file
contains records like

A1 B1 C1 D1
A1 B2 C1 D2
A1 B1 C3 D4
A1 B3 C2 D4
A1 B3 C6 C7

So at the end, we would have output files like -
A1_B1.xtn
---------
A1 B1 C1 D1
A1 B1 C3 D4

A1_B2.xtn
---------
A1 B2 C1 D2

A1_B3.xtn
---------
A1 B3 C2 D4
A1 B3 C6 C7



>      Anyhow, the important thing about this framework is
> that you don't need to rearrange the records, just route
> each to the proper output (or outputs).  You could do this
> by visiting each input record in turn, still in the input
> buffer, and doing a write() to the chosen file (or files).
> But with a large number of records that will be a lot of
> trips in and out of the kernel, and it's probably a good
> idea to try to write more than one record per trip.

True

>      First idea: Set aside a buffer area for each output
> file, copy each record to its proper buffer(s) as you
> encounter it, and do a write() whenever a buffer fills up.
> Advantages: Fewer round-trips to the kernel, simple to code.
> Drawbacks: Every record gets copied at least twice, once
> to "marshal" into its buffer, and a second time to send
> the data to the kernel's file system cache.

I do not know in advance about the number of files that are going to
get opened. Hence that would mean to resort to malloc/new, which I
would like to avoid if I can. The overhead of dynamic memory
management for so many buffers would probably outweigh the benefits of
this approach.

>      Second idea: Use writev() instead of write(), so you
> can send all the Alaska records to the Alaska output
> directly from the input buffer, without copying.  Advantage:
> Fewer round-trips to the kernel, simple to code, only one
> copy.  Drawbacks: Must remember to *do* the pending writev()
> calls before changing the mapping of (or reading more data
> into) the input buffer.

Seems like a good approach to me. Another issue I have is that the
legacy code uses C++ iostream for doing I/O. Would there be any issue
in mixing writev with legacy code in such a case?


>      Third idea: Like the first, but with the output buffers
> mmap()'ed to their output files so all you need to do is
> copy the record to the buffer and let the virtual memory
> system do its magic; no write() or writev().  Advantages:
> Fewest round-trips to the kernel, only one copy.  Drawbacks:
> Using mmap() while a file changes size is a little more work.

Good approach but again requires to resort to dynamic memory
management (Aggravates when the file size changes). I guess it's hard
to get the best of both the worlds :-)


> > 3. I am thinking that probably opening too many files in one single
> > process is also slowing down the I/O.
>
>      How many is "too many?"  I'm not familiar with the limits
> of FreeBSD, but Solaris and Linux can handle tens of thousands
> of open files simultaneously without trouble.  How many do
> you need?

I am expecting some 5000 file handles. I am not sure on the BSD limits
but going by what you say it should not become a bottleneck.


> > So how about this approach -
> > Let the parent hash the output file descriptors into M buckets and
> > accordingly write to M buffers. Once a buffer is full, it would fork a
> > child process that would sort the records based on fd and write to
> > files. It would effectively cut down the number of open file
> > descriptors per process and disk seeks too (by sorting on fd).
>
>      This idea of forking seems to fascinate you in a way I
> can only think of as unhealthy.  Besides, it now sounds like
> you need to copy every record at least twice: Once to dump
> it into one of the M buffers, and again when the child "sorts"
> (that word, again) the records.
>      As for avoiding disk seeks -- Well, I've been assuming
> that the input and output are files, not raw devices.  If

That's correct.

> so, the file system takes care of managing the physical I/O,
> combining multiple writes to adjacent areas into single
> physical writes, optimizing the seek patterns, and so on.

Any idea if older Unices also emply such optimizations (like
FreeBSD4.x)?

> You can exercise some control over the FS behavior from the
> application level, but it's a weakish sort of control, kind
> of like pushing on a rope.

OK

> > Your comments would definitely help me.
>
>      More background on what you're trying to do would help me.
> What are these millions of records, how big are they, how
> many output files are there, why all this talk about sorting?
> (And why do you think fork() is free?)

I have tried to give some background on the problem. The records are
essentially user data records where each record size is variable
(unlike the example that I gave above). I agree with you that it's
essentially an I/O bound process. But we don't seem to use the
multiproc usage even if it's available. Indeed fork isn't free but may
be faster with multiproc around.

Thanks again, your comments would definitely help.




0
Reply km_jr_usenet (66) 12/9/2007 6:33:16 AM

In article 
<1b3b4120-20dd-4a14-acdf-00f45a09ca21@a35g2000prf.googlegroups.com>,
 Kelvin Moss <km_jr_usenet@yahoo.com> wrote:

> On Dec 7, 3:14 pm, Eric Sosman <Eric.Sos...@sun.com> wrote:
> > so, the file system takes care of managing the physical I/O,
> > combining multiple writes to adjacent areas into single
> > physical writes, optimizing the seek patterns, and so on.
> 
> Any idea if older Unices also emply such optimizations (like
> FreeBSD4.x)?

I think most OSes in the past 15-20 years are similar.

> > You can exercise some control over the FS behavior from the
> > application level, but it's a weakish sort of control, kind
> > of like pushing on a rope.
> 
> OK
> 
> > > Your comments would definitely help me.
> >
> >      More background on what you're trying to do would help me.
> > What are these millions of records, how big are they, how
> > many output files are there, why all this talk about sorting?
> > (And why do you think fork() is free?)
> 
> I have tried to give some background on the problem. The records are
> essentially user data records where each record size is variable
> (unlike the example that I gave above). I agree with you that it's
> essentially an I/O bound process. But we don't seem to use the
> multiproc usage even if it's available. Indeed fork isn't free but may
> be faster with multiproc around.

Your application is totally sequential, so there's not really any way to 
benefit from multiple processors at this level.

Behind the scenes the kernel will make use of multiprocessors when 
actually performing the disk I/O.  When your process calls write(), the 
data will simply be copied to a kernel buffer, and any available CPU 
will eventually perform the actual write to the disk.  There may also be 
prefetching, so while your CPU is processing one record, another CPU may 
start the disk read for the next record, and it will already be in the 
kernel buffer by the time your application looks for it.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
0
Reply barmar (6124) 12/9/2007 11:21:28 PM
comp.unix.programmer 10605 articles. 64 followers. Post

7 Replies
60 Views

Similar Articles

[PageSpeed] 27

  • Permalink
  • submit to reddit
  • Email
  • Follow


Reply:

Similar Artilces:

Help making log processing script more efficient
I am currently running a cron job against a web log every minute to see which ips are accessing pages most often. An example output of the log file is: 22.242.52.22 - - [25/Feb/2006:18:59:59 -0500] "GET /dir/page.htm HTTP/1.1" 200 27223 "http://www.domain.com/referrer/page.htm" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.10) Gecko/20050716 Firefox/1.0.6" The script I use is: tail -3000 access_log | #get last 3000 lines from log file (approximately last 3 minutes from current time) grep `date +%d/%b/%Y:%H:%M -d "1 minute ago"` | #get only l...

Being an efficient coder, can VIM make me more efficient?
Hi, all, I'm a programmer (Windows) and now I'm very efficient on coding in Windows style editors like Delphi IDE or other editors. But sometimes I feel dull to do some repeating work such like add a continuous number before each line, remove some pattern text from each line. I think VIM maybe more effective in that area. I have been using VIM now and then for some days, but I've only known the very basically command such like i, dd, o, etc. I think it will take me long time to master VIM. My question is, is it worth for me to try to use VIM instead of other edi...

Child Process Exited while Making Connection to Remote Process
I'm a relative newcomer to MPI. For my class project for my Computer Networks class I'm running a Phylogenetic Analysis program that splits the work out to several different processes, and I'm doing it with MPI to get the parallelism. After writing my code I tried it on an input that should only require one additional processor, so I ran it with just four processors at its disposal. However, I got the output listed below. Does anybody know what this means? I'd include my source code but it's fairly large. I'm cur- rently trying to capture the problem in a smaller ...

Win32::Process, SetProcessAffinityMask for an existing process = perl crash
Hi! Was planning to use Win32::Process to set the ProcessAffinityMask of some processes but this lead to pure and simple crash of perl.exe. This is not a credential problem as I can set the process affinity if I spawn a "process.exe" command to set the affinity. Also, the process affinity is correctly set to the expected value... Seems the crash occurs just after. Here is a sample code causing a perl.exe crash (for example using activeperl 5.6 or 5.8 (latest) on Windows 2000, XP or 2003) : ------------------ use strict; use Win32::Process; use Win32; my $obj; my $pid=2216; ...

!process output post processing script; does one exist?
After getting the !process output, it occured to me that it would be interesting to write a perl script to parse resultant text to build some interesting statistics regarding relative usages of UserTime, KernelTime, pool usage, working set sizes, etc. Before I write one, does anyone know if one already exists? Thanks, Tony Turner ...

NTVDM As A New Process Not A Child Process OF Existing Application
I AHVE A PROBLEM IN RUNNING THE NTVDM AS A SEPARATE PROCESS ,INSTEAD OF THE CHILD PROCESS. I HAVE APPLICATION FROM WHERE I RUN THE WIN 16 BIT APPLICATIONS,FOR THAT NTVDM IS REQUIURED ,BUT THE PROBLEM IS THAT THE NTVDM RUN AS CHILD PROCESS OF THE MAIN APPLICATION. I USE THE " CREATEPROCESS() " API AND ALSO TRIED WITH "SHELLEXECUTE()" API BUT STILL NOT SUCCESSFUL. RetVal = CreateProcess(0&, aProgram, 0&, 0&, 0&, CREATE_NEW_PROCESS_GROUP, 0&, 0&, startinfo, procinfo) I TRIED LOT OF OPTIOINS THAT EXIST.BUT NO SOLUTION. ******TH...

p4_error: Child process exited while making connection to remote process
Hi I am trying to execute mpi jobs on a cluster ,but i m getting an error p4_error: Child process exited while making connection to remote process I have checked the rsh with the other nodes which is asking for password Can the problem will be solved if we do rsh without password? Plz suggest me as early as possible . Regards Jo ...

Child Process Exited while Making Connection to Remote Process #2
I'm a relative newcomer to MPI. For my class project for my Computer Networks class I'm running a Phylogenetic Analysis program that splits the work out to several different processes, and I'm doing it with MPI to get the parallelism. After writing my code I tried it on an input that should only require one additional processor, so I ran it with just four processors at its disposal. However, I got the output listed below. Does anybody know what this means? I'd include my source code but it's fairly large. I'm cur- rently trying to capture the problem in a smaller ...

more efficient
Which is more efficient: Select * from table1 where id in (select id from table2) or Select * from table1 where exists(select * from table2 where table2.id=table1.id) On Wed, 11 Aug 2004 14:53:52 +0100, Trev@Work wrote: >Which is more efficient: > >Select * from table1 where id in (select id from table2) > >or > >Select * from table1 where exists(select * from table2 where >table2.id=table1.id) Hi Trev, That question has no one correct answer; it depends on lots of factors, such as table structures, whether there are indexes, etc. If you really want to know, yo...

make this more efficient?
Is there a more efficient way to do the following code? I tryed reading the file into an array then working with it, but this is what i came up with. For this to work i need to execute it and direct it to an output file like ../code.pl > FS.new I'd appreciate any pointers. Thanks #!/usr/bin/perl open (fs, "FS")|| die "can't open file"; open (fsnew, ">FSYUTBLU.test")|| die "can't open file"; while (<fs>) { s/(.{144})/$1\n/g,$_; #insert newline every 144 chars in the file FS print (fsnew); } close(fsnew); open(fsnew,"FSYU...

if process.exist then . . . ???
im trying to monitor a process, if this process is closed then i want to write a time and date to a logfile... i dont know how to do it but it should be as simple as this... (improper) loop { if process.exist('notepad.exe') then puts 'notepad exist' else my_file = File.new("c:\log.txt", APPEND) my_file.puts "here is the time add date, notepad has been killed" end } -- Posted via http://www.ruby-forum.com/. Here is what i have so far... i have been playing with this for a while now... im getting no were ,-( require 'rubygems'...

exists processing
Could someone explain how this is processed by the database? I'm having trouble formulating these correctly because I don't quite understand how the database processes exists. select count(*) from table1 where (locationid like '%SOMEVALUE1%') and (exists (select job from table2 where (( contains(description,' $Carpenter ') > 0 )) and (job in (select job from jobtable where job is not null and locationid=table1.locationid)))) Actually if there is a good explanation on how to methodically take these suckers apart and figure out how the database processes them tha...

Verify process existance
I want to write a script to send a page if one of 4 processes on our system isn't running. Finding the processes is easy, 4 lines: ps -ef | grep proc1 ps -ef | grep proc2 etc. I could even pipe them to a file, ps -ef | grep proc1 | grep -v grep > jobs.txt ps -ef | grep proc2 | grep -v grep >> jobs.txt ps -ef | grep proc3 | grep -v grep >> jobs.txt ps -ef | grep proc4 | grep -v grep >> jobs.txt I'd like to be able to specify which process isn't runn...

How do you make this matrix efficiently
I would appreciate some help on this one, since I have been struggeling for months to make it go faster. I have a matrix X, size [m,n]. For an element X(i,j), its neighbours are defined to be {X(i+1,j),X(i-1,j),X(i,j+1),X(i,j-1)} (of course, some have only two neighbours(corner points), and some have three(border(but not corner) points)). I need to make a new matrix Y generated by X the following way. For every two neighbours x_i and x_j of X, there will be two equations : x_i - x_j - w_ij = 0 x_j - x_i - w_ij = 0 Where Y is the coefficient matrix on the left hand side. An exampel will ...

How to make this program more efficient?
SUBJECT: How to make this program more efficient? In my program, a thread will check update from server periodically and generate a stl::map for other part of this program to read data from. Let's name the update method as doUpdate and stl::map read methods as getData and copyData. Since stl::map is not thread-safe, we should do synchronization by ourselves. A usable solution is to create a boost::mutex::scoped_lock object in all above methods to make the access to all methods synchronized. But since the doUpdate method will be executed periodically and the interval may be 1 hour or longe...

Make join more efficient
I have a query that involves joining 4 tables. Each of the tables has many columns but my query involves a small subset of the columns from each of the 4 tables. Would it be more efficient if I created 4 temporary tables, each containing the columns of interest and then doing the join on the temporary tables? My original query is fast enough, I just want to be kind to the MySQL server. Thank you, Joe On 3/21/2011 1:55 PM, Joe Hesse wrote: > I have a query that involves joining 4 tables. Each of the tables has > many columns but my query involves a small subset o...

make this snippet efficient
/* Target - read an integer from a binary file */ unsigned int Byte2Int(char *buff) { unsigned char* byte = reinterpret_cast<unsigned char*> (buff); return ((byte[0]<<24)|(byte[1]<<16)|(byte[2]<<8)|(byte[3])); } /* part of main funciton */ ifstream fp("in.bin",ios::binary); char buff[4]; fp.read(buff,4); unsigned int loadSize = Byte2Int(buff); Thank you. KK KK wrote: > /* Target - read an integer from a binary file */ > unsigned int Byte2Int(char *buff) > { > unsigned char* byte = reinterpret_cast<unsigned char*> (buff); > return ((b...

file processing efficiency?
I've got something like this going(reduced to an example) a=20090601 cat ../data/C19.dat | \ awk -v compdate="$a" '{ while ("date --date="$4 " +%Y%m%d"|getline aa) { if(aa >= compdate) print $0 else exit } }' Thing is, I know approximately how records would be processed before it exits. Each dat file is about 800 lines and it would be 220 lines give or take 10, before it exits. So, would I save any time by using head -230 or simply cat the file as above? sivaram -- Sivaram Neelakantan wrote: > I&#...

Do kernel processes exist
Hi, In user space we have user processes and user threads (LWPs) within these processes. But, do kernel processes exist? Kernel threads are created with "kernel_thread", but do these threads run in the context of a kernel process? Or do kernel threads always run on behalf of a user process or lightweight process? As far as I know a thread (kernel or user) always has to run in the context of a process. If there are kernel processes, how do you create one? Thanks and regards, Markus markus wrote: > Hi, > > In user space we have user processes and user threads (LWPs) wi...

making a loop more efficient...
Im sure this has been covered before but i cant find anything relavant anywhere- I want to calculate an average value of 80 different values, 30 times.....this code produces the right answer but takes a long time and im sure there must be a simpler way of doing it.....cheers! J for n=1:30 for i=1:80 x=[tempyp(i,n)]; b(i)=mean(x); end a(n)=mean(b); end On 27 Jan, 14:22, "J B" <tbrowning...@hotmail.com> wrote: > Im sure this has been covered before but i cant find anything relavant anywhere- > > I want to calculate an average value of 80 d...

Make my code more efficient.
How can I simplify my code? I have a if then structure, but I thought The code below the if then clasues should be more effective, but it doesn&#8217;t work. Any suggestions how to get sort this problem. Kind regards Jens if myLight==1 lighting flat; elseif myLight==2 lighting gouraud elseif myLight==3 lighting phong; elseif myLight==4 lighting none end %mylightSelection={@flat;@gouraud;@phong;@none} %lighting (mylightSelection{myLight}) I don't think it's possible without EVAL, and using EVAL in this case would be stupid...

Check process existence
Hi all, I'd like to check if a specific process with a given pid exists. I went through the archives of the mailing list and I found a message dated in 2001, with this piece of code : Process p = Runtime.getRuntime().exec("/bin/kill -0 " + pid); p.waitFor(); boolean alive = p.exitValue() == 0; I tried it, but I get an IOException : java.io.IOException: CreateProcess: \bin\kill -0 3828 error=3 at java.lang.ProcessImpl.create(Native Method) at java.lang.ProcessImpl.<init>(Unknown Source) at java.lang.ProcessImpl.start(Unknown Source) at java.lang.ProcessBuilder.start(...

More efficient array processing
Hello, I'm trying to do the following: datagrid = numpy.zeros(360,180,3,73,20) But I get an error saying that the dimensions are too large? Is there a memory issue here? So, my workaround is this: numpoint = 73 datagrid = numpy.zeros(360,180,3,73,1) for np in range(numpoint): datagrid[:,:,:,np,0] = datagrid[:,:,:,np,0] + concgrid[:,:,:,np,0] But this is SLOW.. what can I do to increase efficiency here? Is there a way to create the larger array? The program loops through several days actually, filling the 5th dimension. Eventually I just sum the 5th dimension anyway (as done in t...

Page Processing Efficiency
If I had a page that was being generated using coldfusion from queries to an oracle table would it be better response time: A) pulling the all the data using 1 query and iterating over the same result table multiple time producing desired arrays, or B) pulling specific data using specific queries with less production of arrays and proccessing after te data was returned? In the example I had 2 date fields called start and comp in a table with other like data and wanted to know what the min value was from both fields. So I have to "select * from the table where mynum='131&#...