f



No more memory available

Hi,

In a loop I am pushing data on a stack and popping it later.  In the 
first part of the calculation I push more than I pop and it eats up all 
the physical memory, and eats also up all the virtual memory available 
for the App.

No more memory available.
Mathematica kernel has shut down.
Try quitting other applications and then retry.

It is with OSX 10.3.5 and Mathematica 5.0.1 on a G4 with 2GB SDRAM.  I 
am wondering what technics are used by others in similar situation.  Do 
you start to write out to the file system your accumulated unprocessed 
data and read it back later for further processing ?

Theoretically I can re-arrange the program so after some amount of 
push, pop all which was pushed and process them before I push more, but 
that requires some rearrangement of the flow of computation and I try 
to avoid that.

I also found Utilities`MemoryConserve` which I will try out.  Any other 
suggestions ?

Thanks ahead,

J�nos
------
"..because Annushka has already bought sunflower oil, and not only 
bought it, but spilled it too."
Bulgakov:  Master and Margarita

0
janos.lobb (370)
10/6/2004 8:49:01 AM
comp.soft-sys.math.mathematica 28821 articles. 0 followers. Follow

3 Replies
590 Views

Similar Articles

[PageSpeed] 40

Hi,

what do you use as stack, a simple list or a folded list, i.e.,

$stack={a,b,c,d}

or

$stack={a,{b,{c,{d,{}}}}}

??

Regards
  Jens

"J�nos" <janos.lobb@yale.edu> schrieb im Newsbeitrag 
news:ck0bht$ns5$1@smc.vnet.net...
> Hi,
>
> In a loop I am pushing data on a stack and popping it later.  In the
> first part of the calculation I push more than I pop and it eats up all
> the physical memory, and eats also up all the virtual memory available
> for the App.
>
> No more memory available.
> Mathematica kernel has shut down.
> Try quitting other applications and then retry.
>
> It is with OSX 10.3.5 and Mathematica 5.0.1 on a G4 with 2GB SDRAM.  I
> am wondering what technics are used by others in similar situation.  Do
> you start to write out to the file system your accumulated unprocessed
> data and read it back later for further processing ?
>
> Theoretically I can re-arrange the program so after some amount of
> push, pop all which was pushed and process them before I push more, but
> that requires some rearrangement of the flow of computation and I try
> to avoid that.
>
> I also found Utilities`MemoryConserve` which I will try out.  Any other
> suggestions ?
>
> Thanks ahead,
>
> J�nos
> ------
> "..because Annushka has already bought sunflower oil, and not only
> bought it, but spilled it too."
> Bulgakov:  Master and Margarita
> 


0
kuska (2791)
10/7/2004 9:31:17 AM
It is stack={{{{{},d},c},b},a}.

In the meantime I measured the depth of the stack and it is normal, I 
mean it is mostly below 100 during a run. So, I was wrong in my 
assumption that my stack is getting too deep.   My apology...   I ran 
out of memory, because the results - different length of strings - I 
collect in a simple list is getting too big.
I will try to compress or pack them before I add them to the list.

In the meantime as I investigated the Depth[of different objects], I 
found that I can crash the kernel with no exception on both of my 
machines with the following one liner:

Depth[Fold[List, "", Table["", {i, 87363}]]]

I looked the Book, but found no limit on nesting other than 
$RecursionLimit, but that one does not apply here looks like anyway.  
So the biggest depth an object can have without crashing the kernel in 
5.0.1 is 87361.  Interestingly it is a multiplicand of two primes 199 
and 439.

Thanks a lot,

J�nos




On Oct 7, 2004, at 5:25 AM, Jens-Peer Kuska wrote:

> Hi,
>
> what do you use as stack, a simple list or a folded list, i.e.,
>
> $stack={a,b,c,d}
>
> or
>
> $stack={a,{b,{c,{d,{}}}}}
>
> ??
>
> Regards
>   Jens
>
> "J�nos" <janos.lobb@yale.edu> schrieb im Newsbeitrag
> news:ck0bht$ns5$1@smc.vnet.net...
>> Hi,
>>
>> In a loop I am pushing data on a stack and popping it later.  In the
>> first part of the calculation I push more than I pop and it eats up 
>> all
>> the physical memory, and eats also up all the virtual memory available
>> for the App.
>>
>> No more memory available.
>> Mathematica kernel has shut down.
>> Try quitting other applications and then retry.
>>
>> It is with OSX 10.3.5 and Mathematica 5.0.1 on a G4 with 2GB SDRAM.  I
>> am wondering what technics are used by others in similar situation.  
>> Do
>> you start to write out to the file system your accumulated unprocessed
>> data and read it back later for further processing ?
>>
>> Theoretically I can re-arrange the program so after some amount of
>> push, pop all which was pushed and process them before I push more, 
>> but
>> that requires some rearrangement of the flow of computation and I try
>> to avoid that.
>>
>> I also found Utilities`MemoryConserve` which I will try out.  Any 
>> other
>> suggestions ?
>>
>> Thanks ahead,
>>
>> J�nos
>> ------
>> "..because Annushka has already bought sunflower oil, and not only
>> bought it, but spilled it too."
>> Bulgakov:  Master and Margarita
>>
>
>
----------------------------------------------
Trying to argue with a politician is like lifting up the head of a 
corpse.
(S. Lem: His Master Voice)

0
janos.lobb (370)
10/8/2004 7:05:08 AM
J�nos wrote:
> It is stack={{{{{},d},c},b},a}.
> 
> In the meantime I measured the depth of the stack and it is normal, I 
> mean it is mostly below 100 during a run. So, I was wrong in my 
> assumption that my stack is getting too deep.   My apology...   I ran 
> out of memory, because the results - different length of strings - I 
> collect in a simple list is getting too big.
> I will try to compress or pack them before I add them to the list.
> 
> In the meantime as I investigated the Depth[of different objects], I 
> found that I can crash the kernel with no exception on both of my 
> machines with the following one liner:
> 
> Depth[Fold[List, "", Table["", {i, 87363}]]]
> 
> I looked the Book, but found no limit on nesting other than 
> $RecursionLimit, but that one does not apply here looks like anyway.  
> So the biggest depth an object can have without crashing the kernel in 
> 5.0.1 is 87361.  Interestingly it is a multiplicand of two primes 199 
> and 439.
> 
> Thanks a lot,
> 
> J�nos
> [...]

In many places in the Mathematica code base recursion is used. Depth is 
an obvious candidate. This has the unfortunate effect of exploding the 
subroutine stack when manipulating deeply nested expressions.

Several years ago (for version 3.0.1, I believe) I fixed this for a few 
important and commonly used functions such as Flatten, LeafCount, and 
our expression freeing code, by using an explicit stack internally to 
avoid the operating system (and maybe compiler) dependent subroutine 
stack. (At least one or two other people contributed to that work so I 
don't want to claim all the credit, or blame, in case people noticed a 
slight slowdown.) Anyway, this was not done for all imaginable 
functions. Possibly we should consider handling a few more in that way. 
Offhand I'm not sure how common it is to do such things as Depth on 
deeply nested expressions.

I do not know if there is some OS-dependent significance to the fact 
that 87361 = 199*439, but clearly it had to be a product of SOME primes.


Daniel Lichtblau
Wolfram Research


0
danl1 (936)
10/9/2004 8:31:51 AM
Reply: