Hi,
I've upgraded to texlive2009 and now my pdflatex runs are extremely
slow. I'm not saying this has anything to do with texlive except that
I must have messed up my configuration somehow during the upgrade.

I've got an 8,000 page document that's taking a little over an hour to
do a single compile using pdflatex. At first it speeds along nicely;
stdout streaming like a firehose, but it gets slower over time...by
the time it's on page 4000, the stdout actually stops and starts,
sometimes taking seconds to move again.

The places it stops aren't at included images. The same doc used to
compile much faster before I upgraded.

Any ideas on what I should look for in the configuration?

BTW, the machine is quad-core, 8cpu, amd64 (FreeBSD 6.3)

thanks,
--Tim Arnold

Tim Arnold wrote:
> Hi,
> I've upgraded to texlive2009 and now my pdflatex runs are extremely
> slow. I'm not saying this has anything to do with texlive except that
> I must have messed up my configuration somehow during the upgrade.
> 
> I've got an 8,000 page document that's taking a little over an hour to
> do a single compile using pdflatex. At first it speeds along nicely;
> stdout streaming like a firehose, but it gets slower over time...by
> the time it's on page 4000, the stdout actually stops and starts,
> sometimes taking seconds to move again.
> 
> The places it stops aren't at included images. The same doc used to
> compile much faster before I upgraded.
> 
> Any ideas on what I should look for in the configuration?
> 
> BTW, the machine is quad-core, 8cpu, amd64 (FreeBSD 6.3)
> 
> thanks,
> --Tim Arnold

I think it denpends on on the complexity of your document. (the use of 
hyperref etc.)

I have a similar experience in my LaTeX book where the index takes a 
very long time to compile when hyperref is enabled.

/daleif

On Tue, 02 Mar 2010 20:28:54 +0100, Lars Madsen <daleif@imf.au.dk>
wrote:

>Tim Arnold wrote:
>> Hi,
>> I've upgraded to texlive2009 and now my pdflatex runs are extremely
>> slow. I'm not saying this has anything to do with texlive except that
>> I must have messed up my configuration somehow during the upgrade.
>> 
>> I've got an 8,000 page document that's taking a little over an hour to
>> do a single compile using pdflatex. At first it speeds along nicely;
>> stdout streaming like a firehose, but it gets slower over time...by
>> the time it's on page 4000, the stdout actually stops and starts,
>> sometimes taking seconds to move again.
>> 
>> The places it stops aren't at included images. The same doc used to
>> compile much faster before I upgraded.
>> 
>> Any ideas on what I should look for in the configuration?
>> 
>> BTW, the machine is quad-core, 8cpu, amd64 (FreeBSD 6.3)
>> 
>> thanks,
>> --Tim Arnold
>
>I think it denpends on on the complexity of your document. (the use of 
>hyperref etc.)
>

This doesn't explain that the same document now
takes much longer. I expect one of the following

 1. pdftex has changed and takes more memory than before.
 2. a package has changed (perhaps hyperref) that now 
    takes more memory than before.
 3. A bug in the management of memory in pdftex.

Certainly pdftex has changed in between tl2008 and 2009, 
but whether that explains the slowdown I don't know. The 
reason I suspect memory is that the slowdown happens 
after so many pages. Usually main memory is pretty well 
cleared after each page, but in the pdf format, some 
data is written only at the end of the file. 

Another bottleneck is disk access. Some packages write 
many temporary files and re-read them. This would cause 
the stated behavior only if these disk accesses began
around page 4000.

I dare say this is an extreme case that the tl2009 
developers never encountered. A bug report to the 
list might be in order (run "texdoc texlive" and read 
section "Getting Help" to find the bug report address).


Dan
To reply by email, change LookInSig to luecking

On Mar 2, 3:30=A0pm, Dan Luecking <LookIn...@uark.edu> wrote:
> On Tue, 02 Mar 2010 20:28:54 +0100, Lars Madsen <dal...@imf.au.dk>
> wrote:
>
>
>
>
>
> >Tim Arnold wrote:
> >> Hi,
> >> I've upgraded to texlive2009 and now my pdflatex runs are extremely
> >> slow. I'm not saying this has anything to do with texlive except that
> >> I must have messed up my configuration somehow during the upgrade.
>
> >> I've got an 8,000 page document that's taking a little over an hour to
> >> do a single compile using pdflatex. At first it speeds along nicely;
> >> stdout streaming like a firehose, but it gets slower over time...by
> >> the time it's on page 4000, the stdout actually stops and starts,
> >> sometimes taking seconds to move again.
>
> >> The places it stops aren't at included images. The same doc used to
> >> compile much faster before I upgraded.
>
> >> Any ideas on what I should look for in the configuration?
>
> >> BTW, the machine is quad-core, 8cpu, amd64 (FreeBSD 6.3)
>
> >> thanks,
> >> --Tim Arnold
>
> >I think it denpends on on the complexity of your document. (the use of
> >hyperref etc.)
>
> This doesn't explain that the same document now
> takes much longer. I expect one of the following
>
> =A01. pdftex has changed and takes more memory than before.
> =A02. a package has changed (perhaps hyperref) that now
> =A0 =A0 takes more memory than before.
> =A03. A bug in the management of memory in pdftex.
>
> Certainly pdftex has changed in between tl2008 and 2009,
> but whether that explains the slowdown I don't know. The
> reason I suspect memory is that the slowdown happens
> after so many pages. Usually main memory is pretty well
> cleared after each page, but in the pdf format, some
> data is written only at the end of the file.
>
> Another bottleneck is disk access. Some packages write
> many temporary files and re-read them. This would cause
> the stated behavior only if these disk accesses began
> around page 4000.
>
> I dare say this is an extreme case that the tl2009
> developers never encountered. A bug report to the
> list might be in order (run "texdoc texlive" and read
> section "Getting Help" to find the bug report address).
>
> Dan
> To reply by email, change LookInSig to luecking

I will make a few more test runs to get more info and then report to
the developers. I just tried setting draft mode on hyperref with the
same results as before. This is what I'm seeing from the 'top' command
during the compile:

CPU states: 12.3% user,  0.0% nice,  0.2% system,  0.1% interrupt,
87.3% idle
Mem: 210M Active, 6463M Inact, 269M Wired, 11M Cache, 214M Buf, 8606M
Free
Swap: 30G Total, 30G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
COMMAND
28771 tiarno        1 101    0 80492K 56384K CPU5   7  43:52 97.71%
pdftex

pdftex takes nearly 100% of any particular cpu (there are 8 cpus on
this machine).

From that data, it appears that memory is not a problem--the numbers
don't change very much throughout the compilation. When I check the IO
stats, it almost always says the same thing:

  PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT
COMMAND
28771 tiarno          0     24      0      0      0      0   0.00%
pdftex

with very occasional 100% usage on the WRITE.

I'll keep testing for a while and report here if I find out what my
own problem is, or to the developers if I get nowhere.

thanks,
--Tim

Tim Arnold <a_jtim@bellsouth.net> writes:

> Hi,
> I've upgraded to texlive2009 and now my pdflatex runs are extremely
> slow. I'm not saying this has anything to do with texlive except that
> I must have messed up my configuration somehow during the upgrade.

Did you remake the format files?


--
Michael D. Sofka               sofkam@rpi.edu
C&MT Sr. Systems Programmer,   Email, HPC, TeX, Epistemology
Rensselaer Polytechnic Institute, Troy, NY.  http://www.rpi.edu/~sofkam/

Tim Arnold wrote:

> I will make a few more test runs to get more info and then report to
> the developers.

My guess is that there's a macro somewhere that has been changed in a 
way that greatly increases its running time.  It might be something to 
do with strings, or searching in arrays.

Here's something you might try.  Interupt the run perhaps 10 times while 
it is slow to see what is executing.  If the interrupts mostly give 
similar traceback stack then perhaps there is your culprit.


Jonathan

On Mar 3, 4:56=A0am, Jonathan Fine <J.F...@open.ac.uk> wrote:
> Tim Arnold wrote:
> > I will make a few more test runs to get more info and then report to
> > the developers.
>
> My guess is that there's a macro somewhere that has been changed in a
> way that greatly increases its running time. =A0It might be something to
> do with strings, or searching in arrays.
>
> Here's something you might try. =A0Interupt the run perhaps 10 times whil=
e
> it is slow to see what is executing. =A0If the interrupts mostly give
> similar traceback stack then perhaps there is your culprit.
>
> Jonathan

Thanks for that suggestion Jonathan. The first 2000 pages go by in
about 4 minutes and things get progressively slower. When I interrupt
the compile at points where it seems to dawdle for a second or two, I
get two types of traceback. Well, sometimes it's writing the
intermediate chapter toc files for minitoc which can take 2-5 seconds
each time.  But except for that, these are the two trackbacks that
appear:

(1)
MT@setupfont ...lias fi MT@tracking MT@check@font
                                                  ifMT@inlist@ else
MT@vinfo...

(2)
MT@in@clist ...xpandafter MT@res@a expandafter ,#2
                                                  ,#1,@empty @nnil

The second one is the likeliest, but I have no idea what it means.
Just to add a bit more data, (pdf version 1.5, compress-level=3D3,
although it makes little difference in timing)

draft mode:
no verbatim input files, no graphics: 6.35 minutes, 14.5mb
verbatim input files, no graphics: 12.17 minutes, 16.5 mb
verbatim and graphic input files: 15.45 minutes, 16.9mb

final mode (with verbatim and graphic files):
use draft for hypersetup only: 59 minutes, 122.7mb
no draft on hypersetup: 1.12 hours 124.6mb

minute  pages
1          800
2         1300
3         1750
4         2100
5         2450
6         2670
7         2790
8         3000
9         3220
10       3360
11       3385
..
13       3700
15       3920
etc.

Still tinkering and thinking.
thanks,
--Tim

On 03.03.10 22:07, Tim Arnold wrote:
> (1)
> MT@setupfont ...lias fi MT@tracking MT@check@font
>                                                    ifMT@inlist@ else
> MT@vinfo...
>
> (2)
> MT@in@clist ...xpandafter MT@res@a expandafter ,#2
>                                                    ,#1,@empty @nnil
>
> The second one is the likeliest, but I have no idea what it means.

These commands are from microtype (it also accepts the draft option, so 
you can verify whether it's really the culprit). \MT@check@font calls 
\MT@in@clist to check whether the font was already set up by matching it 
against a comma-separated list of fonts. This list can get quite long, 
as it will contain all fonts that have been used so far (where every 
combination of NFSS axes/sizes counts as a different font). For example, 
at the end of microtype.dtx, it contains 110 fonts, but without any 
noticeable slowdown.
So either your font setup is enormously complex, with thousands of 
fonts, or there's a bug in microtype so that it adds fonts repeatedly to 
the list (which I cannot reproduce). Maybe you could add \tracingmacros1 
to page 2000 to see what's in the font list. (Sorry for asking this, but 
this would be another possibility: you did not put \tracingmacros1 
anywhere in your document?) Which version of microtype are you using?

Regards,
-- 
  Robert

Am 04.03.2010 02:22, schrieb Robert:
> On 03.03.10 22:07, Tim Arnold wrote:
>> (1)
>> MT@setupfont ...lias fi MT@tracking MT@check@font
>> ifMT@inlist@ else
>> MT@vinfo...
>>
>> (2)
>> MT@in@clist ...xpandafter MT@res@a expandafter ,#2
>> ,#1,@empty @nnil
>>
>> The second one is the likeliest, but I have no idea what it means.
>
> These commands are from microtype (it also accepts the draft option, so
> you can verify whether it's really the culprit). \MT@check@font calls
> \MT@in@clist to check whether the font was already set up by matching it
> against a comma-separated list of fonts. This list can get quite long,
> as it will contain all fonts that have been used so far (where every
> combination of NFSS axes/sizes counts as a different font). For example,
> at the end of microtype.dtx, it contains 110 fonts, but without any
> noticeable slowdown.
> So either your font setup is enormously complex, with thousands of
> fonts, or there's a bug in microtype so that it adds fonts repeatedly to
> the list (which I cannot reproduce). Maybe you could add \tracingmacros1
> to page 2000 to see what's in the font list. (Sorry for asking this, but
> this would be another possibility: you did not put \tracingmacros1
> anywhere in your document?) Which version of microtype are you using?

Could it possibly be microtype's font expansion? What happenes if you 
try \usepackage[expansion=false]{microtype}.
The font expansion feature quite enormously blows the pdf documents up, 
which is the reason why I normally do not use it at all.

Regards,
Christian.

On Mar 4, 1:20=A0am, Christian Justen <christian.jus...@web.de> wrote:
> Am 04.03.2010 02:22, schrieb Robert:
> > On 03.03.10 22:07, Tim Arnold wrote:
> >> (1)
> >> MT@setupfont ...lias fi MT@tracking MT@check@font
> >> ifMT@inlist@ else
> >> MT@vinfo...
>
> >> (2)
> >> MT@in@clist ...xpandafter MT@res@a expandafter ,#2
> >> ,#1,@empty @nnil
>
> >> The second one is the likeliest, but I have no idea what it means.
>
> > These commands are from microtype (it also accepts the draft option, so
> > you can verify whether it's really the culprit). \MT@check@font calls
> > \MT@in@clist to check whether the font was already set up by matching i=
t
> > against a comma-separated list of fonts. This list can get quite long,
> > as it will contain all fonts that have been used so far (where every
> > combination of NFSS axes/sizes counts as a different font). For example=
,
> > at the end of microtype.dtx, it contains 110 fonts, but without any
> > noticeable slowdown.
> > So either your font setup is enormously complex, with thousands of
> > fonts, or there's a bug in microtype so that it adds fonts repeatedly t=
o
> > the list (which I cannot reproduce). Maybe you could add \tracingmacros=
1
> > to page 2000 to see what's in the font list. (Sorry for asking this, bu=
t
> > this would be another possibility: you did not put \tracingmacros1
> > anywhere in your document?) Which version of microtype are you using?
>
> Could it possibly be microtype's font expansion? What happenes if you
> try \usepackage[expansion=3Dfalse]{microtype}.
> The font expansion feature quite enormously blows the pdf documents up,
> which is the reason why I normally do not use it at all.
>
> Regards,
> Christian.

Thanks Christian, Robert, I tried expansion=3Dfalse with no difference
in compile time. But when I omit microtype completely, I get
dramatically faster results:
5 minutes for 8000 pages, and a pdf file size of 123,986,036.

So you guys really helped. Do you know who I should report my result
to in case this is a bug in microtype?

pdflatex 1.40.10 texlive 2009
microtype 2010/01/10 v2.4

I'll be glad to serve as a test runner if needed.

thanks,
--Tim

On Mar 4, 1:20=A0am, Christian Justen <christian.jus...@web.de> wrote:
> Am 04.03.2010 02:22, schrieb Robert:
>
>
>
>
>
> > On 03.03.10 22:07, Tim Arnold wrote:
> >> (1)
> >> MT@setupfont ...lias fi MT@tracking MT@check@font
> >> ifMT@inlist@ else
> >> MT@vinfo...
>
> >> (2)
> >> MT@in@clist ...xpandafter MT@res@a expandafter ,#2
> >> ,#1,@empty @nnil
>
> >> The second one is the likeliest, but I have no idea what it means.
>
> > These commands are from microtype (it also accepts the draft option, so
> > you can verify whether it's really the culprit). \MT@check@font calls
> > \MT@in@clist to check whether the font was already set up by matching i=
t
> > against a comma-separated list of fonts. This list can get quite long,
> > as it will contain all fonts that have been used so far (where every
> > combination of NFSS axes/sizes counts as a different font). For example=
,
> > at the end of microtype.dtx, it contains 110 fonts, but without any
> > noticeable slowdown.
> > So either your font setup is enormously complex, with thousands of
> > fonts, or there's a bug in microtype so that it adds fonts repeatedly t=
o
> > the list (which I cannot reproduce). Maybe you could add \tracingmacros=
1
> > to page 2000 to see what's in the font list. (Sorry for asking this, bu=
t
> > this would be another possibility: you did not put \tracingmacros1
> > anywhere in your document?) Which version of microtype are you using?
>
> Could it possibly be microtype's font expansion? What happenes if you
> try \usepackage[expansion=3Dfalse]{microtype}.
> The font expansion feature quite enormously blows the pdf documents up,
> which is the reason why I normally do not use it at all.
>
> Regards,
> Christian.

one more thing--pdffonts shows the final doc to have 43 fonts.
--Tim