Hi all,

do you have any experience concerning printing speed?

FMP7 and FMP8 where severely broken: PDFs where much bigger than
required. The bug was to include font definitions again and again,
causing file sizes with 100 times the size as required.

FMP6 was working pretty well, both fast and small.

I now gave FMP9 a try. The print size is ok. But the speed is
extremely slow - both for sorting records, as well as for printing.

Before I had about 2 hours for several hundred documents and several
thousand pages (from > 1 M records). FMP9 used 30 hours for this task!

I did not run any comparison yet, taking a single table approach for
..fp7 only. A dump from .fp7 to .txt takes several minutes. An import to
FP6 takes several minutes as well. An import to FP7/8/9 takes several
hours!?

I feel that a find is faster nowadays - but it does not help that much
while sort and print are that slow.

- Martin

On May 5, 8:16 am, Martin Trautmann <t-...@gmx.net> wrote:
> Hi all,
>
> do you have any experience concerning printing speed?
>
> FMP7 and FMP8 where severely broken: PDFs where much bigger than
> required. The bug was to include font definitions again and again,
> causing file sizes with 100 times the size as required.
>
> FMP6 was working pretty well, both fast and small.
>
> I now gave FMP9 a try. The print size is ok. But the speed is
> extremely slow - both for sorting records, as well as for printing.
>
> Before I had about 2 hours for several hundred documents and several
> thousand pages (from > 1 M records). FMP9 used 30 hours for this task!

Is this on a solution that was simply upgraded from FM6 to FM9?
Or did you re-build and/or optimize the solution in FM9?

The architecture shift from FM6 to FM9 is huge. Solutions designed in
FM6 often run miserably slow in later versions of FM. In my experience
i find FM9  very fast, but you absolutely have to code for it. In FM6
inter-table operations necessitated hopping from window to window from
file to file. In FM9 for example these hops are quite costly, but also
completely unnecessary, as you can import all the layouts table
occurrences and scripts for a task into a single file.

The new record locking semantics can also be a source of major
performance problems. But these too are resolveable by rewriting the
problem areas with FM9 optimized code.

-cheers,
Dave

On Mon, 5 May 2008 14:46:44 -0700 (PDT), d-42 wrote:
>  Is this on a solution that was simply upgraded from FM6 to FM9?
>  Or did you re-build and/or optimize the solution in FM9?

It was rebuilt and optimized for FMP7.

>  The architecture shift from FM6 to FM9 is huge. Solutions designed in
>  FM6 often run miserably slow in later versions of FM. 

What kind of solutions? 
>  In my experience
>  i find FM9  very fast, but you absolutely have to code for it.

I don't have any clue how you would prefer to code different for those
tasks. There's some garbage within by now - such as temporary scripts of
some calculated fields. But none of those should cause a major slowdown.

>  In FM6
>  inter-table operations necessitated hopping from window to window from
>  file to file. 

Those were cumbersome, but sometimes were much easier to handle. 

>  In FM9 for example these hops are quite costly, but also
>  completely unnecessary, as you can import all the layouts table
>  occurrences and scripts for a task into a single file.

Which should speed up operations - and doesn't. I'm still annoyed how
slow searches within other tables are - within the same file.
I feel there is still a major design flaw.

>  The new record locking semantics can also be a source of major
>  performance problems. But these too are resolveable by rewriting the
>  problem areas with FM9 optimized code.

All of this does not explain why sort and print are that slow.
I feel that FMP7/8/9 was rewritten completely from scratch - loosing
performance optimisation or basic functionality on several occasions.
Some of this is justified somehow. Some should have become better
instead of worse. Some is not essential for my needs and I'd prefer to
be able to disable it (e.g. Asian language support)

- Martin

Martin Trautmann <t-use@gmx.net> writes:


> [...] I now gave FMP9 a try [...]

How did you migrate your solution from 6 to 9? Did you tidy up the
file references and printer settings saved in scripts?

-Jens

-- 
Free PlugIn for Regular Expressions with FileMaker:
http://jensteich.de/regex-plugin/

On May 5, 3:09 pm, Martin Trautmann <t-...@gmx.net> wrote:
> On Mon, 5 May 2008 14:46:44 -0700 (PDT), d-42 wrote:
> >  Is this on a solution that was simply upgraded from FM6 to FM9?
> >  Or did you re-build and/or optimize the solution in FM9?
>
> It was rebuilt and optimized for FMP7.
>
> >  The architecture shift from FM6 to FM9 is huge. Solutions designed in
> >  FM6 often run miserably slow in later versions of FM.
>
> What kind of solutions?

Pretty much any big project suffered and needed to be rewritten or at
least selectively rewritten for FM7+.

> >  In my experience
> >  i find FM9  very fast, but you absolutely have to code for it.
>
> I don't have any clue how you would prefer to code different for those
> tasks. There's some garbage within by now - such as temporary scripts of
> some calculated fields. But none of those should cause a major slowdown.

You need to use the 'commit records' step in places where you didn't
need to use it before, for a trivial example. (I've seen looping
scripts that went from minutes to seconds with a couple well placed
commit records steps.)


....etc...

> All of this does not explain why sort and print are that slow.

When you say sort and print are 'slow'. What do you mean specifically?

You open up a file, and perform a basic sort on a single indexed field
in the table attached to the current layout, and its orders of
magnitude slower than it was on FM6? Under what conditions is sorting
so much slower?

> I feel that FMP7/8/9 was rewritten completely from scratch - loosing
> performance optimisation or basic functionality on several occasions.
> Some of this is justified somehow. Some should have become better
> instead of worse. Some is not essential for my needs and I'd prefer to
> be able to disable it (e.g. Asian language support)

Agreed on some points. I don't really want to have to browse through
the Kanji date functions either. And would love to just hide that
whole category of features 99% of the time. :)

And its inevitable that FM9 uses more RAM and  resources, but that's
not a big deal... we have more RAM now than we did when FM6 was
current.

I haven't noted any 'basic functionality' taken out though. Please
elaborate.

when I evaluated 7 I had a ton of problems, performance was often
abysmall, and 7 had some killer bugs in it too. But I've been able to
fix all the areas that were slow, and overall I'm very happy with 9.

-cheers,
Dave

On Tue, 06 May 2008 00:24:25 +0200, Jens Teich wrote:
>  Martin Trautmann <t-use@gmx.net> writes:
> 
> 
> > [...] I now gave FMP9 a try [...]
> 
>  How did you migrate your solution from 6 to 9? Did you tidy up the
>  file references and printer settings saved in scripts?

No, they where built within in FMP7. After realising that FMP7 was
broken, I rebuilt it in FMP6.

On Mon, 5 May 2008 16:02:57 -0700 (PDT), d-42 wrote:
> > All of this does not explain why sort and print are that slow.
> 
>  When you say sort and print are 'slow'. What do you mean specifically?

As said before, 2 hours compared to > 24 hours

>  You open up a file, and perform a basic sort on a single indexed field
>  in the table attached to the current layout, and its orders of
>  magnitude slower than it was on FM6? Under what conditions is sorting
>  so much slower?

No - I use one file, copy certain field contents, 

then I go to the main file (external script), enter find mode, paste,
add some extra values, sort the results, print them to a pdf, rename
this pdf and exit this external script

to loop to the next record, copy, and continue as above.

>  I haven't noted any 'basic functionality' taken out though. Please
>  elaborate.

It's not that basic functionality was taken out (apart from some
exceptions, such as search for line break), but that former basic
functions where working faster than they do now. Maybe the FMP 5/6 code
wos optimised for those tasks, maybe FMP 7/8/9 isn't.

>  when I evaluated 7 I had a ton of problems, performance was often
>  abysmall, and 7 had some killer bugs in it too. But I've been able to
>  fix all the areas that were slow, and overall I'm very happy with 9.

I feel that FMP9 is what FMP 7.2 should have been. I still like .fp7,
while I use different tools for my other needs. I never tried to use
those for printing up to now, but while FMP9 is working ok now, it's
just too slow.

- Martin

Martin Trautmann <t-use@gmx.net> writes:

> On Tue, 06 May 2008 00:24:25 +0200, Jens Teich wrote:
>>  Martin Trautmann <t-use@gmx.net> writes:
>> 
>> 
>> > [...] I now gave FMP9 a try [...]
>> 
>>  How did you migrate your solution from 6 to 9? Did you tidy up the
>>  file references and printer settings saved in scripts?
>
> No, they where built within in FMP7. After realising that FMP7 was
> broken, I rebuilt it in FMP6.

If you tried the 7 version with 9 it would be worth a try to make a
new conversion with 9. The conversion tool was strongly improved from
7 to 8.

Jens

On Tue, 06 May 2008 01:40:43 +0200, Jens Teich wrote:
>  Martin Trautmann <t-use@gmx.net> writes:
> > No, they where built within in FMP7. After realising that FMP7 was
> > broken, I rebuilt it in FMP6.
> 
>  If you tried the 7 version with 9 it would be worth a try to make a
>  new conversion with 9. The conversion tool was strongly improved from
>  7 to 8.

Convert from 7 to 9? They still use the same format .fp7, more or less

Martin Trautmann <t-use@gmx.net> writes:

> On Tue, 06 May 2008 01:40:43 +0200, Jens Teich wrote:
>>  Martin Trautmann <t-use@gmx.net> writes:
>> > No, they where built within in FMP7. After realising that FMP7 was
>> > broken, I rebuilt it in FMP6.
>> 
>>  If you tried the 7 version with 9 it would be worth a try to make a
>>  new conversion with 9. The conversion tool was strongly improved from
>>  7 to 8.
>
> Convert from 7 to 9? They still use the same format .fp7, more or less

No from 6 to 9, you said you didn't really work with the 7 version.

Jens

On Tue, 06 May 2008 02:00:27 +0200, Jens Teich wrote:
> > On Tue, 06 May 2008 01:40:43 +0200, Jens Teich wrote:
> >>  Martin Trautmann <t-use@gmx.net> writes:
> >> > No, they where built within in FMP7. After realising that FMP7 was
> >> > broken, I rebuilt it in FMP6.
> >> 
> >>  If you tried the 7 version with 9 it would be worth a try to make a
> >>  new conversion with 9. The conversion tool was strongly improved from
> >>  7 to 8.
> >
> > Convert from 7 to 9? They still use the same format .fp7, more or less
> 
>  No from 6 to 9, you said you didn't really work with the 7 version.

I did a lot of work in FMP 7 - but I rebuilt the solution in FMP 6 as long
as FMP 7 printing  was broken. It was still broken in FMP 8. I tried FMP
9 recently, but I do not use FMP 9 yet.

They broke something different in FMP 8 and FMP 9: Within any version up
to FMP 7 fixed width fonts where fixed width. FMP 7 had a broken font
rendering at the beginning, but this was fixed later on. However, FMP 7
is the only version, where fixed width remained as such.

http://www.fa-technik.adfc.de/Codierung/Codier-Anbieter.png is an image
that was built from ASCII graphics within FMP 7. You can't use ASCII
graphics any longer, based on font size 1 (up to FMP 6) or font size 2
(from FMP 7 on) since FMP 8 upwards do use different width spacing or
kerning even for fixed width fonts.

Thus there are certain jobs where I have to go back to FMP 7 (as long as
I do not use ImageMagic yet for this job).

- Martin

On May 5, 4:33 pm, Martin Trautmann <t-...@gmx.net> wrote:
> On Mon, 5 May 2008 16:02:57 -0700 (PDT), d-42 wrote:
> > > All of this does not explain why sort and print are that slow.
>
> >  When you say sort and print are 'slow'. What do you mean specifically?
>
> As said before, 2 hours compared to > 24 hours

I mean timing for a specfic operation, not the total runtime. How much
longer does a -given- sort take? How much longer does a given print
take? Does it average it 1.2 seconds to sort, and 2.6 seconds generate
a pdf on FM6, but 1.3 seconds to sort, and 12.6 seconds to generate
the PDF on FM9? Or maybe its 12.5 seconds to sort, but 0.5 seconds to
print...


> >  You open up a file, and perform a basic sort on a single indexed field
> >  in the table attached to the current layout, and its orders of
> >  magnitude slower than it was on FM6? Under what conditions is sorting
> >  so much slower?

> No - I use one file, copy certain field contents,
>
> then I go to the main file (external script), enter find mode, paste,
> add some extra values, sort the results, print them to a pdf, rename
> this pdf and exit this external script
>
> to loop to the next record, copy, and continue as above.

Alright, so *is* a process, and worse, no offense, but the script you
describe sounds almost like it was designed to run as poorly as
possible!!

1) Have you tried profiling it at all to isolate what percentage of
time is taken by each step? This can be very instructive. The
profiling will slow it down even further, but the information gained
will tell us where the time is being spent.

If the numbers are low, set up loops for each operation to perform it
1000 times, and then measure the time to do it 1000 times. You'd be
surprised. For example, it takes 15 seconds (on a Core 2 Quad running
at 3GHz ) to copy a text field with 20 characters in it to the clip
board 10,000 times. It takes 1 second to copy that text field to a
local variable 10,000 times.

2) Why use an external script? Why not localize it all in one file.
That tends to make a significant difference to performance in new
versions of FM. (keep the actual data in multiple external files if
you like, but link to them as table occurrences in the file that runs
the script.

3) Don't use copy and paste. That incurs a performance hit. Use a
variable, global variable, or global field depending on the
circumstances. See above, it can be worse by a factor of ~15. Its
probably not what's killing performance in -your- case, but it
underscores the difference suboptimal choices make.

4) Why are you saving and then renaming the file? FM9 can create the
file with the name you want directly.

5) On a looping script make use of freezewindow to prevent needless
repainting. I usually implment a 'progress bar', that repaints every
few seconds or so (so the application doesn't look hung up). It can
make a gigantic difference.

All that said, We can clearly speed it up some, but I can't say it
will solve your problem. It might well simply be the PDF generation is
the bottleneck or the sorting is the bottleneck and there isn't much
that can be done about it. But I'd want to see some hard evidence of
this before throwing in the towel.

-cheers,
Dave

> http://www.fa-technik.adfc.de/Codierung/Codier-Anbieter.pngis an image
> that was built from ASCII graphics within FMP 7. You can't use ASCII
> graphics any longer, based on font size 1 (up to FMP 6) or font size 2
> (from FMP 7 on) since FMP 8 upwards do use different width spacing or
> kerning even for fixed width fonts.

Actually, with FM9 you've got some cool new options. You can use the
web viewer control combined with html generated on the fly in
filemaker to display, well, pretty much anything you want, anyway you
want.

On OSX I -think- you can even use SVG. (You can't on windows because
IE lacks  SVG support... although you might be able to add it via a
plugin -- I know the adobe svg addon doesn't work, but another one
might. I know -some- plugins work, as the flash one does.) However,
you can use the Microsoft web graphics language ... VML if your
targeting windows.

And if you are willing to setup a web server, you can do even crazier
stuff... like have FM send 'url' commands to a the web control which
will pass them to the server, where you can have the web server or cgi
do virtually anything you can imagine.

-cheers
Dave

On Tue, 6 May 2008 03:22:11 -0700 (PDT), d-42 wrote:
> > >  When you say sort and print are 'slow'. What do you mean specifically?
> >
> > As said before, 2 hours compared to > 24 hours
> 
>  I mean timing for a specfic operation, not the total runtime. How much
>  longer does a -given- sort take? How much longer does a given print
>  take? Does it average it 1.2 seconds to sort, and 2.6 seconds generate
>  a pdf on FM6, but 1.3 seconds to sort, and 12.6 seconds to generate
>  the PDF on FM9? Or maybe its 12.5 seconds to sort, but 0.5 seconds to
>  print...

I could try such numbers - but what for? Comparing numbers based on
total amount is much more exact than a random sample which may be
affected by cache, by memory organisation etc.

Since these PDFs are created fully automatically, I may give you numbers
per page, based on the total amount of pages.

> > No - I use one file, copy certain field contents,
> >
> > then I go to the main file (external script), enter find mode, paste,
> > add some extra values, sort the results, print them to a pdf, rename
> > this pdf and exit this external script
> >
> > to loop to the next record, copy, and continue as above.
> 
>  Alright, so *is* a process, and worse, no offense, but the script you
>  describe sounds almost like it was designed to run as poorly as
>  possible!!

Maybe - I doubt so.

>  1) Have you tried profiling it at all to isolate what percentage of
>  time is taken by each step? This can be very instructive. The
>  profiling will slow it down even further, but the information gained
>  will tell us where the time is being spent.

I might add some statistics to log the time for each step, which is
mainly a find, a sort and a print command (multi-column, single font,
different sizes and styles).

>  If the numbers are low, set up loops for each operation to perform it
>  1000 times, and then measure the time to do it 1000 times. You'd be
>  surprised. 

Why should I do the same test procedure again and again? A database
would win that would be optimised for this unreal benchmark, since it
could perform the same operation again and again from cached data. 

This would not tell anything about the real performance for user tasks,
such as not doing the same thing 1000 times, but doing 1000 different
things just once.

>  2) Why use an external script? Why not localize it all in one file.

This would slow down operations even more. It would complicate backup
schemes. It would slow down data dumps etc. 

I do not want to merge data within a single file which has not much to
do with each other.

We are talking about one database for one job with 1M records and 1 GB
file size - and another database with 1M records, where a small subset
of 500 records is used to control the first database.

Just FYI: the first database contains roads, organised by an eight digit
community number. The second database holds those communities, as well
as sub parts and higher level parts. A higher level county shares the
same first five digits of the community number. Within this database I
use relations from a community to its county via left(cnumber,5), as
well as to its state (two digits) of from sub parts to the comunity (8
of up to 12 digits). This community database holds various helper
database for upates and special tasks. So does the road database - and
there are various other databases, each about 1 G sized, for further
information.

I choose to split these community databases even by country, since
handling of single files is much more efficient than merging too many
records and too many tables within a single file.

However, doing everything within a single file is still flawed. 
I just had a simple relation within file "R" which holds tables "data"
and "new". I used a layout which holds records from "new" and related
records from "data", such as "data::status". I did a combined search on
"new::community number" (20 different numbers, 1000 records) and
data::status (with 1 M data records). It just took forever to find these
records. From my point of view, the find should select every matching
record from "new", every matching record from "data" and do a check for
those records which does fulfill all search requests. However, the
current search seems to take a different approach: Find the first
matching new record, find every matching related record for this one.
Then search every matching record for the second one and so one. Thus
search time explodes by the square of records in related tables.

Sometimes it is possible to use OR-ed searches instead of a single AND
search: First search for matching records in "new", then omit
non-matching records via a new request and omit. 

Personally, I could do this job here much faster by canceling the find,
limiting it to a search within the local table, sorting on the related
ones (a 1:1 relation) and omitting the non-matching ones manually. 

Thus I prefer to work within a single table, whenever possible. A search
here does work as expected. Searching within related tables does not
profit that much from merging those tables within a single file - and
one of the missing basic functions is an import not only to one table,
but doing a single import, but splitting matching records within this
step to the proper different tables. I'd expect that opening an exel
spreadsheet with multiple sheets would create a .fp7 database which
could move each sheet to a table of its own. This basic functionality is
still far away, as well as importing external data while adding this as
a new table to a current file.

>  3) Don't use copy and paste. That incurs a performance hit.

copy/paste takes a fraction of about 1 ppm of the actual task, probably
less.

>  Use a
>  variable, global variable, or global field depending on the
>  circumstances. See above, it can be worse by a factor of ~15. Its
>  probably not what's killing performance in -your- case, but it
>  underscores the difference suboptimal choices make.
> 
>  4) Why are you saving and then renaming the file? FM9 can create the
>  file with the name you want directly.

I got a working AppleScript solution which does this job pretty well. I
might change this for FMP9, but I would loose the backwards
compatibility for FMP7

>  5) On a looping script make use of freezewindow to prevent needless
>  repainting. I usually implment a 'progress bar', that repaints every
>  few seconds or so (so the application doesn't look hung up). It can
>  make a gigantic difference.

I just freeze and do no progress bar at all, since this would slow down
the process even more. I monitor the progress within the pdf-folder
itself.

>  All that said, We can clearly speed it up some, but I can't say it
>  will solve your problem. It might well simply be the PDF generation is
>  the bottleneck or the sorting is the bottleneck and there isn't much
>  that can be done about it. But I'd want to see some hard evidence of
>  this before throwing in the towel.

What I observed is that both sort and print did add a significant
slowdown here, where sorting takes about 25 % and printing takes about
70 % of the time. Find is fast (4 %?), while all the other tasks are far
below 1 %.

BTW: yet another annoying "feature" is an occasional Re-Sort of data
after some operations (e.g. working with multiple windows, performing
imports etc.), which is most annoying. It's one of the reasons why I
prefer to keep data in different files and less record numbers.  It's
just so annonying when a window does pop up, telling you that there are
about 200 000 records remaining for sort - and after 200 000 records it
will start to count negative record numbers "still -300 000 records to
go", while you don't have any choice to cancel this operation and don't
have any need for sorted results.

- Martin

On Tue, 6 May 2008 03:48:00 -0700 (PDT), d-42 wrote:
> 
> > http://www.fa-technik.adfc.de/Codierung/Codier-Anbieter.pngis an image
> > that was built from ASCII graphics within FMP 7. You can't use ASCII
> > graphics any longer, based on font size 1 (up to FMP 6) or font size 2
> > (from FMP 7 on) since FMP 8 upwards do use different width spacing or
> > kerning even for fixed width fonts.
> 
>  Actually, with FM9 you've got some cool new options. You can use the
>  web viewer control combined with html generated on the fly in
>  filemaker to display, well, pretty much anything you want, anyway you
>  want.
> 
>  On OSX I -think- you can even use SVG. (You can't on windows because
>  IE lacks  SVG support... although you might be able to add it via a
>  plugin -- I know the adobe svg addon doesn't work, but another one
>  might. I know -some- plugins work, as the flash one does.) However,
>  you can use the Microsoft web graphics language ... VML if your
>  targeting windows.

I choose to use perl instead for those CGI operations. It's just that
much more efficient when someone wants to search for "Schlo�stra�e"
(castle road) and will have the task to search for

  Schlo�stra�e
  Schlossstra�e
  Schlosstra�e
  Schlo�-Stra�e
  Schloss-Stra�e

.... and each variation of stra�e, str., strasse, str etc.
with a first round of an exact match,
a second round for entries that begin with that search term,
a third round which does search for any entries that does include the
search term within,
a fourth round that will cut the part of "Stra�e" and repeat the steps
above (e.g. since the road is not named as road, but ave, alley etc.)

I tried this approach on a single-user FMP-only approach, which is much
slower than a multi-user web-approach via perl, running on the same cpu.

>  And if you are willing to setup a web server, you can do even crazier
>  stuff... like have FM send 'url' commands to a the web control which
>  will pass them to the server, where you can have the web server or cgi
>  do virtually anything you can imagine.

That's nice - but I use perl and WWW:mechanize instead.

- Martin

On May 6, 4:50 am, Martin Trautmann <t-...@gmx.net> wrote:
> On Tue, 6 May 2008 03:48:00 -0700 (PDT), d-42 wrote:
>
> > >http://www.fa-technik.adfc.de/Codierung/Codier-Anbieter.pngisan image
> > > that was built from ASCII graphics within FMP 7. You can't use ASCII
> > > graphics any longer, based on font size 1 (up to FMP 6) or font size 2=

> > > (from FMP 7 on) since FMP 8 upwards do use different width spacing or
> > > kerning even for fixed width fonts.
>
> >  Actually, with FM9 you've got some cool new options. You can use the
> >  web viewer control combined with html generated on the fly in
> >  filemaker to display, well, pretty much anything you want, anyway you
> >  want.
>
> >  On OSX I -think- you can even use SVG. (You can't on windows because
> >  IE lacks  SVG support... although you might be able to add it via a
> >  plugin -- I know the adobe svg addon doesn't work, but another one
> >  might. I know -some- plugins work, as the flash one does.) However,
> >  you can use the Microsoft web graphics language ... VML if your
> >  targeting windows.
>
> I choose to use perl instead for those CGI operations. It's just that
> much more efficient when someone wants to search for "Schlo=DFstra=DFe"
> (castle road) and will have the task to search for
>
>   Schlo=DFstra=DFe
>   Schlossstra=DFe
>   Schlosstra=DFe
>   Schlo=DF-Stra=DFe
>   Schloss-Stra=DFe
>
> ... and each variation of stra=DFe, str., strasse, str etc.
> with a first round of an exact match,
> a second round for entries that begin with that search term,
> a third round which does search for any entries that does include the
> search term within,
> a fourth round that will cut the part of "Stra=DFe" and repeat the steps
> above (e.g. since the road is not named as road, but ave, alley etc.)
>
> I tried this approach on a single-user FMP-only approach, which is much
> slower than a multi-user web-approach via perl, running on the same cpu.
>
> >  And if you are willing to setup a web server, you can do even crazier
> >  stuff... like have FM send 'url' commands to a the web control which
> >  will pass them to the server, where you can have the web server or cgi
> >  do virtually anything you can imagine.
>
> That's nice - but I use perl and WWW:mechanize instead.

The point was that you could bypass the whole 'ascii-art at 1 point'
using the web browser control. I'm really not sure what tangent you
went down.

-Dave

On Tue, 6 May 2008 12:27:24 -0700 (PDT), d-42 wrote:
>  The point was that you could bypass the whole 'ascii-art at 1 point'
>  using the web browser control. I'm really not sure what tangent you
>  went down.

I did not create any SVG by hand yet. I'm not that used to FMP 9 up to
now. I did not even try it within FMP 8.5. At the moment I'd expect that
it is as useless as XML is. Are there any good demos around? What FMP
lists on there web pages, SVG graphing, is .fp5 and creating graphics
via Excel .xslt ...

SVG is not supported by many browsers up to now. Thus ASCII graphics and a
screenshot, saved as .gif is not that bad, although I'd prefer better
solutions. The border lines have to be adjusted by hand up to now. SVG
would not be able to do this job on its own - but I might combine a
static background graphic and just place the different colored and sized
circles/boxes on top of it.

But SVG is a different topic.

- Martin

On May 6, 4:40 am, Martin Trautmann <t-...@gmx.net> wrote:
> On Tue, 6 May 2008 03:22:11 -0700 (PDT), d-42 wrote:
> > > >  When you say sort and print are 'slow'. What do you mean specifically?
>

> I could try such numbers - but what for? Comparing numbers based on
> total amount is much more exact than a random sample which may be
> affected by cache, by memory organisation etc.

But that doesn't tell us squat about about where the performance hit
is.
If the script takes 24 hours to complete in FM9 we need to know WHERE
that time is being spent:

How much time doing the find.
How much time copying and pasting fields
How much time calling and returning from the external script
How much time the save-pdf records script step takes
How much time the applescript takes.
How much time the sort takes.
How long it takes to render a print preview would also be interesting.

However as most of these for a single iteration are going to be sub-
second, we need to capture a 1000+ iterations to get useful numbers.

> This would not tell anything about the real performance for user tasks,
> such as not doing the same thing 1000 times, but doing 1000 different
> things just once.

Your report does the same 10 or so things 1000 times I understand it
right. We are interested in how much time is spent doing each of the
10 things. You are right some things might be distorted by caching if
you just loop on them; so figure out a way around it. Do 1000
iterations of the command but feed it random data each time... there
is always a way.

> >  2) Why use an external script? Why not localize it all in one file.
>
> This would slow down operations even more. It would complicate backup
> schemes. It would slow down data dumps etc.

It would do no such thing.

> I do not want to merge data within a single file which has not much to
> do with each other.

You seem not to understand what I am saying. You don't have to merge
data within a single file. You localize this report generation script
in a single file. The data can remain exactly where it is now, each
its own file.

You do realize you can create fp7 files that contain no tables at all,
and just reference tables in other  database files and then create
layouts for them. This allows you to write high performance scripts
that never call an external function, never hop windows, and yet
operate within the context of multiple tables. You don't even need to
have a single relationship defined.

> >  3) Don't use copy and paste. That incurs a performance hit.
>
> copy/paste takes a fraction of about 1 ppm of the actual task, probably
> less.

As I said in my previous post, I agree this isn't likely the issue in
your script... but it is indicative of weak design. The only time you
should copy/paste is when you absolutely have to, or you actually want
to overwrite/retreive the OS clipboard. Given that you needlessly used
copy/paste in a script that is time sensitive I naturally suspect
you've made other suboptimal design choices.

> >  4) Why are you saving and then renaming the file? FM9 can create the
> >  file with the name you want directly.
>
> I got a working AppleScript solution which does this job pretty well. I
> might change this for FMP9, but I would loose the backwards
> compatibility for FMP7

Its potentially a significant performance hit calling an applescript.
Context switching in OSX is expensive.

> >  5) On a looping script make use of freezewindow to prevent needless
> >  repainting. I usually implment a 'progress bar', that repaints every
> >  few seconds or so (so the application doesn't look hung up). It can
> >  make a gigantic difference.
>
> I just freeze and do no progress bar at all, since this would slow down
> the process even more. I monitor the progress within the pdf-folder
> itself.

If you are hopping windows/files that almost completely undermines any
benefit of freeze window. So its probably not accomplishing as much as
you think.

And a well designed progress bar adds less than 0.5% to the time. On a
1 hour script, it costs maybe 15-20 seconds.


> BTW: yet another annoying "feature" is an occasional Re-Sort of data
> after some operations (e.g. working with multiple windows, performing
> imports etc.), which is most annoying. It's one of the reasons why I
> prefer to keep data in different files and less record numbers.  It's
> just so annonying when a window does pop up, telling you that there are
> about 200 000 records remaining for sort - and after 200 000 records it
> will start to count negative record numbers "still -300 000 records to
> go", while you don't have any choice to cancel this operation and don't
> have any need for sorted results.

That's a clear sign of poor design. If you don't need sorted results
on a particular feature/screen/layout/whatever, don't sort them there.

Bottom line, if its the pdf generation itself that is that much slower
and it might be, then yeah, you are probably hosed. But is this the
case?

You seem to have taken the least optimal route at multiple points, I
can't help but suspect that maybe, just maybe, your performance hit is
still design related. That its taking longer to sort than it has too,
That your freezewindow is having no positive effect because you are
window hopping, that your applescript is taking far more time than you
think...that maybe there is some goofy summary field calculations that
are dragging your print performance down the tubes...

I'm also curious, have you tried running your solution on Windows?
That might also be instructive. Also, what mac platform are you
running all this stuff on? G4? G5? Core2Duo?

-best regards.
Dave

On Tue, 6 May 2008 13:32:48 -0700 (PDT), d-42 wrote:
> > >  2) Why use an external script? Why not localize it all in one file.
> >
> > This would slow down operations even more. It would complicate backup
> > schemes. It would slow down data dumps etc.
> 
>  It would do no such thing.

I misunderstood your answer - I thought your suggestion was to merge all
different tables within a single file.

Ok, maybe you are right that a single master file could to the job which
does take all the external data via layouts and related fields.

Actually, I never tried this approach for this task here. I always
prefered to keep things independently and standalone. But I doubt that
calling the operations one step further away would speed up things. All
my former experience has proven the opposit.

Switching files, layouts may add a speed penalty. But that's perfectly
minor to the operation itself. Doing a search or a sort on related
records instead of working within this table itself never was faster,
but many times was much, much slower (as indicated e.g. within the
described AND search).

>  That's a clear sign of poor design. If you don't need sorted results
>  on a particular feature/screen/layout/whatever, don't sort them there.

I can't reproduce the behavior - but I did not ask to sort everything
completely new. I did not request a sort at all at this moment while it
is enforced by FMP.

I don't know whether this malfunction is gone in FMP 9.
But FMP told me it was resorting a certain amount of records, continuing
in negative numbers, even if the only active window had a sorted subset
only. Usually it may have some related records on the layout which might
trigger sorting of related records. I did not find out the logic behind
yet. You don't know this behavior? You never saw sorting on negative
record counts?

>  Bottom line, if its the pdf generation itself that is that much slower
>  and it might be, then yeah, you are probably hosed. But is this the
>  case?

FMP7 was slow, but it did not matter, since the PDF was useless. I might
compare FMP7 and FMP9. FMP6 feels much faster.

>  You seem to have taken the least optimal route at multiple points, I
>  can't help but suspect that maybe, just maybe, your performance hit is
>  still design related. That its taking longer to sort than it has too,
>  That your freezewindow is having no positive effect because you are
>  window hopping, that your applescript is taking far more time than you
>  think...that maybe there is some goofy summary field calculations that
>  are dragging your print performance down the tubes...

All of this does sound like minor speed improvements only.

>  I'm also curious, have you tried running your solution on Windows?

No, since Win does not offer AppleScript nor integrated PDF creation.

>  That might also be instructive. Also, what mac platform are you
>  running all this stuff on? G4? G5? Core2Duo?

G4, 800 MHz, 1 1/8 GB RAM, MacOSX 10.4

- Martin

On May 6, 1:32 pm, Martin Trautmann <t-...@gmx.net> wrote:
> On Tue, 6 May 2008 12:27:24 -0700 (PDT), d-42 wrote:
> >  The point was that you could bypass the whole 'ascii-art at 1 point'
> >  using the web browser control. I'm really not sure what tangent you
> >  went down.
>
> I did not create any SVG by hand yet. I'm not that used to FMP 9 up to
> now. I did not even try it within FMP 8.5. At the moment I'd expect that
> it is as useless as XML is. Are there any good demos around?

Tons. But forget Filemaker. Its not really part of the equation.

You can display a web page in Filemaker. Pretty well anything that
someone can display in your browser you can now display in filemaker.
And because filemaker can manipulate strings of text you can construct
html pages on the fly and then use the web viewer to render them. You
are limited only by your ability to write html, svg, xhtml,
javascript, css, etc.

And if you add a web server, you can leverage perl cgi, asp.net, php,
etc too.

You can use it to show to show graphs, charts, etc. You can use it to
pull up a map from the internet. You want to display a chart that
sometimes has 3 columns, sometimes 5, you want the row with the
highest total highlighted, and you want to align it on the decimal
point and not use a fixed width font?!

Oh, and you want a simple bar chart to accompany it.

That would be extremely hard to do in filemaker, but almost trivial to
generate the html for, so write a script to generate the html, and
then load it up in the web viewer.


> Thus ASCII graphics and a
> screenshot, saved as .gif is not that bad, although I'd prefer better
> solutions. The border lines have to be adjusted by hand up to now. SVG
> would not be able to do this job on its own - but I might combine a
> static background graphic and just place the different colored and sized
> circles/boxes on top of it.

I think you underestimate what can be done with SVG.

http://www.croczilla.com/svg/samples/lion/screenshot.png
http://www.croczilla.com/svg/samples/butterfly/screenshot.png
http://www.croczilla.com/svg/samples/tiger/screenshot.png

http://www.croczilla.com/svg/samples/

On May 6, 2:04 pm, Martin Trautmann <t-...@gmx.net> wrote:
> On Tue, 6 May 2008 13:32:48 -0700 (PDT), d-42 wrote:
> > > >  2) Why use an external script? Why not localize it all in one file.
>
> > > This would slow down operations even more. It would complicate backup
> > > schemes. It would slow down data dumps etc.
>
> >  It would do no such thing.
>
> I misunderstood your answer - I thought your suggestion was to merge all
> different tables within a single file.

Hell no. :)

> Ok, maybe you are right that a single master file could to the job which
> does take all the external data via layouts and related fields.

NO! YOU DO NOT NEED TO WORK VIA RELATED FIELDS!!

That would be slow.

Define a table occurrence in the relationships graph for your external
table, and then define a layout that uses that table occurrence. You
don't need to define a relationship to do this.

> Actually, I never tried this approach for this task here. I always
> prefered to keep things independently and standalone. But I doubt that
> calling the operations one step further away would speed up things. All
> my former experience has proven the opposit.

Working on an external table defined as I did above is the same speed
as working on it in its own file. At the same time, a big performance
hit is incurred in FM9 when you hop from window to window. (much
bigger than was incurred in FM6) so not doing it is both possible in
FM9 and highly desireable.

Additionally, hopping around in FM9 does NOT commit records. It did in
FM6. Not committing the records and hopping around, especially between
files, can lead to MASSIVE slowdown in FM7+. If you make changes to a
record and then hop to another window, the record you left behind
still has the record locked and the changes have not be committed.

> Switching files, layouts may add a speed penalty. But that's perfectly
> minor to the operation itself. Doing a search or a sort on related
> records instead of working within this table itself never was faster,
> but many times was much, much slower (as indicated e.g. within the
> described AND search).

You were doing it on 'related' records. It shouldn't have beeen done
that way.


> I don't know whether this malfunction is gone in FMP 9.
> But FMP told me it was resorting a certain amount of records, continuing
> in negative numbers, even if the only active window had a sorted subset
> only. Usually it may have some related records on the layout which might
> trigger sorting of related records. I did not find out the logic behind
> yet. You don't know this behavior? You never saw sorting on negative
> record counts?

Can't say that I have. Although I'm very careful to minimize my use of
sorts because sorting is one of the slowest operations you can do in a
database.

> >  You seem to have taken the least optimal route at multiple points, I
> >  can't help but suspect that maybe, just maybe, your performance hit is
> >  still design related. That its taking longer to sort than it has too,
> >  That your freezewindow is having no positive effect because you are
> >  window hopping, that your applescript is taking far more time than you
> >  think...that maybe there is some goofy summary field calculations that
> >  are dragging your print performance down the tubes...
>
> All of this does sound like minor speed improvements only.

Maybe. Maybe not. You've got a script that takes 12x longer than it
used to. Its worth it to dig deep. Record locking issues alone could
account for it.

> >  I'm also curious, have you tried running your solution on Windows?
>
> No, since Win does not offer AppleScript nor integrated PDF creation.

It needs neither.

Applescript as I said is not needed; you can name your files directly
from within filemaker. And FM9 supports PDF creation under both
Windows and OSX. Its a built in feature of filemaker, not the OS.

Hmmm.... How are you generating your PDFs any way? Are you "printing"
them and using the OSX's 'save as pdf' feature? Or are you using
Filemaker's built in PDF engine accessed via the Filemaker Script
step: "Save Records as PDF" ?

You should try "Save Records as PDF" if you haven't.

> >  That might also be instructive. Also, what mac platform are you
> >  running all this stuff on? G4? G5? Core2Duo?
>
> G4, 800 MHz, 1 1/8 GB RAM, MacOSX 10.4

Thanks. At least we know we have the option of throwing more modern
hardware at the problem if all else fails. By modern standards that's
a pretty slow machine.

-cheers,
Dave

I thought I replied to this already, but may have moved on without
hitting send... but rather than rewrite everything... I just wanted to
hit this one point again, in case it got missed...

> >  I'm also curious, have you tried running your solution on Windows?
>
> No, since Win does not offer AppleScript nor integrated PDF creation.

You need neither. FM9 supports PDF creation directly. And you can set
the name of the output file so you don't need applescript.

Given that you seemed not to know this, HOW are you creating PDF
files? Are you using the "Save Records as PDF" script step? You
probably should be.

-cheers,
Dave

On 2008-05-06 14:04:13 -0700, Martin Trautmann <t-use@gmx.net> said:

> I can't reproduce the behavior - but I did not ask to sort everything
> completely new. I did not request a sort at all at this moment while it
> is enforced by FMP.
> 
> I don't know whether this malfunction is gone in FMP 9.
> But FMP told me it was resorting a certain amount of records, continuing
> in negative numbers, even if the only active window had a sorted subset
> only. Usually it may have some related records on the layout which might
> trigger sorting of related records. I did not find out the logic behind
> yet. You don't know this behavior? You never saw sorting on negative
> record counts?

The layout you're using to perform operations can have everything to do 
with performance. If the layout contains displayed summary fields or 
sorted portals, you can encounter huge performance hits.  "It may have 
some related records on the layout" is a big flag for me. If possible, 
layouts used for such looping operations should be as clear of 
extraneous fields as possible. Freeze Window doesn't help much if 
before performing a Set Field (about a zillion times faster than any 
Copy/Paste, btw) it has to summarize a number (which may involve 
sorting) or display a portal which may involve sorting.

Dave is entirely correct. You have made suboptimal design choices at 
more than one decision point. You are not familiar with working with 
Table Occurances from another file, and that fields from such a foreign 
table are *not* "related" fields, but behave and perform as native 
fields on a layout.  You're ignoring obvious performance-damaging 
design elements such as I describe above. I would follow Dave's advice 
very closely in optimizing your files for performance.

Yes, I have seen negative record counts. Usually in files with existing 
corruption which also display other anomalous behaviors, including 
slow/poor performance. Negative record counts are not something you 
want to see in a file.  I'd get the data into a clean file, optimize 
the layouts, remove sorting on any relationships, use Set Field in 
place of Copy/Paste, stop hopping among files, and index fields 
properly for the searches. You'll see an astounding increase in 
performance.
-- 
Lynn Allen
--
www.semiotics.com
Member Filemaker Business Alliance
Long Beach, CA

On Tue, 6 May 2008 14:35:21 -0700 (PDT), d-42 wrote:
>  Define a table occurrence in the relationships graph for your external
>  table, and then define a layout that uses that table occurrence. You
>  don't need to define a relationship to do this.

Ok, this would be possible. But Window and File switching is not what
does make the process timing critical.

>  Additionally, hopping around in FM9 does NOT commit records. 

That's true, but that's no problem here.

> > Switching files, layouts may add a speed penalty. But that's perfectly
> > minor to the operation itself. Doing a search or a sort on related
> > records instead of working within this table itself never was faster,
> > but many times was much, much slower (as indicated e.g. within the
> > described AND search).
> 
>  You were doing it on 'related' records. It shouldn't have beeen done
>  that way.

What's the purpose of related records when you can't use them? I do
prefer to work as far as possible within the table itself. I do use
temporary fields to move related data to the table itself in order to
work faster. But I have split information between files, where 1000
records in one file may share the same info from the other file (e.g.
all roads within a city share the same city information) where I do have
to work on related records.

The search problem occurs while I do merge the current data set with
external updates. There's always a subset of about < 5 % where manual
actions are required, due to typos, abbreviations etc.

I do use related records within printing since each sheet has body
information from the table itself, but takes header, subheaders and
footer from related fields, all of it as merged text.

Correction: I may have found the bottleneck: Each line contains a
calculated field which is composed from related data within the other
file. Thus it is an unstored calculation, while the FMP6 solution takes
a calculated text field instead.

>  Can't say that I have. Although I'm very careful to minimize my use of
>  sorts because sorting is one of the slowest operations you can do in a
>  database.

Sort and find are my major tasks for database operation at all - at
least for my needs. I love table views, where sorting via header clicks
is a major benefit, although I feel that the user concept is broken:
Clicking multiple headers does not sort in the order they were
(cmd-)clicked, but from left to right only.

One of my major wishes for FMP would be multi-cell copy/paste, as I can
do within spreadsheets. Currently I have to go via export/edit
externally/import (replacing field contents), which is always more
dangerous. Oh, BTW: multilevel undo for this edit operation would be
usefull, too :-)

>  Maybe. Maybe not. You've got a script that takes 12x longer than it
>  used to. Its worth it to dig deep. Record locking issues alone could
>  account for it.

Since I do not edit any data externally, I would not expect this. But
who know. All I do within the other file is a loop to goto the next
record, copy field contents, paste it within the other file as a search
and then set a processed date again within the first field. Yes, this is
an edit operation that might slow down things - point taken, I might
ensure to leave fields.

>  Applescript as I said is not needed; you can name your files directly
>  from within filemaker. And FM9 supports PDF creation under both
>  Windows and OSX. Its a built in feature of filemaker, not the OS.
> 
>  Hmmm.... How are you generating your PDFs any way? Are you "printing"
>  them and using the OSX's 'save as pdf' featur? Or are you using
>  Filemaker's built in PDF engine accessed via the Filemaker Script
>  step: "Save Records as PDF" ?

I do the "print" operation, as indicated within the subject.
 
>  You should try "Save Records as PDF" if you haven't.

Here's a first sample, a bigger part with 6210 found records.
9:09:29 begin
9:09:29 find finished
9:10:31 sort finished
9:10:34 window changed to layout, preview
9:13:32 print finished
9:13:32 file rename by AppleScript finished

Thus sorting took 62 seconds, printing took 178 seconds, other jobs
required time slots of minimum length.

Same run with "save as pdf" instead of print to pdf:
9:27:42 begin
9:27:44 find finished
9:28:29 sort finished
9:28:33 window changed to layout, preview
9:31:22 save finished
9:31:23 file rename by AppleScript finished

Sort was faster this time - probably some cached effect, 45 seconds
only. The window switch is something I want since most of the times I do
print just a few files, instead of several hundreds. Thus I do accept
these few seconds each time.
Save took 169 seconds - that's a certain speed improvement.

But the printed PDF takes 416 KB for 57 pages as PDF 1.3,
while the saved PDF takes 1.2 MB for 57 pages as PDF 1.4

Did I miss the point how to use a "save file as"?

Example: I do search for "12345". Thus the first sorted record will use
the key "12345000". This key will catch the related abbreviation "AB".
Thus the pdf file shall be named as "12345-AB.pdf".

I do this via AppleScript to rename the current "pdf/dummy.pdf" to
"AB-12345.pdf", where the AS is computed to use
 town::abbreviation & "-" & left(key,5)

I did not see any option how to compute an output file name yet. That's
why I do use AppleScript on this old Mac. I might build a comparable
solution for Win to rename files, but that's another topic.

> > >  That might also be instructive. Also, what mac platform are you
> > >  running all this stuff on? G4? G5? Core2Duo?
> >
> > G4, 800 MHz, 1 1/8 GB RAM, MacOSX 10.4
> 
>  Thanks. At least we know we have the option of throwing more modern
>  hardware at the problem if all else fails. By modern standards that's
>  a pretty slow machine.

It is - but this work is done for non-profit. I can't justif a new
499-Euro-Mac mini or an even more powerful Mac Pro for 4684 Euro
(Quad-Core Ceon, 3 GHz, 8 GB RAM, 500 GB HD, WiFi) - multiply this by
1.5 for price in USD :-(

- Martin

On Tue, 6 May 2008 16:30:59 -0700, Lynn Allen wrote:
>  If the layout contains displayed summary fields or 
>  sorted portals, you can encounter huge performance hits.

No summaries, no portals. 
As mentionned before, one bottleneck is an unstored calculation based on
related records.

>  "It may have 
>  some related records on the layout" is a big flag for me. If possible, 
>  layouts used for such looping operations should be as clear of 
>  extraneous fields as possible.

Printing is a final step for me where I do expect accuracy and did not
mind that much about time yet. But waiting for (more than) 24 hours was
a negative surprise to me.

>  Dave is entirely correct. You have made suboptimal design choices at 
>  more than one decision point. You are not familiar with working with 
>  Table Occurances from another file, and that fields from such a foreign 
>  table are *not* "related" fields, but behave and perform as native 
>  fields on a layout.  You're ignoring obvious performance-damaging 
>  design elements such as I describe above. I would follow Dave's advice 
>  very closely in optimizing your files for performance.

I may give this a new try with an "printmaster.fp7" file which would
take external tables. But it's sorting and printing which are the time
consuming operations.

Moving layouts from one FMP file to another is an annoying and
unpleasurable task since you can copy fields only, but not the other
layout elements, such als columns, borders and parts.

>  Yes, I have seen negative record counts. Usually in files with existing 
>  corruption which also display other anomalous behaviors, including 
>  slow/poor performance. Negative record counts are not something you 
>  want to see in a file.  I'd get the data into a clean file, optimize 
>  the layouts, remove sorting on any relationships, use Set Field in 
>  place of Copy/Paste, stop hopping among files, and index fields 
>  properly for the searches. You'll see an astounding increase in 
>  performance.

I do not sort on related records. I do not find on related records. But
I do print with related records.

I might give a fresh database a try - but importing 1 M records on this
computer will take several days :-(

- Martin

On Tue, 6 May 2008 14:13:12 -0700 (PDT), d-42 wrote:
>  On May 6, 1:32 pm, Martin Trautmann <t-...@gmx.net> wrote:
> > On Tue, 6 May 2008 12:27:24 -0700 (PDT), d-42 wrote:
> > >  The point was that you could bypass the whole 'ascii-art at 1 point'
> > >  using the web browser control. I'm really not sure what tangent you
> > >  went down.
> >
> > I did not create any SVG by hand yet. I'm not that used to FMP 9 up to
> > now. I did not even try it within FMP 8.5. At the moment I'd expect that
> > it is as useless as XML is. Are there any good demos around?
> 
>  Tons. But forget Filemaker. Its not really part of the equation.

Hm, you mentionned it - and I wondered how I might use FMP to build a
map as shown before.

>  I think you underestimate what can be done with SVG.
> 
>  http://www.croczilla.com/svg/samples/lion/screenshot.png

creator unknown

>  http://www.croczilla.com/svg/samples/butterfly/screenshot.png

Generated by Jasc WebDraw

>  http://www.croczilla.com/svg/samples/tiger/screenshot.png

creator unknown.

Where those images drawn or manipulated within FMP? SVG is just an
exchange format which I might manipulate. Most tools are the usual
vector graphic tools which are able to save/export as SVG.

FMP offers an option to output text as SVG, but I do not see that many
graphing capabilites. Maybe there's a certain set of custom functions
anywhere which does draw pixels, lines, circles at certain locations. 
FMP should be able to import full libraries of custom functions, leaving
them in groups of their own, such as "svg", as there is "text" or
"time" now.

Do you know demos, where FMP built those graphics?
It's no magic to compose
<http://www.croczilla.com/svg/samples/circles2/text_view?obj=circles2.xml>
in order to build 
http://www.croczilla.com/svg/samples/circles2/screenshot.png
http://www.croczilla.com/svg/samples/circles2/circles2.xml

But I'd like to see applications where these infos are served from FMP,
as well as a screenshot how FMP as web browser does display this page.



Apart form that, you mentionned FMP to send URLs to other web severs. I
had this task recently: login, perform a search, catch the output,
extract certain data from it and do corrections. It was an anti-spam-bot
for an external wiki in order to revert spambot vandalism. It is a
typical application where regular expressions are most helpful to
extract certain text parts. That's why regular expressions are one of my
major wishes for goot database operations, never available in FMP up to
now. 

- Martin

On Tue, 6 May 2008 15:05:16 -0700 (PDT), d-42 wrote:
> > >  I'm also curious, have you tried running your solution on Windows?
> >
> > No, since Win does not offer AppleScript nor integrated PDF creation.
> 
>  You need neither. FM9 supports PDF creation directly. And you can set
>  the name of the output file so you don't need applescript.

I do not own FMP9 for Win. Maybe the demo version would to the job? I'll
check later on. Does the Win version offer to compute the output file
name, where the Mac version doesn't, AFAIK?

>  Given that you seemed not to know this, HOW are you creating PDF
>  files? Are you using the "Save Records as PDF" script step? You
>  probably should be.

see other reply

On May 7, 2:19 am, Martin Trautmann <t-...@gmx.net> wrote:

> FMP offers an option to output text as SVG, but I do not see that many
> graphing capabilites. Maybe there's a certain set of custom functions
> anywhere which does draw pixels, lines, circles at certain locations.
> FMP should be able to import full libraries of custom functions, leaving
> them in groups of their own, such as "svg", as there is "text" or
> "time" now.

Yeah, it would be wonderful if FM had a full on set of graphics
functions, and could dynamiclly draw vector graphics right on a
layout. It can't. However the web browser control can be used to get
some of that functionality.

> Do you know demos, where FMP built those graphics?

Built from data? Or built from scratch?

FMP is not a paint program.  Its the last thing I would use to try and
draw a tiger.

But If I had the data for a tiger (not an SVG file, just the data, the
curve parameters, the color data, polygon boundaries, etc. Then it
would be a relatively simple matter to write a script to generate the
SVG or VML and put it in a text field (we've been able to do THAT
since Filemaker 2.0).

But only recently, with FM9 do we have a way of rendering the picture
and taking a look at it within filemaker.


In your case your image had a couple polygons and a bunch of colored
points. I assume you have all the coordinate data in FM for the
polygon boundary and the point colors and coordinates as you would
need this to generate the ascii art on the fly.

So generating an SVG file from it would be a relative breeze, about
the same effort as your existing ascii art.

-Dave

On May 7, 12:50 am, Martin Trautmann <t-...@gmx.net> wrote:

> Correction: I may have found the bottleneck: Each line contains a
> calculated field which is composed from related data within the other
> file. Thus it is an unstored calculation, while the FMP6 solution takes
> a calculated text field instead.

Yes, that could make a difference.

> One of my major wishes for FMP would be multi-cell copy/paste, as I can
> do within spreadsheets. Currently I have to go via export/edit
> externally/import (replacing field contents), which is always more
> dangerous. Oh, BTW: multilevel undo for this edit operation would be
> usefull, too :-)

A multiuser database isn't a spreadsheet. How does a multilevel undo
work in a system where multiple users may have updated the record
prior to your request to undo? What if you've done a relookup on 4
billion records? Where is it going to store the gigabytes of undo
data? I'm not saying I  disagree -- but the semantics of what these
features would be is quite a bit more complicated. And an undo that
only works some of the time is possibly more dangerous than not having
one at all. At least by not having one at all, you'll hopefully take
the time to make backups. :)


> Yes, this is
> an edit operation that might slow down things - point taken, I might
> ensure to leave fields.

FYI 'Leaving fields' does not commit the record in FM7+

> I do the "print" operation, as indicated within the subject.

The word 'print' becomes sort of indeterminate when one is referring
to pdf file generation.

> >  You should try "Save Records as PDF" if you haven't.
>
> Here's a first sample, a bigger part with 6210 found records.
> 9:09:29 begin
> 9:09:29 find finished
> 9:10:31 sort finished
> 9:10:34 window changed to layout, preview
> 9:13:32 print finished
> 9:13:32 file rename by AppleScript finished
>
> Thus sorting took 62 seconds, printing took 178 seconds, other jobs
> required time slots of minimum length.
>
> Same run with "save as pdf" instead of print to pdf:
> 9:27:42 begin
> 9:27:44 find finished
> 9:28:29 sort finished
> 9:28:33 window changed to layout, preview
> 9:31:22 save finished
> 9:31:23 file rename by AppleScript finished
>
> Sort was faster this time - probably some cached effect, 45 seconds
> only. The window switch is something I want since most of the times I do
> print just a few files, instead of several hundreds. Thus I do accept
> these few seconds each time.
> Save took 169 seconds - that's a certain speed improvement.

> But the printed PDF takes 416 KB for 57 pages as PDF 1.3,
> while the saved PDF takes 1.2 MB for 57 pages as PDF 1.4
>
> Did I miss the point how to use a "save file as"?

Different PDF engines and versions will all result in different file
sizes. 1.2MB certainly isn't as good as 416KB but I've seen bigger
differences before. Under specify options try to change the Adobe pdf
version it outputs, that will probably make a difference... I can't
say whether it will make it bigger or smaller though. You can set it
to Acrobat 5+ or Acrobat 6+ on the windows version, not sure what
options you'll have on the mac.

Also check the OSX save-as-pdf options -- there may be differences in
artwork resolutions, page scaling, and so on that might be going on.
(If for some reason its scaling the page even 1% to "fit", for
example, that could be a HUGE performance hit)

Not that you want to run a separate batch process after the fact, but
there is software that can strip out all the needless meta-data crud
in pdf files that isn't required to view them and generally reduce
them by 50-90%. Probably can shrink the OSX pdfs down too.

Might be worth doing in any case, even it takes hours, just to save
space, and speed them up if you serve them over the internet.

----

Given that you've established that of the ~300 seconds report takes to
generate, ~180 seconds are spent in the printing step, I'm now
convinced that at least a significant part of the bottleneck is the
printing step itself.

So lets look at that --  start playing around with your report layout
itself. Eliminate borders and any overlapping objects for example. It
may be the the new filemaker is passing more complex documents to be
rendered into pdf. Sure they might -look- the same, but perhaps inside
they are very different... and the differences may make them much more
time consuming to render.

I'd work up from *completely* blank white pages (set up the 'parts'
but leave them blank) to the full report, trying both the save-records-
as-pdf script steps in all formats as well as the OSX save-as-pdf. So
that we can see what each engine does, and how its affected at each
step.

Then add one piece at a time, and time it in all engines / pdf
versions. I expect you'll find that saving 57 blank pages should take
on the order of 2-3 seconds, and as you build it up the time will
gradually increase... but hopefully somewhere along the way we'll find
something that causes it to suddenly blow a whole whack of time. You
can also monitor file sizes of each engine along the way and see if
there is anything in particular that causes that to jump unexpectedly
too.

Whether its related to page re-scaling, or a (possibly corrupt)
graphic element on the header, or a related field that's needlessly re-
calculating...

> I did not see any option how to compute an output file name yet. That's
> why I do use AppleScript on this old Mac. I might build a comparable
> solution for Win to rename files, but that's another topic.

The save Records as PDF has 4 checkboxes, one of which is 'specify
output file'. To save it to a programmable name that changes with each
iteration you'll need to use a variable that will be set to:
town::abbreviation & "-" & left(key,5). The variable will need to be
set immediately before saving, in each iteration.

It is like this on both OS X and Windows.

Also, I still do recommend trying it on more modern hardware as well
as under windows, just to gauge the results. I understand you are
targeting a non-profit, and a hardware upgrade may not be in the
cards... but if you get wildly varying performance moving between
platforms that may tell us something useful, may indicate something is
corrupt...

-cheers,
Dave

On Wed, 7 May 2008 11:21:17 -0700 (PDT), d-42 wrote:
>  The save Records as PDF has 4 checkboxes, one of which is 'specify
>  output file'. To save it to a programmable name that changes with each
>  iteration you'll need to use a variable that will be set to:
>  town::abbreviation & "-" & left(key,5). The variable will need to be
>  set immediately before saving, in each iteration.

Thanks, I expected to have a direct formula choice. But "Set Variable"
is ok to me.

I'll give the other options a try later on,
Martin

On Wed, 7 May 2008 10:17:10 -0700 (PDT), d-42 wrote:
> > Do you know demos, where FMP built those graphics?
> 
>  Built from data? Or built from scratch?

From data such as dot or line coordinates

>  FMP is not a paint program.  Its the last thing I would use to try and
>  draw a tiger.
> 
>  But If I had the data for a tiger (not an SVG file, just the data, the
>  curve parameters, the color data, polygon boundaries, etc. Then it
>  would be a relatively simple matter to write a script to generate the
>  SVG or VML and put it in a text field (we've been able to do THAT
>  since Filemaker 2.0).

That's just a text conversion which many tools can do, while only few
would be able to display the result.

>  In your case your image had a couple polygons and a bunch of colored
>  points. I assume you have all the coordinate data in FM for the
>  polygon boundary and the point colors and coordinates as you would
>  need this to generate the ascii art on the fly.

exactly - there are custom functions to overlay pixels, circles, lines
and squares by position, size and color
 
>  So generating an SVG file from it would be a relative breeze, about
>  the same effort as your existing ascii art.

even less - a bigger circle in ASCII is lots of calculations, especially
for handling border overlappings. For SVG it's a single command.

- Martin

> >  But If I had the data for a tiger (not an SVG file, just the data, the
> >  curve parameters, the color data, polygon boundaries, etc. Then it
> >  would be a relatively simple matter to write a script to generate the
> >  SVG or VML and put it in a text field (we've been able to do THAT
> >  since Filemaker 2.0).
>
> That's just a text conversion which many tools can do, while only few
> would be able to display the result.

Precisely. FM has had the text/data manipulation tools to create SVG
documents from data since before SVG was a standard.

The point, however, is that recent versions of Filemaker Pro are one
of the tools that *can* display the result (at least on OSX), by
virtue of its web viewer control. (VML works on windows, and there
might be a way to get SVG too.)

> >  So generating an SVG file from it would be a relative breeze, about
> >  the same effort as your existing ascii art.
>
> even less - a bigger circle in ASCII is lots of calculations, especially
> for handling border overlappings. For SVG it's a single command.

Precisely, which is why I suggested it. You complained that your ascii
art system was 'broken' in FM9, and I responded by pointing out that
you have better options now. :)

cheers,
Dave

On Tue, 6 May 2008 03:48:00 -0700 (PDT), d-42 wrote:
>  On OSX I -think- you can even use SVG. 


Hm, I checked by now. The installation lacks Instant Web Publishing.
I heared about a Web Viewer, but I do not have any clue yet how to see
SVG within FMP.

That's why I asked for samples, both where html+svg is served from a FMP
database as well as svg which is displayed within FMP.

www.filemaker.com does not know about svg!? (or there search and
advanced search is useless).

The links to tutorials appear to be broken. 

Any suggestion?

BTW: is the FileMaker tech forum served from FMP itself? fmforums is a
..php forum.

Thanks,
Martin

"Martin Trautmann" <t-use@gmx.net> schreef in bericht 
news:slrng27qpd.9r.t-use@ID-685.user.individual.de...
> On Tue, 6 May 2008 03:48:00 -0700 (PDT), d-42 wrote:
>>  On OSX I -think- you can even use SVG.
>
>
> Hm, I checked by now. The installation lacks Instant Web Publishing.
> I heared about a Web Viewer, but I do not have any clue yet how to see
> SVG within FMP.
>
> That's why I asked for samples, both where html+svg is served from a FMP
> database as well as svg which is displayed within FMP.
>
> www.filemaker.com does not know about svg!? (or there search and
> advanced search is useless).
>
> The links to tutorials appear to be broken.
>
> Any suggestion?
>
> BTW: is the FileMaker tech forum served from FMP itself? fmforums is a
> .php forum.
>
> Thanks,
> Martin

Martin, SVG is not directly supported within filemaker. The newer versions 
support a WebObject, where you are able to show webpages on a layout. Since 
svg is a htm supported format I don't see any reason why that wouldn't work.

Keep well, Ursus 

On Fri, 9 May 2008 11:48:26 +0200, Ursus wrote:
>  Martin, SVG is not directly supported within filemaker. The newer versions 
>  support a WebObject, where you are able to show webpages on a layout. Since 
>  svg is a htm supported format I don't see any reason why that wouldn't work.

Could FMP serve the html code directly or would I have to export from
FMP to an external, static page in order to see this self-generated page
within the Web Viewer again?

Or is there any option in order to build a web page not from the layout
data, but from a single text field which will be used as souce code
(such as html, css, php, svg)?

- Martin

On May 9, 4:45 am, Martin Trautmann <t-...@gmx.net> wrote:
> On Fri, 9 May 2008 11:48:26 +0200, Ursus wrote:
> >  Martin, SVG is not directly supported within filemaker. The newer versions
> >  support a WebObject, where you are able to show webpages on a layout. Since
> >  svg is a htm supported format I don't see any reason why that wouldn't work.
>
> Could FMP serve the html code directly or would I have to export from
> FMP to an external, static page in order to see this self-generated page
> within the Web Viewer again?

> Or is there any option in order to build a web page not from the layout
> data, but from a single text field which will be used as souce code
> (such as html, css, php, svg)?

For small 'documents' you can url encode the entire page in the URI
which accomplishes exactly what you describe. For more information,
see the 'data:' uri scheme. (You know how http:// loads from web
server... ftp:// loads from a ftp server... well data: loads data
inline from the URI.)

Safari supports the data: uri scheme. Internet Explorer 6/7 do not.
(IE8 apparently does). However, on Windows Filemaker handles the data:
uri for you, by behind the scenes creating a temporary file, and then
loading that via a file: uri.

For example you could set a field to be:

data:text/html,<h1>Hello World!</h1>

And set the web viewer to load that URI.

For larger documents, yes, you will need to export to a file and then
load it. Filemaker 9 provides the function 'Get(TempoaryPath)' that
will return a convenient and appropriate place to save these exports.
So building the document in a text field is easy. Exporting a single
text field is easy. (simply export the current record
as ...tempoarypath/somefilename.html in csv format -- because its one
field of one record, there won't be any commas etc) and then set the
webviewer object to load file://temporarypath/somefilename.html

Voila.

PS  regarding the maximum size a data: uri before you have to switch
to an export. I do know that its based on the browsers limit. So for
Mac it will be Safari's limit. For windows, because Filemaker creates
a temporary file behind the scenes anyway, you aren't really limited.

Internet Explorer 5/6/7 is 2kb. IE8 will be at least 32kb (but, per
above, it isn't relevant in FM)
Safari is known to support at least 80kb

So you can create definitely create some pretty sophisticated
documents before you're out of bounds with the data: uri.

-regards,
Dave

On Fri, 9 May 2008 10:17:41 -0700 (PDT), d-42 wrote:
>  For small 'documents' you can url encode the entire page in the URI
>  which accomplishes exactly what you describe. For more information,
>  see the 'data:' uri scheme. (You know how http:// loads from web
>  server... ftp:// loads from a ftp server... well data: loads data
>  inline from the URI.)

Thanks!

It's not the approach I expected, but it's a possibility.

- Martin