Howdy folks,

I've just encountered an unfortunate "feature" of fcopy: it seems that the 
optional -size argument only accepts integer values, not wide ints. This means 
that it cannot be used reliably with large files; they will need several 
successive fcopy with -size under the integer limit, until the desired value is 
reached. Not really difficult to do thanks to -command callbacks and event 
handling (Tcl rocks!), but nevertheless surprising, as I've found no mention of 
this limitation in the manpage or through the intertubes. And the source code 
gives no clue on the reason behind this lack of support for wide ints.

So, bug or feature? In the latter case I think that the doc should be updated 
accordingly.

On Mar 19, 1:35=A0pm, Fr=E9d=E9ric Bonnet <fredericbon...@free.fr> wrote:
> Howdy folks,
>
> I've just encountered an unfortunate "feature" of fcopy: it seems that th=
e
> optional -size argument only accepts integer values, not wide ints. This =
means
> that it cannot be used reliably with large files; they will need several
> successive fcopy with -size under the integer limit, until the desired va=
lue is
> reached. Not really difficult to do thanks to -command callbacks and even=
t
> handling (Tcl rocks!), but nevertheless surprising, as I've found no ment=
ion of
> this limitation in the manpage or through the intertubes. And the source =
code
> gives no clue on the reason behind this lack of support for wide ints.
>
> So, bug or feature? In the latter case I think that the doc should be upd=
ated
> accordingly.


A very good feature. The -size limit is somewhat bogus. Internally
[fcopy] does one buffer size copy and then reschedules itself so that
it shares time with other Tcl events, essentially it automatically
does what the -size feature is supposed to do.

There are numerous document bugs related to this, but since buffer
sizes are not meaningful (they can expand beyond control), who knows
what needs changing. But [fcopy] itself seems to work as
expected...any error would have shown up very quickly.

Personally I wouldn't try to copy 2^32+ bits of data at once, but if
you do, maybe [fcopy] isn't the best choice. Hopefully [fcopy] has an
even lower limit. The -buffersize limit is 1M, so my guess is that 1M
is the max transfer per event.

> This means
> that it cannot be used reliably with large files; they will need several
> successive fcopy with -size under the integer limit, until the desired value is
> reached.

Actually, this is exactly the point of using -size. If you want want a
single transfer, just use fcopy without the "-size" option. "-size" is
meant to be used to break the transfer into smaller chunks, so you can
do something else with it, like transfer monitoring, progress report,
speed control, bandwidth limiting etc.

Tcl Bliss wrote:
>> This means
>> that it cannot be used reliably with large files; they will need several
>> successive fcopy with -size under the integer limit, until the desired value is
>> reached.
> 
> Actually, this is exactly the point of using -size. If you want want a
> single transfer, just use fcopy without the "-size" option. "-size" is
> meant to be used to break the transfer into smaller chunks, so you can
> do something else with it, like transfer monitoring, progress report,
> speed control, bandwidth limiting etc.

Sure, but there are other perfectly good reasons to use -size. I've encountered 
this "feature" with a HTTP Media Server of mine: the client may send a "Range:" 
header with arbitrary values, for example when playing back videos from a given 
position, so it makes sense to issue a single fcopy with the range length given 
to -size. And internally fcopy chunks the output anyway using the channel 
-buffersize. The whole point of fcopy here is to fire-and-forget the channel 
copy and let Tcl manage the background work.

FYI fcopy works fine with -size values of 2GB, and I've worked around this 
limitation without much work. But my point is: if the -size limit makes sense 
(which I doubt), it should be documented. I guess that's the best option anyway, 
as there are certainly other priorities than fixing this arbitrary limitation.

On Mar 20, 2:30=A0am, "tom.rmadilo" <tom.rmad...@gmail.com> wrote:
>
> Personally I wouldn't try to copy 2^32+ bits of data at once, but if
> you do, maybe [fcopy] isn't the best choice. Hopefully [fcopy] has an
> even lower limit. The -buffersize limit is 1M, so my guess is that 1M
> is the max transfer per event.

Per 'event', yes, in the sense of the internal 'fileevents' involved
in async fcopy, but this has nothing to do with the -size limit (fcopy
will keep going until the -size is reached).

-Alex

On Mar 20, 1:23=A0am, Fr=E9d=E9ric Bonnet <fredericbon...@free.fr> wrote:
> Tcl Bliss wrote:
> >> This means
> >> that it cannot be used reliably with large files; they will need sever=
al
> >> successive fcopy with -size under the integer limit, until the desired=
 value is
> >> reached.
>
> > Actually, this is exactly the point of using -size. If you want want a
> > single transfer, just use fcopy without the "-size" option. "-size" is
> > meant to be used to break the transfer into smaller chunks, so you can
> > do something else with it, like transfer monitoring, progress report,
> > speed control, bandwidth limiting etc.
>
> Sure, but there are other perfectly good reasons to use -size. I've encou=
ntered
> this "feature" with a HTTP Media Server of mine: the client may send a "R=
ange:"
> header with arbitrary values, for example when playing back videos from a=
 given
> position, so it makes sense to issue a single fcopy with the range length=
 given
> to -size. And internally fcopy chunks the output anyway using the channel
> -buffersize. The whole point of fcopy here is to fire-and-forget the chan=
nel
> copy and let Tcl manage the background work.
>
> FYI fcopy works fine with -size values of 2GB, and I've worked around thi=
s
> limitation without much work. But my point is: if the -size limit makes s=
ense
> (which I doubt), it should be documented. I guess that's the best option =
anyway,
> as there are certainly other priorities than fixing this arbitrary limita=
tion.

This use makes sense, so do you seek to the starting position and then
use -size as way to interrupt [fcopy] after one background copy (which
probably includes more than one buffersize copy)?

But, for instance, this example in the [fcopy] manpages makes no
sense:

proc CopyMore {in out chunk bytes {error {}}} {
    global total done
    incr total $bytes
    if {([string length $error] !=3D 0) || [eof $in]} {
	set done $total
	close $in
	close $out
    } else {
	fcopy $in $out -command [list CopyMore $in $out $chunk] \
	    -size $chunk
    }
}
set in [open $file1]
set out [socket $server $port]
set chunk 1024
set total 0
fcopy $in $out -command [list CopyMore $in $out $chunk] -size $chunk
vwait done

The same type of code is used in http::geturl in the mistaken belief
that this will keep the client from freezing the rest of the
application. (It will,  internally [fcopy] already handles this
situation...a great feature.)

On 20/03/2010 08:23, Frédéric Bonnet wrote:
> FYI fcopy works fine with -size values of 2GB, and I've worked around
> this limitation without much work. But my point is: if the -size limit
> makes sense (which I doubt), it should be documented. I guess that's the
> best option anyway, as there are certainly other priorities than fixing
> this arbitrary limitation.

I wasn't aware of the limitation before now, but it's now fixed (I
think; testing very large files isn't very portable) in the CVS HEAD.
Also, you would have had problems with the reported number of bytes
transferred when using no -size as that was an int too.

Donal.

On Mar 20, 1:23=A0am, Fr=E9d=E9ric Bonnet <fredericbon...@free.fr> wrote:
> Tcl Bliss wrote:
> >> This means
> >> that it cannot be used reliably with large files; they will need sever=
al
> >> successive fcopy with -size under the integer limit, until the desired=
 value is
> >> reached.
>
> > Actually, this is exactly the point of using -size. If you want want a
> > single transfer, just use fcopy without the "-size" option. "-size" is
> > meant to be used to break the transfer into smaller chunks, so you can
> > do something else with it, like transfer monitoring, progress report,
> > speed control, bandwidth limiting etc.
>
> Sure, but there are other perfectly good reasons to use -size. I've encou=
ntered
> this "feature" with a HTTP Media Server of mine: the client may send a "R=
ange:"
> header with arbitrary values, for example when playing back videos from a=
 given
> position, so it makes sense to issue a single fcopy with the range length=
 given
> to -size. And internally fcopy chunks the output anyway using the channel
> -buffersize. The whole point of fcopy here is to fire-and-forget the chan=
nel
> copy and let Tcl manage the background work.
>
> FYI fcopy works fine with -size values of 2GB, and I've worked around thi=
s
> limitation without much work. But my point is: if the -size limit makes s=
ense
> (which I doubt), it should be documented. I guess that's the best option =
anyway,
> as there are certainly other priorities than fixing this arbitrary limita=
tion.

That's interesting. Implementing "Range:" functionality has been on my
to-do list. Haven't gotten to it yet.
BTW, which media player/file combination support "Range:" requests? I
wasn't aware there were any, except for, may be, Ipod/Iphone.

On 20 Mar, 15:44, "Donal K. Fellows"
<donal.k.fell...@manchester.ac.uk> wrote:
> I wasn't aware of the limitation before now, but it's now fixed (I
> think; testing very large files isn't very portable) in the CVS HEAD.
> Also, you would have had problems with the reported number of bytes
> transferred when using no -size as that was an int too.

BTW, there's no plan to backport this to 8.5 because it impacts an
exported (but unsupported) API. Workaround is just to work with chunks
smaller than 2GB at the script level, and then write Tcl code to
handle doing more than that by calling [fcopy] repeatedly.

Donal.

tom.rmadilo wrote:
> This use makes sense, so do you seek to the starting position and then
> use -size as way to interrupt [fcopy] after one background copy (which
> probably includes more than one buffersize copy)?

Yes, that's the idea. Internally Tcl will potentially issue several I/O 
operations under the buffersize limit, but on the script level you'll only get 
one completion event. Of course to overcome the int size limitation you have to 
issue several consecutive fcopy (by daisy-chaining completion callbacks), so you 
get more events than needed, but this only happens with very large files 
(typically DVD or BD disk images). Unless you WANT periodic events for logging 
or feedback purpose, in this case you'll set -size to a much smaller value as in 
your example.

Donal K. Fellows wrote:
> On 20 Mar, 15:44, "Donal K. Fellows"
> <donal.k.fell...@manchester.ac.uk> wrote:
>> I wasn't aware of the limitation before now, but it's now fixed (I
>> think; testing very large files isn't very portable) in the CVS HEAD.
>> Also, you would have had problems with the reported number of bytes
>> transferred when using no -size as that was an int too.
> 
> BTW, there's no plan to backport this to 8.5 because it impacts an
> exported (but unsupported) API. Workaround is just to work with chunks
> smaller than 2GB at the script level, and then write Tcl code to
> handle doing more than that by calling [fcopy] repeatedly.

Cool! As for 8.5 and older, the workaround works perfectly anyway and is simple 
to code.

Tcl Bliss wrote:
> That's interesting. Implementing "Range:" functionality has been on my
> to-do list. Haven't gotten to it yet.
> BTW, which media player/file combination support "Range:" requests? I
> wasn't aware there were any, except for, may be, Ipod/Iphone.

I'm testing my app on the "Freebox HD" IPTV set-top box (from the French ISP 
"Free"), which embeds a UPnP A/V client. My server gets such requests with media 
types that have metadata at the end of the file, like Matroska and AVI 
containers. The Freebox issues two concurrent HTTP requests: one main request 
for the data stream proper, and a second one with Range: headers for the metadata.

The djmount Linux user-space filesystem driver also sends "Range:" headers to 
emulate seek calls on files mapped from a UPnP A/V Media Server.

On 22 Mar, 21:17, Fr=E9d=E9ric Bonnet <fredericbon...@free.fr> wrote:
> Cool! As for 8.5 and older, the workaround works perfectly anyway and
> is simple to code.

Yes. But I *did* commit code to 8.5 so that if a transfer over 2GB
happens (can only be in the no-size-asked-for case I think) then the
result will be of the correct reported size. That doesn't break any
API (unlike the other, alas). OTOH, I reckon that if you're doing a
big data transfer, it's probably a good idea anyway to put in progress
metering every megabyte or so. If nothing else, it lets you report
current actual transfer rates and estimate the time left. :-)

Donal.