Hi,

As part of a cron job I have running, I'm using gzip -c to append
compressed data to the end of an existing gzip file. According to the
gzip man page this should work, and for the most part it does. It seems,
however, that in some cases this doesn't work. While gunzip and winzip
are both happy opening the file, WinRAR complains about a failed CRC in
the file, and PKUnzip only unzips part of the file.

If I gzip -dc|gzip the file, WinRAR and PKUnzip are perfectly happy to
open the file until next time a new portion is appended to it.

Does anyone have an explaination for this? I don't really want to have
to unzip and re-zip each file every night, as the computational cost
would be far too high.

Any advice would be most appreciated :)

A

-- 
Andy Furnell

Andy wrote:
) As part of a cron job I have running, I'm using gzip -c to append
) compressed data to the end of an existing gzip file. According to the
) gzip man page this should work, and for the most part it does. It seems,
) however, that in some cases this doesn't work. While gunzip and winzip
) are both happy opening the file, WinRAR complains about a failed CRC in
) the file, and PKUnzip only unzips part of the file.
) 
) If I gzip -dc|gzip the file, WinRAR and PKUnzip are perfectly happy to
) open the file until next time a new portion is appended to it.
) 
) Does anyone have an explaination for this? I don't really want to have
) to unzip and re-zip each file every night, as the computational cost
) would be far too high.

The simplest explanation would be that both WinRAR and PKUnzip don't fully
comply with the gzip standard, and can therefore not read all valid gzipped
files.  Complain to the authors of WinRAR and/or PKUnzip.

And ask yourself why you have to use one of those programs to open those
files, and/or simply use gzip to open them.  (Or look for another program
that correctly opens such files, probably any program using zlib.)


SaSW, Willem (at stack dot nl)
-- 
Disclaimer: I am in no way responsible for any of the statements
            made in the above text. For all I know I might be
            drugged or something..
            No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

In article <slrnbfj0dq.224f.no@toad.stack.nl>, Willem wrote:
> Andy wrote:
> ) As part of a cron job I have running, I'm using gzip -c to append
> ) compressed data to the end of an existing gzip file. According to the
> ) gzip man page this should work, and for the most part it does. It seems,
> ) however, that in some cases this doesn't work. While gunzip and winzip
> ) are both happy opening the file, WinRAR complains about a failed CRC in
> ) the file, and PKUnzip only unzips part of the file.
> ) 
> ) If I gzip -dc|gzip the file, WinRAR and PKUnzip are perfectly happy to
> ) open the file until next time a new portion is appended to it.
> ) 
> ) Does anyone have an explaination for this? I don't really want to have
> ) to unzip and re-zip each file every night, as the computational cost
> ) would be far too high.
> 
> The simplest explanation would be that both WinRAR and PKUnzip don't fully
> comply with the gzip standard, and can therefore not read all valid gzipped
> files.  Complain to the authors of WinRAR and/or PKUnzip.
 
Will do. I just figured that this might have been something people had
come across before.
 
> And ask yourself why you have to use one of those programs to open those
> files, and/or simply use gzip to open them.  (Or look for another program
> that correctly opens such files, probably any program using zlib.)

Believe me, if it was up to me I'd ban all customers from using anything
other than gzip to open these files. Unfortunately they seem to have a
mind of their own these days ;)

Thanks for the clarification, anyhow :)

-- 
Andy Furnell

Andy Furnell <andy@ipng.org.uk> wrote in message news:<slrnbfirdk.s26.andy.furnell@penfold.noc.clara.net>...
> Hi,
> 
> As part of a cron job I have running, I'm using gzip -c to append
> compressed data to the end of an existing gzip file. According to the
> gzip man page this should work, and for the most part it does. It seems,
> however, that in some cases this doesn't work. While gunzip and winzip
> are both happy opening the file, WinRAR complains about a failed CRC in
> the file, and PKUnzip only unzips part of the file.

I surprised it was even possible to concatenate compressed files. 
Maybe you could compress each file separately and add it to an
archive.

In article <8a1ed69a.0306250659.217acf79@posting.google.com>, Matt Mahoney wrote:
> Andy Furnell <andy@ipng.org.uk> wrote in message news:<slrnbfirdk.s26.andy.furnell@penfold.noc.clara.net>...
>> Hi,
>> 
>> As part of a cron job I have running, I'm using gzip -c to append
>> compressed data to the end of an existing gzip file. According to the
>> gzip man page this should work, and for the most part it does. It seems,
>> however, that in some cases this doesn't work. While gunzip and winzip
>> are both happy opening the file, WinRAR complains about a failed CRC in
>> the file, and PKUnzip only unzips part of the file.
> 
> I surprised it was even possible to concatenate compressed files. 
> Maybe you could compress each file separately and add it to an
> archive.

The FreeBSD man page would seem to imply that you can:

       Multiple  compressed  files  can  be concatenated. In this
       case, gunzip will extract all members at once.  For  exam-
       ple:
                     
             gzip -c file1  > foo.gz
             gzip -c file2 >> foo.gz

       Then
             gunzip -c foo

       is equivalent to

             cat file1 file2
             
AIUI the only disadvantage to doing this is that you lose compression
compared to concatenating the file, then gzipping it as a whole.

I did consider storing the gzipped files in an archive, but the object
of the exercise is to produce a single contiguous compressed log that
customers can download and make appreciative noises at rather than
coming up with a technically sound solution :)

A

-- 
Andy Furnell

Andy Furnell <andy@ipng.org.uk> wrote in message news:<slrnbfirdk.s26.andy.furnell@penfold.noc.clara.net>...
> While gunzip and winzip
> are both happy opening the file, WinRAR complains about a failed CRC in
> the file, and PKUnzip only unzips part of the file.

gzip (and winzip as well) are written to continue decompressing gzip
streams as long as there is still data in the input stream.  So you
can concatenate the output of independent gzip compressions and expect
gzip to decompress them in a single step, which is what you're doing. 
However it appears that WinRAR is looking to the end of the file for
the CRC (which it shouldn't), and PKUnzip is stopping after the first
gzip segement.

Either you can keep the output of the separate gzip runs in separate
files and decompress them that way, or you can suggest to your users
that they use gzip, which compiles on a large number of platforms.

mark