Hi everyone,

I'm having a problem piping the output of tar (for a backup) through the
following script "cut.sh":

  - - - - - - - - -
#!/bin/bash
SYNTAX="Syntax: $0 [target_file_prefix] [segment_size]"
if [ $# != 2 ]; then
     echo $SYNTAX
     exit -1
fi
TARGET_FILE_PREFIX=$1
SEGMENT_SIZE=$2
PART_NBR=1
while /bin/true; do
     PART_NAME=$TARGET_FILE_PREFIX.$PART_NBR
     dd bs=$SEGMENT_SIZE count=1 2>/tmp/dd.stderr | tee $PART_NAME
     RESULT=`cat /tmp/dd.stderr`
     echo $RESULT | grep -q "0+"
     if [ $? == 0 ]; then
         echo $RESULT | grep -q "0+0"
         if [ $? == 0 ]; then
             # Last part is empty - delete it
             rm -f $PART_NAME
         fi
         exit 0
     fi
     PART_NBR=`expr $PART_NBR + 1`
done
  - - - - - - -

If I do:
   dd if=/dev/urandom bs=10240 count=1 | ./cut.sh output 1024 >/dev/null
it works, and generates 10 files of 1024 bytes: output.1 to output.10

Trouble is, if I pipe the output of tar through ./cut.sh it terminates
unexpectedly, without an error message.

I really can't figure this out :-S

Maybe it's due to the several calls to dd, breaking stdin ?
Please help...
TIA

Jo�l

On Wed, 02 Jul 2003 21:25:02 -0100, Jo�l Bourquard staggered into the
Black Sun and said:
> Here's a much, much simpler example that exhibits the issue:
> 
> tar cv / | dd bs=1M count=1 of=output_file   => output_file 1 MB, ok
> tar cv / | dd bs=1M count=2 of=output_file   => output_file 2 MB, ok
> tar cv / | dd bs=1M count=8 of=output_file   => output_file 5 MB ??????

What are you trying to accomplish here?

When you specify neither "f" nor "O", tar writes to a default device,
usually /dev/tape .  One "v" will make tar write the name of each file
it writes to stdout.  So, "tar cv directory" writes the contents of
directory/ to /dev/tape , and writes a list of all the files in
directory/ to stdout.  Why are you piping the file listing through dd?
Just use > if you're writing to a file.

If you want to split a tar archive up into multiple parts, why not just
use split?  Like so:

tar cO directory | split --bytes=2m - archive

....creates archiveaa , archiveab, archiveac [...] archivezz , each 2M or
smaller, put them all together with cat and you'll get a tar archive.
Extract like so:

cat archive* | tar x

> btw, the platform is Redhat 9 x86.

OK.

-- 
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see
Brainbench MVP for Linux Admin / 
http://www.brainbench.com     /  "He is a rhythmic movement of the
-----------------------------/    penguins, is Tux." --MegaHAL

Dances With Crows wrote:

> What are you trying to accomplish here?
> 
> When you specify neither "f" nor "O", tar writes to a default device,
> usually /dev/tape .  One "v" will make tar write the name of each file
> it writes to stdout.  So, "tar cv directory" writes the contents of
> directory/ to /dev/tape , and writes a list of all the files in
> directory/ to stdout.

Thanks for your answer ! If you can, please read on, though

Actually with tar cv directory | (...)  it happens to work on my system, 
ie: the binary data is piped into the requested file (and verbosity goes 
to the console). Maybe I'm just lucky with the version of tar.

> Why are you piping the file listing through dd?
> Just use > if you're writing to a file.

The purpose was to cut the data in 1GB pieces. Otherwise I would have 
been using '>'.

> If you want to split a tar archive up into multiple parts, why not just
> use split?  Like so:
> 
> tar cO directory | split --bytes=2m - archive

Thanks !!  I didn't know about this great command, just what I needed :-)

> ...creates archiveaa , archiveab, archiveac [...] archivezz , each 2M or
> smaller, put them all together with cat and you'll get a tar archive.
> Extract like so:
> 
> cat archive* | tar x

Neat.

I'm still worried about the following though - if you do this on your 
system:

joel ~]>cat /dev/zero | dd bs=1M count=32 of=/dev/null
7+25 records in
7+25 records out
[joel ~]>cat /dev/zero | dd bs=1M count=32 of=/dev/null
26+6 records in
26+6 records out
[joel ~]>cat /dev/zero | dd bs=1M count=32 of=/dev/null
23+9 records in
23+9 records out
[joel ~]>cat /dev/zero | dd bs=1M count=32 of=/dev/null
11+21 records in
11+21 records out

... do you get inconsistent, non-deterministic results, too ? I find this 
a bit scary, do you know why this happens ?

Best Regards,
Jo�l

Joel Bourquard <nospam---numlock@freesurf.ch> wrote:
> rnichols@interaccess.com wrote:
> Thanks, I didn't know that at all..  when making the script, I thought
> only the last "BLOCKS" input block could be read incompletely, that is
> when dd received an EOF through the pipe. Now I understand why it didn't
> work well.

> The 'dd' command didn't do the exact same thing every time, so I assume
> the 'block completeness' was also a timing issue, wasn't it ?

You want to buffer the output.	I'm surprised you see an effect with
whatever you are doing, though, since stdout is blocking.

>> Even if the 'dd' blocksize matches the 'tar' blocksize, you can still
>> get partial reads due to interaction with the OS's pipe buffer size.

It would be extremely unusual.

> Oh, I see now. But if I do something like:
>	dd if=/dev/hda of=hdd_backup bs=512 count=x

> .. do we have the guarantee that 'x' full blocks will be saved, or is it
> the same thing as before ?

Each block might individually fail. The blocksize for the device is
likely 1024, btw.

>> BTW, not all versions of 'tar' use stdout as the default output device.
>> I suggest _always_ using the "f" option to specify an output device
>> explicitly.

> Do you mean, using -f /dev/stdout ?

No, he means "-f -".

> How can I keep functionality of the following:

>	tar -czv $FILE_SPEC | tee $PIPE1 | split ....

> It sends the binary data to tee's stdin, and displays the filenames on
> the terminal. I suppose they are sent through stderr, but am not sure.

I don't quite know what you mean. I pipe tar through tee and a fifo
often, in order to calculate a running md5sum.

> The $PIPE1 is a file created with mkfifo - very useful to do a checksum
> on-the-fly. The two relevant lines are:

Ah. You too.

> cat $PIPE1 | sha1sum | sed s/-/$TARGET_FILE_NAME/ > $TARGET_FILE.sha1 &
> tar $SWITCHES $FILE_SPEC | tee $PIPE1 | split --bytes=$SEGMENT_SIZE -
> $TARGET_FILE.

> Pipes are very neat, but after a test, it seems the sha1sums don't
> match. Oh well..  script learning is nice :-)

They do match. I do it all the time. Make sure you are not doing
something silly.

Peter

Jo�l Bourquard <nospam---numlock@freesurf.ch> wrote:
> I'm having a problem piping the output of tar (for a backup) through the
> following script "cut.sh":

> If I do:
>    dd if=/dev/urandom bs=10240 count=1 | ./cut.sh output 1024 >/dev/null
> it works, and generates 10 files of 1024 bytes: output.1 to output.10

Does split(1) do what you need instead of cut.sh?

Chris

rnichols@interaccess.com wrote:
> You are probably seeing the result of the pipe containing less than
> 10240 bytes at the time 'dd' performs a read.  I don't know of any way
> to make 'dd' do what you want here because "count=BLOCKS" does not
> guarantee that any of the "BLOCKS" input blocks were actually full, and
> a partial block still counts as "1 block" for satisfying the count.  You
> can easily verify this behavior:
> 
>   (for X in 1 2 3 4 5;do echo "X is $X";sleep 1;done) | dd bs=512 count=4
 > You'll see that 'dd' terminates with far less than 4*512 bytes copied.

Thanks, I didn't know that at all..  when making the script, I thought 
only the last "BLOCKS" input block could be read incompletely, that is 
when dd received an EOF through the pipe. Now I understand why it didn't 
work well.

The 'dd' command didn't do the exact same thing every time, so I assume 
the 'block completeness' was also a timing issue, wasn't it ?

By the way.. the 'split' command is safe (ie: always copies everything), 
isn't it ?

> Even if the 'dd' blocksize matches the 'tar' blocksize, you can still
> get partial reads due to interaction with the OS's pipe buffer size.

Oh, I see now. But if I do something like:
	dd if=/dev/hda of=hdd_backup bs=512 count=x

... do we have the guarantee that 'x' full blocks will be saved, or is it 
the same thing as before ?

> BTW, not all versions of 'tar' use stdout as the default output device.
> I suggest _always_ using the "f" option to specify an output device
> explicitly.

Do you mean, using -f /dev/stdout ?

How can I keep functionality of the following:

	tar -czv $FILE_SPEC | tee $PIPE1 | split ....

It sends the binary data to tee's stdin, and displays the filenames on 
the terminal. I suppose they are sent through stderr, but am not sure.

The $PIPE1 is a file created with mkfifo - very useful to do a checksum 
on-the-fly. The two relevant lines are:

cat $PIPE1 | sha1sum | sed s/-/$TARGET_FILE_NAME/ > $TARGET_FILE.sha1 &
tar $SWITCHES $FILE_SPEC | tee $PIPE1 | split --bytes=$SEGMENT_SIZE - 
$TARGET_FILE.

Pipes are very neat, but after a test, it seems the sha1sums don't 
match. Oh well..  script learning is nice :-)

Best Regards,

Jo�l

Jo�l Bourquard wrote:
> The $PIPE1 is a file created with mkfifo - very useful to do a checksum 
> on-the-fly. The two relevant lines are:
> 
> cat $PIPE1 | sha1sum | sed s/-/$TARGET_FILE_NAME/ > $TARGET_FILE.sha1 &
> tar $SWITCHES $FILE_SPEC | tee $PIPE1 | split --bytes=$SEGMENT_SIZE - 
> $TARGET_FILE.
> 
> Pipes are very neat, but after a test, it seems the sha1sums don't 
> match. Oh well..  script learning is nice :-)

Never mind the part about checksum mismatch - the backup *is* fine, I 
was just cat'ing the wrong files when checking it...

Jo�l

In article <be0rfj$10jp7b$1@ID-195945.news.dfncis.de>,
=?ISO-8859-1?Q?Jo=EBl_Bourquard?=  <nospam---numlock@freesurf.ch> wrote:
:rnichols@interaccess.com wrote:
[SNIP]
:The 'dd' command didn't do the exact same thing every time, so I assume 
:the 'block completeness' was also a timing issue, wasn't it ?

Yes.

:By the way.. the 'split' command is safe (ie: always copies everything), 
:isn't it ?

Yes, 'split' is counting bytes.  AFAIK, 'dd' is unique in it's counting
of read() operations.

:> Even if the 'dd' blocksize matches the 'tar' blocksize, you can still
:> get partial reads due to interaction with the OS's pipe buffer size.
:
:Oh, I see now. But if I do something like:
:	dd if=/dev/hda of=hdd_backup bs=512 count=x
:
:.. do we have the guarantee that 'x' full blocks will be saved, or is it 
:the same thing as before ?

It's dependent on the underlying device driver.  AFAIK, the disk
drivers will always return the full number of bytes requested unless
EOF or an error is encountered.  Pipes, FIFOs, sockets, serial ports,
etc. can return as little as 1 byte.  A tape drive will return 1 tape
block, which might be smaller than the count passed to read().

:> BTW, not all versions of 'tar' use stdout as the default output device.
:> I suggest _always_ using the "f" option to specify an output device
:> explicitly.
:
:Do you mean, using -f /dev/stdout ?

As someone else already noted, "-f -".

:How can I keep functionality of the following:
:
:	tar -czv $FILE_SPEC | tee $PIPE1 | split ....
:
:It sends the binary data to tee's stdin, and displays the filenames on 
:the terminal. I suppose they are sent through stderr, but am not sure.

Tar is fairly intelligent about that.  The "-v" output goes to stdout
_unless_ stdout is being used for the data stream, in which case the
"-v" output goes to stderr.

-- 
Bob Nichols         rnichols@interaccess.com

Peter T. Breuer wrote:
> You want to buffer the output.	I'm surprised you see an effect with
> whatever you are doing, though, since stdout is blocking.

I was surprised too.. at first I naively thought that something like: 
cat /dev/zero | dd bs=1M count=32 of=/dev/null

would behave always the same, but it isn't the case (at least on RH9).

> Each block might individually fail. The blocksize for the device is
> likely 1024, btw.

Ok. So I guess, it could only happen when the drive encounters a 
physical defect ?

>>Do you mean, using -f /dev/stdout ?
> 
> 
> No, he means "-f -".

Ok. Thanks !

>>The $PIPE1 is a file created with mkfifo - very useful to do a checksum
>>on-the-fly. The two relevant lines are:
> 
> 
> Ah. You too.

Yeah, it's a truly great thing.

>>Pipes are very neat, but after a test, it seems the sha1sums don't
>>match. Oh well..  script learning is nice :-)
> 
> 
> They do match. I do it all the time. Make sure you are not doing
> something silly.

Never mind - when verifying the sum after backuping, I cat'd both the 
data and the sha1 output through sha1sum..  silly me :-)

> 
> Peter

Thanks a lot for your reply, Peter. I just got a bit more comfortable 
with pipes !

Regards,
Jo�l

(I mis-posted my answer on the bottom of the whole thread)