Hi

I need to migrate 1000's of files from one server to another. There's
over 100GB in total, and it needs to be copied out of hours if possible
(IE I hope to do this in a few hours rather than a few days!).

Initially I trialled the copy using NFS and cp, and found it takes
about 2 days! Any ideas of a quicker way to copy 100GB of small files?
Many thanks for any tips in advance,
Al.

In article <1106009855.324738.83440@f14g2000cwb.googlegroups.com>,
 "al" <aldoofbanga@hotmail.com> wrote:

> I need to migrate 1000's of files from one server to another. There's
> over 100GB in total, and it needs to be copied out of hours if possible
> (IE I hope to do this in a few hours rather than a few days!).
> 
> Initially I trialled the copy using NFS and cp, and found it takes
> about 2 days! Any ideas of a quicker way to copy 100GB of small files?
> Many thanks for any tips in advance,

If you did a backup/restore from SCSI tape with that volume of data, it 
would take about 30-40 hours, which is close to what you found with NFS.  
This assumes the average restore of 3GB/hour.  Have you tried a 
backup/restore over the network via a pipe?  That might be faster.

With this volume of data, you're more or less screwed.  Unless you can 
mirror it, break the mirror and move the disks over to the new system, 
there's not much more you can do.

Good luck.

-- 
DeeDee, don't press that button!  DeeDee!  NO!  Dee...

On Mon, 17 Jan 2005 16:57:35 -0800, al wrote:

> Initially I trialled the copy using NFS and cp, and found it takes
> about 2 days! Any ideas of a quicker way to copy 100GB of small files?

There is something wrong with your network.  I just copied
sol-9-u7-sparc-v1.iso from a Solaris x86 NFS server to my home directory
on a U60 running s10_b72.  The file size is 691732480 bytes.  The network
is switched 100baseTX.

[ISOS]# time cp sol-9-u7-sparc-v1.iso /export/home/duhring

real    1m11.071s

I.e., the transfer rate was 9.28+ MB/sec and 100 GB should transfer in
11035 sec. (3.07 hr.).

Thanks for the tip. I'll have a further think about the mirroring idea,
I hadn't thought of that.

Thanks, that was the sort of timing I was hoping for. The problem I
have, is that this is not one file, There's actually about 7 million
files! So I think that is where my performance problem is coming from.

PS, any idea how to back/restore using a PIPE? I've never done that
before.
I had tried using "scp" but that didn't seem much different to NFS cp.

In article <1106021172.432838.36580@z14g2000cwz.googlegroups.com>,
 "al" <aldoofbanga@hotmail.com> wrote:

> PS, any idea how to back/restore using a PIPE? I've never done that
> before.
> I had tried using "scp" but that didn't seem much different to NFS cp.

I seem to recall an example of this in the ufsbackup man page (albeit a 
local system example):

ufsdump 0f - /dev/rdsk/c0t0d0s7 | (cd /home; ufsrestore xf -)

You can expand this by setting up a .rhosts file on the remote system. 
Yes, I _know_ ssh is better, but I don't know if that will work here.  
rsh does 'cause I've used it.  Anyway,

ufsdump 0f - /dev/rdsk/c0t0d0s7 | rsh <remote> "cd /home; ufsrestore xf 
-"

This also works with tar, but ufsdump is faster.  Note, you probably 
don't want to do this during prime time or on a 10BaseT link.  FDDI, 
100BaseT or greater will keep the network people out of your cube.

-- 
DeeDee, don't press that button!  DeeDee!  NO!  Dee...

Michael that's not a bad idea, I'll try this. I have never tried
ufsdump/ufsrestore before when the source/target disks are different,
but I guess it should work since the target disk cluster is bigger than
the source.

I'll double check (I'm not physically located near these servers) but I
am sure the network is 100BaseT.

I suspect the "many small files" aspect may be contributing to the issue.
Have you considered tarring and compressing them first? This would reduce
the amount that needs to be transferred over the network.

For bonus points, you could do something like this:

tar c /path/to/files | ssh -C user@remotehost 'tar x'

That'll pipe the tar file straight to network, ssh -C will compress it, and
tar x will unpack it on the remote side. 

Hope this helps.

al wrote:

> Hi
> 
> I need to migrate 1000's of files from one server to another. There's
> over 100GB in total, and it needs to be copied out of hours if possible
> (IE I hope to do this in a few hours rather than a few days!).
> 
> Initially I trialled the copy using NFS and cp, and found it takes
> about 2 days! Any ideas of a quicker way to copy 100GB of small files?
> Many thanks for any tips in advance,
> Al.

On Tue, 18 Jan 2005 16:03:50 +1100, The Locum Sysadmin wrote:

> For bonus points, you could do something like this:
> 
> tar c /path/to/files | ssh -C user@remotehost 'tar x'

For more bonus points one might read the man page for tar.  It is
impossible to pipe output which has been sent to /dev/rmt/0.

> That'll pipe the tar file straight to network, ssh -C will compress it, and
> tar x will unpack it on the remote side. 

The encryption/decryption of the stream will surely speed the transfer.

Yet more great advice, thanks all.

I'm trying Michael's advice at the moment, but having some weird
ufsdump/ufsrestore stuff going on.

I'll try this, it does appear like it will speed things up.

I'm wondering if the bottleneck is the disk, and not the network
however.... 100BaseT should pass this much info in a much shorter time.

Getting a few strange errors with this (see below). Maybe I'll try
tar/ssh, similar to what the other posters suggested...

nohup ufsdump 0f - /dev/rdsk/c2t40d0s6 | rsh [REMHOST] "cd /;
ufsrestore xf -" &
17667
root@lawu # Cannot open("/dev/tty"): No such device or address
root@lawu #
root@lawu # Volume is not in dump format

On Mon, 17 Jan 2005 21:46:20 -0800, al wrote:

> I'm wondering if the bottleneck is the disk, and not the network
> however.... 100BaseT should pass this much info in a much shorter time.

Your data transfer rate is over an order of magnitude less than what it
should be with a 100BaseTX network.  Surely your disks can read/write
faster than 10 MB/sec.

Dave Uhring wrote:

> For more bonus points one might read the man page for tar.  It is
> impossible to pipe output which has been sent to /dev/rmt/0.

Good point, my example syntax would only work for Gnu tar. I'd forgotten you
have to tell Solaris tar about stdout.

>> That'll pipe the tar file straight to network, ssh -C will compress it,
>> and tar x will unpack it on the remote side.
> 
> The encryption/decryption of the stream will surely speed the transfer.

On reasonable hardware, I suspect the encryption penalty will be less than
the penalty for an uncompressed tar file on what appears to be a fairly
slow network, but that's just an off-the-cuff guess. By all means, use
whatever suits you best.

al wrote:
> Hi
> 
> I need to migrate 1000's of files from one server to another. There's
> over 100GB in total, and it needs to be copied out of hours if possible
> (IE I hope to do this in a few hours rather than a few days!).
> 
> Initially I trialled the copy using NFS and cp, and found it takes
> about 2 days! Any ideas of a quicker way to copy 100GB of small files?
> Many thanks for any tips in advance,
> Al.
> 

The fastest way to copy files is using "star".  It can be downloaded at:

http://www.blastwave.org


The manpage for usage is at:

http://www.fokus.fhg.de/research/cc/glone/employees/joerg.schilling/private/man/star.html


An example from the Manpage for copying files:

To copy the directory tree in /home/someuser to the directory /home/fs

use:

example% (cd /home/someuser; star -c .) | (cd /home/fs ; star-xp)



I haven't done any time trials myself, but it's been discussed here in 
the past.

Paul

al <aldoofbanga@hotmail.com> wrote:
> Hi

> I need to migrate 1000's of files from one server to another. There's
> over 100GB in total, and it needs to be copied out of hours if possible
> (IE I hope to do this in a few hours rather than a few days!).

Do you have connectivity prior to the "few hours" window?  If so 'rsync'
may be appropriate here.

Make an initial run early to get most of the data over.  Then when
you're ready for the real cutover, run it again.  Since most of the data
will be in place, it should take only a fraction of the time.

One drawback is that rsync can take a lot of time and memory with the
file directory trees and you've said that you have lots and lots of
files.  However, if it runs on your data, I'd expect it would still be a
time saver.

-- 
Darren Dunham                                           ddunham@taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

In article <1106009855.324738.83440@f14g2000cwb.googlegroups.com>,
al <aldoofbanga@hotmail.com> wrote:

>I need to migrate 1000's of files from one server to another. There's
>over 100GB in total, and it needs to be copied out of hours if possible
>(IE I hope to do this in a few hours rather than a few days!).
>
>Initially I trialled the copy using NFS and cp, and found it takes
>about 2 days! Any ideas of a quicker way to copy 100GB of small files?
>Many thanks for any tips in advance,

The best idea is to use star to copy (see man page) and to make sure
that the source filesystem is the NFS mounted FS while the target FS
is local.

I expect the copy to take about 12 hours.


-- 
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) J�rg Schilling D-13353 Berlin
      js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
      schilling@fokus.fraunhofer.de	(work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

In article <vilain-5E1B3D.18082517012005@news.giganews.com>,
Michael Vilain  <vilain@spamcop.net> wrote:

>If you did a backup/restore from SCSI tape with that volume of data, it 
>would take about 30-40 hours, which is close to what you found with NFS.  
>This assumes the average restore of 3GB/hour.  Have you tried a 
>backup/restore over the network via a pipe?  That might be faster.

star -copy is about 30% faster than ufsdump | ufsrestore.



-- 
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) J�rg Schilling D-13353 Berlin
      js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
      schilling@fokus.fraunhofer.de	(work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

In article <pan.2005.01.18.02.51.55.730542@yahoo.com>,
Dave Uhring  <daveuhring@yahoo.com> wrote:
>On Mon, 17 Jan 2005 16:57:35 -0800, al wrote:
>
>> Initially I trialled the copy using NFS and cp, and found it takes
>> about 2 days! Any ideas of a quicker way to copy 100GB of small files?
>
>There is something wrong with your network.  I just copied
>sol-9-u7-sparc-v1.iso from a Solaris x86 NFS server to my home directory
>on a U60 running s10_b72.  The file size is 691732480 bytes.  The network
>is switched 100baseTX.
>
>[ISOS]# time cp sol-9-u7-sparc-v1.iso /export/home/duhring
>
>real    1m11.071s

Your example is not realistic as there is a big difference between
copying one big file and copying a lot of small files.



-- 
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) J�rg Schilling D-13353 Berlin
      js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
      schilling@fokus.fraunhofer.de	(work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

In article <1106020837.879817.263200@f14g2000cwb.googlegroups.com>,
al <aldoofbanga@hotmail.com> wrote:
>Thanks for the tip. I'll have a further think about the mirroring idea,
>I hadn't thought of that.

If you start miroring, you need to opy everything......

You may use star to do incremental miroring and my experiences are that
approx. 10 GB of dayly changes in the FS take approx. 45 minutes to
sync the target FS.


-- 
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) J�rg Schilling D-13353 Berlin
      js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
      schilling@fokus.fraunhofer.de	(work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

In article <vilain-A087D0.20265417012005@news.giganews.com>,
Michael Vilain  <vilain@spamcop.net> wrote:

>I seem to recall an example of this in the ufsbackup man page (albeit a 
>local system example):
>
>ufsdump 0f - /dev/rdsk/c0t0d0s7 | (cd /home; ufsrestore xf -)
>
>You can expand this by setting up a .rhosts file on the remote system. 
>Yes, I _know_ ssh is better, but I don't know if that will work here.  
>rsh does 'cause I've used it.  Anyway,
>
>ufsdump 0f - /dev/rdsk/c0t0d0s7 | rsh <remote> "cd /home; ufsrestore xf 
>-"
>
>This also works with tar, but ufsdump is faster.  Note, you probably 
>don't want to do this during prime time or on a 10BaseT link.  FDDI, 
>100BaseT or greater will keep the network people out of your cube.

Not correct: using star to copy is typically 30% faster than using the
ufsdump | ufsrestore pipe...

-- 
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) J�rg Schilling D-13353 Berlin
      js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
      schilling@fokus.fraunhofer.de	(work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

In article <K%1Hd.8085$tu2.7514@fe10.lga>, Paul Gress  <pgress@pb.net> wrote:
>al wrote:

>An example from the Manpage for copying files:
>
>To copy the directory tree in /home/someuser to the directory /home/fs
>
>use:
>
>example% (cd /home/someuser; star -c .) | (cd /home/fs ; star-xp)
>
>I haven't done any time trials myself, but it's been discussed here in 
>the past.

Using star -copy is even faster than star -c | star -x


-- 
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) J�rg Schilling D-13353 Berlin
      js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
      schilling@fokus.fraunhofer.de	(work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

On Tue, 18 Jan 2005 11:16:46 +0000, Joerg Schilling wrote:

> Your example is not realistic as there is a big difference between
> copying one big file and copying a lot of small files.

It is sufficiently realistic to give the OP a clue about how to determine
that his network is either 10BaseT or has a duplex mismatch.

al <aldoofbanga@hotmail.com> wrote:
> Hi
> 
> I need to migrate 1000's of files from one server to another. There's
> over 100GB in total, and it needs to be copied out of hours if possible
> (IE I hope to do this in a few hours rather than a few days!).
> 
> Initially I trialled the copy using NFS and cp, and found it takes
> about 2 days! Any ideas of a quicker way to copy 100GB of small files?
> Many thanks for any tips in advance,
> Al.

Have you considered using rsync? You can copy for days (or weeks) in
the background, and then sync in next to no time (relatively speaking)
during your outage. We've migrated terabytes of data in this manner.

Colin

js@cs.tu-berlin.de (Joerg Schilling) writes:

>In article <pan.2005.01.18.02.51.55.730542@yahoo.com>,
>Dave Uhring  <daveuhring@yahoo.com> wrote:
>>On Mon, 17 Jan 2005 16:57:35 -0800, al wrote:
>>
>>> Initially I trialled the copy using NFS and cp, and found it takes
>>> about 2 days! Any ideas of a quicker way to copy 100GB of small files?
>>
>>There is something wrong with your network.  I just copied
>>sol-9-u7-sparc-v1.iso from a Solaris x86 NFS server to my home directory
>>on a U60 running s10_b72.  The file size is 691732480 bytes.  The network
>>is switched 100baseTX.
>>
>>[ISOS]# time cp sol-9-u7-sparc-v1.iso /export/home/duhring
>>
>>real    1m11.071s

>Your example is not realistic as there is a big difference between
>copying one big file and copying a lot of small files.

Especially where the network latency is converned; and the best
suggestions are indeed to bulk read the files on the current system
and then transport the packaged bits and unpack on the remote
host.  This limits the file open and file create latency to
the shortest possible path: local system to directly attached storage.

Casper
-- 
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Darren Dunham wrote:
> al <aldoofbanga@hotmail.com> wrote:
> 
>>Hi
> 
> 
>>I need to migrate 1000's of files from one server to another. There's
>>over 100GB in total, and it needs to be copied out of hours if possible
>>(IE I hope to do this in a few hours rather than a few days!).
> 
> 
> Do you have connectivity prior to the "few hours" window?  If so 'rsync'
> may be appropriate here.
> 
> Make an initial run early to get most of the data over.  Then when
> you're ready for the real cutover, run it again.  Since most of the data
> will be in place, it should take only a fraction of the time.
> 
> One drawback is that rsync can take a lot of time and memory with the
> file directory trees and you've said that you have lots and lots of
> files.  However, if it runs on your data, I'd expect it would still be a
> time saver.
> 

I'd also go with rsync, as like Darren says, you can start it at 
anytime, and then just before your cutover, run it once more to update 
and *then* offline the initial disks.

Rsync will also run with compression, and on doing updates to an 
existing copy of data, it only transfers modified portions of files.

alan.
-- 
Alan Hargreaves - http://blogs.sun.com/tpenta
Kernel/VOSJEC/Performance Engineer
Product Technical Support (APAC)
Sun Microsystems

Thanks again for great advice.

I will experiment with rsync, I haven't used it before, it certainly
sounds like the way to go ! I am about to check the man page, but I
assume it will remove files on the target machine as well, if they have
been deleted from the source.

al <aldoofbanga@hotmail.com> wrote:
> I will experiment with rsync, I haven't used it before, it certainly
> sounds like the way to go ! I am about to check the man page, but I
> assume it will remove files on the target machine as well, if they have
> been deleted from the source.

As I've done something similar some time ago, and did some experiments
on what would be the fastest way, I would recommend a two-step procedure,
too. Copy the whole filesystem with tar while in production, and use
rsync during downtime to update the copy.

For the initial copy I tested rsync, ufsdump|ufsrestore and tar, with
tar being the fastest. I exported the new filesystem via NFS and used:

  tar cf - <dir> | ( cd /net/newhost/newdir && tar xf - )

For the later rsync I'd recommend these options:

  rsync -avH --delete <dir> /net/newhost/<dir>

-H is important so hard links are recreated as they are, otherwise a
copy of the original file is created. --delete is necessary so files
on the destination directory are removed if they don't exist in the
source directory anymore. Use -n before the real run, as it will tell
you what will be copied without actually modifying anything.

The only difference in source and destination after this operation
was that timestamps of symbolic links were set to the current date/time,
and the sizes of some directories were reduced (that's because they
usually aren't shrunk after a lot of files in a directory have been
created and deleted).

mp.
-- 
Systems Administrator | Institute for Software Science | Univ. of Vienna

The Locum Sysadmin (locum_sysad@yahoo.com) wrote:
: >> That'll pipe the tar file straight to network, ssh -C will compress it,
: >> and tar x will unpack it on the remote side.
: > 
: > The encryption/decryption of the stream will surely speed the transfer.
: 
: On reasonable hardware, I suspect the encryption penalty will be less than
: the penalty for an uncompressed tar file on what appears to be a fairly
: slow network, but that's just an off-the-cuff guess. By all means, use
: whatever suits you best.

I do lots of copies over 100mb links.  I find that gzip is faster
than openssh's.  On older slower systems, I've even built openssh
with either arcfour or even the null encryption.

When I need to do these types of copies, I will tend to use find/cpio/scp
to move the old files 1st.  Then use something like rsync to copy
the recent files.

If you need speed and are willing to choke your network, use about
4 parallel FTP sessions to move the files.  If there is no other
traffic on the network, you can get very close to 100%

If you have extra ports, consider a direct link between the two
machines and if you have quad ethernet cards, then use as many in
parallel as you can at once.

-tim
http://web.abnormal.com

In article <41ee1ca6$0$10836$3b214f66@usenet.univie.ac.at>,
Martin Paul  <map@par.univie.ac.at> wrote:

>As I've done something similar some time ago, and did some experiments
>on what would be the fastest way, I would recommend a two-step procedure,
>too. Copy the whole filesystem with tar while in production, and use
>rsync during downtime to update the copy.
>
>For the initial copy I tested rsync, ufsdump|ufsrestore and tar, with
>tar being the fastest. I exported the new filesystem via NFS and used:
>
>  tar cf - <dir> | ( cd /net/newhost/newdir && tar xf - )

Sun's tar implementation is very slow and does not support
sparse files, better use star.

Star also implements a -copy flag that allows to avoid pipe related performance
problems while still managing to decouple the read from the write side as
star implements a shared mmemory based FIFO by default.

The following command lines will work:

-	star -copy [other_options] from_dir to_dir

	copies from_dir into to_dir in a way that the full "from_dir"
	path will appear inside to_dir


-	star -copy [other_options] file_1...file_n to_dir
	
	copies file_1...file_n into to_dir in a way that the full
	path from file_1...file_n will appear inside to_dir 


-	star -copy [other_options] -C from_dir . to_dir

	copies all files from from_dir into to_dir
	the file names visible in from_dir will all appear inside to_dir


Other options:

bs=	set other block size than 10k, useful values e.g. bs=1m
fs=	set other FIFO size than 8 MB, useful values e.g. fs=128m if you have
	enough memory in the machine.

-p		restore file permissions
-acl		copy ACLs
-sparse		Try to retain holes in files
-link-dirs	Try to retain hard linked directories
-xdot		Extract the '.' Directory even if the target is newer
-time		print timing statistics
-no-fsync	Do not call fsync(f) before closing written files.
		This may speed up by 10% on Solaris and by 400% on Linux.
		Note that with -no-fsync cannot tell you whether
		write errors on the extract side did happen.

A useful commandline looks this way:

star -copy -p -acl -sparse -link-dirs -xdot bs=1m fs=128m -time -C from_dir . to_dir

When one filesystem is NFS mounted, try to make the _source_ filesystem
the NFS fs as reading NFS is faster than writing NFS.

-- 
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) J�rg Schilling D-13353 Berlin
      js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
      schilling@fokus.fraunhofer.de	(work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

al <aldoofbanga@hotmail.com> wrote:
> Thanks again for great advice.
> 
> I will experiment with rsync, I haven't used it before, it certainly
> sounds like the way to go ! I am about to check the man page, but I
> assume it will remove files on the target machine as well, if they have
> been deleted from the source.

It doesn't. The idea behind it is to maintain consistency between two
copies of your data.

Colin B. <cbigam@somewhereelse.nucleus.com> wrote:
> al <aldoofbanga@hotmail.com> wrote:
>> Thanks again for great advice.
>> 
>> I will experiment with rsync, I haven't used it before, it certainly
>> sounds like the way to go ! I am about to check the man page, but I
>> assume it will remove files on the target machine as well, if they have
>> been deleted from the source.
> 
> It doesn't. The idea behind it is to maintain consistency between two
> copies of your data.

Gah! I just realised that I utterly misread your assumption. There is a flag
to delete files on the target or not. It's very versatile.

al wrote:
> Getting a few strange errors with this (see below). Maybe I'll try
> tar/ssh, similar to what the other posters suggested...
>
> nohup ufsdump 0f - /dev/rdsk/c2t40d0s6 | rsh [REMHOST] "cd /;
> ufsrestore xf -" &
> 17667
> root@lawu # Cannot open("/dev/tty"): No such device or address
> root@lawu #
> root@lawu # Volume is not in dump format

add -n to your rsh command  and add b to you ufsdump
ufsdump 0bf 256 - /dev/rdsk/c2t40d0s6,  This will increase the blocking
size and make better use of your networks.
Done this a lot and it works great.

js@cs.tu-berlin.de (Joerg Schilling) writes:

> A useful commandline looks this way:

> star -copy -p -acl -sparse -link-dirs -xdot bs=1m fs=128m -time \
> -C >from_dir . to_dir

When I try the following command:

    star -copy -p -sparse -link-dirs -xdot bs=1m fs=128m -time from_dir to_dir

I get the following output:

    star: 0 blocks + 251392 bytes (total of 251392 bytes = 245.50k).
    star: Total time 0.337sec (724 kBytes/sec)
    star: The following problems occurred during archive processing:
    star: Cannot: stat 0, open 0, read/write 0. Size changed 0.
    star: Missing links 0, Name too long 0, File too big 0, Not dumped 0.
    star: Cannot set: time 0, modes 1.
    star: Processed all possible files, despite earlier errors.

So, is this actually reporting a real problem?  If so, what is it telling me?
If there are no actual problems, is there a way to make star be silent unless
a real problem is present?

Regards,
-- 
C. Jeffery Small
jeff@cjsa.com

Yep, I found the --delete flag, cheers.

Thanks yet again for your time guys.

This is roughly what I'm running now:

rsync --rsync-path=/opt/csw/bin/rsync --delete -Harvz /source/
remote:/dest

One strange thing I found is that I have to run it in the foreground,
rsync is very touchy with the shell it is running in. I tried running
it with nohup, and it did not survive. I will try it with cron as
someone suggested before, this may work. Running in the foreground may
be ok, but as I'm physically in a different country it is a bit dodgy.
Thanks
Al.

In article <IAL0CC.A5u@cjsa.com>, Jeffery Small <jeff@cjsa.com> wrote:
>js@cs.tu-berlin.de (Joerg Schilling) writes:
>
>> A useful commandline looks this way:
>
>> star -copy -p -acl -sparse -link-dirs -xdot bs=1m fs=128m -time \
>> -C >from_dir . to_dir
>
>When I try the following command:
>
>    star -copy -p -sparse -link-dirs -xdot bs=1m fs=128m -time from_dir to_dir
>
>I get the following output:
>
>    star: 0 blocks + 251392 bytes (total of 251392 bytes = 245.50k).
>    star: Total time 0.337sec (724 kBytes/sec)
>    star: The following problems occurred during archive processing:
>    star: Cannot: stat 0, open 0, read/write 0. Size changed 0.
>    star: Missing links 0, Name too long 0, File too big 0, Not dumped 0.
>    star: Cannot set: time 0, modes 1.
>    star: Processed all possible files, despite earlier errors.
>
>So, is this actually reporting a real problem?  If so, what is it telling me?
>If there are no actual problems, is there a way to make star be silent unless
>a real problem is present?

As you did cut off the error message that did tell you why the mode 
(permissions) for one file could not be set, only you know why this did happen.

If you read the man page, you will find the error control feature that allows
you to define a pattern/error_type list for errors that should be ignored.
I believe that star is the only copy/backup program that gives you such a
feature.

BTW: 724 kBytes/sec is a bit slow but may be a result of the fact that
you did not copy much and the start up time counts too.

-- 
EMail:joerg@schily.isdn.cs.tu-berlin.de (home) J�rg Schilling D-13353 Berlin
      js@cs.tu-berlin.de		(uni)  If you don't have iso-8859-1
      schilling@fokus.fraunhofer.de	(work) chars I am J"org Schilling
URL:  http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily