Rsync Question for the smart people

Hi All, I'm not great with Linux, so maybe someone can help me with this. I am running Red Hat 9.0. I set up an external USB drive and wrote a small korn shell script to rsync my drive to the external drive as a method of backup. The rsync started at about 8 minutes to complete. On March 22nd it had increased to 30 minutes. Then on March 23rd it jumped to 90 minutes to complete. Since I'd a dumbfuck, maybe someone can help me figure out why the time jumped like that? It should not take 90 minutes to rsync an 80gb drive, which is only 1/2 full.......... Any help or suggestions would be great. Thanks!

amerar@iwc.net <amerar@iwc.net> wrote: > I am running Red Hat 9.0. I set up an external USB drive and wrote a > small korn shell script to rsync my drive to the external drive as a > method of backup. > > The rsync started at about 8 minutes to complete. On March 22nd it had > increased to 30 minutes. Then on March 23rd it jumped to 90 minutes to > complete. Post your rsync command or script (if it's not too long) so that we can help troubleshoot the problem. G'luck... -- PLEASE post a SUMMARY of the answer(s) to your question(s)! Show Windows & Gates to the exit door. Unless otherwise noted, the statements herein reflect my personal opinions and not those of any organization with which I may be affiliated.

Thanks for the reply. Here is the code: rsync -uavx --exclude "mnt" --exclude "var/spool/postfix" --exclude "mnt" --exclude "newroot" --exclude "newcopy" / /mnt/newroot/newcopy It took me a long time to get this to work.....so, if the command is wrong, let me know....

On Fri, 21 Apr 2006 09:24:05 -0700, amerar@iwc.net wrote: > Thanks for the reply. Here is the code: > > rsync -uavx --exclude "mnt" --exclude "var/spool/postfix" --exclude > "mnt" --exclude "newroot" --exclude "newcopy" / /mnt/newroot/newcopy > > It took me a long time to get this to work.....so, if the command is > wrong, let me know.... > If you are using the external drive as a backup, then the "u" switch is obtuse. Explain how there could be newer files on the backup than on the disc itself. The "x" switch is one filesystem. This is redundant because you are manually excluding /mnt. The basic syntax I use is as follows: rsync -lptrv --progress --delete --exclude-from=$EXCLUSIONS $SOURCE $DEST CAUTION: rsync with the "delete" option can do great damage to your system. Practice using it until you are confident and have it down pat, possibly using the "--dry-run" switch in advance before committing. -- Douglas Mayne

amerar@iwc.net wrote: > rsync -uavx --exclude "mnt" --exclude "var/spool/postfix" --exclude > "mnt" --exclude "newroot" --exclude "newcopy" / /mnt/newroot/newcopy > > It took me a long time to get this to work.....so, if the command is > wrong, let me know.... Shouldn't you also exclude proc, sys and dev? -- Timothy Murphy e-mail (<80k only): tim /at/ birdsnest.maths.tcd.ie tel: +353-86-2336090, +353-1-2842366 s-mail: School of Mathematics, Trinity College, Dublin 2, Ireland

On 2006-04-21, amerar@iwc.net <amerar@iwc.net> wrote: > > Showing my ignorance......why exclude those filesystems??? Because they're not real filesystems but figments of the kernel's imagination. -- Robert Riches spamtrap42@verizon.net (Yes, that is one of my email addresses.)

On Friday 21 April 2006 21:51, Robert M. Riches Jr. stood up and spoke the following words to the masses in /comp.os.linux.misc...:/ > On 2006-04-21, amerar@iwc.net <amerar@iwc.net> wrote: >> >> Showing my ignorance......why exclude those filesystems??? > > Because they're not real filesystems but figments of the > kernel's imagination. */dev* is a real filesystem, mind you. It just doesn't hold regular files. ;-) -- With kind regards, *Aragorn* (Registered GNU/Linux user #223157)

On 2006-04-21, Aragorn <stryder@telenet.invalid> wrote: > On Friday 21 April 2006 21:51, Robert M. Riches Jr. stood up and spoke > the following words to the masses in /comp.os.linux.misc...:/ > >> On 2006-04-21, amerar@iwc.net <amerar@iwc.net> wrote: >>> >>> Showing my ignorance......why exclude those filesystems??? >> >> Because they're not real filesystems but figments of the >> kernel's imagination. > > */dev* is a real filesystem, mind you. It just doesn't hold regular > files. ;-) Doesn't that depend on whether you're using udev, devfs, or none of the above? I thought with devfs it was pseudo filesystem like /proc. -- Robert Riches spamtrap42@verizon.net (Yes, that is one of my email addresses.)

"amerar@iwc.net" <amerar@iwc.net> writes: >Hi All, >I'm not great with Linux, so maybe someone can help me with this. >I am running Red Hat 9.0. I set up an external USB drive and wrote a >small korn shell script to rsync my drive to the external drive as a >method of backup. >The rsync started at about 8 minutes to complete. On March 22nd it had >increased to 30 minutes. Then on March 23rd it jumped to 90 minutes to >complete. >Since I'd a dumbfuck, maybe someone can help me figure out why the time >jumped like that? It should not take 90 minutes to rsync an 80gb >drive, which is only 1/2 full.......... Of course we have no idea. There is not enough info here. Maybe your little kid keeps coming in and stopping the process. 80GB is a lot of data, and copying out 80GB is not that fast. On a USB port at 40MB/s (which is more or less the max ideal transfer rate for usb2-- that is 2000 sec which is just under an hour. And that assumes the idea case. Now if yuor files change, your computer has to read off the stuff from the usb drive, compute the md5 sum, and transfer the changed stuff back to the drive. Ie, in this case rsync has no advantage over just copying all of the stuff over. The advantage of rsync, that only the changes are transfered is completely lost since the stuff needs to be all transfered in order to figure out that changes have taken place. Ie, everything has to be read off the drive, the sum calculated, and then the changes sent back onto the drive. Ie, rsync is slower than copying, in the worst case at least twice as slow. >Any help or suggestions would be great. >Thanks!

"amerar@iwc.net" <amerar@iwc.net> writes: >Showing my ignorance......why exclude those filesystems??? Because you do NOT want to transfer them. However the -x should take care of that. Those filesystem do contain files with content. Some of that content can be very slow to read (on a read a request is sent to the kernel to please generate the data that file is supposed to display, and the kernel generates it, which could take a while.)

On 21 Apr 2006 22:40:26 GMT, Unruh staggered into the Black Sun and said: > "amerar@iwc.net" <amerar@iwc.net> writes: >>I am running Red Hat 9.0. Like other people said, you want to update this to something more recent. This has no bearing on your rsync problem, but Redhat 9's been EOLed for a while now and keeping it up to date is/will be difficult. >>I set up an external USB drive and wrote a small korn shell script Most scripts on Linux are in bash, not ksh. Probably won't affect anything, but on Linux systems, bash is guaranteed to be available and ksh isn't. > Now if yuor files change, your computer has to read off the stuff from > the usb drive, compute the md5 sum, and transfer the changed stuff > back to the drive. Not quite. rsync uses mtime and file size, not checksums, unless you pass it the -c or --checksum options. If you didn't include -c, and mtime, path, and file size are the same on both source and dest, rsync moves on to the next file. (And even when you pass it -c, it uses MD4, not MD5, according to the man page.) It does have to stat() all the files on both source and destination, which could be a bit of I/O, but definitely less than md5summing everything. amerar may want to look at the --size-only option to rsync if the destination disk is FAT or the destination disk gets used by non-Linux systems which may handle timestamps differently. > The advantage of rsync, that only the changes are transfered is > completely lost since the stuff needs to be all transfered in order to > figure out that changes have taken place. That's why the rsync developers didn't make checksum use the default behavior. -- Matt G|There is no Darkness in Eternity/But only Light too dim for us to see Brainbench MVP for Linux Admin / mail: TRAP + SPAN don't belong http://www.brainbench.com / "He is a rhythmic movement of the -----------------------------/ penguins, is Tux." --MegaHAL

On 2006-04-21, Dances With Crows wrote: > On 21 Apr 2006 22:40:26 GMT, Unruh staggered into the Black Sun and > said: >> "amerar@iwc.net" <amerar@iwc.net> writes: >>>I am running Red Hat 9.0. > [snip] >>>I set up an external USB drive and wrote a small korn shell script > > Most scripts on Linux are in bash, not ksh. Probably won't affect > anything, but on Linux systems, bash is guaranteed to be available and > ksh isn't. RedHat were late in moving to bash2; I think that bash in RH9.0 may still have been bash1, in which case ksh (actually pdksh) may be the better choice, and is just about as ubiquitous as bash. The great majority of scripts can be written using only POSIX syntax which will work in either bash (even bash1) as well as ksh and [d]ash, etc.. -- Chris F.A. Johnson, author | <http://cfaj.freeshell.org> Shell Scripting Recipes: | My code in this post, if any, A Problem-Solution Approach | is released under the 2005, Apress | GNU General Public Licence

On Friday 21 April 2006 23:56, Robert M. Riches Jr. stood up and spoke the following words to the masses in /comp.os.linux.misc...:/ > On 2006-04-21, Aragorn <stryder@telenet.invalid> wrote: >> On Friday 21 April 2006 21:51, Robert M. Riches Jr. stood up and >> spoke the following words to the masses in /comp.os.linux.misc...:/ >> >>> On 2006-04-21, amerar@iwc.net <amerar@iwc.net> wrote: >>>> >>>> Showing my ignorance......why exclude those filesystems??? >>> >>> Because they're not real filesystems but figments of the >>> kernel's imagination. >> >> */dev* is a real filesystem, mind you. It just doesn't hold regular >> files. ;-) > > Doesn't that depend on whether you're using udev, devfs, or > none of the above? I thought with devfs it was pseudo > filesystem like /proc. You could be right on account of /devfs./ As far as I know, /devfs/ ran in kernel mode, and its contents may have been virtual. I don't really know, as I don't like it too much and never bothered to check it out fully. I do have an older laptop that makes use of it, but I hate the naming conventions for the devices in /devfs./ /udev/ however does create real device nodes - which can be backed up into a tarball - but it typically does so on a /ramfs/ rather than on a hard disk filesystem. -- With kind regards, *Aragorn* (Registered GNU/Linux user #223157)