Posted by: Dakusan
« on: September 10, 2012, 03:44:11 pm »Original post for Automatically resuming rsync can be found at https://www.castledragmire.com/Posts/Automatically_resuming_rsync.
Originally posted on: 09/10/12
Originally posted on: 09/10/12
Rsync is a spectacular bash utility for doing file copying and/or syncing operations. It has a multitude of switches to help optimize and handle any requirements for file copy operation over a local computer or network. However, sometimes networks are less than stable and stalls can happen during an rsync (or scp). This is quite the nuisance when doing very large (i.e. GB) transfers. To solve this, the following script can be used to auto resume a stalled rsync.
export Result=1;
while [ $Result -ne 0 ]; do
echo "STARTING ($Result) @" `date`;
rsync -Pza --timeout=10 COPY_FROM COPY_TO_USER@COPY_TO_HOST:COPY_TO_LOCATION;
Result=$?;
sleep 1;
done
- The -P switch is highly suggested as it activates:
- --partial: This keeps a file even if it doesn’t finish transferring so it can be resumed when rsync is restarted. This is especially important if you have very large files.
- --progress: This shows you the progress of the current file copy operation.
- The -z switch turns on gzip compression during the file transfer, so may only be useful depending on the circumstances.
- The -a switch stands for “archive” and is generally a good idea to use. It includes the switches:
- -r: Recurse into folders
- -t: Preserves file modification time stamp. This is highly recommended for this, and incremental backups, as rsync, by default, skips files whose file sizes and modification times match.
- -l and -D: Preserve file type (i.e. symlinks)
- -p: Preserve file (chmod) permissions
- -g and -o: Preserve file owners.
- The --timeout is the crux of the script, in that if an I/O timeout of 10 seconds occurs, the rsync exits prematurely so it can be restarted.
For more useful switches and information, see the rsync man page.
Script with comments:
export Result=1; #This will hold the result of the rsync. Set to 1 so the first loop check will fail.
while [ $Result -ne 0 ]; do #Loop until rsync result is successful
echo "STARTING ($Result) @" `date`; #Inform the user of the time an rsync is starting and the last rsync failure code
rsync -Pza --timeout=10 COPY_FROM COPY_TO_USER@COPY_TO_HOST:COPY_TO_LOCATION; #See rest of post for switch information
Result=$?; #Store the result of the rsync
sleep 1; #This is an optional 1 second timeout between attempts
done
