Home Page
  • March 29, 2024, 10:51:35 am *
  • Welcome, Guest
Please login or register.

Login with username, password and session length
Advanced search  

News:

Official site launch very soon, hurrah!


Author Topic: Automatically resuming rsync  (Read 8921 times)

Dakusan

  • Programmer Person
  • Administrator
  • Hero Member
  • *****
  • Posts: 535
    • View Profile
    • Dakusan's Domain
Automatically resuming rsync
« on: September 10, 2012, 03:44:11 pm »

Original post for Automatically resuming rsync can be found at https://www.castledragmire.com/Posts/Automatically_resuming_rsync.
Originally posted on: 09/10/12

Rsync is a spectacular bash utility for doing file copying and/or syncing operations. It has a multitude of switches to help optimize and handle any requirements for file copy operation over a local computer or network. However, sometimes networks are less than stable and stalls can happen during an rsync (or scp). This is quite the nuisance when doing very large (i.e. GB) transfers. To solve this, the following script can be used to auto resume a stalled rsync.



export Result=1;
while [ $Result -ne 0 ]; do
 echo "STARTING ($Result) @" `date`;
 rsync -Pza --timeout=10 COPY_FROM COPY_TO_USER@COPY_TO_HOST:COPY_TO_LOCATION;
 Result=$?;
 sleep 1;
done

  • The -P switch is highly suggested as it activates:
    • --partial: This keeps a file even if it doesn’t finish transferring so it can be resumed when rsync is restarted. This is especially important if you have very large files.
    • --progress: This shows you the progress of the current file copy operation.
  • The -z switch turns on gzip compression during the file transfer, so may only be useful depending on the circumstances.
  • The -a switch stands for “archive” and is generally a good idea to use. It includes the switches:
    • -r: Recurse into folders
    • -t: Preserves file modification time stamp. This is highly recommended for this, and incremental backups, as rsync, by default, skips files whose file sizes and modification times match.
    • -l and -D: Preserve file type (i.e. symlinks)
    • -p: Preserve file (chmod) permissions
    • -g and -o: Preserve file owners.
  • The --timeout is the crux of the script, in that if an I/O timeout of 10 seconds occurs, the rsync exits prematurely so it can be restarted.

For more useful switches and information, see the rsync man page.



Script with comments:

export Result=1; #This will hold the result of the rsync. Set to 1 so the first loop check will fail.
while [ $Result -ne 0 ]; do #Loop until rsync result is successful
 echo "STARTING ($Result) @" `date`; #Inform the user of the time an rsync is starting and the last rsync failure code
 rsync -Pza --timeout=10 COPY_FROM COPY_TO_USER@COPY_TO_HOST:COPY_TO_LOCATION; #See rest of post for switch information
 Result=$?; #Store the result of the rsync
 sleep 1; #This is an optional 1 second timeout between attempts
done
Logged