Torben: /* rsync */

2025-01-06T18:38:53Z

rsync

New page

[[Category:Software]][[Category:Linux]][[Category:Windows]]
KEIN BACKUP - KEIN MITLEID

As harddisks are known to break from time to time and re-setting up an old measurement pc is quite some pain in the ass, doing a backup onto an external USB harddisk (40-100€) might be worth thinking about...

Basically we have to distinguish between two types of backups:
# Frequently changed data (your personal data)
# Full image of the PC (important for measurement PCs), that includes all needed drivers, libraries, etc and allows restoring of the PC in no time
The first one should be done very frequently (daily/weekly?), the second one once a year or whenever some greater changes to the PC are done.

Concerning the USB harddisk I prefere small 2.5" ones that work without a separate power supply.

Here I wrote a little guide that might help you. <br>
Happy backing up Torben (18.02.2010)

==Backup of a single folder==

===Windows===
====Directory Compare====
I like the free tool [http://tp.lc.ehu.es/jma/win32/dirco.html Directory Compare] and will now explain you how I use it to backup a single folder like My Documents (De: Eigene Dateien)

* First get the tool. I prefer the .zip file version, since I see no need in installing it.
* Unpack the zip-file or install the tool into some folder (remember this folder!)
* Run the tool by starting "DirCmp.exe"
* Select the folder you want to backup as "Source" and the one you want to copy it to as "Target" <br>(e.g. your external harddisk/networkdrive etc, but keep in mind that the drive has to be connected using the same driveletter before stating the backup...)
* Click on "Scan" as a test
* Under "Options->Configure->Unattended mode" check "use direct API calls..." to prevent the tool to produce message windows
* Click on File->Save to save this settings into a file (e.g. myBackup) that is placed in the program folder. e.g.: <br>c:\program files\Directory Compare\myBackup
* Leave the program
* Use the windows explorer to navigate to the program folder
* Right click on "DirCmp.exe" -> Create new Shortcut
* Rename it to something nice
* Right click on it -> properties (De: Eigenschaften)
* Edit the link target (De: Ziel) and append "myBackup.dcp /m/q" without the "<br>(/m stands for mirror=make target exactly the same as source, overwriting all changes to files done on taget)
* Now just create a desktop shortcut to this file and you are done
Keep in mind, that all changes to files in the target folder are lost and overwritten with each backup, since /m is set, see above.
If you want to keep an older version of the backup, just rename the folder on your external harddisk, so a new copy of all your data is created at the next backup.

In order to increase security, I strongly advice using two different USB harddisks alternating, e.g. one for even months/weeks, one for uneven ones. The reason is, that if your PC breaks while your USB disk is connected you might loose it...
(This also keeps a little more history of changes)

====Zip Folder====
Using [https://sourceforge.net/projects/unxutils/ UnixUtils] for Windows zip util. Should be possible using windows build-in Powershell (Compress-Archive input.txt output.zip).
@echo off

REM set outputfolder
set BACKUPFOLDER="sicher"

REM get CurrDirName
for %%I in (.) do set CurrDirName=%%~nxI

REM use date and time commands to fetch a datestr in format 180830_0647
REM for English Windows this has to be modified. Currently written for
REM date returning '30.08.2018' and time returning ' 6:47:36,10'
set DATESTR=%date:~-2,2%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
REM replace ' ' in hours <10 with 0
set DATESTR=%DATESTR: =0%

REM mkdir
if not exist "%BACKUPFOLDER%" mkdir "%BACKUPFOLDER%"

REM zipping using UnixUtils' zip, excluding backupfolder
%UserProfile%\Documents\Progs\UnixUtils\zip.exe -9 -r "%BACKUPFOLDER%"\%CurrDirName%_%DATESTR%.zip *.* -x *%BACKUPFOLDER%\*
REM -x *sicher\* => exclude folder "sicher"

====rsync for Windows====
There is a port of rsync for Windows using cygwin, [https://www.heise.de/download/product/rsync-1271 download here].

=====Update of 2021: better use robocopy=====
robocopy SOURCE TARGET /MIR /FFT /Z /R:10 /W:5 /MT:8 /log:robocopy.log /NP /NS /NC
/MIR = Mirror
/MT = MultiThreading (default: 8 threads)
/FFT = Assumes FAT file times (two-second precision).
/Z = Copies files in restartable mode
/R = Retries
/W = Wait time between retries
/NP = log: no progress (%)
/NS = log: no file sizes
/NC = log: no file classes
/NFL = log: no file names
/XD = exclude dirs, does not allow wildcards
/XF = exclude files, allows wildcards: *.tmp

For local Backup on USB disk you should remove the retry mechanism and use only 1 thread
robocopy SOURCE TARGET /MIR /Z /R:0 /W:0 /MT:1 /log:robocopy.log /NP /NS /NC

To move instead of backup
Robocopy ".\source" ".\target" /move /e /MT:8 /R:3 /W:5 /LOG:".\robocopy_move.log" /is /it
/e include all subdirectories
/is Includes the same files. Same files are identical in name, size, times, and all attributes.
/it Includes “tweaked” files. Tweaked files have the same name, size, and times, but different attributes.

=====Example 1 - mini=====
@echo off
set rsync="E:\progs\rsync\rsync.exe"
set source=/cygdrive/"D/photos/2017/"
set target=/cygdrive/"E/backup/photos 2017"
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%
rem a -> rlptgoD
rem --modify-window=2 -> 3602 might help sometimes
rem modify-window=2 -> allow for time differences of 2 sec, as for MS FAT

=====Example 2=====
@echo off
set rsync="C:\Users\menketrb\Documents\Progs\rsync\rsync.exe"
set source=/cygdrive/"C/Users/menketrb/Documents/"
set target=/cygdrive/"U/sicher/doks-KOPIE"
set targetReal="U:\sicher\doks-KOPIE"
set exclude=--exclude='My Music' --exclude='My Pictures' --exclude='My Videos'

REM write current date to file inside source folder
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
REM replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
set datefile=%targetReal%\0backup-date-U-%DATESTR%.txt
date /T >> %datefile%
time /T >> %datefile%

REM set target readonly off and on
attrib -r %targetReal%\*.* /s
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%
attrib +r %targetReal%\*.* /s

=====Example 3 - keep daily, weekly and monthly backups=====
@echo off

set rsync="C:\Users\menketrb\Documents\Progs\rsync\rsync.exe"
set source=/cygdrive/"C/Users/menketrb/Documents/"
set sourceReal="C:\Users\menketrb\Documents"
set target=/cygdrive/"C/Users/menketrb/sicher/doksKOPIE"
set targetReal="C:\Users\menketrb\sicher\doksKOPIE"
set exclude=--exclude=Progs --exclude='My Music' --exclude='My Pictures' --exclude='My Videos'

REM write current date to file inside source folder
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
:: replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
set datefile=%sourceReal%\0backup-date-Lokal-%DATESTR%.txt
date /T >> %datefile%
time /T >> %datefile%

:: get date in several formats for name of backups
set monat=%date:~-7,2%

:: Get DayOfWeek as number
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get DayOfWeek') DO (
if "!count!" GTR "0" GOTO next
set dow=%%D
SET /a count+=1
)
:: /a = arithmetisch
:next

:: Get WeekInMonth as number
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get WeekInMonth') DO (
if "!count!" GTR "0" GOTO next
set wim=%%D
SET /a count+=1
)
:: /a = arithmetisch
:next

REM remove write protection
attrib -r %targetReal%\*.* /s

echo =
echo ======= Tagesbackup: rsync-d%dow% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-d%dow%

echo =
echo ======= Wochenbackup: rsync-w%wim% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-w%wim%

echo =
echo ======= Monatsbackup: rsync-m%monat% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-m%monat%

REM set write protection
attrib +r %targetReal%\*.* /s

del %datefile%

===Linux===
====rsync====
Use the command line tool rsync e.g.:
rsync dir1/*.txt dir2/
# delete from target what is not in source
rsync -r --delete dir1/ dir2/

rsync -rvhu --delete --delete-excluded dir1 dir2
rsync is able to speak ssh: (z = compress during transfer)
rsync -rvhuz --no-links --delete --delete-excluded dir1 user@server:dir2

An alternative is [http://rsnapshot.org rsnapshot] [http://wiki.ubuntuusers.de/rsnapshot Gute Deutsche Anleitung] (Thanks to Philipp + Malte)

An alternative with GUI, similar to Apple's TimeMachine is [http://wiki.ubuntuusers.de/Back_In_Time Back In Time]

====rsnapshop====
sudo apt-get install rsnapshot
sudo mkdir /sicher-rsnapshot
in /etc/rsnapshot.conf (use tab to separate key-values pairs!)
# target folder
snapshot_root /sicher-rsnapshot

# retain levels = how many backups to keep per level (alpha, beta, gamma, delta)
# every day
retain alpha 7
# every week
retain beta 4
# every month
retain gamma 12
# every year
retain delta 5

# dirs to backup
# LOCALHOST
backup /home/ localhost/
backup /etc/ localhost/
test configs
rsnapshot configtest
run manually
sudo rsnapshot alpha
run via crontab
sudo crontab -e
# m h dom mon dow command
10 0 * * * /usr/bin/rsnapshot alpha
# every week on Monday
20 0 * * 1 /usr/bin/rsnapshot beta
# every month 1.X.
30 0 1 * * /usr/bin/rsnapshot gamma
# every year 1.1.
40 0 1 1 * /usr/bin/rsnapshot delta

==Full image of a harddisk drive==
I like the free tool [http://www.runtime.org/driveimage-xml.htm DriveImage XML] that be placed on a boot CD in order to do a full backup of a harddisk or partition. It can be found on some bootcds you can find in the net, like UBCD4Win or Hiren's BootCD, but I prefere creating a small personal bootcd without any other tools on it. In order to do so read the next passage or skip it if you already have a suitable boot cd with DriveImage XML on it.

===Build a Boot CD===
For some very strange reasons I was not able to build a working bootcd when running the tools in a virtualbox' WinXP. Using a "real" Windows (Vista 64 in my case) it all worked out fine...
* Fetch [http://www.nu2.nu/pebuilder/#download Bart's PE Builder]
* Ensure you have a WinXP Install CD ready
* Install Bart's PE Builder
* Get the [http://www.runtime.org/driveimage_xml.cab DriveImage XML plugin (.cab file)] for Bart's PE Builder
* Insert your WinXP CD
* Run Bart's PE Builder
* Tell the program where the WinXP install is located (CD Drive)
* Click on plugins and add the previously downloaded DriveImage XML plugin
* Close the plugin window
* Enable the creation of an iso file
* Start building
* Close the program and burn the newly created image onto a cd (not the file, but use the iso-image-mode of your burn program)
Instead of burning directly you might want to try you bootcd first using virtualbox...

===Backing up===
* FIRST: Attach the external hard disk that you want to store the image of your pc on
* Boot from a bootcd that has DriveImage XML installed
* Start the tool DriveImage XML
* Do the backup and ensure that the image is stored on your external harddisk
* Settings: Compression = "Low" makes sense

===Linux and dd, e.g. for SD cards (RasPi)===
from [https://raspberrypi.stackexchange.com/questions/311/how-do-i-backup-my-raspberry-pi]
dd if=/dev/sdX of=/mnt/backup/sdX.dd bs=1M
# (bs = blocksize)

# with on the fly gzip compression
dd if=/dev/sdX | gzip > /mnt/backup/sdX.dd.gz

# with on the fly bzip2 compression (slower, but better compression)
dd if=/dev/sdX | bzip2 > /mnt/backup/sdX.dd.bz2

# with on the fly xz compression (slower, but better compression)
dd if=/dev/sdX | xz > /mnt/backup/sdX.dd.xz

Speed/size comparison for a 16GB SD card with only few GB in use (without filling empty space with zeros, see below)
time bytes MB/s compression
dd 00:43:07 15931539456 5.9 100.0%
gzip 00:43:01 5205027237 1.9 32.7%
bzip2 00:51:47 5114374745 1.6 32.1%
xz 01:01:36 4959001460 1.3 31.1%
after zero-filling of empty space:
gzip 00:43:00 711255877 2,6 4.5%

To improve compression you can fill empty blocks with zeros first on the device, but this takes time and causes write load on the device (maybe bad for ssds)
dd if=/dev/zero of=asdf.txt ; rm asdf.txt
# (assuming your current working dir is on the partition you want to fill the free space with zeros.

To restore the backup, reverse the commands:
dd if=/mnt/backup/sdX.dd of=/dev/sdX bs=1M

# when compressed using gzip
gzip -dc /mnt/backup/sdX.dd.gz | dd of=/dev/sdX bs=1M

# when compressed using bzip2
bunzip2 /mnt/backup/sdX.dd.gz | dd of=/dev/sdX bs=1M

# when compressed using xz
xunz /mnt/backup/sdX.dd.xz | dd of=/dev/sdX bs=1M

==Disaster Recovery==
===Undelete single files===
To enable for recovery, make sure to write as little as possible onto the partition. In Linux unmounting is a good option.
I suggest using the Linux tool photorec, which scans an (unmounted!) partition (ext2/3, Fat32, NTFS, ...) for deleted files and recovers them to another partition.
photorec ships with testdisk, so to install in Ubuntu use
sudo apt-get install testdisk
afterwards run
sudo photorec

===Fix partition table / recover deleted partition===
see [https://www.linuxquestions.org/questions/linux-general-1/how-to-recover-deleted-files-from-a-fat32-volume-in-linux-739511/ TestDisk ]

===securely wipe / overwrite disk===
====Linux/MacOS====
prior to trashing or selling a harddisk you might want to wipe all data
# fill with a zero-filled file
cat /dev/zero > asdf.txt
# or fill whole drive with zeros
dd if=/dev/zero of=/dev/sdX
# or more secure fill with random
dd if=/dev/urandom of=asdf.txt asdf.txt
# or
cat /dev/urandom > asdf.txt
====Windows====
https://www.feyrer.de/g4u/nullfile-1.01_64bit.exe

Backup - Revision history

Torben: /* rsync */