Backup
KEIN BACKUP - KEIN MITLEID
As harddisks are known to break from time to time and re-setting up an old measurement pc is quite some pain in the ass, doing a backup onto an external USB harddisk (40-100€) might be worth thinking about...
Basically we have to distinguish between two types of backups:
- Frequently changed data (your personal data)
- Full image of the PC (important for measurement PCs), that includes all needed drivers, libraries, etc and allows restoring of the PC in no time
The first one should be done very frequently (daily/weekly?), the second one once a year or whenever some greater changes to the PC are done.
Concerning the USB harddisk I prefere small 2.5" ones that work without a separate power supply.
Here I wrote a little guide that might help you.
Happy backing up Torben (18.02.2010)
Backup of a single folder
Windows
Directory Compare
I like the free tool Directory Compare and will now explain you how I use it to backup a single folder like My Documents (De: Eigene Dateien)
- First get the tool. I prefer the .zip file version, since I see no need in installing it.
- Unpack the zip-file or install the tool into some folder (remember this folder!)
- Run the tool by starting "DirCmp.exe"
- Select the folder you want to backup as "Source" and the one you want to copy it to as "Target"
(e.g. your external harddisk/networkdrive etc, but keep in mind that the drive has to be connected using the same driveletter before stating the backup...) - Click on "Scan" as a test
- Under "Options->Configure->Unattended mode" check "use direct API calls..." to prevent the tool to produce message windows
- Click on File->Save to save this settings into a file (e.g. myBackup) that is placed in the program folder. e.g.:
c:\program files\Directory Compare\myBackup - Leave the program
- Use the windows explorer to navigate to the program folder
- Right click on "DirCmp.exe" -> Create new Shortcut
- Rename it to something nice
- Right click on it -> properties (De: Eigenschaften)
- Edit the link target (De: Ziel) and append "myBackup.dcp /m/q" without the "
(/m stands for mirror=make target exactly the same as source, overwriting all changes to files done on taget) - Now just create a desktop shortcut to this file and you are done
Keep in mind, that all changes to files in the target folder are lost and overwritten with each backup, since /m is set, see above. If you want to keep an older version of the backup, just rename the folder on your external harddisk, so a new copy of all your data is created at the next backup.
In order to increase security, I strongly advice using two different USB harddisks alternating, e.g. one for even months/weeks, one for uneven ones. The reason is, that if your PC breaks while your USB disk is connected you might loose it... (This also keeps a little more history of changes)
Zip Folder
Using UnixUtils for Windows zip util. Should be possible using windows build-in Powershell (Compress-Archive input.txt output.zip).
@echo off REM set outputfolder set BACKUPFOLDER="sicher" REM get CurrDirName for %%I in (.) do set CurrDirName=%%~nxI REM use date and time commands to fetch a datestr in format 180830_0647 REM for English Windows this has to be modified. Currently written for REM date returning '30.08.2018' and time returning ' 6:47:36,10' set DATESTR=%date:~-2,2%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2% REM replace ' ' in hours <10 with 0 set DATESTR=%DATESTR: =0% REM mkdir if not exist "%BACKUPFOLDER%" mkdir "%BACKUPFOLDER%" REM zipping using UnixUtils' zip, excluding backupfolder %UserProfile%\Documents\Progs\UnixUtils\zip.exe -9 -r "%BACKUPFOLDER%"\%CurrDirName%_%DATESTR%.zip *.* -x *%BACKUPFOLDER%\* REM -x *sicher\* => exclude folder "sicher"
rsync for Windows
There is a port of rsync for Windows using cygwin, download here.
Update of 2021: better use robocopy
robocopy SOURCE TARGET /MIR /FFT /Z /R:10 /W:5 /MT:8 /log:robocopy.log /NP /NS /NC /MIR = Mirror /MT = MultiThreading (default: 8 threads) /FFT = Assumes FAT file times (two-second precision). /Z = Copies files in restartable mode /R = Retries /W = Wait time between retries /NP = log: no progress (%) /NS = log: no file sizes /NC = log: no file classes /NFL = log: no file names /XD = exclude dirs, does not allow wildcards /XF = exclude files, allows wildcards: *.tmp
For local Backup on USB disk you should remove the retry mechanism and use only 1 thread
robocopy SOURCE TARGET /MIR /Z /R:0 /W:0 /MT:1 /log:robocopy.log /NP /NS /NC
Example 1 - mini
@echo off set rsync="E:\progs\rsync\rsync.exe" set source=/cygdrive/"D/photos/2017/" set target=/cygdrive/"E/backup/photos 2017" %rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target% rem a -> rlptgoD rem --modify-window=2 -> 3602 might help sometimes rem modify-window=2 -> allow for time differences of 2 sec, as for MS FAT
Example 2
@echo off set rsync="C:\Users\menketrb\Documents\Progs\rsync\rsync.exe" set source=/cygdrive/"C/Users/menketrb/Documents/" set target=/cygdrive/"U/sicher/doks-KOPIE" set targetReal="U:\sicher\doks-KOPIE" set exclude=--exclude='My Music' --exclude='My Pictures' --exclude='My Videos'
REM write current date to file inside source folder set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2% REM replace ' ' in small hours with 0 set DATESTR=%DATESTR: =0% set datefile=%targetReal%\0backup-date-U-%DATESTR%.txt date /T >> %datefile% time /T >> %datefile% REM set target readonly off and on attrib -r %targetReal%\*.* /s %rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target% attrib +r %targetReal%\*.* /s
Example 3 - keep daily, weekly and monthly backups
@echo off
set rsync="C:\Users\menketrb\Documents\Progs\rsync\rsync.exe"
set source=/cygdrive/"C/Users/menketrb/Documents/"
set sourceReal="C:\Users\menketrb\Documents"
set target=/cygdrive/"C/Users/menketrb/sicher/doksKOPIE"
set targetReal="C:\Users\menketrb\sicher\doksKOPIE"
set exclude=--exclude=Progs --exclude='My Music' --exclude='My Pictures' --exclude='My Videos'
REM write current date to file inside source folder
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
:: replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
set datefile=%sourceReal%\0backup-date-Lokal-%DATESTR%.txt
date /T >> %datefile%
time /T >> %datefile%
:: get date in several formats for name of backups
set monat=%date:~-7,2%
:: Get DayOfWeek as number
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get DayOfWeek') DO (
if "!count!" GTR "0" GOTO next
set dow=%%D
SET /a count+=1
)
:: /a = arithmetisch
:next
:: Get WeekInMonth as number
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get WeekInMonth') DO (
if "!count!" GTR "0" GOTO next
set wim=%%D
SET /a count+=1
)
:: /a = arithmetisch
:next
REM remove write protection
attrib -r %targetReal%\*.* /s
echo =
echo ======= Tagesbackup: rsync-d%dow% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-d%dow%
echo =
echo ======= Wochenbackup: rsync-w%wim% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-w%wim%
echo =
echo ======= Monatsbackup: rsync-m%monat% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-m%monat%
REM set write protection
attrib +r %targetReal%\*.* /s
del %datefile%
Linux
rsync
Use the command line tool rsync e.g.:
rsync -rvhu --delete --delete-excluded dir1 dir2
rsync is able to speak ssh: (z = compress during transfer)
rsync -rvhuz --no-links --delete --delete-excluded dir1 user@server:dir2
An alternative is rsnapshot Gute Deutsche Anleitung (Thanks to Philipp + Malte)
An alternative with GUI, similar to Apple's TimeMachine is Back In Time
rsnapshop
sudo apt-get install rsnapshot sudo mkdir /sicher-rsnapshot
in /etc/rsnapshot.conf (use tab to separate key-values pairs!)
# target folder snapshot_root /sicher-rsnapshot # retain levels = how many backups to keep per level (alpha, beta, gamma, delta) retain alpha 6 retain beta 7 retain gamma 4 retain delta 12 # dirs to backup # LOCALHOST backup /home/ localhost/ backup /etc/ localhost/
test configs
rsnapshot configtest
run manually
sudo rsnapshot alpha
run via cronjob
sudo crontab -e # m h dom mon dow command 11 */4 * * * /usr/bin/rsnapshot alpha 21 12 * * * /usr/bin/rsnapshot beta 31 12 * * 1 /usr/bin/rsnapshot gamma 41 12 1 * * /usr/bin/rsnapshot delta
Full image of a harddisk drive
I like the free tool DriveImage XML that be placed on a boot CD in order to do a full backup of a harddisk or partition. It can be found on some bootcds you can find in the net, like UBCD4Win or Hiren's BootCD, but I prefere creating a small personal bootcd without any other tools on it. In order to do so read the next passage or skip it if you already have a suitable boot cd with DriveImage XML on it.
Build a Boot CD
For some very strange reasons I was not able to build a working bootcd when running the tools in a virtualbox' WinXP. Using a "real" Windows (Vista 64 in my case) it all worked out fine...
- Fetch Bart's PE Builder
- Ensure you have a WinXP Install CD ready
- Install Bart's PE Builder
- Get the DriveImage XML plugin (.cab file) for Bart's PE Builder
- Insert your WinXP CD
- Run Bart's PE Builder
- Tell the program where the WinXP install is located (CD Drive)
- Click on plugins and add the previously downloaded DriveImage XML plugin
- Close the plugin window
- Enable the creation of an iso file
- Start building
- Close the program and burn the newly created image onto a cd (not the file, but use the iso-image-mode of your burn program)
Instead of burning directly you might want to try you bootcd first using virtualbox...
Backing up
- FIRST: Attach the external hard disk that you want to store the image of your pc on
- Boot from a bootcd that has DriveImage XML installed
- Start the tool DriveImage XML
- Do the backup and ensure that the image is stored on your external harddisk
- Settings: Compression = "Low" makes sense
Linux and dd, e.g. for SD cards (RasPi)
from [1]
dd if=/dev/sdX of=/mnt/backup/sdX.dd bs=1M # (bs = blocksize) # with on the fly gzip compression dd if=/dev/sdX | gzip > /mnt/backup/sdX.dd.gz # with on the fly bzip2 compression (slower, but better compression) dd if=/dev/sdX | bzip2 > /mnt/backup/sdX.dd.bz2 # with on the fly xz compression (slower, but better compression) dd if=/dev/sdX | xz > /mnt/backup/sdX.dd.xz
Speed/size comparison for a 16GB SD card with only few GB in use (without filling empty space with zeros, see below)
time bytes MB/s compression dd 00:43:07 15931539456 5.9 100.0% gzip 00:43:01 5205027237 1.9 32.7% bzip2 00:51:47 5114374745 1.6 32.1% xz 01:01:36 4959001460 1.3 31.1% after zero-filling of empty space: gzip 00:43:00 711255877 2,6 4.5%
To improve compression you can fill empty blocks with zeros first on the device, but this takes time and causes write load on the device (maybe bad for ssds)
dd if=/dev/zero of=asdf.txt ; rm asdf.txt # (assuming your current working dir is on the partition you want to fill the free space with zeros.
To restore the backup, reverse the commands:
dd if=/mnt/backup/sdX.dd of=/dev/sdX bs=1M # when compressed using gzip gzip -dc /mnt/backup/sdX.dd.gz | dd of=/dev/sdX bs=1M # when compressed using bzip2 bunzip2 /mnt/backup/sdX.dd.gz | dd of=/dev/sdX bs=1M # when compressed using xz xunz /mnt/backup/sdX.dd.xz | dd of=/dev/sdX bs=1M
Disaster Recovery
Undelete single files
To enable for recovery, make sure to write as little as possible onto the partition. In Linux unmounting is a good option. I suggest using the Linux tool photorec, which scans an (unmounted!) partition (ext2/3, Fat32, NTFS, ...) for deleted files and recovers them to another partition. photorec ships with testdisk, so to install in Ubuntu use
sudo apt-get install testdisk
afterwards run
sudo photorec
Fix partition table / recover deleted partition
see TestDisk
securely wipe / overwrite disk
Linux/MacOS
prior to trashing or selling a harddisk you might want to wipe all data
# fill with a zero-filled file cat /dev/zero > asdf.txt # or fill whole drive with zeros dd if=/dev/zero of=/dev/sdX # or more secure fill with random dd if=/dev/urandom of=asdf.txt asdf.txt # or cat /dev/urandom > asdf.txt