Backup

From Torben's Wiki

As harddisks are known to break from time to time and re-setting up an old measurement pc is quite some pain in the ass, doing a backup onto an external USB harddisk (40-100€) might be worth thinking about...

Basically we have to distinguish between two types of backups:

  1. Frequently changed data (your personal data)
  2. Full image of the PC (important for measurement PCs), that includes all needed drivers, libraries, etc and allows restoring of the PC in no time

The first one should be done very frequently (daily/weekly?), the second one once a year or whenever some greater changes to the PC are done.

Concerning the USB harddisk I prefere small 2.5" ones that work without a separate power supply.

Here I wrote a little guide that might help you.
Happy backing up Torben (18.02.2010)

Backup of a single folder

Windows

Directory Compare

I like the free tool Directory Compare and will now explain you how I use it to backup a single folder like My Documents (De: Eigene Dateien)

  • First get the tool. I prefer the .zip file version, since I see no need in installing it.
  • Unpack the zip-file or install the tool into some folder (remember this folder!)
  • Run the tool by starting "DirCmp.exe"
  • Select the folder you want to backup as "Source" and the one you want to copy it to as "Target"
    (e.g. your external harddisk/networkdrive etc, but keep in mind that the drive has to be connected using the same driveletter before stating the backup...)
  • Click on "Scan" as a test
  • Under "Options->Configure->Unattended mode" check "use direct API calls..." to prevent the tool to produce message windows
  • Click on File->Save to save this settings into a file (e.g. myBackup) that is placed in the program folder. e.g.:
    c:\program files\Directory Compare\myBackup
  • Leave the program
  • Use the windows explorer to navigate to the program folder
  • Right click on "DirCmp.exe" -> Create new Shortcut
  • Rename it to something nice
  • Right click on it -> properties (De: Eigenschaften)
  • Edit the link target (De: Ziel) and append "myBackup.dcp /m/q" without the "
    (/m stands for mirror=make target exactly the same as source, overwriting all changes to files done on taget)
  • Now just create a desktop shortcut to this file and you are done

Keep in mind, that all changes to files in the target folder are lost and overwritten with each backup, since /m is set, see above. If you want to keep an older version of the backup, just rename the folder on your external harddisk, so a new copy of all your data is created at the next backup.

In order to increase security, I strongly advice using two different USB harddisks alternating, e.g. one for even months/weeks, one for uneven ones. The reason is, that if your PC breaks while your USB disk is connected you might loose it... (This also keeps a little more history of changes)

Zip Folder

Using UnixUtils for Windows zip util. Should be possible using windows build-in Powershell (Compress-Archive input.txt output.zip).

@echo off

REM set outputfolder
set BACKUPFOLDER="sicher"

REM get CurrDirName
for %%I in (.) do set CurrDirName=%%~nxI

REM use date and time commands to fetch a datestr in format 180830_0647
REM for English Windows this has to be modified. Currently written for 
REM date returning '30.08.2018' and time returning ' 6:47:36,10'
set DATESTR=%date:~-2,2%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
REM replace ' ' in hours <10 with 0
set DATESTR=%DATESTR: =0%

REM mkdir
if not exist "%BACKUPFOLDER%" mkdir "%BACKUPFOLDER%"

REM zipping using UnixUtils' zip, excluding backupfolder
%UserProfile%\Documents\Progs\UnixUtils\zip.exe -9 -r "%BACKUPFOLDER%"\%CurrDirName%_%DATESTR%.zip *.* -x *%BACKUPFOLDER%\*
REM -x *sicher\* => exclude folder "sicher"

rsync for Windows

There is a port of rsync for Windows using cygwin, download here.

Example 1 - mini
@echo off
set rsync="E:\progs\rsync\rsync.exe"
set source=/cygdrive/"D/photos/2017/"
set target=/cygdrive/"E/backup/photos 2017"
rem %rsync% -rthvz --modify-window=2 --delete --delete-excluded %source% %target%
%rsync% -ahv --modify-window=2 --delete --delete-excluded %source% %target%
rem a -> rlptgoD
rem modify-window=2 -> allow for time differences of 2 sec, as for MS FAT
Example 2
@echo off
set rsync="C:\Users\menketrb\Documents\Progs\rsync\rsync.exe"
set source=/cygdrive/"C/Users/menketrb/Documents/"
set target=/cygdrive/"U/sicher/doks-KOPIE"
set targetReal="U:\sicher\doks-KOPIE"
set exclude=--exclude='My Music' --exclude='My Pictures' --exclude='My Videos'

REM attrib -r %targetReal%\*.* /s
%rsync% -rthvz --modify-window=2 --delete --delete-excluded %exclude% %source% %target%
REM attrib +r %targetReal%\*.* /s

REM write current date to file inside source folder
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
REM replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
set datefile=%targetReal%\0backup-date-U-%DATESTR%.txt
date /T >> %datefile%
time /T >> %datefile%
Example 3 - keep daily, weekly and monthly backups
@echo off

set rsync="C:\Users\menketrb\Documents\Progs\rsync\rsync.exe"
set source=/cygdrive/"C/Users/menketrb/Documents/"
set sourceReal="C:\Users\menketrb\Documents"
set target=/cygdrive/"C/Users/menketrb/sicher/doksKOPIE"
set targetReal="C:\Users\menketrb\sicher\doksKOPIE"
set exclude=--exclude=Progs --exclude='My Music' --exclude='My Pictures' --exclude='My Videos'

REM write current date to file inside source folder
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
:: replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
set datefile=%sourceReal%\0backup-date-Lokal-%DATESTR%.txt
date /T >> %datefile%
time /T >> %datefile%

:: get date in several formats for name of backups
set monat=%date:~-7,2%

:: Get DayOfWeek as number
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get DayOfWeek') DO (
    if "!count!" GTR "0" GOTO next
    set dow=%%D
    SET /a count+=1
)
:: /a = arithmetisch
:next

:: Get WeekInMonth as number
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get WeekInMonth') DO (
    if "!count!" GTR "0" GOTO next
    set wim=%%D
    SET /a count+=1
)
:: /a = arithmetisch
:next


REM remove write protection
attrib -r %targetReal%\*.* /s


echo =
echo ======= Tagesbackup: rsync-d%dow% ========
echo =
%rsync% -rthvz --modify-window=2 --delete --delete-excluded %exclude% %source% %target%/rsync-d%dow%

echo =
echo ======= Wochenbackup: rsync-w%wim% ========
echo =
%rsync% -rthvz --modify-window=2 --delete --delete-excluded %exclude% %source% %target%/rsync-w%wim%

echo =
echo ======= Monatsbackup: rsync-m%monat% ========
echo =
%rsync% -rthvz --modify-window=2 --delete --delete-excluded %exclude% %source% %target%/rsync-m%monat%


REM set write protection
attrib +r %targetReal%\*.* /s

del %datefile%

Linux

Use the command line tool rsync e.g.:

rsync -rvhu --delete --delete-excluded dir1 dir2

rsync is able to speak ssh:

rsync -rvhu --delete --delete-excluded dir1 user@server:dir2

An alternative is rsnapshot Gute Deutsche Anleitung (Thanks to Philipp + Malte)

An alternative with GUI, similar to Apple's TimeMachine is Back In Time

Full image of a harddisk drive

I like the free tool DriveImage XML that be placed on a boot CD in order to do a full backup of a harddisk or partition. It can be found on some bootcds you can find in the net, like UBCD4Win or Hiren's BootCD, but I prefere creating a small personal bootcd without any other tools on it. In order to do so read the next passage or skip it if you already have a suitable boot cd with DriveImage XML on it.

Build a Boot CD

For some very strange reasons I was not able to build a working bootcd when running the tools in a virtualbox' WinXP. Using a "real" Windows (Vista 64 in my case) it all worked out fine...

  • Fetch Bart's PE Builder
  • Ensure you have a WinXP Install CD ready
  • Install Bart's PE Builder
  • Get the DriveImage XML plugin (.cab file) for Bart's PE Builder
  • Insert your WinXP CD
  • Run Bart's PE Builder
  • Tell the program where the WinXP install is located (CD Drive)
  • Click on plugins and add the previously downloaded DriveImage XML plugin
  • Close the plugin window
  • Enable the creation of an iso file
  • Start building
  • Close the program and burn the newly created image onto a cd (not the file, but use the iso-image-mode of your burn program)

Instead of burning directly you might want to try you bootcd first using virtualbox...

Backing up

  • FIRST: Attach the external hard disk that you want to store the image of your pc on
  • Boot from a bootcd that has DriveImage XML installed
  • Start the tool DriveImage XML
  • Do the backup and ensure that the image is stored on your external harddisk
  • Settings: Compression = "Low" makes sense

Linux and dd, e.g. for SD cards (RasPi)

from [1]

dd if=/dev/sdX of=/mnt/backup/sdX.dd bs=1M
# (bs = blocksize)

# with on the fly gzip compression
dd if=/dev/sdX | gzip > /mnt/backup/sdX.dd.gz

# with on the fly bzip2 compression (slower, but better compression)
dd if=/dev/sdX | bzip2 > /mnt/backup/sdX.dd.bz2

# with on the fly bzip2 compression (slower, but better compression)
dd if=/dev/sdX | xz > /mnt/backup/sdX.dd.xz

Speed/size comparison for a 16GB SD card with only few GB in use (without filling empty space with zeros, see below)

 	time		bytes		MB/s	compression
dd	00:43:07	15931539456	5.9	100.0%
gzip	00:43:01	 5205027237	1.9	 32.7%
bzip2	00:51:47	 5114374745	1.6	 32.1%
xz	01:01:36	 4959001460	1.3	 31.1%
after zero-filling of empty space:
gzip	00:43:00	  711255877	2,6	  4.5%

To improve compression you can fill empty blocks with zeros first on the device, but this takes time and causes write load on the device (maybe bad for ssds)

dd if=/dev/zero of=asdf.txt ; rm asdf.txt
# (assuming your current working dir is on the partition you want to fill the free space with zeros.

To restore the backup, reverse the commands:

dd if=/mnt/backup/sdX.dd of=/dev/sdX bs=1M

# when compressed using gzip
gzip -dc /mnt/backup/sdX.dd.gz | dd of=/dev/sdX bs=1M

# when compressed using bzip2
bunzip2 /mnt/backup/sdX.dd.gz | dd of=/dev/sdX bs=1M

# when compressed using xz
xunz /mnt/backup/sdX.dd.xz | dd of=/dev/sdX bs=1M

Disaster Recovery

Undelete single files

To enable for recovery, make sure to write as little as possible onto the partition. In Linux unmounting is a good option. I suggest using the Linux tool photorec, which scans an (unmounted!) partition (ext2/3, Fat32, NTFS, ...) for deleted files and recovers them to another partition. photorec ships with testdisk, so to install in Ubuntu use

sudo apt-get install testdisk

afterwards run

sudo photorec

Fix partition table / recover deleted partition

see TestDisk

securely wipe / overwrite disk

prior to trashing or selling a harddisk you might want to wipe all data

# fill with zeros
dd if=/dev/zero of=/dev/sdX bs=4096
# fill with random
dd if=/dev/urandom of=/dev/sdX bs=4096