linux-timemachine

Linux Time Machine

Install | Uninstall | TL;DR | Features | How does it work | Restore | Retention | Usage | FAQ | Disclaimer | License

timemachine is a tiny and stable KISS driven and POSIX compliant script that mimics the behavior of OSX’s timemachine. It uses rsync to incrementally back up your data to a different directory, hard disk or remote server via SSH. All operations are incremental, atomic and automatically resumable.

By default it uses the rsync options: --recursive, --perms, --owner, --group, --times and --links. In case your target filesystem does not support any of those options or you cannot use them due to missing permission, you can explicitly disable them via --no-perms, --no-owner, --no-group, --no-times, and --copy-links. See FAQ for examples.

Motivation

The goal of this project is to have a cross-operating system and minimal as possible backup script that can be easily reviewed by anyone without great effort. Additionally it should provide one task only and do it well without the need of external requirements and only rely on default installed tools.

:tada: Install

sudo make install

:no_entry_sign: Uninstall

sudo make uninstall

:coffee: TL;DR

Using POSIX.1-2008 argument syntax:

# Recursive, incremental and atomic backup (locally)
$ timemachine /source/dir /target/dir

# Recursive, incremental and atomic backup (via ssh)
$ timemachine /source/dir user@host:target/dir

# Recursive, incremental and atomic backup (via ssh with non-standard port)
$ timemachine --port 10000 /source/dir user@host:target/dir

# Append rsync options
$ timemachine /source/dir /target/dir -- --specials --progress
$ timemachine /source/dir /target/dir -- --specials --no-perms
$ timemachine /source/dir /target/dir -- --archive --progress

# Make the timemachine script be more verbose
$ timemachine -v /source/dir /target/dir
$ timemachine --verbose /source/dir /target/dir

# Make the timemachine script be even more verbose
$ timemachine -d /source/dir /target/dir
$ timemachine --debug /source/dir /target/dir

# Make the timemachine script and rsync more verbose
$ timemachine -v /source/dir /target/dir -- --verbose
$ timemachine --verbose /source/dir /target/dir -- --verbose

:star: Features

Feature	Description
SSH or local	Local backups as well as backups via SSH are supported.
Incremental	Backups are always done incrementally using rsync’s ability to hardlink to previous backup directories. You can nevertheless always see the full backup on the file system of any incrementally made backup without having to generate it. This will also be true when deleting any of the previously created backup directories. See the Backups section for how this is achieved via rsync. Incremental Backups also mean that only the changes on your source, compared to what is already on the target, have to be backed up. This will save you time as well as disk space on the target disk.
Partial	When backing up, files are transmitted partially, so in case a 2GB movie file backup is interrupted the next run will pick up exactly where it left off at that file and will not start to copy it from scratch.
Resumable	Not only is this script keeping partial files, but also the whole backup run is also resumable. Whenever there is an unfinished backup and you start timemachine again, it will automatically resume it. It will resume any previously failed backup as long as it finally succeeds.
Atomic [1]	The whole backup procedure is atomic. Only if and when the backup procedure succeeds, it is then properly named and symlinked. Any non-successful backup directory is either waiting to be resumed or to be deleted.

• ^[1] The backup process is atomic, but not the backup itself. rsync copies files as it finds them and in the meantime there could already be changes on the source. To achieve an atomic backup, either back up from a read-only volume or from a snapshot.

:information_source: How does it work?

Directory structure

The following directory structure will be created:

$ tree -L 1 /my/backup/folder
.
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/

current will always link to the latest created backup. All backups are incremental except the first created one. You can nevertheless safely remove all previous folders and the remaining folders will still have all of their content.

Backup strategy

Except for the first one, backups are always and automatically done incrementally, so the least amount of space is consumed. Due to rsync’s ability, every directory will still contain all files, even though they are just incremental backups. This is archived via hardlinks.

$ du -hd1 .
497M    ./2018-01-06__18-43-30
24K     ./2018-01-06__18-44-23
24K     ./2018-01-06__18-50-44
24K     ./2018-01-06__18-50-52
497M    .

You can also safely delete the initial full backup directory without having to worry about losing any of your full backup data:

$ rm -rf ./2018-01-06__18-43-30
$ du -hd1 .
497M    ./2018-01-06__18-44-23
24K     ./2018-01-06__18-50-44
24K     ./2018-01-06__18-50-52
497M    .

rsync and hardlinks are magic :-)

Failure handling and resume

In case the timemachine script aborts (self-triggered, disk unavailable or for any other reason) you can simply run it again to automatically resume the last failed run.

This is due to the fact that the backup process is atomic. During a non-complete run, all data will be stored in a directory named .inprogress/. This will hold all already transferred data and will be picked up during the next run. Once the backup is complete, it will be renamed and symlinked to current.

$ tree -a -L 1 /my/backup/folder
.
├── .inprogress/
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/

:recycle: Restore

No special software is required to restore your data. Backed up files can be easily browsed and thus copied back to where you need them. Recall the backup directory structure:

$ tree -L 1 /my/backup/folder
.
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/

Chose a backup directory and simply copy them to where you need it:

# Test it out in dry run mode before applying
$ rsync --archive --progress --dry-run /my/backup/folder/2018-01-06__18-50-52/ /src/

# Apply restoration
$ rsync --archive --progress /my/backup/folder/2018-01-06__18-50-52/ /src/

:repeat: Retention

As decribed above this project is KISS driven and only tries to do one job: back up your data.

Retention is a delicate topic as you want to be sure that data is removed as intended. For this there are already well-established tools that do an excellent job and have proven themselves over time: tmpreaper and tmpwatch.

:lock: Reliability

The script is written and maintained with maximum care. In order to retain a reliable and stable backup solution, a lot of effort goes into a vast amount of integration and regression tests. These tests not only give you measurable confidence, but also help new contributors to not accidentally introduce new or old bugs.

:computer: Usage

Available options

$ timemachine -h

Usage: timemachine [-vd]   <source> <dest> -- [rsync opts]

       timemachine [-vdpi] <source> <host>:<dest>        -- [rsync opts]
       timemachine [-vdpi] <source> <user>@<host>:<dest> -- [rsync opts]
       timemachine [-vdpi] <source> <ssh-alias>:<dest>   -- [rsync opts]

       timemachine [-vdpi] <host>:<source>        <dest> -- [rsync opts]
       timemachine [-vdpi] <user>@<host>:<source> <dest> -- [rsync opts]
       timemachine [-vdpi] <ssh-alias>:<source>   <dest> -- [rsync opts]

       timemachine -V, --version
       timemachine -h, --help

This shell script mimics the behavior of OSX's timemachine.
It uses rsync to incrementally back up your data to a different directory or remote server via SSH.
All operations are incremental, atomic and automatically resumable.

By default it uses --recursive --perms --owner --group --times --links.
In case your target filesystem does not support any of those options, you can explicitly
disable those options via --no-perms --no-owner --no-group --no-times and --copy-links.

Required arguments:
  <source>              Local source directory
  <dest>                Local destination directory.
  <host>:<dest>         SSH host and source/destination directory on server
  <user>@<host>:<dest>  SSH user, SSH host and source/destination directory on server
  <ssh-alias>:<dest>    SSH alias (defined in ~/.ssh/config) and source/destination directory on server

Options:
  -p, --port            Specify alternative SSH port for remote backups if it is not 22.
  -i, --identity        Specify path to SSH key.
  -v, --verbose         Be verbose.
  -d, --debug           Be even more verbose.

Misc Options:
  -V, --version         Print version information and exit
  -h, --help            Show this help screen

Examples:
  Simply back up one directory recursively
      timemachine /home/user /data
  Do the same, but be verbose
      timemachine -v /home/user /data
  Append rsync options and be very verbose
      timemachine -d /home/user /data -- --progress --verbose
  Log to file
      timemachine -v /home/user /data > /var/log/timemachine.log 2> /var/log/timemachine.err

Documentation:
  View more examples at: https://github.com/cytopia/linux-timemachine

Use with cron

The following can be used as an example crontab entry. It assumes you have an external disk (NFS, USB, etc…) that mounts at /backup. Before adding the crontab entry, ensure the filesystem in /backup is mounted and use:

$ touch /backup/mounted

This guards against accidentally backing up to an unmounted directory

Next, add the following to crontab using crontab -e as whichever user you intend to run the backup script as. You may need to place this in the root crontab if you are backing up sensitive files that only root can read

0 2 * * * if [[ -e /backup/mounted ]]; then /usr/local/bin/timemachine /home/someuser /backup; fi

This will cause timemachine to run at 2AM once per day. Since timemachine keeps track of backups with granularity up to the hour, minute and second, you could have it run more than once per day if you want backups to run more often.

:bulb: FAQ

Should I add trailing directory slashes (/)?

Trailing directory slashes only matter for the source directory and will not make a difference if added to the destination directory.

# The following backs up the contents of the src directory
$ timemachine src/ dst/
$ tree -L 2 /dst
.
├── 2018-01-06__18-43-30/
│   └── file.txt
└── current -> 2018-01-06__18-43-30/

# The following backs up the the src directory itself
$ timemachine src dst/
$ tree -L 3 /dst
.
├── 2018-01-06__18-43-30/
│   └── src
│       └── file.txt
└── current -> 2018-01-06__18-43-30/

How to dry-run the backup?

$ timemachine src/ dst/ -- --dry-run

How to use a non-standard SSH port?

$ timemachine --port 1337 src/ user@host:dst/

How to use SSH aliases from ~/.ssh/config?

$ cat ~/.ssh/config
Host my-ssh-alias
  HostName     192.168.0.1
  Port         1234
  User         john
  IdentityFile ~/.ssh/id_rsa__alternative_key

$ timemachine src/ my-ssh-alias:dst/

How to speed up remote backups?

# With  this option, rsync compresses the file data as it is sent to the des‐
# tination machine, which reduces the amount of  data  being  transmitted
$ timemachine src/ user@host:dst/ --compress

How to preserve ACLs?

$ timemachine src/ dst/ -- --acls

How to preserve extended attributes?

$ timemachine src/ dst/ -- --xattrs

How to disable preserving file and directory permissions?

$ timemachine src/ dst/ -- --no-owner --no-perms

How to disable preserving modification time?

$ timemachine src/ dst/ -- --no-times

How to copy the content instead of a symlink?

# This is useful in case your file system does not support symlinks.
# It is recommended to read rsync man page about symlinks to be sure
# about what you are doing
$ timemachine src/ dst/ -- --copy-links --safe-links
$ timemachine src/ dst/ -- --copy-links --safe-links --keep-dirlinks

How to ensure all files in the back up have the ownership of current user?

# Regardless of who owns the files, ensure the backup has uid and gid of current user
# This will only work if you have read-permission on all files.
$ timemachine src/ dst/ -- --no-owner --no-group

# If you do not have permission to read all files, you require sudo or root permission.
# The following will instruct rsync to ensure the backed up data has the uid and gid
# of the desired user.
$ sudo timemachine src/ dst/ -- --chown=<user>:<group>

:exclamation: Disclaimer

Backups are one of the most important things. We all care about our data and want it to be safe, so do not blindly trust scripts when it comes to backups! Do review the code, it is not too complex and kept as short as possible. Have a look at how much effort goes into the integration tests to provide measurable stability.

Learn about rsync it is a very powerful tool and you might even be able to just use this for backups.

There are many other backup solutions out there that might be a better fit for your needs. Do your own research, look at GitHub issues, source code, integration tests and try them out as well.

:octocat: Contributing

See Contributing guidelines to help to improve this project.

:page_facing_up: License

MIT License

Copyright © 2017 cytopia