Backup Service

If you use the XeriomAlertService as well as this service you'll get an email if there's anything wrong with your backups eg if your most recent backup is more than 4 hours old.

<!> This should be considered a beta service. We're pretty confident that it'll work, but it's not been tested under load yet. You should take backups yourself until we're happy this service works as we want it to.

For VMs and colocated hosts

How often are backups taken?

We'll take snapshots 6 times daily (once every four hours), and once a day we'll select the oldest of those backups to promote to a daily backup. Once every seven days we'll promote the oldest daily backup to a weekly backup. Once every calendar month (probably on the first of the month) we'll promote the oldest weekly backup to a monthly backup. We'll retain monthly backups for between three and six months.

After a brief trial it appears that the following package to backup sizes give plenty of space to spare. We'll review them from time-to-time.

Package

Backup Allocation

256Mb RAM / 10Gb disk

2Gb

512Mb RAM / 20Gb disk

4Gb

1024Mb RAM / 40Gb disk

8Gb

These are soft limits and we'll get in touch with you if you exceed them to discuss what we can do to better accommodate your backup requirements.

The way we take backups means that you get to use virtually all your backup allocation, with just a little bit reserved for snapshots going back three months.

Setup

Automatic

Make sure you have the XeriomUbuntuPackagesService setup in /etc/apt/sources.list 

# ...
deb http://packages.xeriom.net/xeriom hardy/
# ...

Install the client package.

sudo apt-get install xeriom-backup-client --yes --force-yes

You still have to finish the setup. Skip to "Finishing up," below.

Manual

Add these lines to /root/.ssh/authorized_keys (and make sure it's chmod go-wrx). 

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,from="193.219.108.249",command="cat /etc/backup.conf" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAtAnZC+ncH1ge+RmGzyLu3syuOP+pI2Twb2EXYx3BU73H68ThJqOk13jMalyYa/octxiXOeT7NlBIfkRQZOeS7m5ZRp63S7AjUV4f6ByalGzpvykSXYWKhhcq8MTrSzP+hzsEoOZvkVEqwH9BbX4cJWlFMI0zTxAcWp5cHaBGUADJdx8Sm9Witw2sxYIo2ceVfarHp5lt4h3mvcAW/oAYPmh2iSBBldXGMxuDn1dQ2Jzhfb+4AcGO9Zv/J/Mng+mNSvwsA2XxTKrLoqztUr/GBI6zdnfAzWUaUryhprM3tIwgzvAm/dtF60wlvdzBmN30Gf3/EYwQFXFK0iP2VJd0dkIdDT209SIFSaWeBfpv/the5eNjmX6VxOgza1niMP5lsy27PB4HRRNiR0HKovim3nmq0uy58BWt7V0oyZBWraHmnW/fnFNSEtKDrjJOxcYzgZ2dn4efyl2FUweQL4MiYUYH74DByIo8LcAFUalFuNkMTlr+J/d22/LXDOO4GJVts2yVJZtftVJCgaJx8CNPKYy6PrWZXavaLbUPE6dGZVIUzpk5Nd9g2One5G33ScS6i5RbYBDizyAiNbugEuaF7tK+uUlWBgoQCs+pOU5klzbNJzC+P+MQ6Lxo29CdIe+KzOvUnyk4g/Zsaho69Svj7HHFZ8S25gz07c4rt0uYgaU= Backup Configuration

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,from="193.219.108.249",command="/usr/local/bin/backup_only" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEAuL0/akkteRgeeq5mLclu/RKvlbLiYpqBPEeLgVrFEWwwCH/cGy2eYhr0XV6bkbO7TG7HYN1tABY5YinA0ylwdOTSvuQxLMoNxpDIYVbt3G5csIZVWI2l20gXpYycCDhXe+30neA3uSFZIgOy7Xkb62WX2oWzsxCe0bHtgxjPBaNFxXk/CrOBiN4WFiQD/O4GeCoihT5LFG63sKqyKevxh92AG7yelem19iTg26oChWYo+/nEyrLE2s3xRGFs4lwWdzgHj5NEH5Fr+qEgcyOS7zOVsh7w0CnuAhPwbIt4kLJ1B6ivAX3irqAP+4KI2apiO1+yTjLe0PEClyTp5B95tX4foHcdARda0oRJO9GsxOGPyzBe5Yn97WUq1LYe3c5sZxSwmmhpCEHwMrXMlnMoMpBkdd1yAcq8qUdq/2IYEl2PEo+Ouq2ffrg7Vxu73fqgBGc2SDMxrz8s0CEzsKEpvepGe2gYQ6CQejt98/0sBE2dTBgUZSeK/TSOSaPq4mbfKJFOU4LHrHwoYPekpUnZZGS0FwugafPfoT5GOEGW6VNyvq1g6lfj1+UujXToAr+xdrIPp8M6FbHGnkAPTz8wO4ia4dFRDt1k4mtqJgLp0C45xK99PJC7lQ2mExgP6BgxeAqrSxbSEdNZqMGFUSalWa7mSK5cZpJAWBg1R+3w8Rk= Backup Agent

Run an sshd on port 22344. You can run one on multiple ports just by adding another Port directive to /etc/ssh/sshd_config like this: 

# ...
Port    22
Port  2222
Port 22344
# ...

Create a file called /usr/local/bin/backup_only that looks like this, and make it chmod +x. 

#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
        *\&*)
                echo "Rejected"
                ;;
        *\(*)
                echo "Rejected"
                ;;
        *\{*)
                echo "Rejected"
                ;;
        *\;*)
                echo "Rejected"
                ;;
        *\<*)
                echo "Rejected"
                ;;
        *\`*)
                echo "Rejected"
                ;;
        rsync\ --server\ --sender\ -logDtprR\ --numeric-ids\ .\ /*)
                $SSH_ORIGINAL_COMMAND
                ;;
        *)
                echo "Rejected"
                ;;
esac

Install rsync. On Ubuntu you'd do this: 

sudo apt-get install rsync --yes

Create a file called /etc/backup.conf that looks something like this (with the directories you want to backup detailed). 

/etc
/home
/var/log

Finishing up

(Stuff you have to do by yourself, because it could be destructive if I do it automatically.)

Enable root logins too. Not fun, and I'll try to find a way around that. 

# In /etc/ssh/sshd_config

PermitRootLogin without-password

# If you never log in as root, consider using this instead:
#
#   PermitRootLogin forced-commands-only
#
# If you need to login as root using a password then use this instead:
#
#   PermitRootLogin yes

Restart sshd if you've made any changes. 

sudo /etc/init.d/ssh restart

Open the firewall to backup.xeriom.net on port 22344. If you're using the monitoring service, also open port 22344 to monitor.xeriom.net. 

sudo iptables -I INPUT 4 -p tcp --dport 22344 -s backup.xeriom.net -j ACCEPT
sudo iptables -I INPUT 4 -p tcp --dport 22344 -s monitor.xeriom.net -j ACCEPT
sudo sh -c "iptables-save -c > /etc/iptables.rules"

Edit /etc/backup.conf. One line per directory you want backed up.

Finally, tell us you'd like your server backed up and we'll turn on the service.

{i} It'll take between one and five hours to start your backups from when we add you to the backup list, so leave it at least a day before accidentally deleting your filesystem ;)

Restoring from backups

Possibly backups points are:

Backup snapshots of your VM are available from any VM or host that's part of your account, and can be mounted over NFS from backup.xeriom.net:/var/cache/rsnapshot/[backup_point]/[canonical_hostname] where [backup_point] is chosen from above and [canonical_hostname] is the result of running dig -x [ip address] with the [ip address] of your VM.

You'll need nfs-common on Ubuntu to mount these. I'm not sure what's required on other flavours of Linux.

A handy backup point is between one and two days ago (ie yesterday). Here's a typical fstab which adds that entry. 

# cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
/dev/xvda1 none swap sw 0 0
/dev/xvda2 / ext3 noatime,nodiratime,errors=remount-ro 0 1
backup.xeriom.net:/var/cache/rsnapshot/daily.0/xeriom-10.vm.xeriom.net /.yesterday nfs rsize=8192,wsize=8192,timeo=14,intr,ro 0 0

Here's how to create the mountpoint and mount the snapshot. 

sudo mkdir /.yesterday
mount /.yesterday

Note that backup snapshots are read only: trying to write to them will fail.

Please don't keep the connection open longer than required. In particular please note that the backups do rotate and your mount point will become stale and start to time out when this happens.

Setup (on the backup host)

RSnapshot

# cat /usr/local/sbin/print_rsnapshot_configuration 
#!/bin/bash

ssh xit.xeriom.net -p 2202 -l xit -i ~/.ssh/backup_configuration "/usr/bin/env RAILS_ENV=production ruby /var/www/xit.xeriom.net/current/script/rsnapshot/print_configuration"


# cat /usr/local/sbin/refresh_rsnapshot_configuration
#!/bin/bash
set -o verbose 

/usr/local/sbin/print_rsnapshot_configuration > ~/xeriom_rsnapshot.tmp
for CONFIG in `grep '# SOURCE' ~/xeriom_rsnapshot.tmp | cut -f 2`; do
  HOST=`basename $CONFIG .conf`
echo $HOST
echo "ssh -o "ConnectTimeout 5" -i ~/.ssh/backup_configuration -p 22344 -l root $HOST cat /etc/backup.conf 2>/dev/null"
  BACKUP_LOCATIONS=`( 
ssh -i ~/.ssh/backup_configuration -p 22344 -l root $HOST cat /etc/backup.conf 2>/dev/null 
) || echo ""`
  echo "" >> ~/xeriom_rsnapshot.tmp
  echo "# BEGIN ${HOST}" >> ~/xeriom_rsnapshot.tmp
  for LOCATION in $BACKUP_LOCATIONS; do
    echo "backup        root@${HOST}:${LOCATION}        $HOST"
  done >> ~/xeriom_rsnapshot.tmp
  echo "# END ${HOST}" >> ~/xeriom_rsnapshot.tmp
done
grep -v '# SOURCE' ~/xeriom_rsnapshot.tmp > /etc/rsnapshot.conf

NFS

sudo apt-get install nfs-kernel-server --yes


# cat /usr/local/sbin/print_nfs_export_configuration
#!/usr/bin/env bash

ssh xit.xeriom.net -p 2202 -l xit -i ~/.ssh/backup_configuration "/usr/bin/env RAILS_ENV=production ruby /var/www/xit.xeriom.net/current/script/nfs/print_configuration"


# cat /usr/local/sbin/refresh_nfs_export_configuration
#!/usr/bin/env bash

/usr/local/sbin/print_nfs_export_configuration > ~/xeriom_nfs.tmp

NEW_MD5=`md5sum ~/xeriom_nfs.tmp |cut -f 1 -d ' '`
OLD_MD5=`md5sum /etc/exports |cut -f 1 -d ' '`

if [ "x${OLD_MD5}" != "x${NEW_MD5}" ]; then
  # There's been an update so copy the new config to the right place.
  mv ~/xeriom_nfs.tmp /etc/exports
  # Make NFS notice the changes.
  exportfs -a
fi

Firewall

root@xeriom-10:~# sudo iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:2202 
ACCEPT     tcp  --  193.219.108.245      0.0.0.0/0           tcp dpt:5666 
ACCEPT     tcp  --  193.219.108.0/24     0.0.0.0/0           tcp dpt:111 
ACCEPT     udp  --  193.219.108.0/24     0.0.0.0/0           udp dpt:111 
ACCEPT     tcp  --  193.219.108.0/24     0.0.0.0/0           tcp dpt:2049 
ACCEPT     udp  --  193.219.108.0/24     0.0.0.0/0           udp dpt:2049 
ACCEPT     tcp  --  193.219.108.0/24     0.0.0.0/0           tcp dpt:4000 
ACCEPT     udp  --  193.219.108.0/24     0.0.0.0/0           udp dpt:4000 
ACCEPT     tcp  --  193.219.108.0/24     0.0.0.0/0           tcp dpt:4001 
ACCEPT     udp  --  193.219.108.0/24     0.0.0.0/0           udp dpt:4001 
ACCEPT     tcp  --  193.219.108.0/24     0.0.0.0/0           tcp dpt:4002 
ACCEPT     udp  --  193.219.108.0/24     0.0.0.0/0           udp dpt:4002 
ACCEPT     icmp --  193.219.108.0/24     0.0.0.0/0           icmp type 3 
ACCEPT     icmp --  193.219.108.245      0.0.0.0/0           icmp type 8 
DROP       all  --  0.0.0.0/0            0.0.0.0/0


craig@xeriom-10:~$ cat /etc/default/nfs-common 
# If you do not set values for the NEED_ options, they will be attempted
# autodetected; this should be sufficient for most people. Valid alternatives
# for the NEED_ options are "yes" and "no".

# Do you want to start the statd daemon? It is not needed for NFSv4.
NEED_STATD=yes

# Options for rpc.statd.
#   Should rpc.statd listen on a specific port? This is especially useful
#   when you have a port-based firewall. To use a fixed port, set this
#   this variable to a statd argument like: "--port 4000 --outgoing-port 4001".
#   For more information, see rpc.statd(8) or http://wiki.debian.org/?SecuringNFS
STATDOPTS="--port 4000 --outgoing-port 4001"

# Do you want to start the idmapd daemon? It is only needed for NFSv4.
NEED_IDMAPD=no

# Do you want to start the gssd daemon? It is required for Kerberos mounts.
NEED_GSSD=no


craig@xeriom-10:~$ cat /etc/default/nfs-kernel-server 
# Number of servers to start up
RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0

# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information, 
# see rpc.mountd(8) or http://wiki.debian.org/?SecuringNFS
RPCMOUNTDOPTS="--port 4002"

# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".
NEED_SVCGSSD=no

# Options for rpc.svcgssd.
RPCSVCGSSDOPTS=


craig@xeriom-10:~$ cat /etc/modprobe.d/options 
# Enable double-buffering so gstreamer et. al. work
options quickcam compatible=2

# Default hostap to managed mode
options hostap_pci iw_mode=2
options hostap_cs iw_mode=2

# Tell the NFS lockd to use port 4001.
options lockd nlm_udpport=4001 nlm_tcpport=4001

Automating configuration

# cat /etc/cron.d/backups
15 *    * * *           root    /usr/local/sbin/refresh_nfs_export_configuration
45 *    * * *           root    /usr/local/sbin/refresh_rsnapshot_configuration
0 */4   * * *           root    /usr/bin/rsnapshot hourly
30 3    * * *           root    /usr/bin/rsnapshot daily
0  3    * * 1           root    /usr/bin/rsnapshot weekly
30 2    1 * *           root    /usr/bin/rsnapshot monthly

XeriomWiki: XeriomBackupService (last edited 2010-01-19 10:22:15 by CraigWebster)