Easy incremental backups using ZFS

Why I need this

I want something to replace current incremental backups because walking a spinning-rust filetree for recently-modified files takes too long. I've seen /src take up to 20 minutes, and incrementals happen every 30.

Basic idea:

begin

files=mktemp() or die
snapname='inc30'
ds=''              # dataset
mp=''              # mountpoint

# Skip redundant or unmounted datasets.
zfs list -d 2 -Ho name,mountpoint -t filesystem | 
    egrep -v '/ROOT/.*-RELEASE|none$' |
while read ds mp |
    snapshot = $ds@$snapname

    if zfs list -t snapshot -H $snapshot > /dev/null 2>&1
    then
        logmsg about diff
        run zfs diff on $snapshot
        get modified, added files
        create full pathname
        append to $files

        logmsg about destroy
        run zfs destroy on $snapshot
    endif

    logmsg about create new
    run zfs create on $snapshot
done

sort $files in place, prepend .
cd /

bdir = /backup/inc/%Y/%m%d/%H%M
mkdir -p $bdir or die

logmsg cpio
cpio -pdum $bdir < $files

logmsg done
rm $files

end

First draft

Got this working -- I can get through an entire set of incremental backups in around 11 seconds!

Results in /var/log/local6log:

Apr 22 02:18:25 furbag zsnap-inc: start
Apr 22 02:18:25 furbag zsnap-inc: SKIPPING tank, top-level pool
Apr 22 02:18:25 furbag zsnap-inc: tank/archive on /archive
Apr 22 02:18:25 furbag zsnap-inc: diff tank/archive@inc30
Apr 22 02:18:25 furbag zsnap-inc: destroy tank/archive@inc30
Apr 22 02:18:26 furbag zsnap-inc: create tank/archive@inc30
[...]
Apr 22 02:18:36 furbag zsnap-inc: zroot/var/mail on /var/mail
Apr 22 02:18:36 furbag zsnap-inc: diff zroot/var/mail@inc30
Apr 22 02:18:36 furbag zsnap-inc: destroy zroot/var/mail@inc30
Apr 22 02:18:36 furbag zsnap-inc: create zroot/var/mail@inc30
Apr 22 02:18:36 furbag zsnap-inc: zroot/var/tmp on /var/tmp
Apr 22 02:18:36 furbag zsnap-inc: diff zroot/var/tmp@inc30
Apr 22 02:18:36 furbag zsnap-inc: destroy zroot/var/tmp@inc30
Apr 22 02:18:36 furbag zsnap-inc: create zroot/var/tmp@inc30
Apr 22 02:18:36 furbag zsnap-inc: backup dir: /backup/inc/2024/0422/0218
Apr 22 02:18:36 furbag zsnap-inc: removing empty directories
Apr 22 02:18:36 furbag zsnap-inc: copied 246 files 157 Kb
Apr 22 02:18:36 furbag zsnap-inc: done

Unfortunately, I can't get du or find to accurately report the backup size, and I don't know why:

root# du -sk /backup/inc/2024/0422/0218
6503    /backup/inc/2024/0422/0218

root# find /backup/inc/2024/0422/0218/ -printf '%k\n' |
        awk '{ total += $1 } END { print total }'
6581

[in output above: 157Kb, not even close]

ZFS takes 5 sec or so to sync a transaction group. After adding that, things worked.

Production results

Average time is 16-21 seconds to back up all mounted filesystems.

The old method finished in under a minute 42% of the time, and finished under 10 minutes 94% of the time. The shortest time was 47 seconds. It ran over 30 minutes 5 times, which overlaps the next incremental.

Anything ending with this is a bogus file:

./var/tmp(on_delete_queue): Cannot stat: No such file or directory

Made a few tweaks to get rid of bogus files like the ones above.

Installation

• Install conf-exclude wherever you put local configuration files. I use /backup/etc in the zsnap-inc script.

• Create a directory to hold incremental backups. I use /backup/inc -- subdirectories YYYY/MMDD will be created. Here's an example:

  +--backup
  |   +--inc
  |   |   +--2024
  |   |   |   +--0801
  |   |   |   |   +--0006             # run at 00:06 on 2024-08-01
  |   |   |   |   |   +--home
  |   |   |   |   |   |   +--vogelke
  |   |   |   |   |   |   |   +--mail
  |   |   |   |   |   |   |   |   +--HEADERS.2024w31
  |   |   |   |   |   |   |   |   +--MAILLOG
  |   |   |   |   |   |   |   |   +--msgid.cache
  |   |   |   |   |   +--var
  |   |   |   |   |   |   +--db
  |   |   |   |   |   |   |   +--entropy
  |   |   |   |   |   |   |   |   +--saved-entropy.1
  |   |   |   |   |   |   +--log
  |   |   |   |   |   |   |   +--authlog
  |   |   |   |   |   |   |   +--maillog
  |   |   |   |   |   |   |   +--syslog
  |   |   |   |   +--0036
  |   |   |   |   |   +--home
  |   |   |   |   |   |   +--vogelke
  
  
  [...]
  
  
  |   |   |   |   +--2336             # run at 23:36 on 2024-08-01
  |   |   |   |   |   +--var
  |   |   |   |   |   |   +--log
  |   |   |   |   |   |   |   +--cron
  |   |   |   |   |   |   |   +--local6log
  |   |   |   |   |   |   |   +--syslog
  

• Move the zsnap-inc script to wherever root's cron scripts live on your system, and use an entry like this in root's crontab file:

  #+--------------------------- Minute (0-59)
  #|   +----------------------- Hour   (0-23)
  #|   |     +----------------- Day    (1-31)
  #|   |     |   +------------- Month  (1-12)
  #|   |     |   |   +--------- Day of week (0-6, 0=Sunday)
  #|   |     |   |   |    +---- Command to be run
  #|   |     |   |   |    |
  #v   v     v   v   v    v
  #=================================================================
  # Backups.
  6,36 *     *   *   *    /path/to/zsnap-inc
  

Comments welcome.