This is a follow up to things I've written down in an answer to another
question:
How does Windows Image Backup decide which files to backup?
Windows/NTFS manages [change journals][2] on each volume, keeping track of which file has changed, how it has changed and because those are part of all NTFS volumes by default, they are available in the VHD(X) of the backup target as well. From my understanding,
`wbengine` simply compares those journals to know which files need backup. This makes it easy to support different backup targets without caring about archive bits of files or stuff like that, because those change journals are bound to volume-unique file-IDs
and those IDs are exactly the same in the backup target:
C:\>fsutil file queryfileid "C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"
Datei-ID: 0x000000000000000000080000000ea5f1
C:\>fsutil file queryfileid "D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe"
Datei-ID: 0x000000000000000000080000000ea5f1
In the above example, `C:\` is my current system volume while `D:\` is the mounted last backup on one of the two available backup targets. Even file sizes, timestamps etc. are all the same:
C:\>dir /A "C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe" Datenträger in Laufwerk C: ist System Volumeseriennummer: 266B-2863 Verzeichnis von C:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin 03.02.2016 17:08 560.640 perlapp.exe 1 Datei(en), 560.640 Bytes 0 Verzeichnis(se), 1.337.040.216.064 Bytes frei C:\>dir /A "D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin\perlapp.exe" Datenträger in Laufwerk D: ist System Volumeseriennummer: 266B-2863 Verzeichnis von D:\Program Files\ActiveState Perl Dev Kit 9.5.1\bin 03.02.2016 17:08 560.640 perlapp.exe 1 Datei(en), 560.640 Bytes 0 Verzeichnis(se), 1.281.302.233.088 Bytes frei
By using this approach, backup can decide at any time with any older VHD(X) which files need to be backed up, as long as the current journal and the one in the image have something in common, which are entry-IDs in my understanding. Without such a shared ID,
e.g. because too many I/O happend productive and the backup is too old, wbengine would simply do a full backup instead of an incremental one.
Using those journals makes it as well pretty fast to know which files to backup, because one doesn't need to iterate the whole tree of files. That's what one actually sees in practice as well: Backup seems to know pretty soon which files to backup and starts
processing those instead of iterating a file tree.
Behaviour in case of network shares as backup target for Windows Server 2012+.
In older versions of Windows, wbengineseems to have always recreated VHD files, making it pretty incompatible
with snapshots supported by some underlying file system like BTRFS or ZFS. Additionally, each and every backup was a full one, simply because wbengine didn't have access to any former backup to compare with.
Things seem to be a little bit different with newer versions of Windows: I'm somewhat sure that one customer of mine ran into troubles with wbadmin and Windows Server 2012 and during debugging those using Process Monitor, I verified that
existing VHDX files were kept instead of deleted and recreated. I've tested this right now with Windows Server 2019 again and multiple invocations ofwbadmin led to successful backups while KEEPING inodes of VHDX files the same:
root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-02 200024" total 549063256 271 0 d---------+ 1 user group 594 Nov 2 20:58 . 266 0 d---------+ 1 user group 108 Nov 2 20:58 .. 273 507813736 ----------+ 1 user group 520061190144 Nov 2 22:02 165c4b13-8376-4c55-9643-a8c1b2c17d6b.vhdx 3814 4 ----------+ 1 user group 776 Nov 1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml 3816 440 ----------+ 1 user group 450488 Nov 1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_Components.xml 3813 8 ----------+ 1 user group 6212 Nov 1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_RegistryExcludes.xml 3815 4 ----------+ 1 user group 1192 Nov 1 22:47 BackupSpecs.xml 272 41249064 ----------+ 1 user group 42289070080 Nov 2 22:03 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-03 132201" total 549318872 271 0 d---------+ 1 user group 594 Nov 2 20:58 . 266 0 d---------+ 1 user group 108 Nov 3 14:19 .. 273 507813736 ----------+ 1 user group 520061190144 Nov 3 14:19 165c4b13-8376-4c55-9643-a8c1b2c17d6b.vhdx 3814 4 ----------+ 1 user group 776 Nov 1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml 3816 440 ----------+ 1 user group 450488 Nov 1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_Components.xml 3813 8 ----------+ 1 user group 6212 Nov 1 22:47 52e1a857-4620-43af-9f6e-a03fb53c7c30_RegistryExcludes.xml 3815 4 ----------+ 1 user group 1192 Nov 1 22:47 BackupSpecs.xml 272 41504680 ----------+ 1 user group 42289070080 Nov 3 14:21 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx root@[...]:~# ls -lisa "/volume1/[...]/Backup 2019-11-03 133308" total 41262660 271 0 d---------+ 1 user group 2504 Nov 3 14:44 . 266 0 d---------+ 1 user group 108 Nov 3 14:30 .. 3851 4 ----------+ 1 user group 776 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_AdditionalFilesc3b9f3c7-5e52-4d5e-8b20-19adc95a34c7.xml 3853 440 ----------+ 1 user group 450488 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Components.xml 3850 8 ----------+ 1 user group 6212 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_RegistryExcludes.xml 3840 4 ----------+ 1 user group 3138 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer0bada1de-01a9-4625-8278-69e735f39dd2.xml 3848 4 ----------+ 1 user group 2284 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer2707761b-2324-473d-88eb-eb007a359533.xml 3844 8 ----------+ 1 user group 5386 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f.xml 3846 4 ----------+ 1 user group 1488 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer542da469-d3e1-473c-9f4f-7847f01fc64f.xml 3839 4 ----------+ 1 user group 1628 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writer75dfb225-e2e4-4d39-9ac9-ffaff65ddf06.xml 3842 24 ----------+ 1 user group 21686 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writera65faa63-5ea8-4ebc-9dbd-a0c4db26912a.xml 3847 4 ----------+ 1 user group 1484 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writera6ad56c2-b509-4e6c-bb19-49d8f43532f0.xml 3845 4 ----------+ 1 user group 2940 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerafbab4a2-367d-4d15-a586-71dbb18f8485.xml 3849 4 ----------+ 1 user group 1850 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerb2014c9e-8711-4c5c-a5a9-3cf384484757.xml 3843 8 ----------+ 1 user group 6048 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerbe000cbe-11fe-4426-9c58-531aa6355fc4.xml 3838 4 ----------+ 1 user group 1746 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writerd61d61c8-d73a-4eee-8cdd-f6f9786b7124.xml 3841 13068 ----------+ 1 user group 13379228 Nov 3 14:44 22c1a052-3d01-438c-87f8-acf63f14f5b5_Writere8132975-6f93-4464-a53e-1050253ae220.xml 3852 4 ----------+ 1 user group 758 Nov 3 14:44 BackupSpecs.xml 272 41249064 ----------+ 1 user group 42289070080 Nov 3 14:44 f7650533-11ca-4db4-80f5-2db42f7b900a.vhdx
So in theory this should allow incremental backups replacing files in-place and efficient snapshots of e.g. underlying BTRFS or ZFS at the same time. Looking at storage of the snapshots of the tested Windows Server 2019, at least some of those really share some space:
Total Exclusive Set shared Filename 424.41GiB 417.69GiB 6.72GiB /volume1/@sharesnap/[...]/GMT+01-2019.10.29-00.00.09 446.53GiB 400.16GiB 46.37GiB /volume1/@sharesnap/[...]/GMT+01-2019.10.30-00.00.08 483.05GiB 448.36GiB 34.70GiB /volume1/@sharesnap/[...]/GMT+01-2019.10.31-00.00.08 553.78GiB 209.26GiB 344.52GiB /volume1/@sharesnap/[...]/GMT+01-2019.11.02-00.00.09 204.68GiB 204.63GiB 0.05GiB /volume1/@sharesnap/[...]/GMT+02-2019.10.01-00.00.07 410.24GiB 405.37GiB 4.87GiB /volume1/@sharesnap/[...]/GMT+02-2019.10.27-00.00.06
It's not as much as I would expect, though. The thing is that it might be that not much space is shared even though VHDX files are reused, because when I subsequently invoke backing up C:\ of that server, backup doesn't seem to get faster.
The first backup might take longer as much data has changed, but after that is finished and with the server doing nothing, a second backup only a few minutes later should be a lot faster, because of only backing up differences of the files. But that doesn't
seem to be the case, instead it seems to take the same time like before. Additionally, while BTRFS can share differences in created snapshots between different invocations of wbadmin with the exact same command line, those are much smaller
than expected when really only backing up changed files:
root@[...]:~# btrfs filesystem du -s --gbytes /volume1/@sharesnap/[a-zA-Z]*/GMT*
Total Exclusive Set shared Filename
446.53GiB 400.20GiB 46.33GiB /volume1/@sharesnap/[...]/GMT+01-2019.10.30-00.00.08
483.05GiB 448.36GiB 34.70GiB /volume1/@sharesnap/[...]/GMT+01-2019.10.31-00.00.08
553.78GiB 546.54GiB 7.24GiB /volume1/@sharesnap/[...]/GMT+01-2019.11.02-00.00.09
39.35GiB 37.68GiB 1.67GiB /volume1/@sharesnap/[...]/GMT+01-2019.11.03-15.36.52
39.35GiB 31.18GiB 8.17GiB /volume1/@sharesnap/[...]/GMT+01-2019.11.03-15.49.03
39.35GiB 37.98GiB 1.37GiB /volume1/@sharesnap/[...]/GMT+01-2019.11.03-16.03.06
410.24GiB 409.72GiB 0.52GiB /volume1/@sharesnap/[...]/GMT+02-2019.10.27-00.00.06That is different to what I see when backing up to my USB-disks, subsequent backups are much faster if nothing has changed. What is interesting is that others seem to be not so sure about how things behave on network shares as well:
If you create a scheduled backup job to network shared folder or a mapped network drive, all the backups will only be performed by full backup because network location is not a volume. If you need to create differential backup or incremental backup to network folder, you need to third party backup software.
https://www.ubackup.com/windows-server/windows-server-backup-differential-4348.html
While the same people write the following, which doesn't make much sense:
Tip: You can also use WBADMIN to create incremental backup to network share, like this: wbadmin start backup –backupTarget: \\backupshare\backup1 -allCritical -include:C: -vssFull –quiet
https://www.ubackup.com/articles/wbadmin-incremental-backup-5740.html
In case of backing up to shares, if VHD is recreated always, like seems to be the case for older versions of Windows, one doesn't get incremental backups. But even though VHDX files are kept in newer versions of Windows, it seems like only backing up changed
files still doesn't work the same way like it does when using USB-disks.
Using -vssFull vs. -vssCopy didn't make any difference in my tests and according to my understanding is not even intended to do so. Those arguments are only relevant to 3rd party software and don't influence which files
are backed up how. Reasons to believe so are documented in my former answer:
Influence of -vssFull and -vssCopy.
https://serverfault.com/a/990394/333397
Questions
Does `wbengine` of Windows Server 2019 support incremental backups when targeting network shares? Does anyone see any improvement in backup times and storage allocations in case of snapshots?
