Gentoo Snapshots and Snapshot Deltas

Posted:: 2023-07-07

Gentoo snapshots are essentially compressed tarballs of the entire Gentoo ebuild repository. They are automatically generated on a daily basis, along with snapshot deltas which contain differences between two consecutive snapshots.

Snapshots and snapshot deltas can be downloaded from any of Gentoo mirrors (e.g., https://gentoo.osuosl.org/snapshots/). When using the tool "emerge-webrsync" to update local ebuild repository, it will download the most recent snapshot and decompress it locally. Alternatively, a tool called "emerge-delta-webrsync" can be used for the same purpose. However its primary use case is to download snapshot deltas and apply them to a previous snapshot in order to construct the latest version.

Here is an example of a snapshot directory tree:

snapshots/
    deltas/
        snapshot-20230706-20230707.patch.bz2
        snapshot-20230706-20230707.patch.bz2.md5sum
    gentoo-20230707.tar.xz
    gentoo-20230707.tar.xz.md5sum
    portage-20230707.tar.bz2
    portage-20230707.tar.bz2.md5sum

Note

Gentoo releases snapshots in two different formats, "portage-yyyymmdd.tar.bz2" (used by the tool "emerge-delta-webrsync") and "gentoo-yyyymmdd.tar.xz" (used by the tool "emerge-webrsync"). Their contents are same, with the only difference being the name of the top-leve directory after decompression - "portage" for "portage-yyyymmdd.tar.bz2" and "gentoo-yyyymmdd" for "gentoo-yyyymmdd.tar.xz".

To prove that, use the tool "rsync" to determine if there are any differences between them:

$ tar -xvf portage-yyyymmdd.tar.bz2
$ tar -xvf gentoo-yyyymmdd.tar.xz

no output indicates there are no differences between them
$ rsync -acn portage gentoo-yyyymmdd

Or, check the content of the decompressed file "metadata/timestamp.commit", which should contain the same git commit id that the snapshot is based on:

$ tar -xvf portage-yyyymmdd.tar.bz2 --strip-components=1 -C portage
$ cat portage/metadata/timestamp.commit

$ tar -xvf gentoo-yyyymmdd.tar.xz --strip-components=1 -C gentoo
$ cat gentoo/metadata/timestamp.commit

Using snapshot deltas instead of full snapshots can save not only bandwidth, but also disk space. For my personal use case, I maintain a personal snapshot mirror, which retains a three-month history of snapshots, allowing me to update to any version from the last three months - unlike Gentoo's official mirrors, which only keep snapshots for the last seven days. I have a daily cron job syncing snapshot deltas from Gentoo's official mirrors and removing any that are older than 3 months. I prefer snapshot deltas over full snapshots mainly to save disk space - a full snapshot is roughly 70M while a snapshot delta is only about 5M.

As mentioned above, snapshot deltas can be downloaded and applied using the tool "emerge-delta-webrsync". A more convenient way is enable it in the Portage repository configuration file "/etc/portage/repos.conf", which tells the tool emerge --sync <REPO> to internally call the command "emerge-delta-webrsync" to retrieve snapshot deltas when available:

# nano /etc/portage/repos.conf/<REPO>.conf
        sync-webrsync-delta = yes

Thanks for reading :)