DwarFS
The Deduplicating Warp-speed Advanced Read-only File System.
A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows.
Table of contents
What is DwarFS (in plain words)?
DwarFS is a mountable archive: you pack a directory into one file
(an “image”), then open it instantly like a normal folder — no full
extraction, no temporary files. It’s read-only when mounted, so you can
browse, open, and run files safely in place. This works particularly well
for big collections with lots of similar or repeated content — think
many versions of a project or dataset, folders of documents and text
files, backups/snapshots that mostly overlap, or libraries with many
near-duplicates. The image can be mounted in fractions of a second, you
can use the contents immediately with no delay, and you extract only if
you actually need to.
Why not just use .zip or .tar.gz?
Traditional archives are fine for storage, but they’re usually slow to
open and awkward for random access (jumping around inside large files or
across many files). DwarFS is built for fast random reads and
space savings. It groups similar files and removes duplication, so
images are often smaller than a simple tar/zip, while reads stay
snappy — even when accessing lots of files simultaneously. In practice,
you keep huge directories compressed, mount them in milliseconds, and
work as if they were already unpacked.
This comparison uses DwarFS 0.14.1, 7-Zip 25.00 (x64), and tar +
pigz for all .tar.gz tests since plain
gzip would have been incredibly slow. Both .tar.gz and 7zip archives
were mounted using fuse-archive
v1.16 with the -olazycache option since the default of caching the entire
archive in memory is infeasible for large archives.
1139 complete Perl installations
| Perl (47.49 GiB, 1.9M files) |
.tar.gz [^pl1] |
.tar.zst [^pl2] |
7zip (-mx=7) |
SquashFS [^pl7] |
DwarFS (lzma) |
DwarFS (zstd) |
| Compression time |
4m 59s |
8m 06s |
23m 27s |
5m 37s |
2m 13s |
5m 3s |
| Compression CPU time |
1h 47m |
2h 18m |
5h 5m |
2h 29m |
31m 17s |
49m 51s |
| Compressed size |
12.17 GiB |
0.387 GiB |
1.219 GiB |
3.245 GiB |
0.310 GiB |
0.352 GiB |
| Compression ratio |
3.902 |
122.7 |
38.96 |
14.63 |
153.2 |
134.9 |
| Decompression time |
2m 19s |
57.2s |
1m 14s |
39.3s |
48.7s |
46.5s |
| Decompression CPU time |
3m 44s |
1m 21s |
2m 28s |
1m 25s |
2m 23s |
1m 59s |
| Mount time |
2m 07s |
❌ [^pl3] |
3.638s |
0.011s |
0.420s |
0.009s |
| Find all 1.9M files [^pl8] |
5.670s |
❌ [^pl3] |
5.695s |
5.311s |
2.800s |
2.821s |
| Checksum 1139 files (2.58 GiB) [^pl4] |
❌ [^pl5] |
❌ [^pl3] |
~5h [^pl6] |
1.541s |
4.330s |
1.134s |
[^pl1]: using pigz -9
[^pl2]: using zstd --long=31 --ultra -22 -T0
[^pl3]: not supported by fuse-archive
[^pl4]: $ ls -1 mnt/*/perl*/bin/perl5* | xargs -d $'\n' -n1 -P16 sha256sum
[^pl5]: killed after making no progress for 15 minutes
[^pl6]: killed when only 78 files were finished after about 20 minutes
[^pl7]: using -comp zstd -Xcompression-level 22 -b 1M -tailends; using squashfuse_ll 0.6.0
[^pl8]: $ fd -t f . mnt | wc -l
All artifacts from 205 DwarFS CI builds
| DwarFS CI (465.2 GiB, 3.6M files) |
.tar.gz (pigz -9) |
7zip (-mx=7) |
DwarFS (zstd) |
| Compression time |
40m 28s |
141m 46s |
68m 6s |
| Compression CPU time |
13h 55m |
42h 47m |
31h 12m |
| Compressed size |
142.9 GiB |
60.04 GiB |
28.63 GiB |
| Compression ratio |
3.255 |
7.748 |
16.25 |
| Decompression time |
26m 17s |
8m 15s |
7m 21s |
| Decompression CPU time |
35m 16s |
41m 29s |
8m 59s |
| Mount time |
22m 18s |
10.5s |
0.024s |
| Run 441 binaries (2.72 GiB) [^ci1] |
❌ [^ci2] |
❌ [^ci3] |
0.774s |
[^ci1]: $ ls -1 mnt/*/dwarfs-*-Linux-x86_64-*-minsize*/bin/mkdwarfs | xargs -d $'\n' -n1 -P16 sh -c '$0 -h'
[^ci2]: killed after making no progress for 15 minutes
[^ci3]: killed after making no progress and consuming more than 32 GiB of RAM
Game audio assets from sonniss.com
| Sonniss (3.072 GiB, 171 files) ^wav1 |
.tar.gz (pigz -9) |
7zip (-mx=7) |
SquashFS [^wav3] |
DwarFS (categorize) |
| Compression time |
5.34s |
6m 21s |
19.52s |
3.98s |
| Compression CPU time |
2m 1s |
19m 14s |
9m 47s |
29.0s |
| Compressed size |
2.725 GiB |
2.255 GiB |
2.711 GiB |
1.664 GiB |
| Compression ratio |
1.127 |
1.362 |
1.133 |
1.846 |
| Decompression time |
13.7s |
1m 50s |
0.774s |
1.32s |
| Decompression CPU time |
17.8s |
1m 52s |
4.60s |
9.15s |
| Mount time |
13.6s |
0.014s |
0.018s |
0.008s |
| Checksum all files [^wav2] |
3m 2s |
30m 42s |
2.81s |
2.64s |
[^wav2]: $ find mnt -type f | xargs -d $'\n' -n1 -P16 sha256sum
[^wav3]: using -comp zstd -Xcompression-level 22 -b 1M -tailends
Quick Start
To create a DwarFS image from a directory, use mkdwarfs:
$ mkdwarfs -i /path/to/input/dir -o /path/to/image.dwarfs
To mount the image, use the dwarfs FUSE driver:
$ mkdir /path/to/mountpoint
$ dwarfs /path/to/image.dwarfs /path/to/mountpoint
$ ls /path/to/mountpoint
To extract the image, use dwarfsextract:
$ dwarfsextract -i /path/to/image.dwarfs -o /path/to/output/dir
There’s also the dwarfsck tool that can be used
for a variety of tasks, for example listing the files in a DwarFS image:
$ dwarfsck /path/to/image.dwarfs -l
It can also be used for generating checksums for each file in the image
in a format that is recognized by the *sum commands:
$ dwarfsck /path/to/image.dwarfs --checksum=sha512 | sha512sum --check
This is useful if you want to verify the integrity of the actual files
stored in the DwarFS image.
Overview
DwarFS is a read-only file system with a focus on achieving very
high compression ratios, particularly for highly redundant data.
This probably doesn’t sound very exciting, because if it’s redundant,
it should compress well. However, I found that other read-only,
compressed file systems don’t do a very good job at making use of
this redundancy. See here for a comparison with other
compressed file systems.
DwarFS also doesn’t compromise on speed; in my use cases, it
performs on par with, or better than, SquashFS. For my primary use
case, DwarFS compression is an order of magnitude better than
SquashFS compression, it’s 6 times faster to build the file
system, it’s typically faster to access files on DwarFS and it uses
less CPU resources.
To give you an idea of what DwarFS is capable of, here’s a quick comparison
of DwarFS and SquashFS on a set of video files with a total size of 39 GiB.
The twist is that each unique video file has two sibling files with a
different set of audio streams (this is
an actual use case).
So there’s redundancy in both the video and audio data, but as the streams
are interleaved and identical blocks are typically very far apart, it’s
challenging to make use of that redundancy for compression. SquashFS
essentially fails to compress the source data at all, whereas DwarFS is
able to reduce the size to nearly one-third, which is close to the
theoretical maximum:
$ du -hs dwarfs-video-test
39G dwarfs-video-test
$ ls -lh dwarfs-video-test.*fs
-rw-r--r-- 1 mhx users 14G Jul 2 13:01 dwarfs-video-test.dwarfs
-rw-r--r-- 1 mhx users 39G Jul 12 09:41 dwarfs-video-test.squashfs
Furthermore, when mounting the SquashFS image and performing a random-read
throughput test using fio-3.34, both
squashfuse and squashfuse_ll top out at around 230 MiB/s:
$ fio --readonly --rw=randread --name=randread --bs=64k --direct=1 \
--opendir=mnt --numjobs=4 --ioengine=libaio --iodepth=32 \
--group_reporting --runtime=60 --time_based
[...]
READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=13.5GiB (14.5GB), run=60004-60004msec
In comparison, DwarFS manages to sustain random read rates of 20 GiB/s:
READ: bw=20.2GiB/s (21.7GB/s), 20.2GiB/s-20.2GiB/s (21.7GB/s-21.7GB/s), io=1212GiB (1301GB), run=60001-60001msec
Distinct features of DwarFS are:
-
Clustering of files by similarity using a similarity hash function.
This makes it easier to exploit the redundancy across file boundaries.
-
Segmentation analysis across file system blocks in order to reduce
the size of the uncompressed file system. This saves memory when
using the compressed file system and thus potentially allows for
higher cache hit rates as more data can be kept in the cache.
-
Categorization framework to categorize
files or even fragments of files and then process individual categories
differently. For example, this allows you to not waste time trying to
compress incompressible files or to compress PCM audio data using FLAC
compression.
-
Highly multi-threaded implementation. Both the
file system creation tool as well as the
FUSE driver are able to make good use of the
many cores of your system.
History
I started working on DwarFS in 2013 and my main use case and major
motivation was that I had several hundred different versions of Perl
that were taking up something around 30 gigabytes of disk space, and
I was unwilling to spend more than 10% of my hard drive keeping them
around for when I happened to need them.
Up until then, I had been using Cromfs
for squeezing them into a manageable size. However, I was getting more
and more annoyed by the time it took to build the file system image
and, to make things worse, more often than not it was crashing after
about an hour or so.
I had obviously also looked into SquashFS,
but never got anywhere close to the compression rates of Cromfs.
This alone wouldn’t have been enough to get me into writing DwarFS,
but at around the same time, I was pretty obsessed with the recent
developments and features of newer C++ standards and really wanted
a C++ hobby project to work on. Also, I’ve wanted to do something
with FUSE
for quite some time. Last but not least, I had been thinking about
the problem of compressed file systems for a bit and had some ideas
that I definitely wanted to try.
A lot of the initial code was written in 2013, then I did a couple
of cleanups, bugfixes and refactors every once in a while, but I
never really got it to a state where I would feel happy releasing
it. It was too awkward to build with its dependency on Facebook’s
folly library and it didn’t
have any documentation.
Digging out the project again in 2020, things didn’t look as grim
as they used to. Folly was building with CMake and so I just pulled
it in as a submodule. Most other dependencies could be satisfied
from packages that should be widely available.
However, the folly and fbthrift dependencies were still a bit of
a pain. DwarFS used only a small fraction of folly, and an even
tinier fraction of fbthrift. Folly had so many dependencies that
it was getting harder and harder to build. Often, when upgrading,
there were new build problems. So in 2026, after some of my pull
requests for folly and fbthrift had been stuck in review with no
feedback for more than half a year, I decided to ditch folly,
fork the bits I needed from fbthrift, and re-write the thrift
code generation in Python. That was quite a bit of work, but less
than I anticipated. A lot of the folly features could be replaced
by Boost or the C++ standard library. Some other, more unique
features had to be re-implemented. This, however, gave me the
freedom to just fix the things I previously had pull requests for,
and it turned out that the new implementation was measurably faster
in some cases. Also, some re-implementations were kept much simpler
than the original folly code, making the resulting binaries smaller.
A bunch of dependencies, such as gflags, glog, libevent,
double-conversion, and fast_float, could be removed entirely.
Building and Installing
Note to Package Maintainers
DwarFS should usually build fine with minimal changes out of the box.
If it doesn’t, please file an issue. I’ve set up
CI jobs
using Docker images for Ubuntu (22.04
and 24.04),
Fedora Rawhide,
openSUSE Tumbleweed,
Arch Linux, and
Debian,
as well as a setup script for FreeBSD,
that can help with determining an up-to-date set of dependencies.
Note that building from the release tarball requires less dependencies
than building from the git repository, notably the ronn tool as well
as Python and the mistletoe Python module are not required when
building from the release tarball.
There are some things to be aware of:
-
While supported by the CMakeLists.txt via -DPREFER_SYSTEM_GTEST=ON,
it is not recommended to use a system-installed version of GoogleTest.
GoogleTest itself recommends downloading it as part of the build, and
issues like this one can
arise quickly if the system-installed version is built with different
compiler flags than the DwarFS build.
-
For other bundled libraries (namely fmt, parallel-hashmap,
range-v3), the system-installed version is used as long as it
meets the minimum required version. Otherwise, the preferred
version is fetched during the build.
Prebuilt Binaries
Each release has pre-built,
statically linked binaries for a number of architectures available
for download. These should run without any dependencies and can be
useful especially on older distributions where you can’t easily build
the tools from source. For example, these binaries will work perfectly
fine on Linux distributions from 2007.
Universal Binaries
In addition to the binary tarballs, there’s a universal binary
available for each architecture. These universal binaries contain all
tools (mkdwarfs, dwarfsck, dwarfsextract and the dwarfs fuse3
driver) in a single executable. These executables are self-extracting
and use either a custom self-extractor or
upx. They are generally optimized to
provide the best performance in the smallest possible binary and are much
smaller than the individual tools combined. However, this also means the
binaries need to be decompressed each time they are run, which can add
significant overhead. If that is an issue, you can either stick to the
“classic” individual binaries or you can decompress the universal binary.
For upx-compressed binaries, you can use:
$ upx -d dwarfs-universal-0.7.0-Linux-aarch64
For the binaries that use the custom self-extractor, you can use:
$ ./dwarfs-universal-riscv64 --extract-wrapped-binary dwarfs-universal
Note that both self-extractors need at least Linux kernel 3.17 to work
properly. If you want to use the FUSE driver, you’ll need to install
the fuse3 tools for your distribution. If you want to run the binaries
on an older kernel, you can unpack the universal binary (unpacking does
not require kernel 3.17). If you’re stuck with fuse2, you must use the
individual dwarfs2 driver instead of the universal binary.
You can run the universal binaries via symbolic links named after
the tool. For example:
$ ln -s dwarfs-universal-0.7.0-Linux-aarch64 mkdwarfs
$ ./mkdwarfs --help
This also works on Windows if the file system supports symbolic links:
> mklink mkdwarfs.exe dwarfs-universal-0.7.0-Windows-AMD64.exe
> .\mkdwarfs.exe --help
Alternatively, you can select the tool by passing --tool=<name> as
the first argument on the command line:
> .\dwarfs-universal-0.7.0-Windows-AMD64.exe --tool=mkdwarfs --help
Note that just like the dwarfs.exe Windows binary, the universal
Windows binary depends on the winfsp-x64.dll from the
WinFsp project. However, for the
universal binary, the DLL is loaded lazily, so you can still use all
other tools without the DLL.
See the Windows Support section for more details.
Dependencies
DwarFS uses CMake as a build tool.
If you build from a release tarball, the build will be faster and
there will be fewer dependencies, because the release tarball ships
with the auto-generated files that would otherwise have to be generated
during the in-repository build.
You can grab an up-to-date list of dependencies from the docker files
used for the CI builds:
For FreeBSD, you can use PKGS from the
script that sets up the jail for the CI builds.
Building
First, unpack the release tarball:
$ tar xvf dwarfs-x.y.z.tar.xz
$ cd dwarfs-x.y.z
Alternatively, you can also clone the git repository, but be aware
that this has more dependencies and the build will likely take longer
because the release archive ships with most of the auto-generated
files that will have to be generated when building from the repository:
$ git clone --recurse-submodules https://github.com/mhx/dwarfs
$ cd dwarfs
Once all dependencies have been installed, you can
build DwarFS using:
$ mkdir build
$ cd build
$ cmake .. -GNinja -DWITH_TESTS=ON
$ ninja
You can then run tests with:
$ ctest -j
All binaries use jemalloc
as a memory allocator by default, as it typically uses much less
system memory compared to the glibc or tcmalloc allocators.
To disable the use of jemalloc, pass -DUSE_JEMALLOC=0 on the
cmake command line.
It is also possible to build/install the DwarFS libraries, tools,
and FUSE driver independently. This is mostly interesting when
packaging DwarFS. Note that the tools and FUSE driver require the
libraries to be either built or already installed. To build just
the libraries, use:
$ cmake .. -GNinja -DWITH_TESTS=ON -DWITH_LIBDWARFS=ON -DWITH_TOOLS=OFF -DWITH_FUSE_DRIVER=OFF
Once the libraries are tested and installed, you can build the
tools (i.e. mkdwarfs, dwarfsck, dwarfsextract) using:
$ cmake .. -GNinja -DWITH_TESTS=ON -DWITH_LIBDWARFS=OFF -DWITH_TOOLS=ON -DWITH_FUSE_DRIVER=OFF
To build the FUSE driver, use:
$ cmake .. -GNinja -DWITH_TESTS=ON -DWITH_LIBDWARFS=OFF -DWITH_TOOLS=OFF -DWITH_FUSE_DRIVER=ON
Installing
Installing is as easy as:
$ sudo ninja install
Though you don’t have to install the tools to play with them.
Static Builds
Attempting to build statically linked binaries is highly discouraged
and not officially supported. That being said, here’s how to set up
an environment where you might be able to build static binaries.
This has been tested with ubuntu-22.04-live-server-amd64.iso. First,
install all the packages listed as dependencies above. Also install:
$ apt install ccache ninja libacl1-dev
ccache and ninja are optional, but help with a speedy compile.
Depending on your distribution, you’ll need to build and install static
versions of some libraries, e.g. libarchive and libmagic for Ubuntu:
$ wget https://github.com/libarchive/libarchive/releases/download/v3.6.2/libarchive-3.6.2.tar.xz
$ tar xf libarchive-3.6.2.tar.xz && cd libarchive-3.6.2
$ ./configure --prefix=/opt/static-libs --without-iconv --without-xml2 --without-expat
$ make && sudo make install
$ wget ftp://ftp.astron.com/pub/file/file-5.44.tar.gz
$ tar xf file-5.44.tar.gz && cd file-5.44
$ ./configure --prefix=/opt/static-libs --enable-static=yes --enable-shared=no
$ make && make install
That’s it! Now you can try building static binaries for DwarFS:
$ git clone --recurse-submodules https://github.com/mhx/dwarfs
$ cd dwarfs && mkdir build && cd build
$ cmake .. -GNinja -DWITH_TESTS=ON -DSTATIC_BUILD_DO_NOT_USE=ON \
-DSTATIC_BUILD_EXTRA_PREFIX=/opt/static-libs
$ ninja
$ ninja test
Usage
Please check out the manual pages for mkdwarfs,
dwarfs, dwarfsck and
dwarfsextract. You can also access the manual
pages using the --man option to each binary, e.g.:
$ mkdwarfs --man
The dwarfs manual page also shows an example for setting up DwarFS
with overlayfs
in order to create a writable file system mount on top of a read-only DwarFS image.
A description of the DwarFS file system format can be found in
dwarfs-format(5).
A high-level overview of the internal operation of mkdwarfs is shown
in this sequence diagram.
Using the Libraries
Using the DwarFS libraries should be pretty straightforward if you’re
using CMake to build your project. For a quick
start, have a look at the example code that uses
the libraries to print information about a DwarFS image (like dwarfsck)
or extract it (like dwarfsextract).
There are five individual libraries:
-
dwarfs_common contains the common code required by all the other
libraries. The interfaces are defined in dwarfs/.
-
dwarfs_reader contains all code required to read data from a
DwarFS image. The interfaces are defined in dwarfs/reader/.
-
dwarfs_extractor contains the code required to extract a DwarFS
image using libarchive. The interfaces
are defined in dwarfs/utility/filesystem_extractor.h.
-
dwarfs_writer contains the code required to create DwarFS images.
The interfaces are defined in dwarfs/writer/.
-
dwarfs_rewrite contains the code to re-write DwarFS images. The
interfaces are defined in dwarfs/utility/rewrite_filesystem.h.
The headers in internal subfolders are only accessible at build
time and won’t be installed. The same goes for the tool subfolder.
The reader and extractor APIs should be fairly stable. The writer
APIs are likely going to change. Note, however, that there are no
guarantees on API stability before this project reaches version 1.0.0.
Windows Support
Support for the Windows operating system is currently experimental.
Having worked pretty much exclusively in a Unix world for the past two
decades, my experience with Windows development is rather limited and
I’d expect there to definitely be bugs and rough edges in the Windows
code.
The Windows version of the DwarFS file system driver relies on the awesome
WinFsp project and its winfsp-x64.dll
must be discoverable by the dwarfs.exe driver.
The different tools should behave pretty much the same whether you’re
using them on Linux or Windows. The file system images can be copied
between Linux and Windows and images created on one OS should work fine
on the other.
There are a few things worth pointing out, though:
-
Windows traditionally treats file names as case-insensitive. However,
the DwarFS driver is case-sensitive by default, even on Windows.
There’s an option -o case_insensitive that can be passed to the
driver to make it behave like a traditional Windows file system.
Note that this is only supported if the DwarFS image contains no
directories or files with the same name differing only in case within
the same directory. Otherwise, the driver will emit a warning when
mounting the image.
-
DwarFS supports both hardlinks and symlinks on Windows, just as it
does on Linux. However, creating hardlinks and symlinks seems to
require admin privileges on Windows, so if, for example, you want to
extract a DwarFS image that contains links of some sort, you might
run into errors if you don’t have the right privileges.
-
Due to a problem in
WinFsp, symlinks cannot currently point outside of the mounted file
system. Furthermore, due to another
problem in WinFsp,
symlinks with a drive letter will appear with a mangled target path.
-
The DwarFS driver on Windows correctly reports hardlink counts via
its API, but currently these counts are not correctly propagated
to the Windows file system layer. This is presumably due to a
problem in WinFsp.
-
When mounting a DwarFS image on Windows, the mount point must not
exist. This is different from Linux, where the mount point must
actually exist. Also, it’s possible to mount a DwarFS image as a
drive letter, e.g.
dwarfs.exe image.dwarfs Z:
-
Filter rules for mkdwarfs always require Unix path separators,
regardless of whether it’s running on Windows or Linux.
Building on Windows
Building on Windows is not too complicated thanks to vcpkg.
You’ll need to install:
WinFsp is expected to be installed in C:\Program Files (x86)\WinFsp;
if it’s not, you’ll need to set WINFSP_PATH when running CMake via
cmake/win.bat.
Clone vcpkg and dwarfs:
> cd %HOMEPATH%
> mkdir git
> cd git
> git clone https://github.com/Microsoft/vcpkg.git
> git clone https://github.com/mhx/dwarfs
Then, bootstrap vcpkg:
> .\vcpkg\bootstrap-vcpkg.bat
And build DwarFS:
> cd dwarfs
> mkdir build
> cd build
> ..\cmake\win.bat
> ninja
Once that’s done, you should be able to run the tests.
Set CTEST_PARALLEL_LEVEL according to the number of CPU cores in
your machine.
> set CTEST_PARALLEL_LEVEL=10
> ninja test
macOS Support
The DwarFS libraries and tools (mkdwarfs, dwarfsck, dwarfsextract)
are now available from Homebrew:
$ brew install dwarfs
$ brew test dwarfs
The macOS version of the DwarFS file system driver relies on the awesome
macFUSE project and is available via gromgit’s
homebrew-fuse tap:
$ brew tap gromgit/homebrew-fuse
$ brew install dwarfs-fuse-mac
Occasionally, when the main Homebrew repository switches to newer library
versions, the bottles in the homebrew-fuse tap can break. In that case,
you will see the FUSE driver fail to run with errors about missing libraries
from dyld. If that happens, you can
file an issue for the
homebrew-fuse tap. You can also temporarily work around the problem by
building the FUSE driver from source:
$ brew reinstall --build-from-source dwarfs-fuse-mac
Use Cases
Astrophotography
Astrophotography can generate huge amounts of raw image data. During a
single night, it’s not unlikely to end up with a few dozen gigabytes
of data. With most dedicated astrophotography cameras, this data ends up
in the form of FITS images. These are usually uncompressed, don’t compress
very well with standard compression algorithms, and while there are certain
compressed FITS formats, these aren’t widely supported.
One of the compression formats (simply called “Rice”) compresses reasonably
well and is really fast. However, its implementation for compressed FITS
has a few drawbacks. The most severe drawbacks are that compression isn’t
quite as good as it could be for color sensors and sensors with a less than
16 bits of resolution.
DwarFS supports the ricepp (Rice++) compression, which builds on the basic
idea of Rice compression, but makes a few enhancements: it compresses color
and low bit depth images significantly better and always searches for the
optimum solution during compression instead of relying on a heuristic.
Let’s look at an example using 129 images (darks, flats and lights) taken
with an ASI1600MM camera. Each image is 32 MiB, so a total of 4 GiB of data.
Compressing these with the standard fpack tool takes about 16.6 seconds
and yields a total output size of 2.2 GiB:
$ time fpack */*.fit */*/*.fit
user 14.992
system 1.592
total 16.616
$ find . -name '*.fz' -print0 | xargs -0 cat | wc -c
2369943360
However, this leaves you with *.fz files that not every application can
actually read.
Using DwarFS, here’s what we get:
$ mkdwarfs -i ASI1600 -o asi1600-20.dwarfs -S 20 --categorize
I 08:47:47.459077 scanning "ASI1600"
I 08:47:47.491492 assigning directory and link inodes...
I 08:47:47.491560 waiting for background scanners...
I 08:47:47.675241 scanning CPU time: 1.051s
I 08:47:47.675271 finalizing file inodes...
I 08:47:47.675330 saved 0 B / 3.941 GiB in 0/258 duplicate files
I 08:47:47.675360 assigning device inodes...
I 08:47:47.675371 assigning pipe/socket inodes...
I 08:47:47.675381 building metadata...
I 08:47:47.675393 building blocks...
I 08:47:47.675398 saving names and symlinks...
I 08:47:47.675514 updating name and link indices...
I 08:47:47.675796 waiting for segmenting/blockifying to finish...
I 08:47:50.274285 total ordering CPU time: 616.3us
I 08:47:50.274329 total segmenting CPU time: 1.132s
I 08:47:50.279476 saving chunks...
I 08:47:50.279622 saving directories...
I 08:47:50.279674 saving shared files table...
I 08:47:50.280745 saving names table... [1.047ms]
I 08:47:50.280768 saving symlinks table... [743ns]
I 08:47:50.282031 waiting for compression to finish...
I 08:47:50.823924 compressed 3.941 GiB to 1.201 GiB (ratio=0.304825)
I 08:47:50.824280 compression CPU time: 17.92s
I 08:47:50.824316 filesystem created without errors [3.366s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish
5 dirs, 0/0 soft/hard links, 258/258 files, 0 other
original size: 3.941 GiB, hashed: 315.4 KiB (18 files, 0 B/s)
scanned: 3.941 GiB (258 files, 117.1 GiB/s), categorizing: 0 B/s
saved by deduplication: 0 B (0 files), saved by segmenting: 0 B
filesystem: 3.941 GiB in 4037 blocks (4550 chunks, 516/516 fragments, 258 inodes)
compressed filesystem: 4037 blocks/1.201 GiB written
In less than 3.4 seconds, it compresses the data down to 1.2 GiB, almost
half the size of the fpack output.
In addition to saving a lot of disk space, this can also be useful when your
data is stored on a NAS. Here’s a comparison of the same set of data accessed
over a 1 Gb/s network connection, first using the uncompressed raw data:
find /mnt/ASI1600 -name '*.fit' -print0 | xargs -0 -P4 -n1 cat | dd of=/dev/null status=progress
4229012160 bytes (4.2 GB, 3.9 GiB) copied, 36.0455 s, 117 MB/s
And next using a DwarFS image on the same share:
$ dwarfs /mnt/asi1600-20.dwarfs mnt
$ find mnt -name '*.fit' -print0 | xargs -0 -P4 -n1 cat | dd of=/dev/null status=progress
4229012160 bytes (4.2 GB, 3.9 GiB) copied, 14.3681 s, 294 MB/s
That’s roughly 2.5 times faster. You can very likely see similar results
with slow external hard drives.
Dealing with Bit Rot
Currently, DwarFS has no built-in ability to add recovery information to a
file system image. However, for archival purposes, it’s a good idea to have
such recovery information in order to be able to repair a damaged image.
This is fortunately relatively straightforward using something like
par2cmdline:
$ par2create -n1 asi1600-20.dwarfs
This will create two additional files that you can place alongside the image
(or on a different storage), as you’ll only need them if DwarFS has detected
an issue with the file system image. If there’s an issue, you can run
$ par2repair asi1600-20.dwarfs
which will very likely be able to recover the image if less than 5% (that’s
the default used by par2create) of the image are damaged.
Extended Attributes
Preserving Extended Attributes in DwarFS Images
Extended attributes are not currently supported. Any extended attributes
stored in the source file system will not currently be preserved when
building a DwarFS image using mkdwarfs.
Extended Attributes exposed by the FUSE Driver
That being said, the root inode of a mounted DwarFS image currently exposes
one or two extended attributes on Linux:
$ attr -l mnt
Attribute "dwarfs.driver.pid" has a 4 byte value for mnt
Attribute "dwarfs.driver.perfmon" has a 4849 byte value for mnt
The dwarfs.driver.pid attribute simply contains the PID of the DwarFS
FUSE driver. The dwarfs.driver.perfmon attribute contains the current
results of the performance monitor.
Furthermore, each regular file exposes an attribute dwarfs.inodeinfo
with information about the underlying inode:
$ attr -l "05 Disappear.caf"
Attribute "dwarfs.inodeinfo" has a 448 byte value for 05 Disappear.caf
The attribute contains a JSON object with information about the
underlying inode:
$ attr -qg dwarfs.inodeinfo "05 Disappear.caf"
{
"chunks": [
{
"block": 2,
"category": "pcmaudio/metadata",
"offset": 270976,
"size": 4096
},
{
"block": 414,
"category": "pcmaudio/waveform",
"offset": 37594368,
"size": 29514492
},
{
"block": 419,
"category": "pcmaudio/waveform",
"offset": 0,
"size": 29385468
}
],
"gid": 100,
"mode": 33188,
"modestring": "----rw-r--r--",
"uid": 1000
}
This is useful, for example, to check how a particular file is spread
across multiple blocks or which categories
have been assigned to the file.
Comparison
The SquashFS, xz, lrzip, zpaq and wimlib tests were all done on
an 8 core Intel® Xeon® E-2286M CPU @ 2.40GHz with 64 GiB of RAM.
The Cromfs tests were done with an older version of DwarFS
on a 6 core Intel® Xeon® CPU D-1528 @ 1.90GHz with 64 GiB of RAM.
The EROFS tests were done using DwarFS v0.9.8 and EROFS v1.7.1 on an
Intel® Core™ i9-13900K with 64 GiB of RAM.
The systems were mostly idle during all of the tests.
With SquashFS
The source directory contained 1139 different Perl installations
from 284 distinct releases, a total of 47.65 GiB of data in 1,927,501
files and 330,733 directories. The source directory was freshly
unpacked from a tar archive to an XFS partition on a 970 EVO Plus 2TB
NVMe drive, so most of its contents were likely cached.
I’m using the same compression type and compression level for
SquashFS that is the default setting for DwarFS:
$ time mksquashfs install perl-install.squashfs -comp zstd -Xcompression-level 22
Parallel mksquashfs: Using 16 processors
Creating 4.0 filesystem on perl-install-zstd.squashfs, block size 131072.
[=========================================================/] 2107401/2107401 100%
Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072
compressed data, compressed metadata, compressed fragments,
compressed xattrs, compressed ids
duplicates are removed
Filesystem size 4637597.63 Kbytes (4528.90 Mbytes)
9.29% of uncompressed filesystem size (49922299.04 Kbytes)
Inode table size 19100802 bytes (18653.13 Kbytes)
26.06% of uncompressed inode table size (73307702 bytes)
Directory table size 19128340 bytes (18680.02 Kbytes)
46.28% of uncompressed directory table size (41335540 bytes)
Number of duplicate files found 1780387
Number of inodes 2255794
Number of files 1925061
Number of fragments 28713
Number of symbolic links 0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 330733
Number of ids (unique uids + gids) 2
Number of uids 1
mhx (1000)
Number of gids 1
users (100)
real 32m54.713s
user 501m46.382s
sys 0m58.528s
For DwarFS, I’m sticking to the defaults:
$ time mkdwarfs -i install -o perl-install.dwarfs
I 11:33:33.310931 scanning install
I 11:33:39.026712 waiting for background scanners...
I 11:33:50.681305 assigning directory and link inodes...
I 11:33:50.888441 finding duplicate files...
I 11:34:01.120800 saved 28.2 GiB / 47.65 GiB in 1782826/1927501 duplicate files
I 11:34:01.122608 waiting for inode scanners...
I 11:34:12.839065 assigning device inodes...
I 11:34:12.875520 assigning pipe/socket inodes...
I 11:34:12.910431 building metadata...
I 11:34:12.910524 building blocks...
I 11:34:12.910594 saving names and links...
I 11:34:12.910691 bloom filter size: 32 KiB
I 11:34:12.910760 ordering 144675 inodes using nilsimsa similarity...
I 11:34:12.915555 nilsimsa: depth=20000 (1000), limit=255
I 11:34:13.052525 updating name and link indices...
I 11:34:13.276233 pre-sorted index (660176 name, 366179 path lookups) [360.6ms]
I 11:35:44.039375 144675 inodes ordered [91.13s]
I 11:35:44.041427 waiting for segmenting/blockifying to finish...
I 11:37:38.823902 bloom filter reject rate: 96.017% (TPR=0.244%, lookups=4740563665)
I 11:37:38.823963 segmentation matches: good=454708, bad=6819, total=464247
I 11:37:38.824005 segmentation collisions: L1=0.008%, L2=0.000% [2233254 hashes]
I 11:37:38.824038 saving chunks...
I 11:37:38.860939 saving directories...
I 11:37:41.318747 waiting for compression to finish...
I 11:38:56.046809 compressed 47.65 GiB to 430.9 MiB (ratio=0.00883101)
I 11:38:56.304922 filesystem created without errors [323s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish
330733 dirs, 0/2440 soft/hard links, 1927501/1927501 files, 0 other
original size: 47.65 GiB, dedupe: 28.2 GiB (1782826 files), segment: 15.19 GiB
filesystem: 4.261 GiB in 273 blocks (319178 chunks, 144675/144675 inodes)
compressed filesystem: 273 blocks/430.9 MiB written [depth: 20000]
█████████████████████████████████████████████████████████████████████████████▏100% |
real 5m23.030s
user 78m7.554s
sys 1m47.968s
So in this comparison, mkdwarfs is more than 6 times faster than mksquashfs,
both in terms of CPU time and wall clock time.
$ ll perl-install.*fs
-rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
-rw-r--r-- 1 mhx users 4748902400 Mar 3 20:10 perl-install.squashfs
In terms of compression ratio, the DwarFS file system is more than 10 times
smaller than the SquashFS file system. With DwarFS, the content has been
compressed down to less than 0.9% (!) of its original size. This compression
ratio only considers the data stored in the individual files, not the actual
disk space used. On the original XFS file system, according to du, the
source folder uses 52 GiB, so the DwarFS image actually only uses 0.8% of
the original space.
Here’s another comparison using lzma compression instead of zstd:
$ time mksquashfs install perl-install-lzma.squashfs -comp lzma
real 13m42.825s
user 205m40.851s
sys 3m29.088s
$ time mkdwarfs -i install -o perl-install-lzma.dwarfs -l9
real 3m43.937s
user 49m45.295s
sys 1m44.550s
$ ll perl-install-lzma.*fs
-rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-lzma.dwarfs
-rw-r--r-- 1 mhx users 3838406656 Mar 3 20:50 perl-install-lzma.squashfs
It’s immediately obvious that the runs are significantly faster and the
resulting images are significantly smaller. Still, mkdwarfs is about
4 times faster and produces and image that’s 12 times smaller than
the SquashFS image. The DwarFS image is only 0.6% of the original file size.
So, why not use lzma instead of zstd by default? The reason is that lzma
is about an order of magnitude slower to decompress than zstd. If you’re
only accessing data on your compressed file system occasionally, this might
not be a big deal, but if you use it extensively, zstd will result in
better performance.
The comparisons above are not completely fair. mksquashfs by default
uses a block size of 128KiB, whereas mkdwarfs uses 16MiB blocks by default,
or even 64MiB blocks with -l9. When using identical block sizes for both
file systems, the difference, quite expectedly, becomes a lot less dramatic:
$ time mksquashfs install perl-install-lzma-1M.squashfs -comp lzma -b 1M
real 15m43.319s
user 139m24.533s
sys 0m45.132s
$ time mkdwarfs -i install -o perl-install-lzma-1M.dwarfs -l9 -S20 -B3
real 4m25.973s
user 52m15.100s
sys 7m41.889s
$ ll perl-install*.*fs
-rw-r--r-- 1 mhx users 935953866 Mar 13 12:12 perl-install-lzma-1M.dwarfs
-rw-r--r-- 1 mhx users 3407474688 Mar 3 21:54 perl-install-lzma-1M.squashfs
Even this is still not entirely fair, as it uses a feature (-B3) that allows
DwarFS to reference file chunks from up to two previous file system blocks.
But the point is that this is really where SquashFS tops out, as it doesn’t
support larger block sizes or back-referencing. And as you’ll see below, the
larger blocks that DwarFS is using by default don’t necessarily negatively
impact performance.
DwarFS also features an option to recompress an existing file system with
a different compression algorithm. This can be useful as it allows relatively
fast experimentation with different algorithms and options without requiring
a full rebuild of the file system. For example, recompressing the above file
system with the best possible compression (-l 9):
$ time mkdwarfs --recompress -i perl-install.dwarfs -o perl-lzma-re.dwarfs -l9
I 20:28:03.246534 filesystem rewritten without errors [148.3s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
filesystem: 4.261 GiB in 273 blocks (0 chunks, 0 inodes)
compressed filesystem: 273/273 blocks/372.7 MiB written
████████████████████████████████████████████████████████████████████▏100% \
real 2m28.279s
user 37m8.825s
sys 0m43.256s
$ ll perl-*.dwarfs
-rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
-rw-r--r-- 1 mhx users 390845518 Mar 4 20:28 perl-lzma-re.dwarfs
-rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-lzma.dwarfs
Note that while the recompressed file system is smaller than the original image,
it is still a lot bigger than the file system we previously build with -l9.
The reason is that the recompressed image still uses the same block size, and
the block size cannot be changed by recompressing.
In terms of how fast the file system is when using it, a quick test
I’ve done is to freshly mount the file system created above and run
each of the 1139 perl executables to print their version.
$ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 1.810 s ± 0.013 s [User: 1.847 s, System: 0.623 s]
Range (min … max): 1.788 s … 1.825 s 10 runs
Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 1.333 s ± 0.009 s [User: 1.993 s, System: 0.656 s]
Range (min … max): 1.321 s … 1.354 s 10 runs
Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 1.181 s ± 0.018 s [User: 2.086 s, System: 0.712 s]
Range (min … max): 1.165 s … 1.214 s 10 runs
Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 1.149 s ± 0.015 s [User: 2.128 s, System: 0.781 s]
Range (min … max): 1.136 s … 1.186 s 10 runs
These timings are for initial runs on a freshly mounted file system,
running 5, 10, 15 and 20 processes in parallel. 1.1 seconds means that
it takes only about 1 millisecond per Perl binary.
Following are timings for subsequent runs, both on DwarFS (at mnt)
and the original XFS (at install). DwarFS is around 15% slower here:
$ hyperfine -P procs 10 20 -D 10 -w1 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'" "ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 347.0 ms ± 7.2 ms [User: 1.755 s, System: 0.452 s]
Range (min … max): 341.3 ms … 365.2 ms 10 runs
Benchmark #2: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 302.5 ms ± 3.3 ms [User: 1.656 s, System: 0.377 s]
Range (min … max): 297.1 ms … 308.7 ms 10 runs
Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 342.2 ms ± 4.1 ms [User: 1.766 s, System: 0.451 s]
Range (min … max): 336.0 ms … 349.7 ms 10 runs
Benchmark #4: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 302.0 ms ± 3.0 ms [User: 1.659 s, System: 0.374 s]
Range (min … max): 297.0 ms … 305.4 ms 10 runs
Summary
'ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'' ran
1.00 ± 0.01 times faster than 'ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null''
1.13 ± 0.02 times faster than 'ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null''
1.15 ± 0.03 times faster than 'ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null''
Using the lzma-compressed file system, the metrics for initial runs look
considerably worse (about an order of magnitude):
$ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install-lzma.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 10.660 s ± 0.057 s [User: 1.952 s, System: 0.729 s]
Range (min … max): 10.615 s … 10.811 s 10 runs
Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 9.092 s ± 0.021 s [User: 1.979 s, System: 0.680 s]
Range (min … max): 9.059 s … 9.126 s 10 runs
Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 9.012 s ± 0.188 s [User: 2.077 s, System: 0.702 s]
Range (min … max): 8.839 s … 9.277 s 10 runs
Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 9.004 s ± 0.298 s [User: 2.134 s, System: 0.736 s]
Range (min … max): 8.611 s … 9.555 s 10 runs
So you might want to consider using zstd instead of lzma if you’d
like to optimize for file system performance. It’s also the default
compression used by mkdwarfs.
Now here’s a comparison with the SquashFS file system:
$ hyperfine -c 'sudo umount mnt' -p 'umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1' -n dwarfs-zstd "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'" -p 'sudo umount mnt; sudo mount -t squashfs perl-install.squashfs mnt; sleep 1' -n squashfs-zstd "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'"
Benchmark #1: dwarfs-zstd
Time (mean ± σ): 1.151 s ± 0.015 s [User: 2.147 s, System: 0.769 s]
Range (min … max): 1.118 s … 1.174 s 10 runs
Benchmark #2: squashfs-zstd
Time (mean ± σ): 6.733 s ± 0.007 s [User: 3.188 s, System: 17.015 s]
Range (min … max): 6.721 s … 6.743 s 10 runs
Summary
'dwarfs-zstd' ran
5.85 ± 0.08 times faster than 'squashfs-zstd'
So, DwarFS is almost six times faster than SquashFS. But what’s more,
SquashFS also uses significantly more CPU power. However, the numbers
shown above for DwarFS obviously don’t include the time spent in the
dwarfs process, so I repeated the test outside of hyperfine:
$ time dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4 -f
real 0m4.569s
user 0m2.154s
sys 0m1.846s
So, in total, DwarFS was using 5.7 seconds of CPU time, whereas
SquashFS was using 20.2 seconds, almost four times as much. Ignore
the ‘real’ time, this is only how long it took me to unmount the
file system again after mounting it.
Another real-life test was to build and test a Perl module with 624
different Perl versions in the compressed file system. The module I’ve
used, Tie::Hash::Indexed,
has an XS component that requires a C compiler to build. So this really
accesses a lot of different stuff in the file system:
-
The perl executables and its shared libraries
-
The Perl modules used for writing the Makefile
-
Perl’s C header files used for building the module
-
More Perl modules used for running the tests
I wrote a little script to be able to run multiple builds in parallel:
#!/bin/bash
set -eu
perl=$1
dir=$(echo "$perl" | cut -d/ --output-delimiter=- -f5,6)
rsync -a Tie-Hash-Indexed/ $dir/
cd $dir
$1 Makefile.PL >/dev/null 2>&1
make test >/dev/null 2>&1
cd ..
rm -rf $dir
echo $perl
The following command will run up to 16 builds in parallel on the 8 core
Xeon CPU, including debug, optimized and threaded versions of all Perl
releases between 5.10.0 and 5.33.3, a total of 624 perl installations:
$ time ls -1 /tmp/perl/install/*/perl-5.??.?/bin/perl5* | sort -t / -k 8 | xargs -d $'\n' -P 16 -n 1 ./build.sh
Tests were done with a cleanly mounted file system to make sure the caches
were empty. ccache was primed to make sure all compiler runs could be
satisfied from the cache. With SquashFS, the timing was:
real 0m52.385s
user 8m10.333s
sys 4m10.056s
And with DwarFS:
real 0m50.469s
user 9m22.597s
sys 1m18.469s
So, frankly, not much of a difference, with DwarFS being just a bit faster.
The dwarfs process itself used:
real 0m56.686s
user 0m18.857s
sys 0m21.058s
So again, DwarFS used less raw CPU power overall, but in terms of wallclock
time, the difference is really marginal.
With SquashFS & xz
This test uses slightly less pathological input data: the root file system of
a recent Raspberry Pi OS release. This file system also contains device inodes,
so in order to preserve those, we pass --with-devices to mkdwarfs:
$ time sudo mkdwarfs -i raspbian -o raspbian.dwarfs --with-devices
I 21:30:29.812562 scanning raspbian
I 21:30:29.908984 waiting for background scanners...
I 21:30:30.217446 assigning directory and link inodes...
I 21:30:30.221941 finding duplicate files...
I 21:30:30.288099 saved 31.05 MiB / 1007 MiB in 1617/34582 duplicate files
I 21:30:30.288143 waiting for inode scanners...
I 21:30:31.393710 assigning device inodes...
I 21:30:31.394481 assigning pipe/socket inodes...
I 21:30:31.395196 building metadata...
I 21:30:31.395230 building blocks...
I 21:30:31.395291 saving names and links...
I 21:30:31.395374 ordering 32965 inodes using nilsimsa similarity...
I 21:30:31.396254 nilsimsa: depth=20000 (1000), limit=255
I 21:30:31.407967 pre-sorted index (46431 name, 2206 path lookups) [11.66ms]
I 21:30:31.410089 updating name and link indices...
I 21:30:38.178505 32965 inodes ordered [6.783s]
I 21:30:38.179417 waiting for segmenting/blockifying to finish...
I 21:31:06.248304 saving chunks...
I 21:31:06.251998 saving directories...
I 21:31:06.402559 waiting for compression to finish...
I 21:31:16.425563 compressed 1007 MiB to 287 MiB (ratio=0.285036)
I 21:31:16.464772 filesystem created without errors [46.65s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish
4435 dirs, 5908/0 soft/hard links, 34582/34582 files, 7 other
original size: 1007 MiB, dedupe: 31.05 MiB (1617 files), segment: 47.23 MiB
filesystem: 928.4 MiB in 59 blocks (38890 chunks, 32965/32965 inodes)
compressed filesystem: 59 blocks/287 MiB written [depth: 20000]
████████████████████████████████████████████████████████████████████▏100% |
real 0m46.711s
user 10m39.038s
sys 0m8.123s
Again, SquashFS uses the same compression options:
$ time sudo mksquashfs raspbian raspbian.squashfs -comp zstd -Xcompression-level 22
Parallel mksquashfs: Using 16 processors
Creating 4.0 filesystem on raspbian.squashfs, block size 131072.
[===============================================================\] 39232/39232 100%
Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072
compressed data, compressed metadata, compressed fragments,
compressed xattrs, compressed ids
duplicates are removed
Filesystem size 371934.50 Kbytes (363.22 Mbytes)
35.98% of uncompressed filesystem size (1033650.60 Kbytes)
Inode table size 399913 bytes (390.54 Kbytes)
26.53% of uncompressed inode table size (1507581 bytes)
Directory table size 408749 bytes (399.17 Kbytes)
42.31% of uncompressed directory table size (966174 bytes)
Number of duplicate files found 1618
Number of inodes 44932
Number of files 34582
Number of fragments 3290
Number of symbolic links 5908
Number of device nodes 7
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 4435
Number of ids (unique uids + gids) 18
Number of uids 5
root (0)
mhx (1000)
unknown (103)
shutdown (6)
unknown (106)
Number of gids 15
root (0)
unknown (109)
unknown (42)
unknown (1000)
users (100)
unknown (43)
tty (5)
unknown (108)
unknown (111)
unknown (110)
unknown (50)
mail (12)
nobody (65534)
adm (4)
mem (8)
real 0m50.124s
user 9m41.708s
sys 0m1.727s
The difference in speed is almost negligible. SquashFS is just a bit
slower here. In terms of compression, the difference also isn’t huge:
$ ls -lh raspbian.* *.xz
-rw-r--r-- 1 mhx users 297M Mar 4 21:32 2020-08-20-raspios-buster-armhf-lite.img.xz
-rw-r--r-- 1 root root 287M Mar 4 21:31 raspbian.dwarfs
-rw-r--r-- 1 root root 364M Mar 4 21:33 raspbian.squashfs
Interestingly, xz actually can’t compress the whole original image
better than DwarFS.
We can even again try to increase the DwarFS compression level:
$ time sudo mkdwarfs -i raspbian -o raspbian-9.dwarfs --with-devices -l9
real 0m54.161s
user 8m40.109s
sys 0m7.101s
Now that actually gets the DwarFS image size well below that of the
xz archive:
$ ls -lh raspbian-9.dwarfs *.xz
-rw-r--r-- 1 root root 244M Mar 4 21:36 raspbian-9.dwarfs
-rw-r--r-- 1 mhx users 297M Mar 4 21:32 2020-08-20-raspios-buster-armhf-lite.img.xz
Even if you actually build a tarball and compress that (instead of
compressing the EXT4 file system itself), xz isn’t quite able to
match the DwarFS image size:
$ time sudo tar cf - raspbian | xz -9 -vT 0 >raspbian.tar.xz
100 % 246.9 MiB / 1,037.2 MiB = 0.238 13 MiB/s 1:18
real 1m18.226s
user 6m35.381s
sys 0m2.205s
$ ls -lh raspbian.tar.xz
-rw-r--r-- 1 mhx users 247M Mar 4 21:40 raspbian.tar.xz
DwarFS also comes with the dwarfsextract tool
that allows extraction of a file system image without the FUSE driver.
So here’s a comparison of the extraction speed:
$ time sudo tar xf raspbian.tar.xz -C out1
real 0m12.846s
user 0m12.313s
sys 0m1.616s
$ time sudo dwarfsextract -i raspbian-9.dwarfs -o out2
real 0m3.825s
user 0m13.234s
sys 0m1.382s
So, dwarfsextract is almost 4 times faster thanks to using multiple
worker threads for decompression. It’s writing about 300 MiB/s in this
example.
Another nice feature of dwarfsextract is that it allows you to directly
output data in an archive format, so you could create a tarball from
your image without extracting the files to disk:
$ dwarfsextract -i raspbian-9.dwarfs -f ustar | xz -9 -T0 >raspbian2.tar.xz
This has the interesting side-effect that the resulting tarball will
likely be smaller than the one built straight from the directory:
$ ls -lh raspbian*.tar.xz
-rw-r--r-- 1 mhx users 247M Mar 4 21:40 raspbian.tar.xz
-rw-r--r-- 1 mhx users 240M Mar 4 23:52 raspbian2.tar.xz
That’s because dwarfsextract writes files in inode-order, and by
default inodes are ordered by similarity for the best possible
compression.
With lrzip
lrzip is a compression utility
targeted e
README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。
评论