zlob

zlob.h

100% POSIX and glibc compatible globbing & file walking library for C, Zig, and Rust that is SIMD-first and platform-optimized, and supports all modern globbing formats.

---

zlob is a C library, zig library and a rust crate that makes globbing and file walking fast. Why? Because the available implementations are just either slow or uncomplete. Zlob is a feature-full file walker that supports all the platform specific nuances, gitignore, multicore file walking, 120% of available wildcard syntax and many more:

• 100% POSIX and glibc compatible with all the flags and features supported
• Faster than glibc up to 10x in specific cases and generally 1.2-1.7x faster. See benchmarks
• Faster than rust’s, node’s, bun’s, python’s implementation by far
• In addition to standard unix wildcard syntax supports ** recursive patterns, braces *.{c,h}, gitignore and bash extglob patterns
• Supports .gitignore out of the box, including .git/info/exclude and git’s core.excludesFile
• Exports api for matching over paths (one or many) without involving file system
• Exposes path length in the output struct for seamless FFI
• Truly optimized for all platforms:
  • usage of getdents64 syscall for faster directory listing
  • usage of getattrslistbulk on macos when requesting metadata from the walker
  • uses ntdll directly on windows
  • windows paths & patterns are normalized at compile time (both “/” and "" accepted and treated the same in patterns)

Built for fff loved by many more amazing projects!

Why it is faster?

zlob is using SIMD first implementation. It is a primary reason it is written in zig to have a native portable SIMD support at a languages level, it significantly reduces certain bottlenecks. But the primary reason of speed is that zlob is firstly analyzes the pattern and then matches paths to this patterns making patterns like ./drivers/**/*.c parsed to [drivers] and *.c which makes it not spend the time on opening useless directories and making leaf matches like suffix for small extensions and other hot and common patterns to be faster because optimized for a hot branch invariant.

One of my favourite optimizations for this project is patterns like ./**/*.{c,rs,zig} this is usually the main reason glob is used and this pattern is the most optimized in the zlob implementation:

• recursive worker is using getdents64 syscall directly which dramatically improves directory listing
• gitignore implementation allows us to optionally skip large subdirectories out of the box
• and the actual *.{c,rs,zig} pattern is precompiled down to the SIMD bitmask matching that allows to match 3 extension at once

Checkout benchmarks yourself.

Compatibility

As much as I could I converted all the tests that I found from the glibc test suite, rust’s glob crate and nodejs’s fs.glob function. It passes 100% of test suite for both file system and string paths matching. As long as this we have a compatibility layer with direct libc and bash for compatibility, check or extend it here ./test/test_libc_comparison.sh

Supported patterns

Any pattern you may think of should be already supported including gnu symbol classes, negation of character group, and event bash’s extglob syntax.

Here are some examples:

Pattern	Required flags	Description
*.c		Match all .c in one dir
./**/*.c		Match all .c in current and subdirs
**/*.c		Match all .c in current and subdirs
./{a,b}/*.c	ZLOB_BRACE	Match all .c in a and b dirs
./{test,src}/**/*.{c,h}	ZLOB_BRACE	Match all .c and .h in test and src dirs
~/*.c	ZLOB_TILDE	Match all .c in home directory
[a-z]*.c		Match all .c starting with lowercase
*.[ch]		Match all .c and .h and .ch
*.[!ch]		Match all files that do not end with .c or .h
@(a | b).c	ZLOB_EXTGLOB	Match all .c that are named a.c or b.c
@(!a).c	ZLOB_EXTGLOB	Match all .c that are not named a.c

API

zlob primarily support 3 public apis: C library, Zig library, and Rust crate

C

The easiest way is to look at include/zlob.h it exposes the same API as POSIX glob requires. Just change prefix glob -> zlob.

#include "zlob.h"

glob_t globbuf;
// recommended flag is default set of settings we recommended, see below
int ret = zlob("*.c", ZLOB_RECOMMENDED, NULL, &globbuf);

if (ret == 0) {
    for (size_t i = 0; i < globbuf.gl_pathc; i++) {
        printf("%s\n", globbuf.gl_pathv[i]);
    }
    zlobfree(&globbuf);
} else {
    fprintf(stderr, "glob error: %d\n", ret);
}

Zig

zlob exposes a native zig module. Add it to your build.zig.zon and import as @import("zlob").

const std = @import("std");
const zlob = @import("zlob");

pub fn main(init: std.process.Init) !void {
    const allocator = init.arena.allocator();

    if (try zlob.match(allocator, init.io, "**/*.{zig,zon}", .{
        .brace = true,
        .gitignore = true,
    })) |*result| {
        defer result.deinit();
        var it = result.iterator();
        while (it.next()) |path| {
            std.debug.print("{s}\n", .{path});
        }
    }

    // or match against an in-memory path list: no allocations, no io
    const paths = [_][]const u8{
        "src/lib.zig",
        "src/main.zig",
        "README.md",
    };
    var result = try zlob.matchPaths(allocator, "*.zig", &paths, .{});
    defer result.deinit();

    var it = result.iterator();
    while (it.next()) |p| {
        std.debug.print("{s}\n", .{p});
    }
}

If you use the parallel walker (zlob.walk), opt out of Zig std’s default 256 KiB thread-local signal stack in your root module. std.Thread instantiates it for every spawned thread and glibc zeroes it inside pthread_create, which costs ~64 page faults and a sigaltstack syscall per worker on every walk (measured 25-35% of wall time on small trees at 16 threads):

pub const std_options: std.Options = .{
    .signal_stack_size = null,
    .enable_segfault_handler = false,
};

Rust

zlob is also shared as an officially supported rust crate, find it on crates.io and read the rust docs

use zlob::{zlob, zlob_match_paths, ZlobFlags};

fn main() -> Result<(), zlob::ZlobError> {
    if let Some(result) = zlob("**/*.{rs,toml}", ZlobFlags::RECOMMENDED)? {
        for path in &result {
            println!("{}", path);
        }
    }

    let paths = ["src/lib.rs", "src/main.rs", "README.md"];
    if let Some(matches) = zlob_match_paths("*.rs", &paths, ZlobFlags::GITIGNORE)? {
        for path in &matches {
            println!("{}", path);
        }
    }

    Ok(())
}

Flags

Behavior is controlled using zlob flags. ZLOB_RECOMMENDED makes zlob behave like a modern glob implementation without sorting the output results, and enables all the modern features that you might need. Additional flags that can be used are:

Included in ZLOB_RECOMMENDED

• ZLOB_NOSORT - do not sort the output results, this is default behavior and it is recommended to use it for better performance
• ZLOB_BRACE - enable support for {a,b} patterns
• ZLOB_TILDE - expand ~ to the home directory
• ZLOB_TILDE_CHECK - checks the user exists when expanding ~user and returns error if it doesn’t

Not included in ZLOB_RECOMMENDED

• ZLOB_GITIGNORE - reads the .gitignore file in the scanning directory first and uses as a prefilter

• ZLOB_PERIOD - Allows to match hidden files using * and ? patterns, by default these patterns do not match hidden files

• ZLOB_EXTGLOB - enable support for bash extglob patterns like @(pattern-list), !(pattern-list), ?(pattern-list), *(pattern-list) and +(pattern-list)

• ZLOB_FOLLOW_SYMLINKS - follow symlinked directories when recursing with ** (see Symlinks and ** below)

• ZLOB_MATCH_BASENAME - a pattern with no / matches file names at any depth, the way find -name, gitignore, rg -g and minimatch’s matchBase treat them (see Basename matching below)

• ZLOB_NOCHECK - if no matches found return the pattern itself as the only result

• ZLOB_NOMAGIC - if the pattern contains no special characters return the pattern itself as the only result

• ZLOB_NOESCAPE - disable backslash escaping

• ZLOB_MARK - append a slash to each directory match

• ZLOB_ONLYDIR - only match directories

• ZLOB_ERR - return error on unreadable directories

• ZLOB_APPEND - append results to the output buffer instead of overwriting it

• ZLOB_DOOFFS - reserve zlo_offs slots in the output buffer for custom use, these slots will be filled with NULL and the actual results will start from zlo_pathv[zlo_offs]

In addition to this zlob exposes zlob_at function that will open specific directory instead of requiring to manipulate CWD

File walker

zlob also exposes its traversal engine directly: a parallel recursive file walker designed to replace the Rust walkdir and ignore crates, available from Zig, Rust and C.

• Parallel: one directory = one task on a work stealing pool
• Bulk metadata: pass a mask of the attributes you need (size, mtime, inode, mode, …) and only those are matched in a platform-specific optimized way
• Only required metadata: you likely don’t need all the available metadata, zlob optimizes fstat to get only what you need.
• .gitignore first: Specifically optimized to handle file ignoring first P.S. Also let’s use reuse assembled (incl nested) gitignore rules after the walk finished

1. Collect - efficient way to collect the results in to the memory if you need to store them
2. Run - call the callback per the matched entries (for grep style print on search and other workflows when you don’t need to store the files list)

Traversal can also be narrowed with a one or many (z)glob patterns.

Here is how you can use it from different languages, starting with Rust:

use zlob::walk::{WalkBuilder, WalkFlags, WalkState, WalkMetadata};

// fastest: get all the results allocated
let results = WalkBuilder::new("/path/to/folder")
    // request metadata
    .metadata(WalkMetadata::SIZE | WalkMetadata::MTIME)
    .build()?;

for entry in results.iter() {
    println!("{} {:?}", entry.path().display(), entry.size());
}

// ignore-crate style traversal: nested .gitignore/.ignore, each repository's
// .git/info/exclude and core.excludesFile (opt out of the last two with
// WalkFlags::NO_GIT_EXCLUDE / WalkFlags::NO_GIT_GLOBAL):
WalkBuilder::new(".").options(WalkFlags::GITIGNORE).run(|entry| {
    println!("{}", entry.path().display());

    WalkState::Continue
})?;

// Also inherit parent ignore rules when walking a Git worktree subtree.
let subtree = WalkBuilder::new("src")
    .options(WalkFlags::GITIGNORE | WalkFlags::GITIGNORE_PARENTS)
    .build()?;

// Glob-scoped traversal: only src/ is ever descended into.
let rs_files = WalkBuilder::new(".").include("src/**/*.rs").build()?;

C (zlob_walk / zlob_walk_collect, see zlob.h):

zlob_walk_options_t opts = {0};
opts.flags = ZLOB_WALK_GITIGNORE;
opts.meta_mask = ZLOB_META_SIZE | ZLOB_META_MTIME;

zlob_walk_result_t res;
if (zlob_walk_collect(".", &opts, &res) == 0) {
    for (size_t i = 0; i < res.count; i++)
        printf("%s (%llu bytes)\n", res.entries[i].path,
               (unsigned long long)res.entries[i].size);
    zlob_walk_result_free(&res);
}

Zig (zlob.walk):

var results = try zlob.walk.collect(allocator, ".", .{
    .meta = .{ .size = true, .mtime = true },
});
defer results.deinit();
for (results.entries) |*e| std.debug.print("{s}\n", .{e.path});

Measured on a real 185k-entry repository (warm cache, i7-14700K, best of 5, cargo bench --bench walk_comparison -- "medium/"):

workload	zlob	competitor
plain walk (185,737 entries)	27.4 ms	walkdir 90.9 ms
gitignore walk, serial (5,884 entries)	2.4 ms	ignore 8.0 ms
gitignore walk, parallel	0.64 ms	ignore parallel 4.9 ms
walk + size/mtime metadata	24.3 ms	walkdir + metadata() 226 ms

Entry counts agree with walkdir and ignore exactly on the same trees.

Basename matching

By default a pattern is matched against the whole (base-relative) path, so foo* only sees entries in the top directory and main.c is one exact path. With ZLOB_MATCH_BASENAME a pattern that contains no / is instead matched against the last path component at any depth, which is what find -name, gitignore rules, rg -g, fd -g and minimatch’s matchBase all do:

Pattern (with ZLOB_MATCH_BASENAME)	Matches	Does not match
libc.so*	libc.so.1, nix/store/x/lib/libc.so.6	libc.so.6/notes.txt
README.md	README.md, docs/api/README.md	README.md.bak
*.{c,h}	src/a.c, include/b.h	a.cc
*foo*	a/foo.txt	a/foo/b.txt
src/*.c, ./main.c, /etc/*	unchanged: anything with a / stays anchored to the base

• *, ? and [...] never cross a / under this flag because they only ever see one component; without the flag * crosses / in zlob_match_paths.
• The filesystem API (zlob(), zlob_at()) globs such patterns as **/pattern, so the whole tree below the base is walked and the usual hidden-file rules of ** apply (ZLOB_PERIOD to enter dot directories). ZLOB_NOCHECK still returns the pattern as you wrote it.
• The in-memory APIs (zlob_match_paths*, zlob_pattern_*) do not look at parent components at all, so a hidden ancestor does not block a match and the mode is as cheap as a suffix compare: a literal is checked from the end of the path without scanning it, prefix* / prefix*.ext / [set] go through the single-component template matchers, and *.ext is already a basename test.
• Not included in ZLOB_RECOMMENDED: with it zlob("*.c", ...) would recurse into every subdirectory instead of listing the current one like a shell.

Symlinks and **

By default ** does not descend into symlinked directories. This matches bash globstar, zsh **, and walkdir — the dominant ** implementations. Symlinks are still matched by name like any other entry; only the recursion stops at them. POSIX glob(3) has no ** at all, so there is no “glibc default” to diverge from here.

Set ZLOB_FOLLOW_SYMLINKS to descend into symlinked directories. zlob keeps a (dev, ino) visited-set so it never loops and never emits the same physical file twice: each real directory is traversed at most once for the whole walk.

This is a deliberate difference from glob crate / nu-glob, which follow symlinks with no cycle tracking. On trees where one directory is reachable through several symlinks (e.g. a Bazel bazel-bin / bazel-out / bazel-<workspace> layout, all pointing into the same cache), those tools emit every aliased path — inflating their count several-fold and looping until PATH_MAX on a true cycle. zlob returns the de-duplicated set instead.

MatchPaths mode

Match paths allowing you to submit a pointer of C strings or rust/zig-like slices and get the matches pointer back.

const char *paths[] = {
    "src/main.c", "src/utils.c", "src/tests/test.h", "readme.md", "src/lib.c",
};
const size_t path_count = sizeof(paths) / sizeof(paths[0]);

zlob_t pzlob;
int result = zlob_match_paths("**/*.c", paths, path_count, ZLOB_RECOMMENDED, &pzlob);

// Make sure that this does NOT free the input paths, the pointer to the paths are owned by the caller
// and have to be freed by the caller only after the zlob_t struct is freed to prevent dangling pointer
zlobfree(&pzlob);

This allows a very fast SIMD processing of the paths and supports NOT ALL the features of the standard FS globbing except ALTDIRFUNC which is not applicable because this mode is done to avoid using ALTDIRFUNC all along. Make sure that if you will use ZLOB_TILDE flag the paths input have to be absolute. Other flags like nomagic might not work as expected because they generally make very little sense.

Compilation

To compile zlob as a C library, install Zig 0.17.1 or newer.

make
make test
make install <PREFIX>

I know it might be annoying to install zig but zig’s linker is currently a decent way to cross compile any native code so I would definitely recommend trying it out.

Benchmarks

Numbers below come from the criterion harness in rust/benches/glob_comparison.rs, comparing the zlob crate against the glob crate and the globset crate (paired with walkdir where it needs to walk the FS). The fixture is a Linux kernel checkout: 93,638 files / 6,157 directories / 36,685 .c files / 99 symlinks.

Here is a benchmark. Median wall time, lower is better.

Pattern (matches)	zlob	glob crate	globset + walkdir
fs/*.c (73)	9.84 µs	31.9 µs   (3.2×)	40.3 ms   (~4 100×)
*/Makefile (21)	11.4 µs	39.1 µs   (3.4×)	40.4 ms   (~3 550×)
[fk]*/*.c (179)	25.0 µs	81.8 µs   (3.3×)	41.6 ms   (~1 660×)
drivers/*/*.c (4 314)	700 µs	1.70 ms   (2.4×)	40.9 ms   (58×)
drivers/**/*.c (22 107)	8.23 ms	16.4 ms   (2.0×)	41.4 ms   (5.0×)
net/**/*.c (1 475)	230 µs	597 µs   (2.6×)	40.2 ms   (175×)
net/**/*.{c,h} (1 747)	266 µs	n/a (no brace support)	40.3 ms   (151×)
**/*.c (36 685)	24.2 ms	49.2 ms   (2.0×)	43.8 ms   (1.8×)

Hardware / config: Linux x86_64 Intel® Core™ i7-14700K, 14 cores (28vcpu), ReleaseFast static build via zig 0.16.0 & cargo bench via criterion

Reproduce with:

# clones a shallow Linux kernel into /tmp/linux if no path is given
scripts/run-benchmarks.sh [/path/to/linux]

The script just sets REPO= and runs cargo bench --bench glob_comparison. The harness prints a match-count parity table at startup so you can see whether each library is doing the same amount of work.

Naming

This is my favourite part. zlob is not just zig zlob but also it means redneck in a various Eastern-European languages but in Polish it means “a manger”, which I find very funny.

License

zlob is licensed under MIT license, see LICENSE file for more details.

P.S. No AI was used in the making of this README.md file thank you for reading it till the end.