fgtk
A set of a misc tools to work with files and processes.
Various oldish helper scripts/binaries I wrote to help myself with
day-to-day tasks.
License for all scripts is WTFPL (public domain-ish - see below),
feel free to just copy and use these in whatever way you like.
Repository URLs:
Contents - links to doc section for each script here:
Scripts
[-root-] Various CLI/system things
File/dir/fs management
File/link/dir and filesystem structure manipulation tools.
Non-interactive CLI tool to keep a list of files to symlink or copy into/from
some “dotfiles” configuration dir or repository, and keep/check/update/restore
metadata manifest for these files.
Keeps track of ACLs, POSIX capabilities and xattrs for metadata, runs file
diffs for file copies and links, supports a bunch of neat symlinking options
(like using relative symlinks, relative symlinks into symlinked repo-dir, etc).
Idea is to keep links and metadata manifest files in some configuration repo,
and run the tool occasionally after system updates or manual changes to pull
updated files into repo, update files on fs from the repo, fix links/permissions
on fs, copy/add new ones, etc - all manifest/maintenance ops done via this script.
Format for links-list looks something like this:
.gitconfig -> .git/config
/usr/share/zoneinfo/Asia/Yekaterinburg -> /etc/localtime
bpf -> /etc/bpf
zshrc > /etc/zsh/zshrc
kernel-config > /usr/src/linux/.config
myapp/secret.conf -> /etc/myapp/secret.conf
myapp/suid.bin -> /usr/local/bin/myapp
myapp/caps.bin -> /usr/local/bin/myapp-helper
And metadata is also a simple plaintext file, with fancier stuff towards the
end of lines, on paths where it’s used/needed:
.gitconfig root:root:644
bpf root:wheel:750
zshrc root:root:644
kernel-config root:wheel:664
myapp/secret.conf root:root:600
myapp/suid.bin root:root:4711
myapp/caps.bin root:root:4700/EP:net_raw/u::rwx,u:netuser:--x,g::r-x,m::r-x,o::---
In addition to lists, there’re separate links/meta exclude-files with regexps of
paths to not warn about being missing in links-list or track metadata for.
Only needs python3 to run, has bundled implementation for parsing/encoding
modern linux ACLs/capabilities extended attributes.
Uses git diff --no-index for --diff-cmd by default, as it is very fast,
has nice colors and should be widely available.
Started as a cfgit project long time ago, evolved away into this more generic
(and not necessarily git-related) tool.
Small tool to match lines from stdin according to ini config file
and run commands for any matching regexps specified there.
Intended as a long-running handler for monitoring some process’ output,
e.g. monitor some log via tail -F file.log, or react to fanotify
filesystem updates from fatrace efficiently.
For example, with myapp-changes.conf file like this:
# Add 10s delay for changes to settle before running commands
delay = 10_000
[data-file-updates]
regexp = : \S*[WD+<>]\S* */srv/myapp/data-files(/[^/]+)?$
run = myapp process-new-data /srv/myapp/data-files
# regexp-env-var = RCP_MATCH -- "run" command will get this in env by default
# regexp-env-group = 1 -- regexp group to put into regexp-env-var, 0 - full match
# regexp-run-group = 1 -- to run/delay/cooldown commands based on matched group
[config-updates]
regexp = : \S*[WD+<>]\S* */srv/myapp/config(/.*)?$
run = pkill -x HUP myapp
…tool can be run as fatrace | run_cmd_pipe myapp-changes.conf (or exec
input-command without shell via ... -- cmd args... by itself), to process
any file-change events and run relevant commands to react to those in a daemon loop.
Can have cooldown and debouncing delay for rules, reloads config-file on SIGHUP,
runs only one process per rule at a time, has small mem footprint, no deps, etc etc.
-h/--help output has more info on configuration format and cli opts.
Build with:
nim c -d:release --opt:size run_cmd_pipe.nim && strip run_cmd_pipe
One interesting use I’ve found in combination with fatrace is to
monitor and synchronize local containers, as well as handle events from those.
Wrapper around GNU find (from findutils) to accept paths at the end of argv
if none are passed before query.
Makes it somewhat more consistent with most other commands that accept
options and a lists of paths (almost always after opts),
but still warns when/if reordering takes place.
No matter how many years I’m using that tool, still sometimes type paths
after query there, so decided to patch around that frustrating issue one day.
Python script to “shift” or “patch” uid/gid values with new container-id
according to systemd-nspawn schema, i.e. set upper 16-bit to specified
container-id value and keep lower 16 bits to uid/gid inside the container.
Similar operation to what systemd-nspawn’s --private-users-chown option does
(described in nspawn-patch-uid.c), but standalone, doesn’t bother with ACLs or
checks on filesystem boundaries.
Main purpose is to update uids when migrating systemd-nspawn containers or
adding paths/filesystems to these without clobbering ownership info there.
Should be safe to use anywhere, as in most non-nspawn cases upper bits of
uid/gid are always zero, hence any changes can be easily reverted by running
this tool again with -c0.
bindfs wrapper script to setup id-mapping from uid of the mountpoint
to uid/gid of the source directory.
I.e. after bindfs-idmap /var/lib/machines/home/src-user ~dst-user/tmp,
~dst-user/tmp will be accessible to dst-user as if they were src-user,
with all operations proxied to src-user’s dir.
Anything created under ~dst-user/tmp will have uid/gid of the src dir.
Useful to allow temporary access to some uid’s files in a local container
to a user id in a main namespace.
For long-term access (e.g. for some daemon), there probably are better options
than such bindfs hack - e.g. bind/idmapped mounts, shared uids/gids, ACLs, etc.
Simple bash script to symlink uppermost “merged” overlayfs layer of a running
docker-compose setup container, to allow easy access to temporary files there.
Useful for testing stuff without the need to rebuild and restart whole container
or a bunch of compose stuff after every one-liner tweak to some script that’s
supposed to be running in there, or to experiment-with and debug things.
These paths are very likely to change between container and docker-compose
restarts for many reasons, so such symlinks are generally only valid during
container runtime, and script needs a re-run to update these too.
Very simple “write 512B, skip N * 512B, repeat” binary for wiping some block
device in a hurry.
Idea is not to erase every trace of data or to hide it, but just to make files
probabilistically unusable due to such junk blocks all over the place.
With low-enough intervals it should also corrupt filesystem pretty badly,
making metadata hard to access.
Fast loop of 512B writes to a device directly will likely hang that binary until
it’s done, as that’s how such direct I/O seem to work on linux.
Writes only stop when write() or lseek() starts returning errors, so using this
on some extendable file will result in it eating up all space available to it.
See head of the file for build and usage info.
More functionality similar to common “ls” tool, to list files in some specific
ways that are occasionally useful. All those are available via various options -
see -h/--help for a full list.
For example, to print up to N -a/--adjacent files (within some specific ordering):
% lsx -aS data/chunk-12345.bin # default up to 10 before/after, w/ S=size ordering
% lsx -a 50as data/chunk-13.bin # only 50 files larger than specified one
% lsx -a 5bt myapp/state.log # up to 5 logs right before state.log by mtime
% lsx -fa a3 logs/20230515.log # 3 log-files (-f/--files) with names after that one
Or files within -t/--mtime vicinity/ranges:
% lsx -t 1h cache/a/bcdefg.json # files created/changed within 1h of that one
% lsx -t 5d/10d cache/*/* # mtime in 5d-10d ago range
% lsx -rt 2024-10-20/2024-10-25 # between those dates in the current dir
% lsx -rt 1am/3:30 logs # logs changed from 1am to 3:30am earlier today
Simple python script with no extra dependencies.
Python script to recursively shorten (truncate) file/directory names
under specified byte-limit, respecting typical filename format, suffixes
and multibyte encodings.
Useful for transferring files from NTFS and similar filesystems
to POSIX/linux ones that have strict 255-byte filename-length limit,
where non-english paths can get very long fast bytewise.
Truncates names decoded to unicode characters to avoid splitting those,
has somewhat complicated rules for how to truncate filenames with dot-suffixes
and multiple dots in them, disambiguates rename destinations on conflicts,
always keeps longest filename possible under -l/--max-len limit,
inserts unicode-ellipsis (…) character to indicate where truncation was made.
Defaults to dry-run mode for safety, only printing all renames to be made.
Safe-ish rm(1) tool replacement for removing files by absolute or relative
pathnames under one specified directory, i.e. restricting all file removals
to that dir.
Does this in a safe way with openat2(checked_realpath, RESOLVE_NO_SYMLINKS)
(plus RESOLVE_NO_XDEV with -x option), which works correctly for any paths,
while preventing all time-of-check-time-of-use race conditions with symlinks
and mountpoints inserted at any time during this operation.
It allows to safely cleanup some arbitrary directory using a file-list with any random
paths in there, and be certain that nothing outside of that directory will be affected,
regardless of how malicious filesystem contents or paths on the list might be.
At the same time, e.g. rmx -d /mnt/storage /tmp/lists/vms/test.img will work with
symlinks in file-path going to /mnt/storage somewhere (which itself can be a symlink too),
as everything is resolved and checked reliably using file-dir realpaths first.
Written in C, can be built with gcc -Wall -O2 -o rmx rmx.c && strip rmx (~15K binary).
Safe rm to restrict file removals blog post has a bit more info on the rationale behind this.
Multi-rename tool to present filenames as a text file, allowing to
edit those in a regular $EDITOR and apply renames based on changes there.
Always presents a diff of name-changes made before applying those,
and requires an interactive confirmation via button prompt.
Has recursive mode, runs a bunch of sanity-checks before doing anything,
allows for file removals as well (by removing lines from a tagged file-list),
detects “delta” or “colordiff” tools for nicer changes’ summary.
There’s a short blog post describing this multi-rename idea in more detail.
Things that manipulate some kind of data formats or mangle generic file/pipe contents.
Ever needed to check if file has newlines or BOM in it, yet every editor is
user-friendly by default and hides these from actual file contents?
One fix is hexdump or switching to binary mode, but these are usually terrible
for looking at text, and tend to display all non-ASCII as “.” instead of nicer
\r \t \n … escapes, not to mention unicode chars.
This trivial script prints each line in a file via python’s repr(), which is
usually very nice, has none of the above issues and doesn’t dump byte codes on
you for anything it can interpret as char/codepoint or some neat escape code.
Has opts for text/byte mode and stripping “universal newlines” (see newline= in
built-in open() func).
Can also do encoding/newline conversion via -c option, as iconv can’t do BOM or
newlines, and sometimes you just want “MS utf-8 mode” (repr -c utf-8-sig+r).
Using that with +i flag as e.g. repr -c utf-8-sig+ri file1 file2 ...
converts encoding+newlines+BOM for files in-place at no extra hassle.
Outputs terminal color sequences, making important output more distinctive.
Also can be used to interleave “tail -f” of several logfiles in the same terminal:
% t -f /var/log/app1.log | color red - &
% t -f /var/log/app2.log | color green - &
% t -f /var/log/app2.log | color blue - &
Or to get color-escape-magic for your bash script: color red bold p
Script (py3) to find all specified (either directly, or by regexp)
hostnames and replace these with corresponding IP addresses, resolved
through getaddrinfo(3).
Examples:
% cat cjdroute.conf
... "fraggod.net:21987": { ... },
"localhost:21987": { ... },
"fraggod.net:12345": { ... }, ...
% resolve-hostnames fraggod.net localhost < cjdroute.conf
... "192.168.0.11:21987": { ... },
"127.0.0.1:21987": { ... },
"192.168.0.11:12345": { ... }, ...
% resolve-hostnames -m '"(?P<name>[\w.]+):\d+"' < cjdroute.conf
% resolve-hostnames fraggod.net:12345 < cjdroute.conf
% resolve-hostnames -a inet6 fraggod.net localhost < cjdroute.conf
...
% cat nftables.conf
define set.gw.ipv4 = { !ipv4.name1.local, !ipv4.name2.local }
define set.gw.ipv6 = { !ipv6.name1.local, !ipv6.name2.local }
...
# Will crash nft-0.6 because it treats names in anonymous sets as AF_INET (ipv4 only)
% resolve-hostnames -rum '!(\S+\.local)\b' -f nftables.conf
define set.gw.ipv4 = { 10.12.34.1, 10.12.34.2 }
define set.gw.ipv6 = { fd04::1, fd04::2 }
...
Useful a as conf-file pre-processor for tools that cannot handle names properly
(e.g. introduce ambiguity, can’t deal with ipv4/ipv6, use weird resolvers, do it
dynamically, etc) or should not be allowed to handle these, convert lists of
names (in some arbitrary format) to IP addresses, and such.
Has all sorts of failure-handling and getaddrinfo-control cli options, can
resolve port/protocol names as well.
Python/Jinja2 script to produce a text file from a template, focused
specifically on templating configuration files, somewhat similar to
“resolve-hostnames” above or templating provided by ansible/saltstack.
Jinja2 env for template has following filters and values:
-
dns(host [, af, proto, sock, default, force_unique=True]) filter/global.
getaddrinfo(3) wrapper to resolve host (name or address) with optional
parameters to a single address, raising exception if it’s non-unique by default.
af/proto/sock values can be either enum value names
(without AF/SOL/SOCK prefix) or integers.
-
hosts - /etc/hosts as a mapping.
For example, hosts-file line 1.2.3.4 sub.host.example.org will produce
following mapping (represented as yaml):
sub.host.example.org: 1.2.3.4
host.example.org:
sub: 1.2.3.4
org:
example:
host:
sub: 1.2.3.4
Can be used as a reliable dns/network-independent names. --hosts-opts
cli option allows some tweaks wrt how that file is parsed.
See also HostsNode object for various helper methods to lookup those.
-
iface - current network interfaces and IPv4/IPv6 addresses
assigned there (fetched from libc getifaddrs via ctypes).
Example value structure (as yaml):
enp1s0:
- 10.0.0.134
- fd00::134
- 2001:470:1f0b:11de::134
- fe80::c646:19ff:fe64:632f
enp2s7:
- 10.0.1.1
lo:
- 127.0.0.1
- ::1
ip_vti0: []
Probably a good idea to use this stuff only when IPs are static and
get assigned strictly before templating.
-
❴% comment_out_if value[, comment-prefix] %❵...❴% comment_out_end %❵
(curly-braces are weird to avoid jinja2 in github-pages - replace with normal ones)
Custom template block to prefix each non-empty line within it with specified
string (defaults to “#”) if value is not false-y.
Can be used when format doesn’t have block comments, but it’s still desirable
to keep disabled things in dst file (e.g. for manual tinkering) instead of
using if-blocks around these, or to make specific lines easier to uncomment manually.
-
it - itertools, zip builtin, szip - zip(a.split(), b.split()),
_v/v_/_v_ - global funcs for adding spaces before/after/around non-empty strings.
-
Whatever is loaded from --conf-file/--conf-dir (JSON/YAML files), if specified.
Use-case is a simple config file pre-processor for autonomous templating on service
startup with a minimal toolbox on top of jinja2, without huge dep-tree or any other
requirements and complexity, that is not scary to run from ExecStartPre= line as root.
Tool to temporarily modify (patch) a file - until reboot or for a specified
amount of time. Uses bind-mounts from tmpfs to make sure file will be reverted
to the original state eventually.
Useful to e.g. patch /etc/hosts with (pre-defined) stuff from LAN on a
laptop (so this changes will be reverted on reboot), or a notification filter
file for a short “busy!” time period (with a time limit, so it’ll auto-revert
after), or stuff like that.
Even though dst file is mounted with “-o ro” by default (there’s “-w” option to
disable that), linux doesn’t seem to care about that option and mounts the thing
as “rw” anyway, so “chmod a-w” gets run on temp file instead to prevent
accidental modification (that can be lost).
There’re also “-t” and “-m” flags to control timestamps during the whole
process.
Python script with various terminal input/output piping helpers and tools.
Has multiple modes for different use-cases, collected in same script mostly
because they’re pretty simple and not worth remembering separate ones.
out-paste
Disables terminal echo and outputs line-buffered stdin to stdout.
Example use-case can be grepping through huge multiline strings
(e.g. webpage source) pasted into terminal, i.e.:
% term-pipe | g -o '\<https?://[^"]\+'
[pasting page here via e.g. Shift+Insert won't cause any echo]
https://www.w3.org/TR/html4/loose.dtd
https://www.bugzilla.org/docs/3.4/en/html/bug_page.html
...
There are better tools for that particular use-case, but this solution
is universal wrt any possible input source.
shell-notify
Filter for screen/tmux/script output to send desktop notification
(using sd-bus lib) when shell prompt is detected on stdin, to enable when some
long job is running for example, so that you’d get notified immediately when it’s done.
Shell prompt detection is done via simple regexp, highly specific to my prompt(s)
and use-case(s), so might need tweaks in the code for different ones.
-l/–log option can be useful when doing that - will print all input lines
(with proper repr() wrapping), which can then be checked for desired patterns
and tested against new detection regexps as necessary.
Example use in tmux.conf:
bind-key r pipe-pane 'exec term-pipe shell-notify'
bind-key R pipe-pane
Should make “r” key (after prefix key) enable notifications and “shift+r” disable them.
Use “pipe-pane -o” to toggle this via same key instead.
“exec …” command there is passed to shell, so to debug errors after any
significant changes, something like “2>/tmp/errors.log” can be added at the end.
Check options of this subcommand for rate-limiting and some other tweaks.
s-run
Go over open screen/tmux sessions, find string there either print
how to attach there or send some command to run there.
Intended to find empty shell in a specific dir and automate pasting “do something”
long-running command from another app, to check on later in that shell/window.
Converts yaml files to an indented json, which is a bit more readable and
editable by hand than the usual compact one-liner serialization.
Due to yaml itself being json superset, can be used to convert json to
pretty-json as well.
Converts yaml/json files to a flat “key: value” lines.
Nested keys are flattened to a dot-separated “level1.level2.level3” keys,
replacing dots, spaces and colons there, to avoid confusing level separators
with the keys themselves.
Values are also processed to always be one-liners, handling long values
and empty lists/dicts and such in a readable manner too.
Output is intended for a human reader, to easily see value paths and such,
and definitely can’t be converted back to yaml or any kind of data safely.
Tool to normalize YAML files’ ordering/formatting and run “git diff | delta”
on those to produce nicely-colorized and useful diffs to inspect in the terminal.
Long YAMLs can be ordered and formatted in wildly different ways, and they often
are, when produced by different tools or edited manually, hence the need for
something to reformat them before running diff tools.
Script can be run on two dirs to compare all yml/yaml files in those recursively
(like “diff -r”), ignoring all other non-yaml files in there, as well as two
specific files.
Also has -f/–reformat option to pretty-print/normalize file(s) without diff,
which can be used to YAML-pretty-print JSON file(s) as well (incl. recursively,
with --fn-re override to match them). Requires python pygments module to be
installed for colorizing YAMLs printed to stdout with this option.
“git diff” can be used without “delta” if --no-delta option is set,
using its own colors (as per gitconfig), but output from delta is usually nicer,
has line numbers and highlights inline diffs.
Binaries and opts to both “git diff” and “delta” tools can be controlled
via env variables printed in -h/–help output.
Same thing as the common “head”, but works with \x00
(aka null char/byte , NUL, ␀, \0, \z, \000, \u0000, %00, ^@) delimeters.
Can be done with putting “tr” in the pipeline before and after “head”,
but this one is maybe a bit less fugly.
Allows replacing input null-bytes with newlines in the output
(–replace-with-newlines option) and vice-versa.
Common use-case is probably has something to do with filenames and xargs, e.g.:
% find -type f -print0 | shuf -z | hz -10 | xargs -0 some-cool-command
% ls -1 | hz -z | xargs -0 some-other-command
I have “h” as an alias for “head” in shells, so “head -z” (if there were such
option) would be aliased neatly to “hz”, hence the script name.
Defaults to reading ALL lines, not just arbitrary number (like 10, which is
default for regular “head”)!
“Log Interleaver And Colorizer” python script.
Reads lines from multiple files, ordering them by the specified field in the
output (default - first field, e.g. ISO8601 timestamp) and outputs each with
(optional) unique-filename-part prefix and unique (ansi-terminal, per-file) color.
Most useful for figuring out sequence of events from multiple timestamped logs.
To have safely-rotated logs with nice timestamps from any arbitrary command’s
output, something like stdbuf -oL <command-and-args> | svlogd -r _ -ttt <log-dir>
can be used.
Note “stdbuf” coreutils tool, used there to tweak output buffering, which usually
breaks such timestamps, and “svlogd” from runit suite (no deps, can be built separately).
See blog post about liac tool for more info.
Script to create “fat” HTML files, embedding all linked images
(as base64-encoded data-urls), stylesheets and js into them.
All src= and href= paths must be local (e.g. “js/script.js” or “/css/main.css”),
and will simply be treated as path components (stripping slashes on the left)
from html dir, nothing external (e.g. “//site.com/stuff.js”) will be fetched.
Doesn’t need anything but python, based on stdlib html.parser module.
Not optimized for huge amounts of embedded data, storing all the substitutions
in memory while it runs, and is unsafe to run on random html files, as it can
embed something sensitive (e.g. <img src="../.ssh/id_rsa">) - no extra checks there.
Use-case is to easily produce single-file webapps or pages to pass around (or
share somewhere), e.g. some d3-based interactive chart page or an html report
with a few embedded images.
Simple and dirty regexp + backreferences something-ML (SGML/HTML/XML) parser to
indent tags/values in a compact way without messing-up anything else in there.
I.e. non-closed tags are FINE, something like <@> doesn’t cause parser to
explode, etc.
Does not add any XML headers, does not mangle (or “canonize”) tags/attrs/values
in any way, except for stripping/adding those spaces.
Kinda like BeautifulSoup, except not limited to html and trivial enough so that
it can be trusted not to do anything unnecessary like stuff mentioned above.
For cases when xmllint --format fail and/or break such kinda-ML-but-not-XML files.
Script to add simple/distinctive base32-encoded content hash to filenames.
For example:
% hashnames -p *.jpg
wallpaper001.jpg -> wallpaper001.kw30e7cqytmmw.jpg
wallpaper893.jpg -> wallpaper893.vbf0t0qht4dd0.jpg
wallpaper895.jpg -> wallpaper895.q5mp0j95bxbdr.jpg
wallpaper898.jpg -> wallpaper898.c9g9yeb06pdbj.jpg
For collecting files with commonly-repeated names into some dir,
like random “wallpaper.jpg” or “image.jpg” images above from the internets.
Can also be used with -t/–tag option to update names for changed files,
which is handy in web-accessible dirs for changing URLs to invalidate caches.
Use -h/–help for info on more useful options.
Produces lower-entropy “human hash” phrase consisting of aspell english
dictionary words for input arg(s) or data on stdin.
It works by first calculating BLAKE2 hash of input string/data via libsodium,
and then encoding it using consistent word-alphabet, exactly like something
like base32 or base64 does.
Example:
% hhash -e AAAAC3NzaC1lZDI1NTE5AAAAIPh5/VmxDwgtJI0HiFBqZkbyV1I1YK+2DVjGjYydNp5o
allan avenues regrade windups flours
entropy-stats: word-count=5 dict-words=126643 word-bits=17.0 total-bits=84.8
Here -e is used to print entropy estimate for produced words.
Note that resulting entropy values can be fractional if word-alphabet ends up
being padded to map exactly to N bits (e.g. 17 bits above), so that words in it
can be repeated, hence not exactly 17 bits of distinct values.
Written in OCAML, linked against libsodium (for BLAKE2 hash function)
via small C glue code. Build it with:
% ocamlopt -o hhash -O2 unix.cmxa str.cmxa \
-cclib -lsodium -ccopt -Wl,--no-as-needed hhash.ml hhash.ml.c
% strip hhash
Caches dictionary into a ~/.cache/hhash.dict (-c option) on first run to produce
consistent results on this machine. Updating that dictionary will change outputs!
Trivial file/stream encryption tool using PyNaCl’s
crypto_secretstream_xchacha20poly1305 authenticated encryption API.
Key can be either specified on the command line for simplicity or read from a
file, and is always processed via scrypt, as it’s likely some short string.
Usage examples:
% crypt -ek my-secret-key secret.tar secret.tar.enc
% crypt -dk my-secret-key secret.tar.enc secret.tar.test
% crypt -ek @~/.secret.key <secret.tar >secret.tar.enc
Intended for an ad-hoc temporary encryption when transferring stuff via a usb
stick, making a temporary backup to a random untrusted disk or whatever.
Does not support any kind of appending/resuming or partial operation, which can
be bad if there’s a flipped bit anywhere in the encrypted data - decryption will
stop and throw error at that point.
Kernel sources/build/version management
Simple stateless script to update sources in /usr/src/linux to some (specified)
stable version.
Looks for “patch-X.Y.Z.xz” files (as provided on kernel.org) under
/usr/src/distfiles (configurable at the top of the script), or downloads them
there from kernel.org.
Does update (or rollback) by grabbing current patchset version from Makefile and
doing essentially patch -R < <patch-current> && patch < <patch-new> - i.e.
rolling-back the current patchset, then applying new patch.
Always does patch --dry-run first to make sure there will be no mess left
over by the tool and updates will be all-or-nothing.
In short, allows to run e.g. kernel-patch 3.14.22 to get 3.14.22 in
/usr/src/linux from any other clean 3.14.* version, or just
kernel-patch to have the latest 3.14 patchset.
Ad-hoc python script to check any random snippet with linux kernel
CONFIG_... values (e.g. “this is stuff you want to set” block on some wiki)
against kernel config file, current config in /proc/config.gz or such.
Reports what matches and what doesn’t to stdout, trivial regexp matching.
Script to remove older kernel versions (as installed by /sbin/installkernel)
from /boot or similar dir.
Always keeps version linked as “vmlinuz”, and prioritizes removal of older
patchset versions from each major one, and only then latest per-major patchset,
until free space goal (specified percentage, 20% by default) is met.
Also keeps specified number of last-to-remove versions, can prioritize cleanup
of “.old” verssion variants, keep config-* files… and other stuff (see --help).
Example:
# clean-boot --debug --dry-run -f 100
DEBUG:root:Preserved versions (linked version, its ".old" variant, --keep-min): 4
DEBUG:root: - 3.9.9.1 - System.map-3.9.9-fg.mf_master
DEBUG:root: - 3.9.9.1 - config-3.9.9-fg.mf_master
DEBUG:root: - 3.9.9.1 - vmlinuz-3.9.9-fg.mf_master
DEBUG:root: - 3.10.27.1 - vmlinuz-3.10.27-fg.mf_master
...
DEBUG:root: - 3.12.19.1 - System.map-3.12.19-fg.mf_master
DEBUG:root: - 3.12.20.1 - config-3.12.20-fg.mf_master
DEBUG:root: - 3.12.20.1 - System.map-3.12.20-fg.mf_master
DEBUG:root: - 3.12.20.1 - vmlinuz-3.12.20-fg.mf_master
DEBUG:root:Removing files for version (df: 58.9%): 3.2.0.1
DEBUG:root: - System.map-3.2.0-fg.mf_master
DEBUG:root: - config-3.2.0-fg.mf_master
DEBUG:root: - vmlinuz-3.2.0-fg.mf_master
DEBUG:root:Removing files for version (df: 58.9%): 3.2.1.0
... (removal of older patchsets for each major version, 3.2 - 3.12)
DEBUG:root:Removing files for version (df: 58.9%): 3.12.18.1
... (this was the last non-latest patchset-per-major)
DEBUG:root:Removing files for version (df: 58.9%): 3.2.16.1
... (removing latest patchset for each major version, starting from oldest - 3.2 here)
DEBUG:root:Removing files for version (df: 58.9%): 3.7.9.1
...
DEBUG:root:Removing files for version (df: 58.9%): 3.8.11.1
...
DEBUG:root:Finished (df: 58.9%, versions left: 4, versions removed: 66).
(“df” doesn’t rise here because of --dry-run, -f 100 =
“remove all non-preserved” - as df can’t really get to 100%)
Note how 3.2.0.1 (non-.old 3.2.0) gets removed first, then 3.2.1, 3.2.2, and so
on, but 3.2.16 (latest of 3.2.X) gets removed towards the very end, among other
“latest patchset for major” versions, except those that are preserved unconditionally
(listed at the top).
ZNC log helpers
Couple scripts to manage ZNC IRC bouncer logs -
archive, view, search, etc.
Tool to process ZNC chat logs, produced by “log” module (one enabled globally,
with default wildcards) and store them using following schema under some -d/–log-dir:
<net>/chat/<channel>__<yy>-<mm>.log.xz
<net>/priv/<nick>__<yy>-<mm>.log.xz
Where “priv” differs from “chat” in latter being prefixed by “#” or “&”.
With values from ZNC log paths: moddata/log/*/<net>/<chan/nick>/<yyyy-mm-dd>.log
Each ZNC-log line gets processed by regexp to add proper date, so that one
doesn’t have to use long timestamps in ZNC itself:
[HH:MM:SS] <nick> some msg -> [yy-mm-dd HH:MM:SS] <nick> some msg.
Latest (current day) logs are skipped.
New logs for each run are concatenated into a monthly .xz file.
Should be safe to stop at any time without data loss -
all resulting .xz’s get written to temporary files and renamed at the very end,
followed by unlinking of the source files, with nothing changed or updated in-place.
All temp files are produced in the destination dir, even with --dry-run,
and should be cleaned-up on any abort/exit/finish.
Idea is to have more convenient hierarchy and less files for easier shell
navigation/grepping (xzless/xzgrep), and without needing to worry about space
usage of uncompressed logs in the long run.
ZNC changed how it stores logs a few times over the years, and this tools
also helped maintain consistent storage schema across these.
Same as znc-log-aggregator above, but seeks/reads specific tail (“last n lines”)
or time range (with additional filtering by channel/nick and network) from all
current and aggregated (via that aggregator script) ZNC logs.
Mostly used to query/grep recent chat logs by approximate channel name from terminal easily.
Python script to list all currently active and non-transient systemd units,
so that these can be tracked as a “system state”,
and e.g. any deviations there detected/reported (simple diff can do it).
Gets unit info by parsing Dump() snapshot fetched via sd-bus API of libsystemd
(using ctypes to wrap it), which is same as e.g. “systemd-analyze dump” gets.
Has -m/–machines option to query state from all registered machines as well,
which requires root (for sd_bus_open_system_machine) due to current systemd limitations.
See Dashboard-for-… blog post for extended rationale,
though it’s probably obsolete otherwise since this thing was rewritten.
Trivial script to ping systemd watchdog and do some trivial actions in-between
to make sure os still works.
Wrote it after yet another silent non-crash, where linux kernel refuses to
create new pids (with some backtraces) and seem to hang on some fs ops, blocking
syslog/journal, but leaving most simple daemons running ok-ish for a while.
So this trivial script, tied into systemd-controlled watchdog timers, tries to
create pids every once in a while, with either hang or crash bubbling-up to
systemd (pid-1), which should reliably reboot/crash the system via hardware wdt.
Example watchdog.service:
[Service]
Type=notify
ExecStart=/usr/local/bin/systemd-watchdog -i30 -n \
-f /var/log/wdt-fail.log \
-x 'ip link' -x 'ip addr' -x 'ip ro' -x 'journalctl -an30'
WatchdogSec=60
TimeoutStartSec=15
Restart=on-failure
RestartSec=20
StartLimitInterval=10min
StartLimitBurst=5
StartLimitAction=reboot-force
[Install]
WantedBy=multi-user.target
(be sure to tweak timeouts and test without “reboot-force” first though,
e.g. pick RestartSec= for transient failures to not trigger StartLimitAction)
Can optionally get IP of (non-local) gateway to 1.1.1.1 (or any specified IPv4)
via libmnl (also used by iproute2, so always available) and check whether it
responds to fping probes, crashing if it does not - see -n/--check-net-gw option.
That’s mainly for remote systems which can become unreachable if kernel network
stack, local firewall, dhcp, ethernet or whatever other link fails (usually due
to some kind of local tinkering), ignoring more mundane internet failures.
To avoid reboot loops (in abscence of any networking), it might be a good idea
to only start script with this option manually (e.g. right before messing up
with the network, or on first successful access).
-f/--fail-log option is to log date/time of any failures for latest boot
and run -x/--fail-log-cmd command(s) on any python exceptions (note: kernel
hangs probably won’t cause these), logging their stdout/stderr there -
e.g. to dump network configuration info as in example above.
Useless without systemd and requires systemd python module, plus fping tool
if -n/--check-net-gw option is used.
Wrapper for systemd.resource control stuff to run commands in transient
scopes within pre-defined slices, as well as wait for these and list pids
within them easily.
Replacement for things like libcgroup, cgmanager and my earlier
cgroup-tools project, compatible with unified cgroup-v2 hierarchy
and working on top of systemd (use systemd.unified_cgroup_hierarchy
on cmdline, if non-default).
Resource limits for cgrc scopes should be defined via hierarchical slices like these:
# apps.slice
[Slice]
CPUWeight=30
IOWeight=30
MemoryHigh=5G
MemoryMax=8G
MemorySwapMax=1G
# apps-browser.slice
[Slice]
CPUWeight=30
IOWeight=30
MemoryHigh=3G
And then script can be used to start things there:
% cgrc apps-browser -- chromium
% cgrc -u ff apps-browser -- firefox --profile myprofile
Where e.g. last command would end up running something like this:
% systemd-run -q --user --scope --unit ff \
--slice apps-browser -- firefox --profile myprofile
Note that .scope cgroups are always transient (vanish after run), and only
.slice ones can be pre-defined with limits.
Both get started/stopped by systemd on as-needed basis.
Tool also allows to check or list pids within scopes/slices with -c/-l options
(to e.g. check if named scope already started or something running in a slice),
as well as waiting on these (-q option, can be used to queue/run commands in sequence)
and manipulating associated cgroup limits easily (-v option).
Run without any args/opts or with -h/–help to get more detailed usage info.
See also “backup” subsection.
ssh-keyscan, but outputting each key in every possible format.
Imagine you have an incoming IM message “hey, someone haxxors me, it says ‘ECDSA
key fingerprint is f5:e5:f9:b6:a4:6b:fd:b3:07:15:f6:d9:0c:f5:47:54’, what do?”,
this tool allows to dump any such fingerprint for a remote host, with:
% ssh-fingerprint congo.fg.nym
...
congo.fg.nym ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNo...zoU04g=
256 MD5:f5:e5:f9:b6:a4:6b:fd:b3:07:15:f6:d9:0c:f5:47:54 /tmp/.ssh_keyscan.key.kc3ur3C (ECDSA)
256 SHA256:lFLzFQR...2ZBmIgQi/w /tmp/.ssh_keyscan.key.kc3ur3C (ECDSA)
---- BEGIN SSH2 PUBLIC KEY ----
...
Only way I know how to get that
“f5:e5:f9:b6:a4:6b:fd:b3:07:15:f6:d9:0c:f5:47:54” secret-sauce is to either do
your own md5 + hexdigest on ssh-keyscan output (and not mess-up due to some
extra space or newline), or store one of the keys from there with first field
cut off into a file and run ssh-keygen -l -E md5 -f key.pub.
Note how “intuitive” it is to confirm something that ssh prints (and it prints
only that md5-fp thing!) for every new host you connect to with just openssh.
With this command, just running it on the remote host - presumably from diff
location, or even localhost - should give (hopefully) any possible gibberish
permutation that openssh (or something else) may decide to throw at you.
Python script to extract raw private key string from ed25519 ssh keys.
Main purpose is easy backup of ssh private keys and derivation of new
secrets from these for other purposes.
For example:
% ssh-keygen -t ed25519 -f test-key
...
% cat test-key
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACDaKUyc/3dnDL+FS4/32JFsF88oQoYb2lU0QYtLgOx+yAAAAJi1Bt0atQbd
GgAAAAtzc2gtZWQyNTUxOQAAACDaKUyc/3dnDL+FS4/32JFsF88oQoYb2lU0QYtLgOx+yA
AAAEAc5IRaYYm2Ss4E65MYY4VewwiwyqWdBNYAZxEhZe9GpNopTJz/d2cMv4VLj/fYkWwX
zyhChhvaVTRBi0uA7H7IAAAAE2ZyYWdnb2RAbWFsZWRpY3Rpb24BAg==
-----END OPENSSH PRIVATE KEY-----
% ssh-keyparse test-key
HOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=
That one line at the end contains 32-byte ed25519 seed (with urlsafe-base64
encoding) - “secret key” - all the necessary info to restore the blob above,
without extra openssh wrapping (as per PROTOCOL.key).
Original OpenSSH format (as produced by ssh-keygen) stores “magic string”,
ciphername (“none”), kdfname (“none”), kdfoptions (empty string), public key and
index for that, two “checkint” numbers, seed + public key string, comment and a
bunch of extra padding at the end. All string values there are length-prefixed,
so take extra 4 bytes, even when empty.
Gist is that it’s a ton of stuff that’s not the actual key, which ssh-keyparse
extracts.
To restore key from seed, use -d/--patch-key option on any existing ed25519 key,
e.g. ssh-keygen -t ed25519 -N '' -f test-key && ssh-keyparse -d <seed> test-key
If key is encrypted with passphrase, ssh-keygen -p will be run on a
temporary copy of it to decrypt, with a big warning in case it’s not desirable.
There’s also an option (--pbkdf2) to run the thing through PBKDF2 (tunable via
--pbkdf2-opts) and various output encodings available:
% ssh-keyparse test-key # default is urlsafe-base64 encoding
HOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=
% ssh-keyparse test-key --hex
1ce4845a6189b64ace04eb931863855ec308b0caa59d04d60067112165ef46a4
% ssh-keyparse test-key --base32
3KJ8-8PK1-H6V4-NKG4-XE9H-GRW5-BV1G-HC6A-MPEG-9NG0-CW8J-2SFF-8TJ0-e
% ssh-keyparse test-key --base32-nodashes
3KJ88PK1H6V4NKG4XE9HGRW5BV1GHC6AMPEG9NG0CW8J2SFF8TJ0e
% ssh-keyparse test-key --raw >test-key.bin
With encoding like --base32 (Douglas Crockford’s human-oriented Base32,
last digit/lowercase-letter there is a checksum), it’s easy to even read the
thing over some voice channel, if necessary.
Bash script to generate (init) ssh key (via ssh-keygen) without asking about
various legacy and uninteresting options and safe against replacing existing
keys.
I.e. don’t ever want RSA, ECDSA or such nonsense (Ed25519 is the norm), don’t
need passwords for 99.999% of the keys, don’t care about any of the ssh-keygen
output, don’t need any interactivity, but do care about silently overwriting
existing key and want the thing to create parent dirs properly (which -f fails
to do).
Has -m option to init key for an nspawn container under /var/lib/machines
(e.g. ssh-key-init -m mymachine) and -r option to replace any existing keys.
Sets uid/gid of the parent path for all new ones and -m700.
Script to keep persistent, unique and reasonably responsive ssh tunnels.
Mostly just a bash wrapper with collection of options for such use-case.
I.e. to run ssh-tunnel -ti 60 2223:nexthop:22 user@host -p2222 instead of
some manual loop (re-)connecting every 60s in the background using something like:
ssh \
-oControlPath=none -oControlMaster=no \
-oConnectTimeout=5 -oServerAliveInterval=3 -oServerAliveCountMax=5 \
-oPasswordAuthentication=no -oNumberOfPasswordPrompts=0 \
-oBatchMode=yes -oExitOnForwardFailure=yes -TnNqy \
-p2222 -L 2223:nexthop:22 user@host
Which are all pretty much required for proper background tunnel operation.
Has opts for reverse-tunnels and using tping tool instead of ssh/sleep loop.
Keeps pidfiles in /tmp and allows to kill running tunnel-script via same command with -k/kill appended.
Python/asyncio scripts to establish multiple ssh reverse-port-forwarding
(“ssh -R”) connections to the same tunnel-server from mutliple hosts using same
exact configuration on each.
Normally, first client host will bind the “ssh -R” listening port and all others
will fail, but these two scripts negotiate unique port within specified range to
each host, so there are no clashes and all tunnels work fine.
Tunnel server also stores allocated ports in a db file, so that each client gets
more-or-less persistent listening port.
Each client negotiates port before exec’ing “ssh -R” command, identifying itself
via --ident-* string (derived from /etc/machine-id by default), and both
client/server need to use same -s/--auth-secret to create/validate MACs in each packet.
Note that all --auth-secret is used for is literally handing-out sequential
numbers, and isn’t expected to be strong protection against anything, unlike ssh
auth that should come after that.
Same thing as ssh-reverse-mux-* scripts above, but for negotiating WireGuard
tunnels, with persistent host tunnel IPs tracked via --ident-* strings with
simple auth via MACs on UDP packets derived from symmetric -s/--auth-secret.
Client identity, wg port, public key and tunnel IPs are sent in the clear with
relatively weak authentication (hmac of -s/--auth-secret string), but wg server
is also authenticated by pre-shared public key (and --wg-psk, if specified).
Such setup is roughly equivalent to a password-protected (--auth-secret) public network.
Runs “wg set” commands to update configuration, which need privileges,
but can be wrapped in sudo or suid/caps via --wg-cmd to avoid root in the
rest of the script.
Does not touch or handle WireGuard private keys in any way by itself,
and probably should not have direct access to these
(though note that unrestricted access to “wg” command can reveal them anyway).
Example systemd unit for server:
# wg.service + auth.secret psk.secret key.secret
# useradd -s /usr/bin/nologin wg && mkdir -m700 ~wg && chown wg: ~wg
# cd ~wg && cp /usr/bin/wg . && chown root:wg wg && chmod 4110 wg
[Unit]
Wants=network.target
After=network.target
[Service]
Type=exec
User=wg
WorkingDirectory=~
Restart=always
RestartSec=60
StandardInput=file:/home/wg/auth.secret
StandardOutput=journal
ExecStartPre=+sh -c 'ip link add wg type wireguard 2>/dev/null; \
ip addr add 10.123.0.1/24 dev wg 2>/dev/null; ip link set wg up'
ExecStartPre=+wg set wg listen-port 1500 private-key key.secret
ExecStart=wg-mux-server --mux-port=1501 --wg-port=1500 \
--wg-net=10.123.0.0/24 --wg-cmd=./wg --wg-psk=psk.secret
[Install]
WantedBy=multi-user.target
Client:
# wg.service + auth.secret psk.secret
# useradd -s /usr/bin/nologin wg && mkdir -m700 ~wg && chown wg: ~wg
# cd ~wg && cp /usr/bin/wg . && chown root:wg wg && chmod 4110 wg
# cd ~wg && cp /usr/bin/ip . && chown root:wg ip && chmod 4110 ip
[Unit]
Wants=network.target
After=network.target
[Service]
Type=exec
User=wg
WorkingDirectory=~
Restart=always
RestartSec=10
StandardInput=file:/home/wg/auth.secret
StandardOutput=journal
ExecStartPre=+sh -c '[ -e key.secret ] || { umask 077; wg genkey >key.secret; }
ExecStartPre=+sh -c '[ -e key.public ] || wg pubkey <key.secret >key.public
ExecStartPre=+sh -c 'ip link add wg type wireguard 2>/dev/null; ip link set wg up'
ExecStartPre=+wg set wg private-key key.secret
ExecStart=wg-mux-client \
20.88.203.92:1501 BcOn/q9D5zcqK0hrWmXGQHtaEKGGf6g5nTxZUZ0P4HY= key.public \
--ident-rpi --wg-net=10.123.0.0/24 --wg-cmd=./wg --ip-cmd=./ip --wg-psk=psk.secret \
--ping-cmd='ping -q -w15 -c3 -i3 10.123.0.1' --ping-silent
[Install]
WantedBy=multi-user.target
When enabled, these should be enough to setup reliable tunnel up on client boot,
and then keep it alive from there indefinitely (via --ping-cmd + systemd restart).
Explicit iface/IP init in these units can be replaced by systemd-networkd
.netdev + .network stuff, as it supports wireguard configuration there.
Bash script to list or kill users’ sshd pids, created for “ssh -R” tunnels, that
don’t have a listening socket associated with them or don’t show ssh protocol
greeting (e.g. “SSH-2.0-OpenSSH_7.4”) there.
These seem to occur when ssh client suddenly dies and reconnects to create new
tunnel - old pid can still hog listening socket (even though there’s nothing on
the other end), but new pid won’t exit and hang around uselessly.
Solution is to a) check for sshd pids that don’t have listenings socket, and
b) connect to sshd pids’ sockets and see if anything responds there, killing
both non-listening and unresponsive pids.
Only picks sshd pids for users with specific prefix, e.g. “tun-” by default, to
be sure not to kill anything useful (i.e. anything that’s not for “ssh -R”).
Uses ps, ss, gawk and ncat (comes with nmap), only prints pids by default
(without -k/--kill option).
Also has -s/--cleanup-sessions option to remove all “abandoned” login sessions
(think loginctl) for user with specified prefix, i.e. any leftover stuff after
killing those useless ssh pids.
See also: autossh and such.
Python wrapper for mosh-server binary to do UDP hole punching through
local NAT setup before starting it.
Comes with mosh-nat-bind.c source for LD_PRELOAD=./mnb.so lib to force
mosh-client on the other side to use specific local port that was used in
“mosh-nat”.
Example usage (server at 84.217.173.225, client at 74.59.38.152):
server% ./mosh-nat 74.59.38.152
mosh-client command:
MNB_PORT=34730 LD_PRELOAD=./mnb.so
MOSH_KEY=rYt2QFJapgKN5GUqKJH2NQ mosh-client <server-addr> 34730
client% MNB_PORT=34730 LD_PRELOAD=./mnb.so \
MOSH_KEY=rYt2QFJapgKN5GUqKJH2NQ mosh-client 84.217.173.225 34730
Notes:
- mnb.so is mosh-nat-bind.c lib. Check its header for command to build it.
- Both mnb.so and mosh-nat only work with IPv4, IPv6 shouldn’t use NAT anyway.
- Should only work like that when NAT on either side doesn’t rewrite src ports.
- 34730 is default for
-c/--client-port and -s/--server-port opts.
- Started mosh-server waits for 60s (default) for mosh-client to connect.
- Continous operation relies on mosh keepalive packets without interruption.
- No roaming of any kind is possible here.
- New MOSH_KEY is generated by mosh-server on every run.
Useful for direct and fast connection when there’s some other means
of access available already, e.g. ssh through some slow/indirect tunnel
or port forwarding setup.
For more hands-off hole-punching, similar approach to what pwnat does can be used.
See mobile-shell/mosh#623 for more info and links on such feature implemented in mosh directly.
Source for LD_PRELOAD lib is based on https://github.com/yongboy/bindp/
Python (asyncio) tool to try connecting to specified TCP port until connection
can be established, then just exit, i.e. to wait until some remote port is accessible.
Can be used to wait for host to reboot before trying to ssh into it, e.g.:
% tping myhost && ssh root@myhost
(default -p/--port is 22 - ssh, see also -s/--ssh option)
Tries establishing new connection (forcing new SYN, IPv4/IPv6 should both work)
every -r/--retry-delay seconds (default: 1), only discarding (closing) “in
progress” connections after -t/--timeout seconds (default: 3), essentially
keeping rotating pool of establishing connections until one of them succeeds.
This means that with e.g. -r1 -t5 there will be 5 establishing connections
(to account for slow-to-respond remote hosts) rotating every second, so ratio of
these delays shouldn’t be too high to avoid spawning too many connections.
Host/port names specified on the command line are resolved synchronously on
script startup (same as with e.g. “ping” tool), so it can’t be used to wait
until hostname resolves, only for connection itself.
Above example can also be shortened via -s/--ssh option, e.g.:
% tping -s myhost 1234
% tping -s root@myhost:1234 # same thing as above
% tping -s -p1234 myhost # same thing as above
Will exec ssh -p1234 root@myhost immediately after successful tcp connection.
Uses python stdlib stuff, namely asyncio, to juggle multiple
README 内容过长，已截取部分内容。完整内容请前往 GitHub 仓库查看。
评论