binaryanalysis-ng
Binary Analysis Next Generation (BANG)
BANG is a framework for processing binary files (like firmware). It consists of
an unpacker that recursively unpacks and classifies/labels files and separate
analysis programs that work on the results of the unpacker.
Some intended uses:
- provenance detection (“what is inside this file”)
- security scans (“are there any known security risks associated with this file”)
Requirements
The recommended way is to use Nix, run
nix-shell to load all the dependencies for the unpacker,
nix-shell maintenance.nix for the maintenance scripts,
nix-shell analysis.nix for the analysis scripts and nix-shell scraper.nix
for the firmware scraper.
nix will make sure that everything is downloaded and installed to run BANG.
Other distributions without Nix
Note: this list is vastly incomplete. It is highly recommended to use Nix.
- a recent Linux distribution (Fedora 36 or higher, or equivalent)
- Python 3.10.x or higher
- pillow (possibly named python3-pillow), a drop in replacement for PIL ( http://python-pillow.github.io/ )
- GNU binutils (for ‘ar’)
- squashfs-tools (for ‘unsquashfs’)
- cabextract
- 7z
- e2tools (for ‘e2ls’ and ‘e2cp’)
- zstd
- python-lz4 (possibly named python3-lz4)
- qemu-img (for VMDK files)
- psycopg2 (possibly named python3-psycopg2)
- python-snappy (possibly named python3-snappy)
- python-tlsh (possibly named python3-tlsh)
- dockerfile-parse (possibly named python3-dockerfile-parse)
- openssl
- rzip
- lzop
- defusedxml (possibly named python3-defusedxml)
- icalendar (possibly named python3-icalendar)
- pyyaml (possibly named python3-pyyaml)
- ncompress
- util-linux (for ‘fsck.cramfs’)
- lz4 (for ‘lz4c’)
- bzip3
and many others (see shell.nix, maintenance.nix and analysis.nix for a
full list).
You will also need to install the Kaitai Struct compiler. This is described in
the file doc/kaitai-struct.md.
Additionally install sasquatch:
https://github.com/onekey-sec/sasquatch
Supported hardware
It is assumed that BANG is run on little endian hardware (such as x86 or x86-64).
Verified unsupported distributions
- Fedora 35 and earlier
- Ubuntu 16.04 and lower (Python version too old)
This doesn’t mean that newer versions of Ubuntu are supported, they just
haven’t been tested.
Docker container (recently untested, assume broken)
docker image build -t bang .
docker container run --rm -it bang
or from the src directory, type
make dockerbuild
Supported file types
The following files can be unpacked, or verified, including carving from a
larger file, unless stated otherwise.
- WebP
- WAV
- ANI
- gzip
- LZMA
- XZ
- timezone files
- tar
- Apple Double encoded files
- ICC (colour profile)
- ZIP (store, deflate, bzip2, but lzma needs some more testing), also JAR, APK (possible with extra Android signing bytes) and other ZIP-based formats, dangling entries without a valid central directory
- U-Boot image
- XAR (no compression, gzip, bzip2, XZ, LZMA)
- ISO9660 (including RockRidge and zisofs)
- lzip
- WOFF (Web Open Font Format)
- TrueType fonts/sfnt-housed fonts
- OpenType fonts
- Vim swap files (whole file only)
- Android sparse data image (regular and Brotli compression, no bsdiff/imgdiff)
- Android backup files
- ICO (MS Windows icons)
- Chrome PAK (version 4 & 5, only if offset starts at 0)
- GNU message catalog
- RPM (gzip, XZ, bzip2, LZMA, zstd, not: delta RPM)
- AIFF/AIFF-C
- terminfo (little endian, regular and extended storage format, not
extended number format)
- AU (Sun/NeXT audio)
- JFFS2 (uncompressed, zlib, rtime, lzo, LZMA from OpenWrt)
- CPIO (various flavours, little endian)
- Sun Raster files (standard type only)
- Intel Hex (text files only)
- Motorola SREC (text files only)
- Quicktime
- Android sparse image files
- Java class file
- Android Dex/Odex (not OAT, just carving)
- ELF
- SWF (uncompressed, zlib, LZMA)
- Android resource files (table type, but possibly not all types, binary XML)
- base64/32/16 (whole file)
- FLV (Macromedia Flash Video)
- Git index files
- JSON (whole file)
- D-Link ROMFS
- bzip2
- GIF (needs PIL)
- JPEG (needs PIL)
- Microsoft Cabinet archives (needs cabextract)
- RZIP (requires rzip)
- 7z (requires external tools), single frame(?)
- Windows Compiled HTML Help (needs external tools, version 3
only)
- Windows Imaging file format (needs external tools, single
image only)
- ext2/3/4 (missing: symbolic link support)
- zstd (needs zstd package)
- SGI image files (needs PIL)
- Apple Icon Image (needs PIL)
- LZ4 (requires LZ4 Python bindings), LZ4 legacy (requires ‘lz4c’)
- VMware VMDK (needs qemu-img, whole file only)
- QEMU qcow2 (needs qemu-img, whole file only)
- VirtualBox VDI (needs qemu-img, whole file only,
Oracle flavour only)
- XML (whole file)
- Snappy framing2 format (needs python-snappy)
- various certificates (PEM, private key, etc., needs openssl)
- lzop
- PNG/APNG (needs PIL)
- ar/deb (needs binutils)
- squashfs (using squashfs-tools), vendor specific exotic variants
(using sasquatch)
- BMP (needs PIL)
- PDF (simple verification, no object streams, incremental updates
at end of the file)
- GIMP brush (needs PIL)
- ZIM (Wikipedia archive format)
- MIDI
- Android tzdata
- Java key store (version 2 only)
- XG3D (proprietary file format from 3D Studio Max, labeling only)
- ACDB (audio callibration database, proprietary file format from Qualcomm, labeling only)
- Microsoft DirectDraw Surface (structure checks and very limited sanity checking)
- Khronos KTX files (version 1)
- Android verified boot image
- SQLite 3
- Linux flattened device tree
- Broadcom TRX
- Photoshop PSD (raw bytes and RLE encoding only)
- minidump files
- PPM files (‘raw’ PPM only)
- PGM files (‘raw’ PGM only)
- PBM files (‘raw’ PBM only)
- Android bootloader image for Qualcomm Snapdragon (MSM)
- Android bootloader image (also a Little Kernel based variant)
- Android bootloader image for Huawei devices
- FAT16 file systems (8.3 file names)
- Coreboot images
- Minix V1 file system (Linux variant)
- Unix compress (needs ‘uncompress’), only if end
of the file is compress’d data
- romfs
- cramfs (version 2 only)
- nb0 Android updates
- Quake PAK files
- Doom WAD files (IWAD only)
- Ambarella firmware files
- Ambarella romfs (used in Ambarella firmware files)
- bFLT
- UBI, fastmap not supported
- GRUB2 font files
- BitTorrent files (subset)
- pcapng (carving, structural checks, little endian only)
- pcap (carving, structural checks)
- serialized Java (block data only, carving, structural checks)
- mapsforge map files (very basic structural checks)
- Parrot PLF files
- PFS file system
- YAFFS2 (including inband tags)
- Qualcomm QCDT files
- Chrome extensions (.crx)
- Windows shell link file (.lnk)
- PCF fonts (that actually follow the specification, little endian only)
- DS_Store
- libminikin hyb text layout format
- Mozilla ARchive (.mar)
- OpenFst (subset, identification only)
- SELinux file context
- Ogg
- Allwinner images
- DFU (Device Firmware Upgrade)
- Key Character Map binary files
- USB Flashing Format (UF2)
- Android VDEX (identification only)
- SEAMA firmware files
- LLVM IR wrapper format (identification only)
- OpenWrt LXL firmware header
- Mediatek BootROM (header only)
- Rockchip RKFW and RKAF
- systemd journal files
- Rockchip rkboot
- Python pickle
- glibc utmp/wtmp
- Android vendor boot
- Android FBPK
- Samsung Tzar
- Qualcomm aboot (version 3 only, no unified boot)
- Rockchip resource files
- Socionext Milbeaut firmware files
- zchunk
- ubifs
- Performance Co-Pilot metadata files
- data URI (PNG, GIF, JPEG, WEBP, certain fonts, SVG, PDF, JSON and octet-stream objects)
- DHTB signed files
- Android AAPT2 container format
- Android update image (version 2 only, full OTA image only)
- Qt resource files (
.rcc)
- glibc locale archive file detection
- Sunplus BRN firmware
- xo65 object files
- DOS MZ, plus COFF for MS-DOS, DJGPP go32 DOS extender
- WinHelp (older formats only)
- PEF (Preferred Executable Format)
- Nano app header (Android)
- WebAssembly binaries
- Android super images
- Qualcomm QTI Chromatix (structural checks only)
- Mediatek images (including logo.bin)
- Android DTO
- Portable Image Format
- ResPack.cfg (seen in CPB firmware format)
- LOD (RDA/Coolsand phone firmware format)
- MediaTek secure ROM(?) info
- lrzip
- SerpentOS .stone package file
- systemd hwdb.bin
- Qt Translation files
- Reolink firmware
- Xiaomi firmware
- HP BDL firmware
- Netgear .chk firmware files
- Instar BNEG firmware files
- Huawei HWNP firmware files
- Reolink ‘logo’ file
- FLS firmware files (IP cameras)
- TP-Link TX6610v4 firmware
- Granite Devices firmware v300
- erofs
- bzip3
- PX4 autopilot firmware files
- SSH known hosts files (whole file)
- Unix passwd files (whole file)
- Unix group files (whole file)
- Unix shadow files (whole file)
- Samba password files
- Linux fstab files
- Java/Android MANIFEST.MF files (whole file)
- Linux kernel configuration files (whole file)
- pkg-config files
- iCalendar (RFC 5545) files (whole file only)
- Windows INI files (text only)
- Linux Software Map files
- exFAT (“no FAT chains” only)
- Subversion hash files (wcprops, all-wcprops, etc.)
- Python PKG-INFO files (whole file)
- Dockerfile files (whole file)
- TRANS.TBL files
- BTF and BTF.ext sections (BPF related)
- NibArchive
- EDID
- ld.so.conf files (subset, no include statements)
- old TP-Link firmware files
- certain VxWorks memfs file systems
- Realtek bootloader (subset of files)
- Linux kernel x86 images
- TP-Link minifs
- ANJVision IP camera firmware files
- Linux
securetty files (whole file)
- EOT fonts
- crontab files (subset)
- modules.dep files
- host.conf files
- lld2d.conf files (from old Realtek SDKs)
- inittab files
- /etc/services files
- /etc/hosts files
- pppoe.conf files
Getting started
This section assumes that you are using Nix to install all dependencies.
git clone this repository- run
nix-shell to install all the dependencies and start a complete environment
cd src- run
make to build all the Kaitai Struct parsers. Please note: this might take
a while!
To unpack a file run (from the src directory):
$ python3 -m bang.cli scan -u /path/to/unpack/directory /path/to/binary
This will output a directory with inside a number of files and directories.
The output directory can serve as input to the analysis scripts (and some
knowledgebase scripts).
To process each file in a directory run (from the src directory):
$ python3 -m bang.cli scan-directory -u /path/to/unpack/directory /path/to/directory/with/binaries
This will create a directory with search results for each file that is scanned,
with the same name as the file that is scanned, using the same structure as if
the file was scanned in regular mode, not directory mode.
License
GNU General Public License, version 3 (GPL-3.0), except where noted.
The code for verifying and labeling Android Verified Boot images was heavily
inspired by code from Android (avbtool) found at:
https://android.googlesource.com/platform/external/avb/+/refs/heads/master/avbtool.py
The original license for avbtool:
Copyright 2016, The Android Open Source Project
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use, copy,
modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
The code for rtime decompression was copied from:
https://github.com/sviehb/jefferson/blob/master/src/jefferson/rtime.py
The original license for jefferson:
The MIT License (MIT)
Copyright (c) 2015 Stefan Viehböck
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Developing
The recommended coding style is described in PEP 8:
https://www.python.org/dev/peps/pep-0008/
It is recommended to run PEP 8 verification tools, for example
python3-flake8 (on Fedora).
Another tool that is highly recommended is pylint.
Acknowledgement
This project has received funding from the European Union’s Horizon 2020
research and innovation programme within the framework of the NGI-POINTER
Project funded under grant agreement No. 871528.
评论