How to use UnBlob to extract files from containers of any format

UnBlob is completely open source and free, and also provides a command-line interface. In addition, this tool can be used as a Python library. These features make UnBlob the perfect tool for file/data extraction, analysis, and reverse engineering of firmware images.

Tool Features

1. High accuracy: It supports using custom rules to identify the starting offset of data blocks and automatically calculates the ending offset of data blocks according to the data format standard;
2. High security: UnBlob can be executed without using advanced permissions and automatically upgrades the dependent third-party components to the latest version;
3. High scalability: UnBlob provides an API, allowing researchers to write custom format processors and data extractors themselves;
4. Fast execution speed: UnBlob runs very fast by default and uses a multi-threading mechanism;

Technical Features

1. UnBlob is developed based on the Python language;
2. To quickly search for code patterns in files, we useHyperscan;
3. To extract identified formats, various types of data extraction tools are used;
4. For ELF analysis, we useLIEFand itsPythonbinding;
5. For CPU-intensive tasks (such as entropy calculation), we useRustto speed up;
6. To provide a more beautiful command-line interface, we useClickLibrary;
7. To provide structured logging, we usestructlogLibrary;

Tool download & installation

This tool is developed based on Python, so we first need to install and configure the Python environment locally. Next, use the following command to clone the source code of this project to the local machine:

git clone https://github.com/onekey-sec/unblob.git

Then use Poetry to install the required dependency components:

cd unblob

poetry install --no-dev

Next, install the required data extractors according to the corresponding operating system:

# Ubuntu 22.04

sudo apt install e2fsprogs p7zip-full unar zlib1g-dev liblzo2-dev lzop lziprecover img2simg libhyperscan-dev zstd

If you need to use squashfs, you can install sasquatch:

curl -L -o sasquatch_1.0_amd64.deb https://github.com/onekey-sec/sasquatch/releases/download/sasquatch-v1.0/sasquatch_1.0_amd64.deb

sudo dpkg -i sasquatch_1.0_amd64.deb

rm sasquatch_1.0_amd64.deb

Finally, please check if all the dependent components have been installed:

Tool Usage

UnBlob provides an easy-to-use command-line interface where we can directly pass the file to be extracted:

$ unblob alpine-minirootfs-3.16.1-x86_64.tar.gz

2022-07-30 06:33.07 [info    ] Start processing file file=openwrt-21.02.2-x86-64-generic-ext4-combined.img.gz pid=7092

This tool will create a new directory named with '_extract':

$ ls -l

total 2656

drwxrwxr-x 3 walkman walkman    4096 Jul 30 08:43 alpine-minirootfs-3.16.1-x86_64.tar.gz_extract

-rw-r--r-- 1 walkman walkman 2711958 Jul 30 08:43 alpine-minirootfs-3.16.1-x86_64.tar.gz

At this time, the tool will recursively extract all known file formats until it reaches the specified recursive path level (default is 10 levels):

License Agreement

The development and release of this project followMITOpen Source License Agreement.

Project Address

UnBlob：【GitHub Gateway】

Reference Materials

https://unblob.org/
https://onekey.com/
https://www.python.org/
https://www.hyperscan.io/
https://lief-project.github.io/
https://pypi.org/project/lief/
https://www.rust-lang.org/
https://click.palletsprojects.com/
https://www.structlog.org/