An Empirical Study on ARM Disassembly Tools

This is the repository for paper "An Empirical Study on ARM Disassembly Tools" accepted to ISSTA 2020 and the paper "A Comprehensive Study on ARM Disassembly Tools" accepted to TSE 2022.

Tools

We evaluate eight different disassembly tools in three different versions across around 2 years. They are

• angr (version 8.19.4.5), (version 9.0.4663), (version 9.0.10534)
• BAP (version 1.6.0), (version 2.1.0), (version 2.3.0)
• Objdump (version 2.30), (version 2.34), (version 2.37)
• Ghidra (version 9.0.4), (version 9.1.2), (version 10.0.4)
• Radare2 (version 3.6.0), (version 5.0.0), (version 5.4.2)
• Binary Ninja (version 1.1.1470), (version 2.2.2487), (version 2.4.2846)
• Hopper (version 4.5.13), (version 4.5.29), (version 5.1.3)
• IDA Pro (version 7.3), (version 7.5), (version 7.6)

Each tool has different method to extract the disassembly result. We read the manual carefully and write a script for each tool to extract the disassembly result. The detail script are listed in Adapters. The code of evaluating the efficiency of different tools are also integrated into the adapters of each tool.

Dataset

Dataset contains the dataset we used in our experiments. However, due to the licensing issues, we cannot share the binaries compiled from SPEC CPU® 2006 directly.

You can take the following tips to build the SPEC CPU 2006 by yourselves. Feel free if you have any questions.

• Prepare the SPEC CPU® 2006
• Install SPEC CPU® 2006 by following the documentation
• I provided two template configuration files (i.e., clang.cfg and gcc.cfg) for GCC and Clang, respectively. You can change the configuration files for different compiling options.
• Use the command runspec --config=/path/to/config/gcc.cfg --action=build --rebuild --tune=base binary name to build every single binary.
• You can glue all of them with your own python or shell script.

Ground Truth

truth.py is the file to extract the ground truth from a binary with debugging information.

Citation

If you use the related script, dataset or the insights we observed in our paper. Please considering cite our paper.

@inproceedings{10.1145/3395363.3397377,
author = {Jiang, Muhui and Zhou, Yajin and Luo, Xiapu and Wang, Ruoyu and Liu, Yang and Ren, Kui},
title = {An Empirical Study on ARM Disassembly Tools},
year = {2020},
isbn = {9781450380089},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3395363.3397377},
doi = {10.1145/3395363.3397377},
booktitle = {Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis},
pages = {401–414},
numpages = {14},
keywords = {Empirical Study, Disassembly Tools, ARM Architecture},
location = {Virtual Event, USA},
series = {ISSTA 2020}
}

@article{jiang2022comprehensive,
  title={A Comprehensive Study on ARM Disassembly Tools},
  author={Jiang, Muhui and Dai, Qinming and Zhang, Wenlong and Chang, Rui and Zhou, Yajin and Luo, Xiapu and Wang, Ruoyu and Liu, Yang and Ren, Kui},
  journal={IEEE Transactions on Software Engineering},
  year={2022},
  publisher={IEEE}
}