Using the DIFF tool to verify that 2 runs of Scarab sent the exact same ops from the PIN tool to Scarab

Whenever we update Scarab, we like to ensure that no new bugs have been introduced. Most of the time this is done by running a set of benchmarks on both the unmodified version of Scarab and the updated version of Scarab and looking for IPC differences. If an IPC difference is detected, we need to look deeper into the two runs (the before and the after run) to see exactly what changed after the update. If the change is not expected, then we found a bug.

To facilitate this process, we developed a diff tool that compares all relevant fields of OPs fetched by Scarab across the two runs.

Here are the steps for using the comparison script: 1- Run your two Scarab version with the following flags: --fetch_off_path_ops 0 --debug_inst_start <START> --debug_inst_stop <STOP> --debug_op_fields 1 2- Run the script: utils/diff_scarab_ops.py path_to_scarab_stdout_1 path_to_scarab_stdout_2

The script will first print a count of differences based on the general category of the line that was different. Then it will print all individual lines that are different.

Here is an example output of the utils/diff_scarab_ops.py script:

mem_type: 31609
---- 2 8: 5792
---- 1 8: 6333
---- 1 4: 15516
---- 2 4: 3968
src: 5774
---- 8 8: 5774
========= diffs for mem_type 2 8 ========
34: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe590 8 --- 34: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe580 8
50: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe588 8 --- 50: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe578 8
66: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe580 8 --- 66: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe570 8
82: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe578 8 --- 82: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe568 8
98: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe570 8 --- 98: DEBUG_OP_FIELDS mem_type addr mem_size: 2 7fffffffe560 8

In each diff, the script prints a histogram of all the diffs (first part) then the line numbers of every diff from the original file (second part). For example, in the log above, we can see that there are 31609 differences that are mem_type. Of those, we can see that 5792 of them are differences where the mem_type was 2 and the mem_size was 8. Then, the second part of the log enumerates all of the diff for mem_type 2 8. The first diff corresponds to line 34 in both files (the line number may be off by 1 in the original file). Now you can open the original scarab stdout files, go to line 34 in both, and look at all the information about the instruction in the two versions. This will give you the PC of the instructions. Finally, look at the objdump of the binary to see which one is correct. For example, the stdout told us that the PC of the instruction at line 34 was 41bd46. We then looked up that PC in the objdump and found that the instructions was: 41bd46: 41 0f b7 84 44 bc 14 movzwl 0x14bc(%r12,%rax,2),%eax.

We found the script really helpful and recommend using it for testing.