Fix load time calculation error in llama_bench

[Septa2112]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Fix #9286

Fix load time calculation error in llama_bench when running multi-benchmark with same model.

From the test results, we can see that the modified load_time remains basically the same and has almost no impact on the benchmark results.

Test Results:

load_time Testing

• Command: ./build/bin/llama-bench -m ../gguf_models/llama-2-7b.Q2_K.gguf -n 1,2,4,8,16,32 -p 0 -v

Original

llama_perf_context_print:        load time =     172.46 ms
llama_perf_context_print:        load time =     596.39 ms
llama_perf_context_print:        load time =    1362.16 ms
llama_perf_context_print:        load time =    2820.51 ms
llama_perf_context_print:        load time =    5678.16 ms
llama_perf_context_print:        load time =   11295.91 ms

After modification

llama_perf_context_print:        load time =     171.97 ms
llama_perf_context_print:        load time =     171.44 ms
llama_perf_context_print:        load time =     167.02 ms
llama_perf_context_print:        load time =     166.92 ms
llama_perf_context_print:        load time =     168.78 ms
llama_perf_context_print:        load time =     167.03 ms

bench result testing

• Command: ./build/bin/llama-bench -m ../gguf_models/llama-2-7b.Q2_K.gguf -n 1,2,4,8,16,32 -p 0

Original

| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |           tg1 |         14.49 ± 0.01 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |           tg2 |         14.48 ± 0.06 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |           tg4 |         14.48 ± 0.02 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |           tg8 |         14.48 ± 0.03 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |          tg16 |         14.47 ± 0.01 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |          tg32 |         14.45 ± 0.00 |

build: 8a308354 (3782)

After modification

| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |           tg1 |         14.49 ± 0.03 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |           tg2 |         14.44 ± 0.04 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |           tg4 |         14.42 ± 0.03 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |           tg8 |         14.47 ± 0.01 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |          tg16 |         14.45 ± 0.02 |
| llama 7B Q2_K - Medium         |   2.63 GiB |     6.74 B | CPU        |       8 |          tg32 |         14.37 ± 0.04 |

build: 216e7d96 (3784)

[slaren]

This is not a good solution, we should not add an API to workaround a bug. Instead, this needs to be fixed in llama.cpp so that the model loading times are not overwritten when creating additional contexts.

[slaren]

Alternatively, simply removing the update of the load time after the first evaluation would be enough. This was done to improve the accuracy of the load time when using mmap, since the model data might not have been read from disk until it is used, but there are better ways to do that, and I am not sure that it is really that important.

[ggerganov]

Alternatively, simply removing the update of the load time after the first evaluation would be enough. This was done to improve the accuracy of the load time when using mmap, since the model data might not have been read from disk until it is used, but there are better ways to do that, and I am not sure that it is really that important.

Yes, it's better to remove this hack.

[Septa2112]

OK, thanks for your suggestions! I will reopen the issue after resolving it by a better way.