UI issue: synchronize graph line colors

[OmarTawfik]

Thanks for the continuous work/effort on this!

Looking at the following project plots:
https://bencher.dev/console/projects/slang/plots

It shows different metrics (for the same 3 benchmarks) across time:

Unfortunately, the same benchmark is rendered with different colors (random?) in each graph:

• Left: Maximum Allocated Bytes plot and yellow is iai::benchmarks::parser benchmark.
• Right: Total Allocated Bytes plot and yellow is iai::benchmarks::query benchmark.

This makes it really hard to read/correlate increases/decreases in different metrics, and can convey the wrong results.

[epompeii]

Unfortunately, the same benchmark is rendered with different colors (random?) in each graph

Ah! The colors are not random. They follow the Tableau 10 color scheme based on their order in the key: https://d3js.org/d3-scale-chromatic/categorical#schemeTableau10

The order in the key is dictated by the order of selection when creating the Perf Plot, before pinning it.

To fix this, you can:

1. Click the "View Plot" button for one of them.
2. In the Perf Plot view, select the "Benchmarks" tab.
3. Click the "Clear Benchmarks" button.
4. Reselect the Benchmarks in the desired order.
5. Click the "Pin" button and give this new plot the same title and save.
6. Delete the old, bad-ordered pinned plot.

This is definitely tedious and preferably this would "just work". However given the high dimensionality of Bencher results (Branch x Testbed x Benchmark x Measure) I'm not sure how to ensure this programmatically.

For example, there could be a pinned plot that is for the same Benchmark across three different Testbeds. What color would it then be best to use?

I'm glad you're taking advantage of pinned Plots. Let me know if you have any idea how to square the circle here.

[OmarTawfik]

To fix this, you can:

I see. Thanks for explaining. I tried it and it fixed the issue for now.

given the high dimensionality of Bencher results (Branch x Testbed x Benchmark x Measure)
What color would it then be best to use?

Thinking more about this, what do you think of:

• for each plot line, create a sorting key of "${branch}-#{testbed}-#{benchmark}".
• sort all lines in the same graph using this key.
• assign each to 1 of the 10 colors above based on their order.

Note: dimensions that are the same between all lines should not participate in the sorting key (for example: the "measure" name). This is what allows comparison between multiple pinned plots in a dashboard.

This will ensure the colors are "stable" across different graphs, and will allow comparing multiple graphs even if a dimension differs between them. It will also work if there is a single dimension that changes between each graph, as the lines will still be sorted similarly on the other dimensions.

[epompeii]

This will ensure the colors are "stable" across different graphs, and will allow comparing multiple graphs even if a dimension differs between them.

This would definitely make the colors stable for your use case. However, I can think of a few instances where this would not be the case. Enforcing a sort order would make it not possible for users to handle these cases themselves and order things as they wish.

With that said, I could see having this kind of sort be an option for the table view of the Perf Plot key: #498

[OmarTawfik]

Fair enough. But I wonder if there is a need to have a different order in each graph, if the underlying data lines don't actually change?

In any case, I think my issue is fixed already by re-ordering the benchmarks manually and recreating the plot. The dashboard above is fixed/consistent now. So I'm happy to close this issue. Thanks for the help!