Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Results aggregation broken #347

Open
kayceesrk opened this issue May 13, 2022 · 2 comments · May be fixed by #374
Open

Results aggregation broken #347

kayceesrk opened this issue May 13, 2022 · 2 comments · May be fixed by #374
Assignees
@punchagan
Labels
good first issue Good for newcomers

Comments

@kayceesrk
Copy link
Contributor

I'm running Sandmark manually with the following run_all_serial.sh file:

#!/bin/bash

TAG='"macro_bench"' make run_config_filtered.json

OPT_WAIT=0 USE_SYS_DUNE_HACK=1 RUN_CONFIG_JSON=run_config_filtered.json make ocaml-versions/5.0.0+trunk.bench
OPT_WAIT=0 USE_SYS_DUNE_HACK=1 RUN_CONFIG_JSON=run_config_filtered.json make ocaml-versions/5.0.0+trunk+volatile_field.bench

When the run completes, I just have results/_1.orun.summary.bench file. The expectation is that I should have two files:

  • 5.0.0+trunk_1.orun.summary.bench
  • 5.0.0+trunk+volatile_field.orun.summary.bench

This is a regression. Not sure what's broken. The README hasn't been updated with additional instructions if there is supposed to be a different workflow.
@kayceesrk
Copy link
Contributor Author

kayceesrk commented May 13, 2022 •
edited
Loading

Looks like there is a SANDMARK_CUSTOM_NAME variable in the Makefile, which is undocumented. My suggestion would be

  1. Either make SANDMARK_CUSTOM_NAME a mandatory argument that terminates the make early if an argument is not given or pick it up automatically from the name of the variant where the variant is ocaml-versions/<variant>.json.
  2. Document this in the README.md file.
  3. Fix the run_all_*.sh files appropriately.

@kayceesrk
Copy link
Contributor Author

Nit: Now that SANDMARK_CUSTOM_NAME looks mandatory, it is better named as VARIANT_NAME.

@shakthimaan shakthimaan added the good first issue Good for newcomers label Jun 21, 2022
@shakthimaan shakthimaan moved this to Backlog in Benchmarking Jun 21, 2022
@shakthimaan shakthimaan moved this from Backlog to Todo in Benchmarking Jun 21, 2022
punchagan added a commit to punchagan/sandmark that referenced this issue Jul 18, 2022
@punchagan
Obtain SANDMARK_CUSTOM_NAME from variant name and make it optional
417de42 
But, also allow overriding it using an environment variable, if required.

Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Jul 18, 2022
@punchagan
Obtain SANDMARK_CUSTOM_NAME from variant name and make it optional
4a6601a 
But, also allow overriding it using an environment variable, if required.

Closes ocaml-bench#347
@punchagan punchagan linked a pull request Jul 21, 2022 that will close this issue
Open
punchagan added a commit to punchagan/sandmark that referenced this issue Jul 26, 2022
@punchagan
Obtain SANDMARK_CUSTOM_NAME from variant name and make it optional
4ee800c 
But, also allow overriding it using an environment variable, if required.

Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Jul 26, 2022
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
3ada19a 
Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Nov 17, 2022
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
af36336 
Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Dec 13, 2022
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
60e2c0f 
Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Dec 13, 2022
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
c8fa52b 
Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Dec 14, 2022
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
2352eb1 
Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Dec 14, 2022
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
6b22dd7 
Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Dec 26, 2022
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
03ccfed 
Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Jan 2, 2023
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
2e7e10a 
Closes ocaml-bench#347
punchagan added a commit to punchagan/sandmark that referenced this issue Jan 2, 2023
@punchagan
Rename SANDMARK_CUSTOM_NAME to VARIANT_NAME
e97dbdf 
Closes ocaml-bench#347
@ElectreAAS ElectreAAS removed their assignment Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@punchagan punchagan

Labels
good first issue Good for newcomers
Projects
Benchmarking
Status: Todo
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix broken result aggregation when SANDMARK_CUSTOM_NAME is not specified punchagan/sandmark
4 participants
@shakthimaan @punchagan @kayceesrk @ElectreAAS