Added CI Job to test new benchmarks (#29)

* Initial commit: Added CI Job to test new benchmarks

* Added test file (copy of run_bv.py)

* Added Matthew's suggestions (#4)

* Initial commit for ci tests

* Change bash script

* Bash script does not need cat

* Added second test file to test

* Sugar is unnecessary

* Added test in mapping

* Mapping takes way longer. Will develop further.

* Delete tests.txt

Deleted old residuals, ready to merge with ci branch

* Added upload artifact action (#6)

* Another test for CI

* Organized CI actions.

* Added both the text file and the original results as artifacts

* Added more aesthetic changes suggested by Matthew (#9)

* Added more changes suggested by Matthew

* Added proper management of __init__.py files (#11)

* Another test for CI

* Organized CI actions.

* Added both the text file and the original results as artifacts

* Added more changes suggested by matthew

* Added method to handle changes to __init__.py files

* Fixed error in main.yml file

* Fixing another error in yaml file

* Fixing scope error in yaml

* Updated changed-files 24 -> 24.1

* Attempt 1

* Attempt 2

* Attempt 3

* Worked. Fixed typo... Now reordering.

* Made a change in init file, let's see if it detects

* Fixed bash code

* Checking for something suspicious

* Checking for something suspicious

* Fixing something suspicious

* Fixed?

* Fixed??

* Test with only __init__ file

* Final commit before merging

* Improved Runtime by removing Washington, Brooklyn and Rochester backends. (#13)

* Remove usage of Washington, Brooklyn and Rockester backends due to slow runtime.

* What if there are no changes?

* Handle no files in a healthy way

* No changes works! Going back to adding changes.

* Created custom action to test each specific benchmark (#15)

* Try customized action

* Added custom acrtion to workflow.

* Fixed typo in workflow.

* Modified action location

* Modified action location 2

* Modified action location 3

* Modified action location 4

* Fixed path for testing

* Fixed typo in line 22 of action.

* Housekeeping, and red-added exclusions.

* Final commit, custom action is ready to be merged.

* Final fixes before merging (#17)

* Final fixes before merging

* Added modifications to CONTRIBUTING.md

* Added last modification to CONTRIBUTING.md

* Delete test.py for final version

.github/actions/changed-benchmark/action.yml

@@ -0,0 +1,33 @@
+name: 'Run Changed Benchmarks'
+description: 'Run Changed Benchmarks'
+inputs:
+  path:
+    description: 'Benchmark paths'
+    required: true
+  default_bench:
+    description: 'Default benchmark'
+    required: true
+runs:
+  using: "composite"
+  steps:
+    - name: Get changes in applications
+      id: changed-benchmarks
+      uses: tj-actions/[email protected]
+      with:
+        files: |
+          ${{ inputs.path }}
+    - name: Test applications
+      run: |
+        if [[ "${{steps.changed-benchmarks.outputs.all_changed_and_modified_files}}" != "" ]]; then
+          if [[ "${{steps.changed-benchmarks.outputs.all_changed_and_modified_files}}" == *"${{ inputs.path }}/__init__.py"* ]]; then
+              no_init=${${{steps.changed-benchmarks.outputs.all_changed_and_modified_files}}//"${{ inputs.path }}/__init__.py"/}
+              if [[ "$no_init" == "" ]]; then
+                  pytest ${{ inputs.default_bench }} -n 2 -k "not backend0 and not backend1 and not backend2" --verbose --store
+              else
+                  pytest "$no_init" -n 2 -k "not backend0 and not backend1 and not backend2" --verbose --store
+              fi
+          else
+            pytest ${{steps.changed-benchmarks.outputs.all_changed_and_modified_files}} -n 2 -k "not backend0 and not backend1 and not backend2" --verbose --store
+          fi
+        fi
+      shell: bash

.github/workflows/main.yml

@@ -30,3 +30,35 @@ jobs:
         run: python -m pip install -U tox
       - name: Run lint
         run: tox -elint
+  benchmark:
+    runs-on: ubuntu-latest
+    name: Run changed benchmarks
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 2
+      - name: Set up Python 3.10
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+      - name: Install Deps
+        run: |
+          python -m pip install --upgrade pip
+          pip install .
+      - name: Run changed application benchmarks
+        uses: ./.github/actions/changed-benchmark
+        with:
+            path: red_queen/games/applications
+            default_bench: red_queen/games/applications/run_bv.py
+      - name: Run changed mapping benchmarks
+        uses: ./.github/actions/changed-benchmark
+        with:
+            path: red_queen/games/mapping
+            default_bench:  red_queen/games/mapping/map_misc.py
+      - name: Print results to log
+        run: python -m report.console_tables --storage results/
+      - name: Upload original result file
+        uses: actions/upload-artifact@v3
+        with:
+          name: results
+          path: results/

CONTRIBUTING.md

@@ -25,3 +25,26 @@ If black returns a code formatting error you can run `tox -eblack` to
 automatically update the code formatting to conform to the style. However,
 if `pylint` returns any error you will have to fix these issues by manually
 updating your code.
+
+## Continuous Integration
+
+The red-queen repo has a Continuous Integration (CI) action that test any new 
+or modified benchmarks being added in a pull request. This action was implemented 
+using [changed-files](https://github.com/tj-actions/changed-files).
+
+In the case that a new type of benchmark category is implemented, a new step
+containing the directory of the python file that will be run with pytest should be 
+added to `.github/workflows/main.yml`. This step should call on our custom action
+`changed-benchmark` which will detect any changed made to any benchmark of a specific 
+type. This action uses [changed-files](https://github.com/tj-actions/changed-files). You must also select
+a default benchmark `<selected-def-bench>` to run in case a change is made to an 
+`__init__.py` file without changing any benchmarks.
+Your step should follow the format below:
+
+```yml
+    - name: Run changed <new_category> benchmarks
+        uses: ./.github/actions/changed-benchmark
+        with:
+            path: red_queen/games/<new_category>
+            default_bench: red_queen/games/<new_category>/<selected-def-bench>
+```