Add query enhancement cypress tests to Github CI #8703

abbyhu2000 · 2024-10-25T06:23:02Z

Description

Add query enhancement cypress tests workflow to github CI to test discover 2.0 functionalities:

sql, ppl queries
dataset selector
...

The tests can be added under folder cypress/integration/core-opensearch-dashboards/opensearch-dashboards/apps/query_enhancement in the functional repo, and will be run in ciGroup10 in the CI.

Issues Resolved

Screenshot

Testing the changes

Changelog

feat: Add query enhancement cypress tests to Github CI

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: abbyhu2000 <[email protected]>

codecov · 2024-10-25T06:43:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.81%. Comparing base (9a25d0d) to head (80b8eaf).
Report is 31 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8703      +/-   ##
==========================================
- Coverage   60.84%   60.81%   -0.03%     
==========================================
  Files        3793     3793              
  Lines       90486    90486              
  Branches    14212    14212              
==========================================
- Hits        55057    55031      -26     
- Misses      31939    32009      +70     
+ Partials     3490     3446      -44

Flag	Coverage Δ
Linux_1	`29.07% <ø> (ø)`
Linux_2	`?`
Linux_3	`37.66% <ø> (-0.01%)`	⬇️
Linux_4	`29.84% <ø> (ø)`
Windows_1	`29.08% <ø> (ø)`
Windows_2	`56.35% <ø> (ø)`
Windows_3	`37.67% <ø> (ø)`
Windows_4	`29.84% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-10-25T08:30:23Z

❌ Invalid Prefix For Manual Changeset Creation

Invalid description prefix. Found "feat". Only "skip" entry option is permitted for manual commit of changeset files.

If you were trying to skip the changelog entry, please use the "skip" entry option in the ##Changelog section of your PR description.

abbyhu2000 · 2024-10-25T18:23:08Z

Cypress runner:
https://github.com/opensearch-project/OpenSearch-Dashboards/actions/runs/11515231239/job/32057142616

CiGroup10 is testing query enhancement tests that are added in this functional repo PR: opensearch-project/opensearch-dashboards-functional-test#1598

All passing except for CiGroup 6, i think currently ciGroup6 is failing on main, should not be relate to this PR.

ananzh · 2024-10-25T20:39:10Z

.github/workflows/cypress_workflow.yml


 jobs:
  cypress-tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
-          group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+        include:


Will this work and help organize?

matrix: config: [standard, query_enhanced, dashboard] group: [1, 2, 3, 4, 5, 6, 7, 8, 9] include: - config: query_enhanced group: 10 test_location: ftr - config: dashboard group: 11 test_location: source exclude: - config: query_enhanced group: [1,2,3,4,5,6,7,8,9] - config: dashboard group: [1,2,3,4,5,6,7,8,9]

+1, using matrix to create the right config could make it shorter. but i do see a benefit of making it this granular as it we have more control. as it leave spaces in the future where we migrate the tests from ftr to this repo

ananzh · 2024-10-25T20:43:29Z

QQ: so for writing tests in ciGroup10, do we need add some code to create a workspace? and test in the workspace?

abbyhu2000 · 2024-10-26T00:11:26Z

QQ: so for writing tests in ciGroup10, do we need add some code to create a workspace? and test in the workspace?

i

QQ: so for writing tests in ciGroup10, do we need add some code to create a workspace? and test in the workspace?

I think i asked @ashwin-pc during standup and he responded that right now we only covers the flows on discover pages.

Do we want to add test to include extra workflows like creating the workspace? or we only test what discover page covers.

kavilla · 2024-10-26T07:00:46Z

.github/workflows/cypress_workflow.yml

+            test_location: ftr
+          - group: 3
+            config: standard
+            test_location: ftr


we have a variable FTR_PATH, should we use this/change FTR_PATH to something that makes more sense to usage and add the other locations? or should we get rid of the FTR_PATH variable

kavilla · 2024-10-26T07:02:52Z

.github/workflows/cypress_workflow.yml

@@ -42,18 +40,65 @@ env:
  COMMENT_TAG: '[MANUAL CYPRESS TEST RUN RESULTS]'
  COMMENT_SUCCESS_MSG: ':white_check_mark: Cypress test run succeeded!'
  COMMENT_FAILURE_MSG: ':x: Cypress test run failed!'
+  LATEST_VERSION: '2.17.0'


maybe in an inline comment on why we are using this. i believe it was because of issues trying to use main and install plugins for github virtual env

one thing we could also do is actually use to make this dynamic is rely on git tags. every release infra will cut a tag for the release in all the repos post release and then we use that to make a github release. we could potentially just get the highest version tag from our repo and assume that is the highest release version of OpenSearch that we can download

kavilla · 2024-10-26T07:18:24Z

.github/workflows/cypress_workflow.yml

-      - name: Setup spec files for Dashboards tests
-        if: ${{ inputs.specs == '' && matrix.group > 9 }}
+      # Run tests based on configuration
+      - name: Run FT repo tests with no query enhancements


nit: historically we called this FTRepo.

and I think the specification with no query enhancements might be unnecessary. we should go with the actual one run the query enhancements to say with query enhancements incase we want to follow this format in the future and add different type of situations.

kavilla · 2024-10-26T07:22:21Z

.github/workflows/cypress_workflow.yml

+      # Run tests based on configuration
+      - name: Run FT repo tests with no query enhancements
+        if: matrix.test_location == 'ftr' && matrix.config == 'standard'
+        uses: cypress-io/github-action@v2


did we want to bump the cypress github action if available and works? might have improvements https://github.com/cypress-io/github-action/blob/master/CHANGELOG.md. but i see somethings in our file will need to change so maybe not worht it at this time

kavilla · 2024-10-26T07:23:07Z

.github/workflows/cypress_workflow.yml

+      - name: Extract OpenSearch
+        if: matrix.test_location == 'ftr' && matrix.config == 'query_enhanced'
+        run: |
+          tar -xzf opensearch-*.tar.gz


we should be able to untar this file and pass the param to call it a specific name so line 238 doesn't have to be a specific version

kavilla · 2024-10-26T07:26:56Z

.github/workflows/cypress_workflow.yml

+        if: matrix.test_location == 'ftr' && matrix.config == 'query_enhanced'
+        run: |
+          /bin/bash -c "./opensearch-2.17.0/opensearch-tar-install.sh &"
+          sleep 30


confused about this one.

wait-on should be actually a better resolution to this: https://github.com/cypress-io/github-action?tab=readme-ov-file#wait-on

are you seeing issues with that?

kavilla · 2024-10-26T07:34:52Z

.github/workflows/cypress_workflow.yml

    container:
      image: docker://opensearchstaging/ci-runner:ci-runner-rockylinux8-opensearch-dashboards-integtest-v2
      options: --user 1001
    env:
+      START_CMD: ${{ matrix.config == 'query_enhanced' &&


i really like this concept of config because it actually follows the format well with how OSD functional tests are structured!

only nit I have here is that this actually is a little bit hard to read. and any fixes or additional updates to this will require us to know to update both places.

so i think minimum we should consider the logic being where we append extra configurations if the matrix config is query enhanced instead of having two.

if that is not good enough then it might be time to start sourcing a .env config file that stores this information so that we can modify this stuff and make this file more approachable.

kavilla · 2024-10-26T07:36:45Z

.github/workflows/cypress_workflow.yml

+            config: standard
+            test_location: ftr
+          - group: 4
+            config: standard


nit: default has been used for this type of situation or undefined and we automatically apply the default config

kavilla · 2024-10-26T07:40:13Z

.github/workflows/cypress_workflow.yml

+          IFS="," read -a SPEC_ARRAY <<< $(yarn --silent osd:ciGroup${{ matrix.group }})
+          DASHBOARDS_SPEC=''
+          for i in "${SPEC_ARRAY[@]}"; do
+          DASHBOARDS_SPEC+="cypress/integration/core_opensearch_dashboards/${i},"


we desperately need to be able to tag cypress test so that when we write them inhouse the test can decide which workflow it will run in

kavilla

i really like the proposal. i mentioned this in the comments moving towards the config route is actually more in line with how OSD functional tests are so the system is better built for that. So nice call!

Some comments though related to future proving the solution

kavilla

everything i think can just be a fast follow

opensearch-trigger-bot · 2024-11-02T02:19:11Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-8703-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 58293032a8c7ee7b985baf54504138eb0086921a
# Push it to GitHub
git push --set-upstream origin backport/backport-8703-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8703-to-2.x.

…8703) * Add cypress ciGroup11 to github workflow for discover 2.0 tests Signed-off-by: abbyhu2000 <[email protected]> * modify matrix logic Signed-off-by: abbyhu2000 <[email protected]> * fix start command Signed-off-by: abbyhu2000 <[email protected]> * more test Signed-off-by: abbyhu2000 <[email protected]> * switch to another way to install sql plugin Signed-off-by: abbyhu2000 <[email protected]> * install wget Signed-off-by: abbyhu2000 <[email protected]> * try 1 Signed-off-by: abbyhu2000 <[email protected]> * add job scheduler Signed-off-by: abbyhu2000 <[email protected]> * use opensearch bundle link Signed-off-by: abbyhu2000 <[email protected]> * get rid of verifying Signed-off-by: abbyhu2000 <[email protected]> * use 2.17.0 Signed-off-by: abbyhu2000 <[email protected]> * Try another way to run opensearch Signed-off-by: abbyhu2000 <[email protected]> * correct conditions Signed-off-by: abbyhu2000 <[email protected]> * dis-enable workspace Signed-off-by: abbyhu2000 <[email protected]> * use test location instead of matrix number Signed-off-by: abbyhu2000 <[email protected]> * remove ciGroup12 in package.json Signed-off-by: abbyhu2000 <[email protected]> --------- Signed-off-by: abbyhu2000 <[email protected]>

Add query enhancement cypress tests to Github CI Signed-off-by: abbyhu2000 <[email protected]>

abbyhu2000 added 16 commits October 23, 2024 23:59

Add cypress ciGroup11 to github workflow for discover 2.0 tests

adb3dfc

Signed-off-by: abbyhu2000 <[email protected]>

modify matrix logic

04fb022

Signed-off-by: abbyhu2000 <[email protected]>

fix start command

0a61085

Signed-off-by: abbyhu2000 <[email protected]>

more test

5b93291

Signed-off-by: abbyhu2000 <[email protected]>

switch to another way to install sql plugin

5b37d74

Signed-off-by: abbyhu2000 <[email protected]>

install wget

4f29a0a

Signed-off-by: abbyhu2000 <[email protected]>

try 1

93f0123

Signed-off-by: abbyhu2000 <[email protected]>

add job scheduler

3ee8452

Signed-off-by: abbyhu2000 <[email protected]>

use opensearch bundle link

4f19024

Signed-off-by: abbyhu2000 <[email protected]>

get rid of verifying

1d1b334

Signed-off-by: abbyhu2000 <[email protected]>

use 2.17.0

0d48f93

Signed-off-by: abbyhu2000 <[email protected]>

Try another way to run opensearch

76cfb80

Signed-off-by: abbyhu2000 <[email protected]>

correct conditions

a39b324

Signed-off-by: abbyhu2000 <[email protected]>

dis-enable workspace

2688de8

Signed-off-by: abbyhu2000 <[email protected]>

use test location instead of matrix number

29e1c41

Signed-off-by: abbyhu2000 <[email protected]>

remove ciGroup12 in package.json

80b8eaf

Signed-off-by: abbyhu2000 <[email protected]>

abbyhu2000 requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe and ruanyl as code owners October 25, 2024 06:23

abbyhu2000 mentioned this pull request Oct 25, 2024

Add GitHub workflow cypress #8684

Closed

7 tasks

ananzh reviewed Oct 25, 2024

View reviewed changes

kavilla reviewed Oct 26, 2024

View reviewed changes

kavilla added ci operational excellence backport 2.x labels Oct 26, 2024

kavilla reviewed Oct 26, 2024

View reviewed changes

ananzh added the v2.19.0 label Oct 30, 2024

kavilla approved these changes Nov 1, 2024

View reviewed changes

kavilla mentioned this pull request Nov 1, 2024

[Discover] Update Version Filter for Data Sources #8785

Merged

7 tasks

ananzh approved these changes Nov 2, 2024

View reviewed changes

ananzh merged commit 5829303 into main Nov 2, 2024
107 of 116 checks passed

opensearch-trigger-bot bot added the failed backport label Nov 2, 2024

abbyhu2000 mentioned this pull request Nov 20, 2024

[Manual backport 2.x] Add query enhancement cypress tests to Github CI (#8703) #8898

Merged

7 tasks

abbyhu2000 added a commit that referenced this pull request Nov 20, 2024

Add query enhancement cypress tests to Github CI (#8703) (#8898)

390c5fc

Add query enhancement cypress tests to Github CI Signed-off-by: abbyhu2000 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add query enhancement cypress tests to Github CI #8703

Add query enhancement cypress tests to Github CI #8703

abbyhu2000 commented Oct 25, 2024 •

edited

Loading

codecov bot commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024

abbyhu2000 commented Oct 25, 2024

ananzh Oct 25, 2024

kavilla Oct 26, 2024

ananzh commented Oct 25, 2024

abbyhu2000 commented Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024

kavilla Oct 26, 2024

kavilla Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024

kavilla left a comment

kavilla left a comment

opensearch-trigger-bot bot commented Nov 2, 2024

Add query enhancement cypress tests to Github CI #8703

Add query enhancement cypress tests to Github CI #8703

Conversation

abbyhu2000 commented Oct 25, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Changelog

Check List

codecov bot commented Oct 25, 2024 • edited Loading

Codecov Report

github-actions bot commented Oct 25, 2024

❌ Invalid Prefix For Manual Changeset Creation

abbyhu2000 commented Oct 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ananzh commented Oct 25, 2024

abbyhu2000 commented Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kavilla Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kavilla Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kavilla left a comment

Choose a reason for hiding this comment

kavilla left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Nov 2, 2024

abbyhu2000 commented Oct 25, 2024 •

edited

Loading

codecov bot commented Oct 25, 2024 •

edited

Loading

abbyhu2000 commented Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024 •

edited

Loading

kavilla Oct 26, 2024 •

edited

Loading