Deploy to Production (#2999)

* Reduce space usage, limit number of fields shown * wip * wip * Color-code tables * Create confirmemail.spec.ts * Delete confirmemail.cy.js * Use test from utils * Make gray default * Fix variants table query * Migrate fields to local API (#2801) * Forward graphql requests to netlify function * Trigger Build * update lockfile * Avoid having to introspect schema for every request * Allow anonymous read only queries * Allow anonymous read only queries * Trigger Build * Add quickadds resolver * Split resolvers into separate files * Separate auth in its own file * Add jest and tests * Add script to boot local api server for development purposes * Add quickadd insert field * Add github workflows * Trigger build remove Trigger build Trigger build fsdfsf Trigger build Trigger build bigint patch use require * Update codegen tool config * Reuse mongodb client from context * Update codege * Strongly typed environment config * Increase function timeout * Update bson version * Make environment variables available to gatsby functions * Fix ts error * Trigger build * Make errors more verbose * Add missing variables * Disable env validation * Use graphql-http to prevent event emitter leaks * Enable env validation * New bson version exports ObjectId instead of ObjectID * Update lockfile * Trigger build * Remove unused query * bue * Add support for filters, sorting and pagination * Remove unused package * Remove unused file * allow api tests to be cancelled * Delete old test file * Run api tests when pushing staging * update test utils * Trigger build * Force exit to let tests exit * remove old api endpoint * Update client side queries * Udpate docs * Use vars for api and group ids * convert to import jic * Trigger build * Trigger build * Update workflows * Trigger build * Add missing variables * Fix error when object id is already serialized * Update workflow file * Improve error message * update readme * Remove unused vars * Remove unused packages * Log errors to rollbar * Trigger build * Add rollbar token variable * Trigger build * fix variable name * Trigger build * Add function to programmatically generate query fields * Add programmatic mutation fields function * Add tests for update methods * update client side queries * Add field queries tests * add tests for insert and update mutations * add delete mutations programmatic tests * Configure fixtures * Tests suite needs to be run in a single process * And missing dynamic fields and improve fixture design * Add roles expectations * delete redundant tests * Add incidents and report queries * Add report translations resolver * add missing dotenv config * Update tests to mock realm api * remove unrelated files * Remove unrelated changes * Remove unrelated changes * Remove access to test user credentials * Remove unused * Use dummy values on api test workflow * Fix wrong database being mocked * Fix schema issue * cleanup * Add incidents and reports field implementation * Implement linkReportsToIncidents mutation * Typed return * Delete old implementation * Add incidents/edit.spec * Add incidents/history.spec * Add incidents/new.spec * Delete incidents/new.cy incidents/history.cy and incidents/edit.cy * Add reports test api fixtures and improve fixtures expressiveness * Migrate entities collection * Add incidents fixture and update fixture structure * Add users fixture * Add utility functions to resolve and configure relationships * Make sure inputs are required * Add special type for available fields * Update test fixtures and fix permissions * add flag report mutation and tests * Add date modified fields * Fix upsertOne issues * Remove leftover * Add test to report to incident linking mutation * Add submissions collection to api * Add full e2e workflow * Trigger Build * update lockfile * Fix history latest version test * Fix new spec errors * Add in memory mongo server * remove comment * Wait for memory server to start * Trigger Build * Remove quotes * Trigger Build * Fix typo in name * Get cite tests to work fully e2e * Rename intercept for consistency * Add missing waitForRequest * get submission form working with new api * Add missing env variable * Trigger Build * Add missing variable * Trigger Build * Update users table components * Add similarity flag mutation * Add update one report mutation * Get incident report edit form to work * update lockfile * Get dinamic citation page to work * Get incident handling to work * Update tests * Update entity type to use date objects * Fix schema stiching warnings * Fix/silence warnings * Rename to prevent artifact collision * Update history related tests * Trigger Build * Add docs to server utils * Silence warning * Run mocked tests separately. * Trigger Build * Make sure to give time to autocomplete * Fix tests * Add createVariant mutation and separate types into files to deal with circular dep * Delete custom mutation from remote * Update variants related components and tests * Trigger Build * Add memory mongo command * Update tests * avoid report artifact collision * Trigger Build * update test * Fix incidents app * Remove debug * Separate incident data type into its own file --------- Co-authored-by: Clara Youdale <[email protected]> * Add full e2e workflows to production and staging * Add classifications collection to api * delete unnecessary merge configs * Migrate taxa collection to api * Revert "Add classifications collection to api" This reverts commit 0da2bc0. * setNavCollapsed for quick access * Change query to filter on Variants table * Update workflows * Remove the outdated `variants.cy.js` and use `playwright/e2e-full/apps/variants.spec.ts` instead. * Enable static type checking of Apollo client calls (#3012) * Configure graphql codegen to work with Apollo client and enable static checking * Update queries to use new codegen function * Fix anonymous mutation * Add updated codegen files * Add comment * Add codegen babel plugin to remove generated query map from production bundle * Add generated fragment maskings * update lockfile * remove unnecessary mock * style * Delete migrated files (#3003) * Delete migrated files * Delete function file * Delete report translations resolver * Change FIND_INCIDENTS query filter to use the new approach * Migrate submissions to new API (#3010) * Migrate submissions to new API * Remove unused adminData * Fix updateSubmission mutation with new API changes * Update SubmissionEditModal.js * Add .ts to eslintrc * Check for incident_ids if null * Add new user query to not fetch adminData * Add generated files for new query * Revert to previous user roles query * Silence schema merge warning (#3024) * Fix staging issues (#3026) * Add fixes to relationship handling from #1072 * Return link validation errors instead of throwing so they are shown in graphql response * Update outdated query variables * Make sure query paramter is int * Add tests --------- Co-authored-by: Luna McNulty <[email protected]> Co-authored-by: Clara Youdale <[email protected]> Co-authored-by: Pablo Costa <[email protected]> Co-authored-by: Kevin <[email protected]>
responsible-ai-collaborative · Aug 19, 2024 · 6c38996 · 6c38996
1 parent f10e79b
commit 6c38996
Show file tree

Hide file tree

Showing 147 changed files with 15,460 additions and 18,510 deletions.
diff --git a/.github/workflows/preview.yml b/.github/workflows/preview.yml
@@ -70,6 +70,16 @@ jobs:
       environment: staging
       runner-label: ${{ vars.PREVIEW_RUNNER_LABEL }}
 
+  call-test-playwright-full:
+    if: ${{ !failure() && !cancelled() }}
+    uses: ./.github/workflows/test-playwright-full.yml
+    needs: permissions-check
+    secrets: inherit
+    with:
+      sha: ${{ github.event.pull_request.head.sha }}
+      environment: staging
+      runner-label: ${{ vars.PREVIEW_RUNNER_LABEL }}
+
   call-deploy:
     if: ${{ !failure() && !cancelled() }}
     uses: ./.github/workflows/deploy.yml

diff --git a/.github/workflows/production.yml b/.github/workflows/production.yml
@@ -71,6 +71,16 @@ jobs:
       environment: production
       runner-label: ${{ vars.PRODUCTION_RUNNER_LABEL }}
 
+  call-test-playwright-full:
+    if: ${{ !failure() && !cancelled() }}
+    uses: ./.github/workflows/test-playwright-full.yml
+    needs: [cache-modifier]
+    secrets: inherit
+    with:
+      sha: ${{ github.sha }}
+      environment: production
+      runner-label: ${{ vars.PREVIEW_RUNNER_LABEL }}
+
   call-deploy:
     if: (inputs.force-deploy == true || success()) && !cancelled()
     uses: ./.github/workflows/deploy.yml

diff --git a/.github/workflows/staging.yml b/.github/workflows/staging.yml
@@ -12,10 +12,10 @@ on:
         default: true
         type: boolean
       force-deploy:
-          description: "Deploy even if tests fail"
-          required: true
-          default: false
-          type: boolean
+        description: "Deploy even if tests fail"
+        required: true
+        default: false
+        type: boolean
 
 jobs:
   cache-modifier:
@@ -59,8 +59,8 @@ jobs:
       sha: ${{ github.sha }}
       environment: staging
       runner-label: ${{ vars.STAGING_RUNNER_LABEL }}
-      
-  call-test: 
+
+  call-test:
     uses: ./.github/workflows/test.yml
     needs: [cache-modifier, call-test-build]
     secrets: inherit
@@ -70,6 +70,16 @@ jobs:
       runner-label: ${{ vars.STAGING_RUNNER_LABEL }}
       cache-modifier: ${{ needs.cache-modifier.outputs.cache-modifier  }}
 
+  call-test-playwright-full:
+    if: ${{ !failure() && !cancelled() }}
+    uses: ./.github/workflows/test-playwright-full.yml
+    needs: [cache-modifier]
+    secrets: inherit
+    with:
+      sha: ${{ github.sha }}
+      environment: staging
+      runner-label: ${{ vars.PREVIEW_RUNNER_LABEL }}
+
   call-deploy:
     if: (inputs.force-deploy == true || success()) && !cancelled()
     uses: ./.github/workflows/deploy.yml

diff --git a/.github/workflows/test-playwright-full.yml b/.github/workflows/test-playwright-full.yml
@@ -0,0 +1,155 @@
+name: Run playwright tests full e2e
+on:
+  workflow_call:
+    inputs:
+      environment:
+        description: The Github environment to load secrets from
+        type: string
+        required: true
+      sha:
+        description: The commit SHA to run the tests against
+        type: string
+        required: true
+      runner-label:
+        description: The label of the runner to use
+        type: string
+      cache-modifier:
+        description: A modifier for the cache key used to bypass existing cache
+        type: string
+        required: false
+        default: ""
+
+jobs:
+  tests:
+    name: Run Playwright tests
+    environment: ${{ inputs.environment }}
+    timeout-minutes: 60
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        shardIndex: [1, 2, 3, 4]
+        shardTotal: [4]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.sha }}
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: lts/*
+
+      - name: Read node modules from cache
+        id: cache-nodemodules
+        uses: actions/cache/restore@v4
+        env:
+          cache-name: cache-install-folder
+        with:
+          path: |
+            site/gatsby-site/node_modules
+            ~/.cache/Cypress
+          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}-${{ inputs.cache-modifier }}
+
+      - name: Install NPM dependencies
+        if: steps.cache-nodemodules.outputs.cache-hit != 'true'
+        run: npm ci
+        working-directory: site/gatsby-site
+
+      - name: start in memory mongodb
+        run: nohup node -r ts-node/register playwright/memory-mongo.ts &
+        working-directory: site/gatsby-site
+        env:
+          MONGODB_CONNECTION_STRING: mongodb://127.0.0.1:4110/
+
+      - name: Wait for memory mongodb to start
+        run: |
+          while ! nc -z localhost 4110; do
+            echo "Waiting for Mongodb memory server to start on port 4110..."
+            sleep 2
+          done
+          echo "MongoDB has started on port 4110."
+
+      - name: Build using Netlify
+        run: netlify build --context preview --offline
+        working-directory: site/gatsby-site
+        env:
+          INSTRUMENT: true
+          NETLIFY_SITE_ID: ${{ vars.NETLIFY_SITE_ID }}
+          NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          E2E_ADMIN_USERNAME: ${{ secrets.E2E_ADMIN_USERNAME }}
+          E2E_ADMIN_PASSWORD: ${{ secrets.E2E_ADMIN_PASSWORD }}
+          ALGOLIA_ADMIN_KEY: ${{ secrets.ALGOLIA_ADMIN_KEY }}
+          GATSBY_ALGOLIA_APP_ID: ${{ vars.GATSBY_ALGOLIA_APP_ID }}
+          GATSBY_ALGOLIA_SEARCH_KEY: ${{ vars.GATSBY_ALGOLIA_SEARCH_KEY }}
+          GATSBY_AVAILABLE_LANGUAGES: ${{ vars.GATSBY_AVAILABLE_LANGUAGES }}
+          GATSBY_REALM_APP_ID: ${{ vars.GATSBY_REALM_APP_ID }}
+          GOOGLE_TRANSLATE_API_KEY: ${{ secrets.GOOGLE_TRANSLATE_API_KEY }}
+          MONGODB_CONNECTION_STRING: mongodb://127.0.0.1:4110/
+          MONGODB_TRANSLATIONS_CONNECTION_STRING: mongodb://127.0.0.1:4110/
+          MONGODB_MIGRATIONS_CONNECTION_STRING: mongodb://127.0.0.1:4110/
+          TRANSLATE_SUBMISSION_DATE_START: ${{ vars.TRANSLATE_SUBMISSION_DATE_START }}
+          GATSBY_REALM_APP_GRAPHQL_URL: ${{ secrets.GATSBY_REALM_APP_GRAPHQL_URL }}
+          GATSBY_PRISMIC_REPO_NAME: ${{ vars.GATSBY_PRISMIC_REPO_NAME }}
+          PRISMIC_ACCESS_TOKEN: ${{ secrets.PRISMIC_ACCESS_TOKEN }}
+          GATSBY_ROLLBAR_TOKEN: ${{ secrets.GATSBY_ROLLBAR_TOKEN }}
+          SKIP_PAGE_CREATOR: ${{ vars.SKIP_PAGE_CREATOR }}
+          CLOUDFLARE_R2_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_R2_ACCOUNT_ID }}
+          CLOUDFLARE_R2_BUCKET_NAME: ${{ secrets.CLOUDFLARE_R2_BUCKET_NAME }}
+          GATSBY_CLOUDFLARE_R2_PUBLIC_BUCKET_URL: ${{ vars.GATSBY_CLOUDFLARE_R2_PUBLIC_BUCKET_URL }}
+          CLOUDFLARE_R2_ACCESS_KEY_ID: ${{ secrets.CLOUDFLARE_R2_ACCESS_KEY_ID }}
+          CLOUDFLARE_R2_SECRET_ACCESS_KEY: ${{ secrets.CLOUDFLARE_R2_SECRET_ACCESS_KEY }}
+          REALM_GRAPHQL_API_KEY: ${{ secrets.REALM_GRAPHQL_API_KEY }}
+          GATSBY_COMMIT_SHA: ${{ inputs.sha }}
+          REALM_API_APP_ID: ${{ vars.REALM_API_APP_ID }}
+          REALM_API_GROUP_ID: ${{ vars.REALM_API_GROUP_ID }}
+          REALM_APP_ID: ${{ vars.GATSBY_REALM_APP_ID }}
+          REALM_API_PUBLIC_KEY: ${{ secrets.REALM_API_PUBLIC_KEY }}
+          REALM_API_PRIVATE_KEY: ${{ secrets.REALM_API_PRIVATE_KEY }}
+          ROLLBAR_POST_SERVER_ITEM_ACCESS_TOKEN: ${{ secrets.GATSBY_ROLLBAR_TOKEN }}
+          API_MONGODB_CONNECTION_STRING: mongodb://127.0.0.1:4110/
+
+      - name: Install playwright browsers
+        run: npx playwright install --with-deps
+        working-directory: site/gatsby-site
+
+      - name: Run playwright tests
+        run: npx playwright test playwright/e2e-full/ --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
+        working-directory: site/gatsby-site
+        env:
+          E2E_ADMIN_PASSWORD: ${{ secrets.E2E_ADMIN_PASSWORD }}
+          E2E_ADMIN_USERNAME: ${{ secrets.E2E_ADMIN_USERNAME }}
+          IS_EMPTY_ENVIRONMENT: ${{ vars.IS_EMPTY_ENVIRONMENT }}
+          MONGODB_CONNECTION_STRING: mongodb://127.0.0.1:4110/
+
+      - uses: actions/upload-artifact@v4
+        if: ${{ !cancelled() }}
+        with:
+          name: blob-report-full-${{ matrix.shardIndex }}
+          path: site/gatsby-site/blob-report/
+          retention-days: 1
+
+  merge-reports:
+    # Merge reports after playwright-tests, even if some shards have failed
+    if: ${{ !cancelled() }}
+    needs: [tests]
+
+    runs-on: ubuntu-latest
+    steps:
+      - name: Download blob reports from GitHub Actions Artifacts
+        uses: actions/download-artifact@v4
+        with:
+          path: all-blob-reports
+          pattern: blob-report-full-*
+          merge-multiple: true
+
+      - name: Merge into HTML Report
+        run: npx playwright merge-reports --reporter html ./all-blob-reports
+
+      - name: Upload HTML report
+        uses: actions/upload-artifact@v4
+        with:
+          name: html-report--attempt-${{ github.run_attempt }}
+          path: playwright-report-full
+          retention-days: 7
diff --git a/.github/workflows/test-playwright.yml b/.github/workflows/test-playwright.yml
@@ -71,7 +71,7 @@ jobs:
         working-directory: site/gatsby-site
 
       - name: Run playwright tests
-        run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
+        run: npx playwright test playwright/e2e/ --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
         working-directory: site/gatsby-site
         env:
           E2E_ADMIN_PASSWORD: ${{ secrets.E2E_ADMIN_PASSWORD }}

diff --git a/site/gatsby-site/.eslintrc.json b/site/gatsby-site/.eslintrc.json
@@ -27,7 +27,7 @@
           ["plugins", "./plugins"],
           ["@reach/router", "@gatsbyjs/reach-router"]
         ],
-        "extensions": [".js", ".jsx"]
+        "extensions": [".js", ".jsx", ".ts"]
       }
     }
   },

diff --git a/site/gatsby-site/codegen.ts b/site/gatsby-site/codegen.ts
@@ -5,9 +5,17 @@ const config: CodegenConfig = {
   overwrite: true,
   schema: ["./server/schema.ts"],
   require: ['ts-node/register', 'dotenv/config'],
+  // TODO: documents option should be 'src/**/!(*.d).{ts,tsx,js}' instead so it parses the entire project but as far as I'm aware there is no way to parse some queries while ignoring others (we have to ignore gatsby's queries because they use a different schema)
+  documents: 'src/graphql/**/!(*.d).{ts,tsx,js}',
+  pluckConfig: {
+    globalIdentifier: 'gql',
+  },
   generates: {
-    "server/generated/graphql.ts": {
-      plugins: ["typescript", "typescript-resolvers", "typescript-mongodb"]
+    "server/generated/": {
+      preset: 'client',
+      presetConfig: {
+        gqlTagName: "gql",
+      }
     }
   }
 };