Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add a config for running CI against the nightly driver MONGOSH-1473 #1780

Merged
merged 7 commits into from
Jan 8, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .evergreen-nightly-driver.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# https://github.com/evergreen-ci/evergreen/blob/main/docs/Project-Configuration/Parameterized-Builds.md#project-config
parameters:
- key: mongodb_driver_version_override
value: nightly

include:
- filename: .evergreen.yml
2 changes: 2 additions & 0 deletions .evergreen.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ functions:
env:
NODE_JS_VERSION: ${node_js_version}
DISTRO_ID: ${distro_id}
MONOGDB_DRIVER_VERSION_OVERRIDE: ${mongodb_driver_version_override}
script: |
source .evergreen/install-node.sh
source .evergreen/install-npm-deps.sh
Expand Down Expand Up @@ -114,6 +115,7 @@ functions:
env:
NODE_JS_VERSION: ${node_js_version}
DISTRO_ID: ${distro_id}
MONOGDB_DRIVER_VERSION_OVERRIDE: ${mongodb_driver_version_override}
script: |
source .evergreen/install-node.sh
source .evergreen/install-npm-deps.sh
Expand Down
2 changes: 2 additions & 0 deletions .evergreen/evergreen.yml.in
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ functions:
env:
NODE_JS_VERSION: ${node_js_version}
DISTRO_ID: ${distro_id}
MONOGDB_DRIVER_VERSION_OVERRIDE: ${mongodb_driver_version_override}
script: |
source .evergreen/install-node.sh
source .evergreen/install-npm-deps.sh
Expand Down Expand Up @@ -181,6 +182,7 @@ functions:
env:
NODE_JS_VERSION: ${node_js_version}
DISTRO_ID: ${distro_id}
MONOGDB_DRIVER_VERSION_OVERRIDE: ${mongodb_driver_version_override}
script: |
source .evergreen/install-node.sh
source .evergreen/install-npm-deps.sh
Expand Down
9 changes: 9 additions & 0 deletions .evergreen/install-npm-deps.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@ set -x

npm ci --verbose

echo "MONOGDB_DRIVER_VERSION_OVERRIDE:$MONOGDB_DRIVER_VERSION_OVERRIDE"

# if MONOGDB_DRIVER_VERSION_OVERRIDE is set, then we want to replace the package version
if [[ -n "$MONOGDB_DRIVER_VERSION_OVERRIDE" ]]; then
export REPLACE_PACKAGE="mongodb:$MONOGDB_DRIVER_VERSION_OVERRIDE"
npm run replace-package
npm ci --verbose --force # force because of issues with peer deps and semver pre-releases
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the evergreen log:

 [2024/01/08 13:03:58.391] Replacing mongodb in . with file:///data/mci/34be6bd4d487b3797e6cba6a385a9abf/src/nightly

That's not an actual file that exists, right? And npm ls still stays [email protected]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

~/mongo/mongosh % export REPLACE_PACKAGE=mongodb:nightly
~/mongo/mongosh % npm run replace-package

> replace-package
> node scripts/replace-package.js

Replacing mongodb in . with file:///Users/leroux.bodenstein/mongo/mongosh/nightly
Replacing mongodb in packages/arg-parser with file:///Users/leroux.bodenstein/mongo/mongosh/nightly
Replacing mongodb in packages/e2e-tests with file:///Users/leroux.bodenstein/mongo/mongosh/nightly
Replacing mongodb in packages/service-provider-core with file:///Users/leroux.bodenstein/mongo/mongosh/nightly
Replacing mongodb in packages/service-provider-server with file:///Users/leroux.bodenstein/mongo/mongosh/nightly
Replacing mongodb in packages/shell-api with file:///Users/leroux.bodenstein/mongo/mongosh/nightly
Replacing mongodb in packages/types with file:///Users/leroux.bodenstein/mongo/mongosh/nightly
~/mongo/mongosh % git status
On branch nightly-driver
Your branch is up to date with 'origin/nightly-driver'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   package-lock.json
	modified:   package.json
	modified:   packages/arg-parser/package.json
	modified:   packages/e2e-tests/package.json
	modified:   packages/service-provider-core/package.json
	modified:   packages/service-provider-server/package.json
	modified:   packages/shell-api/package.json
	modified:   packages/types/package.json

no changes added to commit (use "git add" and/or "git commit -a")
~/mongo/mongosh % git diff package.json
diff --git a/package.json b/package.json
index f9e95a34..e060057f 100644
--- a/package.json
+++ b/package.json
@@ -102,7 +102,7 @@
     "find-up": "^5.0.0",
     "husky": "^8.0.3",
     "mocha": "^10.2.0",
-    "mongodb": "^6.3.0",
+    "mongodb": "nightly",
     "mongodb-runner": "^5.4.6",
     "node-gyp": "^9.0.0",
     "nyc": "^15.1.0",
~/mongo/mongosh % npm i
npm WARN deprecated [email protected]: This package is deprecated. Use Array.isArray.
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: acorn>=7.4 supports numeric separators
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142

> preinstall
> node scripts/sort-workspaces.js


> prepare
> husky install

husky - Git hooks installed

added 2637 packages, and audited 2666 packages in 48s

253 packages are looking for funding
  run `npm fund` for details

13 vulnerabilities (10 moderate, 3 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues possible (including breaking changes), run:
  npm audit fix --force

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.
~/mongo/mongosh % npm ls mongodb
mongosh@ /Users/leroux.bodenstein/mongo/mongosh
├─┬ @mongosh/[email protected] -> ./packages/arg-parser
│ ├─┬ @mongodb-js/[email protected]
│ │ └── [email protected] deduped
│ └── [email protected] deduped
├─┬ @mongosh/[email protected] -> ./packages/e2e-tests
│ └── [email protected] deduped
├─┬ @mongosh/[email protected] -> ./packages/service-provider-core
│ └── [email protected] deduped
├─┬ @mongosh/[email protected] -> ./packages/service-provider-server
│ └── [email protected] deduped
├─┬ @mongosh/[email protected] -> ./packages/shell-api
│ └── [email protected] deduped
├─┬ @mongosh/[email protected] -> ./packages/types
│ └── [email protected] deduped
├─┬ [email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     └── [email protected] deduped
└── [email protected]

yeah.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be good now.


# if we rewrote this script in javascript using just builtin node modules we could skip the npm ci above
npm run mark-ci-required-optional-dependencies

Expand Down
24 changes: 16 additions & 8 deletions scripts/replace-package.js
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
// Replace a package with symlinks to a specific directory.
// Example:
// REPLACE_PACKAGE=mongodb:/home/src/node-mongodb-native node scripts/replace-package.js
// will make 'mongodb' point to '/home/src/node-mongodb-native' in the root
// directory and all lerna packages.
// REPLACE_PACKAGE=mongodb:latest node scripts/replace-package.js
// will replace the 'mongodb' dep's version with latest in the root directory
// and all packages.
'use strict';
const fs = require('fs');
const path = require('path');
const { pathToFileURL } = require('url');
const { execSync } = require('child_process');

const replacement = process.env.REPLACE_PACKAGE;
if (!replacement) {
Expand All @@ -17,19 +16,28 @@ if (!parsed || !parsed.groups.from || !parsed.groups.to) {
throw new Error('Invalid format for REPLACE_PACKAGE');
}

const { from, to } = parsed.groups;
function resolveTag(from, to) {
return execSync(`npm dist-tag ls ${from}@${to} | awk -F ': ' '/^${to}/ {print \$2}'`).toString().trim();
lerouxb marked this conversation as resolved.
Show resolved Hide resolved
}

const { from, to: _to } = parsed.groups;

// npm install doesn't seem to do anything if you're updating a
// package-lock.json file that already has the dep to a tag like nightly, but it
// does do something if you change it to the exact version.
const to = _to === 'nightly' ? resolveTag(from, _to) : _to;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point you might as well just make the script driver-specific? :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thought did cross my mind 😆


for (const dir of ['.', ...fs.readdirSync('packages').map(dir => path.join('packages', dir))]) {
const packageJson = path.join(dir, 'package.json');
if (fs.existsSync(packageJson)) {
const target = pathToFileURL(path.resolve(to)).href;
const contents = JSON.parse(fs.readFileSync(packageJson));
for (const deps of [
contents.dependencies,
contents.devDependencies,
contents.optionalDependencies
]) {
if (deps && deps[from]) {
console.info('Replacing', from, 'in', dir, 'with', target);
console.info(`Replacing deps[${from}]: ${deps[from]} with ${to}`);
deps[from] = to;
}
}
Expand Down