Add a warning if default-members are configured and no package selection arguments are passed in

[milesj]

I've noticed something very weird recently. When running with cargo nextest run, the entire suite would pass with no issue:

Summary [  85.492s] 292 tests run: 292 passed, 0 skipped

While debugging individual tests, I started noticing tests that were "passing" started failing, but after looking into them, they were definitely broken and had to be fixed. I have no idea why nextest was passing. Were this just being ignored/filtered? Was nextest failing to capture/run them? No idea.

When running with cargo test --workspace, there are many failing tests. I'm unable to run the entire suite to determine an exact number without fixing them all (which I'm in the process of doing anyways). You can verify all of this in my branch here: https://github.com/moonrepo/moon/tree/develop-0.21

Things I've noticed that are failing:

• Tests that use #[serial] from serial_test.
• Tests using #[should_panic].

[sunshowers]

!!!! That's a super serious issue. I checked out your branch at commit 7f51e7b5188d8b95d60ebbf9cdb8880b7da569b1 and I don't seem to be able to reproduce this, though: cargo nextest run produced:

Summary [  84.746s] 292 tests run: 281 passed, 11 failed, 0 skipped

(I had to press Ctrl-C because moon_cli::run_node_test inherits_moon_env_vars was taking a long time, over 60 seconds).

This is with cargo-nextest 0.9.45 on Linux x86_64.

What version of cargo-nextest are you using and on what platform?

[sunshowers]

I did another test run and this is what I got:

     Summary [  11.585s] 292 tests run: 283 passed, 9 failed, 0 skipped
        FAIL [   0.005s]                      moon_cli helpers::test::setup_color::forced_color::forces_over_no_color
        FAIL [   0.004s]                      moon_cli helpers::test::setup_color::forced_color::forces_via_arg
        FAIL [   0.054s]         moon_cli::docker_test prune_node::focuses_for_pnpm
        FAIL [   0.038s]           moon_cli::node_test run_script::works_with_pnpm
        FAIL [   0.125s]       moon_cli::run_node_test pnpm::can_install_a_dep
        FAIL [   0.125s]       moon_cli::run_node_test pnpm::can_run_a_deps_bin_hoisted
        FAIL [   0.096s]       moon_cli::run_node_test pnpm::can_run_a_deps_bin_isolated
        FAIL [   0.108s]       moon_cli::run_node_test pnpm::can_run_a_script
        FAIL [   0.388s]     moon_cli::run_system_test unix::handles_ls

[sunshowers]

I also tried with cargo nextest run --workspace and got a much larger number of failing tests:

https://gist.github.com/sunshowers/9149ee5b7c113ae364b68b56430cbe48

Maybe it's just that you were missing the --workspace at the end? This is a well-known issue with cargo (happens while setting default-members) and one I've increasingly become unhappy about.

[milesj]

Maybe --workspace is it. This would explain the increase in bugs lately :[ I was under the assumption nextest did this automatically, but if not, giant whoops on my part!

For reference, here's a PR where I've been going through and fixing then: moonrepo/moon#492

[milesj]

Looks like --workspace fixed it, sorry for this dumb issue :P

If possible, it may be nice to display a warning if Cargo.toml is using workspaces but --workspaces is not passed.

[sunshowers]

Yeah, you're the third person in the last 2 weeks to run into this. I'm going to repurpose this as a feature request to add that warning.

I was under the assumption nextest did this automatically, but if not, giant whoops on my part!

As a general rule, cargo nextest run is equivalent to cargo test -- nextest doesn't add any embellishments to the step where Cargo builds tests, just the run step.

[milesj]

Good to know. Thanks for the quick response 👍

[sunshowers]

I started writing #701 but quickly realized it's a lot of work enumerating all the different cases. If you or someone else would like to do it, please comment here or over there!