Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conditionally mark the test cfg as a well known cfg #15007

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conditionally mark the test cfg as a well known cfg #15007

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
208f817
Add preparatory tests for `lib.test = false` check-cfg changes
Urgau Jan 2, 2025
0f92118
Conditionally mark the `test` cfg as a well known cfg
Urgau Jan 2, 2025
eb3c3a9
Add stderr checking for the conditional `test` well known cfg
Urgau Jan 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 74 additions & 11 deletions tests/testsuite/check_cfg.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,7 +317,7 @@ fn well_known_names_values_doctest() {
.run();
}

#[cargo_test]
#[cargo_test(nightly, reason = "warning currently only on nightly")]
fn test_false_lib() {
let p = project()
.file(
Expand All @@ -332,26 +332,57 @@ fn test_false_lib() {
test = false
"#,
)
.file("src/lib.rs", "")
.file("src/lib.rs", "#[cfg(test)] mod tests {}")
.build();

p.cargo("check -v")
.with_stderr_does_not_contain(x!("rustc" => "cfg" of "docsrs,test"))
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs"))
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (lib) generated 1 warning
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s

"#]])
.run();

p.cargo("clean").run();
p.cargo("test -v")
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs"))
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (lib) generated 1 warning
[FINISHED] `test` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
[DOCTEST] foo
[RUNNING] [..]

"#]])
.run();

p.cargo("clean").run();
p.cargo("test --lib -v")
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs"))
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If user has specified --lib, doesn't it mean that they requested to test the lib crate so cfg(test) is expected to be there?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What the command line arguments are shouldn't matter, otherwise users would get a different behavior between a "normal" cargo test and one with arguments like --all-targets.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, lib.test = false doesn't mean there is no test. It, as of today, is defined as "test" won't run by default. cfg(test) is still relevant regardless if a test is in the default test set.

Maybe I should put the question this way: Why is cfg(test) in a test=false crate unexpected?

Copy link
Member Author

@Urgau Urgau Jan 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO having test = false meaning not tested by default compared to not tests at all is unexpected, at least before reading the documentation I would have assumed that it disabled all the testing.

@jplatte made a further argument in his comment:

Hm, maybe. I guess it depends on what the intention of lib.test = false is. I think the vast majority of projects that set it don't have unit tests at all, and if any were to be introduced, getting a warning would be valuable (whether it results in the test author removing the flag or moving the test code into unit tests). I wonder if there are any projects that use lib.test = false while having unit tests, intentionally.

A similar comment was also put in the URLO thread.

Copy link
Member

@weihanglo weihanglo Jan 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for linking the thread here. Yeah I've read that, and agree that lib.test = false largely implies no test in lib crate at all. That is a valid use case and deserves a lint or warning.

However, fields under Cargo targets are mostly about including in or excluding from the default crate set, and are largely affected by the command-line argument. If Cargo has started emitting unexpected warning for Cargo targets, it may also bite users that disabling tests for other reasons. For example, I've seen people setting test=false because some tests are not expected to run by default for daily development, but only on CI or other special environments.

While that kind of scenario I guess is far less than who don't want tests at all, it is still a valid use case aligning with what the current doc describes. Treating unexpected_cfgs as an indirect way of banning tests in a crate doesn't seem to be the best venue. It is more like a hack because there is no lint for really banning tests.

That being said, I am not surprised if Cargo targets settings don't meet people's expectation. It is always a source of confusions1 😞.

Footnotes

  1. I can actually name a lot of issues here. Let's just have a little peek of them:
    * https://github.com/rust-lang/cargo/issues/8338
    * https://github.com/rust-lang/cargo/issues/10936
    * https://github.com/rust-lang/cargo/issues/10958
    * https://github.com/rust-lang/cargo/issues/13668
    * https://github.com/rust-lang/cargo/issues/13437
    * https://github.com/rust-lang/cargo/issues/13828 ↩

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current behavior is also unexpected and undesirable to me, I agree with the quote and linked comment. I've been keeping an eye on this series of PRs (thanks, @Urgau!) because I'd like to start using lib.test = false for crates that don't have unit tests without risking the silent failure mode mentioned by @jplatte.

For tests that shouldn't run by default, #[ignore] is a more discoverable and more robust choice. Ignored tests are at least called out as such in libtest output, as long as the test binary containing them is built and run. In contrast, cfg(test) code that is never compiled or run at all is mostly invisible. Even if you get in the habit of cargo test --all-targets, you'll have a similar problem with doctests not being run any more! Examples with #[test]s have this problem by default, which the documentation highlights as a pitfall to avoid. The unexpected-cfg warning added by this PR also calls attention to cases where this has been missed.

If any project intentionally uses test = false for crates with unit tests in them, they can still turn off the unexpected-cfg warning by adding cfg(test) to the allowlist in [lints.rust] or build script output. So it's not like this change would make it fundamentally harder to use test = false for tests not built by default -- it just adds the ability to distinguish that usage from "not intended to contain any tests at all".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify a bit, the proposed behavior may be worth for lib.test = false, as it often implies no unit tests. For test.test = false it’s a bit harder to say. We could diverge the logic for different target kinds, but I think nobody wants another confusions added to Cargo targets 😓.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it just adds the ability to distinguish that usage from "not intended to contain any tests at all".

My argument is that this may deserve an unexpected_tests lint. Relying on unexpected_cfgs only partially resolves the issue because people can still have #[test] without any cfg(test).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's a somewhat indirect way of achieving the goal, and it would be nice to also warn on bare #[test] not nested under an explicit cfg(test). Clippy has a lint for that, but ironically it only fires when you run cargo clippy -- --test because otherwise these items are stripped out before Clippy can see them. For the same reason, a dedicated unexpected_tests lint would also have to warn on any occurrence of cfg(... test ...), including nested inside complex cfg expressions, just like unexpected_cfgs would after this PR. I'd also expect a warning for #[cfg_attr(... test ..., blah)] in a crate where tests are "unexpected". So I guess unexpected_tests would be identical to unexpected_cfgs plus warning on #[test]?

But it also wouldn't be a bad solution to just treat bare #[test] as if it came with an implied cfg(test) and emit the same diagnostic if cfg(test) is unexpected. I don't know if that's implementable without unduly complicating unexpected_cfgs and/or #[test] expansion, but it seems semantically appropriate: it matches how #[test] is expanded, except for the subtle difference between rustc --test and rustc --cfg test (that nobody should be relying on). Tailored diagnostics that talk about unit tests in particular can be achieved without an entire new lint. The main thing that's lost is the ability to toggle the lint independently of other unexpected_cfgs warnings, but that seems strange when cfg(... test ...) is part of what the lint would warn about.

Copy link
Member Author

@Urgau Urgau Jan 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unexpected_cfgs lint do trigger on #[test] but only when passed with --test.

$ rustc +nightly --crate-type=lib --test --check-cfg='cfg()' src/lib.rs
warning: unexpected `cfg` condition name: `test`
 --> src/lib.rs:9:5
  |
9 |     #[test]
  |     ^^^^^^^
  |
  = help: expected names are: `clippy`, `debug_assertions`, `doc`, `doctest`, `fmt_debug`, `miri`, `over
flow_checks`, `panic`, `proc_macro`, `relocation_model`, `rustfmt`, `sanitize`, `sanitizer_cfi_generaliz
e_pointers`, `sanitizer_cfi_normalize_integers`, `target_abi`, `target_arch`, `target_endian`, `target_e
nv`, `target_family`, `target_feature`, `target_has_atomic`, `target_has_atomic_equal_alignment`, `target_has_atomic_load_store`, `target_os`, `target_pointer_width`, `target_thread_local`, `target_vendor`, `ub_checks`, `unix`, and `windows`
  = note: using a cfg inside a attribute macro will use the cfgs from the destination crate and not the ones from the defining crate
  = help: try referring to `test` crate for guidance on how handle this unexpected cfg
  = help: to expect this configuration use `--check-cfg=cfg(test)`
  = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg.html> for more information about checking conditional configuration
  = note: `#[warn(unexpected_cfgs)]` on by default
  = note: this warning originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

warning: 1 warning emitted

We can probably make the lint fire without the --test argument.

--> src/lib.rs:1:7
...

[WARNING] `foo` (lib test) generated 1 warning
[FINISHED] `test` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
[RUNNING] `[ROOT]/foo/target/debug/deps/foo-[HASH][EXE]`

"#]])
.run();
}

#[cargo_test]
#[cargo_test(nightly, reason = "warning currently only on nightly")]
fn test_false_bins() {
let p = project()
.file(
Expand All @@ -368,17 +399,26 @@ fn test_false_bins() {
path = "src/deamon.rs"
"#,
)
.file("src/main.rs", "fn main() {}")
.file("src/deamon.rs", "fn main() {}")
.file("src/main.rs", "fn main() {}\n#[cfg(test)] mod tests {}")
.file("src/deamon.rs", "fn main() {}\n#[cfg(test)] mod tests {}")
.build();

p.cargo("check -v")
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs,test")) // for foo
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs")) // for deamon
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (bin "daemon") generated 1 warning
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s

"#]])
.run();
}

#[cargo_test]
#[cargo_test(nightly, reason = "warning currently only on nightly")]
fn test_false_examples() {
let p = project()
.file(
Expand All @@ -398,17 +438,34 @@ fn test_false_examples() {
path = "src/deamon.rs"
"#,
)
.file("src/lib.rs", "")
.file("src/deamon.rs", "fn main() {}")
.file("src/lib.rs", "#[cfg(test)] mod tests {}")
.file("src/deamon.rs", "fn main() {}\n#[cfg(test)] mod tests {}")
.build();

p.cargo("check --examples -v")
.with_stderr_does_not_contain(x!("rustc" => "cfg" of "docsrs,test"))
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs"))
.with_stderr_data(str![[r#"
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (lib) generated 1 warning
...
[WARNING] unexpected `cfg` condition name: `test`
...

[WARNING] `foo` (example "daemon") generated 1 warning
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s

"#]])
.run();
}

#[cargo_test(nightly, reason = "bench is nightly")]
#[cargo_test(
nightly,
reason = "bench is nightly & warning currently only on nightly"
)]
fn test_false_benches() {
let p = project()
.file(
Expand Down Expand Up @@ -436,8 +493,14 @@ fn test_false_benches() {
)
.build();

p.cargo("bench --bench ben1 -v")
.with_stderr_contains(x!("rustc" => "cfg" of "docsrs,test"))
// Benches always require the `test` cfg, there should be no warning.
p.cargo("bench --bench ben1")
.with_stderr_data(str![[r#"
[COMPILING] foo v0.0.0 ([ROOT]/foo)
[FINISHED] `bench` profile [optimized] target(s) in [ELAPSED]s
[RUNNING] benches/ben1.rs (target/release/deps/ben1-[HASH][EXE])

"#]])
.run();
}

Expand Down
Loading