Minitest::Verify

Avoid false negative tests by verifying they fail when key setup is removed.

This is a quick proof-of-concept minitest plugin, but it mostly works fine!

Note

See this lightning talk for an example of how false negative tests are a problem, why catching them programmatically might not work out, and some more practical advice for writing more robust tests in the first place. You probably want to do that before using this gem, but feel free to try it out anyway!

Installation

Install the gem and add to the application's Gemfile by executing:

$ bundle add minitest-verify

If bundler is not being used to manage dependencies, install the gem by executing:

$ gem install minitest-verify

Usage

This is a false negative test. It always passes because post and comment are completely unrelated: there's no reason post.comments would ever include comment.

require "minitest/autorun"

class PostTest < Minitest::Test
  def test_comments_excludes_hidden_comments
    post = create(:post)
    comment = create(:comment, hidden: true)

    refute_includes post.comments, comment
  end
end

We can pull out the key setup and wrap it with verify_fails_without:

require "minitest/autorun"
require "minitest/verify"

class PostTest < Minitest::Test
  include Minitest::Verify

  def test_comments_excludes_hidden_comments
    post = create(:post)
    comment = create(:comment)

    verify_fails_without { comment.update!(hidden: true) }

    refute_includes post.comments, comment
  end
end

Now run the test with the --verify argument:

$ ruby post_test.rb --verify

This will cause the test to run twice. First it runs with the contents of the verify_fails_without block evaluated (normal run). Then it runs without the contents of the verify_fails_without block evaluated (verification run). If the test still passes without having evaluated the code inside the block, it's a false negative and you'll see a verification failure in your test output:

# Running:

F

Finished in 0.000380s, 2631.5783 runs/s, 5263.1565 assertions/s.

  1) Failure:
PostTest#test_comments_excludes_hidden_comments [post_test.rb:11]:
Expected at least one assertion to fail.

1 runs, 2 assertions, 1 failures, 0 errors, 0 skips