Tests using ICollectionFixture are still retried when fixture initialize fails

[vedion]

Hi,

I have a case where an exception is thrown when initializing my fixture used in ICollectionFixture, ex:

[CollectionDefinition(nameof(MyCollectionFixture))]
public class MyCollectionFixture : ICollectionFixture<MyFixture>
{
}

public class MyFixture : IAsyncLifetime
{
   public async Task InitializeAsync()
   {
      throw new NullReferenceException("TEST");
   }
}

[Collection(nameof(MyCollectionFixture))]
public class MyTest
{
   public MyTest(MyFixture myFixture)
   {
       this.fixture = myFixture;
    }

    [RetryFact]
    public When_X_Then_Y()
    {
    }
}

I the above example MyFixture.InitializeAsync will be only be called once and the code in test "When_X_Then_Y" will be hit twice.

Would it be possible to look at "TestCollectionStarting" and "TestCollectionFinished" in MessageTransformer so the tests are not retried if the test failed between state "TestCollectionStarting" and "TestCollectionFinished":
https://github.com/xamarin/xunit/blob/master/src/xunit.execution/Sdk/Frameworks/Runners/TestCollectionRunner.cs

With the current code the exception in the fixture is hidden.

Best Regards,
Anders Havn

[JoshKeegan]

Hi Anders,
Thanks for reporting this, it isn't something I'd considered before now.

I wonder if we should also retry failures creating the fixture, that way a transient failure in the fixture creation would also be retried. I haven't yet looked into whether that would be possible, but would that work for your use-case?

[vedion]

Hi,

Thank you for the response. Yes, that would also work for my use-case :)