New context-based state management model for allure-csharp

[delatrie]

This PR introduces a new model for managing the global state of allure in allure-csharp integrations.

Current implementation

Currently the global state of allure consists of the storage, the step context and the result accessor.

Key elements

Storage

The storage is a thread safe dictionary residing in Allure.Net.Commons/Storage/AllureStorage.cs. It maps a UUID to the corresponding allure item:

namespace Allure.Net.Commons.Storage
{
    internal class AllureStorage
    {
        private readonly ConcurrentDictionary<string, object> storage = new();
        
        /* The rest of the class */
    }
}

This is basically a global bag of allure items, accessed by their UUIDs. An integration that manipulates the lifecycle maintains these UUIDs somewhere and correlates them with objects of the test framework.

Step context

The step context introduces the notion of the current step container, i.e., the object (a test, a fixture, or another step) that becomes a parent of a newly created step without explicitly specified parent. It's defined as a thread safe dictionary mapping keys to a linked list of UUIDs. A key is typically an ID of the current thread, but it could be redefined to denote an object of a test framework. A linked list denotes a stack of the current steps with the current fixture/test as their root parent.

namespace Allure.Net.Commons.Storage
{
    internal class AllureStorage
    {
        /* The rest of the class */
        
        private readonly ConcurrentDictionary<string, LinkedList<string>> stepContext = new();
        private LinkedList<string> Steps => stepContext.GetOrAdd(
            AllureLifecycle.CurrentTestIdGetter(),
            new LinkedList<string>()
        );
        
        /* The rest of the class */
    }
}

Test result accessor

The test result accessor introduces the notion of the current test container and the current test in the current execution context. It resides in Allure.Net.Commons/Steps/CoreStepsHelper.cs as follows:

namespace Allure.Net.Commons.Steps
{
    public class CoreStepsHelper
    {
        private static readonly AsyncLocal<ITestResultAccessor> TestResultAccessorAsyncLocal = new();

        public static ITestResultAccessor TestResultAccessor
        {
            get => TestResultAccessorAsyncLocal.Value;
            set => TestResultAccessorAsyncLocal.Value = value;
        }

    /* The rest of the class */ 

Limitations

Concurrency issues

The current implementation of the step context has limitations on how it could be used in a concurrent environment:

1. It creates a way too strong isolation between threads.
2. It has no isolation of execution contexts that share the same thread.

The following concurrency issues could be observed:

1. Steps can't be created reliably from async continuations that cross thread boundaries:

   async public Task TestMethod1()
   {
       await Task.Delay(1); // paused; it might be continued in a context created for TestMethod2
       Steps.Step("Step in method 2", () => {}); // Throws if the continuation is scheduled in another thread where no step context exists; There might be a chance the step is added in a context of TestMethod2
   }
   
   async public Task TestMethod2()
   {
       await Task.Delay(1);
   }

2. Steps can't be created from a child thread:

   public void TestMethod()
   {
       Parallel.For(0, 2, i =>
       {
           Steps.Step($"Step #i", () => {}); // This throws
       });
   }

3. Steps can't be created reliably from subtasks

   async public Task TestMethod()
   {
       // May lead to an invalid steps structure if Step 1 is continued in a context modified by Step 2
       await Task.WhenAll(
           this.AddStep("Step 1"),
           this.AddStep("Step 2")
       );
   }
   
   async Task AddStep(string name) =>
       await Steps.Step(name, async () => await Task.Delay(0)); // Might be switched to another task after the step is started

See also #83, #286, #325, and #367.

Disparity between the provided API and its usage by test framework integrations

Typically, each integration uses AllureLifecycle's and CoreStepHelper's API to create and maintain the following four structures at each moment of time during the execution:

• the stack of test containers
• the current fixture
• the current test case
• the stack of steps

But the current API primarily requires the author of the integration to manually track UUIDs for most objects leading to unnecessary overhead.

For example, it should be sufficient to just call AllureLifecycle.WriteTestCase() to emit the current test result (i.e., the most recently started one), while currently authors of integrations are required to use AllureLifecycle.WriteTestCase(string uuid) instead. No allure-csharp integration use this method to emit a test result other than the most recently started one, yet they need to store this uuid somewhere and use it.

New implementation

The new implementation moves away from manual management of the global allure state to automatic management of Allure contextual data. We define Allure contextual data as consisting of the following four elements:

1. The stack of current containers, i.e., the container context.
2. The current fixture, i.e., the fixture context.
3. The current test case, i.e., the test context.
4. The stack of current step results, i.e., the step context.

These elements fully describe the state of Allure's execution at each moment of time.
Allure contextual data management is implemented in [Allure.Net.Commons/AllureContext.cs]. It's important to note that this class is immutable. Instances can be safely shared between multiple threads or tasks while remaining isolated.

ExecutionContext and AllureContext

Allure context is declared in Allure.Net.Commons/AllureLifecycle.cs as AsyncLocal of Allure contextual data:

namespace Allure.Net.Commons;
public class AllureLifecycle
{
    readonly AsyncLocal<AllureContext> context = new();
    public AllureContext Context
    {
        get => this.context.Value ??= new AllureContext();
        private set => this.context.Value = value;
    }

    /* The rest of AllureLifecycle */
}

Every AsyncLocal instance is controlled by ExecutionContext that in turn is controlled by .NET runtime to achieve a proper isolation between threads, tasks, async methods, etc. You may read more about ExecutionContext in [1].

Since Allure context is a part of ExecutionContext, it automatically has the following characteristics:

1. It's automatically captured by .NET when a new thread, or a task, or an async method is started.
2. It's automatically restored by .NET when a task continuation is schedules in some thread.
3. When an async function returns, it's automatically restored to the value captured just before the function has been called.

Natural context flow

An execution context of the current thread is captured by any newly started concurrent work, including but not limited to:

• a newly started thread,
• a work queued in a thread pool,
• a TPL's work unit (e.g., a delegate passed into Parallel.For),
• a newly started task,
• an async method call.

Once the context is captured, it becomes fully isolated from its parent. Changes made to the context aren't visible to other concurrent work besides those that in turn capture the changed context.

Manual context capturing

Since Allure context is a part of ExecutionContext, it naturally flows with it. However, sometimes the natural flow of the execution context isn't sufficient. Let's consider the following situation:

1. An async function named A calls an async function named B.
2. B creates a test result activating the test context (a part of Allure context).
3. B then returns and A continues its execution.
4. A tries to access the test context and fails because the execution context is restored including Allure context.

Here is the code to illustrate the problem:

public async Task A()
{
    await this.B();
    /*
     * Code that uses modifications applied by B
     * ...
    */
}

public async Task B()
{
    /*
     * Code that modifies Allure context
     * ...
    */

    await Task.CompletedTasl;
}

Here the test case context created in B isn't visible in A and can't be used because ExecutionContext is restored when B is exited.

To overcome this, capture Allure context manually and use the AllureLifecycle.RunInContext method in a manner, similar to how ExecutionContext.Run works:

public async Task A()
{
    var context = await this.B();
    AllureLifecycle.Instance.RunInContext(context, () => // Use the context captured by B
    {
        /*
         * Code that uses modifications applied by B
         * ...
        */
    });
}

public async Task<AllureContext> B()
{
    /*
     * Code that modifies Allure context
     * ...
    */

    return await Task.FromResult(AllureLifecycle.Context); // Captures Allure context and passes it to the caller
}

API changes

The new mechanism requires lots of API changes. Many previously available API members have lost their meaning and are now deprecated.

Allure context manipulation API

A new set of API methods should be used to manipulate Allure context. Refer to the documentation of a function in the list on what requirements it imposes upon Allure context and how it changes the context.

AllureLifecycle AllureLifecycle.StartTestContainer(TestResultContainer container);
AllureLifecycle AllureLifecycle.UpdateTestContainer(Action<TestResultContainer> update);
AllureLifecycle AllureLifecycle.StopTestContainer();
AllureLifecycle AllureLifecycle.WriteTestContainer();
AllureLifecycle AllureLifecycle.StartBeforeFixture(FixtureResult result);
AllureLifecycle AllureLifecycle.StartAfterFixture(FixtureResult result);
AllureLifecycle AllureLifecycle.UpdateFixture(Action<FixtureResult> update);
AllureLifecycle AllureLifecycle.StopFixture(Action<FixtureResult> beforeStop);
AllureLifecycle AllureLifecycle.StopFixture();
AllureLifecycle AllureLifecycle.StartTestCase(TestResult testResult);
AllureLifecycle AllureLifecycle.UpdateTestCase(Action<TestResult> update);
AllureLifecycle AllureLifecycle.StopTestCase(Action<TestResult> beforeStop);
AllureLifecycle AllureLifecycle.StopTestCase();
AllureLifecycle AllureLifecycle.WriteTestCase();
AllureLifecycle AllureLifecycle.StartStep(StepResult result);
AllureLifecycle AllureLifecycle.UpdateStep(Action<StepResult> update);
AllureLifecycle AllureLifecycle.UpdateStep(Action<StepResult> beforeStop);
AllureLifecycle AllureLifecycle.StopStep();
AllureLifecycle AllureLifecycle.AddScreenDiff(string expectedPng, string actualPng, string diffPng);

Allure context capturing API

The following API can be used to capture Allure context and to execute a custom code in the captured context. Refer to the documentation for more details.

AllureContext AllureLifecycle.Context { get; }
AllureContext AllureLifecycle.RunInContext(AllureContext? context, Action action);

See also the example above.

Deprecated API members

The following API members now does nothing and will be removed in one of the future releases:

Func<string>? AllureLifecycle.CurrentTestIdGetter { get; set; }
ITestResultAccessor? CoreStepsHelper.TestResultAccessor { get; set; }

Lots of API methods that require explicit uuid parameters are also deprecated:

AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartTestContainer(string parentUuid, TestResultContainer container);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.UpdateTestContainer(string uuid, Action<TestResultContainer> update);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StopTestContainer(string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.WriteTestContainer(string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartBeforeFixture(FixtureResult result, out string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartBeforeFixture(string uuid, FixtureResult result);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartBeforeFixture(string parentUuid, FixtureResult result, out string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartBeforeFixture(string parentUuid, string uuid, FixtureResult result);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartAfterFixture(string parentUuid, FixtureResult result, out string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartAfterFixture(string parentUuid, string uuid, FixtureResult result);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.UpdateFixture(string uuid, Action<FixtureResult> update);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StopFixture(string uuid)
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartTestCase(string containerUuid, TestResult testResult);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.UpdateTestCase(string uuid, Action<TestResult> update);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StopTestCase(string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.WriteTestCase(string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartStep(StepResult result, out string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartStep(string uuid, StepResult result);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StartStep(string parentUuid, string uuid, StepResult stepResult);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.UpdateStep(string uuid, Action<StepResult> update);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.StopStep(string uuid);
AllureLifecycle Allure.Net.Commons.AllureLifecycle.AddScreenDiff(string testCaseUuid, string expectedPng, string actualPng, string diffPng);
void Allure.Net.Commons.Steps.CoreStepsHelper.StopFixtureSuppressTestCase(Action<FixtureResult> updateResults = null);
void Allure.Net.Commons.Steps.CoreStepsHelper.PassStep(string uuid, Action<StepResult>? updateResults = null);
void Allure.Net.Commons.Steps.CoreStepsHelper.FailStep(string uuid, Action<StepResult>? updateResults = null);
void Allure.Net.Commons.Steps.CoreStepsHelper.BrokeStep(string uuid, Action<StepResult>? updateResults = null);
void NUnit.Allure.Core.AllureExtensions.AddScreenDiff(AllureLifecycle lifecycle, string expected, string actual, string diff);

Authors that use these methods should migrate to the new Allure context API described above.

Other changes

AllureLifecycle thread safety

Potentially concurrent access to instances of Allure object model in AllureLifecycle is now guarded with a lock to prevent data races in multi-threaded environments.

NUnit OneTimeSetUp fixtures support

Currently the support of NUnit's OneTimeSetUp fixtures is broken in the preview branch. With this PR the support is back:

Allure.SpecFlowPlugin failed feature hook placeholder

Now, if a feature-level hook throws an exception, a broken placeholder scenario is created in the report instead of marking all passed scenarios as broken (in case of an AfterFeature hook).
Before this change such a placeholder had been created in case of a failed BeforeFeature hook only.

C# language version

Each project in this repo now explicitly specifies its language version. For now, it's C# 11.

Null checking

We slowly phase in null checking in the repo. Currently, null checking is enabled globally for the following projects:

• Allure.SpecFlowPlugin

Null checking is enabled on a per file basis for the following files:

• Allure.Net.Commons/AllureLifecycle.cs
• Allure.Net.Commons/AllureContext.cs
• Allure.Net.Commons/Storage/AllureStorage.cs
• Allure.Net.Commons/Steps/CoreStepsHelper.cs
• Allure.XUnit/AllureMessageSink.cs
• Allure.XUnit/AllureXunitHelper.cs

We will continue to embrace this feature until all the code base if affected.

Fixes #83
Fixes #367

Checklist

• Sign Allure CLA
• Provide unit tests

[delatrie]

Thanks for the review, @neparij !

I've also discovered a bug with OneTimeSetUp fixture popping up in every test in the report regardless on whether the test has one or not.
The PR is already quite big, and this bug is most likely present in an existing preview version, although has been hidden by the issue with non-working fixtures until now. So, I've split it out into a new issue #374.