Unable to specify custom test adapter path

[07souravkunda]

Description

I am trying to create a custom test adapter which is intended to run before nunit3.testadapter. I have a requirement of executing tests with custom test adapter and for some conditions tests should run using nunit3.testadapter executor something like this.

I tried out the following solutions:

1. dotnet test --test-adapter-path C:\Adapters
2. vstest.console.exe dll_path /TestAdapterPath:"C:\Adapters"
3. Created a .runsettings file with RunConfiguration.TestAdaptersPaths node as given below and specified this runsettings path to dotnet test

<RunSettings>
  <RunConfiguration>
    <TestAdaptersPaths>C:\Adapters;</TestAdaptersPaths>
  </RunConfiguration>
</RunSettings>

but every time *.TestAdapter.dll from source dll directory is getting picked.

The custom test adapter should be picked in an ideal scenario, but this isn't working. Can someone please help me to achieve this?

Steps to reproduce

Expected behavior

Custom test adapter should be picked.

Actual behavior

*.TestAdapter.dll at source dll directory is picked.

Diagnostic logs

Environment

[MarcoRossignoli]

@Haplois do you think that the extension loading strategy you've implemented here #3380 could help for this use case?

[Haplois]

@MarcoRossignoli, the TestAdapterLoadingStrategy can be used to facilitate this. If you set it to Explicit, DefaultRuntimeProviders, it should only load test adapters under the given path and required runtime providers to run the test. If you also provide the runtime provider, then you can just use Explicit.

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 10 days.

[holbizmetrics]

Ever tried something like that?

   [ExtensionUri("executor://YourTestDiscoverer")]
   public class YourTestExecutor : ITestExecutor
   {
   }
 
    [ExtensionUri("executor://YourTestDiscoverer")]
    [DefaultExecutorUri("executor://YourTestDiscoverer")]
    [FileExtension(".dll")]
    [FileExtension(".exe")]
    [Category("managed")]
    public class YourTestDiscoverer : ITestDiscoverer
    {
    }

Because I just started to create from scratch and with some trial and error, this worked.
Now I got to work on the test discoverer implementation which is empty.

Thus, I get the message that no test is found which is correct so far.

HTH. :-)

[nohwnd]

@holbizmetrics here is a minimal implementation of a discoverer and executor, and a test framework if you need some reference. Biggest hurdle is getting the attributes right (you already got that down), and having the dll named *.TestAdapter.dll. https://github.com/nohwnd/intent

[holbizmetrics]

@nohwnd Yes. Meanwhile even the test appears in the Test Explorer in Visual Studio and one debug commands as well as the traits are working as well. The TestAdapter I already named correctly, but that was a coincidence.
Since I've called my Framework NextUnit, I called the TestAdapter was called NextUnit.TestAdapter. ;-)

What worked for me was this:

[ExtensionUri(DiscovererURI)] //[ExtensionUri("executor://NextUnitTestDiscoverer")]
DefaultExecutorUri(DiscovererURI)]//[DefaultExecutorUri("executor://NextUnitTestDiscoverer")]
[FileExtension(".dll")]
[FileExtension(".exe")]
[Category("managed")]
public class NextUnitTestDiscoverer : ITestDiscoverer
{
...

In the TestDiscoverer.

And in the TestExecutor:

/// <summary>
/// 
/// </summary>
[ExtensionUri(Definitions.DiscovererURI)] //[ExtensionUri("executor://NextUnitTestDiscoverer")]
public class NextUnitTestExecutor : NextUnitBaseExecutor, ITestExecutor
{
...

For anyone who may still encounter the same problem...

P.S.: I don't know why the formatting now looks like this considering the code. Tried to change that multiple times.
But I hope it's clearly readable anyway. Latest when you may post that in your IDE, hopefully.

[nohwnd]

Fixed your comment, you need to use ``` not ` to delimit your code when doing multiline code.

[holbizmetrics]

Firstly, to the original poster:

As far as I understand it is the normally behavior THAT *.TestAdapter.dll will be called.

*.TestAdapter.dll

But other hints have been added already.
And you guys may already even know more about this than me.

An additional info for anyone who's trying to debug a TestAdapter:

Until now, I could do the following:

/// <summary>
/// 
/// </summary>
/// <param name="tests"></param>
/// <param name="runContext"></param>
/// <param name="frameworkHandle"></param>
public void RunTests(IEnumerable<TestCase>? tests, IRunContext? runContext, IFrameworkHandle? frameworkHandle)
{
#if ADAPTER_TEST
	Debugger.Launch();
#endif
	var settings = runContext?.RunSettings?.SettingsXml;
	frameworkHandle.SendMessage(Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging.TestMessageLevel.Informational, "RunTests");
	foreach (TestCase test in tests)
	{
		// Example: Mark the start of the test
		frameworkHandle.RecordStart(test);

Which is:

Adding a Debugger.Launch() statement.
Thus, when the TestRunner was running for the executor, I could debug into the executor.

The same worked for the TestDiscoverer.
I don't know if that's the right way, but that's how I did it.

And it should also be possible to obtain an XUnit debuggable package, I guess.

A hint for now:

This worked flawlessly for quite some time, even with

Visual Studio 2022 version 17.9 Release

Now I got number 10, Visual Studio 2022 version 17.10 Release

Can't seem to make Debugger.Launch() appear/have an effect anymore like before.

Anyone encountering the same problem?

This seems to be broken now, in 17.10 preview. I'll verify at home on another computer. If it will be broken after updating as well it's pretty sure.

[nohwnd]

There is a built in collection of env variables that will wait for you to attach debugger to host or runner:

VSTEST_HOST_DEBUG=1
VSTEST_RUNNER_DEBUG=1

Or if you use our Playground, or build the AttachVS project that is part of our solution, and put the exe on PATH. Then it can attach automatically back to your VS. Either the one that started the execution, or one of your choosing:

VSTEST_RUNNER_DEBUG_ATTACHVS=1
VSTEST_HOST_DEBUG_ATTACHVS=1

If you set 1 in this env variable it will take the current process and search for parent VS. If you set any other positive number, it will try to find that process ID and will try to find parent ID.

https://github.com/microsoft/vstest/blob/main/src/Microsoft.TestPlatform.Execution.Shared/WellKnownDebugEnvironmentVariables.cs

[07souravkunda]

@MarcoRossignoli, the TestAdapterLoadingStrategy can be used to facilitate this. If you set it to Explicit, DefaultRuntimeProviders, it should only load test adapters under the given path and required runtime providers to run the test. If you also provide the runtime provider, then you can just use Explicit.

Sorry for the late reply. Before this suggestion, I had taken a different approach to solving this and lost track of this issue, which works fine for Windows. But it's not working for Unix systems. I was just renaming the test adapter files to achieve my use case. My adapter was named like 00*.adapter.dll, so it will always be picked first. Unfortunately, this only works with the Windows file system, not Mac and Linux.

I tried the above but was still not able to achieve this. It's always picking the nunit/xunit adapters over my adapter. The .runsettings file I used:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <RunConfiguration>
         <!-- <SkipDefaultAdapters>true</SkipDefaultAdapters>  -->
        <TestAdapterLoadingStrategy>Explicit,DefaultRuntimeProviders</TestAdapterLoadingStrategy>
        <!-- <TestAdapterLoadingStrategy>DefaultRuntimeProviders</TestAdapterLoadingStrategy> -->
        <TestAdaptersPaths>/path/to/xunit-browserstack/XUnit-BrowserStack/adapters/00Custom.TestAdapter.dll</TestAdaptersPaths>
    </RunConfiguration>
</RunSettings>

Any help is appreciated @Haplois .

[nohwnd]

We don't allow to specify order in which test adapters will load, nor we can guarantee it. The name migh work, but it also can be trumped by visual studio or other client that will send additional extension to the run etc.

[07souravkunda]

@nohwnd, is there any way to achieve this? Basically, I want my adapter(which only contains the test executor) to be used for Test Execution and the NUnit/Xunit's adapter for Test Discovery.

[nohwnd]

Only way I can see is that you grab the dlls of xunit and nunit adapters, ship them with your adapter (and rename them from *.TestAdapter.dll so vstest won't detect them). And then you delegate your discovery to the code that does discovery in xunit / nunit.

This way vstest is oblivious to the fact that you have xunit and nunit adapters installed, and only uses your adapter.

In other cases all registered executors will try to run tests.

[07souravkunda]

@nohwnd got it, but this will make the nunit/xunit adapter version static. If we can somehow use the discoverer class of nunit/xunit's adapter inside our adapter and remove it from the bin folder during execution, that should also work. Is there a way we can use other adapter's discoverer dynamically?

[nohwnd]

Yes you can do a lot of different hacks, but they don't sound like good maintainable solutions.

[07souravkunda]

I agree but it seems there is no straightforward way to do it. Anything working should be fine for now :\