test: twister: testplan.py use normpath #64272

hakehuang · 2023-10-23T16:50:12Z

when load test plan it is possible the plan is built in another os so the case key would be
'samples/hello_world/samples.baseic.hello_world'
but the testsuite is scaned in current os may in window and the key is like
'samples\hello_world\samples.baseic.hello_world'

use ps.path.normpath, will solve this.

gchwier · 2023-10-25T08:49:49Z

scripts/pylib/twister/twisterlib/testplan.py

@@ -562,7 +562,7 @@ def load_from_file(self, file, filter_platform=[]):
            instance_list = []
            for ts in jtp.get("testsuites", []):
                logger.debug(f"loading {ts['name']}...")
-                testsuite = ts["name"]
+                testsuite = os.path.normpath(ts["name"])


It will work when you generate plan on Linux and load it on Windows:

>>> os.path.normpath('twister-out/twister.log') 'twister-out\\twister.log'

When you generate plan on Windows and try to load it on Linux, it will not work:

>>> os.path.normpath('twister-out\\twister.log') 'twister-out\\twister.log'

IMO testcase name should be also changed in TestSuite class (e.g. to use only / separator).
Or, if we do not need to support that case (generate test plan on Windows and load on Linux), then this change will be enough.

strange, why I get different result, see my log

`
ubuntu@ubuntu-OptiPlex-7050:/home/shared/disk/zephyr_project/zephyr_test/zephyr$ ipython3
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
Type 'copyright', 'credits' or 'license' for more information
ubuntu@ubuntu-OptiPlex-7050:~$ ipython3
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.31.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import os

In [2]: print(os.path.normpath('twister-out\twister.log'))
twister-out\twister.log

In [3]:

`

My first log was from Windows, second from Linux. When you use path with backslashes (generated in Windows) and test it on Linux, then you still have backslashes in the result.

@gchwier thank a lot to point this scenario out, I add a check as below to specially handle it

if testsuite not in self.testsuites: # this could be a path issue in testplan and run envrionment # try convert it # so far we know window plan to linux envirnmont has such problem if "\\" in testsuite: testsuite = testsuite.replace("\\", "/") else: logger.warning(f"{testsuite} is not supported") continue

gchwier

With those changes it should works, but this is rather a workaround. I suggest to use in testplan only / separator, because it works also on Windows. Best place (IMO) is to fix it in TestSuite class, where name is generated:
See:
twisterlib/testsuite.py#L453
and use:
unique = os.path.normpath(os.path.join(relative_ts_root, workdir, name)).replace(os.sep, '/')
But it should be tested.
You could also add some unit test for that change :)

scripts/pylib/twister/twisterlib/testplan.py

hakehuang · 2023-11-01T09:45:28Z

But it should be tested.
You could also add some unit test for that change :)

I test manually, it works, however as we are uing os.sep this binds to the system we are running, do we have a windows CI? if not, we can not test it

scripts/pylib/twister/twisterlib/testsuite.py

scripts/pylib/twister/twisterlib/testplan.py

hakehuang · 2023-11-07T03:55:50Z

@gchwier @nashif can you review the change, it is ok per our testing environment, and the CI failure is fixed in #64551, as the testrunner case check the line number, any line change in testrunner.py will cause CI failure

gchwier · 2023-11-07T09:30:59Z

@gchwier @nashif can you review the change, it is ok per our testing environment, and the CI failure is fixed in #64551, as the testrunner case check the line number, any line change in testrunner.py will cause CI failure

Please move changes related to the testrunner tests from #64551 to separate PR (it will be merged faster and fix tests also here)

hakehuang · 2023-11-08T05:23:21Z

Please move changes related to the testrunner tests from #64551 to separate PR (it will be merged faster and fix tests also here)

@gchwier I create PR #64947, will update this PR once merged

when load test plan it is possible the plan is built in another os so the case key would be 'samples/hello_world/samples.baseic.hello_world' but the testsuite is scaned in current os may in window and the key is like 'samples\\hello_world\\samples.baseic.hello_world' so update the uniq path with only backslash in path Signed-off-by: Hake Huang <[email protected]>

glenn-andrews · 2024-03-15T04:32:57Z

This is breaking something on Windows.

When I try and run tests in an out-of-tree directory it no longer finds the test suite:

(.venv) PS C:\Users\glenn\zephyrproject\my_app> west twister --device-testing --device-serial COM3 --device-serial-baud 115200 -p disco_l475_iot1 -T C:\Users\glenn\zephyrproject\my_app\ -s tests\app\sample.testing.my_app_tests
Renaming output directory to C:\Users\glenn\zephyrproject\my_app\twister-out.18
INFO    - Using Ninja..
INFO    - Zephyr version: zephyr-v3.5.0-1349-gdbed25124906
INFO    - Using 'zephyr' toolchain.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\glenn\zephyrproject\.venv\Scripts\west.exe\__main__.py", line 7, in <module>
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 1085, in main
    app.run(argv or sys.argv[1:])
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 244, in run
    self.run_command(argv, early_args)
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 505, in run_command
    self.run_extension(args.command, argv)
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 654, in run_extension
    self.cmd.run(args, unknown, self.topdir, manifest=self.manifest,
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\commands.py", line 194, in run
    self.do_run(args, unknown)
  File "C:\Users\glenn\zephyrproject\zephyr\scripts\west_commands\twister_cmd.py", line 60, in do_run
    ret = main(options)
          ^^^^^^^^^^^^^
  File "C:\Users\glenn\zephyrproject\zephyr\scripts\pylib\twister\twisterlib\twister_main.py", line 120, in main
    tplan.discover()
  File "C:\Users\glenn\zephyrproject\zephyr\scripts\pylib\twister\twisterlib\testplan.py", line 180, in discover
    raise TwisterRuntimeError("No test cases found at the specified location...")
twisterlib.error.TwisterRuntimeError: No test cases found at the specified location...

hakehuang · 2024-03-15T11:32:50Z

This is breaking something on Windows.

When I try and run tests in an out-of-tree directory it no longer finds the test suite:

(.venv) PS C:\Users\glenn\zephyrproject\my_app> west twister --device-testing --device-serial COM3 --device-serial-baud 115200 -p disco_l475_iot1 -T C:\Users\glenn\zephyrproject\my_app\ -s tests\app\sample.testing.my_app_tests
Renaming output directory to C:\Users\glenn\zephyrproject\my_app\twister-out.18
INFO    - Using Ninja..
INFO    - Zephyr version: zephyr-v3.5.0-1349-gdbed25124906
INFO    - Using 'zephyr' toolchain.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\glenn\zephyrproject\.venv\Scripts\west.exe\__main__.py", line 7, in <module>
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 1085, in main
    app.run(argv or sys.argv[1:])
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 244, in run
    self.run_command(argv, early_args)
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 505, in run_command
    self.run_extension(args.command, argv)
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\app\main.py", line 654, in run_extension
    self.cmd.run(args, unknown, self.topdir, manifest=self.manifest,
  File "C:\Users\glenn\zephyrproject\.venv\Lib\site-packages\west\commands.py", line 194, in run
    self.do_run(args, unknown)
  File "C:\Users\glenn\zephyrproject\zephyr\scripts\west_commands\twister_cmd.py", line 60, in do_run
    ret = main(options)
          ^^^^^^^^^^^^^
  File "C:\Users\glenn\zephyrproject\zephyr\scripts\pylib\twister\twisterlib\twister_main.py", line 120, in main
    tplan.discover()
  File "C:\Users\glenn\zephyrproject\zephyr\scripts\pylib\twister\twisterlib\testplan.py", line 180, in discover
    raise TwisterRuntimeError("No test cases found at the specified location...")
twisterlib.error.TwisterRuntimeError: No test cases found at the specified location...

can you raise an issue?

PerMac · 2024-03-15T11:41:53Z

I am not sure if this is a bug nor windows specific. Twister by default creates tests id including the path name in it. The problem is that it looks for the path as relative to zephyr dir. If it is oot, then the path depends on where it starts scaning, hence with different level of -T it gets different names hence -s doesn't match any more. I found it annoying and tried to fix, but this caused issues in other places. To resolve this problem I added --no-detailed-test-id flag. It removes paths from the names and hence allow to always get the same name for oot tests and match them easily.

In your case please try modifying the last part of your command to -s sample.testing.my_app_tests --no-detailed-test-id and let me know if that worked.

PerMac · 2024-03-15T11:43:11Z

btw at windows you might also need --short-build-path. This flag solves the issue with path lengths limitation on windows

glenn-andrews · 2024-03-15T16:12:43Z

I am not sure if this is a bug nor windows specific. Twister by default creates tests id including the path name in it. The problem is that it looks for the path as relative to zephyr dir. If it is oot, then the path depends on where it starts scaning, hence with different level of -T it gets different names hence -s doesn't match any more. I found it annoying and tried to fix, but this caused issues in other places. To resolve this problem I added --no-detailed-test-id flag. It removes paths from the names and hence allow to always get the same name for oot tests and match them easily.

In your case please try modifying the last part of your command to -s sample.testing.my_app_tests --no-detailed-test-id and let me know if that worked.

It's the backslashes in the path to the test. They changed the direction.

-s tests\app\sample.testing.my_app_tests fails
-s tests/app/sample.testing.my_app_tests works

glenn-andrews · 2024-03-15T16:22:23Z

can you raise an issue?

#70310

hakehuang requested a review from nashif as a code owner October 23, 2023 16:50

hakehuang requested review from tejlmand and PerMac October 23, 2023 16:50

zephyrbot added Trivial Changes that can be reviewed by anyone, i.e. doc changes, minor build system tweaks, etc. area: Twister Twister labels Oct 23, 2023

zephyrbot requested review from galak, gchwier, golowanow and gopiotr October 23, 2023 16:50

zephyrbot assigned nashif Oct 23, 2023

gchwier reviewed Oct 25, 2023

View reviewed changes

hakehuang requested a review from gchwier October 31, 2023 09:57

gchwier reviewed Oct 31, 2023

View reviewed changes

scripts/pylib/twister/twisterlib/testplan.py Outdated Show resolved Hide resolved

scripts/pylib/twister/twisterlib/testplan.py Outdated Show resolved Hide resolved

scripts/pylib/twister/twisterlib/testplan.py Outdated Show resolved Hide resolved

hakehuang requested a review from gchwier November 1, 2023 10:04

gchwier previously approved these changes Nov 2, 2023

View reviewed changes

scripts/pylib/twister/twisterlib/testsuite.py Show resolved Hide resolved

nashif removed the Trivial Changes that can be reviewed by anyone, i.e. doc changes, minor build system tweaks, etc. label Nov 2, 2023

nashif reviewed Nov 3, 2023

View reviewed changes

scripts/pylib/twister/twisterlib/testplan.py Outdated Show resolved Hide resolved

hakehuang dismissed gchwier’s stale review via 3bb7d7e November 3, 2023 15:53

zephyrbot added the Trivial Changes that can be reviewed by anyone, i.e. doc changes, minor build system tweaks, etc. label Nov 5, 2023

hakehuang requested review from nashif and gchwier November 7, 2023 03:50

hakehuang mentioned this pull request Nov 8, 2023

test: twister: test_runner remove line number check in log #64947

Merged

gchwier approved these changes Nov 8, 2023

View reviewed changes

gopiotr approved these changes Nov 8, 2023

View reviewed changes

nashif approved these changes Nov 10, 2023

View reviewed changes

carlescufi merged commit dbed251 into zephyrproject-rtos:main Nov 13, 2023
26 checks passed

glenn-andrews mentioned this pull request Mar 15, 2024

test: twister: -s path requires forward slashes on Windows, didn't before. #70310

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: twister: testplan.py use normpath #64272

test: twister: testplan.py use normpath #64272

hakehuang commented Oct 23, 2023 •

edited

Loading

gchwier Oct 25, 2023

hakehuang Oct 25, 2023 •

edited

Loading

gchwier Oct 25, 2023

hakehuang Oct 26, 2023 •

edited

Loading

gchwier left a comment •

edited

Loading

hakehuang commented Nov 1, 2023

hakehuang commented Nov 7, 2023

gchwier commented Nov 7, 2023

hakehuang commented Nov 8, 2023

glenn-andrews commented Mar 15, 2024

hakehuang commented Mar 15, 2024

PerMac commented Mar 15, 2024

PerMac commented Mar 15, 2024

glenn-andrews commented Mar 15, 2024

glenn-andrews commented Mar 15, 2024

test: twister: testplan.py use normpath #64272

test: twister: testplan.py use normpath #64272

Conversation

hakehuang commented Oct 23, 2023 • edited Loading

gchwier Oct 25, 2023

Choose a reason for hiding this comment

hakehuang Oct 25, 2023 • edited Loading

Choose a reason for hiding this comment

gchwier Oct 25, 2023

Choose a reason for hiding this comment

hakehuang Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

gchwier left a comment • edited Loading

Choose a reason for hiding this comment

hakehuang commented Nov 1, 2023

hakehuang commented Nov 7, 2023

gchwier commented Nov 7, 2023

hakehuang commented Nov 8, 2023

glenn-andrews commented Mar 15, 2024

hakehuang commented Mar 15, 2024

PerMac commented Mar 15, 2024

PerMac commented Mar 15, 2024

glenn-andrews commented Mar 15, 2024

glenn-andrews commented Mar 15, 2024

hakehuang commented Oct 23, 2023 •

edited

Loading

hakehuang Oct 25, 2023 •

edited

Loading

hakehuang Oct 26, 2023 •

edited

Loading

gchwier left a comment •

edited

Loading