Skip to content

Commit

Permalink
Add a script wrapper to run directly and update README
Browse files Browse the repository at this point in the history
Summary:
Add a script outside of testsuite package so we can run it directly.
Update the README in testsuite as well.

Differential Revision: D57984322
  • Loading branch information
lavenzg authored and facebook-github-bot committed May 30, 2024
1 parent e148f3b commit fb7df18
Show file tree
Hide file tree
Showing 3 changed files with 32 additions and 12 deletions.
13 changes: 13 additions & 0 deletions utils/test_runner.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/usr/bin/env fbpython
# Copyright (c) Meta Platforms, Inc. and affiliates.
#
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

import asyncio
import sys

from testsuite.cli import main

if __name__ == "__main__":
sys.exit(asyncio.run(main()))
27 changes: 19 additions & 8 deletions utils/testsuite/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,36 @@

## How to run

The runner (`cli.py`) requires two paths:
1. Path to the test suite directory that we support (i.e., test262, mjsunit, esprima, flow and a CVE testsuite). This can be
The runner requires two paths:
1. Path to the test suite directory that we support (i.e., test262, mjsunit,
esprima, flow and a CVE testsuite). This can be
a list separated by whitespace, so you can run multiple test suites at once.
2. Path to the directory of Hermes binaries, including `hermesc`, `hvm` (and `hermes`, `shermes` depending on extra flags).
2. Path to the directory of Hermes binaries, including `hermes` (and`shermes`
depending on extra flags).

Example usage:

```sh
# ./build is the cmake build directory.

# flow
./utils/testsuite2/cli.py external/flowtest/test/flow -b ./build/bin
./utils/test_runner.py external/flowtest/test/flow -b ./build/bin

# esprima
./utils/testsuite2/cli.py external/esprima/test_fixtures -b ./build/bin
.utils/test_runner.py external/esprima/test_fixtures -b ./build/bin
```

## Skiplist

Run with flag `--test-skiplist` to force running tests that are included in the `skiplist.json` config.

[TODO]
Run with flag `--test-skiplist` to force running tests that are included in the
`skiplist.json` config (but we will scan only "skiplist" and "manual_skiplist"
in the config, and "lazy_skip_list" if `--lazy` is given, "intl_tests" if
`--test-intl` is given).

After it finishes, all passed tests included in the skiplist config file will be
printed, and you can tell the runner to remove these tests from the config by
following the prompt ("manual_list" won't be touched). Note that if some tests
are covered by a folder path in the config, the test runner will first find all
tests under that folder, exclude those passed, and insert the rest into the
config file. If you don't want this behavior for a specific folder, please add
it to the manual list with a comment explaining the reason.
4 changes: 0 additions & 4 deletions utils/testsuite/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -452,7 +452,3 @@ async def main() -> int:
tmp_work_dir.cleanup()

return exit_code


if __name__ == "__main__":
sys.exit(asyncio.run(main()))

0 comments on commit fb7df18

Please sign in to comment.