Skip to content

Commit

Permalink
Reorganise tests (#167)
Browse files Browse the repository at this point in the history
This PR reorganises the `tests/` directory:

- move everything under `tests/golden/` (in preparation for
`tests/unit/`)
- group test files into a common directory

A side effect is that some tests that were previously not detected by
the runner now are. There is also strange cases of tests for which we
have an expectation file but no input file. I will try to figure that
out later.
  • Loading branch information
Niols authored May 10, 2023
1 parent 87e347c commit 8e75944
Show file tree
Hide file tree
Showing 326 changed files with 41 additions and 46 deletions.
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ uninstall: build
dune uninstall $(INSTALL_ARGS)

check: build
@ output=$$(./tests/run 2>&1) ; \
@ output=$$(./tests/golden/run 2>&1) ; \
status=$$? ; \
echo "$$output" | tee tests.org ; \
exit $$status
Expand All @@ -48,7 +48,7 @@ clean:
dune clean
rm -f bin lib doc
rm -f tests.org
tests/run clean || true
tests/golden/run clean || true

PACKAGE=$(shell echo morbig-`cat VERSION`)

Expand Down
33 changes: 16 additions & 17 deletions tests/README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,28 @@
Morbig's Test Suite
===================
# Morbig's Test Suite

Directories
-----------
## Golden tests

- `good/`: these adhere to the POSIX standard, and should be accepted.
Those are found in the `tests/golden/` directory.

- `bad/`: these do not adhere to the standard, and should be rejected.
### Directories

- `unspecified/`: cases where the standard says that the behavior is unspecified
This test suite contains four top-level directories corresponding to four
categories of tests, namely:

- `good/`: these adhere to the POSIX standard, and should be accepted.
- `bad/`: these do not adhere to the standard, and should be rejected.
- `unspecified/`: cases where the standard says that the behavior is unspecified
- `unknown/`: cases where we could not figure out whether there are good, bad,
or unspecified.

Good tests contain an `.expected` result, to which the output of Morbig will be
compared. A good test is said to yield un `unexpected` result if Morbig succeeds
but the output is not what was expected. A good test is said to fail if Morbig
fails.

Good and bad tests may contain an `.open` file that specifies that the test in
question is “open”, that is there is an open issue corresponding to it. Those
tests may fail or succeed without making the whole test suite fail or succeed.
A test is a directory whose name finishes in `.t`. Such tests can be nested as
subdirectories of the four top-level ones at an arbitrary depth. Test
directories contain an input file `input.sh`. Good tests contain an additional
expectations file `expected.json`. Finally, tests may contain an `open` file
that specifies that the test in question has an open issue attached to it; such
tests are expected to fail.

How to add tests
----------------
### How to add tests

Tests are structured by the section of the policy concerned, whenever possible.
Please add a comment at the beginning of the test file, explaining what is being
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit 8e75944

Please sign in to comment.