diff --git a/doc/command_line/print_config.md b/doc/command_line/print_config.md index 04684d32..42041486 100644 --- a/doc/command_line/print_config.md +++ b/doc/command_line/print_config.md @@ -3,38 +3,46 @@ # CLI: Print the Kaocha configuration A Kaocha test run starts with building up a Kaocha configuration map, based on -default values, the contents of `tests.edn`, command line flags, and active -plugins. - -Debugging issues with Kaocha often starts with inspecting the configuration, -which is why a `--print-config` flag is provided. This builds up the -configuration from any available sources, runs it through any active plugins, -and then pretty prints the result, an EDN map. + default values, the contents of `tests.edn`, command line flags, and active + plugins. + + Debugging issues with Kaocha often starts with inspecting the configuration, + which is why a `--print-config` flag is provided. This builds up the + configuration from any available sources, runs it through any active plugins, + and then pretty prints the result, an EDN map. + + Note that the ordering, while not expected to change, is not guaranteed. We + recommend parsing the configuration as EDN and not relying on order. If you + are manipulating the output as text (say, on the command line) and can't + avoid relying on the order, run it through a tool like + [puget](https://github.com/greglook/puget) or + [zprint](https://github.com/kkinnear/zprint) that sorts the keys + alphabetically first. ## Using `--print-config` - When I run `bin/kaocha --print-config` -- Then the output should contain: +- Then the EDN output should contain: ``` clojure {:kaocha.plugin.randomize/randomize? false, :kaocha/reporter [kaocha.report/dots], :kaocha/color? false, - :kaocha/fail-fast? false, + :kaocha/fail-fast? false} ``` -- And the output should contain: +- And the EDN output should contain: ``` clojure - :kaocha/tests + {:kaocha/tests [{:kaocha.testable/type :kaocha.type/clojure.test, :kaocha.testable/id :unit, :kaocha/ns-patterns ["-test$"], :kaocha/source-paths ["src"], :kaocha/test-paths ["test"], - :kaocha.filter/skip-meta [:kaocha/skip]}], + :kaocha.filter/skip-meta [:kaocha/skip]}]} ``` diff --git a/doc/plugins/notifier_plugin.md b/doc/plugins/notifier_plugin.md index 14a870d7..422005f1 100644 --- a/doc/plugins/notifier_plugin.md +++ b/doc/plugins/notifier_plugin.md @@ -3,52 +3,33 @@ # Plugin: Notifier (desktop notifications) Desktop notifications can be enabled with the `:kaocha.plugin/notifier` -plugin. This will pop up a fail/pass notification bubble including a summary -of tests passed/errored/failed at the end of each test run. It's particularly -useful in combination with `--watch`, e.g. `bin/kaocha --plugin notifier ---watch`. - -It does this by invoking a shell command that can be configured, so it can be -used to invoke an arbitrary command or script. By default, it will try to detect -which command to use, using either `notify-send` (Linux) or `terminal-notifier` -(Mac OS X), either of which may need to be installed first. If those commands -aren't available, it uses Java's included notifier. - -Several replacement patterns are available: - -- `%{title}` : The notification title, either `⛔️ Failing` or `✅ Passing` -- `%{message}` : Test result summary, e.g., `5 tests, 12 assertions, 0 failures` -- `%{icon}` : Full local path to an icon to use (currently uses the Clojure icon) -- `%{failed?}` : `true` if any tests failed or errored, `false` otherwise -- `%{count}` : the number of tests -- `%{pass}` : the number of passing assertions -- `%{fail}` : the number of failing assertions -- `%{error}` : the number of errors -- `%{pending}` : the number of pending tests -- `%{urgency}` : `normal` if the tests pass, `critical` otherwise, meant for use with `notify-send` -- '%{timeout}` : configured timeout in milliseconds before the notification should disappear. By - default, it passes -1, which corresponds to the default timeout when using - `notify-send`. - -Note that notifications don't time out on -[GNOME](https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/112) and Ubuntu when -using [Notify -OSD](https://bugs.launchpad.net/ubuntu/+source/notify-osd/+bug/390508). - -If no command is configured, and neither notification command is found, then -the plugin will print a warning. You can explicitly inhibit its behaviour -with `--no-notifications`. - -You can combine the feature with the profiles feature: - - -``` clojure -{:plugins #profile {:default [:kaocha.plugin/notifier] -:ci []}} -``` - -This will silence the "Notification not shown because system does not support -it." warning. + plugin. This will pop up a fail/pass notification bubble including a summary + of tests passed/errored/failed at the end of each test run. It's particularly + useful in combination with `--watch`, e.g. `bin/kaocha --plugin notifier + --watch`. + + It does this by invoking a shell command which can be configured, so it can be + used to invoke an arbitrary command or script. By default it will try to + detect which command to use, using either `notify-send` (Linux) or + `terminal-notifier` (Mac OS X), either of which may need to be installed + first. + + Several replacement patterns are available: + + - `%{title}` : The notification title, either `⛔️ Failing` or `✅ Passing` + - `%{message}` : Test result summary, e.g. `5 tests, 12 assertions, 0 failures` + - `%{icon}` : Full local path to an icon to use (currently uses the Clojure icon) + - `%{failed?}` : `true` if any tests failed or errored, `false` otherwise + - `%{count}` : the number of tests + - `%{pass}` : the number of passing assertions + - `%{fail}` : the number of failing assertions + - `%{error}` : the number of errors + - `%{pending}` : the number of pending tests + - `%{urgency}` : `normal` if the tests pass, `critical` otherwise, meant for use with `notify-send` + + If no command is configured, and neither notification command is found, then + the plugin will silently do nothing. You can explicitly inhibit its behaviour + with `--no-notifications`. ## Enabling Desktop Notifications diff --git a/doc/plugins/orchestra_plugin.md b/doc/plugins/orchestra_plugin.md index d2efe508..4f4c9310 100644 --- a/doc/plugins/orchestra_plugin.md +++ b/doc/plugins/orchestra_plugin.md @@ -59,11 +59,13 @@ instruments your functions. ``` nil ERROR in orchestra-test/spec-fail-test (orchestra_test.clj:11) Just testing simple-fn -Call to #'orchestra-test/simple-fn did not conform to spec. +Call to orchestra-test/simple-fn did not conform to spec. orchestra_test.clj:11 -- Spec failed -------------------- +Return value + "x" should satisfy diff --git a/doc/spec_test_check.md b/doc/spec_test_check.md index 2a13555f..52a7fb97 100644 --- a/doc/spec_test_check.md +++ b/doc/spec_test_check.md @@ -3,48 +3,48 @@ # Automatic spec test check generation Kaocha can discover all of your fdefs and generate `clojure.spec.test.check` -tests for them. This saves you the trouble of writing your own boilerplate, -and gives you the truly "free" generative testing that clojure.spec promises. - -There are two ways you can use this feature: - -1. Adding `:kaocha.type/spec.test.check` test suites to your `tests.edn`: -- `:kaocha.testable/type` = :kaocha.type/spec.test.check -- `:kaocha/source-paths`: Normally your fdefs are with your code, so this -can probably be left defaulted at `["src"]` -- `:kaocha.spec.test.check/checks`: Optional. If you want to -orchestrate multiple "sets" of checks with differing parameters, you can -specify them here. This is a collection of checks, each check being a map -which may contain the following optional keys: -- `:kaocha.spec.test.check/syms`: Currently your only options are either -`:all-fdefs` (default) or to provide a set of the symbols for the fdefs -which you want to test. Eventually we will add `:other-fdefs` to select -all the fdefs that were not specifically mentioned in other checks. -- `:clojure.spec.test.check/instrument?` Turn on orchestra instrumentation -during fdef checks -- `:clojure.spec.test.check/check-asserts?` Run s/check-asserts during fdef -checks -- `:clojure.spec.test.check/opts`: A map containing any of: -- `:num-tests`: Test iterations per fdef -- `:max-size`: Maximum length of generated collections -- All of the keys within each check can also be given in the top-level test -suite map to be merged by default into all checks. -2. The `kaocha.plugin.alpha/spec-test-check` plugin -- This provides a sane default test suite for automatically checking all of -your fdefs. Spec test checking can be configured with more granularity in -tests.edn (as above), but the plugin exists for easy and simplistic CLI -control. -- Regardless of whether you add the test suite(s) to `tests.edn` yourself, -you can also use this plugin to forceably override certain test -parameters: -- `--[no-]stc-instrumentation` = `:clojure.spec.test.check/instrument?` -- `--[no-]stc-asserts` = `:clojure.spec.test.check/check-asserts?` -- `--stc-num-tests NUM` = `:num-tests` -- `--stc-max-size SIZE` = `:max-size` -- By default, this plugin also adds `:no-gen` to `:kaocha.filter/skip-meta`. -You might want to decorate an fdef-ed function with `^:no-gen` if there is -either no good generator for one or more of its arguments or if the -function is side-effectful. + tests for them. This saves you the trouble of writing your own boilerplate, + and gives you the truly "free" generative testing that clojure.spec promises. + + There are two ways you can use this feature: + + 1. Adding `:kaocha.type/spec.test.check` test suites to your `tests.edn`: + - `:kaocha.testable/type` = :kaocha.type/spec.test.check + - `:kaocha/source-paths`: Normally your fdefs are with your code, so this + can probably be left defaulted at `["src"]` + - `:kaocha.spec.test.check/checks`: Optional. If you want to + orchestrate multiple "sets" of checks with differing parameters, you can + specify them here. This is a collection of checks, each check being a map + which may contain the following optional keys: + - `:kaocha.spec.test.check/syms`: Currently your only options are either + `:all-fdefs` (default) or to provide a set of the symbols for the fdefs + which you want to test. Eventually we will add `:other-fdefs` to select + all the fdefs that were not specifically mentioned in other checks. + - `:kaocha.spec.test.check/instrument?` Turn on orchestra instrumentation + during fdef checks + - `:kaocha.spec.test.check/check-asserts?` Run s/check-asserts during fdef + checks + - `:clojure.spec.test.check/opts`: A map containing any of: + - `:num-tests`: Test iterations per fdef + - `:max-size`: Maximum length of generated collections + - All of the keys within each check can also be given in the top-level test + suite map to be merged by default into all checks. + 2. The `kaocha.plugin.alpha/spec-test-check` plugin + - This provides a sane default test suite for automatically checking all of + your fdefs. Spec test checking can be configured with more granularity in + tests.edn (as above), but the plugin exists for easy and simplistic CLI + control. + - Regardless of whether you add the test suite(s) to `tests.edn` yourself, + you can also use this plugin to forceably override certain test + parameters: + - `--[no-]stc-instrumentation` = `:kaocha.spec.test.check/instrument?` + - `--[no-]stc-asserts` = `:kaocha.spec.test.check/check-asserts?` + - `--stc-num-tests NUM` = `:num-tests` + - `--stc-max-size SIZE` = `:max-size` + - By default, this plugin also adds `:no-gen` to `:kaocha.filter/skip-meta`. + You might want to decorate an fdef-ed function with `^:no-gen` if there is + either no good generator for one or more of its arguments or if the + function is side-effectful. ## Detects and checks fdefs using tests.edn diff --git a/doc/syntax_error.md b/doc/syntax_error.md new file mode 100644 index 00000000..1c261188 --- /dev/null +++ b/doc/syntax_error.md @@ -0,0 +1,31 @@ + + +# Syntax errors are preserved + +Syntax errors should be passed along. + +## Show output of failing test + +- Given a file named "test/sample_test.clj" with: + +``` clojure +(ns sample-test + (:require [clojure.test :refer :all])) + +stray-symbol + +(deftest stdout-pass-test + (is (= :same :same))) +``` + + +- When I run `bin/kaocha` + +- Then the output should contain: + +``` nil +Exception: clojure.lang.Compiler$CompilerException +``` + + +