JSON Schema Validation comparison

Feature and performance comparison of different JVM-based implementations of JSON schema validators.

The results of this comparison can be found on here.

Note to maintainers

If you are the maintainer of one of the above implementations, and you feel your implementation is poorly represented, or you maintain an JVM-based implementation not covered yet covered in this comparison, then please feel free to raise a PR. See the Contributing section below.

Contributing

Adding a new validator implementation

Adding a new validator implementation is relatively straight forward and very welcome:

1. First, take a look at the micro-site, as it gives some explanation of what is being tested.
2. Clone the repo and pull it down locally, creating your own branch to work in.
3. Add necessary dependencies to build.gradle.kts.
4. Optional, If the previous step involved add a new repository to download the new dependency, then ensure GitHub's Dependabot will update the new dependency version. This will ensure the site updates when new versions are released. See the Dependabot version updates docs.
5. Add a new implementation of Implementation to the main implementations package for the new validator library. See JavaDocs and other implementations for help.
6. Register your new Implementation type in Implementations.java. This will ensure the new implementation is included in the docs and included in the functional test
7. Run ImplementationTest.java. This unit test will test each implementation, including yours. Ensure tests pass!
8. Manually add appropriate benchmark methods to JsonSerdeBenchmark.java and JsonValidateBenchmark.java. This is currently manual as JMH library does provide a way to generate these automatically. There should be one test per supported draft version. See JavaDocs and the other methods in these classes for examples.
9. Run ./gradlew to format your code, perform static analysis and run the tests. Ensure this passes!
10. Follow these instructions to build and view the website, and ensure your new implementation data is included in tables and charts.
11. Raise a PR with your changes.

Running things locally

Feature comparison

Run the feature comparison locally with ./gradlew runFunctionalTests, or the latest results, or previous runs are available in the GitHub pages workflow runs on GitHub.

Running the functional tests will create result files in the docs/_include directory, ready for Jekyll to inject into the micro-site.

Generated files:

filename	description	use
functional-summary.json	JSON document containing a summary of pass/fail rates of required/optional test cases for each implementation, per supported JSON schema version.	Used to build functional tables and charts in micro-site.
functional-summary.md	Markdown document containing a summary of pass/fail rates of required/optional test cases for each implementation, per supported JSON schema version.	Appended to the GitHub workflow job
per-draft.md	Markdown document containing one table for each implementation and supported schema specification combination, showing the number of test cases that pass and fail in each test file.	Used to build functional tables and charts in micro-site, and appended to the GitHub workflow job

Performance comparison

Run the performance comparison locally with ./gradlew runBenchmarks, or the latest results, or previous runs are available in the GitHub pages workflow runs on GitHub.

See benchmark classes in the performance package .

Running the performance benchmarks will create result files in the docs/_include directory, ready for Jekyll to inject into the micro-site.

Running the performance benchmarks takes a long time. Running ./gradlew runBenchmarkSmokeTest will run the same benchmarks in a matter of minutes, which can be useful for testing and generating data for the micro-site.

Generated files:

filename	description	use
benchmark_results.json	A JSON document containing the results of the performance benchmarking	Used to build functional tables and charts in micro-site.
JsonSerdeBenchmark.md	Markdown document containing the results of this benchmark class.	Appended to the GitHub workflow job
JsonValidateBenchmark.md	Markdown document containing the results of this benchmark class.	Appended to the GitHub workflow job