Set up tests and Github actions workflow

[duncandewhurst]

I don't know what is provided for in https://github.com/OpenDataServices/sphinx-base, but we should set up tests and a Github actions workflow to run them.

We can likely reuse some of the tests from OCDS, which are split across:

• https://github.com/open-contracting/standard/tree/1.2-dev/tests
• https://github.com/open-contracting/standard-maintenance-scripts

If I remember correctly, the reason for the separation is that some tests need to be run across multiple repos (extensions and profiles). For this project, I think we can put all the tests in this repository.

There's a lot of complexity in the OCDS tests relating to extensions, which doesn't need to be reproduced for this project.

[lgs85]

@Lathrisk is going to take a look at this in a couple of weeks, and I can help out/code review etc. I don't think the sphinx base involves any testing, so we'll need to write/borrow from other standards.

[lgs85]

To map this out in a little more detail, here is a suggested starting list for tests to add as Github actions (to be run on each push and pull request)

Generic tests

• python code checking with flake8, black and isort
• Check validity of csv files - we don't have any csv files yet but presumably this can be set up in advance
• Check validity of json files - we don't have any json files yet but presumably this can be set up in advance
• Something to check that the sphinx documentation builds?
• Something to check markdown formatting?

Schema specific tests

• To be decided on completion of the schema. See this example from BODS

Others do please feel free to add/comment.

@Lathrisk in terms of timing, I think we can make a start on the generic tests in the first block of time, then depending on how far we get, finish these off in block two and get going with the schema specific tests?

[duncandewhurst]

Looks good to me.

• Something to check that the sphinx documentation builds?

We should definitely have this, we don't want to merge PRs that break the docs build.

• Something to check markdown formatting?

+1

OCDS has/had some extra tests that we can consider:

• test_search
• test_broken_links (I think that this test would need to result in a warning rather than a failure so that the build isn't overly dependent on externalities)

[jpmckinney]

If you run sphinx-build with -nW you'll enable nit-picky mode and turn warnings into errors, so you won't be able to deploy broken builds.

To check links, you can also run sphinx-build linkcheck. It accepts some options here: https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-the-linkcheck-builder

We don't use linkcheck for OCDS as it's too slow for the number of external links we have.

test_broken_links was removed from test_docs as it's unnecessary (it was only checking internal links, but those are already checked if you build with -nW). test_search only matters if you implement your own custom search page, instead of using the default provided by ReadTheDocs (which is quite better nowadays).

For Markdown formatting, I recommend setting up pre-commit with mdformat.

[duncandewhurst]

Thanks @jpmckinney!

[duncandewhurst]

Another test that would be useful is to check that the descriptions of related properties and codes are in sync:

• Node.physicalInfrastructureProvider, Link.physicalInfrastructureProvider and 'physicalInfrastructureProvider' in organisationRole.csv
• Node.networkProvider, Link.networkProvider and 'networkProvider' in organisationRole.csv
• Phase.funder and 'funder' in organisationRole.csv

[duncandewhurst]

Tests for example data are mentioned in #57 (comment)

[odscjames]

Some Python functionality is now in ofdskit, in that case remove python from here.

[duncandewhurst]

Some Python functionality is now in ofdskit, in that case remove python from here.

We're still going to have a pre-commit Python script for updating derivative schema files and reference documentation so I think we want to have Python code checking as part of the tests.

[odscjames]

I think we want to have Python code checking as part of the tests.

Yes - if there is any large bits of Python in the repository we still want code checking.

I think my comment has been misunderstood - it was posted during our call and refers not to removing Python checking but to removing actual bits of Python code, if that code is now available in external libraries like ofdskit instead (and of course ofdskit does have Python checking!)

We're still going to have a pre-commit Python script for updating derivative schema files and reference documentation

No, this repository is going to have a pre-comit hook to do that. If it's suitable for that code to become a generic Python library in another repository that the hook can just call, we'll do that. Suitable in this case means: If the advantages of code-sharing across standards and/or different places outweigh the disadvantages of dealing with an extra repository. (I'm not saying that is in scope for this ticket, just trying to be clear about the option here that we should consider later!)

[duncandewhurst]

Yes, that all makes sense. I've created new issues.

[odscjames]

I have started looking at this and the draft PR, taking functionality a bit at a time to new PR's to ensure things are merged as soon as they are ready.

[jpmckinney]

Another test we should have - run precommit as far as it is possible

You can use https://pre-commit.ci, which adds a check (which you can make mandatory or not).

[odscjames]

I think everything here has been done or moved to a new issue. Can we close this one?

[duncandewhurst]

Yep. Thanks!