Scenario testing - Restarts and upgrades

[jamshale]

In the acapy tools repo there is some really useful testing using docker containers that allows testing of upgrade scripts and restarts. https://github.com/hyperledger/aries-acapy-tools/tree/main/askar_tools/tests/e2e.

It should be possible to support this in the scenarios directory from acapy itself. It would be awesome to be able to test a supported upgrade path from the last LTS version and test a few different restarts with changed configurations such as making public dids and using seeds etc.

[ianco]

@jamshale are there instructions for running the e2e tests?

[jamshale]

In [acapy-tools](https://github.com/openwallet-foundation/acapy-tools) I don't think there is any instructions :/ . It's pretty simple though. I just poetry install and then navigate to askar_tools and run poetry run pytest ..

You can use other pytest commands to run an individual test.

[ianco]

ok thanks, I'll add this to the README

[ianco]

Probably more useful to be able to test this with a back-end postgres database rather than having to worry about import/export for sqlite wallets (and then the tests could run in a separate docker container, and just point to the external database).

[jamshale]

I agree. For this scenario testing against a postgres db would be easier and better. The sqlite testing was much more difficult with restarts in acapy-tools. Also would be good to have postgres getting tested more here in general.

[ianco]

@jamshale I'm getting errors running the tests.

I ran poetry install --with=dev (to include the dev dependencies, basically acapy-controller), which failed with the following error:

HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /indicio-tech/acapy-minimal-example.git/info/refs?service=git-upload-pack (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))

I updated the project to use acapy-controller = "~=0.2.0" (the published version of the library) and then the install worked.

After running poetry run pytest . however everything failed:

FAILED tests/e2e/test_export.py::TestPgExport::test_export_pg - aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host selfserve.indiciotech.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:...
FAILED tests/e2e/test_export.py::TestPgExport::test_export_sqlite - docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.43/containers/create?name=alice: Conflict ("Conflict. The container name "/alice" is already in use by container "81137e408a975071c15cfb032...
FAILED tests/e2e/test_mt_convert_to_mw.py::TestMultitenantConvertToMultiwallet::test_conversion_pg - aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host selfserve.indiciotech.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:...
FAILED tests/e2e/test_mt_convert_to_mw.py::TestMultitenantConvertToMultiwallet::test_conversion_sqlite - docker.errors.APIError: 409 Client Error for http+docker://localhost/v1.43/containers/create?name=admin: Conflict ("Conflict. The container name "/admin" is already in use by container "14b1b2cf31f1c7e6c9f075f70...
FAILED tests/e2e/test_tenant_import.py::TestTenantImport::test_tenant_import_pg - aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host selfserve.indiciotech.io:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:...

Any thoughts? Looks like a certificate error and then the container ... already is use is because a previous test failed.

[ianco]

(the above is in the askar_tools directory, for the tests under acapy_wallet_upgrade some work and some fail)

(most of the tests fail)

[jamshale]

It doesn't clean up very well when there's a fail. Sometimes you need to manually delete the containers or the network.

I'm not sure about the certificate error though. I'm not getting any and the tests are passing for me. You might be able to switch the genesis file. I'm just trying that now.

[ianco]

What's running at selfserve.indiciotech.io:443?

[jamshale]

The acapy-tools tests use the indicio indy test ledger. I guess that's where it's trying to register a did. I changed the scenario tests in acapy to use http://test.bcovrin.vonx.io/. There's something automatic that the acapy-controller indy_anoncred_onboard function does that requires https that prevents just switching easily :/

Not sure why your getting certificate problems though.

[ianco]

Looks like all that is hard-coded into the unit tests. I'll update to use env vars (if available) and also it looks like some tests have a postgres dependency.

Eventually would be nice to run the tests in docker (I always have to mess around when running locally to get all the proper versions, like python 3.10 which isn't a default on my local ...)

[jamshale]

Ya. These test in acapy-tools aren't the best, but the container process that allows restarts is what I wanted to replicate for acapy.

I got around the hardcoded stuff in the library for the ledger here https://github.com/openwallet-foundation/acapy/blob/main/scenarios/examples/presenting_revoked_credential/example.py#L50.

[ianco]

I notice that the test container logic starts up a bunch of services within the python code. Typically in other projects we do this within a docker-compose file, and then pass parameters in (overrides) via environment variables.

I'd like to convert this project to use a similar structure - docker compose and then build a docker image to run the unit tests.

Before I dive into the code, does this make sense? It should make it easier to use these components within other repo's (like aca-py).

@jamshale @swcurran @dbluhm

[jamshale]

Yes. That would be good. In the scenario tests in acapy we use docker compose files. It's really just about how to persist the storage containers for the length of the test and restart the agents with a different image and configs. The acapy-tools e2e tests do this so I thought we could use a similar design.

[ianco]

It's really just about how to persist the storage containers for the length of the test and restart the agents with a different image and configs.

I'll work on updating the acapy-tools tests to use docker and think about how to do this for the aca-py scenario tests ...

[dbluhm]

The reason the tests start the container in python is because that seemed like the best way to control starting and stopping containers. I also generally prefer to keep it simple and just use docker compose wherever possible but, at the time, I was not finding good solutions that permitted both simple docker compose and controlling startup parameters and starting and stopping the containers. So docker compose got nixed.

[ianco]

@jamshale I'm getting errors running the tests.

FYI managed to fix the certificate issue with this: https://stackoverflow.com/questions/52805115/certificate-verify-failed-unable-to-get-local-issuer-certificate

[ianco]

@jamshale I think the scenario testing approach will have to change to support acapy restarts (especially if we are shutting down one acapy version and then starting up another).

Right now the scenario test all start their agents via docker compose, so there is no mechanism for the test (which runs in one container) to restart the agent (which runs in a different container). (The test creates an instance of an acapy_controller.Controller to control the running agent.)

One possibility is to change the acapy-test container to include the acapy_controller.Controller wrapper, and then expose an additional endpoint to start, stop or restart the agent. (The agent would then be controlled from within the python code, similar to how the askar_tools tests work.) Of course all the other protocols would have to be exposed in a similar way.

Another possibility is to ditch the use of docker compose to start the agents and just manage all the agents within python code, similar to how the askar_tools tests work.

One wrinkle is the requirement to support different versions of acapy - i.e. startup an older version of acapy, run some scenarios, shut down the agent and then start up a new agent (with a newer acapy version) pointing to the same wallet. (Or running a wallet upgrade in between.)

@jamshale @dbluhm any thoughts?

[jamshale]

What if it was separate from the scenario tests? Like, it was a different set of tests called migration tests. Maybe it could be in the same poetry project but started as a different poetry script?

I think the suite of tests runs the tests in the examples folder sequentially. If the migration tests were in another folder, it might be easier to have different setup/structure.

[ianco]

What if it was separate from the scenario tests?

I'll setup a separate test ... One option for the "acapy version" issue is to have two Controller containers based on two different acapy versions (for example the latest acapy release and the local acapy source code) and then starting an agent in one container, executing some protocols, shutting down the agent, and then starting a new agent from the other Controller container (pointing to the same wallet).

The upgrade (if we're running a wallet upgrade) could be run directly from the example code.