diff --git a/docs/creating-a-dataset.md b/docs/creating-a-dataset.md index 33f3c84..a281cab 100644 --- a/docs/creating-a-dataset.md +++ b/docs/creating-a-dataset.md @@ -1,4 +1,4 @@ -The endpoint [`/api/v1/integrations/datasets`](https://api.dev.hdruk.cloud/api/documentation#/Dataset%20Integrations/create_datasets_from_app) should be used to create dataset(s) on the gateway by posting metadata describing these datasets to our API endpoint. +The endpoint [`/api/v1/integrations/datasets`](https://api.dev.hdruk.cloud/api/documentation#/Datasets/create_datasets_integrations) should be used to create dataset(s) on the gateway by posting metadata describing these datasets to our API endpoint. You should POST application/JSON data to the endpoint where your metadata validates against one of [our supported schemas](https://github.com/HDRUK/schemata-2/blob/master/available.json) @@ -17,7 +17,7 @@ You should POST application/JSON data to the endpoint where your metadata valida client_id = app_id = - api_path = "https://api.dev.hdruk.cloud/api/v1/" + api_path = "https://api.dev.hdruk.cloud/api/v1" headers = { @@ -26,9 +26,6 @@ You should POST application/JSON data to the endpoint where your metadata valida "Content-Type": "application/json", } - ``` - - ```python metadata = {"metadata": } @@ -81,6 +78,8 @@ If your app did enable permissions to create a dataset then you'll see the follo } ``` +This will need fixing on the Permissions tab for the app. + #### No x-application-id or x-client-id in your headers ```json @@ -115,4 +114,44 @@ You've copied your App ID or Client ID wrong when you've added them to the heade } ``` -Navigate (on the gateway) to your integrations API management page (Mange API), click on your App, your the application you have copied over the App and Client ID (from the Authentication tab) make sure the App is set to enabled on the 'App info' tab. +Navigate (on the Gateway) to your Integrations > API Management page (Manage API), and select the App. Check that you have copied over the App and Client ID (from the Authentication tab) correctly, and make sure the App is set to 'Enabled' on the 'App Info' tab. + +#### Invalid format + +```json +{ + "message": "metadata is in an unknown format and cannot be processed", + "details": { + "traser_message": { + "message": "Input metadata object matched no known schemas", + "details": { + "available_schemas": { + "HDRUK": [ + "2.1.2", + "2.1.3", + "2.1.0", + "2.0.2", + "2.2.1", + "2.2.0" + ], + "GWDM": [ + "1.0", + "1.1", + "1.2" + ], + "SchemaOrg": [ + "default", + "BioSchema", + "GoogleRecommended" + ] + } + } + }, + "wasTranslated": false, + "metadata": null, + "statusCode": 400 + } +} +``` + +The supplied metadata is not in any valid format. Use the TRASER `find` endpoint (navigate within these docs to `Metadata > Validate`) to understand the reasons for invalidity. diff --git a/docs/creating-metadata.md b/docs/creating-metadata.md index 8ced858..b7038c9 100644 --- a/docs/creating-metadata.md +++ b/docs/creating-metadata.md @@ -1,31 +1,33 @@ -There are multiple [metadata schemas](https://github.com/HDRUK/schemata-2) that we support that your metadata can conform to and can be submitted to our systems. +There are multiple [metadata schemas](https://hdruk.github.io/schemata-2/) that we support that your metadata can conform to and can be submitted to our systems. !!! tip - - **Recommended:** [HDRUK 2.2.0](https://hdruk.github.io/schemata-2/HDRUK/2.2.0/) - our public facing schema, our manual-onboarding forms are based on this schema - - Older supoprted versions: + - **Recommended:** [HDRUK 2.2.0](https://hdruk.github.io/schemata-2/HDRUK/2.2.0/) is our public facing schema. Our manual-onboarding forms are based on this schema. + - Older supported versions: - [HDRUK 2.1.2](https://hdruk.github.io/schemata-2/HDRUK/2.1.2/) - [HDRUK 2.1.0](https://github.com/HDRUK/schemata-2/blob/master/hdr_schemata/models/HDRUK/2.1.0/schema.json) - [HDRUK 2.0.2](https://github.com/HDRUK/schemata-2/blob/master/hdr_schemata/models/HDRUK/2.0.2/schema.json) - **Other Schemas:** - - [BioSchema](https://bioschemas.org/) - we can accept data conforming to this schema definiton, it is just a lot more limited - - [Gateway Data Model (GWDM 1.1)](https://hdruk.github.io/schemata-2/GWDM/1.1/) - a looser and more extensive model, used to store any metadata that we accept from the gatway + - [BioSchema](https://bioschemas.org/) - we can accept data conforming to this schema definition, but it is more limited + - [Gateway Data Model (GWDM 1.1)](https://hdruk.github.io/schemata-2/GWDM/1.1/) - a more general and flexible model. We use this schema internally to store any metadata that we accept from external sources. We also support additional schemas, external to HDRUK, such as [BioSchema](https://bioschemas.org/). -When you create your metadata and `POST` it to our APIs, we convert this metadata, via our [translation service](https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app/docs/) into our Gateway Data Model (GWDM). +When you create your metadata and `POST` it to our APIs, the [translation service](https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app/docs/) converts this metadata into the Gateway Data Model (GWDM) for internal storage. - Example: `HDRUK 2.2.0` → `GWDM 1.1` -There are several types and versions of schemas that we can accept as input, validate and translate into our GWDM. It is also possible to retrieve your data in various different schemas by requesting the stored data is translated back into a specific example. +There are several schemas, each with possibly multiple versions, that we support. Metadata in each can be accepted as input, validated and translated into our GWDM. + +It is also possible to retrieve your data in various different schemas by requesting the stored data is translated into a specific schema. - Example: `GWDM 1.1` → `BioSchema`. -What what inputs to the APIs and hence schema translations to the GWDM we can accept are dependent on which [translation](https://github.com/HDRUK/traser-mapping-files/blob/master/available.json) we can accept. Translations are using written using the transformation language [JSONata](https://jsonata.org/). You can visit our [translation-files-repo](https://github.com/HDRUK/traser-mapping-files) to see these translation maps, request new translations so your schema can be supported, report bugs in translations, or submit your own translation map. +The input-output combinations of schemas we support are dependent on the list of [translations](https://github.com/HDRUK/traser-mapping-files/blob/master/available.json) which are implemented. Translations are using written using the transformation language [JSONata](https://jsonata.org/). You can visit our [translation files repo](https://github.com/HDRUK/traser-mapping-files) to see these translation maps; request new translations so your schema can be supported; report bugs in translations; or submit your own translation map. ## Supported Schemas -To find out what schemas we support, you can use our translation service [(TRASER)](https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app/docs/) to list what models and what versions of the the model we support. +To find out which schemas we support, you can use our translation service [(TRASER)](https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app/docs/) to list them, including the specific versions. You can visit the repository [https://hdruk.github.io/schemata-2/](https://hdruk.github.io/schemata-2/) to see a summary of our schemas and a change log between different versions. @@ -52,11 +54,13 @@ Schemas are published as [pydantic](https://docs.pydantic.dev/latest/) models th "2.1.3", "2.1.0", "2.0.2", + "2.2.1", "2.2.0" ], "GWDM": [ "1.0", - "1.1" + "1.1", + "1.2" ], "SchemaOrg": [ "default", @@ -74,7 +78,7 @@ Schemas are published as [pydantic](https://docs.pydantic.dev/latest/) models th Returns: ```bash - {"HDRUK":["2.1.2","2.1.3","2.1.0","2.0.2","2.2.0"],"GWDM":["1.0","1.1"],"SchemaOrg":["default","BioSchema","GoogleRecommended"]} + {"HDRUK":["2.1.2","2.1.3","2.1.0","2.0.2","2.2.1","2.2.0"],"GWDM":["1.0","1.1","1.2"],"SchemaOrg":["default","BioSchema","GoogleRecommended"]} ``` === "Web Browser" @@ -92,7 +96,7 @@ Our translation service will also allow you to see what available translations t import requests import json - traser_uri = "ttps://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app" + traser_uri = "https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app" response = requests.get(f"{traser_uri}/list/templates") print(json.dumps(response.json(), indent=6)) @@ -137,6 +141,36 @@ Our translation service will also allow you to see what available translations t "input_model": "HDRUK", "input_version": "2.1.2" }, + { + "output_model": "HDRUK", + "output_version": "2.2.1", + "input_model": "HDRUK", + "input_version": "2.2.0" + }, + { + "output_model": "HDRUK", + "output_version": "2.2.1", + "input_model": "GWDM", + "input_version": "1.2" + }, + { + "output_model": "HDRUK", + "output_version": "2.2.0", + "input_model": "HDRUK", + "input_version": "2.1.2" + }, + { + "output_model": "HDRUK", + "output_version": "2.2.0", + "input_model": "HDRUK", + "input_version": "2.2.1" + }, + { + "output_model": "HDRUK", + "output_version": "2.2.0", + "input_model": "GWDM", + "input_version": "1.1" + }, { "output_model": "GWDM", "output_version": "1.0", @@ -161,12 +195,42 @@ Our translation service will also allow you to see what available translations t "input_model": "HDRUK", "input_version": "2.1.2" }, + { + "output_model": "GWDM", + "output_version": "1.1", + "input_model": "HDRUK", + "input_version": "2.2.0" + }, { "output_model": "GWDM", "output_version": "1.1", "input_model": "GWDM", "input_version": "1.0" }, + { + "output_model": "GWDM", + "output_version": "1.1", + "input_model": "GWDM", + "input_version": "1.2" + }, + { + "output_model": "GWDM", + "output_version": "1.1", + "input_model": "SchemaOrg", + "input_version": "BioSchema" + }, + { + "output_model": "GWDM", + "output_version": "1.2", + "input_model": "HDRUK", + "input_version": "2.2.1" + }, + { + "output_model": "GWDM", + "output_version": "1.2", + "input_model": "GWDM", + "input_version": "1.1" + }, { "output_model": "SchemaOrg", "output_version": "default", @@ -185,6 +249,12 @@ Our translation service will also allow you to see what available translations t "input_model": "GWDM", "input_version": "1.0" }, + { + "output_model": "SchemaOrg", + "output_version": "BioSchema", + "input_model": "GWDM", + "input_version": "1.1" + }, { "output_model": "SchemaOrg", "output_version": "GoogleRecommended", @@ -209,7 +279,7 @@ Our translation service will also allow you to see what available translations t Returns: ```bash - [{"output_model":"HDRUK","output_version":"2.1.2","input_model":"HDRUK","input_version":"datasetv2"},{"output_model":"HDRUK","output_version":"2.1.2","input_model":"HDRUK","input_version":"2.1.3"},{"output_model":"HDRUK","output_version":"2.1.2","input_model":"HDRUK","input_version":"2.0.2"},{"output_model":"HDRUK","output_version":"2.1.2","input_model":"GWDM","input_version":"1.0"},{"output_model":"HDRUK","output_version":"2.1.2","input_model":"GWDM","input_version":"1.1"},{"output_model":"HDRUK","output_version":"2.1.3","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"GWDM","output_version":"1.0","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"GWDM","output_version":"1.0","input_model":"GWDM","input_version":"1.1"},{"output_model":"GWDM","output_version":"1.0","input_model":"SchemaOrg","input_version":"default"},{"output_model":"GWDM","output_version":"1.1","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"GWDM","output_version":"1.1","input_model":"GWDM","input_version":"1.0"},{"output_model":"SchemaOrg","output_version":"default","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"SchemaOrg","output_version":"default","input_model":"GWDM","input_version":"1.1"},{"output_model":"SchemaOrg","output_version":"BioSchema","input_model":"GWDM","input_version":"1.0"},{"output_model":"SchemaOrg","output_version":"GoogleRecommended","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"SchemaOrg","output_version":"GoogleRecommended","input_model":"GWDM","input_version":"1.0"}] + [{"output_model":"HDRUK","output_version":"2.1.2","input_model":"HDRUK","input_version":"datasetv2"},{"output_model":"HDRUK","output_version":"2.1.2","input_model":"HDRUK","input_version":"2.1.3"},{"output_model":"HDRUK","output_version":"2.1.2","input_model":"HDRUK","input_version":"2.0.2"},{"output_model":"HDRUK","output_version":"2.1.2","input_model":"GWDM","input_version":"1.0"},{"output_model":"HDRUK","output_version":"2.1.2","input_model":"GWDM","input_version":"1.1"},{"output_model":"HDRUK","output_version":"2.1.3","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"HDRUK","output_version":"2.2.1","input_model":"HDRUK","input_version":"2.2.0"},{"output_model":"HDRUK","output_version":"2.2.1","input_model":"GWDM","input_version":"1.2"},{"output_model":"HDRUK","output_version":"2.2.0","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"HDRUK","output_version":"2.2.0","input_model":"HDRUK","input_version":"2.2.1"},{"output_model":"HDRUK","output_version":"2.2.0","input_model":"GWDM","input_version":"1.1"},{"output_model":"GWDM","output_version":"1.0","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"GWDM","output_version":"1.0","input_model":"GWDM","input_version":"1.1"},{"output_model":"GWDM","output_version":"1.0","input_model":"SchemaOrg","input_version":"default"},{"output_model":"GWDM","output_version":"1.1","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"GWDM","output_version":"1.1","input_model":"HDRUK","input_version":"2.2.0"},{"output_model":"GWDM","output_version":"1.1","input_model":"GWDM","input_version":"1.0"},{"output_model":"GWDM","output_version":"1.1","input_model":"GWDM","input_version":"1.2"},{"output_model":"GWDM","output_version":"1.1","input_model":"SchemaOrg","input_version":"BioSchema"},{"output_model":"GWDM","output_version":"1.2","input_model":"HDRUK","input_version":"2.2.1"},{"output_model":"GWDM","output_version":"1.2","input_model":"GWDM","input_version":"1.1"},{"output_model":"SchemaOrg","output_version":"default","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"SchemaOrg","output_version":"default","input_model":"GWDM","input_version":"1.1"},{"output_model":"SchemaOrg","output_version":"BioSchema","input_model":"GWDM","input_version":"1.0"},{"output_model":"SchemaOrg","output_version":"BioSchema","input_model":"GWDM","input_version":"1.1"},{"output_model":"SchemaOrg","output_version":"GoogleRecommended","input_model":"HDRUK","input_version":"2.1.2"},{"output_model":"SchemaOrg","output_version":"GoogleRecommended","input_model":"GWDM","input_version":"1.0"}] ``` === "Web Browser" diff --git a/docs/deleting-a-dataset.md b/docs/deleting-a-dataset.md index 0379cf8..2f2a6ec 100644 --- a/docs/deleting-a-dataset.md +++ b/docs/deleting-a-dataset.md @@ -1,10 +1,10 @@ -Deleting an dataset using `DELETE` on the endpoint [`/api/v1/integrations/datasets/{id}`](https://api.dev.hdruk.cloud/api/documentation#/Dataset%20Integrations/create_datasets_from_app) +Deleting an dataset using `DELETE` on the endpoint [`/api/v1/integrations/datasets/{id}`](https://api.dev.hdruk.cloud/api/documentation#/Datasets/delete_datasets_integrations) === " python " ```python response = requests.delete( - f"{api_path}/integrations/datasets/", + f"{api_path}integrations/datasets/", headers=headers ) @@ -20,10 +20,12 @@ Deleting an dataset using `DELETE` on the endpoint [`/api/v1/integrations/datase --header 'x-client-id: ' \ ``` -Running this returns the message if it was successfull +If successful, this returns ``` { "message": "success" } ``` + +Please see the previous page regarding dataset creation (`Using the API > Datasets > Create`) for details of the likely error messages you may receive from the `DELETE` endpoint. diff --git a/docs/fe-creating-an-app.md b/docs/fe-creating-an-app.md index 9d46c32..f1df420 100644 --- a/docs/fe-creating-an-app.md +++ b/docs/fe-creating-an-app.md @@ -1,12 +1,12 @@ ## Introduction The API Management Service enables data custodians to streamline metadata transfer to the Gateway -through the creation of API keys. This guide offers step-by-step instructions on setting up and managing the +through the creation of API keys. This guide offers step-by-step instructions for setting up and managing the API self-service on the Gateway. ## Create a new API key -The API management feature allows data custodians to create API keys and link them to the Gateway. +The API management feature allows data custodians to create API keys and ?link them to the Gateway. Please note that HDR UK cannot create application registrations on behalf of users. ### Step 1: Sign into the Gateway @@ -16,7 +16,7 @@ you need assistance with this step, contact the HDR UK technology team using the [https://www.healthdatagateway.org/about/contact-us](https://www.healthdatagateway.org/about/contact-us) -### Step 2: Access the Gateway API management +### Step 2: Access the Gateway API management page If you have the necessary permissions (Team admin or Developer), you can access the service by following these steps: @@ -29,11 +29,12 @@ _Fig 1: Create a new API key_ ### Step 3: Create a new API Key -When creating a new API, provide the following information (Fig 2): -Public app name: Name of the app you wish to create. -Description: A brief description of the app. -Notification contacts: Email addresses for relevant notification recipients. -After completing the required fields, click "Save & Continue" to proceed to the permissions page. +When creating a new API Key, provide the following information (Fig 2): + +- Public app name: Name of the app you wish to create. +- Description: A brief description of the app. +- Notification contacts: User accounts for relevant notification recipients. +- After completing the required fields, click "Save & Continue" to proceed to the permissions page. ![image](https://github.com/HDRUK/gateway-2-integrations-testing/assets/69473770/5667e21e-576d-4aec-9a96-92448c4375fc) _Fig 2: App info_ @@ -49,11 +50,13 @@ _Fig 3: Permissions matrix_ It is recommended that when creating Apps on the Gateway you only assign permissions that the app will need. We would suggest a minimal permissions approach, for instance: if an App is going to be managing -datasets we would recommend datasets.create, datasets.read, datasets.update and datasets.delete, there -would be no need to grant the app permissions to manage tools or Data Uses. -Once the desired permissions scope is set, click "Save & finish" to complete the API setup. Please note, at -the current time, only functionality for datasets has been enabled on the Gateway. -The API key will be revealed in the authentication tab after completing the app setup (Fig 4). The Client ID is +Datasets we would recommend `datasets.create`, `datasets.read`, `datasets.update` and `datasets.delete`. There +would be no need to grant the app permissions to manage Tools or Data Uses. + +Once the desired permissions scope is set, click "Save & Finish" to complete the API setup. Please note, at +the current time, only functionality for Datasets has been enabled on the Gateway. + +Click on the newly-created app\s entry. The API key will be revealed in the "Authentication" tab after completing the app setup (Fig 4). The Client ID is crucial for configuring your API settings; ensure to copy it and apply it to your API settings accordingly. ![image](https://github.com/HDRUK/gateway-2-integrations-testing/assets/69473770/2c0ec13f-bbe8-4a07-94ee-adc9b8c5620a) diff --git a/docs/fe-creating-fma.md b/docs/fe-creating-fma.md index 3258c70..96760e2 100644 --- a/docs/fe-creating-fma.md +++ b/docs/fe-creating-fma.md @@ -33,13 +33,13 @@ When creating a new integration, the following information needs to be provided: - Authentication Type: Select one of the authentication methods - API_Key, Bearer, or No_Auth. - **API_Key**: Provides a simple way for APIs to verify the systems accessing them. - **Bearer**: The service's script supports a static bearer token. It is strongly recommended to use HTTPS at all times to ensure security. If secure HTTP is not available, it is advisable not to use the service to prevent potential exploitation. - - **No_Auth**: Choose this option when authentication is not required to access your catalogue. + - **NO_Auth**: Choose this option when authentication is not required to access your catalogue. - Synchronisation Time: Specify the time at which the synchronisation process starts pulling data each day. - Base URL: Enter the main domain name of the API. - Datasets Endpoint: Provide the URL for listing all datasets available in the metadata catalogue. - Dataset Endpoint: Specify the URL that lists the latest version of metadata on the data custodians' servers. Please manually fill in this field to avoid making assumptions during the process. -- Auth Token: Enter the API key generated by the data custodian from their data server. -- Notification Contacts: Add the relevant email addresses for receiving notifications. +- Authorisation Token: (If `API_Key` or `Bearer` selected in "Authentication Type" above.) Enter the API key generated by the data custodian from their data server. +- Notification Contacts: Add the relevant individuals for receiving notifications. Once all the required fields are filled, click on “Save configuration” to store the information on the Gateway (Fig 3). The next step is to run a test to ensure the API connection works without any errors. @@ -101,14 +101,14 @@ The Interface Diagram below (Fig 6) shows how the Gateway integration ingestion The Gateway first contacts the /datasets endpoint you provide and interprets the response. It then compares the returned information with the existing records in the Gateway database. Based on the comparison, a decision will be made for each dataset on how the metadata will be handled. There are generally three scenarios: -### New Dataset +### 1. New Dataset If new data is detected through the ingestion script, it will be retrieved and stored in the Gateway database, and it will be made visible on the Gateway. -### Updated Dataset +### 2. Updated Dataset The Gateway ingestion script determines if a dataset has changed since the last synchronisation. It specifically compares the ID of the dataset and the version that was last provided with the current version. The script does not check for a newer version number but rather a different version number. This accounts for cases where a dataset may be reverted to a previous version. Updates to datasets are automatically made live on the Gateway, and the previous version of the dataset will be archived following existing Gateway processes. -### Delete Dataset +### 3.Delete Dataset The ingestion script can detect datasets that have been removed from the custodian metadata catalogue. If a dataset ID is no longer found in the /datasets endpoint, it will be considered a deleted dataset. A "deleted" dataset will be archived on the Gateway, along with all previous versions, and will no longer be visible on the Gateway following existing processes. diff --git a/docs/fe-login.md b/docs/fe-login.md index 9590fbd..1dcd61b 100644 --- a/docs/fe-login.md +++ b/docs/fe-login.md @@ -4,7 +4,7 @@ Login to the Gateway for beta testing: [https://web.dev.hdruk.cloud/en/sign-in]( Once you have logged in, in the top right corner, when you expand your profile, you will see all the teams that you are a member of. -You will then see the main page for you team: +Clicking on any team name, you will then see the main page for you team:
image diff --git a/docs/fma-setup.md b/docs/fma-setup.md deleted file mode 100644 index 58643f3..0000000 --- a/docs/fma-setup.md +++ /dev/null @@ -1,11 +0,0 @@ -The following information can be entered when creating (and then updating) an FMA integration with our FMA test service - -- Integration Type: Datasets -- Authentication Type: NO_Auth -- Base URL: https://hdr-gateway-fma-test-server-dev-qmnkcg5qjq-ew.a.run.app -- Datasets Endpoint: /api/v1/noauth/datasets -- Dataset Endpoint: /api/v1/noauth/dataset/{id} - -image - -As can be seen from the screenshot above, the successful testing (black box on right hand side) of this endpoint allows us to enable its usage. diff --git a/docs/fma-test-server.md b/docs/fma-test-server.md index e2c63ef..ff0b7ab 100644 --- a/docs/fma-test-server.md +++ b/docs/fma-test-server.md @@ -1,6 +1,8 @@ +We provide an example test server that fits the specification defined in [the FMA Setup page](https://hdruk.github.io/gateway-2-integrations-testing/fe-creating-fma/). + ### FMA Datasets -The main `datasets` payload for matches the [FMA schema](https://github.com/HDRUK/schemata-2/blob/master/hdr_schemata/models/FMA/fma.schema.json) and allows FMA to be able to pull metadata for your datasets: +The main `datasets` payload matches the [FMA schema](https://github.com/HDRUK/schemata-2/blob/master/hdr_schemata/models/FMA/fma.schema.json) and allows FMA to be able to pull metadata for your datasets: [https://hdr-gateway-fma-test-server-dev-qmnkcg5qjq-ew.a.run.app/api/v1/noauth/exemplar/datasets](https://hdr-gateway-fma-test-server-dev-qmnkcg5qjq-ew.a.run.app/api/v1/noauth/exemplar/datasets) @@ -50,7 +52,7 @@ The main `datasets` payload for matches the [FMA schema](https://github.com/HDRU ### Dataset Example -For example one of the datasets listed in the above datasets payload directs the service to the location of where to find metadata for a dataset that matches the HDRUK 2.1.2 schema. +Meanwhile, the `/datasets/{id}` endpoint provides information about the selected dataset in the HDRUK 2.1.2 schema. Example: @@ -82,3 +84,15 @@ Example: ... } ``` + +The following information can be entered when creating (and then updating) an FMA integration with the FMA test service: + +- Integration Type: Datasets +- Authentication Type: NO_Auth +- Base URL: https://hdr-gateway-fma-test-server-dev-qmnkcg5qjq-ew.a.run.app +- Datasets Endpoint: /api/v1/noauth/datasets +- Dataset Endpoint: /api/v1/noauth/dataset/{id} + +image + +As can be seen from the screenshot above, the successful testing (black box on right hand side) of this endpoint allows us to enable its usage. diff --git a/docs/metadata-validation.md b/docs/metadata-validation.md index 11e46fb..c293f50 100644 --- a/docs/metadata-validation.md +++ b/docs/metadata-validation.md @@ -1,11 +1,16 @@ ## Schema Validation -Once you have created your metadata, for example for the HDRUK schema `2.1.2`, you can use [various routes](https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app/docs/) of the translation service to check to see if it validates against this schema: +Once you have created your metadata, for example for the HDRUK schema `2.1.2`, you can use [various routes](https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app/docs/) of the translation service to check to see if it validates against this schema. + +Using the example file found at [https://github.com/HDRUK/gateway-2-integrations-testing/blob/master/example-hdruk212.json](https://github.com/HDRUK/gateway-2-integrations-testing/blob/master/example-hdruk212.json): === "python" Using python `requests` + ``` python + import json + import requests metadata = json.load(open("example-hdruk212.json")) @@ -25,9 +30,10 @@ Once you have created your metadata, for example for the HDRUK schema `2.1.2`, y === "CURL" ```bash - curl --location 'http://localhost:3001/find?with_errors=1' \ - --header 'Content-Type: application/json' \ - --data-raw '{}' + metadata='{"metadata": '$(cat example-hdruk212.json)'}' + curl --location 'https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app/find?with_errors=1'\ + --header 'Content-Type: application/json' --data \ + --data "$metadata" ``` Responds with: @@ -54,6 +60,22 @@ Responds with: }, "message": "must have required property 'publishingFrequency'" }, + { + "instancePath": "/provenance/temporal", + "schemaPath": "#/properties/temporal/anyOf/1/type", + "keyword": "type", + "params": { + "type": "null" + }, + "message": "must be null" + }, + { + "instancePath": "/provenance/temporal", + "schemaPath": "#/properties/temporal/anyOf", + "keyword": "anyOf", + "params": {}, + "message": "must match a schema in anyOf" + }, { "instancePath": "/provenance", "schemaPath": "#/properties/provenance/anyOf/1/type", @@ -72,7 +94,54 @@ Responds with: } ] }, - .... + { + "name": "HDRUK", + "version": "2.1.0", + "matches": false, + "errors": [ + { + "instancePath": "/summary/publisher/logo", + "schemaPath": "#/definitions/url/format", + "keyword": "format", + "params": { + "format": "uri" + }, + "message": "must match format \"uri\"" + } + ] + }, + { + "name": "HDRUK", + "version": "2.0.2", + "matches": false, + "errors": [ + { + "instancePath": "", + "schemaPath": "#/additionalProperties", + "keyword": "additionalProperties", + "params": { + "additionalProperty": "structuralMetadata" + }, + "message": "must NOT have additional properties" + } + ] + }, + { + "name": "HDRUK", + "version": "2.2.1", + "matches": false, + "errors": [ + { + "instancePath": "/summary", + "schemaPath": "#/required", + "keyword": "required", + "params": { + "missingProperty": "datasetType" + }, + "message": "must have required property 'datasetType'" + } + ] + }, { "name": "HDRUK", "version": "2.2.0", @@ -105,7 +174,54 @@ Responds with: } ] }, - ... + { + "name": "GWDM", + "version": "1.1", + "matches": false, + "errors": [ + { + "instancePath": "", + "schemaPath": "#/required", + "keyword": "required", + "params": { + "missingProperty": "required" + }, + "message": "must have required property 'required'" + } + ] + }, + { + "name": "GWDM", + "version": "1.2", + "matches": false, + "errors": [ + { + "instancePath": "", + "schemaPath": "#/required", + "keyword": "required", + "params": { + "missingProperty": "required" + }, + "message": "must have required property 'required'" + } + ] + }, + { + "name": "SchemaOrg", + "version": "default", + "matches": false, + "errors": [ + { + "instancePath": "", + "schemaPath": "#/required", + "keyword": "required", + "params": { + "missingProperty": "name" + }, + "message": "must have required property 'name'" + } + ] + }, { "name": "SchemaOrg", "version": "BioSchema", @@ -122,7 +238,22 @@ Responds with: } ] }, - ... + { + "name": "SchemaOrg", + "version": "GoogleRecommended", + "matches": false, + "errors": [ + { + "instancePath": "", + "schemaPath": "#/required", + "keyword": "required", + "params": { + "missingProperty": "name" + }, + "message": "must have required property 'name'" + } + ] + } ] ``` @@ -132,8 +263,17 @@ You can use the route `validate` instead of `find` to validate a payload against ```python + import json + import requests + metadata = json.load(open("example-hdruk212.json")) + headers = { + "Content-Type": "application/json", + } + + traser_uri = "https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app" + response = requests.post( f"{traser_uri}/validate?input_schema=HDRUK&input_version=2.1.2", headers=headers, json={"metadata":metadata} @@ -181,7 +321,8 @@ You can use the route `validate` instead of `find` to validate a payload against === "CURL" ``` - curl --location 'http://localhost:3001/validate?input_schema=HDRUK&input_version=2.1.2' \ + metadata='{"metadata": '$(cat example-hdruk212.json)'}' + curl --location 'https://hdr-gateway-traser-dev-qmnkcg5qjq-ew.a.run.app/validate?input_schema=HDRUK&input_version=2.1.2' \ --header 'Content-Type: application/json' \ - --data-raw '{}' + --data "$metadata" ``` diff --git a/docs/retrieving-a-dataset.md b/docs/retrieving-a-dataset.md index 4ff7f2a..01c6fa4 100644 --- a/docs/retrieving-a-dataset.md +++ b/docs/retrieving-a-dataset.md @@ -1,4 +1,4 @@ -The endpoint [`/api/v1/integrations/datasets/{id}`](https://api.dev.hdruk.cloud/api/documentation#/Dataset%20Integrations/create_datasets_from_app) can now be used to retrieve your data that you have just `POST` to the HDRUK gateway. +The endpoint [`/api/v1/integrations/datasets/{id}`](https://api.dev.hdruk.cloud/api/documentation#/Datasets/fetch_datasets_integrations) can now be used to retrieve your data that you have just `POST`ed to the HDRUK gateway. === " python " @@ -19,31 +19,71 @@ The endpoint [`/api/v1/integrations/datasets/{id}`](https://api.dev.hdruk.cloud/ --header 'x-client-id: ' ``` -Running this returns the dat that we store for your metadata +Running this returns the data that we store for your metadata ``` - { - "id": 16, - "created_at": "2024-02-09T12:10:23.000000Z", - "updated_at": "2024-02-09T12:10:23.000000Z", - "deleted_at": null, - "dataset_id": 16, - "metadata": { - "metadata": { - "required": { - "gatewayId": "16", - "gatewayPid": "5537be3a-e448-4214-8946-8ce1e75a74c8", - "issued": "2024-02-09T12:10:23.591675Z", - "modified": "2024-02-09T12:10:23.591698Z", - "revisions": [], - "version": "3.0.0" - }, - ... +{ + "message": "success", + "data": { + "id": 16, + "mongo_object_id": null, + "mongo_id": null, + "mongo_pid": null, + "datasetid": null, + "pid": "9b76e4c6-a9fc-40e9-9ab3-22c6a93f22c2", + "source": null, + "discourse_topic_id": 0, + "is_cohort_discovery": false, + "commercial_use": 0, + "state_id": 0, + "uploader_id": 0, + "metadataquality_id": 0, + "user_id": 5, + "team_id": 1, + "views_count": 0, + "views_prev_count": 0, + "has_technical_details": 1, + "created": "2024-04-04 13:29:30", + "updated": "2024-04-04 13:29:30", + "submitted": "2024-04-04 13:29:30", + "published": null, + "created_at": "2024-04-04T13:29:30.000000Z", + "updated_at": "2024-04-04T13:29:30.000000Z", + "deleted_at": null, + "create_origin": "API", + "status": "ACTIVE", + "named_entities": [], + "collections": [], + "versions": [ + { + "id": 662, + "created_at": "2024-04-04T13:29:30.000000Z", + "updated_at": "2024-04-04T13:29:30.000000Z", + "deleted_at": null, + "dataset_id": 662, + "metadata": { + "metadata": { + "required": { + "gatewayId": "662", + "gatewayPid": "9b76e4c6-a9fc-40e9-9ab3-22c6a93f22c2", + "issued": "2024-04-04T13:29:30.039074Z", + "modified": "2024-04-04T13:29:30.039087Z", + "revisions": [], + "version": "3.0.0" + }, + ... + } + } + }, + ... + ] + } +} ``` ### Alternative metadata schema -You can request to get your dataset metadata back using a different schema model/version, depending on what we have supported (see previous sections on available schemas and translations). +You can request to get your dataset metadata back using a different schema model/version, depending on what we support (see previous sections on available schemas and translations), using the `schema_model` and `scheme_version` query parameters. #### BioSchema @@ -66,7 +106,7 @@ You can request to get your dataset metadata back using a different schema model --header 'x-client-id: ' \ ``` -Will return the `data` payload with your metadata in your requested model (and version), if it is possible to translated between our GWDM and the output model. +Sample output: ```json @@ -84,9 +124,7 @@ Will return the `data` payload with your metadata in your requested model (and v #### HDRUK 2.2.0 (public schema) -Using `/api/v1/integrations/datasets/?schema_model=HDRUK&schema_version=2.2.0` - -You can get back your metadata conforming to our public schema from our gateway data model (GWDM) +Using `/api/v1/integrations/datasets/?schema_model=HDRUK&schema_version=2.2.0` you can get back your metadata conforming to our public schema from our Gateway Data Model (GWDM): ```json { @@ -111,7 +149,7 @@ You can get back your metadata conforming to our public schema from our gateway Please find below a summary of likely errors and their meanings. -#### Cannot find dataset by +#### Cannot find dataset by `dataset_id` If the `` that you use to retrieve your dataset (`/api/v1/integrations/datasets/`) does not exist or is invalid then you'll see the following error: @@ -141,7 +179,7 @@ If a dataset has been created and then subsquently deleted you will get the foll #### Unauthorised permissions to retrieve -If your app did enable permissions to retrieve a dataset then you'll see the following response (code `400`). +If your app does not have permissions to retrieve a dataset then you'll see the following response (code `400`): ```json { @@ -153,8 +191,7 @@ If your app did enable permissions to retrieve a dataset then you'll see the fol #### No schema version -If you request the data in another model, you must use `/integrations/datasets/{dataset_id}?schema_model={model}&schema_verson={version}"` -otherwise you may get a `500` response if you forgot to specify both: +If you request the data in another model, you must supply both `schema_model` and `schema_version`, e.g. `/integrations/datasets/{dataset_id}?schema_model={model}&schema_version={version}"`. Supplying only one will generate the following: ```json { @@ -166,7 +203,7 @@ otherwise you may get a `500` response if you forgot to specify both: #### Unknown output schema -If you request the output schema that does not exist or is not supported, example `/integrations/datasets/{dataset_id}?schema_model=HDRUK&schema_version=2.2.3`. You'll get the following response and error coming from our translation service. Navigate to 'creating metadata' to learn about what models and translations are supported. +If you request an output schema that does not exist or is not supported, for example `/integrations/datasets/{dataset_id}?schema_model=HDRUK&schema_version=2.2.3`, you will receive the following response and error from the translation service. ```json { @@ -183,3 +220,4 @@ If you request the output schema that does not exist or is not supported, exampl } } ``` +Navigate within these docs to `Metadata > Create` to learn about what models and translations are supported. diff --git a/docs/updating-a-dataset.md b/docs/updating-a-dataset.md index 0fd95e1..facf78a 100644 --- a/docs/updating-a-dataset.md +++ b/docs/updating-a-dataset.md @@ -1,4 +1,4 @@ -Method `PUT` [`/api/v1/integrations/datasets/{id}`](https://api.dev.hdruk.cloud/api/documentation#/Dataset%20Integrations/create_datasets_from_app) you can update your dataset metadata with a new version +Using the `PUT` method to [`/api/v1/integrations/datasets/{id}`](https://api.dev.hdruk.cloud/api/documentation#/Datasets/update_datasets_integrations) you can update your dataset metadata with a new _version_. === " python " @@ -40,14 +40,18 @@ Running this returns a structure like: "id": 875, ... }, + { + "id": 882, + ... + }, ... ] } } ``` -It returns what data we now hold for this dataset (i.e. multiple versions as you have now updated your dataset with a new version) +This shows the data we now hold for this dataset (i.e. multiple versions as you have now updated your dataset with a new version). ### Errors -Please see the previous page for dataset creation, the errors you may receive from an update will be similar to errors you can get from creating a dataset - they'll most likely be due to incorrect metadata or non-existing a non-existing dataset. +Please see the previous page regarding dataset creation (`Using the API > Datasets > Create`) for details of the likely error messages you may receive from the `PUT` endpoint. diff --git a/mkdocs.yml b/mkdocs.yml index 8061892..a856f7f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -33,8 +33,7 @@ nav: - Update: updating-a-dataset.md - Delete: deleting-a-dataset.md - Using FMA: - - Test Server: fma-test-server.md - - Setup on the Gateway: fma-setup.md + - Test Server Example: fma-test-server.md plugins: - search