Cannot get data object from .body attribute

[alexiswl]

Hello,

Trying some basic ProjectData get requests via the ICA SDK python.

I have a simple python script here:

Details

#!/usr/bin/env python3

"""
Test get data object with icasdk
"""

# Standard Imports
from os import environ
import logging
import json

# ICASDK Imports
import icasdk
from icasdk import Configuration, ApiClient, ApiException
from icasdk.apis.tags.project_data_api import ProjectDataApi

# Set Globals
DATA_ID = "fil.09fd831171954940ece408db581818a8"

# Set logging
logging.basicConfig(level=logging.DEBUG)

# Set configuration
logging.info("Set ICAv2 Configuration Construct")
configuration = Configuration(
    host=environ["ICAV2_BASE_URL"],  # https://ica.illumina.com/ica/rest
    access_token=environ["ICAV2_ACCESS_TOKEN"]  # icav2 tenants enter umccr-test
)

logging.info("Getting api instance context to collect Data")
with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = ProjectDataApi(api_client)

# example passing only required values which don't have defaults set
try:
    # Retrieve a project data.
    api_response = api_instance.get_project_data(
        path_params={
            'projectId': environ["ICAV2_PROJECT_ID"],
            'dataId': DATA_ID,
        },
    )
except ApiException as e:
    logging.error("Exception when calling ProjectDataApi->get_project_data: %s\n" % e)
    raise ApiException

logging.info("Getting 'body' attribute of api response to collect data object")
data_obj = api_response.body

logging.info("Data Object Looks like the following")
logging.info(f"{data_obj}")

logging.info("However if we try and look at the underlying attributes we get blank!")
logging.info(f"{data_obj.__dict__}")

logging.info("And collecting the data attribute (which the IDE seems to think is appropriate) also fails")

try:
    data_id = data_obj.data.id
except AttributeError as error:
    logging.error(f"Got the error {error}")

Which yields the following logs

Details

INFO:root:Set ICAv2 Configuration Construct
INFO:root:Getting api instance context to collect Data
INFO:root:Getting 'body' attribute of api response to collect data object
INFO:root:Data Object Looks like the following
INFO:root:DynamicSchema({'data': DynamicSchema({'id': 'fil.09fd831171954940ece408db581818a8', 'urn': 'urn:ilmn:ica:region:1efd315d-6309-4d7e-826b-d3824b0b5acb:data:fil.09fd831171954940ece408db581818a8#/illumina_reference_data/dragen_somatic_demo_data/HCC1187BL_S1_L001_R2_001.fastq.gz', 'details': DynamicSchema({'timeCreated': '2023-05-24T13:52:53Z', 'timeModified': '2023-05-26T01:52:23Z', 'tenantId': '194b3adf-3614-4bf7-a2a7-7561a059e416', 'tenantName': 'umccr', 'owningProjectId': '9c20aeb0-f780-4e9a-ac42-739b30ed91f3', 'owningProjectName': 'playground_v2', 'name': 'HCC1187BL_S1_L001_R2_001.fastq.gz', 'path': '/illumina_reference_data/dragen_somatic_demo_data/HCC1187BL_S1_L001_R2_001.fastq.gz', 'fileSizeInBytes': Decimal('20655195768'), 'status': <DynamicSchema: 'AVAILABLE'>, 'tags': DynamicSchema({'technicalTags': (), 'userTags': (), 'connectorTags': (), 'runInTags': (), 'runOutTags': (), 'referenceTags': ()}), 'format': DynamicSchema({'id': '7674cc19-2ab7-42fe-bac3-be16c83c720c', 'timeCreated': '2021-10-29T09:44:27Z', 'timeModified': '2021-10-29T09:44:27Z', 'ownerId': '499bfe9d-85bd-4588-ba70-fbc2f664bb9e', 'tenantId': '9fec8354-853b-4ee5-b211-eb03e484d876', 'tenantName': 'Illumina', 'code': 'FASTQ', 'description': 'FASTQ format is a text-based format for storing both a biological sequence (usually nucleotide sequence) and its corresponding quality scores.'}), 'dataType': <DynamicSchema: 'FILE'>, 'objectETag': '966531f6b9483d59b10ac913aedd7f39-2463', 'storedForTheFirstTimeAt': '2023-05-24T13:52:53Z', 'region': DynamicSchema({'id': '1efd315d-6309-4d7e-826b-d3824b0b5acb', 'timeCreated': '2021-11-05T10:12:33Z', 'timeModified': '2023-03-29T15:51:39Z', 'ownerId': '8ec463f6-1acb-341b-b321-043c39d8716a', 'tenantId': 'f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3', 'tenantName': 'ica-cp-admin', 'code': 'AU', 'country': DynamicSchema({'id': '68d3ebe4-83d4-4945-9e96-c140779ea107', 'timeCreated': '2021-10-29T09:44:25Z', 'timeModified': '2021-10-29T09:44:25Z', 'ownerId': '499bfe9d-85bd-4588-ba70-fbc2f664bb9e', 'tenantId': '9fec8354-853b-4ee5-b211-eb03e484d876', 'tenantName': 'Illumina', 'code': 'AU', 'name': 'Australia', 'region': 'earth'}), 'cityName': 'Sydney'})})}), 'projectId': 'eb511e09-1b6a-4ebd-9da4-bbdecbfb934d'})
INFO:root:However if we try and look at the underlying attributes we get blank!
INFO:root:{}
INFO:root:And collecting the data attribute (which the IDE seems to think is appropriate) also fails
ERROR:root:Got the error DynamicSchema({'data': DynamicSchema({'id': 'fil.09fd831171954940ece408db581818a8', 'urn': 'urn:ilmn:ica:region:1efd315d-6309-4d7e-826b-d3824b0b5acb:data:fil.09fd831171954940ece408db581818a8#/illumina_reference_data/dragen_somatic_demo_data/HCC1187BL_S1_L001_R2_001.fastq.gz', 'details': DynamicSchema({'timeCreated': '2023-05-24T13:52:53Z', 'timeModified': '2023-05-26T01:52:23Z', 'tenantId': '194b3adf-3614-4bf7-a2a7-7561a059e416', 'tenantName': 'umccr', 'owningProjectId': '9c20aeb0-f780-4e9a-ac42-739b30ed91f3', 'owningProjectName': 'playground_v2', 'name': 'HCC1187BL_S1_L001_R2_001.fastq.gz', 'path': '/illumina_reference_data/dragen_somatic_demo_data/HCC1187BL_S1_L001_R2_001.fastq.gz', 'fileSizeInBytes': Decimal('20655195768'), 'status': <DynamicSchema: 'AVAILABLE'>, 'tags': DynamicSchema({'technicalTags': (), 'userTags': (), 'connectorTags': (), 'runInTags': (), 'runOutTags': (), 'referenceTags': ()}), 'format': DynamicSchema({'id': '7674cc19-2ab7-42fe-bac3-be16c83c720c', 'timeCreated': '2021-10-29T09:44:27Z', 'timeModified': '2021-10-29T09:44:27Z', 'ownerId': '499bfe9d-85bd-4588-ba70-fbc2f664bb9e', 'tenantId': '9fec8354-853b-4ee5-b211-eb03e484d876', 'tenantName': 'Illumina', 'code': 'FASTQ', 'description': 'FASTQ format is a text-based format for storing both a biological sequence (usually nucleotide sequence) and its corresponding quality scores.'}), 'dataType': <DynamicSchema: 'FILE'>, 'objectETag': '966531f6b9483d59b10ac913aedd7f39-2463', 'storedForTheFirstTimeAt': '2023-05-24T13:52:53Z', 'region': DynamicSchema({'id': '1efd315d-6309-4d7e-826b-d3824b0b5acb', 'timeCreated': '2021-11-05T10:12:33Z', 'timeModified': '2023-03-29T15:51:39Z', 'ownerId': '8ec463f6-1acb-341b-b321-043c39d8716a', 'tenantId': 'f91bb1a0-c55f-4bce-8014-b2e60c0ec7d3', 'tenantName': 'ica-cp-admin', 'code': 'AU', 'country': DynamicSchema({'id': '68d3ebe4-83d4-4945-9e96-c140779ea107', 'timeCreated': '2021-10-29T09:44:25Z', 'timeModified': '2021-10-29T09:44:25Z', 'ownerId': '499bfe9d-85bd-4588-ba70-fbc2f664bb9e', 'tenantId': '9fec8354-853b-4ee5-b211-eb03e484d876', 'tenantName': 'Illumina', 'code': 'AU', 'name': 'Australia', 'region': 'earth'}), 'cityName': 'Sydney'})})}), 'projectId': 'eb511e09-1b6a-4ebd-9da4-bbdecbfb934d'}) has no attribute 'data'

Screen shot of IDE thinking it should be able to collect the .data attribute

And the child 'id' attribute

How am I supposed to use this SDK? Can you provide an example that collects the API response?

Alexis

[spacether]

@alexiswl

However if we try and look at the underlying attributes we get blank!

This is because the instance inherits from frozendict. Treat it as a normal dictionary and you will be able to access key value pairs.

From these docs:
https://github.com/Illumina/ica-sdk-python/blob/main/docs/models/ProjectData.md
We see that an instance of ProjectData is returned in api_response.body. One can access the properties data/projectId with instance.projectId or instance['projectId']. You can confirm this by making an instance of ProjectData in the interpreter and accessing its properties.

[alexiswl]

Thanks @spacether using the following works

project_id = data_obj['projectId']
data_id = data_obj['data']['id']

Just a little confused why the IDE thinks that it's okay to retrieve these as attributes of the object?
The benefit of having an SDK like this allow for faster code development through the perks of type-hinting.

Using attributes works for libica which uses a similar openapi generation technique, so would be great to see the type-hinting features replicated here too

[spacether]

Per the readme payloads subclass all validated schema classed so dot property access is available at runtime. If this project updates their generator to openapi Json schema generator v3 or higher property methods will be written and are seen in the idea.

[alexiswl]

so dot property access is available at runtime.

I'm not seeing this behaviour when I run it.

Traceback (most recent call last):
  File "/tmp/tmp.Evrx4k5DC6/sdk-test.py", line 54, in <module>
    print(data_obj.projectId)
    ^^^^^^^^^^^^^^^^
    raise AttributeError(f"{self} has no attribute '{name}'")

[spacether]

Hmm would have to check. It could be because the properties are optional so properties weren't written for them in an older version.

[alexiswl]

@spacether these properties definitely are not optional.

The swagger this sdk is derived from is here: https://ica.illumina.com/ica/api/swagger/index.html

With the ProjectData output here - https://ica.illumina.com/ica/api/swagger/index.html#/Project%20Data/getProjectData

[alexiswl]

@kcutler0 do you have an ETA on when you expect this be solved?

[kcutler0]

I'll take a look this week and provide an update or at least a working example.

[spacether]

If this project is upgraded to use the latest version of the python client, which is in this repo
https://github.com/openapi-json-schema-tools/openapi-json-schema-generator
Then required an optional property acessors will always be generated, and optional properties will return an Unset instance of the property was not set. You can see this working here:
https://github.com/openapi-json-schema-tools/openapi-json-schema-generator/blob/master/samples/client/petstore/python/src/petstore_api/components/schema/animal.py#L33
In the new version the returned instance does not subclass all validated schemas, just the first schema defined in the document. Under the hood all validation is run.

[alexiswl]

@kcutler0 how'd you go with this?

[kcutler0]

Working on that now for v2.21 branch. I checked the differences in parameters between the openapi generators (openapi-generator-cli vs openapi-generator-cli.jar) and didn't see major differences that would change the actual issue, but I did update to later version of the generator v7.1 as well as add some of the parameters from your makefile https://github.com/umccr-illumina/libica/blob/main/syncapi2.sh

--global-property=apiDocs=true,apiTests=true,modelDocs=true,modelTests=true
I'm working on updating some local config issues and am hoping to have a solution next week.

[alexiswl]

@victorskl might have a bit more insight on the libica parameters

[alexiswl]

@kcutler0 how did you go with this?

[kcutler0]

@alexiswl I promise I'm not ignoring the issue, I'm just having trouble getting the later version of openapi-generator-cli that include pydantic validation to allow the import of the generated module. I don't have a workaround yet but am still working on it

[alexiswl]

@kcutler0 any updates on this project?

[alexiswl]

@kcutler0 following this up