GA collection download failing

[evacougnon]

This issue tracker is only for AODN Portal issues.

For reference: https://aodn.testrail.com/index.php?/tests/view/84288

Steps to reproduce the issue

1. https://portal-systest.aodn.org.au/search?uuid=d887e71a-71dc-4851-94a9-920f7b7cc7e5
2. Go to step 3 and select download as GeoTIFF several times (10+ times)

Expected behaviour

CAPTCHA property should activate
The download should commence and notification in the pop up informs us

Actual behaviour

Warning on step 2 of this collection "There is a problem with the availability of this collection" , and no filters available
After clicking download several time (and accept the license screen), still don't get any notifications that the download started and no CAPTCHA

Screenshots.

What version of the aodn portal are you using?

4.42.97

Which browser are you using?

Chrome

Any extra information that might be useful in the debugging process.

[craigrose]

GA WMS is failing:
That is a wrong url. Should be warehouse.ausseabed.gov.au

[craigrose]

Endpoint:
https://warehouse.ausseabed.gov.au/geoserver/wms?request=GetCapabilities
Possibly layers (search for Southern_Indian_Ocean):
ausseabed:Southern_Indian_Ocean__MH370__Bathymetry_2017_150m_HS
ausseabed:Southern_Indian_Ocean__MH370__Bathymetry_2017_150m_L0_Coverage (WFS)
ausseabed:Southern_Indian_Ocean__MH370__Bathymetry_2017_150m
ausseabed:Southern_Indian_Ocean__MH370__Bathymetry_2017_150m_OV
But this needs to be checked.

Using this layer: ausseabed:Southern_Indian_Ocean__MH370__Bathymetry_2017_150m

Harvest is not updating the record correctly:
POT - https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/metadata/100315

On harvest there is a transform that uses:

https://github.com/aodn/collection-config/blob/262522cbed8c3f0d75001f5b4751542ff33e1b9a/geoscience-australia/d887e71a-71dc-4851-94a9-920f7b7cc7e5.xml

We probably need to update that.

The configs get applied here: https://github.com/aodn/geonetwork-build/blob/5490a5221f4ec0562eb9ff331790821fbcd2c17f/main/src/main/webapp/xsl/conversion/common/add-collection-config.xsl#L134

PRs:
WMS/WCS Layers: https://github.com/aodn/collection-config/tree/issue-2942-fix-mh370
WFS Filter config: https://github.com/aodn/filter-config/tree/issue-2942-fix-mh370 (merged to 'test' branch)

Test metadata record: http://geonetwork3-cmrose0.dev.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/d887e71a-71dc-4851-94a9-920f7b7cc7e5
Portal layer: http://aodnportal-cmrose0.dev.aodn.org.au/search?uuid=d887e71a-71dc-4851-94a9-920f7b7cc7e5

TODO:

• Update https://github.com/aodn/filter-config/blob/2d86dd05fa2cda88fae9e97e62a4e34b48357052/ga-geoserver/marine/bathymetry_150m_all_3857.xml
• Update https://github.com/aodn/collection-config/blob/262522cbed8c3f0d75001f5b4751542ff33e1b9a/geoscience-australia/d887e71a-71dc-4851-94a9-920f7b7cc7e5.xml
• Check that the test is OK

[Chetnamann]

What version of the aodn portal are you using?
4.42.104

Which browser are you using?
Chrome

[craigrose]

Should be using "ausseabed:Southern_Indian_Ocean__MH370__Bathymetry_2017_150m_OV".
But this layer does not have any WFS service and hence no filters.
Portal does not seem to handle the idea of no filters for a layer. The layer works but we get a message saying the layer is unavailable.
Tried a config that removes this message but now we have "Loading subset parameters‥" remaining.
Faking the date filter works but need to confirm this doesn't break the download. Also had to add a "fake" geom filter.

Download

The available download for this layer uses WCS returned as a GeoTiff. It supports GET and POST (with XML payload) requests. The complete download is ~3 GBytes.

To access this service, the AODN portal uses the class JsonService extends AsyncDownloadService. AsyncDownloadService is the base class for other services such as aws-wps, gogoduck etc. JsonService uses POST with a json body to build the wcs request.

As the GA service is not configured to accept POST with json payload this fails.

http://warehouse.ausseabed.gov.au/geoserver/ows?service=WCS&acceptversions=2.0.1&request=GetCapabilities

http://warehouse.ausseabed.gov.au/geoserver/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageID=ausseabed__Southern_Indian_Ocean__MH370__Bathymetry_2017_150m_OV

To fix this we need to eithier write/rewrite a service in Portal to form the required POST request or have GA reconfigure the layer. The details of reconfiguring the layer to match our current code are yet to be determined. If they were to reconfigure, it would also be useful to have them set up WFS as well so we can get Portal to be able to create some filters. Such filters should be used in the WCS request

Request:

POST http://{{host}}/geoserver/wcs
{
    "bboxInOutputCRS": "69.389648,-70.665039,200.610352,17.665039",
    "emailAddress": "[email protected]",
    "outputCRS": "EPSG:4326",
    "format": "GeoTIFF",
    "coverages": "ausseabed:Southern_Indian_Ocean__MH370__Bathymetry_2017_150m_OV"
}

Response:

<?xml version="1.0" encoding="UTF-8"?>
<ows:ExceptionReport xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ows="http://www.opengis.net/ows" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0.0" xsi:schemaLocation="http://www.opengis.net/ows http://warehouse.ausseabed.gov.au/geoserver/schemas/ows/1.0.0/owsExceptionReport.xsd">
    <ows:Exception exceptionCode="NoApplicableCode">
        <ows:ExceptionText>com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '{' (code 123) in prolog; expected '&lt;'
 at [row,col {unknown-source}]: [1,1]
Unexpected character '{' (code 123) in prolog; expected '&lt;'
 at [row,col {unknown-source}]: [1,1]</ows:ExceptionText>
    </ows:Exception>
</ows:ExceptionReport>

[craigrose]

Just dumping this here for prosperity:

From my observations I can see that the layer https://portal.aodn.org.au/search?uuid=a05f7892-eef2-7506-e044-00144fdd4fa6 appears to have flow control turned on (Set-Cookie: GS_FLOW_CONTROL=GS_CFLOW_-5123ca63:1891f7e897d:-15c3) which is probably helping somewhat but layer loading is slow and I still see failed wms requests. The slow speed suggests that geowebcache is not being used. In fact, I don't think it is because the response headers do not contain the expected keys:

eg:
geowebcache-tile-index: [1, 1, 2]
geowebcache-cache-result: HIT
geowebcache-tile-index: [1, 1, 2]
geowebcache-tile-bounds: -90.0,0.0,0.0,90.0
geowebcache-gridset: EPSG:4326
geowebcache-crs: EPSG:4326

Failed wms requests will most often (but not always) appear when zooming, panning or doing a page reload after an initial successful load of the layer. Overall I feel like it has improved - but that is subjective.

As for the MH370 layer - I'm still working on getting the download to work. Availability of the collection was affected by the fact that the layer does not have any WFS service and hence cannot find any filters. Portal assumes that there will be at least one filter and breaks if there isn't. I have found a fix for that, but still need to get the download working.

[vfisaac]

@Chetnamann - the issue with the GA collection download failing is unrelated to the Test case which is testing this issue
"Captcha stops download commencement"
As the test case is already testing the issue with gridded data collection - could you please remove Steps 2 and 3 from the test
and create a new test case for testing the GA download.

Let me know if you have any questions

[craigrose]

Had a meeting with Natalie (GA). They will look into what they can do with this. Meanwhile it is blocked. See @atkinsn for details.