Globally added X-Robots-Tag: noindex HTTP response header #324

perronld · 2023-05-01T17:35:51Z

Overview

Search engines currently indexes dev/staging installations of Birdhouse. This may create SEO issues for organizations.

Changes

Add optional-components/x-robots-tags-header and X_ROBOTS_TAGS_HEADER variable to allow setting the desired
header value server-wide.

Breaking changes

X-Robots-Tag: noindex response header is globally added to nginx. This changes is breaking only if you wish to be indexed by search engines.

github-actions

Thanks for submitting a PR.
Make sure you have looked at CONTRIBUTING guidelines.

crim-jenkins-bot · 2023-05-01T18:50:02Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1524/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : x-robots-tag
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-118.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1105/

NOTEBOOK TEST RESULTS

    
[2023-05-01T17:53:07.428Z] ============================= test session starts ==============================
[2023-05-01T17:53:07.428Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-05-01T17:53:07.428Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-05-01T17:53:07.429Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-05-01T17:53:07.429Z] collected 268 items
[2023-05-01T17:53:07.429Z] 
[2023-05-01T17:53:16.381Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-05-01T17:53:24.107Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-05-01T17:53:31.453Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-05-01T17:53:41.389Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-05-01T17:53:44.326Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-05-01T18:01:14.738Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-05-01T18:01:14.738Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-05-01T18:01:18.640Z] ...............                                                          [ 26%]
[2023-05-01T18:01:28.174Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-05-01T18:02:01.105Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-05-01T18:02:17.903Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-05-01T18:02:19.288Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-05-01T18:02:24.591Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-05-01T18:02:30.983Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-05-01T18:06:33.401Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-05-01T18:07:41.319Z] .............                                                            [ 47%]
[2023-05-01T18:07:46.019Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-05-01T18:07:48.240Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-05-01T18:08:15.505Z] .................                                                        [ 58%]
[2023-05-01T18:08:23.131Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-05-01T18:08:24.521Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-05-01T18:08:41.002Z] .........                                                                [ 64%]
[2023-05-01T18:08:53.524Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-05-01T18:09:02.887Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-05-01T18:09:39.919Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-05-01T18:09:41.838Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-05-01T18:09:49.072Z] ......                                                                   [ 81%]
[2023-05-01T18:12:25.641Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-05-01T18:13:16.365Z] .............                                                            [ 86%]
[2023-05-01T18:13:28.610Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-05-01T18:16:00.965Z] ....s.                                                                   [ 89%]
[2023-05-01T18:16:09.102Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-05-01T18:16:14.202Z] ...                                                                      [ 91%]
[2023-05-01T18:16:29.129Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-05-01T18:16:51.214Z] ......                                                                   [ 93%]
[2023-05-01T18:16:53.659Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-05-01T18:20:39.333Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-05-01T18:20:39.333Z] 
[2023-05-01T18:20:39.333Z] ================= 265 passed, 3 skipped in 1644.90s (0:27:24) ==================

fmigneault

To make this non-breaking for everyone, it should be defined as an optional-component.
In this component, a nginx conf would be added and mounted in the proxy docker-compose service, such that in can be loaded by the include /etc/nginx/conf.d/*.conf; step.
The instances that want to make use of this component will have to add it explicitly in EXTRA_CONF_DIRS of their env.local file.

perronld · 2023-05-01T19:28:34Z

Before working further on this PR, lets see first if anyone actually wish to be indexed by public search engines.

huard · 2023-05-01T19:31:36Z

Not sure I understand 100%, but I don't see why we'd want the staging and dev instances to be indexed.

fmigneault · 2023-05-01T19:41:28Z

For staging and dev instances, I agree that indexing is most probably irrelevant. However, this definition can also be used for prod. So we must allow toggling the option if any indexing should be permitted for any use case.

perronld · 2023-05-01T20:14:34Z

I think that the indexing is a problem even in prod.
In our case, Googlebot crawled Geoserver and Thredds (including wps_outputs).

fmigneault · 2023-05-01T23:19:30Z

Maybe in the case of CRIM's instances, even prod indexing is not relevant because of our use cases, but other organizations could be taking advantage of it. For example, https://pavics.ouranos.ca/ entrypoint has a lot of documentation that indexes could facilitate search/matches. It could be the same eventually for U-Toronto that intends to swap the Jupyter Notebook interface by their own. For this reason, it should be an "opt-in" feature.

tlvu · 2023-05-02T02:29:08Z

I think for Ouranos, we want the homepage of PAVICS to be indexed so we can be discovered.

Not sure I want all the data in Thredds and Geoserver to be indexed though. And no indexing for staging and test servers of course.

Is there a way to exclude some path from being indexed?

@huard @tlogan2000 Any problems to have our data on Thredds and Geoserver being indexed by Google?

tlvu · 2023-05-02T02:30:41Z

In our case, Googlebot crawled Geoserver and Thredds (including wps_outputs).

Oh geez ! Which log files we see this and what kind of log pattern? I'd like to check if we've been crawled too.

tlogan2000 · 2023-05-02T11:54:39Z

Any problems to have our data on Thredds and Geoserver being indexed by Google?

I think I would prefer only the homepage as you suggest but have no real idea how much trouble it is to avoid having indexing on geoserver / thredds. Aany ballpark estimate in terms of amount of effort involved?

perronld · 2023-05-02T13:26:15Z

I got the URL list from a report in Google Search console. If you want to look in the logs, you can always filter for specific user-agents, such as "Googlebot". Is the PAVICS homepage deployed using birdhouse-deploy too? If that's the case, then the nginx design makes things easy:

There could be several add_header directives. These directives are inherited from the previous configuration level if and only if there are no add_header directives defined on the current level.

So, for example, in the location block of the homepage, all we have to do is to add add_header X-Dummy-Header asdf to cancel the X-Robots-Tag that I added in the http block.

fmigneault · 2023-05-02T18:10:57Z

Isn't there support for a robots.txt file at the root of the website as well? With this, you can normally have specific paths you want included or excluded for crawling, and by which bots.

huard · 2023-05-02T19:17:06Z

Note the existence of https://developers.google.com/search/docs/appearance/structured-data/dataset

So one option would be to turn off THREDDS indexing, but add formatted metadata information to our landing page describing the datasets we want to be indexed.

fmigneault · 2023-05-02T21:44:10Z

There's a lot of overlap between the JSON-LD Dataset definition and the STAC definitions.
Maybe the JSON-LD Dataset could be generated by an output plugin with https://github.com/cedadev/stac-generator while the THREDDS contents are generated for STAC collections themselves. However, we will have to figure out how to inject the resulting JSON-LD definitions in the HTML <head> of each relevant dataset...

tlvu · 2023-05-03T13:31:05Z

So, for example, in the location block of the homepage, all we have to do is to add add_header X-Dummy-Header asdf to cancel the X-Robots-Tag that I added in the http block.

The homepage is immediately at the root so I think overriding the root with add_header X-Dummy-Header asdf will not work.

However, /thredds and /geoserver are under their own location directives so something like this might work? See examples from https://yoast.com/x-robots-tag/

location ~* \.(doc|pdf)$ {
    add_header  X-Robots-Tag "noindex, noarchive, nosnippet";
}

This means we should not forget to add this header for each new location directive we add to the proxy. Not so ideal but not sure what are other solutions since the homepage is at the root.

Another solution is move the homepage to /homepage and the root will simply redirect to /homepage by default.

In all cases, I think this X-Robots-Tag should probably be optional and opt-in, unless we all agree that allow crawling of Thredds and Geoserver is a bad idea.

perronld · 2023-05-03T13:37:31Z

Unfortunately, I don't know how the homepage is configured, but even if it's at the root, it should be within a location / { ... } block

tlvu · 2023-05-03T13:46:15Z

Unfortunately, I don't know how the homepage is configured, but even if it's at the root, it should be within a location / { ... } block

Root location block here

birdhouse-deploy/birdhouse/config/proxy/conf.d/all-services.include.template

Lines 1 to 3 in 1981a1d

    
           location / { 
        
               ${PROXY_ROOT_LOCATION} 
        
           }

Default value when homepage not activated

birdhouse-deploy/birdhouse/config/proxy/default.env

Lines 9 to 11 in 1981a1d

    
           # Content of "location /" in file config/proxy/conf.d/all-services.include.template 
        
           # Useful to have a custom homepage. 
        
           export PROXY_ROOT_LOCATION="return 302 https://\$host/jupyter/hub/login;"

Value when homepage activated

birdhouse-deploy/birdhouse/env.local.example

Line 219 in 1981a1d

#export PROXY_ROOT_LOCATION="alias /data/homepage/;"

But given this is the root location directive, doesn't it recursively set the value for all the sub location directives? It's possible I misunderstood how multiple location directives work together.

perronld · 2023-05-03T13:49:39Z

I added the X-Robots-Tag header in the http block, which is parent to all location blocks, so if any location block has the add_header directive, then if will cancel the one from the http block.
So, the fix for the homepage would be to set the variable like this:
export PROXY_ROOT_LOCATION="alias /data/homepage/; add_header X-Dummy-Header asdf;"

tlvu · 2023-05-03T18:42:16Z

I added the X-Robots-Tag header in the http block, which is parent to all location blocks, so if any location block has the add_header directive, then if will cancel the one from the http block.
So, the fix for the homepage would be to set the variable like this:
export PROXY_ROOT_LOCATION="alias /data/homepage/; add_header X-Dummy-Header asdf;"

That seems to work !

@perronld Please document this opt-out for the homepage and describe this change in the CHANGES.md file (Usually some form of copy/paste of the PR description).

@mishaschwartz @eyvorchuk any concern from UofT and PCIC side, about this globally denying seach engine indexing of PAVICS instances? Note an opt-out exists for the homepage.

Before:

$ curl --include --silent https://boreas.ouranos.ca/geoserver/ | head -20                                  
HTTP/1.1 302 
Server: nginx/1.23.4
Date: Wed, 03 May 2023 18:33:00 GMT
Content-Length: 0
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Set-Cookie: JSESSIONID=A34F9B545B03CBAC2282CCE2025C5EC4; Path=/geoserver; HttpOnly
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Access-Control-Allow-Origin,Access-Control-Allow-Credentials
Location: /geoserver/index.html


$ curl --include --silent https://boreas.ouranos.ca/index.html | head -20
HTTP/1.1 200 OK
Server: nginx/1.23.4
Date: Wed, 03 May 2023 18:31:05 GMT
Content-Type: text/html
Content-Length: 1475
Last-Modified: Fri, 10 Mar 2023 17:17:47 GMT
Connection: keep-alive
ETag: "640b663b-5c3"
X-Robots-Tags: noindex
Accept-Ranges: bytes

<!doctype html>
<html>
  <head>

After:

$ curl --include --silent https://boreas.ouranos.ca/geoserver/ | head -20                                                                                            
HTTP/1.1 302 
Server: nginx/1.23.4
Date: Wed, 03 May 2023 18:29:23 GMT
Content-Length: 0
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Set-Cookie: JSESSIONID=74625D96DA69A90A4A3CBE082A834CE4; Path=/geoserver; HttpOnly
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Access-Control-Allow-Origin,Access-Control-Allow-Credentials
Location: /geoserver/index.html
X-Robots-Tags: noindex

$ curl --include --silent https://boreas.ouranos.ca/index.html | head -20                                                                                            
HTTP/1.1 200 OK
Server: nginx/1.23.4
Date: Wed, 03 May 2023 18:29:42 GMT
Content-Type: text/html
Content-Length: 1475
Last-Modified: Fri, 10 Mar 2023 17:17:47 GMT
Connection: keep-alive
ETag: "640b663b-5c3"
X-Dummy-Header: asdf
Accept-Ranges: bytes

<!doctype html>
<html>
  <head>

tlvu · 2023-05-03T19:19:12Z

@fmigneault does this Global X-Robots-Tag impact later implementation of #325 ? We want to advertize data on Thredds to various web crawlers/indexers?

fmigneault · 2023-05-03T23:07:35Z

if any location block has the add_header directive, then if will cancel the one from the http block.
So, the fix for the homepage would be to set the variable like this:
export PROXY_ROOT_LOCATION="alias /data/homepage/; add_header X-Dummy-Header asdf;"

does this Global X-Robots-Tag impact later implementation of #325 ? We want to advertize data on Thredds to various web crawlers/indexers?

Each location block would need a specific variable, similar to PROXY_ROOT_LOCATION, to allow per-endpoint overrides.
I don't think this solution is quite the right one. The X-Robots-Tags: noindex is still present in the /geoserver/ response, so crawlers are not available for any other endpoint other than the entrypoint. If we wanted to allow also /thredds/, there is no way to allow both / and /thredds/, and keep others like /geoserver/ still with disallowed indexing.

Not sure why the example before case of /geoserver/ did not have the X-Robots-Tags: noindex? Seems like this produces the opposite of what was expected (allow crawl only on the entrypoint, and not anywhere else).

I don't really like the solution to inject a random header to cancel another one. Seems like a very obscure workaround that most won't understand intuitively. Seems like it would be easier to only add X-Robots-Tags: noindex where we want them instead.

mishaschwartz · 2023-05-04T13:54:16Z

I don't really like the solution to inject a random header to cancel another one

What if we used the default all value to override subpaths instead of a random string like asdf

The all value is documented here: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#directives

fmigneault · 2023-05-04T21:26:01Z

What if we used the default all value to override subpaths instead of a random string like asdf

That makes more sense since we will be working with the same X-Robots-Tag header.

tlvu · 2023-05-04T21:36:37Z

The all value is documented here: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#directives

Should we be extra sure and add all the "deny" there? noindex, nofollow, noarchive, nositelinkssearchbox, nosnippet, notranslate, noimageindex

fmigneault · 2023-08-07T16:27:27Z

I think noindex, nofollow are sufficient.
The others seem to depend on the index to occur to be generated, or are irrelevant with the contents that we have.

fmigneault · 2023-09-01T00:13:18Z

@perronld @huard @tlvu @mishaschwartz
I pushed some commits to transfer the original change proposed into a separate optional-component.

@perronld
On our side, that would mean only adding the following to our daccs-config env.local for all staging/dev/prod instances.

EXTRA_CONFS_DIR='
   <...>
   optional-components/x-robots-tags-header
'

crim-jenkins-bot · 2023-09-01T01:40:24Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1998/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : x-robots-tag
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-67.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1279/

NOTEBOOK TEST RESULTS

    
[2023-09-01T01:00:00.759Z] ============================= test session starts ==============================
[2023-09-01T01:00:00.759Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-01T01:00:00.759Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-09-01T01:00:00.759Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-01T01:00:00.759Z] collected 236 items
[2023-09-01T01:00:00.759Z] 
[2023-09-01T01:00:10.261Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-01T01:00:20.489Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-01T01:00:30.465Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-01T01:00:35.913Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ......FF       [ 13%]
[2023-09-01T01:12:35.963Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-01T01:12:35.963Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 19%]
[2023-09-01T01:12:41.338Z] ...............                                                          [ 25%]
[2023-09-01T01:12:50.355Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-01T01:12:57.064Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-09-01T01:13:12.447Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-01T01:13:13.826Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 33%]
[2023-09-01T01:13:20.673Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 36%]
[2023-09-01T01:13:24.947Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 38%]
[2023-09-01T01:16:38.459Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-01T01:17:53.938Z] .............                                                            [ 50%]
[2023-09-01T01:17:56.675Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-01T01:17:58.890Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 55%]
[2023-09-01T01:18:16.578Z] .................                                                        [ 62%]
[2023-09-01T01:18:23.815Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-01T01:18:25.196Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb F [ 65%]
[2023-09-01T01:18:40.194Z] .FF......                                                                [ 69%]
[2023-09-01T01:18:49.548Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-01T01:18:58.909Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 76%]
[2023-09-01T01:19:00.820Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-01T01:19:03.562Z] ......                                                                   [ 79%]
[2023-09-01T01:19:10.149Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-01T01:19:26.618Z] .............                                                            [ 85%]
[2023-09-01T01:19:36.625Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-01T01:20:12.168Z] ....s.                                                                   [ 88%]
[2023-09-01T01:20:20.309Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-01T01:20:33.890Z] ...                                                                      [ 89%]
[2023-09-01T01:20:48.815Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-09-01T01:21:10.517Z] ......                                                                   [ 92%]
[2023-09-01T01:21:12.996Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-01T01:24:17.152Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-09-01T01:24:17.152Z] 
[2023-09-01T01:24:17.152Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-09-01T02:22:31Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1999/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : x-robots-tag
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-166.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1280/

NOTEBOOK TEST RESULTS

    
[2023-09-01T01:40:46.310Z] ============================= test session starts ==============================
[2023-09-01T01:40:46.310Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-01T01:40:46.310Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-09-01T01:40:46.310Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-01T01:40:46.310Z] collected 236 items
[2023-09-01T01:40:46.310Z] 
[2023-09-01T01:40:54.380Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-01T01:41:02.600Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-01T01:41:12.318Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-01T01:41:17.754Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .FFFFFFF       [ 13%]
[2023-09-01T01:54:43.750Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-01T01:54:43.750Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 19%]
[2023-09-01T01:54:47.940Z] ...............                                                          [ 25%]
[2023-09-01T01:54:57.583Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-01T01:55:04.897Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-09-01T01:55:21.649Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-01T01:55:23.036Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 33%]
[2023-09-01T01:55:25.718Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb EEEEEEE            [ 36%]
[2023-09-01T01:55:30.137Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 38%]
[2023-09-01T01:58:12.387Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-01T01:59:31.941Z] .............                                                            [ 50%]
[2023-09-01T01:59:34.682Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-01T01:59:37.278Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 55%]
[2023-09-01T01:59:54.534Z] .................                                                        [ 62%]
[2023-09-01T02:00:02.963Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-01T02:00:04.877Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb F [ 65%]
[2023-09-01T02:00:20.169Z] ..F......                                                                [ 69%]
[2023-09-01T02:00:29.280Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-01T02:00:38.612Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 76%]
[2023-09-01T02:00:40.025Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-01T02:00:43.411Z] ......                                                                   [ 79%]
[2023-09-01T02:00:55.633Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-01T02:01:14.087Z] .............                                                            [ 85%]
[2023-09-01T02:01:26.314Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-01T02:02:12.835Z] ....s.                                                                   [ 88%]
[2023-09-01T02:02:20.968Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-01T02:02:35.405Z] ...                                                                      [ 89%]
[2023-09-01T02:02:50.293Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-09-01T02:03:12.572Z] ......                                                                   [ 92%]
[2023-09-01T02:03:15.317Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-01T02:06:19.427Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-09-01T02:06:19.427Z] 
[2023-09-01T02:06:19.427Z] ==================================== ERRORS ====================================

crim-jenkins-bot · 2023-09-01T02:23:07Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2000/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : x-robots-tag
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-149.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1281/

NOTEBOOK TEST RESULTS

    
[2023-09-01T01:41:44.907Z] ============================= test session starts ==============================
[2023-09-01T01:41:44.908Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-01T01:41:44.908Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-09-01T01:41:44.908Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-01T01:41:44.908Z] collected 236 items
[2023-09-01T01:41:44.908Z] 
[2023-09-01T01:41:52.970Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-01T01:42:06.344Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-01T01:42:27.563Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-01T01:42:38.437Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 13%]
[2023-09-01T01:54:59.655Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-01T01:54:59.655Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 19%]
[2023-09-01T01:54:59.655Z] ...............                                                          [ 25%]
[2023-09-01T01:55:08.553Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-01T01:55:17.397Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-09-01T01:55:33.558Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-01T01:55:35.210Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 33%]
[2023-09-01T01:55:40.231Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 36%]
[2023-09-01T01:55:44.394Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 38%]
[2023-09-01T01:58:12.796Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-01T01:59:27.821Z] .............                                                            [ 50%]
[2023-09-01T01:59:30.677Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-01T01:59:33.119Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 55%]
[2023-09-01T01:59:50.461Z] .................                                                        [ 62%]
[2023-09-01T01:59:57.945Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-01T01:59:59.325Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb F [ 65%]
[2023-09-01T02:01:09.736Z] ..F...FFF                                                                [ 69%]
[2023-09-01T02:01:20.299Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-01T02:01:29.362Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 76%]
[2023-09-01T02:01:31.269Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-01T02:01:34.605Z] ......                                                                   [ 79%]
[2023-09-01T02:01:49.558Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-01T02:02:09.360Z] .............                                                            [ 85%]
[2023-09-01T02:02:19.364Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-01T02:02:59.534Z] ....s.                                                                   [ 88%]
[2023-09-01T02:03:07.665Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-01T02:03:23.271Z] ...                                                                      [ 89%]
[2023-09-01T02:03:35.507Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-09-01T02:03:55.941Z] ......                                                                   [ 92%]
[2023-09-01T02:03:58.942Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-01T02:06:59.590Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-09-01T02:06:59.590Z] 
[2023-09-01T02:06:59.590Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-09-01T17:55:31Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2003/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : x-robots-tag
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1282/

NOTEBOOK TEST RESULTS

    
[2023-09-01T17:19:22.872Z] ============================= test session starts ==============================
[2023-09-01T17:19:22.872Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-01T17:19:22.872Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-09-01T17:19:22.872Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-01T17:19:22.872Z] collected 236 items
[2023-09-01T17:19:22.872Z] 
[2023-09-01T17:19:31.355Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-01T17:19:41.191Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-01T17:19:50.897Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-01T17:19:57.858Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 13%]
[2023-09-01T17:27:33.701Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-01T17:27:35.610Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 19%]
[2023-09-01T17:27:43.044Z] ...............                                                          [ 25%]
[2023-09-01T17:27:53.492Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-01T17:28:01.288Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-09-01T17:28:17.565Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-01T17:28:18.957Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 33%]
[2023-09-01T17:28:27.026Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 36%]
[2023-09-01T17:28:31.545Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 38%]
[2023-09-01T17:32:30.285Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-01T17:33:45.272Z] .............                                                            [ 50%]
[2023-09-01T17:33:47.182Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-01T17:33:49.763Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 55%]
[2023-09-01T17:34:06.165Z] .................                                                        [ 62%]
[2023-09-01T17:34:14.064Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-01T17:34:15.974Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb F [ 65%]
[2023-09-01T17:34:17.031Z] .FFFFFFFF                                                                [ 69%]
[2023-09-01T17:34:27.440Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-01T17:34:37.176Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 76%]
[2023-09-01T17:34:39.090Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-01T17:34:41.892Z] ......                                                                   [ 79%]
[2023-09-01T17:34:50.032Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-01T17:35:04.801Z] .............                                                            [ 85%]
[2023-09-01T17:35:14.822Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-01T17:35:56.954Z] ....s.                                                                   [ 88%]
[2023-09-01T17:36:05.082Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-01T17:36:19.832Z] ...                                                                      [ 89%]
[2023-09-01T17:36:32.062Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-09-01T17:36:53.735Z] ......                                                                   [ 92%]
[2023-09-01T17:36:56.576Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-01T17:40:00.734Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-09-01T17:40:00.734Z] 
[2023-09-01T17:40:00.734Z] =================================== FAILURES ===================================

mishaschwartz

I really like this idea. I just have one questions about the way its set up:

Is there a reason why this needs to be configured in an optional component? Why not just make it an environment variable in the proxy component? That seems like it would be simpler since this setting only changes a single line in the proxy configuration.

Or are you thinking that this could be expanded later to update additional header related settings that are not specifically for the proxy?

fmigneault · 2023-09-06T14:19:53Z

Using only an environment variable would work as well.
However, the user would have to write it properly to include the full directive, e.g.:

export X_ROBOTS_TAGS='add_header X-Robots-Tags "noindex, nofollow";'

I felt it was less error-prone to abstract that need for them to have the exact nginx syntax.
I also took advantage of the usual optional-component documentation approach to provide more details about the feature.

For now, the component only overrides X-Robots-Tags globally, but we could have additional variables later on if need be for per-location overrides.

tlvu

+1 for me as well. Please document the opt-out procedure in the optional-components/README.rst

For example, to allow crawling for the homepage, we would do this export PROXY_ROOT_LOCATION="alias /data/homepage/; add_header X-Robots-Tags: all;" in env.local

fmigneault · 2023-09-06T16:49:40Z

@tlvu Added in 497ded4

crim-jenkins-bot · 2023-09-06T17:36:04Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2010/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : x-robots-tag
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1287/

NOTEBOOK TEST RESULTS

    
[2023-09-06T16:56:24.351Z] ============================= test session starts ==============================
[2023-09-06T16:56:24.352Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-06T16:56:24.352Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-09-06T16:56:24.352Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-06T16:56:24.352Z] collected 236 items
[2023-09-06T16:56:24.352Z] 
[2023-09-06T16:56:31.902Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-06T16:56:40.999Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-06T16:56:51.762Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-06T16:56:58.455Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 13%]
[2023-09-06T17:07:07.186Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-06T17:07:07.186Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 19%]
[2023-09-06T17:07:11.948Z] ...............                                                          [ 25%]
[2023-09-06T17:07:20.724Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-06T17:07:27.479Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-09-06T17:07:43.464Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-06T17:07:44.848Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 33%]
[2023-09-06T17:07:52.967Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 36%]
[2023-09-06T17:07:57.484Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 38%]
[2023-09-06T17:12:40.199Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-06T17:13:44.186Z] .............                                                            [ 50%]
[2023-09-06T17:13:48.648Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-06T17:13:50.856Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 55%]
[2023-09-06T17:14:06.453Z] .................                                                        [ 62%]
[2023-09-06T17:14:14.291Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-06T17:14:15.679Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 65%]
[2023-09-06T17:14:30.336Z] .F.......                                                                [ 69%]
[2023-09-06T17:14:39.334Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-06T17:14:48.314Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 76%]
[2023-09-06T17:14:49.695Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-06T17:14:52.757Z] ......                                                                   [ 79%]
[2023-09-06T17:14:59.346Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-06T17:15:19.286Z] .............                                                            [ 85%]
[2023-09-06T17:15:31.547Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-06T17:16:04.881Z] ....s.                                                                   [ 88%]
[2023-09-06T17:16:13.017Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-06T17:16:25.588Z] ...                                                                      [ 89%]
[2023-09-06T17:16:40.501Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-09-06T17:17:01.351Z] ......                                                                   [ 92%]
[2023-09-06T17:17:03.647Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-06T17:19:58.265Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-09-06T17:19:58.265Z] 
[2023-09-06T17:19:58.265Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-09-06T18:00:42Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2011/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : x-robots-tag
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov

## Overview Please include a summary of the changes and which issues are fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Changes **Non-breaking changes** - Fix incorrect tag ``X-Robots-Tags`` header to appropriate ``X-Robots-Tag`` (no final ``s``) name for ``optional-components/x-robots-tags-header``. **Breaking changes** - n/a ## Related Issue / Discussion - Relates to #324

Globally added X-Robots-Tag: noindex HTTP response header

ed0a5e5

perronld requested review from huard and tlvu May 1, 2023 17:35

github-actions bot reviewed May 1, 2023

View reviewed changes

fmigneault requested changes May 1, 2023

View reviewed changes

fmigneault mentioned this pull request May 2, 2023

💡 [Feature] Generate JSON-LD Dataset definition for crawlers using STAC metadata #325

Open

fmigneault added 2 commits August 31, 2023 19:50

Merge remote-tracking branch 'origin/master' into x-robots-tag

77214f1

use optional component x-robots-tags-header

d2b2116

github-actions bot added the documentation Improvements or additions to documentation label Sep 1, 2023

fmigneault added 2 commits August 31, 2023 20:16

update changelogs

ad34b4b

remove hardcoded x-robots-tags header

5c6b208

add x-robots-tags quotes to handle comma-separated value

f8542d3

mishaschwartz reviewed Sep 6, 2023

View reviewed changes

tlvu approved these changes Sep 6, 2023

View reviewed changes

mishaschwartz approved these changes Sep 6, 2023

View reviewed changes

add mention about x-robots-tags nginx override for homepage location

497ded4

tlvu approved these changes Sep 6, 2023

View reviewed changes

Bump version: 1.29.2 → 1.30.0

76453b3

github-actions bot added the ci/operations Continuous Integration components label Sep 6, 2023

fmigneault approved these changes Sep 6, 2023

View reviewed changes

fmigneault merged commit d06df29 into master Sep 6, 2023
3 checks passed

fmigneault deleted the x-robots-tag branch September 6, 2023 17:41

fmigneault mentioned this pull request Sep 11, 2023

fix X-Robots-Tag header name #379

Merged

Globally added X-Robots-Tag: noindex HTTP response header #324

Globally added X-Robots-Tag: noindex HTTP response header #324

Conversation

perronld commented May 1, 2023 • edited by fmigneault Loading

Overview

Changes

github-actions bot left a comment

Choose a reason for hiding this comment

crim-jenkins-bot commented May 1, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

fmigneault left a comment

Choose a reason for hiding this comment

perronld commented May 1, 2023

huard commented May 1, 2023

fmigneault commented May 1, 2023

perronld commented May 1, 2023

fmigneault commented May 1, 2023

tlvu commented May 2, 2023

tlvu commented May 2, 2023 • edited Loading

tlogan2000 commented May 2, 2023

perronld commented May 2, 2023 • edited Loading

fmigneault commented May 2, 2023

huard commented May 2, 2023

fmigneault commented May 2, 2023

tlvu commented May 3, 2023

perronld commented May 3, 2023 • edited Loading

tlvu commented May 3, 2023

perronld commented May 3, 2023

tlvu commented May 3, 2023

tlvu commented May 3, 2023 • edited Loading

fmigneault commented May 3, 2023

mishaschwartz commented May 4, 2023

fmigneault commented May 4, 2023

tlvu commented May 4, 2023 • edited Loading

fmigneault commented Aug 7, 2023

fmigneault commented Sep 1, 2023 • edited Loading

crim-jenkins-bot commented Sep 1, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Sep 1, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Sep 1, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Sep 1, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

mishaschwartz left a comment

Choose a reason for hiding this comment

fmigneault commented Sep 6, 2023

tlvu left a comment

Choose a reason for hiding this comment

fmigneault commented Sep 6, 2023

crim-jenkins-bot commented Sep 6, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Sep 6, 2023

E2E Test Results

DACCS-iac Pipeline Results

perronld commented May 1, 2023 •

edited by fmigneault

Loading

tlvu commented May 2, 2023 •

edited

Loading

perronld commented May 2, 2023 •

edited

Loading

perronld commented May 3, 2023 •

edited

Loading

tlvu commented May 3, 2023 •

edited

Loading

tlvu commented May 4, 2023 •

edited

Loading

fmigneault commented Sep 1, 2023 •

edited

Loading