linkcheck inconsistent behavior from client

[fmigneault]

Describe the bug

Running on this project: https://github.com/crim-ca/weaver/

Local Ubuntu 20.04.6 LTS instance, Python 3.10.13 produces :

(         changes: line 1633) broken    https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff - 406 Client Error: Not Acceptable for url: https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff
(autoapi/weaver/base/index: line   72) redirect  https://stackoverflow.com/a/5191224 - with Found to https://stackoverflow.com/questions/5189699/how-to-make-a-class-property/5191224#5191224

On GitHub CI, Python 3.10.13 produces (https://github.com/crim-ca/weaver/actions/runs/7962895777/job/22037962351?pr=604) :

(         changes: line 1633) ok        https://github.com/geopython/pywps/commit/343d82539576b1e73eee3102654749c3d3137cff
(autoapi/weaver/base/index: line   72) broken    https://stackoverflow.com/a/5191224 - 403 Client Error: Forbidden for url: https://stackoverflow.com/a/5191224

Both locations run Sphinx 7.2.6

The errors for the same URLs are replicated over multiple runs, but somehow do not align on different machines.

How to Reproduce

Use the listed URLs in some RST document, and see if they get flagged by linkcheck.
The command make check-links can be used from https://github.com/crim-ca/weaver/ to test exactly the repo where the problem occurs.

Environment Information

Platform:              linux; (Linux-5.15.0-97-generic-x86_64-with-glibc2.31)
Python version:        3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0])
Python implementation: CPython
Sphinx version:        7.2.6
Docutils version:      0.18.1
Jinja2 version:        3.1.2
Pygments version:      2.17.1

> pip freeze
alabaster==0.7.13
alembic==1.12.1
amqp==5.2.0
anyascii==0.3.2
argcomplete==3.1.6
asciitree==0.3.3
astor==0.8.1
astroid==2.15.8
attrs==23.1.0
Babel==2.13.1
bandit==1.7.5
Beaker==1.12.1
beautifulsoup4==4.12.2
billiard==4.2.0
boto3==1.29.4
boto3-stubs==1.29.4
botocore==1.32.4
botocore-stubs==1.32.4
bump2version==1.0.1
CacheControl==0.13.1
celery==5.3.5
certifi==2023.11.17
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
cligj==0.7.2
codacy-coverage==1.3.11
colander==1.8.3
coloredlogs==15.0.1
contextlib2==21.6.0
cornice==6.0.1
cornice-swagger @ git+https://github.com/fmigneault/cornice.ext.swagger.git@b66bd4d85f5f5550f39d046742ae9799aee1f153
coverage==7.3.2
cryptography==41.0.5
cwl-upgrader==1.2.10
cwl-utils==0.31
cwltool==3.1.20230906142556
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.7
dnspython==2.4.2
doc8==0.11.2
docformatter==1.5.0
docker==6.1.3
docutils==0.18.1
dparse==0.6.3
duration==1.1.1
esgf-compute-api @ git+https://github.com/ESGF/esgf-compute-api.git@8faf829abb3ba0434715c4db44440cfb9c71b55f
execnet==2.0.2
fasteners==0.19
filelock==3.13.1
fiona==1.9.5
flake8==5.0.4
flynt==1.0.1
future==0.18.3
geotiff==0.2.10
gitdb==4.0.11
GitPython==3.1.40
greenlet==3.0.1
gunicorn==21.2.0
humanfriendly==10.0
humanize==4.8.0
hupper==1.12
idna==3.4
imagesize==1.4.1
importlib-resources==6.1.1
iniconfig==2.0.0
iso8601==2.1.0
isodate==0.6.1
isort==5.12.0
Jinja2==3.1.2
jmespath==1.0.1
json2xml==4.1.0
jsonschema==4.20.0
jsonschema-specifications==2023.11.1
kombu==5.3.4
lazy-object-proxy==1.9.0
lxml==4.9.3
Mako==1.3.0
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mccabe==0.7.0
mdurl==0.1.2
mistune==2.0.5
mock==3.0.5
moto==4.2.14
msgpack==1.0.7
mypy==1.7.0
mypy-boto3-s3==1.29.3
mypy-extensions==1.0.0
networkx==3.2.1
numcodecs==0.12.1
numpy==1.26.2
oauthlib==3.2.2
OWSLib==0.29.3
packaging==21.3
parameterized==0.9.0
Paste==3.7.1
PasteDeploy==3.1.0
path==16.7.1
path.py==12.5.0
pbr==6.0.0
Pint==0.22
plaster==1.1.2
plaster-pastedeploy==1.0.1
platformdirs==4.0.0
pluggy==1.3.0
prompt-toolkit==3.0.41
prov==1.5.1
psutil==5.9.6
py==1.11.0
pycodestyle==2.9.1
pycparser==2.21
pydocstyle==6.3.0
pydot==1.4.2
pyflakes==2.5.0
Pygments==2.17.1
pylint==2.17.7
pylint-per-file-ignores==1.3.2
pylint-quotes==0.2.3
pymongo==4.6.0
pyparsing==3.1.1
pyproj==3.6.1
pyramid==2.0.2
pyramid-beaker==0.8
pyramid-celery @ git+https://github.com/crim-ca/pyramid_celery.git@3d4c5096661176b176cab4b7d31b282317c70301
pyramid-mako==1.1.0
pyramid-rewrite==0.2
pyramid-storage==1.3.0
pytest==6.2.5
pytest-fixture-config==1.7.0
pytest-httpserver==1.0.8
pytest-rerunfailures==12.0
pytest-server-fixtures==1.7.0
pytest-shutil==1.7.0
python-dateutil==2.8.2
pytz==2023.3.post1
pywps==4.6.0
PyYAML==6.0.1
rdflib==7.0.0
referencing==0.31.0
requests==2.31.0
requests-file==1.5.1
requests-magpie @ git+https://github.com/Ouranosinc/requests-magpie@b26ed3c2cee4395c1bd745c7e102a442ba2199ae
responses==0.24.1
restructuredtext-lint==1.4.0
retry==0.9.2
rich==13.7.0
rpds-py==0.13.1
ruamel.yaml==0.17.40
ruamel.yaml.clib==0.2.8
s3transfer==0.7.0
safety==2.3.5
schema-salad==8.4.20231117150958
shapely==2.0.2
shellescape==3.8.1
simplejson==3.19.2
six==1.16.0
smmap==5.0.1
snowballstemmer==2.2.0
soupsieve==2.5
Sphinx==7.2.6
sphinx-argparse==0.4.0
sphinx-autoapi==3.0.0
sphinx-autodoc-typehints==1.25.2
sphinx-paramlinks==0.6.0
sphinx-rtd-theme==1.3.0
sphinxcontrib-applehelp==1.0.7
sphinxcontrib-devhelp==1.0.5
sphinxcontrib-htmlhelp==2.0.4
sphinxcontrib-jquery==4.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.6
sphinxcontrib-redoc==1.6.0
sphinxcontrib-serializinghtml==1.1.9
SQLAlchemy==2.0.23
stevedore==5.1.0
stopit==1.1.2
termcolor==2.3.0
tifffile==2023.9.26
toml==0.10.2
tomli==2.0.1
tomlkit==0.12.3
translationstring==1.4
types-awscrt==0.19.13
types-s3transfer==0.7.0
typing_extensions==4.8.0
tzdata==2023.3
untokenize==0.1.1
urllib3==2.0.7
urlmatch==1.0.1
venusian==3.1.0
vine==5.1.0
waitress==2.1.2
wcwidth==0.2.11
-e git+ssh://git@github-crim/crim-ca/weaver.git@781dac1bb5429beba99580a4f1118224ef46b6a1#egg=weaver
WebOb==1.8.7
websocket-client==1.6.4
WebTest==3.0.0
Werkzeug==3.0.1
wrapt==1.16.0
WSGIProxy==0.2.2
WSGIProxy2==0.5.1
xmltodict==0.13.0
zarr==2.12.0
zope.deprecation==5.0
zope.interface==6.1

Sphinx extensions

https://github.com/crim-ca/weaver/blob/d911c161721c2e25ee4663d34a49d2b218261710/docs/source/conf.py#L54-L68

Additional context

Error seen on crim-ca/weaver#604 (but unrelated changes)
Last run that did not raise any issue by linkcheck as on crim-ca/weaver@d234e11 (ie: https://github.com/crim-ca/weaver/actions/runs/7890099969/job/21545963203)
No apparent change to documentation links between them (crim-ca/weaver@d234e11...cmd-err-handling).

[jayaddison]

Hi @fmigneault - unfortunately I think that this is most likely to be due to stackoverflow choosing to respond differently to HTTP requests depending on the origin. If that's so, then in the short-term may have to adapt by using linkcheck configuration options.

I'm going to convert this issue into a discussion topic since it isn't immediately clear that there is a bug or enhancement request for the Sphinx codebase here - but perhaps we'll discover one as a result.

[fmigneault]

I'm very familiar with Python, urllib and requests, but I assume linkcheck does some additional operations to handle specific use cases? I didn't dig deeper into the rate-limiting/retries logic it employs.

[fmigneault]

I reran linkcheck again:
https://github.com/crim-ca/weaver/actions/runs/7962895492/job/22587462163

Just to make it harder by not helping with the inconsistencies, this time, both links that caused 403 and 406 are OK
Instead, I got another link that was working just fine, and reporting 405

(           index: line  290) broken    https://app.dimensions.ai/details/grant/grant.8105745 - 405 Client Error: Not Allowed for url: https://app.dimensions.ai/details/grant/grant.8105745

Following that link with the (I expect) GET request works.
So, either linkchek did another HTTP method (OPTIONS, HEAD)? that the server did not like, or something else happened, but either way it shouldn't fail.

[fmigneault]

Reran linkcheck for a 3rd time: https://github.com/crim-ca/weaver/actions/runs/7962895777/job/22625974814?pr=604

I still get the above 405, and the 403/406 are gone.

I'm not sure what happens when running in a GitHub action, but I cannot replicate the problem using either curl, requests or Sphinx's own requests util, using corresponding parameters to what are performed by the code:

sphinx/sphinx/builders/linkcheck.py

Lines 430 to 437 in 5523c9b

	with retrieval_method(
	url=req_url, auth=auth_info,
	headers=headers,
	timeout=self.timeout,
	**kwargs,
	_user_agent=self.user_agent,
	_tls_info=(self.tls_verify, self.tls_cacerts),
	) as response:

❯ curl --head  https://app.dimensions.ai/details/grant/grant.8105745 -H 'Accept: text/html,application/xhtml+xml;q=0.9,*/*;q=0.8' -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
HTTP/1.1 200 OK
Date: Wed, 13 Mar 2024 19:04:16 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 20015
Connection: keep-alive
Set-Cookie: session=dj4sQCP_3-wvWwf71Dpk9729-MhM9iMZAVjpbe3o6dMGk3j5t50megBF7aOl2d1bjIMl5lKZb0MoL-oFHupeIVsxNzEwMzU2NjU2LCAxNzEwMzU2NjU2Ljc0MDA4NTQsIHsiX2NzcmZ0XyI6ICJlZDVkYjlkNTQ2NmE0OWVjNmE1ODY3NjY2NDVjNTgxMjIwMzZhOWY2In1d; Path=/; secure; HttpOnly; SameSite=Lax
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

import requests
r = requests.head("https://app.dimensions.ai/details/grant/grant.8105745", headers={"Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8", "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"})
r.status_code
202

from sphinx.util import requests
r = requests._Session().head("https://app.dimensions.ai/details/grant/grant.8105745", headers={"Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8"}, _user_agent="Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0")
r.status_code
202

[fmigneault]

Seems like the 405 might be related to robots handling (https://app.dimensions.ai/robots.txt).
When doing the request in my browser, I can see for a brief moment an intermediate request for human verification.
Some kind of javascript process seems to be involved, and this causes the site to detect linkcheck requests as a robot.
Following some requests issues, I have found this thread: psf/requests#5003
There is a proposal of using requests_html.HTMLSession instead of a typical requests.Session to simulate a browser that can perform the javascript handling as if it was accessed by an actual browser.
Maybe an option to use an alternate session class could be a solution?

[fmigneault]

As a demonstration about how inconsistent linkcheck can be regarding which status responses are obtained, I ran the 2 following PRs at roughly the same time:

• let sphinx pick a more recent user-agent for docs linkcheck crim-ca/weaver#609
• use typing Unpack for request options crim-ca/weaver#610

The first one failed a few times until I gave up, and applied an explicit ignore rule for the problematic URL.
The second PR, however, did not encounter any of the problem HTTP codes, even though it did not change anything regarding doc links.
This is comparison to crim-ca/weaver@c079e4e (at that time master for the 2 PRs), which was still indicating a broken link due to HTTP 405.