Survey of installed Python dependencies (requirements.txt, commands.json, tox.ini, etc.)

[foolip]

Similar to #28801, I've had a look at what Python dependencies we install, rather than vendoring them into tool/third_party/. These are the mechanisms I'm aware of:

• requirements.txt and requirements_*.txt files
• install fields in commands.json files (and requirements fields which can refer to requirements.txt files)
• deps sections in tox.ini files (which often refer to requirements.txt files)
• direct calls to pip install in scripts or CI steps

Here are the dependencies, starting with some special groups and then alphabetically...

marionette_driver + mozcrash + mozdebug + mozinstall + mozleak + moznetwork + mozprofile + mozrunner + mozversion

These are dependencies for Firefox required only by tools/wptrunner/requirements_firefox.txt. (Some others are used more broadly, see below.)

flake8 + pep8-naming + pycodestyle + pyflakes

Dependencies for Python coding style in tools/tox.ini, via tools/requirements_flake8.txt. Not installed by any ./wpt command.

mypy + mypy-extensions + typed-ast

Dependencies for Python type checking in tools/tox.ini, via tools/requirements_mypy.txt. Not installed by any ./wpt command.

pytest + pytest-cov + hypothesis + mock

Dependencies for tests for tools/ in tools/tox.ini, via tools/requirements_pytest.txt. These are not the same as the pytest + dependencies used to run webdriver/tests/, those are separately vendored with different versions, see #28801.

requests + taskcluster + pyyaml + json-e + jsonschema

Dependencies in tools/tox.ini used in CI. Not installed by any ./wpt command. These aren't pinned, but I haven't seen things suddenly break because of them.

Sphinx + sphinx-argparse + recommonmark

These are in docs/requirements.txt, dependencies for building web-platform-tests.org, wired up to ./wpt build-docs but in CI that isn't used.

Template-Python + html5lib + lxml + mercurial + webencodings

These are the dependencies for css/build-css-testsuites.sh via css/requirements.txt. Seemingly not run in CI?

aioquic

Dependency for ./wpt serve-quic-transport in tools/quic/requirements.txt, but this is also a vendored dependency, see #28801. The versions aren't the same, which one ends up being used?

html5lib

In tools/wptrunner/requirements.txt, but is also a vendored dependency, see #28801. Notably, the webencodings dependency isn't in tools/wptrunner/requirements.txt...

Also in css/requirements.txt, see above.

Also in resources/test/tox.ini, used to extract inline test expectations.

mozinfo

In tools/wpt/requirements.txt and tools/wptrunner/requirements.txt, but seems like it's ultimately only used for Firefox, since it's related to mozinfo.json files that are probably only for Firefox.

mozinstall

Required by ./wpt install, via tools/wpt/commands.json.

mozlog

In tools/wptrunner/requirements.txt, used for logging in lots of places.

mozprocess

This is listed in many files:

tools/wptrunner/requirements_android_webview.txt
tools/wptrunner/requirements_chrome.txt
tools/wptrunner/requirements_chrome_android.txt
tools/wptrunner/requirements_chrome_ios.txt
tools/wptrunner/requirements_edge.txt
tools/wptrunner/requirements_edge_chromium.txt
tools/wptrunner/requirements_epiphany.txt
tools/wptrunner/requirements_firefox.txt
tools/wptrunner/requirements_ie.txt
tools/wptrunner/requirements_opera.txt
tools/wptrunner/requirements_safari.txt
tools/wptrunner/requirements_sauce.txt
tools/wptrunner/requirements_servo.txt
tools/wptrunner/requirements_webkit.txt

All the usage is with mozprocess.ProcessHandler, in these files:

tools/wptrunner/wptrunner/browsers/chrome_android.py
tools/wptrunner/wptrunner/browsers/firefox.py
tools/wptrunner/wptrunner/browsers/servodriver.py
tools/wptrunner/wptrunner/executors/executorservo.py
tools/wptrunner/wptrunner/webdriver_server.py

Maybe this can be simplified somehow?

pillow

A dependency in tools/wptrunner/requirements.txt needed for RefTestImplementation.

psutil

From tools/wptrunner/requirements_safari.txt, used for killing Safari processes.

Also in tools/wptrunner/requirements_firefox.txt but seemingly not used for running Firefox?

requests

In these files:
tools/wpt/requirements.txt
tools/wptrunner/requirements.txt
tools/wptrunner/requirements_sauce.txt

Used for requests.get and requests.post as you'd expect.

Also in resources/test/tox.ini and tools/tox.ini, but see #28777.

Also in tools/ci/commands.json, tools/docker/commands.json and tools/wpt/commands.json.

selenium

A dependency of anything built on tools/wptrunner/wptrunner/executors/executorselenium.py:

tools/wptrunner/requirements_edge.txt
tools/wptrunner/requirements_ie.txt
tools/wptrunner/requirements_opera.txt
tools/wptrunner/requirements_sauce.txt

six

In tools/wptrunner/requirements.txt, #28776 tracks getting rid of wpt-internal usage. (It would still reamain as a transitive in dependency.)

Also in css/requirements.txt, see above.

Also in resources/test/tox.ini, but see #28777.

ua-parser

The only dependency in tools/wave/requirements.txt, used in tools/wave/utils/user_agent_parser.py.

urllib3

Listed in tools/wptrunner/requirements.txt, it's a dependency of selenium and requests, never used directly in wpt.

virtualenv

The only dependency for the ./wpt CLI beyond Python itself. Installed in CI, but no specific version is required. #20019 discusses avoiding this dependency.

wptserve

Listed in annotation-protocol/requirements.txt and can presumably be removed since wptserve is part of wpt now, see #28810.

zstandard

Required by ./wpt run, via tools/wpt/commands.json.

[foolip]

@jgraham pointed out install sections in commands.json files, so I updated the description, but didn't list every last dependency. pygithub and pyyaml are also in use.