Don't require virtualenv

[foolip]

When working on #19940 I had this question: do pip and virtualenv really need to be on PATH, or could the instructions be simplified by using them as modules instead?

It looks like ./wpt run never invokes pip internally, and running without pip on PATH works.

For virtualenv, however, not having it on PATH runs into this code:

wpt/tools/wpt/virtualenv.py

Lines 28 to 30 in ccbf1f9

	self.virtualenv = find_executable("virtualenv")
	if not self.virtualenv:
	raise ValueError("virtualenv must be installed and on the PATH")

Finding virtualenv in a way similar to pip here would remove the requirement:

wpt/tools/wpt/virtualenv.py

Lines 48 to 59 in ccbf1f9

	@property
	def bin_path(self):
	if sys.platform in ("win32", "cygwin"):
	return os.path.join(self.path, "Scripts")
	return os.path.join(self.path, "bin")
	@property
	def pip_path(self):
	path = find_executable("pip", self.bin_path)
	if path is None:
	raise ValueError("pip not found")
	return path

Alternatively, using pip and virtualenv as python -m pip/virtualenv or as Python modules could remove the requirement.

@jgraham are there complexities hiding here that I haven't run into yet?

@LukeZielinski how is this dealt with in Chromium?

[foolip]

Looks like the answer for Chromium is --skip-venv-setup.

[foolip]

Actually, that pip_path helper is not analogous to what I'm suggesting. That'll be the _venv/bin/pip binary, it doesn't find pip outside of PATH, like in ~/Library/Python/2.7/bin/pip.

[LukeZielinski]

In Chromium, we set --skip-venv-setup and also set --venv to a directory with an already-setup venv.

[foolip]

I tried #20026 and for me locally on macOS, it means PATH doesn't need to be modified.

[foolip]

After struggling in #27377 I wonder if we still need virtualenv or could simplify for contributors by relying on just venv instead? As far as I can tell, what we need is:

• an isolated environment using whatever python executable ./wpt ended up using
• pip at some minimum version (19.2.3 doesn't work for me)
• installing requirements using pip
• ensuring those installed dependencies are on PYTHONPATH, if that's how it works

The _venv directory is also used to save browser and driver binaries, but I'm not sure if anything needs to be put on PATH or not.

@jgraham @gsnedders before I go too far down this path, are there complications here I'm not aware of yet?

[jgraham]

In general I like this idea. I just poked at it and quickly ran into https://stackoverflow.com/questions/27462582/how-can-i-activate-a-pyvenv-vitrualenv-from-within-python-activate-this-py-was (we're using execfile with activate_this.py into the python environment to set up the required paths &c. but venv doesn't support that, so we'd need a different solution there).

[foolip]

Aha, so that explains something. I didn't understand what exec() in activate() was, and it's like eval() in JavaScript.

I also noticed there was no activate_this.py, but I wonder what parts of it we need? Might we be able to get away with just using pip to install into _venv/something, and then setting PYTHONPATH?

[jgraham]

I would at least be newrous if we didn't copy ~all of activate_this's behavior. So we could just vendor in that file I guess and hope that Python doesn't change in ways that require it to be updated.

[gsnedders]

How difficult would it be to move to creating a subprocess and invoking the venv python?

[foolip]

@gsnedders I've experimented with that, invoking python -m venv from the main process works, but I haven't worked out how to update the main process to use the resulting directory without activate_this.py. I think that updating sys.path is actually sufficient, and if it is then a hack would be to invoke a script in the venv which prints sys.path, and then update the main process to match.

But I'm not too sure about that. I'm considering this also in light of web-platform-tests/rfcs#82, which might end up with a suggestion that changes how we currently use virtualenv, so right now I'm not attempting to just swap out virtualenv for python3 -m venv.

[gsnedders]

@gsnedders I've experimented with that, invoking python -m venv from the main process works, but I haven't worked out how to update the main process to use the resulting directory without activate_this.py. I think that updating sys.path is actually sufficient, and if it is then a hack would be to invoke a script in the venv which prints sys.path, and then update the main process to match.

Note that there may still be differences due to sys.modules already being populated (and sys.exec_prefix etc. being wrong).

What I was suggesting was that the main process wouldn't run anything within the venv, but instead we'd run the majority of the application within a subprocess, which therefore eliminates the need for anything like virtualenv's activate_this.py.

[foolip]

@gsnedders Yeah, that's an idea worth exploring. When using pipenv run python wpt run (in my experiment) that's pretty much what ends up happening. I'm not sure what the added startup time from the child process would be, and also how much memory the parent process would end up sitting on, but probably it'd be OK.