pdm installed scripts break in with venv + non-src layout

[ditsuke]

• I have searched the issue tracker and believe that this is not a duplicate.

Steps to reproduce

1. Project layout

.
├── co.py
├── nvim.local.log
├── pdm.lock
├── pyproject.toml
└── src
    └── a.py

2. pyproject.toml

[project]
name = "llama-cpp-scripts"
version = "0.0.0"
requires-python = ">=3.9,<3.12"
dependencies = ["numpy<2.0.0,>=1.25.0"]

[tool.pdm.build]
includes = ["*.py"]

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

[project.scripts]
co = "co:main"

3. Run script

❯ pdm install && pdm run co
INFO: The saved Python interpreter doesn't match the project's requirement. Trying to find another one.
WARNING: Project requires a python version of <3.12,>=3.9, The virtualenv is being created for you as it cannot be matched to the right 
version.
INFO: python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /home/dt/Documents/temp/fail-pdm/.venv
Synchronizing working set with resolved packages: 1 to add, 0 to update, 0 to remove

  ✔ Install numpy 1.26.4 successful
Installing the project as an editable package...
  ✔ Install llama-cpp-scripts 0.0.0 successful

🎉 All complete!

Traceback (most recent call last):
  File "/home/dt/Documents/temp/fail-pdm/.venv/bin/co", line 5, in <module>
    from co import main
ModuleNotFoundError: No module named 'co'

Actual behavior

Script run fails with a ModuleNotFoundError. Ref: ggerganov/llama.cpp#5745

Expected behavior

Script run succeeds. pdm shouldn't be thrown off by an src directory, it should instead compute sys.path through paths of file in include.

Environment Information

❯ pdm info && pdm info --env
PDM version:
  2.16.1
Python Interpreter:
  /home/dt/Documents/temp/fail-pdm/.venv/bin/python (3.11)
Project Root:
  /home/dt/Documents/temp/fail-pdm
Local Packages:
  
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.9",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "6.9.6-200.fc40.x86_64",
  "platform_system": "Linux",
  "platform_version": "#1 SMP PREEMPT_DYNAMIC Fri Jun 21 15:48:21 UTC 2024",
  "python_full_version": "3.11.9",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "linux"
}

Minimum reproducible project archive: minrepoduction.tar.gz

[pawamoy]

You might be confusing [project.scripts] and [tool.poetry.scripts] (or [tool.pdm.scripts]), they are not the same thing.

[project.scripts] co = "co:main" will create an executable in a venv's bin folder, which will try to import main from co and run it. Except there are no co module installed. This is reproducible with pip:

rm -rf .venv
python -m venv .venv
. .venv/bin/activate
pip install -e .
co

...gives the same error:

Traceback (most recent call last):
  File "/tmp/t/.venv/bin/co", line 5, in <module>
    from co import main
ModuleNotFoundError: No module named 'co'

You might want to use user scripts instead, or actually put your co.py module into your sources, so that it is installed and the co:main script can find it.

[ditsuke]

I believe for pdm, the proper analogue to tool.poetry.scripts is project.scripts as tool.pdm.scripts does not register as project entrypoints (which is a hard requirement here).

To demonstrate further, this behavior changes when the src directory is removed:

❯ cat --plain co.py 
import sys


def main():
    print(f"sys.path: {sys.path}")

❯ mv src src2 && pdm install && pdm run co
All packages are synced to date, nothing to do.
Installing the project as an editable package...
  ✔ Update llama-cpp-scripts 0.0.0 -> 0.0.0 successful

🎉 All complete!

sys.path: ['/home/dt/Documents/temp/fail-pdm/.venv/bin', '/home/dt/.local/share/mise/installs/python/3.11/lib/python311.zip', '/home/dt/.
local/share/mise/installs/python/3.11/lib/python3.11', '/home/dt/.local/share/mise/installs/python/3.11/lib/python3.11/lib-dynload', '/ho
me/dt/Documents/temp/fail-pdm/.venv/lib/python3.11/site-packages', '/home/dt/Documents/temp/fail-pdm']

And also when the script is in src:

❯ mv co.py src && pdm install && pdm run co
All packages are synced to date, nothing to do.
Installing the project as an editable package...
  ✔ Update llama-cpp-scripts 0.0.0 -> 0.0.0 successful

🎉 All complete!

sys.path: ['/home/dt/Documents/temp/fail-pdm/.venv/bin', '/home/dt/.local/share/mise/installs/python/3.11/lib/python311.zip', '/home/dt/.
local/share/mise/installs/python/3.11/lib/python3.11', '/home/dt/.local/share/mise/installs/python/3.11/lib/python3.11/lib-dynload', '/ho
me/dt/Documents/temp/fail-pdm/.venv/lib/python3.11/site-packages', '/home/dt/Documents/temp/fail-pdm/src']

Curiously, the sys.path changes with the presence of the src folder. I don't observe this behavior with poetry, where sys.path always includes project root. I'm unable to discern how pdm manages this.

---

Interestingly also reproducible with pip, so not which part is introducing the sys.path detection now

❯ mv src src2                             

❯ python -m venv .venv   

❯ source ./.venv/bin/activate

❯ pip install -e .

❯ co
sys.path: ['/home/dt/Documents/temp/fail-pdm/.venv/bin', '/home/dt/.local/share/mise/installs/python/3.11/lib/python311.zip', '/home/dt/.
local/share/mise/installs/python/3.11/lib/python3.11', '/home/dt/.local/share/mise/installs/python/3.11/lib/python3.11/lib-dynload', '/ho
me/dt/Documents/temp/fail-pdm/.venv/lib/python3.11/site-packages', '/home/dt/Documents/temp/fail-pdm']

[ditsuke]

or actually put your co.py module into your sources, so that it is installed and the co:main script can find it.

Could you clarify if this means putting it in include (already done) or some other key?

[pawamoy]

I believe for pdm, the proper analogue to tool.poetry.scripts is project.scripts as tool.pdm.scripts does not register as project entrypoints (which is a hard requirement here).

Sorry, yes, I was the one confused 🙂 So you really want [project.scripts], OK.

Interestingly also reproducible with pip, so not which part is introducing the sys.path detection now

Interesting indeed! Thanks for investigating further. Let me try as well.

[pawamoy]

With src, after pip install -e .:

% cat .venv/lib/python3.11/site-packages/llama_cpp_scripts.pth
/tmp/t/src

In that case the co.py module isn't found and calling your entrypoint fails.

---

With src2, after pip install -e .:

% cat .venv/lib/python3.11/site-packages/llama_cpp_scripts.pth
/tmp/t

Indeed if the src folder is found, it is used as source folder, and if it isn't found, then the repository root is used as source folder. In that case, the co.py module is found and calling your entrypoint works.

---

So the issue comes from the automatic detection of src by PDM backend (when installing the project in editable mode, or generally). But I wouldn't call this an issue. Detection of src is advertised and expected.

---

Could you clarify if this means putting it in include (already done) or some other key?

I meant to move co.py into your src folder (i.e. into your package, since you seem to want to bundle it anyway).

[frostming]

This is a bit tricky, the project is detected as src layout but src is not even included by the build.

[ditsuke]

Thank you for the fix!