diff --git a/docs/building-extension-modules.md b/docs/building-extension-modules.md new file mode 100644 index 00000000000..e1e54b1ab8a --- /dev/null +++ b/docs/building-extension-modules.md @@ -0,0 +1,377 @@ +--- +title: "Building extension modules" +draft: false +type: docs +layout: single + +menu: + docs: + weight: 125 +--- + +# Building Extension Modules + +{{% warning %}} +While this feature has been around since almost the beginning of the Poetry project and has needed minimal changes, +it is still considered unstable. You can participate in the discussions about stabilizing this feature +[here](https://github.com/python-poetry/poetry/issues/2740). + +And as always, your contributions towards the goal of improving this feature are also welcome. +{{% /warning %}} + +Poetry allows a project developer to introduce support for, build and distribute native extensions within their project. +In order to achieve this, at the highest level, the following steps are required. + +{{< steps >}} +{{< step >}} +**Add Build Dependencies** + +The build dependencies, in this context, refer to those Python packages that are required in order to successfully execute +your build script. Common examples include `cython`, `meson`, `maturin`, `setuptools` etc., depending on how your +extension is built. + +{{% note %}} +You must assume that only Python built-ins are available by default in a build environment. This means, if you need +even packages like `setuptools`, it must be explicitly declared. +{{% /note %}} + +The necessary build dependencies must be added to the `build-system.requires` section of your `pyproject.toml` file. + +```toml +[build-system] +requires = ["poetry-core", "setuptools", "cython"] +build-backend = "poetry.core.masonry.api" +``` + +{{% note %}} +It is recommended that you consider specifying version constraints to all entries in `build-system.requires` in order to +avoid surprises if one of the packages introduce a breaking change. For example, you can set `cython` to +`cython>=3.0.11,<4.0.0` to ensure no major version upgrades are used. +{{% /note %}} + +{{% note %}} +If you wish to develop the build script within your project's virtual environment, then you must also add the +dependencies to your project explicitly to a dependency group - the name of which is not important. + +```sh +poetry add --group=build setuptools cython +``` +{{% /note %}} + +{{< /step >}} + +{{< step >}} +**Add Build Script** + +The build script can be a free-form Python script that uses any dependency specified in the previous step. This can be +named as needed, but **must** be located within the project root directory (or a subdirectory) and also **must** +be included in your source distribution. You can see the [example snippets section]({{< relref "#example-snippets" >}}) +for inspiration. + +{{% note %}} +The build script is always executed from the project root. And it is expected to move files around to their destinations +as expected by Poetry as per your `pyproject.toml` file. +{{% /note %}} + +```toml +[tool.poetry.build] +script = "relative/path/to/build-extension.py" +``` + +{{% note %}} +The name of the build script is arbitrary. Common practice has been to name it `build.py`, however this is not +mandatory. You **should** consider [using a subdirectory]({{< relref "#can-i-store-the-build-script-in-a-subdirectory" >}}) +if feasible. +{{% /note %}} + +{{< /step >}} + +{{< step >}} +**Specify Distribution Files** + +{{% warning %}} +The following is an example, and should not be considered as complete. +{{% /warning %}} + +```toml +[tool.poetry] +... +include = [ + { path = "package/**/*.so", format = "wheel" }, + # sources must be present in sdist, can be ignored if you only have *.pyx sources + { path = "package/**/*.c", format = "sdist" }, +] +``` + +The key takeaway here should be the following. You can refer to the [`pyproject.toml`]({{< relref "pyproject#exclude-and-include" >}}) +documentation for information on each of the relevant sections. + +1. Include your build outputs in your wheel. +2. Exclude your build inputs from your wheel. +3. Include your build inputs to your source distribution. + +{{< /step >}} + +{{< /steps >}} + +## Example Snippets + +### Cython + +{{< tabs tabTotal="3" tabID1="cython-pyproject" tabName1="pyproject.toml" tabID2="cython-build-script" tabName2="build-extension.py" tabID3="cython-src-tree" tabName3="Source Tree">}} + +{{< tab tabID="cython-pyproject" >}} + +```toml +[build-system] +requires = ["poetry-core", "cython", "setuptools"] +build-backend = "poetry.core.masonry.api" + +[tool.poetry] +... +packages = [ + { include = "package", from = "src"}, +] +include = [ + { path = "src/package/**/*.so", format = "wheel" }, +] +# if not already excluded via .gitignore +exclude = [ + "**/*.c" +] + +[tool.poetry.build] +script = "scripts/build-extension.py" +``` + +{{< /tab >}} + +{{< tab tabID="cython-build-script" >}} + +```py +from __future__ import annotations + +import os +import shutil + +from pathlib import Path + +from Cython.Build import cythonize +from setuptools import Distribution +from setuptools import Extension +from setuptools.command.build_ext import build_ext + + +COMPILE_ARGS = ["-march=native", "-O3", "-msse", "-msse2", "-mfma", "-mfpmath=sse"] +LINK_ARGS = [] +INCLUDE_DIRS = [] +LIBRARIES = ["m"] + + +def build() -> None: + extensions = [ + Extension( + "*", + ["src/package/*.pyx"], + extra_compile_args=COMPILE_ARGS, + extra_link_args=LINK_ARGS, + include_dirs=INCLUDE_DIRS, + libraries=LIBRARIES, + ) + ] + ext_modules = cythonize( + extensions, + include_path=INCLUDE_DIRS, + compiler_directives={"binding": True, "language_level": 3}, + ) + + distribution = Distribution({ + "name": "package", + "ext_modules": ext_modules + }) + + cmd = build_ext(distribution) + cmd.ensure_finalized() + cmd.run() + + # Copy built extensions back to the project + for output in cmd.get_outputs(): + output = Path(output) + relative_extension = Path("src") / output.relative_to(cmd.build_lib) + + shutil.copyfile(output, relative_extension) + mode = os.stat(relative_extension).st_mode + mode |= (mode & 0o444) >> 2 + os.chmod(relative_extension, mode) + + +if __name__ == "__main__": + build() +``` + +{{< /tab >}} + +{{< tab tabID="cython-src-tree" >}} + +``` +scripts/ +└── build-extension.py +src/ +└── package + ├── example.pyx + └── __init__.py +``` + +{{< /tab >}} + +{{< /tabs >}} + +### Meson + +{{< tabs tabTotal="2" tabID1="meson-pyproject" tabName1="pyproject.toml" tabID2="meson-build-script" tabName2="build-extension.py">}} + +{{< tab tabID="meson-pyproject" >}} + +```toml +[tool.poetry.build] +script = "build-extension.py" + +[build-system] +requires = ["poetry-core", "meson"] +build-backend = "poetry.core.masonry.api" +``` + +{{< /tab >}} + +{{< tab tabID="meson-build-script" >}} + +```py +from __future__ import annotations + +import subprocess + +from pathlib import Path + + +def meson(*args): + subprocess.call(["meson", *args]) + + +def build(): + build_dir = Path(__file__).parent.joinpath("build") + build_dir.mkdir(parents=True, exist_ok=True) + + meson("setup", build_dir.as_posix()) + meson("compile", "-C", build_dir.as_posix()) + meson("install", "-C", build_dir.as_posix()) + + +if __name__ == "__main__": + build() +``` + +{{< /tab >}} + +{{< /tabs >}} + +### Maturin + +{{< tabs tabTotal="2" tabID1="maturin-pyproject" tabName1="pyproject.toml" tabID2="maturin-build-script" tabName2="build-extension.py">}} + +{{< tab tabID="maturin-pyproject" >}} + +```toml +[tool.poetry.build] +script = "build-extension.py" + +[build-system] +requires = ["poetry-core", "maturin"] +build-backend = "poetry.core.masonry.api" +``` + +{{< /tab >}} + +{{< tab tabID="maturin-build-script" >}} + +```py +import os +import shlex +import shutil +import subprocess +import zipfile + +from pathlib import Path + + +def maturin(*args): + subprocess.call(["maturin", *list(args)]) + + +def build(): + build_dir = Path(__file__).parent.joinpath("build") + build_dir.mkdir(parents=True, exist_ok=True) + + wheels_dir = Path(__file__).parent.joinpath("target/wheels") + if wheels_dir.exists(): + shutil.rmtree(wheels_dir) + + cargo_args = [] + if os.getenv("MATURIN_BUILD_ARGS"): + cargo_args = shlex.split(os.getenv("MATURIN_BUILD_ARGS", "")) + + maturin("build", "-r", *cargo_args) + + # We won't use the wheel built by maturin directly since + # we want Poetry to build it but, we need to retrieve the + # compiled extensions from the maturin wheel. + wheel = next(iter(wheels_dir.glob("*.whl"))) + with zipfile.ZipFile(wheel.as_posix()) as whl: + whl.extractall(wheels_dir.as_posix()) + + for extension in wheels_dir.rglob("**/*.so"): + shutil.copyfile(extension, Path(__file__).parent.joinpath(extension.name)) + + shutil.rmtree(wheels_dir) + + +if __name__ == "__main__": + build() +``` + +{{< /tab >}} + +{{< /tabs >}} + +## FAQ +### When is my build script executed? +If your project uses a build script, it is run implicitly in the following scenarios. + +1. When `poetry install` is run, it is executed prior to installing the project's root package. +2. When `poetry build` is run, it is executed prior to building distributions. +3. When a PEP 517 build is triggered from source or sdist by another build frontend. + +### How does Poetry ensure my build script's dependencies are met? +Prior to executing the build script, Poetry creates a temporary virtual environment with your project's active Python +version and then installs all dependencies specified under `build-system.requires` into this environment. It should be +noted that no packages will be present in this environment at the time of creation. + +### Can I store the build script in a subdirectory? +Yes you can. If storing the script in a subdirectory, your `pyproject.toml` might look something like this. + +```toml +[tool.poetry] +... +packages = [ + { include = "package", from = "src"} +] +include = [ + { path = "src/package/**/*.so", format = "wheel" }, +] +# exclude any intermediate source files +exclude = [ + "**/*.c" +] + +[tool.poetry.build] +script = "scripts/build-extension.py" +```