doc: add documentation for using extension modules (#9864)

docs/building-extension-modules.md

@@ -0,0 +1,377 @@
+---
+title: "Building extension modules"
+draft: false
+type: docs
+layout: single
+
+menu:
+  docs:
+    weight: 125
+---
+
+# Building Extension Modules
+
+{{% warning %}}
+While this feature has been around since almost the beginning of the Poetry project and has needed minimal changes,
+it is still considered unstable. You can participate in the discussions about stabilizing this feature
+[here](https://github.com/python-poetry/poetry/issues/2740).
+
+And as always, your contributions towards the goal of improving this feature are also welcome.
+{{% /warning %}}
+
+Poetry allows a project developer to introduce support for, build and distribute native extensions within their project.
+In order to achieve this, at the highest level, the following steps are required.
+
+{{< steps >}}
+{{< step >}}
+**Add Build Dependencies**
+
+The build dependencies, in this context, refer to those Python packages that are required in order to successfully execute
+your build script. Common examples include `cython`, `meson`, `maturin`, `setuptools` etc., depending on how your
+extension is built.
+
+{{% note %}}
+You must assume that only Python built-ins are available by default in a build environment. This means, if you need
+even packages like `setuptools`, it must be explicitly declared.
+{{% /note %}}
+
+The necessary build dependencies must be added to the `build-system.requires` section of your `pyproject.toml` file.
+
+```toml
+[build-system]
+requires = ["poetry-core", "setuptools", "cython"]
+build-backend = "poetry.core.masonry.api"
+```
+
+{{% note %}}
+It is recommended that you consider specifying version constraints to all entries in `build-system.requires` in order to
+avoid surprises if one of the packages introduce a breaking change. For example, you can set `cython` to
+`cython>=3.0.11,<4.0.0` to ensure no major version upgrades are used.
+{{% /note %}}
+
+{{% note %}}
+If you wish to develop the build script within your project's virtual environment, then you must also add the
+dependencies to your project explicitly to a dependency group - the name of which is not important.
+
+```sh
+poetry add --group=build setuptools cython
+```
+{{% /note %}}
+
+{{< /step >}}
+
+{{< step >}}
+**Add Build Script**
+
+The build script can be a free-form Python script that uses any dependency specified in the previous step. This can be
+named as needed, but **must** be located within the project root directory (or a subdirectory) and also **must**
+be included in your source distribution. You can see the [example snippets section]({{< relref "#example-snippets" >}})
+for inspiration.
+
+{{% note %}}
+The build script is always executed from the project root. And it is expected to move files around to their destinations
+as expected by Poetry as per your `pyproject.toml` file.
+{{% /note %}}
+
+```toml
+[tool.poetry.build]
+script = "relative/path/to/build-extension.py"
+```
+
+{{% note %}}
+The name of the build script is arbitrary. Common practice has been to name it `build.py`, however this is not
+mandatory. You **should** consider [using a subdirectory]({{< relref "#can-i-store-the-build-script-in-a-subdirectory" >}})
+if feasible.
+{{% /note %}}
+
+{{< /step >}}
+
+{{< step >}}
+**Specify Distribution Files**
+
+{{% warning %}}
+The following is an example, and should not be considered as complete.
+{{% /warning %}}
+
+```toml
+[tool.poetry]
+...
+include = [
+    { path = "package/**/*.so", format = "wheel" },
+    # sources must be present in sdist, can be ignored if you only have *.pyx sources
+    { path = "package/**/*.c", format = "sdist" },
+]
+```
+
+The key takeaway here should be the following. You can refer to the [`pyproject.toml`]({{< relref "pyproject#exclude-and-include" >}})
+documentation for information on each of the relevant sections.
+
+1. Include your build outputs in your wheel.
+2. Exclude your build inputs from your wheel.
+3. Include your build inputs to your source distribution.
+
+{{< /step >}}
+
+{{< /steps >}}
+
+## Example Snippets
+
+### Cython
+
+{{< tabs tabTotal="3" tabID1="cython-pyproject" tabName1="pyproject.toml" tabID2="cython-build-script" tabName2="build-extension.py" tabID3="cython-src-tree" tabName3="Source Tree">}}
+
+{{< tab tabID="cython-pyproject" >}}
+
+```toml
+[build-system]
+requires = ["poetry-core", "cython", "setuptools"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.poetry]
+...
+packages = [
+    { include = "package", from = "src"},
+]
+include = [
+    { path = "src/package/**/*.so", format = "wheel" },
+]
+# if not already excluded via .gitignore
+exclude = [
+    "**/*.c"
+]
+
+[tool.poetry.build]
+script = "scripts/build-extension.py"
+```
+
+{{< /tab >}}
+
+{{< tab tabID="cython-build-script" >}}
+
+```py
+from __future__ import annotations
+
+import os
+import shutil
+
+from pathlib import Path
+
+from Cython.Build import cythonize
+from setuptools import Distribution
+from setuptools import Extension
+from setuptools.command.build_ext import build_ext
+
+
+COMPILE_ARGS = ["-march=native", "-O3", "-msse", "-msse2", "-mfma", "-mfpmath=sse"]
+LINK_ARGS = []
+INCLUDE_DIRS = []
+LIBRARIES = ["m"]
+
+
+def build() -> None:
+    extensions = [
+        Extension(
+            "*",
+            ["src/package/*.pyx"],
+            extra_compile_args=COMPILE_ARGS,
+            extra_link_args=LINK_ARGS,
+            include_dirs=INCLUDE_DIRS,
+            libraries=LIBRARIES,
+        )
+    ]
+    ext_modules = cythonize(
+        extensions,
+        include_path=INCLUDE_DIRS,
+        compiler_directives={"binding": True, "language_level": 3},
+    )
+
+    distribution = Distribution({
+        "name": "package",
+        "ext_modules": ext_modules
+    })
+
+    cmd = build_ext(distribution)
+    cmd.ensure_finalized()
+    cmd.run()
+
+    # Copy built extensions back to the project
+    for output in cmd.get_outputs():
+        output = Path(output)
+        relative_extension = Path("src") / output.relative_to(cmd.build_lib)
+
+        shutil.copyfile(output, relative_extension)
+        mode = os.stat(relative_extension).st_mode
+        mode |= (mode & 0o444) >> 2
+        os.chmod(relative_extension, mode)
+
+
+if __name__ == "__main__":
+    build()
+```
+
+{{< /tab >}}
+
+{{< tab tabID="cython-src-tree" >}}
+
+```
+scripts/
+└── build-extension.py
+src/
+└── package
+    ├── example.pyx
+    └── __init__.py
+```
+
+{{< /tab >}}
+
+{{< /tabs >}}
+
+### Meson
+
+{{< tabs tabTotal="2" tabID1="meson-pyproject" tabName1="pyproject.toml" tabID2="meson-build-script" tabName2="build-extension.py">}}
+
+{{< tab tabID="meson-pyproject" >}}
+
+```toml
+[tool.poetry.build]
+script = "build-extension.py"
+
+[build-system]
+requires = ["poetry-core", "meson"]
+build-backend = "poetry.core.masonry.api"
+```
+
+{{< /tab >}}
+
+{{< tab tabID="meson-build-script" >}}
+
+```py
+from __future__ import annotations
+
+import subprocess
+
+from pathlib import Path
+
+
+def meson(*args):
+    subprocess.call(["meson", *args])
+
+
+def build():
+    build_dir = Path(__file__).parent.joinpath("build")
+    build_dir.mkdir(parents=True, exist_ok=True)
+
+    meson("setup", build_dir.as_posix())
+    meson("compile", "-C", build_dir.as_posix())
+    meson("install", "-C", build_dir.as_posix())
+
+
+if __name__ == "__main__":
+    build()
+```
+
+{{< /tab >}}
+
+{{< /tabs >}}
+
+### Maturin
+
+{{< tabs tabTotal="2" tabID1="maturin-pyproject" tabName1="pyproject.toml" tabID2="maturin-build-script" tabName2="build-extension.py">}}
+
+{{< tab tabID="maturin-pyproject" >}}
+
+```toml
+[tool.poetry.build]
+script = "build-extension.py"
+
+[build-system]
+requires = ["poetry-core", "maturin"]
+build-backend = "poetry.core.masonry.api"
+```
+
+{{< /tab >}}
+
+{{< tab tabID="maturin-build-script" >}}
+
+```py
+import os
+import shlex
+import shutil
+import subprocess
+import zipfile
+
+from pathlib import Path
+
+
+def maturin(*args):
+    subprocess.call(["maturin", *list(args)])
+
+
+def build():
+    build_dir = Path(__file__).parent.joinpath("build")
+    build_dir.mkdir(parents=True, exist_ok=True)
+
+    wheels_dir = Path(__file__).parent.joinpath("target/wheels")
+    if wheels_dir.exists():
+        shutil.rmtree(wheels_dir)
+
+    cargo_args = []
+    if os.getenv("MATURIN_BUILD_ARGS"):
+        cargo_args = shlex.split(os.getenv("MATURIN_BUILD_ARGS", ""))
+
+    maturin("build", "-r", *cargo_args)
+
+    # We won't use the wheel built by maturin directly since
+    # we want Poetry to build it but, we need to retrieve the
+    # compiled extensions from the maturin wheel.
+    wheel = next(iter(wheels_dir.glob("*.whl")))
+    with zipfile.ZipFile(wheel.as_posix()) as whl:
+        whl.extractall(wheels_dir.as_posix())
+
+        for extension in wheels_dir.rglob("**/*.so"):
+            shutil.copyfile(extension, Path(__file__).parent.joinpath(extension.name))
+
+    shutil.rmtree(wheels_dir)
+
+
+if __name__ == "__main__":
+    build()
+```
+
+{{< /tab >}}
+
+{{< /tabs >}}
+
+## FAQ
+### When is my build script executed?
+If your project uses a build script, it is run implicitly in the following scenarios.
+
+1. When `poetry install` is run, it is executed prior to installing the project's root package.
+2. When `poetry build` is run, it is executed prior to building distributions.
+3. When a PEP 517 build is triggered from source or sdist by another build frontend.
+
+### How does Poetry ensure my build script's dependencies are met?
+Prior to executing the build script, Poetry creates a temporary virtual environment with your project's active Python
+version and then installs all dependencies specified under `build-system.requires` into this environment. It should be
+noted that no packages will be present in this environment at the time of creation.
+
+### Can I store the build script in a subdirectory?
+Yes you can. If storing the script in a subdirectory, your `pyproject.toml` might look something like this.
+
+```toml
+[tool.poetry]
+...
+packages = [
+    { include = "package", from = "src"}
+]
+include = [
+    { path = "src/package/**/*.so", format = "wheel" },
+]
+# exclude any intermediate source files
+exclude = [
+    "**/*.c"
+]
+
+[tool.poetry.build]
+script = "scripts/build-extension.py"
+```