poetry2nix turns Poetry projects into Nix derivations without the need to actually write Nix expressions. It does so by parsing pyproject.toml
and poetry.lock
and converting them to Nix derivations on the fly.
For more information, see the announcement post on the Tweag blog.
The creator & long-term maintainer of poetry2nix (@adisbladis) is no longer using Poetry or Poetry2nix. This means that poetry2nix is looking for maintainers.
Poetry has a number of upcoming large changes, which poetry2nix is unlikely to gain support for:
If you are not already using Poetry & Poetry2nix, do consider uv
& uv2nix
.
You can turn your Python application into a Nix package with a few lines
by adding a default.nix
next to your pyproject.toml
and poetry.lock
files:
# file: default.nix
let
sources = import ./nix/sources.nix;
pkgs = import sources.nixpkgs { };
# Let all API attributes like "poetry2nix.mkPoetryApplication"
# use the packages and versions (python3, poetry etc.) from our pinned nixpkgs above
# under the hood:
poetry2nix = import sources.poetry2nix { inherit pkgs; };
myPythonApp = poetry2nix.mkPoetryApplication { projectDir = ./.; };
in
myPythonApp
The Nix code being executed by import sources.poetry2nix { inherit pkgs; }
is ./default.nix.
The resulting poetry2nix
attribute set contains (only) the API attributes like
mkPoetryApplication
.
Hint:
This example assumes that nixpkgs
and poetry2nix
are managed and pinned by
the handy niv tool. In your terminal just run:
nix-shell -p niv
niv init
niv add nix-community/poetry2nix
You can then build your Python application with Nix by running:
nix-build default.nix
Finally, you can run your Python application from the new ./result
symlinked folder:
# replace <script> with the name in the [tool.poetry.scripts] section of your pyproject.toml
./result/bin/<script>
If your project uses the experimental flake.nix
schema, you don't need niv.
This repository provides poetry2nix as a flake as well for you to import
as a flake input. For example:
# file: flake.nix
{
description = "Python application packaged using poetry2nix";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.poetry2nix.url = "github:nix-community/poetry2nix";
outputs = { self, nixpkgs, poetry2nix }:
let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
# create a custom "mkPoetryApplication" API function that under the hood uses
# the packages and versions (python3, poetry etc.) from our pinned nixpkgs above:
inherit (poetry2nix.lib.mkPoetry2Nix { inherit pkgs; }) mkPoetryApplication;
myPythonApp = mkPoetryApplication { projectDir = ./.; };
in
{
apps.${system}.default = {
type = "app";
# replace <script> with the name in the [tool.poetry.scripts] section of your pyproject.toml
program = "${myPythonApp}/bin/<script>";
};
};
}
You can then (build and) run your Python app with
nix run .
A larger real-world setup can be found in ./templates/app/flake.nix. This example is also exported as a flake template so that you can start your poetry2nix project conveniently through:
nix flake init --template github:nix-community/poetry2nix
Additionally, this project flake provides an overlay
to merge poetry2nix
into your pkgs
and access it as pkgs.poetry2nix
.
Just replace the three lines pkgs = ...
, inherit ...
and myPythonApp = ...
above with:
pkgs = nixpkgs.legacyPackages.${system}.extend poetry2nix.overlays.default;
myPythonApp = pkgs.poetry2nix.mkPoetryApplication { projectDir = self; };
The poetry2nix public API consists of the following attributes:
- mkPoetryApplication: Creates a Python application.
- mkPoetryEnv: Creates a Python environment with an interpreter and all packages from
poetry.lock
. - mkPoetryPackages: Creates an attribute set providing access to the generated packages and other artifacts.
- mkPoetryScriptsPackage: Creates a package containing the scripts from
tool.poetry.scripts
of thepyproject.toml
. - mkPoetryEditablePackage: Creates a package containing editable sources. Changes in the specified paths will be reflected in an interactive nix-shell session without the need to restart it.
- defaultPoetryOverrides: A set of bundled overrides fixing problems with Python packages.
- overrides.withDefaults: A convenience function for specifying overrides on top of the defaults.
- overrides.withoutDefaults: A convenience function for specifying overrides without defaults.
- cleanPythonSources: A function to create a source filter for python projects.
Creates a Python application using the Python interpreter specified based on the designated poetry project and lock files. mkPoetryApplication
takes an attribute set with the following attributes (attributes without default are mandatory):
- projectDir: path to the root of the project.
- src: project source (default:
cleanPythonSources { src = projectDir; }
). - pyproject: path to
pyproject.toml
(default:projectDir + "/pyproject.toml"
). - poetrylock:
poetry.lock
file path (default:projectDir + "/poetry.lock"
). - overrides: Python overrides to apply (default:
defaultPoetryOverrides
). - meta: application meta data (default:
{}
). - python: The Python interpreter to use (default:
pkgs.python3
). - preferWheels : Use wheels rather than sdist as much as possible (default:
false
). - groups: Which Poetry 1.2.0+ dependency groups to install (default:
[ "main" ]
). - checkGroups: Which Poetry 1.2.0+ dependency groups to install (independently of groups) to run unit tests (default:
[ "dev" ]
). - extras: Which Poetry
extras
to install (default:[ "*" ]
, all extras).
Other attributes are passed through to buildPythonApplication
.
Make sure to add in your pyproject.toml
the py-object for your main()
. Otherwise, the result is empty.
[tool.poetry.scripts]
poetry = "poetry.console.application:main"
poetry2nix.mkPoetryApplication {
projectDir = ./.;
}
The resulting derivation also has the passthru attribute dependencyEnv
, which is an environment with a python interpreter, all non-development dependencies and your application.
This can be used if your application doesn't provide any binaries on its own and instead relies on dependencies binaries to call its modules (as is often the case with celery
or gunicorn
).
For example, if your application defines a CLI for the module admin
and a gunicorn app for the module web
, a working default.nix
would contain
let
app = poetry2nix.mkPoetryApplication {
projectDir = ./.;
};
in app.dependencyEnv
After building this expression, your CLI and app can be called with these commands
./result/bin/python -m admin
./result/bin/gunicorn web:app
If you prefer to build a single binary that runs gunicorn web:app
, use pkgs.writeShellApplication
for a simple wrapper.
Note: If you need to perform overrides on the application, use app.dependencyEnv.override { app = app.override { ... }; }
. See ./tests/dependency-environment/default.nix for a full example.
Creates an environment that provides a Python interpreter along with all dependencies declared by the designated poetry project and lock files. Also allows package sources of an application to be installed in editable mode for fast development. mkPoetryEnv
takes an attribute set with the following attributes (attributes without default are mandatory):
- projectDir: path to the root of the project.
- pyproject: path to
pyproject.toml
(default:projectDir + "/pyproject.toml"
). - poetrylock:
poetry.lock
file path (default:projectDir + "/poetry.lock"
). - overrides: Python overrides to apply (default:
defaultPoetryOverrides
). - python: The Python interpreter to use (default:
pkgs.python3
). - editablePackageSources: A mapping from package name to source directory, these will be installed in editable mode. Note that path dependencies with
develop = true
will be installed in editable mode unless explicitly passed toeditablePackageSources
asnull
. (default:{}
). - extraPackages: A function taking a Python package set and returning a list of extra packages to include in the environment. This is intended for packages deliberately not added to
pyproject.toml
that you still want to include. An example of such a package may bepip
. (default:(ps: [ ])
). - preferWheels : Use wheels rather than sdist as much as possible (default:
false
). - groups: Which Poetry 1.2.0+ dependency groups to install (default:
[ "main" "dev" ]
). - checkGroups: Which Poetry 1.2.0+ dependency groups to install (independently of groups) to run unit tests (default:
[ "dev" ]
). - extras: Which Poetry
extras
to install (default:[ "*" ]
, all extras).
poetry2nix.mkPoetryEnv {
projectDir = ./.;
}
See ./tests/env/default.nix for a working example.
poetry2nix.mkPoetryEnv {
projectDir = ./.;
editablePackageSources = {
my-app = ./src;
};
}
See ./tests/editable/default.nix for a working example of an editable package.
The env
attribute of the attribute set created by mkPoetryEnv
contains a shell environment.
{ pkgs ? import <nixpkgs> {} }:
let
myAppEnv = pkgs.poetry2nix.mkPoetryEnv {
projectDir = ./.;
editablePackageSources = {
my-app = ./src;
};
};
in myAppEnv.env
For a shell environment including external dependencies, override the env
to add dependency packages (for example, pkgs.hello
) as build inputs.
{ pkgs ? import <nixpkgs> {} }:
let
myAppEnv = pkgs.poetry2nix.mkPoetryEnv {
projectDir = ./.;
editablePackageSources = {
my-app = ./src;
};
};
in myAppEnv.env.overrideAttrs (oldAttrs: {
buildInputs = [ pkgs.hello ];
})
Creates an attribute set of the shape { python, poetryPackages, pyProject, poetryLock }
. Where python
is the interpreter specified, poetryPackages
is a list of all generated python packages, pyProject
is the parsed pyproject.toml
and poetryLock
is the parsed poetry.lock
file. mkPoetryPackages
takes an attribute set with the following attributes (attributes without default are mandatory):
- projectDir: path to the root of the project.
- pyproject: path to
pyproject.toml
(default:projectDir + "/pyproject.toml"
). - poetrylock:
poetry.lock
file path (default:projectDir + "/poetry.lock"
). - overrides: Python overrides to apply (default:
defaultPoetryOverrides
). - python: The Python interpreter to use (default:
pkgs.python3
). - editablePackageSources: A mapping from package name to source directory, these will be installed in editable mode (default:
{}
). - preferWheels : Use wheels rather than sdist as much as possible (default:
false
). - groups: Which Poetry 1.2.0+ dependency groups to install (default:
[ ]
). - checkGroups: Which Poetry 1.2.0+ dependency groups to install (independently of groups) to run unit tests (default:
[ "dev" ]
). - extras: Which Poetry
extras
to install (default:[ "*" ]
, all extras).
poetry2nix.mkPoetryPackages {
projectDir = ./.;
python3 = python39;
}
Creates a package containing the scripts from tool.poetry.scripts
of the pyproject.toml
:
- projectDir: path to the root of the project.
- pyproject: path to
pyproject.toml
(default:projectDir + "/pyproject.toml"
). - python: The Python interpreter to use (default:
pkgs.python3
).
poetry2nix.mkPoetryScriptsPackage {
projectDir = ./.;
python3 = python39;
}
Creates a package containing editable sources. Changes in the specified paths will be reflected in an interactive nix-shell session without the need to restart it:
- projectDir: path to the root of the project.
- pyproject: path to
pyproject.toml
(default:projectDir + "/pyproject.toml"
). - python: The Python interpreter to use (default:
pkgs.python3
). - editablePackageSources: A mapping from package name to source directory, these will be installed in editable mode (default:
{}
).
poetry2nix.mkPoetryEditablePackage {
projectDir = ./.;
python3 = python39;
editablePackageSources = {
my-app = ./src;
};
}
poetry2nix bundles a set of default overrides that fix problems with various Python packages. These overrides are implemented in overrides.
Returns a list containing the specified overlay and defaultPoetryOverrides
.
Takes an attribute set with the following attributes (attributes without default are mandatory):
- src: project source directory
poetry2nix.mkPoetryEnv {
projectDir = ./.;
overrides = poetry2nix.overrides.withDefaults (final: prev: { foo = null; });
}
See ./tests/override-support/default.nix for a working example.
Returns a list containing just the specified overlay, ignoring defaultPoetryOverrides
.
poetry2nix.mkPoetryEnv {
projectDir = ./.;
overrides = poetry2nix.overrides.withoutDefaults (final: prev: { foo = null; });
}
Provides a source filtering mechanism that:
- Filters gitignore's
- Filters pycache/pyc files
- Uses cleanSourceFilter to filter out .git/.hg, .o/.so, editor backup files & nix result symlinks
poetry2nix.cleanPythonSources {
src = ./.;
}
Sometimes when it can be convenient to create a custom instance of poetry2nix
with a different set of default overrides.
let
# final & prev refers to poetry2nix
p2nix = poetry2nix.overrideScope (final: prev: {
# pyself & pyprev refers to python packages
defaultPoetryOverrides = prev.defaultPoetryOverrides.extend (pyfinal: pyprev: {
my-custom-pkg = pyprev.my-custom-pkg.overridePythonAttrs (oldAttrs: { });
});
});
in
p2nix.mkPoetryApplication {
projectDir = ./.;
}
or as a nixpkgs overlay:
let
pkgs = import <nixpkgs> {
overlays = [
# final & prev refers to nixpkgs
(final: prev: {
# p2nixfinal & p2nixprev refers to poetry2nix
poetry2nix = prev.poetry2nix.overrideScope (p2nixfinal: p2nixprev: {
# pyfinal & pyprev refers to python packages
defaultPoetryOverrides = p2nixprev.defaultPoetryOverrides.extend (pyfinal: pyprev: {
my-custom-pkg = pyprev.my-custom-pkg.overridePythonAttrs (oldAttrs: { });
});
});
})
];
};
in pkgs.poetry2nix.mkPoetryApplication {
projectDir = ./.;
}
Poetry by default downloads Python packages (wheels, sources, etc.) from PyPI
but supports to specify one or more alternative repositories in a
"package source" section
in the pyproject.toml
file:
[[tool.poetry.source]]
name = "private-repository"
url = "https://example.org/simple/"
priority = "primary"
...
Poetry then bakes the individual source repository urls for each Python package together with
a cryptographic hash of the package into its poetry.lock
file.
This is great for reproducibility as Poetry knows where to download packages from later
and can ensure that the packages haven't been modified.
Poetry2nix downloads the same packages from the same repository urls in the lock file and reuses the hashes. However, many private Python repositories require authentication with credentials like username and password token, especially in companies.
While Poetry supports several methods of authentication like through
a NETRC file
environment variables
a custom crendentials file
and others,
poetry2nix only supports one: the NETRC
file method that secretly adds credentials to your
http calls to the repository url, e.g. https://example.org/simple/
.
For this to work, follow these three steps:
- Create or locate your
NETRC
file into your computer, usually in your home folder/home/user/<username>/.netrc
or/etc/nix/netrc
with credentials like:
machine https://example.org
login <repository-username>
password <repository-password-or-token>
- Mount the
NETRC
file into the Nix build sandbox with Nix extra-sandbox-paths setting; otherwise poetry2nix is not able to access that file from within the Nix sandbox. You can mount the file either through the global Nix/NixOS config, usually/etc/nix/nix.conf
:
# file: nix.conf
extra-sandbox-paths /etc/nix/netrc`
This is not recommended as you expose your secrets to all Nix builds.
Better just mount it for single, specific poetry2nix builds directly in the terminal:
# non-flake project
nix-build --option extra-sandbox-paths /etc/nix/netrc default.nix
# flake project
nix build . --extra-sandbox-paths /etc/nix/netrc
Note that the username you're executing this command with must be a
"trusted-user"
in the global Nix/NixOS config, usually /etc/nix/nix.conf
:
# file: nix.conf
trusted-users <username>
If you are not a trusted user, this extra setting will be silently ignored and package downloads will fail.
- Tell poetry2nix where to find the
NETRC
file inside the Nix sandbox. For that you have to pass an environment variable calledNETRC
into the sandbox containing the path to the file. Depending on whether you use flakes or not you have the following options:
For flakes the only option is to add the environment variable to the "nix-daemon", the process that actually creates sandboxes and performs builds on your behalf.
On NixOS you can add the env variable to the nix-daemon through its systemd configuration:
systemd.services.nix-daemon = {
serviceConfig = {
Environment = "NETRC=/etc/nix/netrc";
};
};
This environment variable will automatically be passed to all your builds so you can keep using the build commands as before;
# non-flake project
nix-build --option extra-sandbox-paths /etc/nix/netrc default.nix
# flake project
nix build . --extra-sandbox-paths /etc/nix/netrc
For a non-flake project you can alternatively pass the NETRC
path value through
a fake Nix search path -I NETRC=<netrc-path>
argument in the terminal; such a search path doesn't work with flakes.
poetry2nix contains special code to forward this variable as an environment variable into any Python sandbox.
# non-flake project
nix-build -I NETRC=/etc/nix/netrc --option extra-sandbox-paths /etc/nix/netrc default.nix
Note: The alternative to pass the NETRC
path environment variable
into the sandbox via the (impureEnvVars setting](https://nixos.org/manual/nix/stable/language/advanced-attributes.html##adv-attr-impureEnvVars)
doesn't work.
Q. Does poetry2nix install wheels or sdists?
A. By default, poetry2nix installs from source. If you want to give precedence to wheels, look at the preferWheel
and preferWheels
attributes.
Q. Does poetry2nix use package definitions from nixpkgs' Python package set?
A. poetry2nix overlays packages taken from the poetry.lock
file on top of nixpkgs, in such a way that overlaid packages in nixpkgs are completely ignored.
Any package that is used, but isn't in the poetry.lock
file (most commonly build dependencies) is taken from nixpkgs.
Q. How to prefer wheel installation for a single package?
A. Override it and set preferWheel = true
in that single package:
poetry2nix.mkPoetryApplication {
projectDir = ./.;
overrides = poetry2nix.overrides.withDefaults (final: prev: {
# Notice that using .overridePythonAttrs or .overrideAttrs won't work!
some-dependency = prev.some-dependency.override {
preferWheel = true;
};
});
}
Q. I'm experiencing one of the following errors, what do I do?
- ModuleNotFoundError: No module named 'setuptools'
- ModuleNotFoundError: No module named 'pdm'
- ModuleNotFoundError: No module named 'setuptools-scm'
- ModuleNotFoundError: No module named 'poetry-core'
- ModuleNotFoundError: No module named 'flit'
- ModuleNotFoundError: No module named 'flit-core'
- ModuleNotFoundError: No module named 'flit-scm'
A. Have a look at the following document edgecase.md
- Package and deploy Python apps faster with Poetry and Nix This is a short (11 minutes) video tutorial by Steve Purcell from Tweag walking you through how to get started with a small web development project.
Contributions to this project are welcome in the form of GitHub PRs. Please consider the following before creating PRs:
- You can use
nix fmt
to format everything and sortoverrides/build-systems.json
. - If you are planning to make any considerable changes, you should first present your plans in a GitHub issue so it can be discussed.
- If you add new features please consider adding tests. You can run them locally as follows:
nix-build --keep-going --show-trace tests/default.nix
To list test names:
nix eval --impure --expr 'let pkgs = import <nixpkgs> {}; in pkgs.lib.attrNames (import ./tests/default.nix {})'
To run specific tests, add --attr NAME
to the nix-build
command above. For example, to run the bcrypt
and jq
tests:
nix-build --attr bcrypt --attr jq --keep-going --show-trace tests/default.nix
To test with a specific channel:
nix-build --expr 'with import <unstable> {}; callPackage ./tests/default.nix {}'
We have a Matrix room at #poetry2nix:blad.is.
Development of poetry2nix
has been supported by Tweag.
poetry2nix is released under the terms of the MIT license.