mpia-python-template

This repository contains a Python template suitable for starting new projects. This template has three main goals:

• To create a properly packaged Python project which requires minimal set-up effort for the user. Following the seven steps below you will have the structure of a python package in ~15 minutes.
• To encourage proper testing. This template includes a GitHub actions workflow, .github/workflows/ci.yml, that will run continuous integration on the tests directory at every push or pull request.
• To encourage code and package documentation. This template includes pre-configured documentation in the docs directory which will automatically pull the information from the code docstrings. A GitHub actions workflow, .github/workflows/docs.yml, also publishes the documentation to a GitHub Pages webpage at every push or pull request.

This template does not assume GitHub tags, it is not set up for PyPi releases and it does not produce ReadTheDocs documentation, however such extensions can be easily added on to the template by motivated users. Please contact the template authors if you need help with these tasks.

There is a lot of (sometimes contradictory) information on how to package a Python project. Here we have generally followed the current recommendations of the Python Packaging Authority.

How to Use This Template

Click on the green "Use This Template" button in the upper right corner of the screen to make a copy of the template.

Select location for the new repository. For projects led by MPIA staff and/or with an MPIA advisor, we encourage you to create your copy within the mpi-astronomy organization but you can also create a copy under your own account.

Select a name for your repository, select if it should be Public (recommended) or Private. Here as an example we will use new_project as the name of your new repository.

Click on the Create Repository from Template button.

Customize the Template in 7 Easy Steps

1. We recommend creating a new environment for your project. This will allow you to track dependencies a lot easier. There are several different ways to do this (virtualenv, pipenv). Here we show an example with conda:

    conda create --name new_environment python
    conda activate new_environment
   

   Do not install dependencies that are necessary for your code with conda. All dependencies should be specified in the setup.cfg files (see step 2). If you need some libraries to do diagnostic work, you can install them now (e.g., pip install numpy matplotlib ipython) but make sure that they are included in setup.cfg if your package needs them. We recommend using pip for installation rather than conda (see some notes on the differences here), unless you need the binaries for a given library. Note: if you do not have conda installed, we recommend miniconda.

2. Make a copy of the repository on your computer. Click on the green Code button and copy the HTTPS or SSH link, depending on how you authenticate with GitHub.

    git clone https://github.com/mpi-astronomy/new_project.git
   

   or

    git clone [email protected]:mpi-astronomy/new_project.git
   

   If you have not set up HTTPS or SSH authentication for GitHub on your computer, then follow the instructions here for HTTPS or here for SSH.

3. Edit the pyproject.toml file.

   Specifically change the following lines:

    [project]
    name = "new_package_name"
    authors = [{name = "Example Author", email = "[email protected]"}]
    description = "An example package"
   
    [project.urls]
    "Bug Tracker" = "https://github.com/mpi-astronomy/new_project/issues"
    "Source Code" = "https://github.com/mpi-astronomy/new_project"
   
    [tool.setuptools_scm]
    write_to = "src/new_package_name/_version.py"
   

   The my_package_name here will be the name of your package, it does not have to be the same as the name of the repository (as shown here). This will be the name used to import your package once installed. See the Python style guide here for package naming conventions (tl;dr: use lower case letters and underscores only).

   As you develop your code, you will need to specify any dependencies, i.e. packages from which you have direct imports. Add any new dependencies here (you get numpy for free):

    dependencies = [
        "numpy",
    ]
   

   Test and documentation dependencies can be specified here:

    [project.optional-dependencies]
    docs = [
        "sphinx",
        "sphinx-automodapi",
        "numpydoc",
    ]
    test = [
        "pytest",
        "pytest-doctestplus",
        "flake8",
        "flake8-pyproject",
        "codecov",
        "pytest-cov",
    ]
   

   Edit as needed.

4. Rename your package.

   If you chose a different name for your package in name you should now rename your directory name as well. For example, in the previous step I changed the name of this package from the default my_package to new_package_name. Here we rename the directory too:

    cd src/
    git mv my_package new_package_name
   

5. Edit the README.md. Add a short description of your package instead of this text.

6. Edit the CITATION.cff file to specify how you want your code cited. The CITATION.cff file allows you to generate a BibTeX citation directly from the repository page (see Cite this repository in the upper right corner of this page). The fields we include here are the minimum recommended in the AstroBetter post on citing astronomical software with no DOI. Please review the recommendations therein if you would like to add more information to the citation. Additional information about CITATION.cff files can be found here.

7. Add your code to the src/new_package_name/ directory. All your code should be *.py files in this directory. Do not add other directories inside src/ unless you know what you are doing. Add your tests to the tests/ directory. Edit the docs/index.rst file to create documentation for your project.

🎉 You have a package.

What not to Change

1. Do not change the LICENSE file and definitely do not delete it. For more information, see this article.

2. If the copy is within the mpi-astronomy GitHub organization, do not edit the CODE_OF_CONDUCT.md file. If the copy is under your personal account, please customize the CODE_OF_CONDUCT.md file. One easy way to do that is to remove the text in square braces and to edit the reporting section to read "Any violations of the Code of Conduct should be reported to the owners of this repository."

3. Using this Python packaging template gives you the most up-to-date way to package your Python code. You do not need to have setup.py, setup.cfg or requirements.txt files. The setup.* files are legacy formats that are now deprecated for most use cases. All your requirements should be listed in the pyproject.toml file. If you have a more complex package with C or Fortran code that needs to be compiled during install, you may also need to create a setup.py file, but we are going to keep things here simple.

Then What?

Once you have made a local copy of the repository, you can install the package and start developing, testing and documenting it.

Versioning Your Package

This package template does versioning for your package automatically using setuptools-scm which generates a version based on git tags in the standard semantic versioning format A.B.c. For more information see setuptools-scm and Python packaging Version specifiers.

This means that when you have version of your package that you want to tag and release, make sure HEAD is pointing the the commit where you want to tag, and use git tag A.B.c to tag it with a proper semantic version. The next time you install the package, it should show you the updated version based on your tag.

If you do not want to do the automatic versioning (your code is not a git repo or you just want to do your own thing) then use the static_version branch of the repo. This branch can also be downloaded as a ZIP file and used locally. This is not the recommended workflow however.

Installation

With the package structure used here, you do not have to point Python to the location of your package. You absolutely SHOULD NOT be adding the package directory to your $PYTHONPATH. Instead, you can use pip to install it locally:

cd ~/path/to/new_project
pip install -e .

pip will install all the dependencies specified in the pyproject.toml file. The -e flag makes the install editable which means that you do not have to install the package again and again after each change. Changes to your files inside the project folder will automatically reflect in changes on your installed package. If you are working in an interactive environment (ipython, Jupyter) you will need to re-import any modules that have changed. For example, after editing module_x.py you will need to do the following to have the changes available in the Python interpreter:

import importlib
importlib.reload(module_x)

An accessible description of pip install can be found here.

To install a non-editable version, do:

cd ~/path/to/new_project
pip install .

This is how you can use your package once you are no longer developing it. Any users who are not contributing code can installing your package with:

pip install git+https://github.com/mpi-astronomy/new_project

Commit early and often

As you make changes to your package, get into the habit of committing changes early and often. Every time you add a new function, a new test, edit the docstring:

git add new_module.py
git commit -m "Added a function to reverse the sprocket of the whoosle."

And every few commits:

git push

Testing your code

Ideally, you should be writing tests along with the new code. To test your code, first install the package in editable mode with the optional test dependencies:

pip install -e ".[test]"

Then run the tests from the git repo root directory:

pytest --cov=src/<my_package> tests

changing <my_package> to whatever your package name is. This generates a report on how much of your code is covered by tests. Ideally this should be >80%.

Checking Code Style

To check for compliance with the Python style guide, run flake8:

flake8

This will do basic linting of your code, making sure you following best coding practices and will often also find bugs.

Continuous Integration

This repository come pre-set with continuous integration using GitHub Actions. Every time you push a commit or make a pull request, all tests will be automatically run by GitHub. On the GitHub page for your repository you should have an Action tab (forth from the left). This tab will show you the test results. While you can (and should) run the test suite locally, these runs are usually only on your operating system against one version of python. The advantage of CI is that you can test your code against different versions of python, different versions of key libraries and different operating systems. Here we have set up a simple test matrix which runs against 3 different versions of python. You can make the CI more complex if you need. You can disable/enable actions as shown here or by deleting the .github/workflows/ci.yml file (use git rm).

Creating documentation

If/when you want to update the auto-generated sphinx documentation, you can edit the docs/index.rst file. This file is in reStructuredText format. More information on making your docs pretty is available in the sphinx docs.

To generate the documentation, you need to first install the dependencies and then make the pages:

pip install -e ".[docs]"
cd docs/
make html
open _build/html/index.html

sphinx can also generate a PDF of your docs, but this is left as an exercise for the user.

This repository is also set to auto-generate an HTML page with the documentation and creates a GitHub pages webpage. While the files are auto-generated, the page must be made visible in the first place. Go to the Settings tab in GitHub and in the left-hand menu navigate to the Pages option. Select the gh-pages branch in the drop down Source menu. This is a one-time setting. The URL for your documentation will be displayed in the green banner. The example documentation page for this repository can be found at https://mpi-astronomy.github.io/mpia-python-template/.

You can disable/enable the auto-generated documentation builds as shown here or by deleting the .github/workflows/docs.yml file (use git rm). To unpublish the documentation page, you also need to delete the gh-pages branch, see instructions here.