feat(extensions): enable async Gunicorn workers for Flask and Django (#…

…747)

* Feat(extension): Enable Async Gunicorn workers for Flask and Django extensions

* chore(lint): Ran static checkers

* chore(): Fix flask cli unit test

* chore(docs): Docs lint

* chore(docs): Improved wordlist

* Chore(): Update docs, update cli init test, update `rockcraft.yaml` file verification

* Chore(): Try to make linter happy

* Chore(docs): Lint docs

* Chore(docs): Update async doc link

* Run CI

* Chore(Docs): Small doc improvements

* Chore(lint): Doc lint

* Chore(docs): Add words to list

* Chore(): Undone import format

* Chore(): Addressed comments.

* Chore(): Consistent spacing

* Chore(Format): Format imports

* Chore(Lint): Add word

* Chore(): Update docs, update gunicorn conf for async

* Chore(lint): Format

* Chore(tests): Fix tests

* Chore(docs): Fix refs

* Chore(docs): Update refs

* Chore(): Updated docs, async conf file

* Chore(docs): Fix lint

* Chore(): Add a spread test for async

* Chore(test): Removed unnecessary fixture

* Chore(): Format

* Merge main

* Chore(test): Simplify test and fix typo

* Chore(): Improve code

* Chore(lint): Make linter happy

* Chore(): Make linter happier

* Chore(test): Fix unit test

* Chore(lint): Make Mypy happy

* Chore(): Fix shell lint

* Chore(test): Fix test comment

* Chore(test): Add async flask/django tests. Created test dir

* Chore(lint): Run autoformat

* Chore(test): Ignore import errors in django data

* Chore(test): Fix cli test

* Chore(format): format

* Chore(): Change async flask implementation

* Chore(test): Fix spread test

* Chore(): Update docs and spread test

* Chore(test): Removed unnecessary django files

* Chore(doc): Lint docs

* Update docs/reference/extensions/django-framework.rst

Co-authored-by: Erin Conley <[email protected]>

* Update docs/reference/extensions/django-framework.rst

Co-authored-by: Erin Conley <[email protected]>

* Update docs/reference/extensions/django-framework.rst

Co-authored-by: Erin Conley <[email protected]>

* Update docs/reference/extensions/django-framework.rst

Co-authored-by: Erin Conley <[email protected]>

* Chore(): Update gevent install check

* Chore(docs): Style update

* Chore(): Updated pip package check

* Chore(Deps): Add `packaging` dependency

* Chore(lint): Format code

* Chore(format): Format

* Chore(docs): Format doc

* Chore(): Fix spread test, change async check

* Chore(): Except specific exception

* Chore(): Uncomment restore

* Update docs/reference/extensions/django-framework.rst

Co-authored-by: Tiago Nobrega <[email protected]>

* Update docs/reference/extensions/flask-framework.rst

Co-authored-by: Tiago Nobrega <[email protected]>

* Chore(): Addressed comments. Add comment to test_cli.py. fix doc issue. Remove double dependency.

* Chore(test): Add unit async gunicorn tests

* Chore(lint): Format code

* Chore(): Changed copy_tree() to shutil.copytree()

* chore(doc): Fix doc format

* chore(doc): Format

* chore(doc): Format

* chore(doc): Format

* chore(doc): Format

* chore(test): Update tests

* chore(doc): Change testing port in doc

* chore(update): Update statsd-exporter version in flask/django plugin

* chore(): Revert version change in statsd

* chore(doc): Update async docs

* Update docs/reference/extensions/django-framework.rst

Co-authored-by: Michael DuBelko <[email protected]>

* Update docs/reference/extensions/flask-framework.rst

Co-authored-by: Michael DuBelko <[email protected]>

---------

Co-authored-by: Erin Conley <[email protected]>
Co-authored-by: Tiago Nobrega <[email protected]>
Co-authored-by: Michael DuBelko <[email protected]>

docs/.wordlist.txt

@@ -3,6 +3,7 @@ AMD
 amd
 ARGS
 ASGI
+async
 Autotools
 autotools
 boolean
@@ -44,10 +45,12 @@ filesystems
 filetree
 fs
 gc
+gevent
 GiB
 GID
 github
 GPG
+gunicorn
 Gunicorn
 gzipped
 hardcoded
@@ -144,6 +147,7 @@ triaged
 ubuntu
 unbuilt
 UID
+uncomment
 usrmerge
 Uvicorn
 VENV

docs/reference/extensions/django-framework.rst

@@ -17,6 +17,9 @@ server metrics.
     The Django extension is compatible with the ``bare``, ``[email protected]``
     and ``[email protected]`` bases.
 
+The Django extension supports both synchronous and asynchronous
+Gunicorn workers.
+
 Project requirements
 ====================
 
@@ -29,6 +32,11 @@ There are 2 requirements to be able to use the ``django-framework`` extension:
    ``./<Rock name with - replaced by _>/<Rock name with - replaced by _>/manage.py``
    relative to the ``rockcraft.yaml`` file.
 
+For the project to make use of asynchronous Gunicorn workers:
+
+- The ``requirements.txt`` file must include ``gevent`` as a dependency.
+
+
 ``parts`` > ``django-framework/dependencies:`` > ``stage-packages``
 ===================================================================
 
@@ -43,6 +51,23 @@ application. In the following example we use it to specify ``libpq-dev``:
         # list required packages or slices for your Django application below.
         - libpq-dev
 
+.. _django-gunicorn-worker-selection:
+
+Gunicorn worker selection
+=========================
+
+If the project has gevent as a dependency, Rockcraft automatically updates the
+pebble plan to spawn asynchronous Gunicorn workers.
+
+When the project instead needs synchronous workers, you can override the worker
+type by adding ``--args django sync`` to the Docker command that launches the
+rock:
+
+.. code-block:: bash
+
+   docker run --name django-container -d -p 8000:8000 django-image:1.0 \
+   --args django sync
+
 Useful links
 ============
 

docs/reference/extensions/flask-framework.rst

@@ -16,6 +16,9 @@ server metrics.
     The Flask extension is compatible with the ``bare``, ``[email protected]``
     and ``[email protected]`` bases.
 
+The Flask extension supports both synchronous and asynchronous
+Gunicorn workers.
+
 Project requirements
 ====================
 
@@ -25,7 +28,12 @@ There are 2 requirements to be able to use the ``flask-framework`` extension:
    ``Flask`` declared as a dependency
 2. The project must include a WSGI app with the path ``app:app``. This means
    there must be an ``app.py`` file at the root of the project with the name
-   of the Flask object is set to ``app``
+   of the Flask object is set to ``app``.
+
+For the project to make use of asynchronous Gunicorn workers:
+
+- The ``requirements.txt`` file must include ``gevent`` as a dependency.
+
 
 ``parts`` > ``flask-framework/dependencies`` > ``stage-packages``
 =================================================================
@@ -41,6 +49,23 @@ application. In the following example we use it to specify ``libpq-dev``:
         # list required packages or slices for your flask app below.
         - libpq-dev
 
+.. _flask-gunicorn-worker-selection:
+
+Gunicorn worker selection
+=========================
+
+If the project has gevent as a dependency, Rockcraft automatically updates the
+pebble plan to spawn asynchronous Gunicorn workers.
+
+When the project instead needs synchronous workers, you can override the worker
+type by adding ``--args flask sync`` to the Docker command that launches the
+rock:
+
+.. code-block:: bash
+
+   docker run --name flask-container -d -p 8000:8000 flask-image:1.0 \
+   --args flask sync
+
 ``parts`` > ``flask-framework/install-app`` > ``prime``
 =======================================================
 

rockcraft/extensions/gunicorn.py

@@ -23,6 +23,7 @@
 from typing import Any
 
 from overrides import override
+from packaging.requirements import InvalidRequirement, Requirement
 
 from ..errors import ExtensionError
 from ._python_utils import has_global_variable
@@ -112,6 +113,19 @@ def _gen_parts(self) -> dict:
             }
         return parts
 
+    def _check_async(self) -> str:
+        """Check if gevent package installed in requirements.txt."""
+        requirements_file = self.project_root / "requirements.txt"
+        requirements_text = requirements_file.read_text()
+        for line in requirements_text.splitlines():
+            try:
+                req = Requirement(line)
+                if req.name == "gevent":
+                    return "gevent"
+            except InvalidRequirement:
+                pass
+        return "sync"
+
     @override
     def get_root_snippet(self) -> dict[str, Any]:
         """Fill in some default root components.
@@ -130,7 +144,7 @@ def get_root_snippet(self) -> dict[str, Any]:
                 self.framework: {
                     "override": "replace",
                     "startup": "enabled",
-                    "command": f"/bin/python3 -m gunicorn -c /{self.framework}/gunicorn.conf.py {self.wsgi_path}",
+                    "command": f"/bin/python3 -m gunicorn -c /{self.framework}/gunicorn.conf.py {self.wsgi_path} -k [ {self._check_async()} ]",
                     "after": ["statsd-exporter"],
                     "user": "_daemon_",
                 },

rockcraft/models/project.py

@@ -483,7 +483,7 @@ def load_project(filename: Path) -> dict[str, Any]:
         msg = err.strerror or "unknown"
         if err.filename:
             msg = f"{msg}: {err.filename!r}."
-        raise ProjectLoadError(msg) from err
+        raise ProjectLoadError(str(msg)) from err
 
     return transform_yaml(filename.parent, yaml_data)
 

tests/data/django/expected_rockcraft.yaml

@@ -0,0 +1,43 @@
+name: test-name
+# see https://documentation.ubuntu.com/rockcraft/en/latest/explanation/bases/
+# for more information about bases and using 'bare' bases for chiselled rocks
+base: [email protected] # the base environment for this Django application
+version: '0.1' # just for humans. Semantic versioning is recommended
+summary: A summary of your Django application # 79 char long summary
+description: |
+    This is test-name's description. You have a paragraph or two to tell the
+    most important story about it. Keep it under 100 words though,
+    we live in tweetspace and your description wants to look good in the
+    container registries out there.
+# the platforms this rock should be built on and run on.
+# you can check your architecture with `dpkg --print-architecture`
+platforms:
+    amd64:
+    # arm64:
+    # ppc64el:
+    # s390x:
+
+# to ensure the django-framework extension functions properly, your Django project
+# should have a structure similar to the following with ./test_name/test_name/wsgi.py
+# being the WSGI entry point and contain an application object.
+# +-- test_name
+# |   |-- test_name
+# |   |   |-- wsgi.py
+# |   |   +-- ...
+# |   |-- manage.py
+# |   |-- migrate.sh
+# |   +-- some_app
+# |       |-- views.py
+# |       +-- ...
+# |-- requirements.txt
+# +-- rockcraft.yaml
+
+extensions:
+    - django-framework
+
+# uncomment the sections you need and adjust according to your requirements.
+# parts:
+#   django-framework/dependencies:
+#     stage-packages:
+#       # list required packages or slices for your Django application below.
+#       - libpq-dev

tests/data/django/requirements.txt

@@ -0,0 +1 @@
+Django

tests/data/django/test_name/test_name/wsgi.py

@@ -0,0 +1,7 @@
+import os
+
+# Rockcraft repo doesn't need Django installed so ignore module-not-found mypy error
+from django.core.wsgi import get_wsgi_application  # type: ignore
+
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "django_app.settings")
+application = get_wsgi_application()

tests/data/flask/app.py

@@ -0,0 +1 @@
+app = object()

tests/data/flask/expected_rockcraft.yaml

@@ -0,0 +1,71 @@
+name: test-name
+# see https://documentation.ubuntu.com/rockcraft/en/latest/explanation/bases/
+# for more information about bases and using 'bare' bases for chiselled rocks
+base: [email protected] # the base environment for this Flask application
+version: '0.1' # just for humans. Semantic versioning is recommended
+summary: A summary of your Flask application # 79 char long summary
+description: |
+    This is test-name's description. You have a paragraph or two to tell the
+    most important story about it. Keep it under 100 words though,
+    we live in tweetspace and your description wants to look good in the
+    container registries out there.
+# the platforms this rock should be built on and run on.
+# you can check your architecture with `dpkg --print-architecture`
+platforms:
+    amd64:
+    # arm64:
+    # ppc64el:
+    # s390x:
+
+# to ensure the flask-framework extension works properly, your Flask application
+# should have an `app.py` file with an `app` object as the WSGI entrypoint.
+# a `requirements.txt` file with at least the flask package should also exist.
+# see https://documentation.ubuntu.com/rockcraft/en/latest/reference/extensions/flask-framework
+# for more information.
+extensions:
+    - flask-framework
+
+# uncomment the sections you need and adjust according to your requirements.
+# parts:  # you need to uncomment this line to add or update any part.
+
+#   flask-framework/install-app:
+#     prime:
+#       # by default, only the files in app/, templates/, static/, migrate, migrate.sh,
+#       # migrate.py and app.py are copied into the image. You can modify the list
+#       # below to override the default list and include or exclude specific
+#       # files/directories in your project.
+#       # note: prefix each entry with "flask/app/" followed by the local path.
+#       - flask/app/.env
+#       - flask/app/app.py
+#       - flask/app/webapp
+#       - flask/app/templates
+#       - flask/app/static
+
+# you may need Ubuntu packages to build a python dependency. Add them here if necessary.
+#   flask-framework/dependencies:
+#     build-packages:
+#       # for example, if you need pkg-config and libxmlsec1-dev to build one
+#       # of your packages:
+#       - pkg-config
+#       - libxmlsec1-dev
+
+# you can add package slices or Debian packages to the image.
+# package slices are subsets of Debian packages, which result
+# in smaller and more secure images.
+# see https://documentation.ubuntu.com/rockcraft/en/latest/explanation/chisel/
+
+# add this part if you want to add packages slices to your image.
+# you can find a list of packages slices at https://github.com/canonical/chisel-releases
+#   runtime-slices:
+#     plugin: nil
+#     stage-packages:
+#       # list the required package slices for your flask application below.
+#       # for example, for the slice libs of libpq5:
+#       - libpq5_libs
+
+# if you want to add a Debian package to your image, add the next part
+#   runtime-debs:
+#     plugin: nil
+#     stage-packages:
+#       # list required Debian packages for your flask application below.
+#       - libpq5

tests/data/flask/requirements.txt

@@ -0,0 +1 @@
+Flask

tests/spread/rockcraft/extension-flask-async/README

@@ -0,0 +1 @@
+test

tests/spread/rockcraft/extension-flask-async/app.py

@@ -0,0 +1,16 @@
+from time import sleep
+
+from flask import Flask  # pyright: ignore[reportMissingImports]
+
+app = Flask(__name__)
+
+
+@app.route("/")
+def ok():
+    return "ok"
+
+
+@app.route("/io")
+def pseudo_io():
+    sleep(2)
+    return "ok"

tests/spread/rockcraft/extension-flask-async/requirements.txt

@@ -0,0 +1,2 @@
+flask
+gevent

tests/spread/rockcraft/extension-flask-async/static/js/test.js

@@ -0,0 +1 @@
+console.log("hello")