feanil/fix uwsgi xml issues

.github/workflows/release.yml

@@ -22,7 +22,7 @@ jobs:
           # It's important that we build the tutor binaries with the *oldest* possible
           # OS releases and Python version. See these docs for more information:
           # https://pyinstaller.org/en/stable/usage.html#making-gnu-linux-apps-forward-compatible
-          - os: ubuntu-20.04
+          - os: ubuntu-22.04
             locale: C.UTF-8
           # https://endoflife.date/macos
           - os: macos-12

Makefile

@@ -90,7 +90,7 @@ bootstrap-dev-plugins: bootstrap-dev ## Install dev requirements and all support
 	pip install -r requirements/plugins.txt
 
 pull-base-images: # Manually pull base images
-	docker image pull docker.io/ubuntu:20.04
+	docker image pull docker.io/ubuntu:22.04
 
 ci-info: ## Print info about environment
 	python --version

changelog.d/20240520_150131_danyal.faheem_upgrade_mysql_utf8mb4.md

@@ -0,0 +1,7 @@
+- 💥[Feature] Upgrade default charset and collation of mysql to utf8mb4 and utf8mb4_unicode_ci respectively (by @Danyal-Faheem)
+    - Add do command to upgrade the charset and collation of tables in mysql.
+    - The command will perform the following upgrades:
+      - Upgrade all `utf8mb3` charset to `utf8mb4`
+      - Upgrade collation `utf8mb3_general_ci` to `utf8mb4_unicode_ci`
+      - Upgrade collation `utf8mb3_bin` to `utf8mb4_bin`
+      - Upgrade collation `utf8mb3_*` to `utf8mb4_*`

changelog.d/20240926_151223_dave_private_media.md

@@ -0,0 +1 @@
+- [Feature] Create a new /data/openedx-media-private volume to store media files that are not publicly accessible via the browser, and create configuration that Learning Core (openedx-learning) will use to access this storage space. Learning Core will use a sub-directory of /data/openedx-media-private to store Content Library file uploads, but other apps can create their own sub-directories for their own use. (by @ormsbee)

changelog.d/20241010_122752_dawoud.sheraz_ubuntu_upgrade.md

@@ -0,0 +1,2 @@
+- 💥[Feature] Update Open edX Image to use Ubuntu 22.04 as base OS. (by @dawoudsheraz)
+  - Adds xmlsec related dependencies to fix xmlsec import issues during translations build

changelog.d/20241010_131903_regis_local_openedx_io.md

@@ -0,0 +1 @@
+- 💥[Feature] Migrate from local.edly.io to local.openedx.io. (by @regisb)

docs/dev.rst

@@ -28,7 +28,7 @@ This will perform several tasks. It will:
 * build the "openedx-dev" Docker image, which is based on the "openedx" production image but is `specialized for developer usage`_ (eventually with your fork),
 * stop any existing locally-running Tutor containers,
 * disable HTTPS,
-* set ``LMS_HOST`` to `local.edly.io <http://local.edly.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.edly.io>`_),
+* set ``LMS_HOST`` to `local.openedx.io <http://local.openedx.io>`_ (a convenience domain that simply `points at 127.0.0.1 <https://dnschecker.org/#A/local.openedx.io>`_),
 * prompt for a platform details (with suitable defaults),
 * start LMS, CMS, supporting services, and any plugged-in services,
 * ensure databases are created and migrated, and
@@ -42,8 +42,8 @@ Additionally, when a local clone of edx-platform is bind-mounted, it will:
 
 Once setup is complete, the platform will be running in the background:
 
-* LMS will be accessible at `http://local.edly.io:8000 <http://local.edly.io:8000>`_.
-* CMS will be accessible at `http://studio.local.edly.io:8001 <http://studio.local.edly.io:8001>`_.
+* LMS will be accessible at `http://local.openedx.io:8000 <http://local.openedx.io:8000>`_.
+* CMS will be accessible at `http://studio.local.openedx.io:8001 <http://studio.local.openedx.io:8001>`_.
 * Plugged-in services should be accessible at their documented URLs.
 
 Now, use the ``tutor dev ...`` command-line interface to manage the development environment. Some common commands are described below.

docs/local.rst

@@ -141,6 +141,33 @@ The default Open edX theme is rather bland, so Tutor makes it easy to switch to
 
 Out of the box, only the default "open-edx" theme is available. We also developed `Indigo, a beautiful, customizable theme <https://github.com/overhangio/indigo>`__ which is easy to install with Tutor.
 
+Changing the mysql charset and collation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note:: This command has been tested only for users upgrading from Quince. While it is expected to work for users on earlier releases, please use it with caution as it has not been tested with those versions.
+
+The database's charset and collation might not support specific characters or emojis. Tutor will function normally without this change unless specific characters are used in the instance.
+
+.. warning:: This change is potentially irreversible. It is recommended to make a backup of the MySQL database. See the :ref:`database dump instructions <database_dumps>` to create a DB dump.
+
+To change the charset and collation of all the tables in the openedx database, run::
+
+    tutor local do convert-mysql-utf8mb4-charset
+
+Alternatively, to change the charset and collation of certain tables or to exclude certain tables, the ``--include`` or ``--exclude`` options can be used. These options take comma separated names of tables/apps with no space in-between. To upgrade the ``courseware_studentmodule`` and ``courseware_studentmodulehistory`` tables, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --include=courseware_studentmodule,courseware_studentmodulehistory
+
+Tutor performs pattern matching from the start of the table name so just the name of the app is enough to include/exclude all the tables under that app. To upgrade all the tables in the database except the ones under the student and wiki apps, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --exclude=student,wiki
+
+In the above command, all the tables whose name starts with either student or wiki will be excluded from the upgrade process.
+
+By default, only the tables in the openedx database are changed. For upgrading tables in any additional databases used by plugins, the ``--database`` option can be used to upgrade them. To upgrade all the tables in the discovery database, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --database=discovery
+
 Running arbitrary ``manage.py`` commands
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/tutorials/datamigration.rst

@@ -22,6 +22,8 @@ With Tutor, all data are stored in a single folder. This means that it's extreme
 
     tutor local start -d
 
+.. _database_dumps:
+
 Making database dumps
 ---------------------
 

docs/tutorials/theming.rst

@@ -46,12 +46,12 @@ Then, run a local webserver::
 
     tutor dev start lms
 
-The LMS can then be accessed at http://local.edly.io:8000. You will then have to :ref:`enable that theme <settheme>`::
+The LMS can then be accessed at http://local.openedx.io:8000. You will then have to :ref:`enable that theme <settheme>`::
 
     tutor dev do settheme mythemename
 
 Watch the themes folders for changes (in a different terminal)::
 
     tutor dev run watchthemes
 
-Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://local.edly.io:8000.
+Make changes to some of the files inside the theme directory: the theme assets should be automatically recompiled and visible at http://local.openedx.io:8000.

requirements/plugins.txt

@@ -1,14 +1,14 @@
-# change version ranges when upgrading from redwood
-tutor-android>=18.0.0,<19.0.0
-tutor-cairn>=18.0.0,<19.0.0
-tutor-credentials>=18.0.0,<19.0.0
-tutor-discovery>=18.0.0,<19.0.0
-tutor-ecommerce>=18.0.0,<19.0.0
-tutor-forum>=18.0.0,<19.0.0
-tutor-indigo>=18.0.0,<19.0.0
-tutor-jupyter>=18.0.0,<19.0.0
-tutor-mfe>=18.0.0,<19.0.0
-tutor-minio>=18.0.0,<19.0.0
-tutor-notes>=18.0.0,<19.0.0
-tutor-webui>=18.0.0,<19.0.0
-tutor-xqueue>=18.0.0,<19.0.0
+# # For Tutor Nightly, we install plugins from their nightly branches instead of from PyPI
+tutor-android@git+https://github.com/overhangio/tutor-android@nightly
+tutor-cairn@git+https://github.com/overhangio/tutor-cairn@nightly
+tutor-credentials@git+https://github.com/overhangio/tutor-credentials@nightly
+tutor-discovery@git+https://github.com/overhangio/tutor-discovery@nightly
+tutor-ecommerce@git+https://github.com/overhangio/tutor-ecommerce@nightly
+tutor-forum@git+https://github.com/overhangio/tutor-forum@nightly
+tutor-indigo@git+https://github.com/overhangio/tutor-indigo@nightly
+tutor-jupyter@git+https://github.com/overhangio/tutor-jupyter@nightly
+tutor-mfe@git+https://github.com/overhangio/tutor-mfe@nightly
+tutor-minio@git+https://github.com/overhangio/tutor-minio@nightly
+tutor-notes@git+https://github.com/overhangio/tutor-notes@nightly
+tutor-webui@git+https://github.com/overhangio/tutor-webui@nightly
+tutor-xqueue@git+https://github.com/overhangio/tutor-xqueue@nightly

tests/commands/test_jobs.py

@@ -90,3 +90,78 @@ def test_set_theme(self) -> None:
             self.assertIn("lms-job", dc_args)
             self.assertIn("assign_theme('beautiful', 'domain1')", dc_args[-1])
             self.assertIn("assign_theme('beautiful', 'domain2')", dc_args[-1])
+
+    def test_convert_mysql_utf8mb4_charset_all_tables(self) -> None:
+        with temporary_root() as root:
+            self.invoke_in_root(root, ["config", "save"])
+            with patch("tutor.utils.docker_compose") as mock_docker_compose:
+                result = self.invoke_in_root(
+                    root,
+                    [
+                        "local",
+                        "do",
+                        "convert-mysql-utf8mb4-charset",
+                        "--non-interactive",
+                    ],
+                )
+                dc_args, _dc_kwargs = mock_docker_compose.call_args
+
+            self.assertIsNone(result.exception)
+            self.assertEqual(0, result.exit_code)
+            self.assertIn("lms-job", dc_args)
+            self.assertIn("utf8mb4", dc_args[-1])
+            self.assertIn("openedx", dc_args[-1])
+            self.assertIn("utf8mb4_unicode_ci", dc_args[-1])
+            self.assertNotIn("regexp", dc_args[-1])
+
+    def test_convert_mysql_utf8mb4_charset_include_tables(self) -> None:
+        with temporary_root() as root:
+            self.invoke_in_root(root, ["config", "save"])
+            with patch("tutor.utils.docker_compose") as mock_docker_compose:
+                result = self.invoke_in_root(
+                    root,
+                    [
+                        "local",
+                        "do",
+                        "convert-mysql-utf8mb4-charset",
+                        "--include=courseware_studentmodule,xblock",
+                    ],
+                )
+                dc_args, _dc_kwargs = mock_docker_compose.call_args
+
+            self.assertIsNone(result.exception)
+            self.assertEqual(0, result.exit_code)
+            self.assertIn("lms-job", dc_args)
+            self.assertIn("openedx", dc_args[-1])
+            self.assertIn("utf8mb4", dc_args[-1])
+            self.assertIn("utf8mb4_unicode_ci", dc_args[-1])
+            self.assertIn("regexp", dc_args[-1])
+            self.assertIn("courseware_studentmodule", dc_args[-1])
+            self.assertIn("xblock", dc_args[-1])
+
+    def test_convert_mysql_utf8mb4_charset_exclude_tables(self) -> None:
+        with temporary_root() as root:
+            self.invoke_in_root(root, ["config", "save"])
+            with patch("tutor.utils.docker_compose") as mock_docker_compose:
+                result = self.invoke_in_root(
+                    root,
+                    [
+                        "local",
+                        "do",
+                        "convert-mysql-utf8mb4-charset",
+                        "--database=discovery",
+                        "--exclude=course,auth",
+                    ],
+                )
+                dc_args, _dc_kwargs = mock_docker_compose.call_args
+
+            self.assertIsNone(result.exception)
+            self.assertEqual(0, result.exit_code)
+            self.assertIn("lms-job", dc_args)
+            self.assertIn("utf8mb4", dc_args[-1])
+            self.assertIn("utf8mb4_unicode_ci", dc_args[-1])
+            self.assertIn("discovery", dc_args[-1])
+            self.assertIn("regexp", dc_args[-1])
+            self.assertIn("NOT", dc_args[-1])
+            self.assertIn("course", dc_args[-1])
+            self.assertIn("auth", dc_args[-1])

tutor/__about__.py

@@ -10,7 +10,7 @@
 # the nightly branch.
 # The suffix is cleanly separated from the __version__ in this module to avoid
 # conflicts when merging branches.
-__version_suffix__ = ""
+__version_suffix__ = "nightly"
 
 # The app name will be used to define the name of the default tutor root and
 # plugin directory. To avoid conflicts between multiple locally-installed

tutor/commands/jobs.py

@@ -13,6 +13,8 @@
 
 from tutor import config as tutor_config
 from tutor import env, fmt, hooks
+from tutor.commands.context import Context
+from tutor.commands.jobs_utils import get_mysql_change_charset_query
 from tutor.hooks import priorities
 
 
@@ -315,6 +317,109 @@ def sqlshell(args: list[str]) -> t.Iterable[tuple[str, str]]:
     yield ("lms", command)
 
 
+@click.command(
+    short_help="Convert the charset and collation of mysql to utf8mb4.",
+    help=(
+        "Convert the charset and collation of mysql to utf8mb4. You can either upgrade all tables, specify only certain tables to upgrade or specify certain tables to exclude from the upgrade process"
+    ),
+    context_settings={"ignore_unknown_options": True},
+)
+@click.option(
+    "--include",
+    is_flag=False,
+    nargs=1,
+    help="Apps/Tables to include in the upgrade process. Requires comma-seperated values with no space in-between.",
+)
+@click.option(
+    "--exclude",
+    is_flag=False,
+    nargs=1,
+    help="Apps/Tables to exclude from the upgrade process. Requires comma-seperated values with no space in-between.",
+)
+@click.option(
+    "--database",
+    is_flag=False,
+    nargs=1,
+    default="{{ OPENEDX_MYSQL_DATABASE }}",
+    show_default=True,
+    required=True,
+    type=str,
+    help="The database of which the tables are to be upgraded",
+)
+@click.option("-I", "--non-interactive", is_flag=True, help="Run non-interactively")
+@click.pass_obj
+def convert_mysql_utf8mb4_charset(
+    context: Context,
+    include: str,
+    exclude: str,
+    database: str,
+    non_interactive: bool,
+) -> t.Iterable[tuple[str, str]]:
+    """
+    Do command to upgrade the charset and collation of tables in MySQL
+
+    Can specify whether to upgrade all tables, or include certain tables/apps or to exclude certain tables/apps
+    """
+
+    config = tutor_config.load(context.root)
+
+    if not config["RUN_MYSQL"]:
+        fmt.echo_info(
+            f"You are not running MySQL (RUN_MYSQL=false). It is your "
+            f"responsibility to upgrade the charset and collation of your MySQL instance."
+        )
+        return
+
+    # Prompt user for confirmation of upgrading all tables
+    if not include and not exclude and not non_interactive:
+        upgrade_all_tables = click.confirm(
+            "Are you sure you want to upgrade all tables? This process is potentially irreversible and may take a long time.",
+            prompt_suffix=" ",
+        )
+        if not upgrade_all_tables:
+            return
+
+    charset_to_upgrade_from = "utf8mb3"
+    charset = "utf8mb4"
+    collation = "utf8mb4_unicode_ci"
+
+    query_to_append = ""
+    if include or exclude:
+
+        def generate_query_to_append(tables: list[str], exclude: bool = False) -> str:
+            include = "NOT" if exclude else ""
+            table_names = f"^{tables[0]}"
+            for i in range(1, len(tables)):
+                table_names += f"|^{tables[i]}"
+            # We use regexp for pattern matching the names from the start of the tablename
+            query_to_append = f"AND table_name {include} regexp '{table_names}' "
+            return query_to_append
+
+        query_to_append += (
+            generate_query_to_append(include.split(",")) if include else ""
+        )
+        query_to_append += (
+            generate_query_to_append(exclude.split(","), exclude=True)
+            if exclude
+            else ""
+        )
+    click.echo(
+        fmt.title(
+            f"Updating charset and collation of tables in the {database} database to {charset} and {collation} respectively."
+        )
+    )
+    query = get_mysql_change_charset_query(
+        database, charset, collation, query_to_append, charset_to_upgrade_from
+    )
+
+    mysql_command = (
+        "mysql --user={{ MYSQL_ROOT_USERNAME }} --password={{ MYSQL_ROOT_PASSWORD }} --host={{ MYSQL_HOST }} --port={{ MYSQL_PORT }} --skip-column-names --silent "
+        + shlex.join([f"--database={database}", "-e", query])
+    )
+    yield ("lms", mysql_command)
+    fmt.echo_info("MySQL charset and collation successfully upgraded")
+
+
 def add_job_commands(do_command_group: click.Group) -> None:
     """
     This is meant to be called with the `local/dev/k8s do` group commands, to add the
@@ -390,6 +495,7 @@ def do_callback(service_commands: t.Iterable[tuple[str, str]]) -> None:
 
 hooks.Filters.CLI_DO_COMMANDS.add_items(
     [
+        convert_mysql_utf8mb4_charset,
         createuser,
         importdemocourse,
         importdemolibraries,