feat: add do command to change MySQL charset to utf8mb4 #1079

Danyal-Faheem · 2024-06-14T12:57:51Z

Additional PR to close #938.

The first part of this work can be found in #1084.

Introduces a new do command convert_mysql_utf8mb4_charset that converts the default and current character and collation of the tables to utf8mb4.

The command has the following options:

database: the database in which the tables are to be converted. Default: {{ OPENEDX_MYSQL_DATABSE }}
include: comma separated list of tables on which the upgrade process is to be applied
exclude: comma separated list of tables to exclude from the upgrade process

The command will perform the following upgrades:
- Upgrade all utf8mb3 charset to utf8mb4
- Upgrade collation utf8mb3_general_ci to utf8mb4_unicode_ci
- Upgrade collation utf8mb3_bin to utf8mb4_bin
- Upgrade collation utf8mb3_* to utf8mb4_*

DawoudSheraz · 2024-07-29T13:56:28Z

@ormsbee FYI

ztraboo · 2024-08-06T16:27:35Z

Will these changes work with the Nutmeg release? I'm considering this to help with putting emojis in section and subsection header titles.
https://discuss.openedx.org/t/emoji-in-subsection-title-not-supported-since-nutmeg/7856

DawoudSheraz · 2024-08-15T11:36:32Z

docs/local.rst

+Changing the mysql charset and collation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note:: This command is only for users upgrading from quince. 


I am not entirely sure if this is correct.

I've updated this note.

DawoudSheraz · 2024-08-15T11:40:55Z

tutor/commands/jobs.py

+    if not config["RUN_MYSQL"]:
+        fmt.echo_info(
+            f"You are not running MySQL (RUN_MYSQL=false). It is your "
+            f"responsibility to update your MySQL instance to {charset} charset and {collation} collation."
+        )
+        return


should we not move this on top? If MySQL is not running, we don't need to do any actions.

Yes, we should. Thank you for picking this out. I will update it.

DawoudSheraz · 2024-08-15T11:55:49Z

tutor/utils.py

+                    CASE 
+                        WHEN COLLATION_NAME LIKE CONCAT('{charset_to_upgrade_from}', '_general_ci') THEN 'utf8mb4_unicode_ci'
+                        WHEN COLLATION_NAME LIKE CONCAT('{charset_to_upgrade_from}', '_%') THEN CONCAT('{charset}', SUBSTRING_INDEX(COLLATION_NAME, '{charset_to_upgrade_from}', -1))
+                        ELSE COLLATION_NAME 
+                    END AS COLLATION_NAME 


Nice, I did not know about CASE.

DawoudSheraz · 2024-08-15T11:58:28Z

tutor/utils.py

+                    # Then, upgrade the default charset and collation of each column
+                    # This sequence of table -> column is necessary to preserve column defaults


IF we are updating table again in UpdateTables, how will it not cause any issues?

UpdateTables will only update the tables that have the default collation of something other than utf8mb4_*, i.e, tables that weren't updated in UpdateColumns.

Reference: https://github.com/overhangio/tutor/pull/1079/files#diff-46b7a04b09b0ffeb02b4f36ed494cbed6eaa7ea1efb4c46f9951b4ff6b0a3a37R464

…bles

ormsbee · 2024-08-30T15:19:04Z

I am not really competent at writing MySQL stored procedures, but this seems right to me. Certainly the logical mappings in the description and the options you give in the Python script for inclusion/exclusion are good. I realize this is a pain to get right–thank you for sticking to it.

Danyal-Faheem · 2024-09-03T09:26:51Z

Hi @regisb, can you review this PR for me?

tutor/commands/jobs.py

tutor/utils.py

tutor/commands/jobs_utils.py

DawoudSheraz · 2024-09-16T07:51:54Z

docs/local.rst

+
+.. note:: This command has been tested only for users upgrading from Quince. While it is expected to work for users on earlier releases, please use it with caution as it has not been tested with those versions.
+
+Your database's charset and collation might not support specific characters or emojis. Tutor will run fine without this change unless you explicity use specific characters in your instance.
+
+.. warning:: This change is potentially irreversible. It is recommended to make a backup of the MySQL database. See the :ref:`database dump instructions <database_dumps>` to create a DB dump.
+
+To change the charset and collation of all the tables in the openedx database, run::
+
+    tutor local do convert-mysql-utf8mb4-charset
+
+Alternatively, if you only want to change the charset and collation of certain tables or exclude certain tables, you can use the ``--include`` or ``--exclude`` options. These options take comma separated names of tables/apps with no space in-between. To upgrade the ``courseware_studentmodule`` and ``courseware_studentmodulehistory`` tables, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --include=courseware_studentmodule,courseware_studentmodulehistory
+
+Tutor performs pattern matching from the start of the table name so you can just enter the name of the app to include/exclude all the tables under that app. To upgrade all the tables in the database except the ones under the student and wiki apps, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --exclude=student,wiki
+
+In the above command, all the tables whose name starts with either student or wiki will be excluded from the upgrade process.
+
+By default, only the tables in the openedx database are changed. If you are running any plugins with their own databases, you can upgrade them by utilizing the ``--database`` option. To upgrade all the tables in the discovery database, run::
+
+    tutor local do convert-mysql-utf8mb4-charset --database=discovery


nit: the usage of you/second person pronouns in the guide should be reduced/removed.

I've reworded the docs to reduce the usage.

DawoudSheraz · 2024-09-16T07:54:05Z

tutor/commands/jobs_utils.py

+    charset_to_upgrade_from: str,
+) -> str:
+    """
+    Helper function to generate the mysql query to upgrade the charset and collation of tables


it is not just tables, it is columns, tables and database.

DawoudSheraz · 2024-10-18T09:50:22Z

tutor/commands/upgrade/compose.py

@@ -164,6 +164,10 @@ def upgrade_from_quince(context: click.Context, config: Config) -> None:
    upgrade_mongodb(context, config, "5.0.26", "5.0")
    upgrade_mongodb(context, config, "6.0.14", "6.0")
    upgrade_mongodb(context, config, "7.0.7", "7.0")
+    fmt.echo_alert(


nit: this would be moved to upgrade from sumac section because we are targeting this for nightly and makes sense to have this there.

You mean to upgrade to Sumac, right? I agree, but we don't have an upgrade_from_redwood command yet, so we can make that change later in the sumac branch.

Oh yeah, right. I meant upgrade_from_redwood section of sumac branch. I will take care of that in sumac branch once this branch is merged.

I have updated this in sumac branch 2ad4be0

regisb

LGTM! Awesome work.

Danyal-Faheem marked this pull request as draft June 14, 2024 13:11

Danyal-Faheem force-pushed the danyalfaheem/mysql-charset-change-command branch from 1a18449 to 8a279f0 Compare June 14, 2024 13:16

Danyal-Faheem mentioned this pull request Jun 14, 2024

feat!: upgrade mysql charset and collation to utf8mb4 #1065

Closed

Danyal-Faheem mentioned this pull request Jun 25, 2024

feat!: upgrade default mysql charset and collation to utf8mb4 #1084

Merged

Danyal-Faheem force-pushed the danyalfaheem/mysql-charset-change-command branch from 8a279f0 to aef70b6 Compare July 26, 2024 13:21

Danyal-Faheem requested a review from DawoudSheraz July 26, 2024 13:22

Danyal-Faheem marked this pull request as ready for review July 26, 2024 13:22

DawoudSheraz requested a review from regisb July 29, 2024 13:55

DawoudSheraz reviewed Aug 15, 2024

View reviewed changes

feat: add do command to upgrade the charset and collation of MySQL ta…

7dc81d9

…bles

Danyal-Faheem force-pushed the danyalfaheem/mysql-charset-change-command branch from f4fbab9 to 7dc81d9 Compare August 21, 2024 13:39

docs: update alert message wording

c7090ba

regisb requested changes Sep 3, 2024

View reviewed changes

tutor/commands/jobs.py Outdated Show resolved Hide resolved

tutor/commands/jobs.py Outdated Show resolved Hide resolved

tutor/commands/jobs.py Outdated Show resolved Hide resolved

tutor/utils.py Outdated Show resolved Hide resolved

fix: make requested changes

89104e5

DawoudSheraz reviewed Sep 4, 2024

View reviewed changes

tutor/commands/jobs_utils.py Show resolved Hide resolved

Danyal-Faheem added 2 commits September 4, 2024 13:17

docs: add file docstring

128958d

fix: format files

2a92c49

DawoudSheraz reviewed Sep 16, 2024

View reviewed changes

DawoudSheraz approved these changes Sep 16, 2024

View reviewed changes

DawoudSheraz reviewed Sep 16, 2024

View reviewed changes

docs: reduce second person pronoun usage

dfd5f5f

Danyal-Faheem mentioned this pull request Oct 18, 2024

feat: add do command to update the authentication plugin of MySQL users #1097

Open

DawoudSheraz reviewed Oct 18, 2024

View reviewed changes

regisb approved these changes Nov 6, 2024

View reviewed changes

DawoudSheraz merged commit d4e6c6c into overhangio:nightly Nov 11, 2024
2 checks passed

feanil mentioned this pull request Nov 12, 2024

feanil/fix uwsgi xml issues #1156

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add do command to change MySQL charset to utf8mb4 #1079

feat: add do command to change MySQL charset to utf8mb4 #1079

Danyal-Faheem commented Jun 14, 2024 •

edited

Loading

DawoudSheraz commented Jul 29, 2024

ztraboo commented Aug 6, 2024

DawoudSheraz Aug 15, 2024

Danyal-Faheem Aug 21, 2024

DawoudSheraz Aug 15, 2024

Danyal-Faheem Aug 15, 2024

DawoudSheraz Aug 15, 2024

DawoudSheraz Aug 15, 2024

Danyal-Faheem Aug 15, 2024

ormsbee commented Aug 30, 2024

Danyal-Faheem commented Sep 3, 2024

DawoudSheraz Sep 16, 2024

Danyal-Faheem Sep 16, 2024

DawoudSheraz Sep 16, 2024

Danyal-Faheem Sep 16, 2024

DawoudSheraz Oct 18, 2024

regisb Oct 18, 2024

DawoudSheraz Oct 18, 2024

DawoudSheraz Nov 22, 2024

regisb left a comment

		# Then, upgrade the default charset and collation of each column
		# This sequence of table -> column is necessary to preserve column defaults

feat: add do command to change MySQL charset to utf8mb4 #1079

feat: add do command to change MySQL charset to utf8mb4 #1079

Conversation

Danyal-Faheem commented Jun 14, 2024 • edited Loading

DawoudSheraz commented Jul 29, 2024

ztraboo commented Aug 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ormsbee commented Aug 30, 2024

Danyal-Faheem commented Sep 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

regisb left a comment

Choose a reason for hiding this comment

Danyal-Faheem commented Jun 14, 2024 •

edited

Loading