Upgrade to Celery protocol 2

[timmc-edx]

edx-platform, registrar, and enterprise-catalog are all using Celery task protocol 1. Upgrading to protocol 2 may remove some technical debt, and as a side benefit may help with #692. We also may have seen a bug where something in celery (or an associated library) was adding headers to a v1 message as if it were a v2 message, which caused a bug in ddtrace; such things may become more likely over time as code is written with the assumption of v2 messages.

Implementation:

• Make protocol version configurable [PR]
• Test in stage via Django setting [PR 1, PR 2]
• Roll out to edge, prod via Django setting [edge, prod]
• Change default to 2 [PR]
• Remove overrides from all envs [stage, edge, prod]
• Document in Open edX release notes: https://openedx.atlassian.net/wiki/spaces/COMM/pages/4570710024/Next+Release+Teak+-+Operator+Dev+Notes
• Inform owners of other relevant services that they should be able to safely upgrade
  • Titans: https://github.com/openedx/enterprise-catalog/blob/b0b2832046c6f83f0a4dee414f845286811aaf78/enterprise_catalog/celery.py#L11
  • Cosmonauts: https://github.com/edx/registrar/blob/51751e6e56b5b86b58ddfb2e2c849577c8175e3e/registrar/celery.py#L19

Notes:

• This can be identified by looking for APP.conf.task_protocol = 1 in an IDA's celery config file [edxapp example, added during celery 4 upgrade]
• https://docs.celeryq.dev/en/stable/history/whatsnew-4.0.html doesn't mention anything about a stop-the-world aspect to the upgrade (just needing to get everything on 3.1.25 first) -- so this might be an easy rollout with no blue/green compatibility issues.
  • Asking for confirmation here: Migration path from task protocol version 1 to 2 celery/celery#9400

[robrap]

@timmc-edx: Are you also planning on opening a github issue against celery if you think they are missing test coverage, and potentially have a bug around this?

[timmc-edx]

Unlikely. Too much of a rabbit hole to go down right now, unfortunately. (It's not even necessarily celery -- could be some other library that interacts with it.)

[robrap]

[inform] This work was scheduled because even though the issue will hopefully first have been fixed by DD in another way, this change will remove technical debt that would have avoided this situation altogether, and may help avoid future related situations.