feat(celery): add support for Python 3.11, 3.12

[Yun-Kim]

Officially adds support for Python 3.11 and 3.12 for the Celery integration.

Note: looks like an issue with task exceptions that didn't translate well to Python 3.11. We'll have to investigate this first.

• It looks like celery>=5.3 uses a newer version of billiard>=4.1.0 and also changed how billiard.ExceptionInfo gets constructed in celery's handle_failures callback. Note we were previously using billiard==3.6.4 and celery==5.2.7.
• This is now changing the value of the exception einfo we check for in our traced trace_failure callback, which makes things weird when we try to ignore some Exceptions in our integration tests for celery.

I've verified this hypothesis (that the new release of celery>=5.3.0 changed how they construct ExceptionInfo in handle_failures() breaks our integration tests) by updating the Python 3.10 test suite to use celery==5.3.1, and this also results in the same failures. We'll need to make a bigger effort to make our integration compatible with the latest celery>=5.3.0.

Checklist

• Change(s) are motivated and described in the PR description.
• Testing strategy is described if automated tests are not included in the PR.
• Risk is outlined (performance impact, potential for breakage, maintainability, etc).
• Change is maintainable (easy to change, telemetry, documentation).
• Library release note guidelines are followed. If no release note is required, add label changelog/no-changelog.
• Documentation is included (in-code, generated user docs, public corp docs).
• Backport labels are set (if applicable)

Reviewer Checklist

• Title is accurate.
• No unnecessary changes are introduced.
• Description motivates each change.
• Avoids breaking API changes unless absolutely necessary.
• Testing strategy adequately addresses listed risk(s).
• Change is maintainable (easy to change, telemetry, documentation).
• Release note makes sense to a user of the library.
• Reviewer has explicitly acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment.
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[emmettbutler]

Looks like some of the newly-added tests are failing.

[pr-commenter]

Benchmarks

Benchmark execution time: 2023-11-10 18:02:51

Comparing candidate commit a9c34e6 in PR branch yunkim/celery-3-11 with baseline commit 1a6dc42 in branch 2.x.

Found 1 performance improvements and 2 performance regressions! Performance is the same for 87 metrics, 0 unstable metrics.

scenario:flasksimple-appsec-get

• 🟥 max_rss_usage [+740.617KB; +1007.146KB] or [+2.109%; +2.868%]

scenario:flasksimple-appsec-post

• 🟥 max_rss_usage [+748.834KB; +967.800KB] or [+2.137%; +2.761%]

scenario:span-add-metrics

• 🟩 max_rss_usage [-17.300MB; -17.153MB] or [-29.082%; -28.835%]

[mabdinur]

@Yun-Kim, is there a feature request for supporting celery for python3.11? Do we need to prioritize this work?

[Yun-Kim]

@mabdinur we got an indirect FR in this issue: #4149 (comment)

There are some tasks with higher priority that I'm bogged down by, but I think I've written enough in the PR description for anyone else to take this over in the meantime.

[kochb]

Should I understand that there is a formal feature request procedure that would help with resuming work on this? We're a datadog customer using Celery under Python 3.11, and we're currently encountering the ‘_Code’ object has no attribute ‘co_positions’ error mentioned in #4149.

[Yun-Kim]

Hi @kochb, thanks for reaching out!

Should I understand that there is a formal feature request procedure that would help with resuming work on this?

Yes, making a feature request via a support ticket would be the fastest way to resume work on this. Feel free to create an issue on this repository as well!

[Yun-Kim]

Looks like tests are passing for 3.11 and 3.12. Thanks @mabdinur for taking this over to the finish line! 🙇

[kochb]

Thank you for your work on this!