feat(tracer): add Python 3.11 support #4125

Yun-Kim · 2022-08-22T21:23:34Z

Description

Python 3.11 has made some new changes to their C internal API that we need to address in a couple of places, mostly in our profiler. I've moved the profiler changes into #4343 and changed this PR to focus on the tracer and integrations supporting Python 3.11. Note that some of the profiler changes were kept in this PR to ensure that the profiler can compile with Python 3.11 per our setup/install rules.

The major tracer-related fix is:

Python 3.11 moved _PyFloat_Pack8() into their internal C API and replaced it with PyFloat_Pack8() to their public C API (Link to CPython issue). We use it once in ddtrace.internal.pack_template.h where I've added a Python version check to use the correct corresponding function.

Note: grpcio with pytest-asyncio is causing Python segmentation faults on the test_unary_exception and test_unary_cancellation test cases, and from the traceback it doesn't look like dd-trace-py is involved. You can reproduce this using this gist which does not involve dd-trace-py. As a result I am skipping the grpcio_asyncio tests with Python 3.11.

Checklist

Title must conform to conventional commit.
Add additional sections for feat and fix pull requests.
Ensure tests are passing for affected code.
Library documentation and/or Datadog's documentation site is updated. Link to doc PR in description.

Reviewer Checklist

…rarily commented out profiler compilation

ddtrace/internal/pack_template.h

docker/.python-version

setup.py

…regarding python version alphas

ddtrace/internal/pack_template.h

releasenotes/notes/python311-a7077c0e461622d2.yaml

P403n1x87 · 2022-09-16T13:18:38Z

Just a note that the bytecode module doesn't support Python 3.11 yet, so the debugger and internal test suites won't pass.

… depend on bytecode

Yun-Kim · 2022-09-21T17:20:25Z

Just a note that the bytecode module doesn't support Python 3.11 yet, so the debugger and internal test suites won't pass.

As discussed, the bytecode library currently does not support Python 3.11, and is used by the internal, debugger, graphene, graphql components/integrations. Until then, I'll exclude those test suites from running Python 3.11.

Yun-Kim · 2022-10-26T19:03:03Z

@Mergifyio backport 1.6

mergify · 2022-10-26T19:03:08Z

backport 1.6

✅ Backports have been created

#4387 feat(tracer): add Python 3.11 support (backport #4125) has been created for branch 1.6

Python 3.11 has made some new changes to their C internal API that we need to address in a couple of places, mostly in our profiler. I've moved the profiler changes into #4343 and changed this PR to focus on the tracer and integrations supporting Python 3.11. Note that some of the profiler changes were kept in this PR to ensure that the profiler can compile with Python 3.11 per our setup/install rules. The major tracer-related fix is: Python 3.11 moved _PyFloat_Pack8() into their internal C API and replaced it with PyFloat_Pack8() to their public C API (Link to CPython issue). We use it once in ddtrace.internal.pack_template.h where I've added a Python version check to use the correct corresponding function. Note: grpcio with pytest-asyncio is causing Python segmentation faults on the test_unary_exception and test_unary_cancellation test cases, and from the traceback it doesn't look like dd-trace-py is involved. You can reproduce this using this gist which does not involve dd-trace-py. As a result I am skipping the grpcio_asyncio tests with Python 3.11. Co-authored-by: Kyle Verhoog <[email protected]> (cherry picked from commit c94cc14) # Conflicts: # tox.ini

This is an automatic backport of pull request #4125 done by [Mergify](https://mergify.com). <details> <summary>Mergify commands and options</summary> <br /> More conditions and actions can be found in the [documentation](https://docs.mergify.com/). You can also trigger Mergify actions by commenting on this pull request: - `@Mergifyio refresh` will re-evaluate the rules - `@Mergifyio rebase` will rebase this PR on its base branch - `@Mergifyio update` will merge the base branch into this PR - `@Mergifyio backport <destination>` will backport this PR on `<destination>` branch Additionally, on Mergify [dashboard](https://dashboard.mergify.com/) you can: - look at your merge queues - generate the Mergify configuration with the config editor. Finally, you can contact us on https://mergify.com </details>

## Description Blocked by #4125. This PR removes the skipping of building Python 3.11 wheels in our build_deploy job. ## Checklist - [x] Title must conform to [conventional commit](https://github.com/conventional-changelog/commitlint/tree/master/%40commitlint/config-conventional). - [x] Add additional sections for `feat` and `fix` pull requests. - [x] Ensure tests are passing for affected code. - [x] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description. ## Motivation ## Design ## Testing strategy ## Relevant issue(s) ## Testing strategy ## Reviewer Checklist - [x] Title is accurate. - [x] Description motivates each change. - [x] No unnecessary changes were introduced in this PR. - [x] PR cannot be broken up into smaller PRs. - [x] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [x] Tests provided or description of manual testing performed is included in the code or PR. - [x] Release note has been added for fixes and features, or else `changelog/no-changelog` label added. - [x] All relevant GitHub issues are correctly linked. - [x] Backports are identified and tagged with Mergifyio. - [x] Add to milestone.

## Description Blocked by #4125. This PR removes the skipping of building Python 3.11 wheels in our build_deploy job. ## Checklist - [x] Title must conform to [conventional commit](https://github.com/conventional-changelog/commitlint/tree/master/%40commitlint/config-conventional). - [x] Add additional sections for `feat` and `fix` pull requests. - [x] Ensure tests are passing for affected code. - [x] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description. ## Motivation ## Design ## Testing strategy ## Relevant issue(s) ## Testing strategy ## Reviewer Checklist - [x] Title is accurate. - [x] Description motivates each change. - [x] No unnecessary changes were introduced in this PR. - [x] PR cannot be broken up into smaller PRs. - [x] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [x] Tests provided or description of manual testing performed is included in the code or PR. - [x] Release note has been added for fixes and features, or else `changelog/no-changelog` label added. - [x] All relevant GitHub issues are correctly linked. - [x] Backports are identified and tagged with Mergifyio. - [x] Add to milestone. (cherry picked from commit 62852e3)

…4448) From #4125, two sections in the stack collector were modified to reflect Python 3.11 changes. However, those changes are not the exact same logic as the former code, including version gating and exception representation. I'm specifically applying these changes only if Python 3.11 or higher are being run. Otherwise, it is being reverted to what it was prior to #4125. While I'm unsure what is exactly causing the issue, we saw an issue (only when the profiler was activated) when running dogweb CI where tests involving sqlalchemy failed on setup with sessions becoming immediately inactive before any queries were run. Furthermore, we saw failures in asyncpg framework tests and major memory/CPU usage spikes when #4125 was merged into 1.x, both of which went away with this fix.

…4448) From #4125, two sections in the stack collector were modified to reflect Python 3.11 changes. However, those changes are not the exact same logic as the former code, including version gating and exception representation. I'm specifically applying these changes only if Python 3.11 or higher are being run. Otherwise, it is being reverted to what it was prior to #4125. While I'm unsure what is exactly causing the issue, we saw an issue (only when the profiler was activated) when running dogweb CI where tests involving sqlalchemy failed on setup with sessions becoming immediately inactive before any queries were run. Furthermore, we saw failures in asyncpg framework tests and major memory/CPU usage spikes when #4125 was merged into 1.x, both of which went away with this fix. (cherry picked from commit 6bc6ff1)

…4448) (#4452) From #4125, two sections in the stack collector were modified to reflect Python 3.11 changes. However, those changes are not the exact same logic as the former code, including version gating and exception representation. I'm specifically applying these changes only if Python 3.11 or higher are being run. Otherwise, it is being reverted to what it was prior to #4125. While I'm unsure what is exactly causing the issue, we saw an issue (only when the profiler was activated) when running dogweb CI where tests involving sqlalchemy failed on setup with sessions becoming immediately inactive before any queries were run. Furthermore, we saw failures in asyncpg framework tests and major memory/CPU usage spikes when #4125 was merged into 1.x, both of which went away with this fix. (cherry picked from commit 6bc6ff1) Co-authored-by: Yun Kim <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Kyle Verhoog <[email protected]>

…ect in python 3.11 (#4901) ## Description #4125 added the use of a getter function `PyThread_GetFrame()` to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, `PyThread_GetFrame()` returns a strong reference to a `PyFrameObject` and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in #4899. ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Backports are identified and tagged with Mergifyio. Co-authored-by: Brett Langdon <[email protected]>

…ect in python 3.11 (DataDog#4901) ## Description DataDog#4125 added the use of a getter function `PyThread_GetFrame()` to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, `PyThread_GetFrame()` returns a strong reference to a `PyFrameObject` and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in DataDog#4899. ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Backports are identified and tagged with Mergifyio. Co-authored-by: Brett Langdon <[email protected]>

…ect in python 3.11 (#4901) ## Description #4125 added the use of a getter function `PyThread_GetFrame()` to provide compatibility for Python 3.11 in the profiler (specifically the stack collector). However, `PyThread_GetFrame()` returns a strong reference to a `PyFrameObject` and its reference count was not decremented in that change. This means that the frame object will not be garbage collected and likely the cause behind the memory leak mentioned in #4899. ## Checklist - [ ] Followed the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines) when writing a release note. - [ ] Add additional sections for `feat` and `fix` pull requests. - [ ] [Library documentation](https://github.com/DataDog/dd-trace-py/tree/1.x/docs) and/or [Datadog's documentation site](https://github.com/DataDog/documentation/) is updated. Link to doc PR in description.   ## Motivation  ## Design  ## Testing strategy   ## Relevant issue(s)  ## Testing strategy   ## Reviewer Checklist - [ ] Title is accurate. - [ ] Description motivates each change. - [ ] No unnecessary changes were introduced in this PR. - [ ] Avoid breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes unless absolutely necessary. - [ ] Tests provided or description of manual testing performed is included in the code or PR. - [ ] Release note has been added and follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/contributing.html#Release-Note-Guidelines), or else `changelog/no-changelog` label added. - [ ] All relevant GitHub issues are correctly linked. - [ ] Backports are identified and tagged with Mergifyio. Co-authored-by: Brett Langdon <[email protected]>

Added Py 3.11 to Riot, added version check for PyFloat_Pack8(), tempo…

9070cdf

…rarily commented out profiler compilation

brettlangdon reviewed Aug 22, 2022

View reviewed changes

ddtrace/internal/pack_template.h Outdated Show resolved Hide resolved

docker/.python-version Outdated Show resolved Hide resolved

setup.py Outdated Show resolved Hide resolved

Reverted 3.11 version pin in dockerfile, removed unnecessary comment …

1c29dff

…regarding python version alphas

Yun-Kim force-pushed the yun-kim/py311 branch from fc80eab to 1c29dff Compare August 23, 2022 20:23

Yun-Kim and others added 2 commits August 23, 2022 16:28

Reverse condition

f1c0af5

Merge branch '1.x' into yun-kim/py311

20c705a

Yun-Kim changed the title ~~Support Python 3.11~~ feat: Support Python 3.11 Aug 24, 2022

Made collector/stack.pyx 3.11 compatible

d96a4c9

Yun-Kim changed the title ~~feat: Support Python 3.11~~ feat: support Python 3.11 Aug 25, 2022

Uncommented Appsec setup

77e4d17

Yun-Kim force-pushed the yun-kim/py311 branch from 4531bec to 77e4d17 Compare August 25, 2022 19:13

Merge branch '1.x' into yun-kim/py311

508d5c6

P403n1x87 reviewed Aug 26, 2022

View reviewed changes

ddtrace/internal/pack_template.h Outdated Show resolved Hide resolved

P403n1x87 reviewed Aug 26, 2022

View reviewed changes

ddtrace/internal/pack_template.h Outdated Show resolved Hide resolved

P403n1x87 reviewed Aug 26, 2022

View reviewed changes

releasenotes/notes/python311-a7077c0e461622d2.yaml Outdated Show resolved Hide resolved

Yun-Kim mentioned this pull request Sep 1, 2022

Support Python 3.11 #4149

Closed

Yun-Kim and others added 2 commits September 15, 2022 12:10

Typos

f8f62c1

Merge branch '1.x' into yun-kim/py311

0fa9bff

Exclude python 3.11 for internal/debugger/graphql/graphene since they…

ac33541

… depend on bytecode

Yun-Kim and others added 9 commits September 21, 2022 13:25

Merge branch '1.x' into yun-kim/py311

2193159

Add none check for exceptions in stack collector

d08e99b

Merge branch '1.x' into yun-kim/py311

c7eddc9

Pinned cherrypy, pynamodb, django, mariadb test suite Python versions

e5a1cba

Pinned python version for psycopg test suite

28708f6

Update grpc test suites for python 3.11

d3ba946

Fixed Starlette test app for Py 3.11

a5d2aa1

Disable aioredis tests for Python 3.11

f7fd78b

Remove gevent as dependency of ddtracerun tests

bf0b7c9

Kyle-Verhoog previously approved these changes Oct 25, 2022

View reviewed changes

Remove unsupported version of pynamodb for python 3.11 test suite

92c2742

Yun-Kim dismissed Kyle-Verhoog’s stale review via 92c2742 October 25, 2022 22:53

Kyle-Verhoog approved these changes Oct 26, 2022

View reviewed changes

majorgreys approved these changes Oct 26, 2022

View reviewed changes

mergify bot added 4 commits October 26, 2022 15:02

Merge branch '1.x' into yun-kim/py311

e491de5

Merge branch '1.x' into yun-kim/py311

e5a926b

Merge branch '1.x' into yun-kim/py311

3caf622

Merge branch '1.x' into yun-kim/py311

7079b38

Kyle-Verhoog merged commit c94cc14 into DataDog:1.x Oct 26, 2022

mergify bot mentioned this pull request Oct 26, 2022

feat(tracer): add Python 3.11 support (backport #4125) #4387

Merged

Yun-Kim deleted the yun-kim/py311 branch October 26, 2022 21:19

Kyle-Verhoog mentioned this pull request Nov 2, 2022

chore(asyncpg): skip incompatible framework test #4442

Closed

10 tasks

Yun-Kim mentioned this pull request Nov 2, 2022

fix(profiler): version gate python 3.11 changes to stack collector #4448

Merged

10 tasks

Yun-Kim mentioned this pull request Jan 13, 2023

fix(profiler): decrement refcount given strong reference of frame object in python 3.11 #4901

Merged

11 tasks

flodolo mentioned this pull request Feb 22, 2024

Update workflows and docker to Python 3.10 mozilla/pontoon#3112

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(tracer): add Python 3.11 support #4125

feat(tracer): add Python 3.11 support #4125

Yun-Kim commented Aug 22, 2022 •

edited by Kyle-Verhoog

Loading

P403n1x87 commented Sep 16, 2022

Yun-Kim commented Sep 21, 2022 •

edited

Loading

Yun-Kim commented Oct 26, 2022

mergify bot commented Oct 26, 2022 •

edited

Loading

feat(tracer): add Python 3.11 support #4125

feat(tracer): add Python 3.11 support #4125

Conversation

Yun-Kim commented Aug 22, 2022 • edited by Kyle-Verhoog Loading

Description

Checklist

Reviewer Checklist

P403n1x87 commented Sep 16, 2022

Yun-Kim commented Sep 21, 2022 • edited Loading

Yun-Kim commented Oct 26, 2022

mergify bot commented Oct 26, 2022 • edited Loading

✅ Backports have been created

Yun-Kim commented Aug 22, 2022 •

edited by Kyle-Verhoog

Loading

Yun-Kim commented Sep 21, 2022 •

edited

Loading

mergify bot commented Oct 26, 2022 •

edited

Loading