Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(tracer): add Python 3.11 support #4125

Merged
merged 79 commits into from
Oct 26, 2022
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
9070cdf
Added Py 3.11 to Riot, added version check for PyFloat_Pack8(), tempo…
Yun-Kim Aug 22, 2022
1c29dff
Reverted 3.11 version pin in dockerfile, removed unnecessary comment …
Yun-Kim Aug 23, 2022
f1c0af5
Reverse condition
Yun-Kim Aug 23, 2022
20c705a
Merge branch '1.x' into yun-kim/py311
Yun-Kim Aug 23, 2022
d96a4c9
Made collector/stack.pyx 3.11 compatible
Yun-Kim Aug 25, 2022
77e4d17
Uncommented Appsec setup
Yun-Kim Aug 25, 2022
508d5c6
Merge branch '1.x' into yun-kim/py311
Yun-Kim Aug 25, 2022
f8f62c1
Typos
Yun-Kim Sep 15, 2022
0fa9bff
Merge branch '1.x' into yun-kim/py311
Yun-Kim Sep 15, 2022
ac33541
Exclude python 3.11 for internal/debugger/graphql/graphene since they…
Yun-Kim Sep 20, 2022
2193159
Merge branch '1.x' into yun-kim/py311
Yun-Kim Sep 21, 2022
d08e99b
Add none check for exceptions in stack collector
Yun-Kim Sep 22, 2022
c7eddc9
Merge branch '1.x' into yun-kim/py311
Yun-Kim Sep 22, 2022
e5a1cba
Pinned cherrypy, pynamodb, django, mariadb test suite Python versions
Yun-Kim Sep 23, 2022
28708f6
Pinned python version for psycopg test suite
Yun-Kim Sep 23, 2022
d3ba946
Update grpc test suites for python 3.11
Yun-Kim Sep 23, 2022
a5d2aa1
Fixed Starlette test app for Py 3.11
Yun-Kim Sep 23, 2022
f7fd78b
Disable aioredis tests for Python 3.11
Yun-Kim Sep 23, 2022
bf0b7c9
Remove gevent as dependency of ddtracerun tests
Yun-Kim Sep 26, 2022
086d157
Separate ddtracerun test suite into pre-3.11 and post-3.11, skip geve…
Yun-Kim Sep 26, 2022
de7fa2e
Added py 3.11 test suites into tox
Yun-Kim Sep 26, 2022
aad8d6f
Pinned dogpile version in tox, reformatted riot testsuites for grpc
Yun-Kim Sep 26, 2022
fcd1311
Merge branch '1.x' into yun-kim/py311
Yun-Kim Sep 26, 2022
a4f1f1b
Merge branch '1.x' into yun-kim/py311
Yun-Kim Sep 27, 2022
5a6d9b7
Merge branch '1.x' into yun-kim/py311
Yun-Kim Sep 27, 2022
cdd1ccf
Added dogpile==1.1 to tox
Yun-Kim Sep 27, 2022
d8a6b88
Merge branch '1.x' into yun-kim/py311
Yun-Kim Sep 28, 2022
e2f0727
Unpin debugger, pin asyncpg
Yun-Kim Sep 28, 2022
a752988
Pin vine dependency for kombu python 3.11 tests, repin debugger tests
Yun-Kim Sep 29, 2022
1d0993f
Upgraded kombu tests to run Python 3.11 with kombu >= 5.0
Yun-Kim Sep 29, 2022
d5460ce
Unpin hypothesis
Yun-Kim Sep 30, 2022
146db8e
Pin hypothesis to 6.55
Yun-Kim Sep 30, 2022
8a1f245
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 3, 2022
4e0b8ff
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 3, 2022
02a25d9
Unpin hypothesis
Yun-Kim Oct 3, 2022
9bb5161
Separate test_http.py tracer test suite from riot to tox
Yun-Kim Oct 4, 2022
13c826d
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 4, 2022
253f1d9
Add tracer test_http test suite into circleci config
Yun-Kim Oct 4, 2022
67bde40
Add pys for test_http test suite in tox
Yun-Kim Oct 4, 2022
fb52184
Remove closing regex for circleci tracer test_http job
Yun-Kim Oct 4, 2022
6d7c004
Try different pattern
Yun-Kim Oct 4, 2022
6219277
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 5, 2022
8f7e7cb
Merge riot tracer test suites
Yun-Kim Oct 5, 2022
a1e1e79
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 6, 2022
b69ddaa
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 14, 2022
4d81ce7
Remove celery tests for python 3.11 until billiard dependency compatible
Yun-Kim Oct 14, 2022
3e39392
Replace removed inspect function with py 3.11 compatible function
Yun-Kim Oct 14, 2022
3a5b3ee
Updated starlette and fasapi test suites to use python 3.11 compatibl…
Yun-Kim Oct 17, 2022
bc5b544
Remove gevent tests with Python 3.11
Yun-Kim Oct 17, 2022
39450f6
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 18, 2022
91b96d4
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 18, 2022
0693351
Replace comment in riotfile
Yun-Kim Oct 18, 2022
c70c42b
Revert profiler changes
Yun-Kim Oct 20, 2022
f41a26a
Re-add profiler changes to allow profiler to compile in py311
Yun-Kim Oct 20, 2022
c806dbe
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 20, 2022
b9b8fa7
Remove grpc_aio tests with 3.11
Yun-Kim Oct 20, 2022
72d99e8
Address comments
Yun-Kim Oct 20, 2022
45bfb7d
Skip bytecode related tests
Kyle-Verhoog Oct 21, 2022
45bd4c3
Skip py36 wrapping tests as well
Kyle-Verhoog Oct 21, 2022
aa55193
Skip internal gevent test
Kyle-Verhoog Oct 21, 2022
f41db8f
Merge branch '1.x' into yun-kim/py311
Kyle-Verhoog Oct 21, 2022
c167e49
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 24, 2022
d20a8bf
Address comments, update changelog
Yun-Kim Oct 24, 2022
33fbdfb
Revert removing py27 from black target_version, capitalize changelog …
Yun-Kim Oct 24, 2022
521695b
Remove 3.10 from opentracer gevent tests tox
Yun-Kim Oct 24, 2022
a291b93
Disable profiler and dynamic instrumentation if running with Python 3.11
Yun-Kim Oct 24, 2022
3187689
Remove gevent skips as gevent 22.8 is compatible with Python 3.11
Yun-Kim Oct 25, 2022
922dedf
Update changelog, add whitespace to comment
Yun-Kim Oct 25, 2022
de9d63f
Pin grpcio to 1.49 for Python 3.11
Yun-Kim Oct 25, 2022
e4649c0
Fix skip in grpc_aio with Python 3.11
Yun-Kim Oct 25, 2022
35d0318
Merge branch '1.x' into yun-kim/py311
Kyle-Verhoog Oct 25, 2022
244da69
Merge branch '1.x' into yun-kim/py311
Kyle-Verhoog Oct 25, 2022
49f1a2e
Skip additional segfaulting grpc_aio tests
Kyle-Verhoog Oct 25, 2022
a848af6
Merge branch '1.x' into yun-kim/py311
Yun-Kim Oct 25, 2022
92c2742
Remove unsupported version of pynamodb for python 3.11 test suite
Yun-Kim Oct 25, 2022
e491de5
Merge branch '1.x' into yun-kim/py311
mergify[bot] Oct 26, 2022
e5a926b
Merge branch '1.x' into yun-kim/py311
mergify[bot] Oct 26, 2022
3caf622
Merge branch '1.x' into yun-kim/py311
mergify[bot] Oct 26, 2022
7079b38
Merge branch '1.x' into yun-kim/py311
mergify[bot] Oct 26, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion ddtrace/internal/pack_template.h
Original file line number Diff line number Diff line change
Expand Up @@ -558,7 +558,12 @@ static inline int msgpack_pack_double(msgpack_packer* x, double d)
{
unsigned char buf[9];
buf[0] = 0xcb;
_PyFloat_Pack8(d, &buf[1], 0);
// Python 3.11 introduced PyFloat_Pack8() to the public C API and moved PyFloat_Pack8() to the internal C API
Yun-Kim marked this conversation as resolved.
Show resolved Hide resolved
#if PY_VERSION_HEX <= 0x030B00A1
Yun-Kim marked this conversation as resolved.
Show resolved Hide resolved
_PyFloat_Pack8(d, &buf[1], 0);
#else
PyFloat_Pack8(d, &buf[1], 0);
#endif
msgpack_pack_append_buffer(x, buf, 9);
}

Expand Down
28 changes: 21 additions & 7 deletions ddtrace/profiling/collector/stack.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,14 @@ IF UNAME_SYSNAME != "Windows" and PY_MAJOR_VERSION >= 3 and PY_MINOR_VERSION >=
PyObject* frame

_PyErr_StackItem * _PyErr_GetTopmostException(PyThreadState *tstate)

ctypedef struct _PyErr_StackItem:
PyObject* exc_type
PyObject* exc_value
PyObject* exc_traceback

PyObject* PyException_GetTraceback(PyObject* exc)
PyObject* Py_TYPE(PyObject* ob)

IF PY_MINOR_VERSION == 7:
# Python 3.7
cdef extern from "<internal/pystate.h>":
Expand Down Expand Up @@ -200,6 +202,8 @@ IF UNAME_SYSNAME != "Windows" and PY_MAJOR_VERSION >= 3 and PY_MINOR_VERSION >=
# Needed for accessing _PyGC_FINALIZED when we build with -DPy_BUILD_CORE
cdef extern from "<internal/pycore_gc.h>":
pass
cdef extern from "<Python.h>":
PyObject* PyThreadState_GetFrame(PyThreadState* tstate)
ELSE:
from cpython.ref cimport Py_DECREF

Expand All @@ -216,7 +220,8 @@ cdef collect_threads(thread_id_ignore_list, thread_time, thread_span_links) with
cdef PyThreadState* tstate
cdef _PyErr_StackItem* exc_info
cdef PyThread_type_lock lmutex = _PyRuntime.interpreters.mutex

cdef PyObject* exc_type
cdef PyObject* exc_tb
cdef dict running_threads = {}

# This is an internal lock but we do need it.
Expand All @@ -232,12 +237,21 @@ cdef collect_threads(thread_id_ignore_list, thread_time, thread_span_links) with
tstate = PyInterpreterState_ThreadHead(interp)
while tstate:
# The frame can be NULL
if tstate.frame:
running_threads[tstate.thread_id] = <object>tstate.frame

# Python 3.9 added helper function to public C API to access PyFrameObject from tstate,
# Python 3.11 moved PyFrameObject to internal C API and cannot be directly accessed from tstate
IF PY_MINOR_VERSION >= 9:
Kyle-Verhoog marked this conversation as resolved.
Show resolved Hide resolved
frame = PyThreadState_GetFrame(tstate)
ELSE:
frame = tstate.frame
if frame:
running_threads[tstate.thread_id] = <object>frame
exc_info = _PyErr_GetTopmostException(tstate)
if exc_info and exc_info.exc_type and exc_info.exc_traceback:
current_exceptions[tstate.thread_id] = (<object>exc_info.exc_type, <object>exc_info.exc_traceback)
# Python 3.11 removed exc_type, exc_traceback from exception representations, can instead
# derive exc_type and exc_traceback from remaining exc_value field
exc_type = Py_TYPE(exc_info.exc_value)
exc_tb = PyException_GetTraceback(exc_info.exc_value)
if exc_info.exc_value and exc_type and exc_tb:
current_exceptions[tstate.thread_id] = (<object>exc_type, <object>exc_tb)

tstate = PyThreadState_Next(tstate)

Expand Down
4 changes: 4 additions & 0 deletions releasenotes/notes/python311-a7077c0e461622d2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
features:
- |
Added official support for Python 3.11
Yun-Kim marked this conversation as resolved.
Show resolved Hide resolved
15 changes: 12 additions & 3 deletions riotfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,16 @@
from riot import latest


SUPPORTED_PYTHON_VERSIONS = [(2, 7), (3, 5), (3, 6), (3, 7), (3, 8), (3, 9), (3, 10)] # type: List[Tuple[int, int]]
SUPPORTED_PYTHON_VERSIONS = [
(2, 7),
(3, 5),
(3, 6),
(3, 7),
(3, 8),
(3, 9),
(3, 10),
(3, 11),
] # type: List[Tuple[int, int]]


def version_to_str(version):
Expand Down Expand Up @@ -58,9 +67,9 @@ def select_pys(min_version=MIN_PYTHON_VERSION, max_version=MAX_PYTHON_VERSION):
"""Helper to select python versions from the list of versions we support

>>> select_pys()
['2.7', '3.5', '3.6', '3.7', '3.8', '3.9', '3.10']
['2.7', '3.5', '3.6', '3.7', '3.8', '3.9', '3.10', '3.11']
>>> select_pys(min_version='3')
['3.5', '3.6', '3.7', '3.8', '3.9', '3.10']
['3.5', '3.6', '3.7', '3.8', '3.9', '3.10', '3.11']
>>> select_pys(max_version='3')
['2.7']
>>> select_pys(min_version='3.5', max_version='3.8')
Expand Down
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,7 @@ def get_exts_for(name):
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
],
use_scm_version={"write_to": "ddtrace/_version.py"},
setup_requires=["setuptools_scm[toml]>=4,<6.1", "cython", "cmake", "ninja"],
Expand Down