Datadog instrumentation breaks pymongo with ddtrace<2.7.9

[timmc-edx]

A/C:

• Upgrade to at least ddtrace 2.7.9 or 2.8.2
• Remove DD_TRACE_PYMONGO_ENABLED=false from configuration

Important note: Follow roll-out of proving out on Stage, then moving to Edge, then moving to Prod.

When we tried instrumenting edxapp with Datadog APM (via ddtrace-run) it caused some pymongo calls to break. This was with ddtrace-run 2.7.3 and pymongo 3.13.0. Using DD_TRACE_PYMONGO_ENABLED=false prevents the errors.

Versions 2.7.9 and 2.8.2 contain a fix; I've verified the fix in 2.7.9 in devstack.

Stack trace:

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/trace_utils.py", line 334, in wrapper
    return func(mod, pin, wrapped, instance, args, kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/django/patch.py", line 301, in wrapped
    return func(*args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/django/utils/deprecation.py", line 133, in __call__
    response = self.process_request(request)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/trace_utils.py", line 334, in wrapper
    return func(mod, pin, wrapped, instance, args, kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/django/patch.py", line 301, in wrapped
    return func(*args, **kwargs)
  File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/contentserver/middleware.py", line 84, in process_request
    content = self.load_asset_from_location(loc)
  File "/edx/app/edxapp/edx-platform/openedx/core/djangoapps/contentserver/middleware.py", line 292, in load_asset_from_location
    content = AssetManager.find(location, as_stream=True)
  File "/edx/app/edxapp/edx-platform/xmodule/assetstore/assetmgr.py", line 33, in find
    return contentstore().find(asset_key, throw_on_not_found, as_stream)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/mongodb_proxy.py", line 53, in wrapper
    return func(*args, **kwargs)
  File "/edx/app/edxapp/edx-platform/xmodule/contentstore/mongo.py", line 134, in find
    fp = self.fs.get(content_id)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/gridfs/__init__.py", line 153, in get
    gout._ensure_file()
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/gridfs/grid_file.py", line 482, in _ensure_file
    self._file = self.__files.find_one({"_id": self.__file_id}, session=self._session)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/collection.py", line 1491, in find_one
    for result in cursor.limit(-1):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/cursor.py", line 1280, in next
    if len(self.__data) or self._refresh():
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/cursor.py", line 1195, in _refresh
    self.__send_message(q)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/cursor.py", line 1078, in __send_message
    response = client._run_operation(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1515, in _run_operation
    return self._retryable_read(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1617, in _retryable_read
    return func(session, server, sock_info, secondary_ok)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1511, in _cmd
    return server.run_operation(
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/ddtrace/contrib/pymongo/client.py", line 172, in run_operation
    result = self.__wrapped__.run_operation(sock_info, operation, *args, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/server.py", line 94, in run_operation
    use_cmd = operation.use_command(sock_info)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/message.py", line 341, in use_command
    sock_info.validate_session(self.client, self.session)
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/pymongo/pool.py", line 892, in validate_session
    raise InvalidOperation(
pymongo.errors.InvalidOperation: Can only use session with the MongoClient that started it

Private support ticket: https://help.datadoghq.com/hc/en-us/requests/1600528

[jristau1984]

@timmc-edx is this ticket blocked until the DD side investigation has returned some findings?

[timmc-edx]

Yes, and we can leave it either in No Status or in Blocked.

[jristau1984]

Ticket scope has been updated, so this ticket is now Ready to be picked up.

[timmc-edx]

Completed in edx/configuration#59