[bug] Internal Server Error for url: /obj/create.

[kerriewu-sunrun]

Hello,

I'm using weave to make a call a weave.op() decorated function from AWS Lambda and am running into this error intermittently. If anyone is able to provide a pointer about how to resolve this, let me know.

It seems to be a recent issue as we have been using weave in the same way for a few months, and only in the past couple weeks noticed an increase in these types of errors.

Thanks a ton!

LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] HTTPError: 500 Server Error: Internal Server Error for url: /obj/create. Reason: Internal Server Error
Traceback (most recent call last):
  File "/var/lang/lib/python3.11/site-packages/aws_lambda_powertools/logging/logger.py", line 447, in decorate
    return lambda_handler(event, context, *args, **kwargs)
  File "/var/lang/lib/python3.11/site-packages/aws_lambda_powertools/metrics/provider/base.py", line 205, in decorate
    response = lambda_handler(event, context, *args, **kwargs)
  File "/var/task/app.py", line 189, in lambda_handler
    return app.resolve(event, context)
  File "/var/lang/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/api_gateway.py", line 1918, in resolve
    response = self._resolve().build(self.current_event, self._cors)
  File "/var/lang/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/api_gateway.py", line 2025, in _resolve
    return self._call_route(route, route_keys)  # pass fn args
  File "/var/lang/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/api_gateway.py", line 2103, in _call_route
    route(router_middlewares=self._router_middlewares, app=self, route_arguments=route_arguments),
  File "/var/lang/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/api_gateway.py", line 407, in __call__
    return self._middleware_stack(app)
  File "/var/lang/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/api_gateway.py", line 1314, in __call__
    return self.current_middleware(app, self.next_middleware)
  File "/var/lang/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/api_gateway.py", line 1346, in _registered_api_adapter
    return app._to_response(next_middleware(**route_args))
  File "/var/task/app.py", line 69, in llm
    responses = get_completions(
  File "/var/task/app.py", line 120, in get_completions
    responses = loop.run_until_complete(
  File "/var/lang/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
  File "/var/lang/lib/python3.11/site-packages/weave/trace/op.py", line 347, in wrapper
    call = _create_call(wrapper, *args, **kwargs)  # type: ignore
  File "/var/lang/lib/python3.11/site-packages/weave/trace/op.py", line 174, in _create_call
    return client.create_call(
  File "/var/lang/lib/python3.11/site-packages/weave/trace/trace_sentry.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/var/lang/lib/python3.11/site-packages/weave/trace/weave_client.py", line 525, in create_call
    op_def_ref = self._save_op(unbound_op)
  File "/var/lang/lib/python3.11/site-packages/weave/trace/weave_client.py", line 908, in _save_op
    return self._save_and_attach_ref(op, name)
  File "/var/lang/lib/python3.11/site-packages/weave/trace/weave_client.py", line 914, in _save_and_attach_ref
    op_def_ref = self._save_object_basic(op, name)
  File "/var/lang/lib/python3.11/site-packages/weave/trace/weave_client.py", line 802, in _save_object_basic
    response = self.server.obj_create(
  File "/var/lang/lib/python3.11/site-packages/weave/trace_server_bindings/remote_http_trace_server.py", line 349, in obj_create
    return self._generic_request(
  File "/var/lang/lib/python3.11/site-packages/weave/trace_server_bindings/remote_http_trace_server.py", line 220, in _generic_request
    r = self._generic_request_executor(url, req)
  File "/var/lang/lib/python3.11/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
  File "/var/lang/lib/python3.11/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
  File "/var/lang/lib/python3.11/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/var/lang/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
  File "/var/lang/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
  File "/var/lang/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/var/lang/lib/python3.11/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
  File "/var/lang/lib/python3.11/site-packages/weave/trace_server_bindings/remote_http_trace_server.py", line 203, in _generic_request_executor
    raise requests.HTTPError(

``

[jwlee64]

Hi @kerriewu-sunrun, do you have any code snippets/mocks that would help us debug this issue?