Display full message in GRPC exception log

[Prabha-Veerubhotla]

Feature Request

If this is a feature request, please fill out the following form in full:

Describe the problem the feature is intended to solve

While using Tensorflow serving, the exception message log is TRUNCATED.
Ex:

Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: xxxx...TRUNCATED

Describe the solution

Full logs should be displayed without any truncation.

Additional context

The Tensorflow serving client side config for max inbound message size
.maxInboundMessageSize()
is set to
int32max to match with server side config

serving/tensorflow_serving/model_servers/server.cc

Line 385 in 6a9d0fd

builder.SetMaxMessageSize(tensorflow::kint32max);

The origination code seems to be the following as per stack trace:
https://github.com/grpc/grpc-java/blob/master/stub/src/main/java/io/grpc/stub/ClientCalls.java#L275

System information

• TensorFlow Serving version: 2.11.0.5

Source code / logs

Prediction in Tensorflow serving:
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_service.proto#L23

[singhniraj08]

@Prabha-Veerubhotla,

TF serving binary has C++ dependencies. C++ logging has a hard coded limit on log size 15K which results in this truncation of log messages. A typical workaround of this problem is to output every line as a separate record. This approach can rapidly increase memory usage which can result in crash due to a memory exceeded error.
Please let us know, if this is completely blocking you, we can try looking for alternatives. Thank you!

[Prabha-Veerubhotla]

Thank you for looking into this @singhniraj08. I actually found that this is the reason for the log truncation

serving/tensorflow_serving/model_servers/grpc_status_util.cc

Line 29 in 701e5c0

"...TRUNCATED");

Do we have a command line param to have custom error message limit instead of the default 1024 .

[singhniraj08]

@Prabha-Veerubhotla,

Going through complete list of available command line flags, I couldn't find any flag or param to set custom error message limit. Thanks.

[Prabha-Veerubhotla]

@singhniraj08 how about adding a new command line argument to set the custom error message limit here.

I am currently blocked with this as I cannot look at the complete error message. There are some features that are failing in TF serving.

[singhniraj08]

@Prabha-Veerubhotla,

Let us discuss this feature implementation internally and we will update this thread with updates. Thanks.

[Prabha-Veerubhotla]

thank you @singhniraj08

[Prabha-Veerubhotla]

hi @bmzhao is there an update on this ?

[Prabha-Veerubhotla]

@singhniraj08 were you able to follow up on this ?

[Prabha-Veerubhotla]

Happy to contribute with a pr if there is an agreement. We want this change to be part of 2.11 version.

[Prabha-Veerubhotla]

@singhniraj08 , @bmzhao any update on this issue ?

[ndeepesh]

@singhniraj08 @bmzhao Is there an update to this issue?

[asamadiya]

@ndeepesh @Prabha-Veerubhotla @singhniraj08 @bmzhao
Can't we just do this? #2185

[Prabha-Veerubhotla]

@asamadiya this should work. I am not sure if there will be any concern of printing a large message.
The original PR was part of an internal change from tensor flow team c49fd96