Bug: Hardcoded GRPC limits in store gateway client prevents data from quering

[moskitone]

What is the bug?

When trying to query data with a large time range, one can face the GRPC error in the mimir-quirier.
"rpc error: code = ResourceExhausted desc = grpc: received message larger than max (133997632 vs. 104857600)"

Increasing grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size does nothing because it is hardcoded in the
https://github.com/grafana/mimir/blob/main/pkg/querier/store_gateway_client.go#L80

How to reproduce it?

It fails only on a limited set of queries, and it is not easy to reproduce without having the same metrics data stored in storage.

What did you think would happen?

The data size reached hardcoded limits.

What was your environment?

Kubernetes, mimir-distributed helm chart, GCS bucket.

Any additional context to share?