Feature request: allowing to return (pass-through) json literal string to avoid loads()/dumps() or escaping

[Talkless]

Our use case for flask-jsonrpc is to implement API server for PostgreSQL database.

PostgreSQL returns results as JSON objects, but that "forces" psycopg2 to parse it (using json.lodas?) to build Python dict/list.

But our flask-jsonrpc app DOES NOT need to have JSON parsed into Python objects (dict/list), no need to do any work on it, it just has to pass it to the API user (JavaScript webapp). And these JSON objects can be HUGE!

So in naive/straightforward scenario:

1. PostgreSQL returns JSON object (travels as SQL string of course via network).
2. psycopg2 parses it, returns Python object (dict/list). That's expensive work (JSON result objects can be huge).
3. Our view returns that Python object (dict/list).
4. flask-jsonrpc builds final JSON-RPC response dict with id, jsonrcp version and result Python object (dict/list) from our view.
5. JSON-RPC response dict that includes Python object (dict/list) is rendered to string (using json.dumps probably?) to return to API client (browser). That's expensive work (JSON result objects can be huge).
6. API client parses JSON-RPC response object and we get PostgreSQL JSON result inside JavaScript.

To avoid loading/dumping JSON in flask-jsonrpc API server, we decided to return JSON result object from PostgreSQL as string.

Since SQL query's result not "officially" JSON, psycopg2 does not parse it, but now flask-jsonrcp's result JSON has it escaped, it looks like this:

Finally, JavaScript webapp has to parse JSON-RPC response object AND then parse escaped "result" string into JavaScript object again.

This kind of escaping/de-escaping of json-as-string is much better that parsing/dumping JSON object inside Python, but still feels "nonsensical", strange for webdevs, sub-optimal.

I wonder could there be a way to allow to "pass through" JSON-as-string result from PostgreSQL/psycopg2 into flask-jsonrcp response object?

Maybe ability to override rendering of valid (non-error) result could work (that assumes our views always returns JSON string literals) :

@jsonrpc.validresultformatter()
def my_valid_result_formatter(id, version, result):
    return f'{{id: "{id}", jsonrpc: "{version}", result: {result}}}'

Or, maybe some special type annotating that we want to return literal JSON string to be emplaced without escaping into final JSON-RPC response object:

@jsonrpc_client.method('object.trajectory', notification=False)
@jwt_required()
def object__trajectory(
        object_id: int,
        time_from: str,
        time_to: str
) -> Optional[flask_jsonrpc.JsonStringLiteral]: # can return None that should become `result: null` in JSON-RPC response.
    # ...

Or this is would be considered "obscure", "rare", "out-of-scope" optimization"?

[alexted]

• A similar problem is discussed here: Manually Controlling JSONB Serialization/Deserialization sqlalchemy/sqlalchemy#10944. It's very interesting conversation!
• Have you tried using third-party libraries for data de/serialization, such as the awesome msgspec and orjson?

[Talkless]

psycopg has set_json_loads that we could implement to avoid parsing into Python dict/list,

but that still leaves string escaping issue. If you put json-as-string into JSON value, it will be escaped.

Not sure about second point. We don't need to parse PostgreSQL's returned JSON, just pass it to the web browser basically.

It's not question on to have "faster/slower" JSON parser, it's just not needed at at, as it's nonsesical to invoke maybe thousands of allocations in Python to build dict/list from PostgreSQL JSON result, just for later (after split second) to be looped and rendered into string again.

[nycholas]

@Talkless You can define your own JSONRPCView and overwrite the method post and use whatever you can intend of make_response(jsonify(response), status_code, headers), similarly, you can do the same with JSONRPCSite and overwrite the method to_json, so you have full control over the requests and responses.

class JSONRPC(JSONRPCDecoratorMixin):
    def __init__(
        self: Self,
        app: Flask | None = None,
        service_url: str = '/api',
        jsonrpc_site: type[JSONRPCSite] = default_jsonrpc_site,
        jsonrpc_site_api: type[JSONRPCView] = default_jsonrpc_site_api,  # ------------------> Here
        enable_web_browsable_api: bool = False,
    ) -> None:

See here the example of use.

Also, does the capability to custom serializer/deserializer help in that use case? It's very interesting to have on the project, as @alexted mentioned it could be valuable for other scenarios as well.

Thank you.

[Talkless]

So if I understood correctly, we create custom view OptimizedJsonAPI with post methon overridden,
and then just use it like this:

jsonrpc= JSONRPC(app, '/api', enable_web_browsable_api=True, jsonrpc_site_api=OptimizedJsonAPI)

In OptimizedJsonAPI.post() we basically copy-paste original JSONRPCView.post() with lines return make_response(jsonify(response), status_code, headers) changed as we like?

Also, does the capability to custom serializer/deserializer help in that use case?

Not sure what you mean by that exactly, but if lines with return make_response(...) would be changed into return self.make_response(...), we could override ONLY that JSONRPCView.make_response() (leaving original .post() logic intact), making more future-compatible & maintainable I guess.

[Talkless]

Or maybe even better to have JSONRPCView.jsonify() overridable?

[nycholas]

@Talkless

In OptimizedJsonAPI.post() we basically copy-paste original JSONRPCView.post() with lines return make_response(jsonify(response), status_code, headers) changed as we like?

Yes, that's it. The downside here is to copy-and-paste the entire method post, because of that the proposal of customizer the serializer/deserializer is welcome.

Not sure what you mean by that exactly, but if lines with

It is the exact feature that you are asking for, the capability to overwrite the serializer/deserializer (json.loads and jsonify) by configuration. By the way, it is a very good proposal, I will create an issue for that.

Does it make sense to you?

Thank you.

[Talkless]

@nycholas all clear, thanks!

Maybe we can just rename this issue?

[nycholas]

No worries, let's use this issue 💪.

[nycholas]

👋 @Talkless good news, as the project uses the Flask.json.provider.DefaultJSONProvider, you can overwrite the default provider as documented here. Below an example of using that customization.

from flask import Flask
from flask.json.provider import DefaultJSONProvider

from flask_jsonrpc import JSONRPC


class JSONIdentifyProvider(DefaultJSONProvider):
    def loads(self, s, **kwargs):
        return super().loads(s)

    def dumps(self, obj, **kwargs):
        return obj

# Flask application
app = Flask('application')
app.json = JSONIdentifyProvider(app)

# Flask-JSONRPC
jsonrpc = JSONRPC(app, '/api', enable_web_browsable_api=True)


@jsonrpc.method('App.index')
def index() -> str:
    return '{"message": "Welcome to Flask JSON-RPC"}'


if __name__ == '__main__':
    app.run(host='0.0.0.0', debug=True)

Testing it by doing the request:

 curl -i -X POST \
   -H "Content-Type: application/json; indent=4" \
   -d '{
    "jsonrpc": "2.0",
    "method": "App.index",
    "params": {},
    "id": "1"
}' http://localhost:5000/api  
HTTP/1.1 200 OK
Server: Werkzeug/3.0.6 Python/3.13.0
Date: Fri, 01 Nov 2024 17:29:44 GMT
Content-Type: application/json
Content-Length: 84
Connection: close

{'id': '1', 'jsonrpc': '2.0', 'result': '{"message": "Welcome to Flask JSON-RPC"}'}

Let me know if solves your use case, please.

Thank you.

[Talkless]

app.json = JSONIdentifyProvider(app)

Wow, that looks much better solution, thanks!

[Talkless]

Wait, {"message": "Welcome to Flask JSON-RPC"} is inside quotes, i.e. your custom JSON provided did not construct whole response, just process result part, injecting it as string, not as literal JSON segment.

Result needed is:

{'id': '1', 'jsonrpc': '2.0', 'result': {"message": "Welcome to Flask JSON-RPC"}}

instead of:

{'id': '1', 'jsonrpc': '2.0', 'result': '{"message": "Welcome to Flask JSON-RPC"}'

[nycholas]

@Talkless 👋

The custom JSON provider there are two methods, and each one of them will treat the request and response. The method loads receives (request) the byte string below:

b'{\n    "jsonrpc": "2.0",\n    "method": "App.index",\n    "params": {},\n    "id": "1"\n}'

and the method dumps receives (response) the Python dictionary below:

{'id': '1', 'jsonrpc': '2.0', 'result': '{"message": "Welcome to Flask JSON-RPC"}'}

So, I just did an example of how to customize the JSON provider, it wasn't the intention to create a JSON provider that will suit your use case, now you can change it and use it for your use case.

Thank you!

[nycholas]

@Talkless I'm closing this issue as you didn't pronounce yourself, I understand that the solution fixed your use case, any other questions or doubts, please reopen the issue.

Thank you.

[Talkless]

Thanks @nycholas I might get back to this question some time in the future, currency too busy with other stuff.