null characters in strings

[EmielBruijntjes]

Hi Rainer,

Would you be willing to overthink your decision to not handle NULL bytes inside strings? It would be nice if we can rely on the library to recognize all valid json input, even when it contains these NULL bytes.

Emiel

[rgerhards]

Emiel, let me ask a question first: would you be ready to model your program around whatever string library libfastjson would choose (or implement) to provide this functionality?

[EmielBruijntjes]

Yes, I would be ready to model the program around the fastjson api -- as long as it is not ridiculous of coure.

But I don't get why you would need a special string api. As long as a user of the fastjson library uses the functions fjson_object_get_string() and fjson_object_get_string_len() functions to retrieve strings, and the function fjson_object_new_string_len() for setting data, it is possible to store and retrieve all possible binary data inside json, including NULL values. From my perspective, the current API is powerful enough to support NULL values.

[rgerhards]

My point was to make sure you had thought about the implications of that use. You obviously have. The core point is that with NUL character support, the caller can no longer use the regular C runtime functions. Many json-c programs I have seen would fail miserably in that case (and json-c at least partly supports NUL characters). Furthermore, NUL support means that libfastjson-c needs to apply different (and slower) handling in internal functions, which at least is contrary to the prime design goal of being fast.

That said, I would not totally oppose support for NUL chars, but I would say

• it needs to be optional
• to support NUL, the caller must explicitely opt-in (e.g. during an init call)
• the performance in the non-NULL case must not suffer

The third requirement probably means that we need to duplicate a (large?) number of internal functions.

Also note that json-c, from which we forked, claims full compatibility, but only has very partial support for NUL chars. In practice, they are almost unusable. Nevertheless, it got very widely adpoted, which makes me think NUL char support is something that is not really demanded (just to clarify my line of thought).

Side note: I prefer the term "NUL character", based on the US-ASCII and Unicode tables over "NULL value". While both are technical the integer 0, "NULL value" I think is too easily mistaken for NULL pointers and IMHO also not not 100% technically correct when talking about the 0x00 US-ASCII control character.