Upgrade from 1.5.7 to 1.6 breaks listing of recordings in TV / Recordings section

[nkichukov]

I am raising a separate issue here to track this problem that was partially discussed under:
#134

The new version of the app stopped listing my recordings on various kodi 19 builds: 19 Alpha 1, 19 RC1.

The recordings were listing totally fine in 1.5.7.
The recordings stoppled being listed in 1.6. The error is:
https://user-images.githubusercontent.com/1557784/107213222-26a4e700-6a19-11eb-868b-f661538965c8.jpg

Setting up a new tvheadend server with only a few recordings appears to work fine, so it is not a generic problem with the change, there is something in the actual list of recordings, perhaps in the payload: the character set/encoding or a special escape sequence or a character in the returned json response that breaks the app.

How can I identify which recording(s) make it fail?

This is a regression in my point of view.

[kambala-decapitator]

How can I identify which recording(s) make it fail?

check them one by one

[wutschel]

Searching the web for the displayed error message "The data couldn´t be read because it isn´t in the correct format." points towards an internal error in the JSON deserialization. With 1.6 the App was migrated to use NSJSONSerialization, which was the key to bring back functionality under iOS 14. So, there´s no way back. From what I read this issue pops up when data is not received completely or when the received data violates the expected format.

@nkichukov, can you try to figure which recording causes the error?

[joethefox]

@wutschel, I remember that in the past I had this kind of issue solved with this commit 369d130

maybe can be related

[wutschel]

@joethefox, good to see you here :)

Do you know if this would this be equivalent to the option NSJSONReadingFragmentsAllowed when using NSJSONSerialization?

NSDictionary *jsonResult = [NSJSONSerialization JSONObjectWithData:connectionData options:NSJSONReadingFragmentsAllowed error:&error];

[kambala-decapitator]

@wutschel how'd that be related to ill-formed unicode?

Specifies that the parser should allow top-level objects that are not an instance of NSArray or NSDictionary.

[wutschel]

It´s not, forget about my comment.

[nkichukov]

@wutschel , eventually I will figure it out, however my list of recordings is very big. None of the dvr log files contain invalid utf8 bytes this time.

In the past, where I did have invalid utf8 bytes in the dvr logs due to a bug in tvheadend file name truncate mechanism, I filed another bug report for different iOS app, see here:
zipleen/tvheadend-iphone-client#246

In the end, the developer gave up on the fix and I had to work with tvheadend upstream to figure out a solution, see:
https://tvheadend.org/issues/5668

Now, this appears to be a very similar problem, however:
1.) It was working in 1.5.7
2.) There are no invalid bytes in the dvr log files or their names.

Running the json rpc api call with CURL does return the complete json result set and there are no obvious problems with it.

Is there a way for me to run my result-set through the iOS app engine that does the parsing, outside of the app and analyze where it could fail?

Thanks,
-Nikolay

[kambala-decapitator]

Is there a way for me to run my result-set through the iOS app engine that does the parsing, outside of the app and analyze where it could fail?

The simplest would be building and running the following test code on macOS: https://gist.github.com/kambala-decapitator/94adb76c6849ddee2dc41c8fd6e05930

To build: (I can also provide prebuilt binary)

clang -framework Foundation test_json.m

Run example:

❯ ./a.out '{"jsonrpc":"2.0","method":"Application.GetProperties","id":"asd","params":["version"]}'
2021-02-16 20:10:48.088 a.out[17092:280468] {
    id = asd;
    jsonrpc = "2.0";
    method = "Application.GetProperties";
    params =     (
        version
    );
}

In output you'll see either decoded JSON or an error.

You could also just post your curl output.

[nkichukov]

Hello @kambala-decapitator

I have managed to isolate the issue. I could identify the recordings that cause the problem and those seem to be the ones that have large description fields ('plot' field in kodi).

Then I tried the code you provided, however, because I could not preserve the invalid byte when copying and pasting the response, I had to adapt it to read it from a file. It was failing to parse the json from the file.

In addition to this, I've tried the linux utility 'json_reformat' which is also unable to parse the output that contains invalid utf8 bytes, the failure is: 'lexical error: invalid bytes in UTF8 string.'.

I added upstream bug report to tvheadend, see https://tvheadend.org/issues/6003 .

However, I am aware this could be due to kodi-pvr-hts, kodi or kodi json rpc api. At this time, I cannot really tell.

Now I wonder how this worked before you switched to NSJSONSerialization?

Apparently any tool that has to deal with invalid utf8 bytes struggles to ignore/survive those, such as tvhClient/official-kodi-remote-ios/json_reformat.

[kambala-decapitator]

old parser engine had an option to ignore broken UTF-8, see here: bbc955c#diff-fdf9f7d097ab403c5114e09e302e2ed12c9f65a85aa6c9ef652332997a135565L278

unfortunately, NSJSONSerialization doesn't have a counterpart.

[joethefox]

@wutschel, you are doing an awesome job, kudos :)

I don't have a solution too sadly :(

[kambala-decapitator]

@nkichukov could you post sample JSON that gives failures?

[nkichukov]

You can find more details regarding the JSON payload here:
https://tvheadend.org/issues/6003

Again, this is more likely a tvheadend bug, as their software returns invalid bytes at version 4.2.8 when 'description' field is too long and they truncate it. They might have fixed this in there release candidate version 4.3, but I have not tested it.

I have worked around the problem by truncating all long fields in my recordings, which no longer causes the problem.
Thanks!