If CSV has encoding problem, Polars loads a dataset with zero rows

[mccalluc]

Hit this during the hackathon, and it was confusing. It's not really our problem... but with DP people aren't supposed to look at the data, so maybe we need to work harder to parse the CSV than polars out of the box, which assumes user interaction.

[Shoeboxam]

The behavior I'm seeing with your xfail example is that it parses any rows with unusual characters (like é) as null. Values without those characters parse correctly.

[mccalluc]

I thought I saw in the hackathon that no rows would load if there was a misencoding anywhere. I can expand the test.

[SalilVadhan]

It happened on the squirrel dataset.

…

----- Salil Vadhan (he/they)

On Wed, Aug 28, 2024 at 9:48 AM Chuck McCallum ***@***.***> wrote: I thought I saw in the hackathon that no rows would load if there was a misencoding anywhere. I can expand the test. — Reply to this email directly, view it on GitHub <#1976 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4KIGONBOFK2G5ZCEWY5HTZTXILLAVCNFSM6AAAAABNAVOUKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJVGM3TCMZSGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mccalluc]

Solution proposed in

• Handle non-utf8 CSVs dp-wizard#29

is to use encoding="utf8-lossy": It's not perfect, but unless two grouping strings are identical apart from characters that are misencoded, it wouldn't harm the statistics. Somehow getting the correct encoding and re-encoding the CSV feels like a big hassle for very little gain.