Sorry for not reacting to this. You were correct that this is a problem of floating point precision and the solution is to calculate transformations on the CPU using 64 bit precision. However a correct fix is much more complex than this.

Firstly and mainly, the decision when to calculate the transformation on CPU should not be based on the number of points but on the magnitude (determinant?) of the transformation matrix, when zoom or translation is to large to be represented accurately by 32 bits it should be calculated on the CPU.

Secondly, even then the transformation should not be done for each render step, otherwise you lose the speed up from using the GPU.
Instead one can split the transformation into two steps. The first part is doing the transformation on CPU and caching the resulting vertex positions as well as the transformation. The second is to calculate the relative transformation from that cached transformation during each render step and only calculate that relative transformation on the GPU.
And every time the relative transformation gets to large for representing it with 32 bits we update the cached positions and transformation with the current transformation.
That way we only calculate the transformation once in a while on the CPU and can keep the GPU for handling most of the render calls.

This just for completeness if anyone would like redo this PR.

I remember the PR was to use CPU to compute small number of points, but still use GPU for large number of points.
A fast, hacky workaround

Ah, I see now! Yes indeed a fast hacky workaround :).

Out of curiosity: Did you actually really end up using this library? I could never really get it useful enough for my use cases.

Yes I ended up using the library for a while. It helped a bit. Later we switched our focus. When we revisit plotting, we decided to use streamlit, plotly and clickhouse(for downsampling)

Plotly sounds much more sensible, designed for interactive plots and with GPU acceleration built in. Maybe I should recommend it in my Readme as a better alternative.