Spurious structure for sparse matrices?

[deklanw]

I was playing with UMAP, first with a dense dataset and then later with a sparse dataset.

Here's the result I got with all defaults for the dense dataset

Looked interesting. To sanity check it I just used np.random.rand to generate data of the same shape,

Looks like a random mess. So far so good.

I tried the same thing with a sparse binary real-world dataset. Result:

And, then sanity checked with a random sparse binary matrix with the same density and shape as my original:

R_density = R.nnz / np.multiply(*R.shape)
random_binary = scipy.sparse.random(*R.shape, density=R_density, format='csr')

# replace nonzero with 1
random_binary.data[:] = 1

Result:

This doesn't look like a random mess. I removed the binary requirement and got the same result. Anyone know what would cause this?

[lmcinnes]

The only real difference is in the nearest neighbor search which did have some minor bugs in the sparse computations at one time. That is all farmed out to PyNNDescent these days and those bugs have been fixed there, but perhaps there is something odd outstanding? I am not aware of any reason why this should occur other than, perhaps, the sparse rp-trees giving some weird trouble.