UMAP diagnostic type "neighborhood" results in backend error

[cjy8s]

I have been going through the UMAP documentation, and other than having to change what is called "umap" through my package imports, the examples have worked for my data. The only part that is not working is the last part, for calculating Jaccard scores using the neighborhood diagnostics. The same embedding_sanity_check2D_fit object works with all of the other examples in the documentation, so I do not think that is the issue.

Do you have any ideas as to why this is not working? Thanks!

import umap.umap_ as umap

# PCA dimension reduction to 50 dimensions
pca = sklearn.decomposition.PCA(n_components=50)
pca_result = pca.fit_transform(embeddings_array)

# Compute UMAP with 2 dimensions on the PCA-reduced data
embedding_sanity_check2D_fit = umap.UMAP(min_dist=0.5, spread=1).fit(pca_result)
embedding_sanity_check2D_trans = embedding_sanity_check2D_fit.transform(pca_result) 

import umap.plot

umap.plot.diagnostic(embedding_sanity_check2D_fit, diagnostic_type='neighborhood')

---------------------------------------------------------------------------
TypingError                               Traceback (most recent call last)
Cell In[155], [line 1](vscode-notebook-cell:?execution_count=155&line=1)
----> [1](vscode-notebook-cell:?execution_count=155&line=1) umap.plot.diagnostic(embedding_sanity_check2D_fit, diagnostic_type='neighborhood')

File [~/.conda/.../lib/python3.11/site-packages/umap/plot.py:1124), in diagnostic(umap_object, diagnostic_type, nhood_size, local_variance_threshold, ax, cmap, point_size, background, width, height)
....

File [~/.conda/.../lib/python3.11/site-packages/numba/core/dispatcher.py:409), in _DispatcherBase._compile_for_args.<locals>.error_rewrite(e, issue_type)
    [407](https://.../lib/python3.11/site-packages/numba/core/dispatcher.py:407)     raise e
    [408](https://.../lib/python3.11/site-packages/numba/core/dispatcher.py:408) else:
--> [409](https://.../lib/python3.11/site-packages/numba/core/dispatcher.py:409)     raise e.with_traceback(None)

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<function intersect1d at 0x7f84449b7ce0>) found for signature:
 
 >>> intersect1d(array(int32, 1d, C), array(int32, 1d, C), assume_unique=Literal[bool](True))
 
There are 2 candidate implementations:
  - Of which 2 did not match due to:
  Overload in function 'jit_np_intersect1d': File: numba/np/arraymath.py: Line 3586.
    With argument(s): '(array(int32, 1d, C), array(int32, 1d, C), assume_unique=bool)':
   Rejected as the implementation raised a specific error:
     TypingError: got an unexpected keyword argument 'assume_unique'
  raised from /.../lib/python3.11/site-packages/numba/core/typing/templates.py:784

During: resolving callee type: Function(<function intersect1d at 0x7f84449b7ce0>)
During: typing of call at /.../lib/python3.11/site-packages/umap/plot.py (209)


File "../.../lib/python3.11/site-packages/umap/plot.py", line 209:
def _nhood_compare(indices_left, indices_right):
    <source elided>
    for i in range(indices_left.shape[0]):
        intersection_size = np.intersect1d(indices_left[i], indices_right[i]).shape[0]
        ^

[lmcinnes]

A workaround for now:

Add this just above the function _nhood_compare in plot.py in your umap install:

@numba.njit(cache=True)
def arr_intersect(ar1, ar2):
    aux = np.concatenate((ar1, ar2))
    aux.sort()
    return aux[:-1][aux[1:] == aux[:-1]]

and then chance the line intersection_size = np.intersect1d(indices_left[i], indices_right[i]).shape[0] to intersection_size = arr_intersect(indices_left[i], indices_right[i]).shape[0]

That may solve the issue for you.

[cjy8s]

It's a different error now, so that is progress right?

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<function hstack at 0x7f7a301dbb00>) found for signature:
 
 >>> hstack(list(array(int32, 1d, C))<iv=None>)
 
There are 2 candidate implementations:
  - Of which 2 did not match due to:
  Overload of function 'hstack': File: numba/np/arrayobj.py: Line 6007.
    With argument(s): '(list(array(int32, 1d, C))<iv=None>)':
   No match.

During: resolving callee type: Function(<function hstack at 0x7f7a301dbb00>)
During: typing of call at /.../lib/python3.11/site-packages/umap/plot.py (216)


File "../.../lib/python3.11/site-packages/umap/plot.py", line 216:
# This is the addition that I input according to lmcinnes's advice on GitHub
@numba.njit(cache=True)
def arr_intersect(ar1, ar2):
    aux = np.concatenate((ar1, ar2))
    aux.sort()
    return aux[:-1][aux[1:] == aux[:-1]]

@numba.jit()
def _nhood_compare(indices_left, indices_right):
    """Compute Jaccard index of two neighborhoods"""
    result = np.empty(indices_left.shape[0])

    for i in range(indices_left.shape[0]):
        intersection_size = arr_intersect(indices_left[i], indices_right[i]).shape[0]
->      union_size = np.unique(np.hstack([indices_left[i], indices_right[i]])).shape[0]
        result[i] = float(intersection_size) / float(union_size)

    return result