Analysis tool for measuring manifold classification capacity, manifold radius, and manifold dimension, implemented for the results in Untangling in Invariant Speech Recognition, (2019) NeurIPS. The code implements the technique first described in Classification and Geometry of General Perceptual Manifolds, (2018) Physical Review X. and refined in Separability and Geometry of Object Manifolds in Deep Neural Networks, (2019) BioRxiv; (2020) Nature Communications.
If you find this code useful for your research, please cite our paper:
@inproceedings{stephenson2019untangling,
title={Untangling in Invariant Speech Recognition},
author={Stephenson, Cory and Feather, Jenelle and Padhy, Suchismita and Elibol, Oguz and Tang, Hanlin and McDermott, Josh and Chung, SueYeon},
booktitle={Advances in Neural Information Processing Systems},
pages={14368--14378},
year={2019}
}
First install required dependencies with
pip install -r requirements.txt
Then install the package via
pip install -e .
The following contains usage instructions for constructing data and feeding it to the analysis tool. An example analysis of a deep neural network implemented in PyTorch along with some higher level tools can be found in this example notebook.
Sample P classes from the dataset, and M examples from each class. Calculate the activations at the desired layer for each example, and package them into an array of shape (N, M) where N is the dimensionality of the layer. In the current implementation, M should be less than N. Assemble each of these into an iterable (ex: a list like X = [(N, M1), (N, M2),..., (N,MP)]
)
After specifying the margin kappa
and the number of t-vectors to sample n_t
(a good default is 200), compute the average mean-field theoretic manifold capacity, manifold radius, and and dimensionality.
First, import the required packages. manifold_analysis_correlation
contains the mean-field theoretic analysis including center correlations.
import numpy as np
from mftma.manifold_analysis_correlation import manifold_analysis_corr
Set up the data to analyze. Data should be a sequence of numpy arrays of shape (num_features, num_samples). Here, we show an example with random data with N=5000 features, M=50 samples per manifold, and P=100 manifolds.
np.random.seed(0)
X = [np.random.randn(5000, 50) for i in range(100)] # Replace this with data to analyze
Analyze the data. In this example, the analysis is done at zero margin (kappa=0
) with n_t=200
samples.
kappa = 0
n_t = 200
capacity_all, radius_all, dimension_all, center_correlation, K = manifold_analysis_corr(X, kappa, n_t)
To compute the average manifold capacity, the average should be done inversely. For manifold radius and dimension, the usual average is appropriate.
avg_capacity = 1/np.mean(1/capacity_all)
avg_radius = np.mean(radius_all)
avg_dimension = np.mean(dimension_all)
For this random data example, the result is (up to small differences due to n_t=200
samples)
avg_capacity = 0.04
avg_radius = 1.47
avg_dimension = 36.17
center_correlation = 0.01
K = 1
In addition to the manifold geometry metrics, the total dimensionality of the data can be computed. We use two different measures of total dimension. The first computes the number of dimensions required to explain a given percentage (default 90%) of the total variance. The second computes the participation ratio of the data. Using the random data from above, this looks like
from mftma.alldata_dimension_analysis import alldata_dimension_analysis
percentage = 0.90
D_participation_ratio, D_explained_variance, D_feature = alldata_dimension_analysis(X, perc=percentage)
For the random data, this results in the large values of
D_participation_ratio = 2499
D_explained_variance = 2548
D_feature = 5000 # By construction of the dataset and returned simply for convenience
Some of the manifold datasets used in Untangling in Invariant Speech Recognition are available for download.
The LibriSpeech word manifolds dataset is avalaible here: https://www.dropbox.com/sh/rh0wrsw88e77azd/AABG_YjDitkiYzfx6K45StXMa?dl=0
The LibriSpeech speaker manifolds dataset is available here: https://www.dropbox.com/sh/wej6hq24c70irwl/AAAoQ6f6Sa5xOCSYnRDt9w4Ga?dl=0
Email [email protected] or [email protected] with any questions.