Improve intermediate layer extraction explanation #1338
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dbogdanov
requested changes
May 29, 2023
| "Note: The output of this algorithm is 2D, which is suitable for extracting embeddings or " | ||
| "class activations (the output shape is, e.g., [time, number of classes]). If the output " | ||
| "parameter is set to an intermediate layer with more dimensions, the output will be " | ||
| "flattened to 2D.\n" |
There was a problem hiding this comment.
Rephrased version (trying to simplify):
Note: The algorithm outputs a time series of class activations or embedding vectors, with a 2D shape [time, feature vector]. Feature vector values will be flattened if the
outputparameter is set to extract an intermediate layer with multiple dimensions.
| "class activations (the output shape is, e.g., [time, number of classes]). If the output " | ||
| "parameter is set to an intermediate layer with more dimensions, the output will be " | ||
| "flattened to 2D.\n" | ||
| "\n" |
There was a problem hiding this comment.
Same comments as for TensorflowPredictEffnetDiscogs
| "Note: The output of this algorithm is 2D, which is suitable for extracting embeddings or " | ||
| "class activations (the output shape is, e.g., [time, number of classes]). If the output " | ||
| "parameter is set to an intermediate layer with more dimensions, the output will be " | ||
| "flattened to 2D.\n" |
There was a problem hiding this comment.
Same comment as for TensorflowPredictEffnetDiscogs
| "Note: The output of this algorithm is 2D, which is suitable for extracting embeddings or " | ||
| "class activations (the output shape is, e.g., [time, number of classes]). If the output " | ||
| "parameter is set to an intermediate layer with more dimensions, the output will be " | ||
| "flattened to 2D.\n" |
There was a problem hiding this comment.
Same comment as for TensorflowPredictEffnetDiscogs
| @@ -66,6 +68,11 @@ AlgorithmStatus TensorToVectorReal::process() { | |||
| _timeStamps = tensor.dimension(2); | |||
| _featsSize = tensor.dimension(3); | |||
|
|
|||
| if (_channels != 1 && !_warned) { | |||
| E_WARNING("TensorToVectorReal: The channel axis (dimension 1) of the input tensor has size larger than 1, but the output of this algorithm is 2D. The batch, channel, and time axes (dimensions 0, 1, 2) will be flattened to the first dimension of the output matrix."); | |||
There was a problem hiding this comment.
We output a vector of vector of reals, so the "matrix" terminology may be misleading.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TensorToVectorReal converts tensors to 2D arrays by flattening all axis but the last one into the first dimension.
model-specific prediction algorithms (e.g., TensorflowPredictVGGish) use this algorithm to return 2D arrays since they are primarily intended for time-wise predictions or embeddings. However, it is possible to use these algorithms to extract intermediate layers of the models that may have more than two dimensions. In this case, all dimensions but the last one will be flattened. To address this:
Note that it is also possible to retrieve intermediate layers with their original shape using TensorflowPredict as discussed here.