About the encoder layer output

[Shar-01]

As understood, the MCG-NJU/videomae-base encoder outputs tensor of size (batch, 1568,768). Can we interpret this as 1567 patch tokens with emb size of 768 and the first token as the cls token? If so, for further downstream tasks, can we only take the first (cls token)and discard the rest 1567 patch tokens? I.e., Can we just take (batch, 1, 768) for further downstream tasks? Can someone say if they tried this for further downstream tasks and how much information is lost/retained compared to taking the whole 1568 tokens? (if someone compared it).