Implementation of RAG

[abizard]

Hi, can developments like RAG be implemented in this Lag-Llama model?

[kashif]

in theory the very last state vector before being fed to the prediction head can be used as a representation of the past, although this vector has been trained for the next time-step prediction task rather than a representation of the context time series... thus for RAG type applications one would need to train the pre-trained lag-llama with an appropriate loss (not the next time step prediction task) using perhaps contrastive learning... hope that helps!