Hello!

[lucidrains]

This is great! Anyway I can help out? Are you all doing this for school or part of a lab?

[lucidrains]

Have you seen https://github.com/kheyer/Genomic-ULMFiT

[Lev1ty]

Hi Phil! Thank you for reaching out! We were learning from some of the repositories you've created just earlier yesterday. To answer your questions... Yes! We would appreciate any help; we're currently at the stage of implementing the Performer, Sub-linear memory performer, and nystromformer. We're prioritizing nystromformer because it seems easier to implement and apparently fairs better in terms of time to convergence and long range attention compared to performer. This project is a part of a student research team/club at our university called iGEM McMaster and our focus is on genetic engineering and synthetic biology. We found that there's some missing literature on efficient transformers for analyzing longer gene sequences (100k+ nucleotides) so we've decided to open up this project. The genomic-ULMFiT seems cool and the reading links on that repository are definitely relevant. If you're interested, we could reach out to you over email to further discuss collaboration.

-Adam