[WIP] add support for mixtral

[tohrnii]

Mixtral WIP

[danielhanchen]

Fantastic and fabulous work @tohrnii!!! Super appreciate it! I will take a look later today!

[kaykyr]

Any update about this pull request?

[cm2435]

@kaykyr @danielhanchen @tohrnii You guys open to some collaboration on this? I think I just my Phi2 implementation done (big touch wood) so I'm happy to take a look

[tohrnii]

Apologies, I got stuck on something else. I'd love to collaborate @cm2435. If however you are close to completing the implementation, I'm happy to close this PR in favor of yours.

[danielhanchen]

Hey - thanks again on the PR @tohrnii and super appreciate it again :) Ye more than happy to make this happen and collab with you all @kaykyr @cm2435 - I was just a bit bogged down recently on chat templates and making a UI - I will be much more free next week, then we can make Mixtral happen :)

[kaykyr]

For sure! I am trying to fine tune a MoE pretrained if I had progress I will create pull requests guys. I also able to offer my small server (2x RTX 3090 with NVLink + i9 11900HK + 64GB DDR4) for collaborators who wanna run tests with multi-gpu.

[danielhanchen]

@kaykyr Oh thanks for the kind offer!!! I'll take up for that offer later in the month :)

[cm2435]

@kaykyr funny you mention that- I've got almost the exact same setup! I'm going to be very sad when they deprecate the SLI bridge as cuda supported hardware

[ilkersigirci]

Great work. Is there any estimate about when this will be merged?