Add Della merge method

[Tej-Deep]

Adds a new merging method della. Della first ranks parameters in each row of delta parameters and assigns drop probabilities adaptively, inversely proportional to their magnitudes. Delta parameters with higher magnitudes are assigned lower drop probabilities. After assigning drop probabilities, the delta parameters are dropped and rescaled in a manner similar to the DARE method. The Della-merging paper can be found here

[cg123]

Thanks for the PR! Glad to get your method into mergekit.

It would be great to also add Della to the README with a link to your paper. Happy to do this myself if you'd like.

[Tej-Deep]

Thank you. I have added Della to the README. Please feel free to make any changes.

[cg123]

Looks great! Thanks for the PR.

[Se-Hun]

@Tej-Deep

Hello, Thanks for sharing your approach and code.
I read your paper(https://arxiv.org/abs/2406.11617) and wanna apply this to my model.
So, How can i configure yaml file for mergekit ?
Can i get some examples ??

[Tej-Deep]

@Se-Hun

Hello. The configure yaml for Della is similar to Ties and Dare. Here is an example config yaml file:

models:
  - model: MODEL_1_PATH
    parameters:
      weight: 1.0
  - model: MODEL_2_PATH
    parameters:
      weight: 1.0
merge_method: della
base_model: BASE_MODEL_PATH
parameters:
  density: 0.7
  lambda: 1.1
  epsilon: 0.2

[Se-Hun]

Oh, Thank you for quick answer.