Testing tracker

[felipemello1]

The purpose of this issue is to keep track of all changes we need to make in our tests for them to be more robust

• FSDP sharding

  • Unit test for training.shard_model (see [bug] fix sharding multimodal #1889)
  • Unit test for training.get_shard_conditions
  • Test custom_sharded_layers
  • Test if if memory(FSDP) < memory(single_device)

• Test correct initialization of LoRA params when using meta device (a la test_lora_meta_device_init_fsdp)

• Better testing for eval recipe: add test for generation and log likelihood tasks #1873

• Integration testing for DPO  #1411

• Regression testing in torchtune #1306

• Add recipe tests which use DoRA (see [bug] DoRA is broken #1903)

• Config CI: [WIP] Config Continous Integration (CCI) #1717

  • In the config CI, add test to make sure that component is always the first item
  • check if the module isnt deprecated gemma2 had wrong path to scheduler #2013 (comment)

• Add regression test for peak memory (especially with gradient accumulation enabled)

• We should have tests that run for 2 epochs, instead of 1, to make sure intermediate_checkpoint are saved correctly

• Add tests for Gemma2 attention

• Too many of our tests use torchtune-format checkpoints (e.g.). We should have a couple of these, but the majority of our test checkpoints should be HF format.

Tasks

Beta Give feedback

1. Add regression tests for adapter accuracy (maybe manually merge original checkpoint + adapter and resume from checkpoint))
2. Define manual test to resume from checkpoint
3. Adding MM eval tests / attention bugfixes #1989
   CLA Signed +1
4. Add recipe tests for vision models