Batch evaluation script

[jjbuschhoff]

Sbatch script that performs evaluation on a given set of tasks for a given collection of model checkpoints using the Megatron-LM-client-server inference solution.

[janEbert]

Multiple paths in the sbatch script are not properly quoted and can cause issues if they include spaces, for example. For more safety it would be preferred to correctly quote all variable expansions in Bash scripts.

Also, the whole argument handling portion of run_server_no_opt.py is way too manual. There should be more abstraction here. For example, many command line flags are already missing. Continued experimentation would imply that every single change that adds arguments would also need to manually add them here, lest they would be ignored.

I noted lots of things that would need some love, but in general this is super useful and I'm very happy you created the script!

[janEbert]

I'd remove the comment above since it is confusing and uncomment these lines so that FlashAttention is activated if args.bf16 or args.fp16. FlashAttention is an exact algorithm, so we do not gain anything from deactivating it.

[jjbuschhoff]

Also, the whole argument handling portion of run_server_no_opt.py is way too manual. There should be more abstraction here. For example, many command line flags are already missing. Continued experimentation would imply that every single change that adds arguments would also need to manually add them here, lest they would be ignored.

I agree, I had a look into this and it seems that it is possible to call run_text_generation_server.py --use_checkpoint_args directly, however, this only sets some of the hyperparameters as per the checkpoint (see load_args_from_checkpoint in megatron.checkpointing), likely those that are necessary for training. The checkpoint_args are returned but unused in megatron.initialize.initialize_megatron(). I'm looking into a way to merge them that doesn't lead validate_args failing.

[janEbert]

That's a great find!