Input opt

[farakiko]

Additional standardize_inputs: Bool flag in the config.yml which if True will standardize the input features before running a training (can also be simply passed using command line arg --standardize-input).

The standardization only makes use of a subset of the train_dataset (default: 10k events) to extract the mean and the std. The standardization is performed separately for each PFelement type.

Below is a comparison with and without standardization of inputs using the pyg-clic.yml config, running with the following commands,

without
python mlpf/pyg_pipeline.py --gpus 1 --train --dataset clic --data-dir /pfvolcentral/tensorflow_datasets --config parameters/pytorch/pyg-clic.yaml --conv-type attention --attention-type math --dtype float32 --ntrain 10000 --nvalid 1000 --num-epochs 100 --patience 20 --lr 1e-5 --num-workers 2 --prefetch-factor 2 --gpu-batch-multiplier 20

with
python mlpf/pyg_pipeline.py --gpus 1 --train --dataset clic --data-dir /pfvolcentral/tensorflow_datasets --config parameters/pytorch/pyg-clic.yaml --conv-type attention --attention-type math --dtype float32 --ntrain 10000 --nvalid 1000 --num-epochs 100 --patience 20 --lr 1e-5 --num-workers 2 --prefetch-factor 2 --gpu-batch-multiplier 20 --standardize-input

Below are plots of the validation loss. Three runs are overlayed for each case, the blue curves are runs with input standardization, and the orange curves are runs without input standardization.