Training SD1.5 Always tends to noise instead of positive imporves?

[SilverDragon-RY]

Hi, I am trying to train SD15 on 17k random samples from laion2b. This is the the setting I am using:

accelerate launch
--main_process_port 11111
--num_processes 1
--num_cpu_threads_per_process 2
perflow_accelerate_sd.py
--data_root "???"
--resolution 512 --dataloader_num_workers 4 --train_batch_size 16 --gradient_accumulation_steps 1
--pretrained_model_name_or_path "pt-sk/stable-diffusion-1.5"
--unet_model_path ""
--pred_type "diff_eps" --loss_type "noise_matching"
--windows 4 --solving_steps 10 --support_cfg --cfg_sync
--learning_rate 2e-5 --lr_scheduler "constant" --lr_warmup_steps 500
--mixed_precision "fp16"
--output_dir "../exps/sd15_laion_org"
--validation_steps 100 --inference_steps "8" --inference_cfg "5" --save_ckpt_state --checkpointing_steps 1000
--max_train_steps 400000 \

as a result, the output image quickly fades to noise and seen no improvements till iter 37000. I have tried on a variety settings of dataset/LR settings/base-models but all the same. Does this phenomenon fits your observation or is there something wrong I am doing?

Ps: sample below