Seeking guidance on resuming training process

[keysongkang]

I am trying to resume the training process due to the wall time of our HPC system.
After setting the resume flag and rerunning the code, it appears that the code package recognizes the existence of a previously trained model (trained until Epoch 3106), but it starts training from Epoch 0 when it resumes. Is there anything else I should add to the input?

Graph data file: HGraph-h5-Si_soc-5l-FromDFT.pkl
Use existing graph data file
Atomic types: [14]
Finish loading the processed 100 structures (spinful: False, the number of atomic types: 1), cost 19 seconds
number of train set: 60
number of val set: 20
number of test set: 20
{'normalizer': False, 'boxcox': False}
Output features length of single edge: 324
The model you built has: 513205 parameters
=> loaded model at "/u/scratch/Si/output/trained_model/best_state_dict.pkl" (epoch 3106)
VAL loss each out:
[2.1e-02, 2.2e-04, 3.6e-03, 3.0e-03, 5.8e-04, 6.1e-05, 8.4e-05, 4.0e-03, 3.7e-03, 4.1e-03, 1.3e-03, 1.2e-03, 1.8e-03, 5.8e-04, 2.6e-05, 2.6e-04, 3.1e-05, 7.0e-05, 6.8e-04, 1.1e-03, 2.2e-03, 1.9e-03, 1.0e-04, 8.4e-04, 1.1e-03, 1.6e-03, 1.4e-03, 1.5e-03, 7.1e-04, 7.3e-04, 4.6e-04, 3.6e-05, 3.1e-05, 2.5e-05, 5.8e-05, 1.7e-05, 3.8e-03, 2.8e-03, 1.6e-03, 1.6e-03, 5.6e-04, 2.5e-04, 2.8e-04, 2.0e-03, 1.5e-03, 1.7e-03, 4.0e-04, 8.7e-04, 3.5e-04, 1.7e-04, 2.0e-04, 1.9e-04, 1.5e-04, 6.8e-04, 1.3e-03, 7.5e-04, 2.3e-04, 1.8e-04, 2.2e-04, 1.5e-04, 1.3e-03, 9.1e-04, 2.4e-04, 1.3e-04, 4.2e-05, 7.1e-05, 2.2e-04, 9.3e-05, 1.3e-04, 3.2e-04, 2.8e-05, 1.0e-04, 1.6e-04, 1.2e-04, 4.9e-04, 1.8e-04, 4.5e-04, 3.8e-05, 7.2e-04, 9.0e-04, 6.1e-04, 6.4e-04, 2.9e-04, 8.5e-04, 2.2e-04, 5.2e-04, 9.5e-05, 1.1e-04, 8.2e-05, 4.9e-05, 2.0e-04, 5.0e-05, 4.6e-04, 1.1e-04, 1.2e-04, 1.8e-04, 5.1e-04, 6.9e-04, 4.1e-04, 2.5e-04, 4.2e-04, 1.5e-04, 7.6e-04, 2.6e-04, 1.1e-04, 3.0e-05, 3.0e-04, 4.5e-04, 2.3e-04, 4.4e-05, 2.6e-04, 1.6e-04, 5.2e-04, 3.0e-05, 3.0e-04, 8.5e-04, 4.1e-04, 3.7e-04, 2.6e-04, 1.1e-04, 5.3e-04, 2.1e-04, 2.8e-05, 3.3e-04, 1.7e-03, 3.9e-04, 4.7e-03, 1.5e-03, 2.8e-03, 1.4e-04, 1.2e-03, 1.3e-03, 1.4e-03, 3.0e-03, 1.3e-03, 1.3e-03, 2.8e-04, 1.1e-04, 1.5e-04, 2.4e-04, 2.5e-04, 1.6e-04, 5.0e-04, 2.0e-04, 3.6e-03, 1.5e-03, 1.4e-03, 2.4e-04, 5.7e-04, 3.4e-04, 2.1e-04, 1.4e-03, 7.4e-04, 7.4e-04, 1.5e-04, 1.8e-04, 2.1e-04, 1.6e-04, 1.9e-04, 1.1e-04, 6.4e-05, 1.3e-04, 4.5e-03, 1.5e-03, 1.5e-03, 9.6e-05, 6.6e-04, 2.1e-04, 4.6e-04, 1.1e-03, 1.2e-03, 9.1e-04, 1.1e-03, 9.9e-05, 2.3e-04, 1.3e-04, 7.5e-05, 6.3e-05, 3.7e-04, 1.3e-04, 1.2e-03, 5.9e-04, 3.9e-04, 7.2e-05, 2.7e-04, 3.3e-04, 1.9e-04, 5.7e-04, 1.3e-04, 4.3e-04, 2.8e-04, 1.9e-04, 1.3e-04, 4.1e-04, 5.5e-04, 4.0e-05, 1.6e-04, 1.9e-05, 1.6e-03, 4.8e-04, 2.9e-04, 3.8e-05, 2.8e-04, 1.5e-04, 5.6e-04, 1.7e-04, 1.2e-04, 3.2e-04, 6.7e-04, 3.4e-05, 4.4e-04, 2.6e-05, 3.0e-05, 1.6e-05, 4.6e-05, 8.1e-05, 1.4e-03, 5.1e-04, 4.4e-04, 8.8e-05, 2.5e-04, 2.1e-04, 1.6e-04, 1.4e-04, 2.0e-03, 2.1e-04, 1.7e-05, 2.2e-03, 2.8e-05, 5.5e-04, 5.3e-05, 4.6e-05, 9.3e-04, 2.0e-05, 1.4e-04, 2.3e-05, 2.8e-04, 3.3e-04, 2.0e-04, 3.4e-05, 3.3e-05, 1.3e-04, 1.3e-04, 1.7e-04, 3.2e-05, 4.0e-05, 5.1e-04, 2.1e-05, 3.7e-05, 3.0e-04, 2.3e-04, 2.3e-05, 1.7e-04, 6.3e-05, 2.3e-04, 6.2e-04, 5.9e-05, 1.5e-03, 5.0e-05, 2.3e-04, 3.2e-04, 8.4e-05, 8.0e-04, 1.4e-04, 3.4e-04, 4.4e-04, 1.3e-03, 3.5e-05, 3.9e-04, 1.6e-04, 5.6e-05, 2.2e-05, 4.0e-04, 5.1e-04, 7.5e-05, 4.5e-04, 7.0e-05, 9.4e-05, 1.1e-04, 1.1e-03, 1.9e-05, 2.2e-04, 1.3e-05, 5.9e-04, 1.5e-04, 4.7e-04, 2.5e-04, 3.4e-05, 1.0e-04, 1.8e-05, 1.2e-03, 5.0e-05, 7.0e-05, 1.2e-03, 7.2e-05, 1.0e-04, 2.6e-04, 3.0e-04, 3.7e-04, 2.2e-04, 9.0e-05, 1.4e-04, 4.8e-04, 5.9e-05, 9.7e-04, 2.4e-05, 1.5e-04, 1.7e-05, 1.4e-04, 1.4e-04, 8.5e-04, 1.4e-04, 8.1e-05, 1.2e-04, 8.5e-04, 6.8e-05, 8.0e-05, 6.4e-04, 6.3e-04, 3.9e-04, 4.1e-05, 1.7e-04, 8.8e-05, 1.6e-04]
max orbital: 2.1e-02 (0-based index: 0)
Epoch #0 | Learning rate: 1.00e-04 | Epoch time: 33.22 | Train loss: 0.00052358 | Val loss: 0.00060493 | Best val loss: 0.00060493.
VAL loss each out:

[mzjb]

The program is only displaying 'Epoch 0', and I'm not quite clear on what exactly you're confused about.

Additionally, if the training process is time-consuming, you might consider using DeepH-E3 as an alternative. It utilizes the same data format as DeepH-pack.

[keysongkang]

Thank you for your comments! Let me clarify what I was trying to achieve:

1. Initially, I set the number of epochs for training to be 10,000.
2. However, due to the time constraints of our HPC system, the calculation was terminated around the 3,106th epoch.
3. I resumed the calculation, but it seems that it started training again (printed output shows it starts from epoch 0), reading the last recorded epoch as 3,106.
4. After repeating this process four times, the total number of epochs should have exceeded 13,000, but the training did not complete.

I had assumed that the training would terminate once it reached the target number of epochs, but it appears that this is not the case. I was wondering if the training restarts from the beginning each time. Therefore, I wanted to confirm this aspect. However, based on your response, it seems that my understanding that the calculation is just not finished and keep training over 10000 epoch is correct.

I also have an additional question. Can the trained model from DeepH interface with FHI-aims? Currently, it seems that it is compatible with OpenMX and ABACUS. Would that be possible to calculate the overlap matrix calculation using FHI-aims?

[mzjb]

To study large-scale material systems using DeepH, it is necessary to obtain the overlap matrix of the materials efficiently. For OpenMX and Siesta, we have implemented fast overlap calculations. For ABACUS, their developers have also implemented fast overlap calculations. However, we have not implemented a separate overlap calculation feature in FHI-aims. If you are familiar with FHI-aims, you can implement the overlap calculation independently without performing the SCF calculations.

[keysongkang]

Thank you for providing your valuable feedback! I have a couple of follow-up questions regarding your comments:

1. I came across an instruction in the documentation that states, "For overlap matrix calculation, you need to use the same basis set and DFT software when preparing the dataset." Given this, I'm curious if it is fine to compute the overlap matrix using OpenMX and use it for the trained model with FHI-aims data. Could you shed some light on this matter?

2. If it turns out that I need to employ FHI-aims to obtain the overlap matrix, I'm considering attempting to generate it. Could you please provide further clarification on a few points?
   a. My understanding is that I need to calculate the overlap matrix elements of the atomic basis in FHI-aims. Could you confirm if my understanding is correct?
   b. It appears that the output is in HDF5 format. Can you provide more details about the output files? What does each number in the output file name represent?
   c. Within the output file, is there a specific format or structure that I should be aware of?

I hope these questions provide more clarity, and I look forward to your response.

[mzjb]

1. FHI-aims and OpenMX use different basis sets, so it is not possible to directly use the FHI-aims Hamiltonian with the overlap matrix calculated from OpenMX.
2. a. Yes.
   b. The output files are in HDF5 format. The key in the HDF5 file is a string consisting of five integers in the format [Rx, Ry, Rz, i, j]. The corresponding value is the matrix block H_{i, R0}. H_{i, Rj} represents the hopping between the i-th atom in the (0, 0, 0) unit cell and the j-th atom in the (Rx, Ry, Rz) unit cell.
   c. I recommend referring to the FHI-aims preprocess script in DeepH, which arranges the angular part of the atomic basis in the specific order required by DeepH.