SLURM HPC architecture for running AGNOSTOS

[mhyleung]

Dear all

Our server has a following architecture. We currently have the same head and compute node. It seems to be ok submitting SLURM jobs as single nodes, but we are currently struggling to set up multi-nodes with our server. We suspect that SLURM requires some sort of networking between computer and head nodes with open ports. We just have 96cpus, but they seem to be separated into 2 NUMA nodes between the 96 CPUs.

We were wondering if AGNOSTOS can work via SLURM operating on a single node, or is it necessary to have AGNOSTOS running on a multi-node system?

Thank you very much

Regards

Marcus

[genomewalker]

Hi @mhyleung
it should be able to run in a single node and run multiple jobs in a single node. It is difficult to say as this is very architecture dependent and not related to AGNOSTOS but to your setup and how SLURM is configured.
If you only have one node, you should be careful as some steps can be memory and I/O intensive. So be conservative on how many jobs you run in a single node.

[mhyleung]

Thank you Antonio. Just to be safe, we have changed the cluster.yaml file and set the runs to a terabyte memory. Running it on 10 contig sequences at the moment and so far no glaring errors.

Just a related question: of the different steps within AGNOSTOS, which one is the most computationally intensive to the point of needing MPI? We are now at the mmseq_clustering_update snakemake (step 4 or 5?) without activating mpi, but it seems to be running fine for now. It this simply because we are running on a small dataset, or we cannot get away with using openmpi because now the database is so big even if the input set of sequences is small?

Regards

Marcus

[genomewalker]

If you only have one node, you should be fine if you remove any MPI as you will not be able to upscale to hundreds of processes. Maybe one bottleneck will be in the step when you do the all-vs-all HHblits search to get the communities.

[mhyleung]

Thanks Antonio as usual.

Just out of curiosity if I may ask, when you and your team run AGNOSTOS on multiple nodes, how much cpu and mem is used per node? Thanks

Marcus

[ChiaraVanni]

Hi Marcus!
this usually depends on the rule you are running. You can find the amount of resources that we used to build the AGNOSTOS DBs in the cluster.yaml file (https://github.com/functional-dark-side/agnostos-wf/blob/master/db_update/config/cluster.yaml)