Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slurm tests #3606

Merged
merged 12 commits into from
Sep 6, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/parsl+slurm.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Test Slurm Scheduler
on:
pull_request:

jobs:
build:
runs-on: ubuntu-22.04
permissions:
packages: read
strategy:
fail-fast: false
matrix:
container: ["ghcr.io/tylern4/slurm-standalone:slurm-24-05-0-1"]
timeout-minutes: 30

container:
image: ${{ matrix.container }}
options: "--platform=linux/amd64 --rm -h node01"

name: ${{ matrix.container }}
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Install Dependencies and Parsl
run: |
CC=/usr/lib64/openmpi/bin/mpicc pip3 install . -r test-requirements.txt
- name: Verify Parsl Installation
run: |
pytest parsl/tests/ -k "not cleannet and not unix_filesystem_permissions_required" --config parsl/tests/configs/local_threads.py --random-order --durations 10
- name: Test Parsl with Slurm Config
run: |
./parsl/tests/slurm-entrypoint.sh pytest parsl/tests/ -k "not cleannet and not unix_filesystem_permissions_required" --config parsl/tests/configs/slurm_local.py --random-order --durations 10
29 changes: 29 additions & 0 deletions parsl/tests/configs/slurm_local.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
from parsl.channels import LocalChannel
from parsl.config import Config
from parsl.executors import HighThroughputExecutor
from parsl.launchers import SrunLauncher
from parsl.providers import SlurmProvider


def fresh_config():
return Config(
executors=[
HighThroughputExecutor(
label="docker_slurm",
encrypted=True,
provider=SlurmProvider(
cmd_timeout=60, # Add extra time for slow scheduler responses
channel=LocalChannel(),
nodes_per_block=1,
init_blocks=1,
min_blocks=1,
max_blocks=1,
walltime='00:10:00',
launcher=SrunLauncher(),
),
)
],
)


config = fresh_config()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this line isn't needed and you got it by cut and pasting it from somewhere else that also doesn't need it... it also causes trouble in some situations by making configs be initialized at import (so every single test config gets initialised on every test run), rather than when the config is actually wanted.

16 changes: 16 additions & 0 deletions parsl/tests/slurm-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/bin/bash
set -e

echo "---> Starting the MUNGE Authentication service (munged) ..."
gosu munge /usr/sbin/munged

echo "---> Starting the slurmctld ..."
exec gosu slurm /usr/sbin/slurmctld -i -Dvvv &

echo "---> Waiting for slurmctld to become active before starting slurmd..."

echo "---> Starting the Slurm Node Daemon (slurmd) ..."
exec /usr/sbin/slurmd -Dvvv &

echo "---> Running user command '${@}'"
exec "$@"
Loading