Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let bsub retry on LSF message "Request from non-LSF host rejected" #9195

Merged
merged 7 commits into from
Nov 13, 2024
Merged

Let bsub retry on LSF message "Request from non-LSF host rejected" #9195

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7d6025e
Freeze websockets at < 14
JHolba Nov 11, 2024
404a7ea
Assume non-LSF host error is flaky
berland Nov 12, 2024
29298aa
Pin qtpy to 2.4.1 for Everest
berland Nov 13, 2024
10cb1ba
Mitigate flakyness on busy test nodes
berland Nov 11, 2024
ef7d79f
Mitigate flakiness in memory profiling
berland Nov 1, 2024
a870a48
Use maxprocesses=2 for cli tests
JHolba Nov 1, 2024
c39bb3a
Fix bug with inconsistent numbers in flaky test
berland Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ dependencies = [
"tqdm>=4.62.0",
"typing_extensions>=4.5",
"uvicorn >= 0.17.0",
"websockets",
"websockets < 14",
"xarray",
"xtgeo >= 3.3.0",
]
Expand Down
9 changes: 8 additions & 1 deletion src/ert/scheduler/lsf_driver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,14 @@ class RunningJob:
LSF_INFO_JSON_FILENAME = "lsf_info.json"
FLAKY_SSH_RETURNCODE = 255
JOB_ALREADY_FINISHED_BKILL_MSG = "Job has already finished"
BSUB_FAILURE_MESSAGES = ("Job not submitted",)
BSUB_FAILURE_MESSAGES = (
"Error in rusage section",
"Expeced number, string",
"No such queue",
"Too many processors requested",
"cannot be used in the resource requirement section",
"duplicate section",
)


def _parse_jobs_dict(jobs: Mapping[str, JobState]) -> dict[str, AnyJob]:
Expand Down
3 changes: 2 additions & 1 deletion tests/unit_tests/scheduler/test_lsf_driver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -578,7 +578,6 @@ async def test_that_bsub_will_retry_and_fail(
" '&' cannot be used in the resource requirement section. Job not submitted.",
),
(255, "Error in rusage section. Job not submitted."),
(255, "Job not submitted."),
],
)
async def test_that_bsub_will_fail_without_retries(
Expand All @@ -604,6 +603,8 @@ async def test_that_bsub_will_fail_without_retries(
[
(0, "void"),
(FLAKY_SSH_RETURNCODE, ""),
(0, "Request from non-LSF host rejected"),
(FLAKY_SSH_RETURNCODE, "Request from non-LSF host rejected"),
],
)
async def test_that_bsub_will_retry_and_succeed(
Expand Down
Loading