Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

better logging of fam failures #16967

Merged
merged 1 commit into from
Nov 26, 2024
Merged

better logging of fam failures #16967

merged 1 commit into from
Nov 26, 2024

Conversation

evgeni
Copy link
Member

@evgeni evgeni commented Nov 20, 2024 •
edited
Loading

Problem Statement

The output of FAM tests is mangled due to how pytest tries to print the big result object.
There are no linebreaks and it's super hard to find the failure.

Additionally the double assert is pointless, as it asserts twice the same thing: the result is successful.

Solution

Stop asserting PASSED in stdout (additionally saves the case where there is the string PASSED in the output for some reason, but the test actually failed).
Pass a custom error output to the status == 0 assertion, including stdout, stderr and the status code.

Related Issues

@evgeni
Copy link
Member Author

evgeni commented Nov 20, 2024 

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9416
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 336 warnings in 1228.67s (0:20:28) ===========

@Satellite-QE Satellite-QE added the PRT-Failed Indicates that latest PRT run is failed for the PR label Nov 20, 2024
@evgeni
Copy link
Member Author

evgeni commented Nov 20, 2024 

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

@evgeni evgeni added CherryPick PR needs CherryPick to previous branches 6.14.z Introduced in or relating directly to Satellite 6.14 6.15.z Introduced in or relating directly to Satellite 6.15 6.16.z Introduced in or relating directly to Satellite 6.16 labels Nov 20, 2024
@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9421
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 328 warnings in 1120.24s (0:18:40) ===========

@evgeni
Copy link
Member Author

evgeni commented Nov 20, 2024 

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9423
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 337 warnings in 1113.46s (0:18:33) ===========

@evgeni
Copy link
Member Author

evgeni commented Nov 20, 2024 

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9424
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 339 warnings in 1260.80s (0:21:00) ===========

@evgeni
Copy link
Member Author

evgeni commented Nov 20, 2024 

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9425
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 16 failed, 94 passed, 341 warnings in 4650.37s (1:17:30) ===========

@evgeni evgeni force-pushed the better-fam-log branch 2 times, most recently from 4221c48 to 7c562af Compare November 21, 2024 09:52
@evgeni
Copy link
Member Author

evgeni commented Nov 21, 2024 

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py

@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9431
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 16 failed, 94 passed, 333 warnings in 4732.56s (1:18:52) ===========

@evgeni evgeni removed 6.14.z Introduced in or relating directly to Satellite 6.14 6.15.z Introduced in or relating directly to Satellite 6.15 labels Nov 21, 2024
evgeni
evgeni commented Nov 21, 2024
View reviewed changes
tests/foreman/sys/test_fam.py Outdated
Comment on lines 197 to 198
module_target_sat.get('/tmp/pytest.stdout', 'pytest.stdout')
module_target_sat.get('/tmp/pytest.stderr', 'pytest.stderr')
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can't be the right approach :(

@evgeni evgeni marked this pull request as ready for review November 21, 2024 13:20
@evgeni evgeni requested a review from a team as a code owner November 21, 2024 13:20
@evgeni
Copy link
Member Author

evgeni commented Nov 22, 2024 

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py -vv

@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9445
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py -vv --external-logging
Test Result : =========== 16 failed, 94 passed, 356 warnings in 5042.30s (1:24:02) ===========

@evgeni
Copy link
Member Author

evgeni commented Nov 22, 2024 

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[smart_class_parameter_override_value]

@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9447
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[smart_class_parameter_override_value] --external-logging
Test Result : ================= 1 failed, 11 warnings in 1118.29s (0:18:38) ==================

rmynar
rmynar approved these changes Nov 25, 2024
View reviewed changes
Copy link
Contributor

@rmynar rmynar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't be easier :)

@rmynar
Copy link
Contributor

rmynar commented Nov 25, 2024

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[activation_key]

@rmynar
Copy link
Contributor

rmynar commented Nov 25, 2024

trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_import_run_roles

@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9455
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[activation_key] --external-logging
Test Result : ================== 1 passed, 11 warnings in 986.83s (0:16:26) ==================

@Satellite-QE Satellite-QE added PRT-Passed Indicates that latest PRT run is passed for the PR and removed PRT-Failed Indicates that latest PRT run is failed for the PR labels Nov 25, 2024
@Satellite-QE
Copy link
Collaborator

PRT Result

Build Number: 9456
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/sys/test_fam.py::test_positive_import_run_roles --external-logging
Test Result : ================== 1 passed, 9 warnings in 599.90s (0:09:59) ===================

@evgeni evgeni merged commit 04976cc into SatelliteQE:master Nov 26, 2024
11 checks passed
@evgeni evgeni deleted the better-fam-log branch November 26, 2024 12:43
github-actions bot pushed a commit that referenced this pull request Nov 26, 2024
@evgeni @web-flow
better logging of fam failures (#16967)
fb7e207 
(cherry picked from commit 04976cc)
@Satellite-QE Satellite-QE mentioned this pull request Nov 26, 2024
Merged
pondrejk pushed a commit that referenced this pull request Nov 29, 2024
@Satellite-QE @evgeni
[6.16.z] better logging of fam failures (#17001)
8d533ec 
better logging of fam failures (#16967)

(cherry picked from commit 04976cc)

Co-authored-by: Evgeni Golov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rmynar rmynar rmynar approved these changes

Assignees

@evgeni evgeni

Labels
6.16.z Introduced in or relating directly to Satellite 6.16 CherryPick PR needs CherryPick to previous branches PRT-Passed Indicates that latest PRT run is passed for the PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@evgeni @Satellite-QE @rmynar