Adding a fixture to set scheduler to slower speeds and revert it back.

[rraghav-cisco]

Description of PR

Summary:
Fixes the flakiness of DWRR testcase. The PR adds a new fixture that slows down the scheduler without changing the underlying algorithm. This allows the dWRR test to pass consitently.

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 202012
• 202205
• 202305
• 202311
• 202405

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[rraghav-cisco]

Result from my run:

=========================================================================================================================== PASSES ===========================================================================================================================
______________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic] ______________________________________________________________________________________________________
---------------------------------------------------- generated xml file: /run_logs/dwrr/2024-11-24-04-39-38/1/L-400g-LL-masic-gb/qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr_2024-11-24-04-39-38.xml -----------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------------------------------------------------------------------- live log sessionfinish -------------------------------------------------------------------------------------------------------------------
04:56:32 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================================== short test summary info ===================================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic]
========================================================================================================= 1 passed, 1 warning in 1012.38s (0:16:52) ==========================================================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$

[rraghav-cisco]

Finally found a way to repeat tests:

====================================================================================================================== warnings summary ======================================================================================================================
../../usr/local/lib/python3.8/dist-packages/paramiko/transport.py:236
  /usr/local/lib/python3.8/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
    "class": algorithms.Blowfish,

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================================================================================================== PASSES ===========================================================================================================================
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-1-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-2-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-3-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-4-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-5-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-6-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-7-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-8-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-9-10] ____________________________________________________________________________________________________
___________________________________________________________________________________________________ TestQosSai.testQosSaiDwrr[single_dut_multi_asic-10-10] ___________________________________________________________________________________________________
---------------------------------------------------- generated xml file: /run_logs/dwrr/2024-11-24-19-05-56/1/L-400g-LL-masic-gb/qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr_2024-11-24-19-05-56.xml -----------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------------------------------------------------------------------- live log sessionfinish -------------------------------------------------------------------------------------------------------------------
19:35:30 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================================== short test summary info ===================================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-1-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-2-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-3-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-4-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-5-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-6-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-7-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-8-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-9-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-10-10]
========================================================================================================= 10 passed, 1 warning in 1773.04s (0:29:33) =========================================================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$ 

[rraghav-cisco]

100G/S - 100G/S, mAsic:

INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------------------------------------------------------------------- live log sessionfinish -------------------------------------------------------------------------------------------------------------------
01:56:02 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================================== short test summary info ===================================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-1-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-1-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-2-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-2-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-3-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-3-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-4-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-4-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-5-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-5-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-6-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-6-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-7-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-7-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-8-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-8-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-9-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-9-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-10-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-10-10]
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-1-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-1-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-2-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-2-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-3-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-3-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-4-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-4-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-5-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-5-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-6-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-6-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-7-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-7-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-8-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-8-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-9-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-9-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_dut_multi_asic-10-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
ERROR qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_dut_multi_asic-10-10] - Failed: Processes "['analyze_logs--<MultiAsicSonicHost xx37-lc7>']" failed with exit code "1"
=========================================================================================== 20 passed, 470 deselected, 1 warning, 20 errors in 2973.01s (0:49:33) ============================================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$ 

[rraghav-cisco]

100G/S - 100G/S - sAsic:

---------------------------------------------------------- generated xml file: /run_logs/dwrr-cir/2024-11-26-02-31-41/1/O-100g-SS-gb-sasic/qos/test_qos_sai.py::TestQosSai_2024-11-26-02-31-41.xml -----------------------------------------------------------
INFO:root:Can not get Allure report URL. Please check logs
------------------------------------------------------------------------------------------------------------------- live log sessionfinish -------------------------------------------------------------------------------------------------------------------
03:05:14 __init__.pytest_terminal_summary         L0067 INFO   | Can not get Allure report URL. Please check logs
================================================================================================================== short test summary info ===================================================================================================================
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-1-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-1-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-2-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-2-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-3-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-3-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-4-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-4-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-5-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-5-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-6-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-6-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-7-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-7-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-8-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-8-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-9-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-9-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrr[single_asic-10-10]
PASSED qos/test_qos_sai.py::TestQosSai::testQosSaiDwrrWeightChange[single_asic-10-10]
================================================================================================= 20 passed, 470 deselected, 1 warning in 2011.91s (0:33:31) =================================================================================================
sonic@202405-qos-sonic-mgmt-prod:/data/tests$