a lot of tests fail since werkzeug 2.1

[symphorien]

tested on current master

venv $  pytest
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.9.12, pytest-7.1.2, pluggy-1.0.0
rootdir: /tmp/flask-testing
collected 41 items                                                                                                                                                                           

tests/test_twill.py FF                                                                                                                                                                 [  4%]
tests/test_utils.py .................F....F......FF..FF.F..                                                                                                                            [100%]

========================================================================================== FAILURES ==========================================================================================
_______________________________________________________________________________ TestTwill.test_make_twill_url ________________________________________________________________________________

self = <tests.test_twill.TestTwill testMethod=test_make_twill_url>

    def test_make_twill_url(self):
>       with Twill(self.app) as t:

tests/test_twill.py:23: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <flask_testing.Error object at 0x7f5fc5718b80>, args = (<Flask 'tests.flask_app'>,), kwargs = {}, msg = "'twill' package is required for Error"

    def __init__(self, *args, **kwargs):
        msg = "'twill' package is required for %s" % (
              self.__class__.__name__)
>       raise ImportError(msg)
E       ImportError: 'twill' package is required for Error

flask_testing/__init__.py:26: ImportError
_________________________________________________________________________________ TestTwill.test_twill_setup _________________________________________________________________________________

self = <tests.test_twill.TestTwill testMethod=test_twill_setup>

    def test_twill_setup(self):
    
>       twill = Twill(self.app)

tests/test_twill.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <flask_testing.Error object at 0x7f5fc56913a0>, args = (<Flask 'tests.flask_app'>,), kwargs = {}, msg = "'twill' package is required for Error"

    def __init__(self, *args, **kwargs):
        msg = "'twill' package is required for %s" % (
              self.__class__.__name__)
>       raise ImportError(msg)
E       ImportError: 'twill' package is required for Error

flask_testing/__init__.py:26: ImportError
___________________________________________________________________________ TestClientUtils.test_assert_redirects ____________________________________________________________________________

self = <tests.test_utils.TestClientUtils testMethod=test_assert_redirects>

    def test_assert_redirects(self):
        response = self.client.get("/redirect/")
>       self.assertRedirects(response, "/")

tests/test_utils.py:88: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
flask_testing/utils.py:314: in assertRedirects
    self.assertEqual(response.location, expected_location, message)
E   AssertionError: '/' != 'http://localhost/'
E   - /
E   + http://localhost/
__________________________________________________________________ TestClientUtils.test_assert_redirects_valid_status_codes __________________________________________________________________

self = <tests.test_utils.TestClientUtils testMethod=test_assert_redirects_valid_status_codes>

    def test_assert_redirects_valid_status_codes(self):
        valid_redirect_status_codes = (301, 302, 303, 305, 307)
    
        for status_code in valid_redirect_status_codes:
            response = self.client.get("/redirect/?code=" + str(status_code))
>           self.assertRedirects(response, "/")

tests/test_utils.py:113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
flask_testing/utils.py:314: in assertRedirects
    self.assertEqual(response.location, expected_location, message)
E   AssertionError: '/' != 'http://localhost/'
E   - /
E   + http://localhost/
__________________________________________________________________________ BaseTestLiveServer.test_server_listening __________________________________________________________________________

self = <tests.test_utils.BaseTestLiveServer testMethod=test_server_listening>, result = <TestCaseFunction test_server_listening>

    def __call__(self, result=None):
        """
        Does the required setup, doing it here means you don't have to
        call super.setUp in subclasses.
        """
    
        # Get the app
>       self.app = self.create_app()

flask_testing/utils.py:441: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.test_utils.BaseTestLiveServer testMethod=test_server_listening>

    def create_app(self):
        """
        Create your Flask app here, with any
        configuration you need.
        """
>       raise NotImplementedError
E       NotImplementedError

flask_testing/utils.py:432: NotImplementedError
_____________________________________________________________________ BaseTestLiveServer.test_server_process_is_spawned ______________________________________________________________________

self = <tests.test_utils.BaseTestLiveServer testMethod=test_server_process_is_spawned>, result = <TestCaseFunction test_server_process_is_spawned>

    def __call__(self, result=None):
        """
        Does the required setup, doing it here means you don't have to
        call super.setUp in subclasses.
        """
    
        # Get the app
>       self.app = self.create_app()

flask_testing/utils.py:441: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.test_utils.BaseTestLiveServer testMethod=test_server_process_is_spawned>

    def create_app(self):
        """
        Create your Flask app here, with any
        configuration you need.
        """
>       raise NotImplementedError
E       NotImplementedError

flask_testing/utils.py:432: NotImplementedError
______________________________________________________________________ TestLiveServerOSPicksPort.test_server_listening _______________________________________________________________________

self = <tests.test_utils.TestLiveServerOSPicksPort testMethod=test_server_listening>, result = <TestCaseFunction test_server_listening>

    def __call__(self, result=None):
        """
        Does the required setup, doing it here means you don't have to
        call super.setUp in subclasses.
        """
    
        # Get the app
        self.app = self.create_app()
    
        self._configured_port = self.app.config.get('LIVESERVER_PORT', 5000)
        self._port_value = multiprocessing.Value('i', self._configured_port)
    
        # We need to create a context in order for extensions to catch up
        self._ctx = self.app.test_request_context()
        self._ctx.push()
    
        try:
>           self._spawn_live_server()

flask_testing/utils.py:451: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.test_utils.TestLiveServerOSPicksPort testMethod=test_server_listening>

    def _spawn_live_server(self):
        self._process = None
        port_value = self._port_value
    
        def worker(app, port):
            # Based on solution: http://stackoverflow.com/a/27598916
            # Monkey-patch the server_bind so we can determine the port bound by Flask.
            # This handles the case where the port specified is `0`, which means that
            # the OS chooses the port. This is the only known way (currently) of getting
            # the port out of Flask once we call `run`.
            original_socket_bind = socketserver.TCPServer.server_bind
            def socket_bind_wrapper(self):
                ret = original_socket_bind(self)
    
                # Get the port and save it into the port_value, so the parent process
                # can read it.
                (_, port) = self.socket.getsockname()
                port_value.value = port
                socketserver.TCPServer.server_bind = original_socket_bind
                return ret
    
            socketserver.TCPServer.server_bind = socket_bind_wrapper
            app.run(port=port, use_reloader=False)
    
        self._process = multiprocessing.Process(
            target=worker, args=(self.app, self._configured_port)
        )
    
        self._process.start()
    
        # We must wait for the server to start listening, but give up
        # after a specified maximum timeout
        timeout = self.app.config.get('LIVESERVER_TIMEOUT', 5)
        start_time = time.time()
    
        while True:
            elapsed_time = (time.time() - start_time)
            if elapsed_time > timeout:
>               raise RuntimeError(
                    "Failed to start the server after %d seconds. " % timeout
                )
E               RuntimeError: Failed to start the server after 5 seconds.

flask_testing/utils.py:501: RuntimeError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
 * Serving Flask app 'tests.flask_app' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
__________________________________________________________________ TestLiveServerOSPicksPort.test_server_process_is_spawned __________________________________________________________________

self = <tests.test_utils.TestLiveServerOSPicksPort testMethod=test_server_process_is_spawned>, result = <TestCaseFunction test_server_process_is_spawned>

    def __call__(self, result=None):
        """
        Does the required setup, doing it here means you don't have to
        call super.setUp in subclasses.
        """
    
        # Get the app
        self.app = self.create_app()
    
        self._configured_port = self.app.config.get('LIVESERVER_PORT', 5000)
        self._port_value = multiprocessing.Value('i', self._configured_port)
    
        # We need to create a context in order for extensions to catch up
        self._ctx = self.app.test_request_context()
        self._ctx.push()
    
        try:
>           self._spawn_live_server()

flask_testing/utils.py:451: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.test_utils.TestLiveServerOSPicksPort testMethod=test_server_process_is_spawned>

    def _spawn_live_server(self):
        self._process = None
        port_value = self._port_value
    
        def worker(app, port):
            # Based on solution: http://stackoverflow.com/a/27598916
            # Monkey-patch the server_bind so we can determine the port bound by Flask.
            # This handles the case where the port specified is `0`, which means that
            # the OS chooses the port. This is the only known way (currently) of getting
            # the port out of Flask once we call `run`.
            original_socket_bind = socketserver.TCPServer.server_bind
            def socket_bind_wrapper(self):
                ret = original_socket_bind(self)
    
                # Get the port and save it into the port_value, so the parent process
                # can read it.
                (_, port) = self.socket.getsockname()
                port_value.value = port
                socketserver.TCPServer.server_bind = original_socket_bind
                return ret
    
            socketserver.TCPServer.server_bind = socket_bind_wrapper
            app.run(port=port, use_reloader=False)
    
        self._process = multiprocessing.Process(
            target=worker, args=(self.app, self._configured_port)
        )
    
        self._process.start()
    
        # We must wait for the server to start listening, but give up
        # after a specified maximum timeout
        timeout = self.app.config.get('LIVESERVER_TIMEOUT', 5)
        start_time = time.time()
    
        while True:
            elapsed_time = (time.time() - start_time)
            if elapsed_time > timeout:
>               raise RuntimeError(
                    "Failed to start the server after %d seconds. " % timeout
                )
E               RuntimeError: Failed to start the server after 5 seconds.

flask_testing/utils.py:501: RuntimeError
------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------
 * Serving Flask app 'tests.flask_app' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
______________________________________________________________ TestNotRenderTemplates.test_assert_template_rendered_signal_sent ______________________________________________________________

self = <tests.test_utils.TestNotRenderTemplates testMethod=test_assert_template_rendered_signal_sent>

    def test_assert_template_rendered_signal_sent(self):
        self.client.get("/template/")
    
>       self.assert_template_used('index.html')

tests/test_utils.py:259: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
flask_testing/utils.py:244: in assertTemplateUsed
    _check_for_signals_support()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def _check_for_signals_support():
        if not _is_signals:
>           raise RuntimeError(
                "Your version of Flask doesn't support signals. "
                "This requires Flask 0.6+ with the blinker module installed."
            )
E           RuntimeError: Your version of Flask doesn't support signals. This requires Flask 0.6+ with the blinker module installed.

flask_testing/utils.py:112: RuntimeError
====================================================================================== warnings summary ======================================================================================
tests/test_twill.py:27
  /tmp/flask-testing/tests/test_twill.py:27: PytestCollectionWarning: cannot collect test class 'TestTwillDeprecated' because it has a __init__ constructor (from: tests/test_twill.py)
    class TestTwillDeprecated(TwillTestCase):

../../nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/lib/python3.9/unittest/result.py:24
  /nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/lib/python3.9/unittest/result.py:24: PytestCollectionWarning: cannot collect test class 'TestResult' because it has a __init__ constructor (from: tests/test_utils.py)
    class TestResult(object):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ===================================================================================
FAILED tests/test_twill.py::TestTwill::test_make_twill_url - ImportError: 'twill' package is required for Error
FAILED tests/test_twill.py::TestTwill::test_twill_setup - ImportError: 'twill' package is required for Error
FAILED tests/test_utils.py::TestClientUtils::test_assert_redirects - AssertionError: '/' != 'http://localhost/'
FAILED tests/test_utils.py::TestClientUtils::test_assert_redirects_valid_status_codes - AssertionError: '/' != 'http://localhost/'
FAILED tests/test_utils.py::BaseTestLiveServer::test_server_listening - NotImplementedError
FAILED tests/test_utils.py::BaseTestLiveServer::test_server_process_is_spawned - NotImplementedError
FAILED tests/test_utils.py::TestLiveServerOSPicksPort::test_server_listening - RuntimeError: Failed to start the server after 5 seconds.
FAILED tests/test_utils.py::TestLiveServerOSPicksPort::test_server_process_is_spawned - RuntimeError: Failed to start the server after 5 seconds.
FAILED tests/test_utils.py::TestNotRenderTemplates::test_assert_template_rendered_signal_sent - RuntimeError: Your version of Flask doesn't support signals. This requires Flask 0.6+ with ...
========================================================================= 9 failed, 32 passed, 2 warnings in 10.47s ==========================================================================
                                                                                                                                                                                              
[1] /tmp/flask-testing master 11s 
venv $  pip freeze
attrs==21.4.0
backports.entry-points-selectable @ file:///build/backports.entry_points_selectable-1.1.1/dist/backports.entry_points_selectable-1.1.1-py2.py3-none-any.whl
click==8.1.3
distlib @ file:///build/distlib-0.3.4/dist/distlib-0.3.4-py3-none-any.whl
filelock @ file:///build/filelock-3.6.0/dist/filelock-3.6.0-py3-none-any.whl
Flask==2.1.2
Flask-Testing==0.8.1
importlib-metadata==4.11.3
iniconfig==1.1.1
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.1
packaging==21.3
platformdirs @ file:///build/source/dist/platformdirs-2.5.1-py3-none-any.whl
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.8
pytest==7.1.2
six @ file:///build/six-1.16.0/dist/six-1.16.0-py2.py3-none-any.whl
tomli==2.0.1
virtualenv @ file:///build/virtualenv-20.14.0/dist/virtualenv-20.14.0-py2.py3-none-any.whl
Werkzeug==2.1.2
zipp==3.8.0

[klarose]

I have also run into this. It appears that at least in our case, the RuntimeError: Failed to start the server after 5 seconds. is triggered when setting LIVESERVER_PORT to 0. In particular, the following code is never invoked, which makes me suspect the patch it's doing is failing:

flask-testing/flask_testing/utils.py

Line 474 in 5107691

def socket_bind_wrapper(self):

I can work around it by setting a specific port, but that isn't very resilient.