Guacamole user sync timeout

[JimMadge]

✅ Checklist

• I have searched open and closed issues for duplicates.
• This is a problem observed when managing a Data Safe Haven.
• I can reproduce this with the latest version.
• I have read through the documentation.
• This isn't an open-ended question (open a discussion if it is).

💻 System information

• Operating System:
• Data Safe Haven version: 5.2.0

🚫 Describe the problem

The Guacamole user sync container may encounter a timeout, at which point it seems to stall and not crash/restart/try to sync again.
This seems to happen sometimes on first deployment.

2024-12-10 09:09:03 [ERROR   ] Exception during reset or similar
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/pool/base.py", line 986, in _finalize_fairy
    fairy._reset(
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/pool/base.py", line 1432, in _reset
    pool._dialect.do_rollback(self)
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/default.py", line 699, in do_rollback
    dbapi_connection.rollback()
  File "/usr/local/lib/python3.11/dist-packages/psycopg/connection.py", line 261, in rollback
    self.wait(self._rollback_gen())
  File "/usr/local/lib/python3.11/dist-packages/psycopg/connection.py", line 394, in wait
    return waiting.wait(gen, self.pgconn.socket, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "psycopg_c/_psycopg/waiting.pyx", line 213, in psycopg_c._psycopg.wait_c
  File "/usr/local/lib/python3.11/dist-packages/psycopg/_connection_base.py", line 602, in _rollback_gen
    yield from self._exec_command(b"ROLLBACK")
  File "/usr/local/lib/python3.11/dist-packages/psycopg/_connection_base.py", line 469, in _exec_command
    result = (yield from generators.execute(self.pgconn))[-1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "psycopg_c/_psycopg/generators.pyx", line 122, in execute
  File "psycopg_c/_psycopg/generators.pyx", line 176, in fetch_many
  File "psycopg_c/_psycopg/generators.pyx", line 229, in fetch
psycopg.OperationalError: consuming input failed: could not receive data from server: Connection timed out
SSL SYSCALL error: Connection timed out

Steps to reproduce

1. Deploy a brand new SRE, and add a new user to it. We've seen it happening on the green SHM.
2. Use this user to access the brand new SRE, via Guacamole. There won't be any SRD available:
   
3. Changing browsers or using Incognito tabs won't help.
4. Going to portal, and checking the logs of the container guacamole-user-sync shows the following in logs:

2024-12-10 09:09:03 [ERROR   ] Exception during reset or similar
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/pool/base.py", line 986, in _finalize_fairy
    fairy._reset(
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/pool/base.py", line 1432, in _reset
    pool._dialect.do_rollback(self)
  File "/usr/local/lib/python3.11/dist-packages/sqlalchemy/engine/default.py", line 699, in do_rollback
    dbapi_connection.rollback()
  File "/usr/local/lib/python3.11/dist-packages/psycopg/connection.py", line 261, in rollback
    self.wait(self._rollback_gen())
  File "/usr/local/lib/python3.11/dist-packages/psycopg/connection.py", line 394, in wait
    return waiting.wait(gen, self.pgconn.socket, interval=interval)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "psycopg_c/_psycopg/waiting.pyx", line 213, in psycopg_c._psycopg.wait_c
  File "/usr/local/lib/python3.11/dist-packages/psycopg/_connection_base.py", line 602, in _rollback_gen
    yield from self._exec_command(b"ROLLBACK")
  File "/usr/local/lib/python3.11/dist-packages/psycopg/_connection_base.py", line 469, in _exec_command
    result = (yield from generators.execute(self.pgconn))[-1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "psycopg_c/_psycopg/generators.pyx", line 122, in execute
  File "psycopg_c/_psycopg/generators.pyx", line 176, in fetch_many
  File "psycopg_c/_psycopg/generators.pyx", line 229, in fetch
psycopg.OperationalError: consuming input failed: could not receive data from server: Connection timed out
SSL SYSCALL error: Connection timed out

6. Restarting the -container-group-remote-desktop containers solves the problem.

🚂 Workarounds or solutions

Restarting the container group can fix the problem. Presumably because Entra/Apricot are now ready to receive requests

[JimMadge]

@cptanalatriste anything to add here?

[cptanalatriste]

@JimMadge I added some steps to reproduce