Incompatibility with a running gpg-agent as ssh-agent?

[freswa]

Expected behavior:
upterm host, Ctrl-c gives me a ssh command to connect.
I enter the ssh command on another host and I'm sharing a terminal.

What actually happens:
I'm running an gpg-agent in ssh-agent mode.
upterm host leads to my gpg-agent authenticating against uptermd.upterm.dev. If I don't supply the password, I get an FATA[0016] ssh dial error: ssh: handshake failed: agent: failed to sign challenge. If I supply the password, the session is protected with a pubkey and entering the ssh <token>@uptermd.upterm.dev command on another host leads to permission denied (publickey). I can't share a session even if I delete my .ssh/config.

Is this expected behavior? How can I achieve the desired behavior similar to tmate?