Ubuntu 23.04 and Yubikey not always asking for U2F

[omriasta]

I have a Lenovo X1 Carbon 10th Gen running Ubuntu 23.04. I have setup the Yubikey as per the instructions.
When used in sudo or console TTY login it behaves as expected:
I get prompted first to use my fingerprint scanner (if not used in 10 seconds, it asks for password) and after that, I get prompted to touch the Yubikey.
I have also added the same line in /etc/pam.d/gdm-password (see below file).
I have noticed that this only works occasionally.
For the most part, if I boot up the computer, it doesn't ask for the yubikey at alll and just lets me in after the fingerprint is read. It seems to prompt for the yubikey more often after I lock the screen or resume from suspend but even then it is just most of the times and not always. I haven't been able to figure out why this works sometimes randomly.
I have noticed that occasionally, even using sudo, the key is not detected properly. When this happens though, the login is just stalled and does nothing. If I unplug the key and plug it right back in, it will prompt me to start over with the fingerprint and then yubikey as expected (as opposed to just letting me in without the yubikey).
I'm not sure how much of this is related, just wanted to give as much info as possible. Anybody have any ideas?

gdm-password

%PAM-1.0
auth    requisite       pam_nologin.so
auth    required        pam_succeed_if.so user != root quiet_success
@include common-auth
auth required pam_u2f.so authfile=/etc/Yubico/u2f_keys cue
auth    optional        pam_gnome_keyring.so
@include common-account
# SELinux needs to be the first session rule. This ensures that any 
# lingering context has been cleared. Without this it is possible 
# that a module could execute code in the wrong domain.
session [success=ok ignore=ignore module_unknown=ignore default=bad]        pam_selinux.so close
session required        pam_loginuid.so
# SELinux needs to intervene at login time to ensure that the process
# starts in the proper default security context. Only sessions which are
# intended to run in the user's context should be run after this.
# pam_selinux.so changes the SELinux context of the used TTY and configures
# SELinux in order to transition to the user context with the next execve()
# call.
session [success=ok ignore=ignore module_unknown=ignore default=bad]        pam_selinux.so open
session optional        pam_keyinit.so force revoke
session required        pam_limits.so
session required        pam_env.so readenv=1
session required        pam_env.so readenv=1 user_readenv=1 envfile=/etc/default/locale
@include common-session
session optional        pam_gnome_keyring.so auto_start
@include common-password

common auth

#
# /etc/pam.d/common-auth - authentication settings common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of the authentication modules that define
# the central authentication scheme for use on the system
# (e.g., /etc/shadow, LDAP, Kerberos, etc.).  The default is to use the
# traditional Unix authentication mechanisms.
#
# As of pam 1.0.1-6, this file is managed by pam-auth-update by default.
# To take advantage of this, it is recommended that you configure any
# local modules either before or after the default block, and use
# pam-auth-update to manage selection of other modules.  See
# pam-auth-update(8) for details.

# here are the per-package modules (the "Primary" block)
auth    [success=3 default=ignore]      pam_fprintd.so max-tries=1 timeout=10 # debug
auth    [success=2 default=ignore]      pam_unix.so nullok try_first_pass
auth    [success=1 default=ignore]      pam_sss.so use_first_pass
# here's the fallback if no module succeeds
auth    requisite                       pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
auth    required                        pam_permit.so
# and here are more per-package modules (the "Additional" block)
auth    optional                        pam_cap.so
# end of pam-auth-update config

[omriasta]

For anyone that has this problem, I think I was able to fix this by removing the entries I made in the pam.d files for sudo and gdm-password and just adding the same line at the end of the common-auth file. Now it appears to be prompting me always. I'm still not sure why this doesn't work consistently in the gdm-password file...or what causes the random behaviour.