Improve options for showing menu during minibuffer use

- Use the `window-preserve-size' window properly instead of the
  more aggressive `window-size-fixed' variable, as recommended in
  the documentation.

- Delay fixing the size until the window is actually being displayed.

- Add a new option `transient-show-during-minibuffer-read', replacing
  `transient-hide-during-minibuffer-read'.  This is necessary because
  there are now multiple values that show the menu, but in different
  ways, which can only be expressed using distinct non-nil values.

- When the user chose that the size should be `fixed', but we know that
  won't fly because the menu window uses the full frame height, then
  override that choice.  This is necessary because it will soon become
  possible for an individual prefix to use a different display action,
  which may have different needs from the global value the user
  optimized for.

- Allow displaying the fixed menu if possible, while also falling back
  to not displaying it either at all or while potentially resizing it,
  if that is not possible.

Closes #330.

CHANGELOG

@@ -1,6 +1,12 @@
 # -*- mode: org -*-
 * v0.8.0    UNRELEASED
 
+- While the minibuffer is in use, the menu window is now hidden by
+  default.  The new option ~transient-show-during-minibuffer-read~ not
+  only controls whether the menu stays visible while the minibuffer
+  is in use, but also whether it may be resized if necessary.  This
+  new option replaces ~transient-hide-during-minibuffer-read~.  #330
+
 - When returning to a prefix whose ~refresh-suffixes~ slot is non-nil,
   its suffixes are now re-initialized.  #327
 
@@ -20,12 +26,6 @@
 - ~transient-setup-buffer-hook~ is now run later to allow overriding
   more default settings.
 
-- While the minibuffer is in use, the menu buffer is now hidden by
-  default.  Customize ~transient-hide-during-minibuffer-read~ if you
-  prefer the old default.
-
-- Further refinements to when the size of the menu window is fixed.
-
 Bug fixes:
 
 - Fixes some menu navigation edge-cases.

docs/transient.org

@@ -623,10 +623,35 @@ Also see [[* Common Suffix Commands]].
   from the user.  If ~nil~, there is no initial input and the first
   element has to be accessed the same way as the older elements.
 
-- User Option: transient-hide-during-minibuffer-read ::
-
-  This option controls whether the transient buffer is hidden while
-  user input is being read in the minibuffer.
+- User Option: transient-show-during-minibuffer-read ::
+
+  This option controls whether the transient menu continues to be
+  displayed while the minibuffer is used to read user input.
+
+  This is only relevant to commands that do not close the menu, such as
+  commands that set infix arguments.  If a command exits the menu, and
+  uses the minibuffer, then the menu is always closed before the
+  minibuffer is entered, irrespective of the value of this option.
+
+  When ~nil~ (the default), hide the menu while the minibuffer is in use.
+  When ~t~, keep showing the menu, but allow for the menu window to be
+  resized to ensure that completion candidates can be displayed.  When
+  ~fixed~, keep showing the menu and prevent it from being resized.
+
+  If the value is ~fixed~ and the menu window uses the full height of its
+  frame, then the former is ignored and resizing is allowed anyway.  This
+  is necessary because individual menus may use custom display actions,
+  which the user may not be aware of.
+
+  When using a third-party mode, which automatically resizes windows
+  (e.g., by calling ~balance-windows~ on ~post-command-hook~), then ~fixed~
+  is likely a better choice than ~t~.
+
+  The value can also be an integer, in which case the behavior depends on
+  whether at least that many lines are left to display windows other than
+  the menu window.  If that is the case, display the menu and preserve the
+  size of the window.  Otherwise, hide the menu if the number is positive,
+  or allow resizing if it is negative.
 
 - User Option: transient-align-variable-pitch ::
 

docs/transient.texi

@@ -782,9 +782,34 @@ from the user.  If @code{nil}, there is no initial input and the first
 element has to be accessed the same way as the older elements.
 @end defopt
 
-@defopt transient-hide-during-minibuffer-read
-This option controls whether the transient buffer is hidden while
-user input is being read in the minibuffer.
+@defopt transient-show-during-minibuffer-read
+This option controls whether the transient menu continues to be
+displayed while the minibuffer is used to read user input.
+
+This is only relevant to commands that do not close the menu, such as
+commands that set infix arguments.  If a command exits the menu, and
+uses the minibuffer, then the menu is always closed before the
+minibuffer is entered, irrespective of the value of this option.
+
+When @code{nil} (the default), hide the menu while the minibuffer is in use.
+When @code{t}, keep showing the menu, but allow for the menu window to be
+resized to ensure that completion candidates can be displayed.  When
+@code{fixed}, keep showing the menu and prevent it from being resized.
+
+If the value is @code{fixed} and the menu window uses the full height of its
+frame, then the former is ignored and resizing is allowed anyway.  This
+is necessary because individual menus may use custom display actions,
+which the user may not be aware of.
+
+When using a third-party mode, which automatically resizes windows
+(e.g., by calling @code{balance-windows} on @code{post-command-hook}), then @code{fixed}
+is likely a better choice than @code{t}.
+
+The value can also be an integer, in which case the behavior depends on
+whether at least that many lines are left to display windows other than
+the menu window.  If that is the case, display the menu and preserve the
+size of the window.  Otherwise, hide the menu if the number is positive,
+or allow resizing if it is negative.
 @end defopt
 
 @defopt transient-align-variable-pitch

lisp/transient.el

@@ -95,6 +95,10 @@ TYPE is a type descriptor as accepted by `cl-typep', which see."
           `(pred (pcase--flip cl-typep ',type))
         `(pred (cl-typep _ ',type))))))
 
+(make-obsolete-variable 'transient-hide-during-minibuffer-read
+                        "use `transient-show-during-minibuffer-read' instead."
+                        "0.8.0")
+
 (defmacro transient--with-emergency-exit (id &rest body)
   (declare (indent defun))
   (unless (keywordp id)
@@ -269,6 +273,46 @@ of this variable use \"C-x t\" when a transient is active."
   :group 'transient
   :type 'boolean)
 
+(defcustom transient-show-during-minibuffer-read nil
+  "Whether to show the transient menu while reading in the minibuffer.
+
+This is only relevant to commands that do not close the menu, such as
+commands that set infix arguments.  If a command exits the menu, and
+uses the minibuffer, then the menu is always closed before the
+minibuffer is entered, irrespective of the value of this option.
+
+When nil (the default), hide the menu while the minibuffer is in use.
+When t, keep showing the menu, but allow for the menu window to be
+resized to ensure that completion candidates can be displayed.  When
+`fixed', keep showing the menu and prevent it from being resized.
+
+If the value is `fixed' and the menu window uses the full height of its
+frame, then the former is ignored and resizing is allowed anyway.  This
+is necessary because individual menus may use custom display actions,
+which the user may not be aware of.
+
+When using a third-party mode, which automatically resizes windows
+\(e.g., by calling `balance-windows' on `post-command-hook'), then
+`fixed' is likely a better choice than t.
+
+The value can also be an integer, in which case the behavior depends on
+whether at least that many lines are left to display windows other than
+the menu window.  If that is the case, display the menu and preserve the
+size of the window.  Otherwise, hide the menu if the number is positive,
+or allow resizing if it is negative."
+  :package-version '(transient . "0.8.0")
+  :group 'transient
+  :type '(choice
+          (const :tag "Hide menu" nil)
+          (const :tag "Show menu and preserve size" fixed)
+          (const :tag "Show menu and allow resizing" t)
+          (natnum :tag "Show menu, allow resizing if less than N lines left"
+                  :format "\n   %t: %v"
+                  :value 20)
+          (integer :tag "Show menu, except if less than N lines left"
+                   :format "\n   %t: %v"
+                   :value -20)))
+
 (defcustom transient-read-with-initial-input nil
   "Whether to use the last history element as initial minibuffer input."
   :package-version '(transient . "0.2.0")
@@ -390,17 +434,6 @@ text and might otherwise have to scroll in two dimensions."
   :group 'transient
   :type 'boolean)
 
-(defcustom transient-hide-during-minibuffer-read t
-  "Whether to hide the transient buffer while reading in the minibuffer.
-
-This is only relevant to commands that do not close the menu, such
-as commands that set infix arguments.  If a command exits the menu,
-and uses the minibuffer, then the menu is always closed before the
-minibuffer is entered, irrespective of the value of this option."
-  :package-version '(transient . "0.8.0")
-  :group 'transient
-  :type 'boolean)
-
 (defconst transient--max-level 7)
 (defconst transient--default-child-level 1)
 (defconst transient--default-prefix-level 4)
@@ -2442,24 +2475,38 @@ value.  Otherwise return CHILDREN as is."
 (defun transient--suspend-override (&optional nohide)
   (transient--debug 'suspend-override)
   (transient--timer-cancel)
-  (cond ((and (not nohide) transient-hide-during-minibuffer-read)
-         (transient--delete-window))
-        ((and transient--prefix transient--redisplay-key)
-         (setq transient--redisplay-key nil)
-         (when transient--showp
-           (if-let ((win (minibuffer-selected-window)))
-               (with-selected-window win
-                 (transient--show))
-             (transient--show)))))
+  (let ((show (if nohide 'fixed transient-show-during-minibuffer-read)))
+    (when (and (integerp show)
+               (< (frame-height (window-frame transient--window))
+                  (+ (abs show)
+                     (window-height transient--window))))
+      (setq show (natnump show)))
+    (cond ((not show)
+           (transient--delete-window))
+          ((and transient--prefix transient--redisplay-key)
+           (setq transient--redisplay-key nil)
+           (when transient--showp
+             (if-let ((win (minibuffer-selected-window)))
+                 (with-selected-window win
+                   (transient--show))
+               (transient--show)))))
+    (when (and (window-live-p transient--window)
+               (and show
+                    (not (eq show 'fixed))
+                    (not (window-full-height-p transient--window))))
+      (set-window-parameter transient--window 'window-preserved-size
+                            (list (window-buffer transient--window) nil nil))))
   (transient--pop-keymap 'transient--transient-map)
   (transient--pop-keymap 'transient--redisplay-map)
   (remove-hook 'pre-command-hook  #'transient--pre-command)
   (remove-hook 'post-command-hook #'transient--post-command))
 
 (defun transient--resume-override (&optional _ignore)
   (transient--debug 'resume-override)
-  (when (and transient--showp transient-hide-during-minibuffer-read)
-    (transient--show))
+  (cond ((and transient--showp (not (window-live-p transient--window)))
+         (transient--show))
+        ((window-live-p transient--window)
+         (transient--fit-window-to-buffer transient--window)))
   (transient--push-keymap 'transient--transient-map)
   (transient--push-keymap 'transient--redisplay-map)
   (add-hook 'pre-command-hook  #'transient--pre-command)
@@ -3826,14 +3873,6 @@ have a history of their own.")
       (erase-buffer)
       (when transient-force-fixed-pitch
         (transient--force-fixed-pitch))
-      (setq window-size-fixed
-            ;; If necessary, make sure the height of the minibuffer
-            ;; can be increased to display completion candidates.
-            ;; See https://github.com/minad/vertico/issues/532.
-            (if (and (not transient-hide-during-minibuffer-read)
-                     (window-full-height-p))
-                'width
-              t))
       (when (bound-and-true-p tab-line-format)
         (setq tab-line-format nil))
       (setq header-line-format nil)
@@ -3879,7 +3918,11 @@ have a history of their own.")
         ;; Grow but never shrink window that previously displayed
         ;; another buffer and is going to display that again.
         (fit-window-to-buffer window nil (window-height window))
-      (fit-window-to-buffer window nil 1))))
+      (fit-window-to-buffer window nil 1)))
+  (set-window-parameter window 'window-preserved-size
+                        (list (window-buffer window)
+                              (window-body-width window t)
+                              (window-body-height window t))))
 
 (defun transient--separator-line ()
   (and-let* ((height (cond ((not window-system) nil)