xshift/yshift do not apply to anchors when set layers is used

[schtandard]

When using set layers in a tikzpicture and shifting an axis using xshift/yshift, that shift is not applied to the axis' anchors.

\documentclass{article}

\usepackage{pgfplots}
\pgfplotsset{compat=1.18}

\begin{document}

\begin{tikzpicture}
  \pgfplotsset{set layers}
  \begin{axis}[name=ax, xshift=1cm, yshift=0.5cm]
    \addplot+ {x^2};
  \end{axis}
  \draw[red] (ax.south west) rectangle (ax.north east);
\end{tikzpicture}

\end{document}

I believe this is a bug.

(related tex.sx question)