The AUR should not be a supported platform

[dvzrv]

Hi! 👋

I package libguestfs for Arch Linux. When investigationg libguestfs/libguestfs#139 I noticed the following:

:: Retrieving packages...
 gcc-libs-13.2.1-5-x86_64 downloading...
 perl-5.38.2-1-x86_64 downloading...
 icu-74.2-2-x86_64 downloading...
 glibc-2.39-1-x86_64 downloading...
 systemd-255.4-2-x86_64 downloading...
 binutils-2.42-2-x86_64 downloading...
 vim-runtime-9.1.0252-1-x86_64 downloading...
 grub-2:2.12-2-x86_64 downloading...
 glib2-2.80.0-2-x86_64 downloading...
 openssl-3.2.1-1-x86_64 downloading...
 util-linux-2.40-2-x86_64 downloading...
 gnutls-3.8.5-1-x86_64 downloading...
 coreutils-9.5-1-x86_64 downloading...
 qemu-common-8.2.2-2-x86_64 downloading...
 gettext-0.22.4-1-x86_64 downloading...
 bash-5.2.026-2-x86_64 downloading...
 lvm2-2.03.23-3-x86_64 downloading...
 sqlite-3.45.2-1-x86_64 downloading...
 hwdata-0.381-1-any downloading...
 pcre2-10.43-3-x86_64 downloading...
 gawk-5.3.0-1-x86_64 downloading...
 krb5-1.21.2-2-x86_64 downloading...
 kbd-2.6.4-1-x86_64 downloading...
 linux-api-headers-6.7-1-any downloading...
 db5.3-5.3.28-4-x86_64 downloading...
 e2fsprogs-1.47.0-2-x86_64 downloading...
 ncurses-6.4_20230520-1-x86_64 downloading...
 shadow-4.15.1-2-x86_64 downloading...
 openssh-9.7p1-1-x86_64 downloading...
 systemd-libs-255.4-2-x86_64 downloading...
 curl-8.7.1-5-x86_64 downloading...
 thin-provisioning-tools-1.0.12-1-x86_64 downloading...
 cdrtools-3.02a09-5-x86_64 downloading...
 procps-ng-4.0.4-3-x86_64 downloading...
 pam-1.6.1-2-x86_64 downloading...
 wolfssl-5.7.0-1-x86_64 downloading...
 libxml2-2.12.6-1-x86_64 downloading...
 cryptsetup-2.7.2-1-x86_64 downloading...
 libcap-2.69-4-x86_64 downloading...
 libunistring-1.2-1-x86_64 downloading...
 xz-5.6.1-3-x86_64 downloading...
 libgcrypt-1.10.3-1-x86_64 downloading...
 libelf-0.191-1-x86_64 downloading...
 libarchive-3.7.3-1-x86_64 downloading...
 libp11-kit-0.25.3-1-x86_64 downloading...
 zstd-1.5.5-1-x86_64 downloading...
 util-linux-libs-2.40-2-x86_64 downloading...
 nettle-3.9.1-1-x86_64 downloading...
 gmp-6.3.0-1-x86_64 downloading...
 iptables-1:1.8.10-1-x86_64 downloading...
 mpfr-4.2.1-2-x86_64 downloading...
 findutils-4.9.0-3-x86_64 downloading...
 libnl-3.9.0-1-x86_64 downloading...
 iana-etc-20240222-1-any downloading...
error: failed retrieving file 'iana-etc-20240222-1-any.pkg.tar.zst' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'iptables-1:1.8.10-1-x86_64.pkg.tar.zst.sig' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'findutils-4.9.0-3-x86_64.pkg.tar.zst' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
warning: too many errors from repos.archlinux.org, skipping for the remainder of this transaction
error: failed retrieving file 'libnl-3.9.0-1-x86_64.pkg.tar.zst' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
error: failed retrieving file 'mpfr-4.2.1-2-x86_64.pkg.tar.zst.sig' from repos.archlinux.org : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds
warning: failed to retrieve some files
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.
--2024-04-15 14:38:02--  https://aur.archlinux.org/packages/cd/cdrtools/cdrtools.tar.gz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving aur.archlinux.org (aur.archlinux.org)... 95.216.144.15, 2a01:4f9:c010:50::1
Connecting to aur.archlinux.org (aur.archlinux.org)|95.216.144.15|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-04-15 14:38:02 ERROR 404: Not Found.

--2024-04-15 14:38:02--  https://aur.archlinux.org/packages/cd/cdrtools/cdrtools.tar.gz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving aur.archlinux.org (aur.archlinux.org)... 95.216.144.15, 2a01:4f9:c010:50::1
Connecting to aur.archlinux.org (aur.archlinux.org)|95.216.144.15|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-04-15 14:38:02 ERROR 404: Not Found.

supermin: set -e
          umask 0000
          cd '/var/tmp/supermind89798.tmpdir/ytx0efty'
          wget 'https://aur.archlinux.org/packages/cd/cdrtools/cdrtools.tar.gz'
          tar xf 'cdrtools.tar.gz'
          cd 'cdrtools'
          /usr/bin/makepkg
          mv 'cdrtools'-*.pkg.tar.xz '/var/tmp/supermind89798.tmpdir/ytx0efty'
       : command failed, see earlier errors
make[2]: *** [Makefile:1080: stamp-supermin] Error 1
make[2]: Leaving directory '/build/libguestfs/src/libguestfs-1.52.0/appliance'
make[1]: *** [Makefile:1089: all-recursive] Error 1
make[1]: Leaving directory '/build/libguestfs/src/libguestfs-1.52.0'
make: *** [Makefile:995: all] Error 2

This appears to be implemented in

supermin/src/ph_pacman.ml

Lines 196 to 234 in 5a44ffc

	if Sys.command cmd <> 0 then (
	(* The package may not be in the main repos, check the AUR. *)
	List.iter (
	fun name ->
	let cmd = sprintf "\
	set -e
	umask 0000
	cd %s
	wget %s
	tar xf %s
	cd %s
	%s
	mv %s-*.pkg.tar.xz %s
	"
	(quote tdir)
	(quote ("https://aur.archlinux.org/packages/" ^
	(String.sub name 0 2) ^
	"/" ^ name ^ "/" ^ name ^ ".tar.gz"))
	(quote (name ^ ".tar.gz"))
	(quote name) (* cd *)
	Config.makepkg
	(quote name) (quote tdir) (* mv *) in
	if !settings.debug >= 2 then printf "%s" cmd;
	run_command cmd
	) names;
	);
	(* Unpack the downloaded packages. *)
	let cmd =
	sprintf "
	umask 0000
	for f in %s/*.pkg.tar.*; do
	[[ $f == *.sig ]] && continue
	tar -xf \"$f\" -C %s
	done
	"
	(quote tdir) (quote dir) in
	if !settings.debug >= 2 then printf "%s" cmd;
	run_command cmd

Frankly, this is quite the horrific approach. The AUR is unsupported for a reason and should never be used in the context of distribution packages.
In this particular case supermin attempts to build a package that is available in the repositories (cdrtools), but not in the AUR.

If the packages can not be retrieved, the build process should not fall back to building from unverified and untrusted sources but instead just fail!
To be more specific: Please do not automatically build from the AUR at all!

[rwmjones]

Sure, we can drop AUR if you like. Are there other packages which are not in the base Arch distribution that we need?

[dvzrv]

That's a good question. Where do I need to look to find out?

Are these the relevant blocks?
https://github.com/libguestfs/libguestfs/blob/6ba64125d960c820b75727e3856b0df0ed6b264c/appliance/packagelist.in#L99-L123
https://github.com/libguestfs/libguestfs/blob/6ba64125d960c820b75727e3856b0df0ed6b264c/appliance/packagelist.in#L243-L298

I know that e.g. zerofree is not packaged for the official repos, but it doesn't appear to be built from the AUR (see https://gitlab.archlinux.org/archlinux/packaging/packages/libguestfs/-/issues/1).

Could you explain which of the packages are hard dependencies?

[rwmjones]

Yes it's those. Supermin actually ignores packages that it doesn't know about (by design), so it's rather hard to know which ones are really required and which aren't. A number of them will just enable optional features in libguestfs but it'll work without them.

zerofree is definitely not important.

I was just wonder if there are major packages in AUR which are not in base Arch that we should know about, however if not I guess we can just try it.

[dvzrv]

Yes it's those.

Just to make sure: Are all those pkgs only build time dependencies?
With libguestfs I always had a hard time figuring out what is build time and what is runtime dependency :S

[dvzrv]

The following dependencies do not exist (neither in the official repositories nor in the AUR):

• ntfs-3g-system-compression
• gostsum
• module-init-tools
• procps (we have procps-ng though)
• util-linux-ng (we have util-linux though and I have never heard of util-linux-ng)
• exfat-fuse
• fuse-exfat

The following only exist in the AUR:

• scrub
• zerofree

[rwmjones]

Yes it's those.

Just to make sure: Are all those pkgs only build time dependencies? With libguestfs I always had a hard time figuring out what is build time and what is runtime dependency :S

They're both. Libguestfs builds a skeleton appliance and fills it in with files from the host filesystem at runtime (https://rwmj.wordpress.com/2014/03/08/supermin-version-5/).

Looking at the list above, I would say the only one which is important is util-linux{-ng}. I'm sure you must be packaging util-linux-ng, but probably it's been renamed as util-linux. Old util-linux was deprecated years ago. (https://en.wikipedia.org/wiki/Util-linux)