Turn all applications into application bundles

[probonopd]

Point in case:
From one day to the next, Firefox stopped working because I had installed some other applications using pkg install, which had seemingly upgraded some system libraries, resulting in the old Firefox version no longer working properly.

Hence I was factually forced to update Firefox as well.

As a result, meet.jit.si no longer works in Firefox, now giving this:

about:config, network.http.spdy.enforce-tls-profile makes Firefox crash.

It had been working before.

There is no obvious way to downgrade Firefox. (Correction: actually there is the firefox-esr package, but that's only the case for very, very few apps.)

This is exactly what makes me dislike the concept of traditional package managers for managing graphical user-facing applicatiions.

For this exact reason we need to come up with a system that turns each application into an Application Bundle and makes it possible to:

• Get new applications without changing the rest of the system, especially without affecting other applications
• Keep multiple versions of the same applications around in case applications suddenly change in unwanted ways (example given above)

How to do this best, in a way that works for most already-ported applications without too much manual effort?

cc @mszoek

[probonopd]

As for using Jitsi Meet on FreeBSD (helloSystem):

sudo pkg install firefox-esr # 78.14.0,1
sudo killall pulseaudio
firefox

and then doing about:config, network.http.spdy.enforce-tls-profile false makes it work!

[mszoek]

I am not sure there is a simple or easy way to package things into Bundles. In general, the build script/makefile has to specifically target a Bundle. One of the things I hope airyxOS will accomplish is to make it a more common way to package software.

That said... have a look at what GoboLinux does. A scheme like that lets multiple versions of things co-exist. Although they are not movable like true Bundles or AppDirs are, something like this is 80% of the way to a bundle. The rest could be a clever AppRun script that sets up the right LD_LIBRARY_PATHs and launches the app, plus whatever app-specific tweaking is needed for each one to work properly.

If your libraries are all done the same way, apps won't break unless you remove the versions they use. You can add new software without affecting old.

Maybe something like this could be a start towards making "real" bundles?

[bitfanatic]

Disclaimer:
I am a noob but interested Linux so if im wrong
Just forget what i wrote.

I think using appimages would solve this problem.
Not sure if appimages Work on BSD

Side note:
I recently tried out Nitrux 1.6
They have made their own software manager
Which only have appimages on it
Very good experience.

[kettle-7]

AppImages use the same bundle concept, only they are in one file that gets mounted as a disk instead of a folder.

[kettle-7]

So in response to your comment, yeah, that's the idea.

[kettle-7]

Unfortunately, many apps don't respond well to being put in bundles, or AppImages, because they use hard-coded paths and stuff. Also, probono made AppImage, so if that was the canonical option, he would have gone with it.

[bitfanatic]

Lol your right i overlooked that iceberg🤣

I Will go back to being silent

[probonopd]

AppImages are basically a workaround for certain constraints on the "Desktop Linux Platform", which in reality is a giant mess of different distributions that cannot agree on anything and are made by competing vendors, and most of which won't put any efforts into making application bundles a reality.

On FreeBSD/helloSystem, we have another set of constraints, which allows other solutions. Here, the concern is more how we can re-use the large catalog of ports and packages by making them into bundles without too much manual interaction.

GoboLinux needs everything to be recompiled. I don't have the resources for that. I am looking for ways to convert existing packages. Thinking along the lines of overlay/union filesystems, user-mode chroots, LD_PRELOAD filesystem redirections, and things like that.

[kettle-7]

We maybe could use a chroot sandbox, that way the apps are effectively installed but in a folder you can move anywhere, and we could have some "common" filesystems containing base stuff a little bit like flatpak runtimes, but there'd need to be less of them.

[probonopd]

Exactly - I was thinking about something along those lines. Especially since non-root chroots are said to be coming to FreeBSD.

[kettle-7]

I'd like to know more about these 'non-root chroots'

[kettle-7]

In general, the build script/makefile has to specifically target a Bundle.

Yes and no. It's relatively simple to bundle most autotools or cmake based packages. Firefox is little short of a nightmare, and Thunderbird to go with it. I haven't figured out a good solution for qmake yet, and of course nothing works anywhere other than /usr if it uses hardcoded paths to such locations. Most packages that can go in /opt can be bundled.

[probonopd]

Making a proof-of-concept Firefox.app

cc @mszoek

Say I have the firefox-esr package installed on the system but for whatever reason I need to run Firefox 92.0 as well (without messing with the system-wide installed package).

We can produce a simple application bundle from the existing pkg with:

mkdir -p "Firefox 92.0.app/Resources"
cd "Firefox 92.0.app/Resources"
wget "http://pkg.FreeBSD.org/freebsd:12:x86:64/quarterly/All/firefox-92.0_2,2.txz"
tar xf *.txz
rm *.txz
cp usr/local/share/pixmaps/firefox.png "Firefox 92.0.png"
cd ..
cat > "Firefox 92.0" <<\EOF
#!/bin/sh

HERE="$(dirname "$(readlink -f "${0}")")"
exec "${HERE}/Resources/usr/local/lib/firefox/firefox" "$@"
EOF
chmod +x "Firefox 92.0"
cd ..
zip --symlinks -r "Firefox 92.0.app.zip" "Firefox 92.0.app"
mv "Firefox 92.0.app" /Applications
launch "Firefox 92.0" # launch bug fixed in https://github.com/helloSystem/launch/commit/43567d61f2054930a3c5c8c831f310044d26d077, will be available from ISO build 0F51 on

Works :+1

We can now open Firefox 92.0 from the System menu, and can use Firefox 92.0.app.zip to transport the application bundle to another machine.

Now, the real fun begins when we need to bundle dependencies as well.
We can download all of them with pkg fetch -d -o ~/ffbundle firefox but this will download many more packages than we actually need, because it does not consider which packages are already part of the system.

So, we need a way to calculate the dependencies taking into consideration the set of packages that helloSystem comes with (not the set of packages that happen to be locally installed at the point of making the application bundle, or else the application bundle will only run on that very machine).

How can we do this?

[probonopd]

@mszoek

I would try using a chrooted package build where the chroot environment is a pristine helloSystem install. That way you will only get packages that are not part of the default system.

That's certainly one way of doing it, albeit a high-overhead one (every installed, "historically grown" helloSystem installation would need to keep a pristine helloSystem image around just for the purpose of making application bundles).

If I am not mistaken, pkg keeps the information about what has been installed on the system in a sqlite database. If we can trick it into using the sqlite database of a pristine helloSystem ISO, then we should be golden - wdyt?

[probonopd]

So basically there's a read-only directory containing a pristine helloSystem copy, that is mounted from an image file, then using unionfs mounts we could get the root filesystem used by the system, with /home, /var, /etc etc. mounted from separate ZFS subvolumes, /tmp mounted tmpfs, and use a similar overlay for apps.

This concept will become really interesting once there is non-root (unprivileged) chroot. If I remember correctly, @emaste mentioned recently that this is in the works. (People have been dreaming about this for almost 20 years.)

[mszoek]

That's certainly one way of doing it, albeit a high-overhead one (every installed, "historically grown" helloSystem installation would need to keep a pristine helloSystem image around just for the purpose of making application bundles).

If I am not mistaken, pkg keeps the information about what has been installed on the system in a sqlite database. If we can trick it into using the sqlite database of a pristine helloSystem ISO, then we should be golden - wdyt?

That's essentially why a chroot works. It's an alternate version of the package DB and installed packages.

I must have misunderstood what you're trying to do here because the comment about every helloSystem install needing a pristine chroot puzzles me. Are you trying to dynamically make bundles on the end-user system? I had assumed we were talking about making app bundles on a build system for distribution.

If the goal is just to figure out the delta in packages from "pristine install" then yes, running something like pkg install --dry-run or pkg info -d could give you a list of dependent packages to install. pkg install --fetch-only might also be useful. You can force pkg to run against an alternate database by setting env var PKG_DBDIR

[grahamperrin]

#203 (comment)

non-root chroots are said to be coming to FreeBSD.

https://cgit.freebsd.org/src/commit/?id=a40cf4175c90142442d0c6515f6c83956336699b was committed in July.

[grahamperrin]

Firefox stopped working

#161 (comment)

meet.jit.si no longer works in Firefox,

https://user-images.githubusercontent.com/192271/133346768-0d240cc8-06fb-4544-a86c-ae4a4d8f5692.png

about:config, network.http.spdy.enforce-tls-profile makes Firefox crash.

No such crash here, on any system. The given preference is a default.

[kettle-7]

If the goal is just to figure out the delta in packages from "pristine install" then yes, running something like pkg install --dry-run or pkg info -d could give you a list of dependent packages to install. pkg install --fetch-only might also be useful. You can force pkg to run against an alternate database by setting env var PKG_DBDIR

Yes, the idea is that you have one pristine root folder (ideally mounted read-only from an image) and every individual app, as well as the system itself, (although the latter would probably need an initramfs, which I'm not sure FreeBSD does) and you union mount everything on that.

[kettle-7]

That's certainly one way of doing it, albeit a high-overhead one (every installed, "historically grown" helloSystem installation would need to keep a pristine helloSystem image around just for the purpose of making application bundles).

High overhead? If flatpak had one runtime for a clean, say, Fedora install the overhead would be tiny, particularly if it was the same set of files ostree used for the root filesystem.

[probonopd]

I must have misunderstood what you're trying to do here because the comment about every helloSystem install needing a pristine chroot puzzles me. Are you trying to dynamically make bundles on the end-user system? I had assumed we were talking about making app bundles on a build system for distribution.

Indeed, my idea is that the user runs app install firefox (or something like this) on the local machine, and an application bundle is created on the local machine from the official FreeBSD pkgs. Wouldn't that be neat?

alternate database by setting env var PKG_DBDIR

That's it! Thanks a lot.

[mszoek]

It would be a cool hack for sure. So the idea is to unpack the app pkg and relocate its files into an AppDir type structure, plus install any dependent packages?

[probonopd]

Exactly. Unpack into an AppDir, along with all dependencies that are not part of the ISO.

[kettle-7]

Then maybe we could make a graphical frontend for pkg...

...like the AppShop was going to be ;~)

[grahamperrin]

… downgrade …

find /var/cache/pkg -type file -print

pkg-add(8)

[probonopd]

Work has begun to produce application bundles.
Call for testers: #227

...like the AppShop was going to be ;~)

Maybe it can be a frontend for https://github.com/helloSystem/Applications/releases/ one day?

[kettle-7]

Maybe it can be a frontend for https://github.com/helloSystem/Applications/releases/ one day?

Potentially. But having seen your script, there's a lot of conditional branches being run for specific apps. So long as that is used, with every single app made you end up having to keep adding new pieces of code for tiny compatibility things. I'd imagine if you wanted it on a reasonable scale as far as how many apps there are, that is not the way to go about it. I do see that much of that script is copying certain icons, but I could make a python program to parse the freedesktop stuff and then it's just a matter of running that. Or maybe you could rewrite the shell script in python and add that functionality. I'm a little busy with exams at the moment but should soon be free to do more work for this project.

Anyway, any case in which you need to change the core components (as this I believe shall one time soon become) in order to tailor for specific apps will, not can, but will, present problems.

[probonopd]

https://github.com/helloSystem/Applications/blob/2c76829ab12b20aa808ced41211cdf6d5df736b3/makebundle#L8-L13

#### TODO:
#### Before investing much more time into this, consider rewriting it in Python
#### (port to /usr/local/bin/app)
#### because right now it fails on blanks in directory names and is cumbersome
#### to write and maintain in sh - and we have existing Python code for many
#### aspects including GUI

I could make a python program to parse the freedesktop stuff and then it's just a matter of running that.

In fact, we already have that in /usr/local/bin/app...

The bash script is really a proof-of-concept because it was quicker to code.
The real thing should be done properly, and Python is well-suited.

[probonopd]

It's also the question whether we want to run this on the server or on the local machine (with a GUI like the current application installer GUI).

[kettle-7]

Ideally server side if it takes a long time and works only for specific apps, but if you can get it to work universally, then it should be run client side to ease the demand on the server.

[probonopd]

https://github.com/therontarigo/freebsd-user-namespace may also be useful for situations in which an application insists on loading its resources from compiled-in absolute paths:

Intercepts libc filesystem operations to hide the real filesystem from the process and provide instead a configurable namespace mapping virtual locations to real locations.

This works by using LD_PRELOAD.

Reference:
therontarigo/freebsd-user-namespace#1

[grahamperrin]

Turn all applications into application bundles

Will this increase the size of the ISO? helloSystem/ISO#80

one day

helloSystem/ISO#141

[grahamperrin]

Cross-reference: #272