[Out of date] Proxmox + ZFS + LXC + Docker (+ Nextcloud AIO)? Fix your Docker storage driver!

[iGadget]

Update 20241104: As per #1490 (reply in thread) this workaround is no longer needed and in fact may produce issues on the latest PVE version(s).

TLDR: Using Proxmox 8.1 / ZFS 2.2 (or newer) no workarounds are needed. overlay2 should be the default driver and should not cause any problems with any docker image.

Old info:

If you're deploying Nextcloud AIO (or probably any other resource intensive Dockerized app) on Proxmox + LXC and your storage is ZFS, you'll soon find that:
a) Performance is poor
b) Your /var/lib/docker dir is HUGE (i.e. 40+ GB after fresh install) and only increasing in size
c) Updating containers via the Nextcloud AIO interface fails

This is because Docker falls back to the vfs storage driver when deployed on ZFS storage.
The vfs storage driver is intended for testing purposes only, serves as a fallback and should not be used in production.
An often cited, relatively easy to deploy alternative is the fuse-overlayfs driver, but there have been reports of PVE nodes crashing when this is used, so I'd avoid that if I were you.

The latest version of the Nextcloud AIO installation script should warn you if the vfs storage driver or fuse-overlayfs driver is being used (as soon as this patch has been released).

Besides the 'bad' examples above, Docker provides several other storage driver options. The zfs option mentioned in the docs seems logical, but I haven't been able to get that working in LXC so far.
So for LXC + Docker on ZFS you'll want overlay2.

The problem here is that overlay2 only supports EXT4 and XFS as backing filesystems, not ZFS. Fortunately, a zvol can be formatted as EXT4 or XFS. But... by default, Proxmox only allows zvols to be used with VMs, not LXCs. For LXC, Proxmox uses ZFS subvols, but ZFS subvols cannot be formatted with a different filesystem.... A catch 22?

Luckily, no. With a few lines of shell code directly on the PVE node and a number of clicks in the PVE WebGUI we can work around this issue:

1. Assuming your cluster storage pool is 'local-zfs', referred to locally as 'rpool/data/' and the LXC is number 104, do
   zfs create -s -V 100G rpool/data/vm-104-disk-1 (yes, that's vm-104, not lxc-104, you read that right. Proxmox weirdiness. If you use anything else than this naming scheme, the zvol will not be picked up by the PVE storage manager)
2. Format the disk with ext4:
   mkfs.ext4 /dev/zvol/rpool/data/vm-104-disk-1
3. Create temp dir and mount the zvol there:
   mkdir /tmp/docker-ext4
mount /dev/zvol/rpool/data/vm-104-disk-1 /tmp/docker-ext4
4. Fix permissions:
   chown -R 100000:100000 /tmp/docker-ext4
5. Stop Docker on the LXC (or shut it down)
6. Move contents of LXC's /var/lib/docker to /tmp/docker-ext4
7. Unmount zvol:
   umount /tmp/docker-ext4
8. Then make sure it's added to LXC 104's config in Proxmox:
   pct rescan
   You should see some feedback in the console and then the should be visible in the Proxmox WebGUI under
   PVE node -> 104 -> Resources as
   'Unused Disk 0 - local-zfs:vm-104-disk-1'.
9. Select the disk, click 'Edit'
10. At 'Path:' enter /var/lib/docker
11. Click 'Create'
12. Start LXC
13. Check output of docker info - it should now state:
    Storage Driver: overlay2
Backing Filesystem: extfs
14. Test if creating a snapshot via WebGUI works
15. If you have a PVE cluster, test if migrating the LXC via WebGUI works
16. Enjoy your faster, leaner and more stable Docker setup :-)

If you have any suggestions for improving the steps above, please share!

[joost00719]

Thank you so much! This worked flawlessly for me. I really appreciate it that you took the time to document this.

[fdcastel]

Thank you @iGadget. I'm using this solution for some time, but until now I haven't found a solution for the snapshots/migration problem. Your vm-* naming scheme was a nice catch! It is working now.

I've published a collection of scripts I'm using here, if anyone is interested.

[joost00719]

Gave you a star! I'm sure I'll be using at least one of them in the future.

[iGadget]

Thanks for your comments, @joost00719 & @fdcastel !
However, it seems there are still some unresolved issues with this approach, just spotted this. Apparently, when you try to restore a backup, the vm-* gets restored as a subvol. Fortunately, the data itself is intact, but requires quite a few steps (described in the link) to convert back to vm-*. @techsolo12 has created a few scripts (which I haven't tried myself yet), but seeing your awesome work, perhaps you can assist in improving those, @fdcastel ?

[techsolo12]

@fdcastel @iGadget Hey Guys! I'm open for any kind of improvement. Since coding isn't one of my skills, I think there are many possible things which can improve. Perhaps I can learn a lot. Just send me an PR or contact me. :)

[fdcastel]

Hi guys!

I already refactored the logic to just rebuild the zfs volume in (this script) which should be used during restore. In my tests it seems to be working fine.

I'm just finishing the the docs for everything. Please give me a little more time and it will all be there. 😉

[fdcastel]

All done, guys! Instructions here.

Please test and comment (preferably, with new issues in that repository, not here)

Merry Christmas! 🎅

[Sieboldianus]

Just a note: I'll need to look into this more closely, but OP suggestions generally seem to follow what I described here
https://du.nkel.dev/blog/2021-03-25_proxmox_docker/

The difference seems to be the naming schema, which seem to solve the migration/snapshot issues, thanks for sharing!

[iGadget]

The difference seems to be the naming schema, which seem to solve the migration/snapshot issues, thanks for sharing!

@Sieboldianus You're most welcome. Without all the info I gathered from others, including your excellent article, I would never have been able to figure it out. However, please also check @fdcastel 's scripts mentioned above as they work around the issue(s) which might (will?) arise when restoring backups.

[frenzymind]

Hello. It seems we no need these dances with tambourine anymore.
I try fresh installation of pve, create lxc container, install docker. Pve host on zfs. Container unprivileged and nesting enabled.
It shows me overlay2 is using right from this point:

docker info | grep -A 7 "Storage Driver:"
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd

Yes, it still has some sort of issues as mentioned here porxmox forum and here du.nkel.dev, but they related to unprivileged mode.

[11:36:14 2023] overlayfs: upper fs does not support RENAME_WHITEOUT.
[11:36:14 2023] overlayfs: failed to set xattr on upper
[11:36:14 2023] overlayfs: ...falling back to xino=off.
[11:36:17 2023] overlayfs: upper fs does not support RENAME_WHITEOUT.
[11:36:17 2023] overlayfs: fs on '/var/lib/docker/metacopy-check1870527761/l1' does not support file handles, falling back to xino=off.

I run some docker containers and it seems really overlay2 is working well. It give me 1.6G instead of 6,9G used space with vfs for the same docker-compose setup file.
1.6G /var/lib/docker/overlay2
6,9G /var/lib/docker/vfs

Snapshots and vzdump backups is working too.

@iGadget @Sieboldianus What do you think? Proxmox stuff make it easy for now? Or I miss something?
Thanks for answers!

Here is PVE versions

proxmox-ve: 7.3-1 (running kernel: 5.15.102-1-pve)
pve-manager: 7.3-6 (running version: 7.3-6/723bb6ec)
pve-kernel-helper: 7.3-7
pve-kernel-5.15: 7.3-3
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-2
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-6
libpve-storage-perl: 7.3-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.5
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20221111-1
pve-firewall: 4.2-7
pve-firmware: 3.6-4
pve-ha-manager: 3.5.1
pve-i18n: 2.8-3
pve-qemu-kvm: 7.2.0-7
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

[Sieboldianus]

Thanks for sharing! This looks good. I will definitely give it a try.

There was one report that points to some potential issue with the ZFS storage driver:

Hi folks,

i too started a few containers and most of them just run fine. But with that one:
"ghcr.io/paperless-ngx/paperless-ngx:latest" i get a :

/usr/local/lib/python3.9/site-packages/pkg_resources/tests/data/my-test-package-source: 
invalid argument

The same docker-compose file works both on overlay2 with ext4 and xfs. Only with
overlay2 backed by zfs the stack would not start. So there may still be some workarounds
needed.

I did not verify this. Could be some hardlink issue or something.

[tcamargo]

Same(?) problem with pihole/pihole.

ee827dbc8bf8: Extracting [==================================================>]  30.34MB/30.34MB
6c22f91571c7: Download complete
67421bc419c4: Download complete
ERROR: failed to register layer: ApplyLayer exit status 1 stdout:  stderr: unlinkat /var/cache/apt/archives: invalid argument

[fdcastel]

Just passing to say that I did a PVE fresh install and I can confirm @frenzymind findings: 🙌

# docker info | grep -A 7 "Storage Driver:"
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd

Remember to create the LXC container with --features keyctl=1,nesting=1 (more info here, search for keyctl=<boolean>)

It appears the solution came with Docker Engine v23.0 but I'm not sure.

I didn't made any more exhaustive tests, yet. I will build a test environment with my production containers in the following days and will report back here in case of problems.

[Zoey2936]

you are right: moby/moby#42661

[fdcastel]

you are right: moby/moby#42661

Yeah. But I find strange that this PR was merged... back in 2021 (!?)

[Zoey2936]

I don't know what their release strategy is… But yes, that is strange.

[fdcastel]

Same problem here. :( Pihole image running on unprivileged / keyctl=1 / nesting=1 LXC ```bash Pulling pihole (pihole/pihole:latest)... latest: Pulling from pihole/pihole bb263680fed1: Pull complete de9654fb76c7: Pull complete 4f4fb700ef54: Pull complete cbe6380c3a6b: Pull complete 34575e9e3344: Pull complete 44048c8f578a: Pull complete ee827dbc8bf8: Extracting [==================================================>] 30.34MB/30.34MB 6c22f91571c7: Download complete 67421bc419c4: Download complete ERROR: failed to register layer: ApplyLayer exit status 1 stdout: stderr: unlinkat /var/cache/apt/archives: invalid argument ```

[techsolo12]

I can confirm that paperless not work as expected Luke in this topic at proxmox forum.
I got exactly the same error.

[fdcastel]

What we've found until now:

This problem appears to be caused by UIDs out of allowed ranges used in lower images:

moby/moby#43576 (comment)

https://gitlab.com/Shinobi-Systems/Shinobi/-/issues/452

[martin-77]

Install works with /var/lib/docker formatted as ext4 vm-disk, but warning: if you backup and restore, the disk gets restored as normal zfs subvol and crashes any updates of the container.
Result: Will install it as regular VM until this is fixed.

[ammgws]

Isn't that doc out of date now that docker works fine with overlay2 on zfs?

>sudo docker info
...
Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 23.0.2
 **Storage Driver: overlay2**
  **Backing Filesystem: zfs**
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true

[tcamargo]

Overlay2 on ZFS is supported, but there are issues with some docker images and unprivileged lxc containers. See #1490 (comment) and #1490 (reply in thread).

[ammgws]

I came across that as well, but if you are able to build the docker image in the LXC itself then the uid/gid issue doesn't occur.

[martin-77]

Do you know, how to build the aio-image yourself? Or do you have a guide?

[Sieboldianus]

It should all be in the repository. docker compose build .

[frenzymind]

I try to make some kind of conclusion here about failed layer registered error.

I get 'failed to register layer' error with overlay2 and zfs

Overlayfs is not supported in 2.1.x versions of zfs : read here
Officially it will work out of the box in zfs 2.2.0, which in release candidate state for now: releases
So, if you get this error - you can do nothing.
One possible exception if your error related to uid/gid mapping (lchown in error text), then you can fix it with increasing uid/gid ranges like here
If problem with unlinkat (I met only this, may be there are others exist) again - you can do nothing.

Why docker says it works then ?

Generally docker is lie here and incorrectly let you pick overlay2 driver. dmesg shows in log that something is wrong.

[136965.616563] overlayfs: upper fs does not support RENAME_WHITEOUT.
[136965.616624] overlayfs: fs on '/var/lib/docker/overlay2/l/DKSJIGBRZ2GYTCO7RX44W4XALM' does not support file handles, falling back to xino=off.

Some of the system calls docker use don't work in every scenario. That why some images works, but others don't.
Things might appear to work with rootfull container, but that should be an illusion.
Read this two messages: one and two

What to do ?

1. If you have no errors with layers, use that images. I have some installations with overlay2 and nothing horrible happening with that.

2. This guide from iGadget and this guide from Sieboldianus Running docker inside an unprivileged LXC container on Proxmox works well and let you escape 'failed to register layer' error.

But remember: In both 1 and 2 cases there is still some dirt happening under the hood, even it looks fine outside.

3. Fallback to vfs

4. Use vm

Hope this helps. Thanks.

[fdcastel]

For the sake of completeness, I would add a third point to your list:

c) LXC allows you to use Bind Mount Points to share files between host and container.

[cpadil]

@fdcastel thanks for the follow up on this topic, it has been really helpful
I just updated to Proxmox 8.1.. upgraded the zfs pool... and planning to rollback the workaround for the LXC + Docker + ZFS but I was wondering if the LXC has to be privileged or can it be unprivileged?

[fdcastel]

@cpadil Wait! If you upgraded to Proxmox 8.1 you already have ZFS 2.2 which DOESN'T NEED any workaround at all!

You may use both privileged and unprivileged containers. Prefer the later whenever possible.

[fdcastel]

Sorry. My bad. I read your sentence too quickly.

You ALREADY HAVE the workaround and now you wish to REMOVE IT. Fine! 😉

[cpadil]

@fdcastel thanks, I just removed the workaround, the unprivileged LXC is just using regular zfs fs, docker seems to be running fine....

however, I still see this errors in the pve host

overlayfs: fs on '/var/lib/docker/overlay2/l/XXX' does not support file handles, falling back to xino=off.

I've tried to restart the LXC but those warnings remain

Found this post from someone in the same situation:
openzfs/zfs#9414 (comment)

Hopefully those warnings are not a concern

[SciLor]

I am having trouble using it.

I am on PVE8.
When trying to boot my Docker LXC I get following error:
TASK ERROR: zfs error: cannot open 'rpool/proxmox/container/subvol-2111-disk-4': operation not applicable to datasets of this type

I need to change the /etc/pve/lxc/*.conf so I am using /dev/zvol/rpool/proxmox/container/subvol-2111-disk-4 instead of subvol-2111-disk-4. Then it started working fine.

[db260179]

Just thought i add my dirty solution for this on the promox lxc unprivileged mode:

Basically if you want to maintain the zfs volumes and use docker in lxc unprivileged mode, then the simple fix is just mount the

zfs volume as a directory i.e. a mounted zfs volume is called /apps in proxmox host, so just create a folder /apps/containers
(adding this in the web ui will do this for you), then move the container root-disk to this folder in the web ui config of the lxc container - this will create a extfs raw image file thus making docker happy again.

[qupfer]

I don't like these loopback-ext4-workarounds, so I switched to fuse-overlayfs again.
Enable the "fuse" checkbox in the container options via webgui (or on cli).

In the lxc container:
systemctl stop docker
rm -rf /var/lib/docker
apt -y install fuse-overlayfs and add to /etc/docker/daemon.json

{
  "storage-driver": "fuse-overlayfs"
}

reboot.

Somewhere I read taht fuse-overlayfs is also needed on the lxc host (proxmox) itself.

[fdcastel]

Nice to know it works for you.

BTW: And how is the performance?

Because "File System" and "Userspace" should never appear in the same sentence.

[lorenzo93]

I seems that ZFS 2.2.0 (that was released yesterday) will solve all our problems!
Does anyone know how much does it take to bring the package from the github release to the proxmox repos?

[AlexanderMcColl]

Just here to say thank you so much for sharing your knowledge, worked perfectly first time to get Immich up and running in a container for me.
No way I could dedicate the 4 cores and 6GB of RAM it wants to a VM on my humble homelab.
Now running happily in an unpriv container. Can't wait for ZFS 2.2 to hopefully make this redundant.

[pongraczi]

It seems ZFS 2.2 is available in testing:
https://forum.proxmox.com/threads/docker-support-in-proxmox.27474/page-7#post-603240

[pongraczi]

More info here:
https://forum.proxmox.com/threads/opt-in-linux-6-5-kernel-with-zfs-2-2-for-proxmox-ve-8-available-on-test.135635/

[techsolo12]

Proxmox VE 8.1 was yesterday released with official support for ZFS 2.2.x and kernel 6.5 included (interesting for AMD servers).
You only have to upgrade your zfs pools after a apt update && apt dist-upgrade -y to the new version with a simple zpool upgrade <zfspool> for each pool. Please verify before and after upgrade the health status of the pools with a zpool status <zfspool>

[johntdavis84]

Is there documentation from Proxmox on upgrading the zpools?

I've upgraded to PVE 8.1 but didn't see anything about upgrading the zpools in any instructions. I've never done it before and I'm a bit unclear on why it's necessary/what it does.

EDIT: My understanding after some googling is that upgrading the zpools enables them to use features provided by new versions of ZFS that otherwise aren't enabled yet because the zpools were created on older versions of ZFS.

I went ahead and did it on my Proxmox 8.1 host, and it seemed to work fine. (I did it with LXCs and VM instances running.)

Everything's still working after a reboot, as well.

[Sieboldianus]

Yes, you should just be careful upgrading the root pool if you enabled unusual features and if you are using grub as the bootloader (the default). You can upgrade selected pools by specifying the name: zpool upgrade tank_hdd, zpool upgrade tank_ssd, zpool upgrade rpool. This can be done while the pool/server is mounted/ running.

[fdcastel]

@Sieboldianus

Besides, I observed an interesting heavy drop in memory consumption after the 2.2.0 upgrade on my Proxmox (48 TB ZFS pools running 18 docker services in nested unprivileged LXC):

1) If you did a *clean install *using the latest proxmox ISO it may be due to [this change](https://bugzilla.proxmox.com/show_bug.cgi?id=4829) From Proxmox VE Changelog: The arc_max parameter for installations on ZFS can now be set in the Advanced Options. If not explicitly set by the user, it is set to a value targeting 10% of system memory instead of 50%, which is a better fit for a virtualization workload (issue 4829). 2) However, if you did an *in-place upgrade* I believe -- to the best of my understanding -- that the `arc_max` parameter shouldn't be changed. *Edit: Bad formatting due reply-by-email.*

[Sieboldianus]

This was an in-place upgrade. The arc_max parameter is unset:

cat /sys/module/zfs/parameters/zfs_arc_max
> 0

So perhaps you are right and they simply lowered the arc_max default memory consumption, if nothing was set here. I don't complain, now I can finally setup mailcow 👍

[webysther]

If you're deploying Nextcloud AIO (or probably any other resource intensive Dockerized app) on Proxmox + LXC and your storage is ZFS, you'll soon find that: a) Performance is poor b) Your /var/lib/docker dir is HUGE (i.e. 40+ GB after fresh install) and only increasing in size c) Updating containers via the Nextcloud AIO interface fails

This is because Docker falls back to the vfs storage driver when deployed on ZFS storage. The vfs storage driver is intended for testing purposes only, serves as a fallback and should not be used in production. An often cited, relatively easy to deploy alternative is the fuse-overlayfs driver, but there have been reports of PVE nodes crashing when this is used, so I'd avoid that if I were you.

The latest version of the Nextcloud AIO installation script should warn you if the vfs storage driver or fuse-overlayfs driver is being used (as soon as this patch has been released).

Besides the 'bad' examples above, Docker provides several other storage driver options. The zfs option mentioned in the docs seems logical, but I haven't been able to get that working in LXC so far. So for LXC + Docker on ZFS you'll want overlay2.

The problem here is that overlay2 only supports EXT4 and XFS as backing filesystems, not ZFS. Fortunately, a zvol can be formatted as EXT4 or XFS. But... by default, Proxmox only allows zvols to be used with VMs, not LXCs. For LXC, Proxmox uses ZFS subvols, but ZFS subvols cannot be formatted with a different filesystem.... A catch 22?

Luckily, no. With a few lines of shell code directly on the PVE node and a number of clicks in the PVE WebGUI we can work around this issue:

1. Assuming your cluster storage pool is 'local-zfs', referred to locally as 'rpool/data/' and the LXC is number 104, do
   zfs create -s -V 100G rpool/data/vm-104-disk-1 (yes, that's vm-104, not lxc-104, you read that right. Proxmox weirdiness. If you use anything else than this naming scheme, the zvol will not be picked up by the PVE storage manager)
2. Format the disk with ext4:
   mkfs.ext4 /dev/zvol/rpool/data/vm-104-disk-1
3. Create temp dir and mount the zvol there:
   mkdir /tmp/docker-ext4
mount /dev/zvol/rpool/data/vm-104-disk-1 /tmp/docker-ext4
4. Fix permissions:
   chown -R 100000:100000 /tmp/docker-ext4
5. Stop Docker on the LXC (or shut it down)
6. Move contents of LXC's /var/lib/docker to /tmp/docker-ext4
7. Unmount zvol:
   umount /tmp/docker-ext4
8. Then make sure it's added to LXC 104's config in Proxmox:
   pct rescan
   You should see some feedback in the console and then the should be visible in the Proxmox WebGUI under
   PVE node -> 104 -> Resources as
   'Unused Disk 0 - local-zfs:vm-104-disk-1'.
9. Select the disk, click 'Edit'
10. At 'Path:' enter /var/lib/docker
11. Click 'Create'
12. Start LXC
13. Check output of docker info - it should now state:
    Storage Driver: overlay2
Backing Filesystem: extfs
14. Test if creating a snapshot via WebGUI works
15. If you have a PVE cluster, test if migrating the LXC via WebGUI works
16. Enjoy your faster, leaner and more stable Docker setup :-)

If you have any suggestions for improving the steps above, please share!

How much poor we talk about? With NVME (Ext4), bare metal nextcloud don't get less than 1sec to open a page...

[fdcastel]

@webysther Many things have changed / improved since this message you are replying to was posted.

But suffice to say: if you are using vfs (mainly due to an old version of ZFS / Proxmox) you're gonna have a baaaad time.

As @iGadget said, it was never meant to be used in production. For example: Your /var/lib/docker dir will always grow with each new layer / image.

[thmsdelange]

I've been reading through this discussion and several others on the proxmox forums for a few days but I fail to understand what part of the workaround applies to the old and the new (ZFS 2.2) situation.

I am running proxmox 8.1.3 and upgraded my pools to ZFS 2.2, created an unprivileged LXC with its root filesystem on a ZFS pool (I did not create a zvol, just selected the pool). Then I followed the steps mentioned in [https://du.nkel.dev/blog/2021-03-25_proxmox_docker/](this blog) until right before the "ZFS" section (as this should work natively with ZFS 2.2, right?).

I am confused how I should setup the environment, do I use the ZFS docker storage driver or overlay2? The output of sudo docker info | grep -A 7 reads

"Storage Driver:"
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd

Do I need to do something else? Please enlighten me

[fdcastel]

The workarounds are not needed for ZFS 2.2.

You should use overlay2. To the best of my knowledge docker zfs storage driver never took off.

Using Proxmox 8.1 / ZFS 2.2 no workarounds are needed. overlay2 should be the default driver and should not cause any problems with any docker image.

For the full story, read this.