buildkit rootless 0.17.0 with fuse-overlayfs giving setxattr user.overlay.impure operation not permitted errors #5478
Comments
|
Please post a (minimized) complete reproducer. Is this issue specific to Dockerfiles that refer to |
|
I'll see what I can do to get a reproducer. Is there any reason to believe this would be a problem with 0.17.0 but not 0.16.0 Nothing else seemed to change and rolling back the buildkit image works fine. I couldn't see anything in the 0.17.0 changelog that might obviously imply a change here, other than perhaps the go lib updates to various things. |
|
There's a relatively minimal reproducer at https://github.com/chadlwilson/buildkit-rootless-issue incl GHA workflow that shows the problem with an Ubuntu 24.04 host.
Actions run that shows error: https://github.com/chadlwilson/buildkit-rootless-issue/actions/runs/11667854159/job/32486137082 Diff of docker info for $ diff official-docker.txt centos-docker.txt
1c1
< Client:
---
> Client: Docker Engine - Community
8,12c8
< Path: /usr/local/libexec/docker/cli-plugins/docker-buildx
< compose: Docker Compose (Docker Inc.)
< Version: v2.30.1
< Path: /usr/local/libexec/docker/cli-plugins/docker-compose
<
---
> Path: /usr/libexec/docker/cli-plugins/docker-buildx
20,25c16
< Storage Driver: overlay2
< Backing Filesystem: extfs
< Supports d_type: true
< Using metacopy: false
< Native Overlay Diff: false
< userxattr: false
---
> Storage Driver: vfs
45c36
< Operating System: Alpine Linux v3.20
---
> Operating System: CentOS Stream 9
50,51c41,42
< Name: 77e260121212
< ID: 65e06b61-35b9-489b-8042-db38e2593de5
---
> Name: f807bea0881d
> ID: 5d1f9b14-1f29-422e-a6e8-9701ca8b4bd6Diff of buildkit config on 3,4c3
< Last Activity: 2024-11-04 15:29:21 +0000 UTC
<
---
> Last Activity: 2024-11-04 15:43:50 +0000 UTC
15c14
< org.mobyproject.buildkit.worker.hostname: fdc4515ccde9
---
> org.mobyproject.buildkit.worker.hostname: c226a50182b5
19c18
< org.mobyproject.buildkit.worker.snapshotter: overlayfs
---
> org.mobyproject.buildkit.worker.snapshotter: fuse-overlayfsNot quite sure what to conclude about these differences:
|
chadlwilson
changed the title
v0.17.1 (now stable) has regressions: moby/buildkit#5478
|
v0.17.1 has the same problem (and perhaps surprisingly was marked as stable) |
|
Can you try if this works? diff --git a/vendor/github.com/containerd/containerd/archive/tar.go b/vendor/github.com/containerd/containerd/archive/tar.go
index c61f89ec8..4c16ee810 100644
--- a/vendor/github.com/containerd/containerd/archive/tar.go
+++ b/vendor/github.com/containerd/containerd/archive/tar.go
@@ -408,6 +408,12 @@ func createTarFile(ctx context.Context, path, extractDir string, hdr *tar.Header
key = key[len(paxSchilyXattr):]
if err := setxattr(path, key, value); err != nil {
if errors.Is(err, syscall.EPERM) && strings.HasPrefix(key, userXattrPrefix) {
+ if key == "user.overlay.impure" {
+ // Only occurs with images built with Red Hat's buildah?
+ // https://github.com/moby/buildkit/issues/5478
+ log.G(ctx).WithError(err).Debugf("ignored xattr %s in archive", key)
+ continue
+ }
// In the user.* namespace, only regular files and directories can have extended attributes.
// See https://man7.org/linux/man-pages/man7/xattr.7.html for details.
if fi, err := os.Lstat(path); err == nil && (!fi.Mode().IsRegular() && !fi.Mode().IsDir()) { |
|
I don't have the ability to build from source right now, never done that before. Not sure how easy it is. FWIW, problem was introduced between rc1 and rc2 so somewhere in v0.17.0-rc1...v0.17.0-rc2 |
|
Could you try |
Sure, if i have a build env set up. :) Also need to create a container image to give to buildx as well, and do so consistently with how the images here are built. I don't have a local env to replicate this right now, so have to iterate on cloud infra which is slow. |
Tried bisecting, and can't replicate the issue. The change here that appears to have caused the problem seems to be While it's a bit messy since apks on Alpine can't be downgraded, running on latest buildkit code and downgrading to Alpine
This slight adjustment seemed to work with the newer fuse-overlayfs diff --git a/vendor/github.com/containerd/containerd/archive/tar.go b/vendor/github.com/containerd/containerd/archive/tar.go
index c61f89ec8..2dc53fb1d 100644
--- a/vendor/github.com/containerd/containerd/archive/tar.go
+++ b/vendor/github.com/containerd/containerd/archive/tar.go
@@ -408,6 +408,12 @@ func createTarFile(ctx context.Context, path, extractDir string, hdr *tar.Header
key = key[len(paxSchilyXattr):]
if err := setxattr(path, key, value); err != nil {
if errors.Is(err, syscall.EPERM) && strings.HasPrefix(key, userXattrPrefix) {
+ if key == "user.overlay.impure" || key == "user.overlay.origin" {
+ // Only occurs with images built with Red Hat's buildah?
+ // https://github.com/moby/buildkit/issues/5478
+ log.G(ctx).WithError(err).Debugf("ignored xattr %s in archive", key)
+ continue
+ }
// In the user.* namespace, only regular files and directories can have extended attributes.
// See https://man7.org/linux/man-pages/man7/xattr.7.html for details.
if fi, err := os.Lstat(path); err == nil && (!fi.Mode().IsRegular() && !fi.Mode().IsDir()) { |
|
So anyway, the curl image (in this case) does seem to be the problem, as it's built with buildah So I don't know. Perhaps this is a "won't fix" from the perspective of Docker, moby, fuse-overlayfs etc. But what a PITA. :-) |
chadlwilson
changed the title
When building via buildx our builds starting giving errors like the below after an implicit upgrade to use the
moby/buildkit:v0.17.0-rootlessimage (previouslyv0.16.0-rootless).#9 ERROR: mount callback failed on /run/user/1000/containerd-mount2444245211: failed to setxattr "/run/user/1000/containerd-mount2444245211/etc" for key "user.overlay.impure": operation not permittedEnvironment is
Host OS:
Linux 5.10.219-208.866.amzn2.x86_64 amd64(Amazon Linux 2)Host Docker:
20.10.27(yeah, I know it's EOL - long story)DIND image OS: Centos Stream 9 (if it matters)
DIND image Docker:
27.3.1DIND image Docker buildx plugin:
0.17.1Fuller log
If this is related to the outdated host Docker version and this expected on this configuration, feel free to close/ignore and let me know.
Perhaps related to moby/moby#47605 and/or moby/moby#43626 and use of native overlay?
The text was updated successfully, but these errors were encountered: