VirtioFS does not work on Apple silicon

[jgoerz]

Description

I'm not sure if this is a bug in Docker or in Hashicorp's nomad. Close/reject this at your discretion. I was trying to do nomad's quick start walk through (pytechco) on an Apple M2 Max Macbook.

In case it matters, nomad info:

Nomad v1.7.3
BuildDate 2024-01-15T16:55:40Z
Revision 60ee328f97d19d2d2d9761251b895b06d82eb1a1

Nomad starts up fine, but when running the redis job it fails with:

Driver Failure: Failed to start container
8ee2e5f1585f299ef96828e644fdd0948113caa3a27ef93a859696d78d56a3fb: API error
(500): error while creating mount source path
'/host_mnt/private/tmp/NomadClient3722121065/9365107d-8527-25f7-7096-f24e45fc6b62/alloc':
mkdir /host_mnt/private/tmp/NomadClient3722121065: permission denied

I tried using /etc/synthetic.conf to create a map from the read-only filesystem mount point /host_mnt to something else and VirtioFS still did not work. Neither did gRPC FUSE nor the osxfs (Legacy). In order to make it work, I needed the following:

• Settings => General
  • Unchecked: Use Rosetta for x86/amd64 emulation on Apple Silicon
  • Selected: gRPC FUSE (Choose file sharing implementation for your containers)

Reproduce

1. download and install docker for desktop (Mac) 4.26.1 on Apple Silicon hardware
   • Ensure that VirtioFS is selected as the file sharing implementation
   • Ensure "use rosetta" emulation is turned off (or on, either breaks it with VirtioFS)
2. download and install nomad 1.7.3
3. clone the repo for the nomad quickstart
4. Run the nomad agent in dev mode (creates the cluster)

   sudo nomad agent -dev \
     -bind 0.0.0.0 \
     -network-interface='{{ GetDefaultInterfaces | attr "name" }}'
   

5. Run the redis job

   nomad job run pytechco-redis.nomad.hcl
   

The job will fail because it can't create mounts for the container.

Expected behavior

I would expect VirtioFS to work.

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.7
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.0-desktop.2
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.3-desktop.2
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  0.1
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.10
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/jgoerz/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/jgoerz/.docker/cli-plugins/docker-buildx-orig" is not valid: plugin candidate "buildx-orig" did not match "^[a-z][a-z0-9]*$"
WARNING: Plugin "/Users/jgoerz/.docker/cli-plugins/docker-compose-orig" is not valid: plugin candidate "compose-orig" did not match "^[a-z][a-z0-9]*$"

Server:
 Containers: 5
  Running: 1
  Paused: 0
  Stopped: 4
 Images: 25
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.5.11-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.662GiB
 Name: docker-desktop
 ID: e0e8f8cd-43c1-4802-b3bd-e0cc8411cf81
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

1160199C-6563-4AFA-A51E-013AB0F7E37B/20240125161418

Additional Info

No response

[codymullins]

I'm seeing this error running a job on Apple silicon. Are you still having this issue too?

[chengyuehsha]

I'm seeing this error running a job on Apple silicon. Are you still having this issue too?

I encountered the same problem and Apple silicon too.

[chengyuehsha]

Additionally, I think this issue is the same matter.