Cluster mode without elevated privilages

[balakrishnapenumarti]

Running garnet in cluster mode (within kubernetes) with non-elevated privileges is resulting in

Unable to initialize server due to exception: Access to the path '/app/cluster' is denied.
or
Unable to initialize server due to exception: Read-only file system : '/app/cluster'

depending on the user and filesystems settings.
Is this something already supported with other settings instead of working around with root user ?

[vazois]

Running garnet in cluster mode (within kubernetes) with non-elevated privileges is resulting in

Unable to initialize server due to exception: Access to the path '/app/cluster' is denied. or Unable to initialize server due to exception: Read-only file system : '/app/cluster'

depending on the user and filesystems settings. Is this something already supported with other settings instead of working around with root user ?

Try using --checkpointdir parameter when you start the instance. One option is using /tmp that does not require root permissions

[balakrishnapenumarti]

Works with /tmp but with write access on root file system for a non root user.
Using /data as checkpoint dir works with both read/write config of root filesystem and with a non root user.

[badrishc]

so what is the resolution here? we need to give permission to the default cluster/checkpoint directory? or was this because you specified a different directory that was not covered by the docker file? cc @akoken

[balakrishnapenumarti]

As checkpoint will need write access, but as its cluster managed and should not be modifiable in the scope of non root, I am thinking its best to create a default checkpoint directory with needed permissions also (while giving flexibility to users to change it on demand). That would ideally not cause errors in starting cluster mode as listed above or with other workarounds like using tmp
Unable to initialize server due to exception: Access to the path '/tmp/cluster' is denied.
Not sure if thats seen as a needed feature, but for now i am happy to work with /data location.

[akoken]

We prioritize container security, which is why we use a non-root user (app) by default, inherited from the base .NET image. Permissions are explicitly granted to the data and app folders for this user, as seen here.

I’m unable to reproduce the error you mentioned. Garnet runs in cluster mode without issues using the following command:

docker run --rm -p 6379:6379 --ulimit memlock=-1 ghcr.io/microsoft/garnet:latest --cluster true

    _________
   /_||___||_\      Garnet 1.0.39 64 bit; cluster mode
   '. \   / .'      Port: 6379
     '.\ /.'        https://aka.ms/GetGarnet
       '.'

* Ready to accept connections

$ ls -lah cluster
total 16K
drwxr-xr-x 2 app app 4.0K Nov 15 19:05 .
drwxr-xr-x 1 app app 4.0K Nov 15 19:05 ..
-rw-r--r-- 1 app app  512 Nov 15 19:05 nodes.conf.0
-rw-r--r-- 1 app app  512 Nov 15 19:05 replication.conf.0
$ whoami
app
$ 

In general, container folder structures should remain static to ensure consistency and reliability. If dynamic folders are required, I'd recommend users mount a local directory or volume to the container folder, as is common practice in other applications. If it needs to be readonly, just pass the readonly flag.

# For docker:
-v /path/my-data:/data:ro

# For K8s
volumes:
    - name: mnt
      hostPath:
        path: /my-data
  containers:
    - volumeMounts:
        - name: mnt
          mountPath: /data
          readOnly: true