Permission denied on internal db #1811

Slyke · 2024-08-12T20:50:05Z

Helm version:

version.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}

Harbor version:

v2.11.0

Logs:

init DB, DB version:15
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with this locale configuration:
  provider:    libc
  LC_COLLATE:  en_US.UTF-8
  LC_CTYPE:    en_US.UTF-8
  LC_MESSAGES: C
initdb: error: could not access directory "/var/lib/postgresql/data/pgdata/pg15": Permission denied
  LC_MONETARY: C
  LC_NUMERIC:  C
  LC_TIME:     C
The default text search configuration will be set to "english".

Data page checksums are disabled.

Pod description:

Name:         harbor-test-database-0
Namespace:    XXXXXXXXXXXXXXXX
Priority:     0
Node:         XXXXXXXXXXXXXXXXXXXXXX
Start Time:   Mon, 12 Aug 2024 20:45:07 +0000
Labels:       app=harbor
              app.kubernetes.io/component=database
              app.kubernetes.io/instance=harbor-test
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=harbor
              app.kubernetes.io/part-of=harbor
              app.kubernetes.io/version=2.11.0
              chart=harbor
              component=database
              controller-revision-hash=harbor-test-database-5d9b7cf448
              heritage=Helm
              release=harbor-test
              statefulset.kubernetes.io/pod-name=harbor-test-database-0
Annotations:  checksum/secret: a2b0779dc9dfd5560e4dd773aa858315c10c217e309c425bd6afa29964c21ff6
              container.seccomp.security.alpha.kubernetes.io/data-permissions-ensurer: runtime/default
              container.seccomp.security.alpha.kubernetes.io/database: runtime/default
Status:       Running
IP:           10.244.0.248
IPs:
  IP:           10.244.0.248
Controlled By:  StatefulSet/harbor-test-database
Init Containers:
  data-permissions-ensurer:
    Container ID:  containerd://8c0e5fd88285b8601c3c2f586201da2567bb22d86ac8a41f2277f99b5a8b12a1
    Image:         goharbor/harbor-db:v2.11.0
    Image ID:      docker.io/goharbor/harbor-db@sha256:103820016e01b0f6a2c1ee88692d3774ff4dd345c85b58e7c928e3a2b958bda3
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      chmod -R 700 /var/lib/postgresql/data/pgdata || true
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 12 Aug 2024 20:45:08 +0000
      Finished:     Mon, 12 Aug 2024 20:45:08 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/postgresql/data from database-data (rw)
Containers:
  database:
    Container ID:   containerd://893adb6931833f1d36f9c20b5be1fcb0346df85baafee0c2667f6dd3e37d4a5d
    Image:          goharbor/harbor-db:v2.11.0
    Image ID:       docker.io/goharbor/harbor-db@sha256:103820016e01b0f6a2c1ee88692d3774ff4dd345c85b58e7c928e3a2b958bda3
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 12 Aug 2024 20:46:01 +0000
      Finished:     Mon, 12 Aug 2024 20:46:01 +0000
    Ready:          False
    Restart Count:  3
    Liveness:       exec [/docker-healthcheck.sh] delay=300s timeout=1s period=10s #success=1 #failure=3
    Readiness:      exec [/docker-healthcheck.sh] delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      harbor-test-database  Secret  Optional: false
    Environment:
      PGDATA:  /var/lib/postgresql/data/pgdata
    Mounts:
      /dev/shm from shm-volume (rw)
      /var/lib/postgresql/data from database-data (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  shm-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  512Mi
  database-data:
    Type:        PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:   harbor-db-pvc
    ReadOnly:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  76s                 default-scheduler  Successfully assigned cicd/harbor-test-database-0 to k-w-003
  Normal   Pulled     75s                 kubelet            Container image "goharbor/harbor-db:v2.11.0" already present on machine
  Normal   Created    75s                 kubelet            Created container data-permissions-ensurer
  Normal   Started    75s                 kubelet            Started container data-permissions-ensurer
  Normal   Pulled     23s (x4 over 75s)   kubelet            Container image "goharbor/harbor-db:v2.11.0" already present on machine
  Normal   Created    23s (x4 over 75s)   kubelet            Created container database
  Normal   Started    22s (x4 over 75s)   kubelet            Started container database
  Warning  BackOff    16s (x10 over 73s)  kubelet            Back-off restarting failed container

The text was updated successfully, but these errors were encountered:

crlsgms · 2024-08-30T03:58:17Z

I'm also getting an error updating from 1.14.3 to 1.15.0 (or straigh to 1.15.1) database, core, jobservice, redis and registry in crashloopbackoff

database

database upgrade DB from 14 to 15
database popen failure: Cannot allocate memory
database initdb: error: program "postgres" is needed by initdb but was not found in the same directory as "/usr/pgsql/15/bin/initdb"

core

goroutine 1 gp=0xc0000061c0 m=0 mp=0x4418680 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00013e750 sp=0xc00013e740 pc=0x474a08
runtime.main()
    /usr/local/go/src/runtime/proc.go:171 +0x67 fp=0xc00013e7e0 sp=0xc00013e750 pc=0x442467
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00013e7e8 sp=0xc00013e7e0 pc=0x476a21

rax    0x0
rbx    0x1
rcx    0x7f9186eb2dbc
rdx    0x6
rdi    0x1
rsi    0x1
rbp    0x7f9186e25740
rsp    0x7fffe32d0950
r8     0x0
r9     0x73
r10    0x8
r11    0x246
r12    0x6
r13    0x0
r14    0x4417840
r15    0x1ffffffffffffff
rip    0x7f9186eb2dbc
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

jobserver also like the core pod, runtime errors on same asm_amd64

redis

1:C 30 Aug 2024 03:52:13.902 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 30 Aug 2024 03:52:13.902 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 30 Aug 2024 03:52:13.902 * Configuration loaded
1:M 30 Aug 2024 03:52:13.903 * monotonic clock: POSIX clock_gettime
1:M 30 Aug 2024 03:52:13.903 # Failed to write PID file: Permission denied
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 7.2.4 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

1:M 30 Aug 2024 03:52:13.903 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 30 Aug 2024 03:52:13.903 # Fatal: Can't initialize Background Jobs. Error message: Operation not permitted

registry even start for a while, then goes on the same error as core and jobservice

registry 10.20.91.16 - - [30/Aug/2024:03:55:52 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
registry 10.20.91.16 - - [30/Aug/2024:03:56:02 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
registry 10.20.91.16 - - [30/Aug/2024:03:56:02 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
registry 10.20.91.16 - - [30/Aug/2024:03:56:12 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.26"
registryctl goroutine 1 gp=0xc0000061c0 m=0 mp=0x1f3ebc0 [running]:
registryctl runtime.systemstack_switch()
registryctl     /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00011a750 sp=0xc00011a740 pc=0x472888
registryctl runtime.main()
registryctl     /usr/local/go/src/runtime/proc.go:171 +0x67 fp=0xc00011a7e0 sp=0xc00011a750 pc=0x441427
registryctl runtime.goexit({})
registryctl     /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00011a7e8 sp=0xc00011a7e0 pc=0x4748a1
registryctl
registryctl rax    0x0
registryctl rbx    0x1
registryctl rcx    0x7ff5d1eb6dbc
registryctl rdx    0x6
registryctl rdi    0x1
registryctl rsi    0x1
registryctl rbp    0x7ff5d1e29740
registryctl rsp    0x7ffc7da59ac0
registryctl r8     0x0
registryctl r9     0x73
registryctl r10    0x8
registryctl r11    0x246
registryctl r12    0x6
registryctl r13    0x0
registryctl r14    0x1f3e180
registryctl r15    0x1ffffffffffffff
registryctl rip    0x7ff5d1eb6dbc
registryctl rflags 0x246
registryctl cs     0x33
registryctl fs     0x0
registryctl gs     0x0

anyhow its now impossible to upgrade from version 1.14.3 to whatever new 1.15 version (I'm using rancher btw)

MinerYang · 2024-09-02T07:27:27Z

How do you upgrade your helm harbor? I cannot reproduce this issue at my side. May need more contexts here. e.g. any configurations apply to your database settings?

github-actions · 2024-11-01T09:05:27Z

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

Slyke · 2024-11-01T10:28:59Z

Is it possible to put an init container on pg to set permissions?

s4ndalHat · 2024-11-21T12:52:48Z

getting the same error here with the assembly code in the core, and redis during upgrade, have been it fixed ?

DrackThor · 2024-12-04T09:01:30Z

same problem here with chart version 1.15.2 and harbor v2.11.2. We're using a plain helm installation, our PVs are provided by NetApp. Any suggestions anybody?
Looks like we run into permission errors on trivy, redis and postgres. At least on trivy I was able to fix this with a initContainer

Slyke · 2024-12-04T09:48:22Z

@MinerYang this might be caused by having the PV on the node being an NFS mounted directory.

DrackThor · 2024-12-04T10:22:08Z

I was able to fix this issue for postgres, redis and trivy like this.. might not be pretty, but at least Harbor starts now.
(and I think I found at least one of this fixes online in another GH issue, so sorry for copycatting)

postgres:

extrInitContainers:
      - name: db-permission-fix
        securityContext:
          runAsUser: 0  # Run as root user
        image: busybox
        command: [ 'sh', '-c', 'mkdir -p /var/lib/postgresql/data/; chown -R 999:999 /var/lib/postgresql/data/; chmod -R 700 /var/lib/postgresql/data/' ]
        volumeMounts:
          - name: database-data
            mountPath: /var/lib/postgresql/data

trivy:

  initContainers:
    - name: trivy-permission-fix
      securityContext:
        runAsUser: 0  # Run as root user
      image: busybox
      command: [ 'sh', '-c', 'chown -R 10000:10000 /home/scanner' ]
      volumeMounts:
        - name: data
          mountPath: /home/scanner

redis:
I still get a Failed to write PID file: Permission denied log message, but redis appears to work fine.

    initContainers:
      - name: redis-permission-fix
        securityContext:
          runAsUser: 0  # Run as root user
        image: busybox
        command: [ 'sh', '-c', 'chown -R 999:999 /var/lib/redis' ]
        volumeMounts:
          - name: data
            mountPath: /var/lib/redis

s4ndalHat · 2024-12-04T17:04:42Z

So in my case, currently the harbor only runs on a NFS mounted PVC by netapp,

There is no official workaround ? It was fine on every version except this...

I don't think that my CTO would br okay to change our current setup on harbor

DaanSelen · 2024-12-12T15:55:30Z

I was able to fix this issue for postgres, redis and trivy like this.. might not be pretty, but at least Harbor starts now. (and I think I found at least one of this fixes online in another GH issue, so sorry for copycatting)

postgres:

extrInitContainers:
      - name: db-permission-fix
        securityContext:
          runAsUser: 0  # Run as root user
        image: busybox
        command: [ 'sh', '-c', 'mkdir -p /var/lib/postgresql/data/; chown -R 999:999 /var/lib/postgresql/data/; chmod -R 700 /var/lib/postgresql/data/' ]
        volumeMounts:
          - name: database-data
            mountPath: /var/lib/postgresql/data

trivy:

  initContainers:
    - name: trivy-permission-fix
      securityContext:
        runAsUser: 0  # Run as root user
      image: busybox
      command: [ 'sh', '-c', 'chown -R 10000:10000 /home/scanner' ]
      volumeMounts:
        - name: data
          mountPath: /home/scanner

redis: I still get a Failed to write PID file: Permission denied log message, but redis appears to work fine.

    initContainers:
      - name: redis-permission-fix
        securityContext:
          runAsUser: 0  # Run as root user
        image: busybox
        command: [ 'sh', '-c', 'chown -R 999:999 /var/lib/redis' ]
        volumeMounts:
          - name: data
            mountPath: /var/lib/redis

Where to apply these?

DrackThor · 2024-12-13T09:09:55Z

@DaanSelen in the values.yaml of the Harbor HELM Chart

DaanSelen · 2024-12-13T09:50:36Z

@DaanSelen in the values.yaml of the Harbor HELM Chart

Thanks! But I am now unsure my issue is the same: #1889

MinerYang added the follow-up label Sep 2, 2024

MinerYang assigned wy65701436 Sep 2, 2024

github-actions bot added the Stale label Nov 1, 2024

github-actions bot removed the Stale label Nov 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Permission denied on internal db #1811

Permission denied on internal db #1811

Slyke commented Aug 12, 2024 •

edited

Loading

crlsgms commented Aug 30, 2024

MinerYang commented Sep 2, 2024

github-actions bot commented Nov 1, 2024

Slyke commented Nov 1, 2024

s4ndalHat commented Nov 21, 2024 •

edited

Loading

DrackThor commented Dec 4, 2024

Slyke commented Dec 4, 2024

DrackThor commented Dec 4, 2024

s4ndalHat commented Dec 4, 2024

DaanSelen commented Dec 12, 2024

DrackThor commented Dec 13, 2024

DaanSelen commented Dec 13, 2024

Permission denied on internal db #1811

Permission denied on internal db #1811

Comments

Slyke commented Aug 12, 2024 • edited Loading

crlsgms commented Aug 30, 2024

MinerYang commented Sep 2, 2024

github-actions bot commented Nov 1, 2024

Slyke commented Nov 1, 2024

s4ndalHat commented Nov 21, 2024 • edited Loading

DrackThor commented Dec 4, 2024

Slyke commented Dec 4, 2024

DrackThor commented Dec 4, 2024

s4ndalHat commented Dec 4, 2024

DaanSelen commented Dec 12, 2024

DrackThor commented Dec 13, 2024

DaanSelen commented Dec 13, 2024

Slyke commented Aug 12, 2024 •

edited

Loading

s4ndalHat commented Nov 21, 2024 •

edited

Loading