deploy pod "nacosdb-mysql-0" Init:CrashLoopBackOff

[TenBid]

Summary

deploy pod "nacosdb-mysql-0" Init:CrashLoopBackOff

Expected behaviour

pod nacosdb-mysql-0 should run

Current behaviour

deploy pod "nacosdb-mysql-0" Init:CrashLoopBackOff

Steps to reproduce

root@Master:/home/train-ticket# make deploy
args num: 2
Parse DeployArgs
Start deployment Step <1/3>------------------------------------
Start to deploy mysql cluster for nacos.
NAME: nacosdb
LAST DEPLOYED: Thu May  4 20:26:16 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The cluster is comprised of 3 pods: 1 leader and 2 followers. Each instance is accessible within the cluster through:<pod-name>.nacosdb-mysql

To connect to your database:

1. Get mysql user `nacos`'s password:

    kubectl get secret -n default nacosdb-mysql -o jsonpath="{.data.mysql-password}" | base64 --decode; echo

2. Run an Ubuntu pod that you can use as a client:

    kubectl run ubuntu -n default --image=ubuntu:focal -it --rm --restart='Never' -- bash -il

3. Install the mysql client:

    apt-get update && apt-get install mysql-client -y

4. To connect to leader service in the Ubuntu pod:

    mysql -h nacosdb-mysql-leader -u nacos -p

5. To connect to follower service (read-only) in the Ubuntu pod:

    mysql -h nacosdb-mysql-follower -u nacos -p
Waiting for mysql cluster of nacos to be ready ......
Waiting for 3 pods to be ready...

Your environment

OS(e.g: cat /etc/os-release):
root@Master:/home/train-ticket# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

containerd version (e.g containerd -v):
root@Master:/home/train-ticket# containerd -v
containerd github.com/containerd/containerd v1.6.14 9ba4b250366a5ddde94bb7c9d1def331423aa323

Kubernetes version(use kubectl version):
root@Master:/home/train-ticket# kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.1
Kustomize Version: v5.0.1
Server Version: v1.27.0

Additional context

I run the kubectl logs nacosdb-mysql-0 command to view pod logs. The following information occurs。The problem has not been solved and I would be grateful if anyone could help me。

root@Master:/home/train-ticket# kubectl logs nacosdb-mysql-0
Defaulted container "mysql" out of: mysql, xenon, slowlog, init-mysql (init)
Initializing database
2023-05-04T20:26:21.270405+08:00 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestaserver option (see documentation for more details).
2023-05-04T20:26:22.692394+08:00 0 [Warning] InnoDB: New log files created, LSN=45790
2023-05-04T20:26:23.501390+08:00 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2023-05-04T20:26:23.573486+08:00 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has n started. Generating a new UUID: e1ec02f0-ea76-11ed-81eb-5e76d1f997e3.
2023-05-04T20:26:23.575750+08:00 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2023-05-04T20:26:23.703937+08:00 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialiinsecure option.
Database initialized
Initializing certificates
Ignoring -days; not generating a certificate
Generating a RSA private key
.........................................................................+++++
....................+++++
writing new private key to 'ca-key.pem'
-----
Ignoring -days; not generating a certificate
Generating a RSA private key
.......................................+++++
.......................+++++
writing new private key to 'server-key.pem'
-----
Ignoring -days; not generating a certificate
Generating a RSA private key
..............................+++++
....................................................................................................+++++
writing new private key to 'client-key.pem'
-----
Certificates initialized
MySQL init process in progress...
2023-05-04T20:26:28.509645+08:00 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestaserver option (see documentation for more details).
2023-05-04T20:26:28.590181+08:00 0 [Note] mysqld (mysqld 5.7.34-37-log) starting as process 67 ...
2023-05-04T20:26:28.618174+08:00 0 [Note] InnoDB: PUNCH HOLE support available
2023-05-04T20:26:28.618203+08:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-05-04T20:26:28.618207+08:00 0 [Note] InnoDB: Uses event mutexes
2023-05-04T20:26:28.618209+08:00 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-05-04T20:26:28.618210+08:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-05-04T20:26:28.618212+08:00 0 [Note] InnoDB: Using Linux native AIO
MySQL init process in progress...
2023-05-04T20:26:29.681183+08:00 0 [Note] InnoDB: Number of pools: 1
2023-05-04T20:26:29.681667+08:00 0 [Note] InnoDB: Using CPU crc32 instructions
2023-05-04T20:26:29.710084+08:00 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2023-05-04T20:26:29.871613+08:00 0 [Note] InnoDB: Completed initialization of buffer pool
2023-05-04T20:26:29.917486+08:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. Sthe man page of setpriority().
2023-05-04T20:26:30.073728+08:00 0 [Note] InnoDB: Crash recovery did not find the parallel doublewrite buffer at /var/lib/mysql/xb_doublewri
2023-05-04T20:26:30.080640+08:00 0 [Note] InnoDB: Highest supported file format is Barracuda.
MySQL init process in progress...
2023-05-04T20:26:30.191453+08:00 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 3932160 bytes
2023-05-04T20:26:30.288513+08:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-05-04T20:26:30.288581+08:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-05-04T20:26:30.593351+08:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-05-04T20:26:30.598091+08:00 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2023-05-04T20:26:30.598119+08:00 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2023-05-04T20:26:30.603332+08:00 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.34-37 started; log sequence number 2752616
2023-05-04T20:26:30.604887+08:00 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-05-04T20:26:30.686484+08:00 0 [Note] InnoDB: Buffer pool(s) load completed at 230504 20:26:30
2023-05-04T20:26:30.809166+08:00 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
**2023-05-04T20:26:30.810281+08:00 0 [ERROR] SSL error: Unable to get private key from '/var/lib/mysql/server-key.pem'**
2023-05-04T20:26:30.810715+08:00 0 [Warning] Failed to set up SSL because of the following SSL library error: Unable to get private key

I run the kubectl describe pod nacosdb-mysql-0 command to view pod describe.

Events:
  Type     Reason               Age                     From     Message
  ----     ------               ----                    ----     -------
  Warning  FailedPostStartHook  5m46s (x115 over 15h)   kubelet  PostStartHook failed
  Warning  BackOff              3m25s (x2861 over 15h)  kubelet  Back-off restarting failed container xenon in pod nacosdb-mysql-0_default(6eb03919-8dba-4cbe-890a-711a08638016)

[lingdie]

check k8s cni and network. ping pods by dns.

[TenBid]

check k8s cni and network. ping pods by dns.

Thank you very much. I have solved this problem。
The specific methods are as follows:
Note:The reason is that the k8s version is too high to be compatible.For this problem, you need to roll back the version。

root@master:~# docker version
Client: Docker Engine - Community
Version: 20.10.18
API version: 1.41
Go version: go1.18.6
Git commit: b40c2f6
Built: Thu Sep 8 23:11:43 2022
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.18
API version: 1.41 (minimum version 1.12)
Go version: go1.18.6
Git commit: e42327a
Built: Thu Sep 8 23:09:30 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
root@master:~# kubectl version --short
Client Version: v1.22.4
Server Version: v1.22.17

[06-18]

check k8s cni and network. ping pods by dns.

Thank you very much. I have solved this problem。 The specific methods are as follows: Note:The reason is that the k8s version is too high to be compatible.For this problem, you need to roll back the version。

root@master:~# docker version Client: Docker Engine - Community Version: 20.10.18 API version: 1.41 Go version: go1.18.6 Git commit: b40c2f6 Built: Thu Sep 8 23:11:43 2022 OS/Arch: linux/amd64 Context: default Experimental: true

Server: Docker Engine - Community Engine: Version: 20.10.18 API version: 1.41 (minimum version 1.12) Go version: go1.18.6 Git commit: e42327a Built: Thu Sep 8 23:09:30 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.5.11 GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8 runc: Version: 1.0.3 GitCommit: v1.0.3-0-gf46b6ba docker-init: Version: 0.19.0 GitCommit: de40ad0 root@master:~# kubectl version --short Client Version: v1.22.4 Server Version: v1.22.17

Hello！I encountered the same issue as you did. May I ask, which version of Kubernetes did you roll back to? I will be immensely grateful.

[jaylenwang7]

check k8s cni and network. ping pods by dns.

Thank you very much. I have solved this problem。 The specific methods are as follows: Note:The reason is that the k8s version is too high to be compatible.For this problem, you need to roll back the version。
root@master:# docker version Client: Docker Engine - Community Version: 20.10.18 API version: 1.41 Go version: go1.18.6 Git commit: b40c2f6 Built: Thu Sep 8 23:11:43 2022 OS/Arch: linux/amd64 Context: default Experimental: true
Server: Docker Engine - Community Engine: Version: 20.10.18 API version: 1.41 (minimum version 1.12) Go version: go1.18.6 Git commit: e42327a Built: Thu Sep 8 23:09:30 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.5.11 GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8 runc: Version: 1.0.3 GitCommit: v1.0.3-0-gf46b6ba docker-init: Version: 0.19.0 GitCommit: de40ad0 root@master:# kubectl version --short Client Version: v1.22.4 Server Version: v1.22.17

Hello！I encountered the same issue as you did. May I ask, which version of Kubernetes did you roll back to? I will be immensely grateful.

You can see the version they used in the last line: root@master:~# kubectl version --short Client Version: v1.22.4 Server Version: v1.22.17