Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update k8s version #2174

Merged
merged 2 commits into from
Sep 7, 2024
Merged

update k8s version #2174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9bb07a2
update k8s version
outductor Sep 7, 2024
b6c4541
update versions
outductor Sep 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/pluto.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ jobs:

- name: Use pluto
run: |
pluto detect-files -d seichi-onp-k8s/manifests/seichi-kubernetes/apps --target-versions k8s=v1.28.5
pluto detect-files -d seichi-onp-k8s/manifests/seichi-kubernetes/apps --target-versions k8s=v1.30.4

- name: Use pluto
run: |
pluto detect-files -d seichi-onp-k8s/manifests/seichi-kubernetes/app-templates --target-versions k8s=v1.28.5
pluto detect-files -d seichi-onp-k8s/manifests/seichi-kubernetes/app-templates --target-versions k8s=v1.30.4
6 changes: 3 additions & 3 deletions seichi-onp-k8s/cluster-boot-up/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

### VM 環境

VM環境は `Proxmox Virtual Environment 8.1.3` を利用しています。
VM環境は `Proxmox Virtual Environment 8.2.4` を利用しています。

- ベアメタル3ノード

Expand All @@ -34,7 +34,7 @@ VM環境は `Proxmox Virtual Environment 8.1.3` を利用しています。
参考: [Role of SSH in Proxmox VE Clusters - proxmox wiki](https://pve.proxmox.com/wiki/Cluster_Manager#_role_of_ssh_in_proxmox_ve_clusters)

KubernetesノードのVMは cloudinit イメージで作成されています。
この cloudinit イメージのベースには `Ubuntu 22.04 LTS` を利用しています。
この cloudinit イメージのベースには `Ubuntu 24.04 LTS` を利用しています。

### ストレージ

Expand Down Expand Up @@ -64,7 +64,7 @@ KubernetesノードのVMは cloudinit イメージで作成されています。
2022/05/23現在、クラスタは (3 control plane nodes + 3 worker nodes) の構成で[作成されています](https://github.com/GiganticMinecraft/seichi_infra/blob/91999d509e52905eaff16fc8928fbe5316f1eaeb/seichi-onp-k8s/cluster-boot-up/scripts/proxmox-host-terminal/deploy-vm.sh#L15-L20)。

クラスタの作成は以下のツール群で行っています。
- kubeadm, kubectl, kubelet v1.28.5
- kubeadm, kubectl, kubelet v1.30.4

CNI には Cilium を利用しています。

Expand Down
2 changes: 1 addition & 1 deletion seichi-onp-k8s/cluster-boot-up/scripts/local-terminal/obtain-cloudflared-cert.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
cert_pem_downloaded_location="${HOME}/.cloudflared/cert.pem"

# 利用する cloudflared のバージョン等
cloudflared_release="2022.5.1"
cloudflared_release="2024.8.3"
cloudflared_binary="https://github.com/cloudflare/cloudflared/releases/download/${cloudflared_release}/cloudflared-linux-amd64"

# endregion
Expand Down
33 changes: 23 additions & 10 deletions seichi-onp-k8s/cluster-boot-up/scripts/nodes/k8s-node-setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,21 @@ EOF
# Apply sysctl params without reboot
sudo sysctl --system

## Install containerd
sudo apt-get update && sudo apt-get install -y containerd apt-transport-https curl gnupg2
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl gnupg2
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install containerd.io

# Configure containerd
sudo mkdir -p /etc/containerd
Expand All @@ -95,7 +108,7 @@ fi
sudo systemctl restart containerd

# Modify kernel parameters for Kubernetes
# inotify instance number is very limited in Ubuntu 22.04 and it has to be at least more than pod number * 2
# inotify instance number is very limited in Ubuntu 24.04 and it has to be at least more than pod number * 2
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
Expand All @@ -113,9 +126,9 @@ EOF
sysctl --system

# Install kubeadm
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get install -y kubeadm=1.28.5-1.1 kubectl=1.28.5-1.1 kubelet=1.28.5-1.1
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get install -y kubeadm=1.30.4-1.1 kubectl=1.30.4-1.1 kubelet=1.30.4-1.1
apt-mark hold kubelet kubectl

# Disable swap
Expand All @@ -141,8 +154,8 @@ esac

# Install HAProxy
apt-get install -y --no-install-recommends software-properties-common
add-apt-repository ppa:vbernat/haproxy-2.8 -y
sudo apt-get install -y haproxy=2.8.\*
add-apt-repository ppa:vbernat/haproxy-3.0 -y
sudo apt-get install -y haproxy=3.0.\*

cat > /etc/haproxy/haproxy.cfg <<EOF
global
Expand Down Expand Up @@ -249,7 +262,7 @@ systemctl reload haproxy
kubeadm config images pull

# install k9s
wget https://github.com/derailed/k9s/releases/download/v0.28.2/k9s_Linux_amd64.tar.gz -O - | tar -zxvf - k9s && sudo mv ./k9s /usr/local/bin/
wget https://github.com/derailed/k9s/releases/download/v0.32.5/k9s_Linux_amd64.tar.gz -O - | tar -zxvf - k9s && sudo mv ./k9s /usr/local/bin/

# Ends except first-control-plane
case $1 in
Expand Down Expand Up @@ -294,7 +307,7 @@ etcd:
local:
extraArgs:
listen-metrics-urls: http://0.0.0.0:2381
kubernetesVersion: "v1.28.5"
kubernetesVersion: "v1.30.4"
controlPlaneEndpoint: "${KUBE_API_SERVER_VIP}:8443"
apiServer:
certSANs:
Expand Down
10 changes: 5 additions & 5 deletions seichi-onp-k8s/cluster-boot-up/scripts/proxmox-host-terminal/deploy-vm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,20 +26,20 @@ VM_LIST=(

#region create-template

# download the image(ubuntu 22.04 LTS)
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
# download the image(ubuntu 24.04 LTS)
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

# install qemu-guest-agent to image using libguestfs-tools
apt-get update && apt-get install libguestfs-tools -y
virt-customize -a jammy-server-cloudimg-amd64.img --install liburing2 --install qemu-guest-agent
virt-customize -a noble-server-cloudimg-amd64.img --install liburing2 --install qemu-guest-agent

# create a new VM and attach Network Adaptor
# vmbr0=Service Network Segment (192.168.0.0/20)
# vmbr1=Storage Network Segment (192.168.16.0/22)
qm create $TEMPLATE_VMID --cores 2 --memory 4096 --net0 virtio,bridge=vmbr0 --net1 virtio,bridge=vmbr1 --net2 virtio,bridge=vmbr2 --agent enabled=1,fstrim_cloned_disks=1 --name seichi-onp-k8s-template

# import the downloaded disk to $TEMPLATE_BOOT_IMAGE_TARGET_VOLUME storage
qm importdisk $TEMPLATE_VMID jammy-server-cloudimg-amd64.img $TEMPLATE_BOOT_IMAGE_TARGET_VOLUME
qm importdisk $TEMPLATE_VMID noble-server-cloudimg-amd64.img $TEMPLATE_BOOT_IMAGE_TARGET_VOLUME

# finally attach the new disk to the VM as scsi drive
qm set $TEMPLATE_VMID --scsihw virtio-scsi-pci --scsi0 $TEMPLATE_BOOT_IMAGE_TARGET_VOLUME:vm-$TEMPLATE_VMID-disk-0
Expand All @@ -57,7 +57,7 @@ qm set $TEMPLATE_VMID --serial0 socket --vga serial0
qm template $TEMPLATE_VMID

# cleanup
rm jammy-server-cloudimg-amd64.img
rm noble-server-cloudimg-amd64.img

#endregion

Expand Down