kubernetes 版本升级(V1.21-V1.22)

#前言:


由于阿里云 ack 集群 1.20 版本会于 2023/4 月份停止维护,线上环境可以直接走阿里云的控制台升级,但是测试环境要和线上同步版本,所以此将此次测试环境升级记录一下v1.21.1-v1.22.1 那么,开始梭哈。

⚠️ 本教程只适用于使用kubeadm部署的 Kubernetes 环境

集群环境:

1
2
3
4
5
[root@k8s-master ~]# kubectl get  nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 558d v1.21.1
k8s-node-1 Ready <none> 558d v1.21.1
k8s-node-2 Ready <none> 558d v1.21.1

#Master 节点升级:


查看 kubeadm 可升级版本:

1
2
3
4
[root@k8s-master ~]# yum list --showduplicates kubeadm --disableexcludes=kubernetes

find the latest 1.22.1 version in the list
kubeadm.x86_64 1.22.1-0 kubernetes

下载安装 Kubeadm:

1
[root@k8s-master ~]# yum install -y kubeadm-1.22.1-0 --disableexcludes=kubernetes

验证升级结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@k8s-master ~]# kubeadm version -o json  #"gitVersion": "v1.22.1"

{
"clientVersion": {
"major": "1",
"minor": "22",
"gitVersion": "v1.22.1",
"gitCommit": "632ed300f2c34f6d6d15ca4cef3d3c707341xxxx",
"gitTreeState": "clean",
"buildDate": "2021-08-19T15:44:22Z",
"goVersion": "go1.16.7",
"compiler": "gc",
"platform": "linux/amd64"
}
}

驱逐前该节点上的 Pod 如下:

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s-master ~]# kubectl get pod --all-namespaces -owide --field-selector spec.nodeName=k8s-master

NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-545d6fc579-9cmc4 1/1 Running 0 3d22h 10.244.0.15 k8s-master <none> <none>
kube-system csi-plugin-n99kp 4/4 Running 0 96d 172.23.130.165 k8s-master <none> <none>
kube-system csi-provisioner-68498f646b-xbghf 8/8 Running 0 3d23h 172.23.130.165 k8s-master <none> <none>
kube-system etcd-k8s-master 1/1 Running 0 3d23h 172.23.130.165 k8s-master <none> <none>
kube-system kube-apiserver-k8s-master 1/1 Running 0 3d23h 172.23.130.165 k8s-master <none> <none>
kube-system kube-controller-manager-k8s-master 1/1 Running 0 3d23h 172.23.130.165 k8s-master <none> <none>
kube-system kube-flannel-ds-k5mvp 1/1 Running 2 562d 172.23.130.165 k8s-master <none> <none>
kube-system kube-proxy-8w8qr 1/1 Running 0 3d23h 172.23.130.165 k8s-master <none> <none>
kube-system kube-scheduler-k8s-master 1/1 Running 0 3d23h 172.23.130.165 k8s-master <none> <none>

驱逐 Master 节点:

通过将节点标记为不可调度并驱逐工作负载来准备节点进行维护,将 pod 优雅的中止。

1
[root@k8s-master ~]# kubectl drain k8s-master --delete-emptydir-data  --ignore-daemonsets

如果节点上存在绑定了本地存储的 Pod 或是一些守护进程集管理的 Pod,会提示“error: unable to drain node “k8s-master”, aborting bash…”。驱逐命令将不会生效。

  • –delete-emptydir-data:强制驱逐节点上绑定了本地存储的 Pod,例如 coredns。
  • –ignore-daemonsets:强制驱逐节点上的守护进程集 Pod,例如 everest-csi-controller。

Example:
再具体点就是

1
2
3
4
5
6
7
"--delete-emptydir-data" 是一个 Kubernetes Pod 的参数,用于在删除 Pod 时同时删除其 EmptyDir 卷中的数据。下面是更具体的解释:

在 Kubernetes 中,EmptyDir 卷是一种临时存储卷,用于在 Pod 生命周期内共享数据。当 Pod 被创建时,EmptyDir 卷会被创建并挂载到 Pod 的容器中,容器可以将数据写入该卷中。当 Pod 被删除时,EmptyDir 卷会被卸载并删除。

默认情况下,当 Pod 被删除时,EmptyDir 卷中的数据也会被删除。但是,如果在 Pod 运行期间手动更改了 EmptyDir 卷中的数据,那么在删除 Pod 时该数据就不会被自动删除。此时,如果需要在下次创建 Pod 时使用该 EmptyDir 卷,可能需要手动清理该卷中的数据。

"--delete-emptydir-data" 参数可以在删除 Pod 时自动删除 EmptyDir 卷中的数据。这个参数只对 EmptyDir 卷有效,不会影响其他类型的卷。使用此参数时,如果在 Pod 运行期间手动更改了 EmptyDir 卷中的数据,该数据将在删除 Pod 时自动删除,而不需要手动清理。需要注意的是,这个参数只能在删除 Pod 时使用,不能用于修改已经存在的 Pod。

1
2
3
4
5
6
7
8
9
10

"--ignore-daemonsets" 是一个 Kubernetes 命令的参数,用于在删除一个节点上的 Pod 时忽略 DaemonSet 类型的 Pod。下面是更具体的解释:

在 Kubernetes 集群中,DaemonSet 是一种 Pod 的控制器,它会确保在集群的每个节点上都运行一个 Pod 的副本。DaemonSet 类型的 Pod 是无法通过普通的 ReplicaSet 或 Deployment 控制器来管理的,因为它们必须在每个节点上运行一个副本。因此,在删除一个节点上的 Pod 时,如果这个节点上运行了 DaemonSet 类型的 Pod,那么这些 Pod 就不能被删除。

"--ignore-daemonsets" 参数可以在删除一个节点上的 Pod 时忽略 DaemonSet 类型的 Pod。使用这个参数时,如果在节点上运行了 DaemonSet 类型的 Pod,这些 Pod 将不会被删除,并且删除操作将继续进行,直到删除非 DaemonSet 类型的 Pod 为止。

需要注意的是,使用 "--ignore-daemonsets" 参数时应谨慎,因为它可能会导致节点上的某些 Pod 无法删除。如果你需要删除某个节点上的所有 Pod,可以先手动删除 DaemonSet 类型的 Pod,然后再使用 "--ignore-daemonsets" 参数删除剩余的 Pod。另外,如果你需要保留节点上的 DaemonSet 类型的 Pod,可以使用其他参数或命令来管理它们。
-- Data from chatgpt

因为我这里存在临时存储的卷所以需要添加--delete-emptydir-data否则会报下面错误。

1
2
3
4
5
6
7
8
[root@k8s-master ~]# kubectl drain k8s-master --ignore-daemonsets  
node/k8s-master cordoned
error: unable to drain node "k8s-master", aborting bash...

There are pending nodes to be drained:
k8s-master
error: cannot delete Pods with local storage (use --delete-emptydir-data to override): argocd/devops-argocd-application-controller-0, argocd/devops-argocd-dex-server-5f4c69cdb8-5jzbb, default/redis-master-0, default/redis-replicas-0, default/redis-replicas-2, kube-system/csi-provisioner-68498f646b-2bz25

此时查看 master 节点已经被打上了污点
Pasted image 20230213175219

验证升级计划 :

1
[root@k8s-master ~]#  kubeadm upgrade plan

选择升级的版本:

执行升级命令

1
2
3
4
5
6
[root@k8s-master ~]# kubeadm upgrade apply v1.22.1

#执行完毕显示 successful 升级成功
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.22.1". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

更新工作组件:

1
2
3
4
[root@k8s-master ~]# kubeadm upgrade node

[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

更新 kubelet & kubectl:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[root@k8s-master ~]# yum install -y kubelet-1.22.1-0 kubectl-1.22.1-0 --disableexcludes=kubernetes

[root@k8s-master ~]# kubectl version -o json # 确认升级成功 "gitVersion": "v1.22.1"
{
"clientVersion": {
"major": "1",
"minor": "22",
"gitVersion": "v1.22.1",
"gitCommit": "632ed300f2c34f6d6d15ca4cef3d3c7073412xxx",
"gitTreeState": "clean",
"buildDate": "2021-08-19T15:45:37Z",
"goVersion": "go1.16.7",
"compiler": "gc",
"platform": "linux/amd64"
},
"serverVersion": {
"major": "1",
"minor": "22",
"gitVersion": "v1.22.1",
"gitCommit": "632ed300f2c34f6d6d15ca4cef3d3c7073412xxx",
"gitTreeState": "clean",
"buildDate": "2021-08-19T15:39:34Z",
"goVersion": "go1.16.7",
"compiler": "gc",
"platform": "linux/amd64"
}
}

重启 kebelet 插件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@k8s-master ~]# sudo systemctl daemon-reload
[root@k8s-master ~]# sudo systemctl restart kubelet
[root@k8s-master ~]# sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2023-02-13 18:01:11 CST; 13s ago
Docs: https://kubernetes.io/docs/
Main PID: 2201 (kubelet)
Tasks: 18
Memory: 67.9M
CGroup: /system.slice/kubelet.service
└─2201 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2

取消 Master 节点 SchedulingDisabled 状态:

将节点标记为可调度,让节点重新上线

1
2
3
#所有将节点上下线的操作均在 master 节点上执行
[root@k8s-master ~]# kubectl uncordon k8s-master
node/k8s-master uncordoned

#Node节点升级:


驱逐 Node 节点:

Master 节点上执行

1
2
3
4
5
6
7
8
9
10
11
[root@k8s-master ~]# kubectl drain k8s-node-1 --delete-emptydir-data  --ignore-daemonsets
node/k8s-node-1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/csi-plugin-jvpdk, kube-system/kube-flannel-ds-lf77m, kube-system/kube-proxy-gtwrq, kubesphere-logging-system/fluent-bit-62m75, kubesphere-monitoring-system/node-exporter-8svnv
evicting pod argocd/devops-argocd-notifications-controller-6d86f8974f-7dqm6
evicting pod argocd/devops-argocd-applicationset-controller-b88d4b875-nb5mq
evicting pod argocd/devops-argocd-dex-server-5f4c69cdb8-g7s56
evicting pod kubesphere-devops-system/devops-controller-7dfc7c6cfc-8d9dh
evicting pod argocd/devops-argocd-application-controller-0
evicting pod argocd/devops-argocd-redis-655969589d-wvxtp
evicting pod argocd/devops-argocd-repo-server-859666d69d-hr4dh
evicting pod slots/message-58648d7fdf-vngpq

在 Node 节点上执行:

下载安装 Kubeadm:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@k8s-node-1 ~]# yum install -y kubeadm-1.22.1-0 --disableexcludes=kubernetes

[root@k8s-node-1 ~]# kubeadm version -o json
{
"clientVersion": {
"major": "1",
"minor": "22",
"gitVersion": "v1.22.1", #显示 v1.22.1 升级成功
"gitCommit": "632ed300f2c34f6d6d15ca4cef3d3c7073412xxxx",
"gitTreeState": "clean",
"buildDate": "2021-08-19T15:44:22Z",
"goVersion": "go1.16.7",
"compiler": "gc",
"platform": "linux/amd64"
}
}

更新工作组件:

1
2
3
4
5
6
7
8
9
10
[root@k8s-node-1 ~]# kubeadm upgrade node

[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks
[preflight] Skipping prepull. Not a control plane node.
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.

更新 kubelet & kubectl:

1
[root@k8s-master ~]# yum install -y kubelet-1.22.1-0 kubectl-1.22.1-0 --disableexcludes=kubernetes

重启 kebelet 插件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@k8s-node-1 ~]# systemctl daemon-reload
[root@k8s-node-1 ~]# systemctl restart kubelet
[root@k8s-node-1 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2023-02-13 18:43:13 CST; 8s ago
Docs: https://kubernetes.io/docs/
Main PID: 20066 (kubelet)
Tasks: 19
Memory: 69.7M
CGroup: /system.slice/kubelet.service
└─20066 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2

取消 Node 节点 SchedulingDisabled 状态:

将节点标记为可调度,让节点重新上线

1
2
3
[root@k8s-master ~]# kubectl uncordon k8s-node-1

node/k8s-node-1 uncordoned

验证:

1
2
3
4
5
6
7
[root@k8s-master ~]# kubectl get  nodes 

#SchedulingDisabled 状态已经取消。
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 565d v1.22.1
k8s-node-1 Ready <none> 565d v1.22.1
k8s-node-2 Ready <none> 565d v1.21.1

node-1节点此时也已经升级完成。node-2节点这里就不演示了。重复和node-1节点同样的操作即可。