1.4.4.8.6 添加新的 master 节点
在另外两台主机上进行操作：
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join 10.0.0.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:3a83a61437d3deb9fa80a5b5c9e1f50973833ce68f412814600f49ef7ce9919f \
--control-plane 将上面生成的证书填写进来，需要加--certificate-key选项后再加证书
kubeadm join 10.0.0.100:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:3a83a61437d3deb9fa80a5b5c9e1f50973833ce68f412814600f49ef7ce9919f \
--control-plane --certificate-key 76c855a0e485eaa87b396782fe8465cc5536223062e7aefbce5f20c9444fb6de
# 在10.0.0.19和10.0.0.29上分别执行上面命令
[root@k8s-master2 ~]# kubeadm join 10.0.0.100:6443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:3a83a61437d3deb9fa80a5b5c9e1f50973833ce68f412814600f49ef7ce9919f \
> --control-plane --certificate-key 76c855a0e485eaa87b396782fe8465cc5536223062e7aefbce5f20c9444fb6de
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master2.waluna.top localhost] and IPs [10.0.0.19 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master2.waluna.top localhost] and IPs [10.0.0.19 127.0.0.1 ::1]
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master2.waluna.top kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.waluna.local] and IPs [10.20.0.1 10.0.0.19 10.0.0.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node k8s-master2.waluna.top as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master2.waluna.top as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
[root@k8s-master2 ~]# 1.4.4.8.7 kubeadm init 创建 k8s 集群流程
https://kubernetes.io/zh/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-workflow1.4.5 验证master 状态
验证 master 状态，需等待一两分钟才能变为ready，因为在创建flannel，需要时间
[root@k8s-master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master1.waluna.top Ready control-plane,master 21m v1.21.1
k8s-master2.waluna.top Ready control-plane,master 2m11s v1.21.1
k8s-master3.waluna.top Ready control-plane,master 64s v1.21.1
[root@k8s-master1 ~]#
# 查看所有pod信息
[root@k8s-master1 ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-558bd4d5db-2z2w6 1/1 Running 0 21m
kube-system coredns-558bd4d5db-m9tbs 1/1 Running 0 21m
kube-system etcd-k8s-master1.waluna.top 1/1 Running 0 21m
kube-system etcd-k8s-master2.waluna.top 1/1 Running 0 110s
kube-system etcd-k8s-master3.waluna.top 1/1 Running 0 43s
kube-system kube-apiserver-k8s-master1.waluna.top 1/1 Running 0 21m
kube-system kube-apiserver-k8s-master2.waluna.top 1/1 Running 0 114s
kube-system kube-apiserver-k8s-master3.waluna.top 1/1 Running 0 47s
kube-system kube-controller-manager-k8s-master1.waluna.top 1/1 Running 1 9m8s
kube-system kube-controller-manager-k8s-master2.waluna.top 1/1 Running 0 114s
kube-system kube-controller-manager-k8s-master3.waluna.top 1/1 Running 0 47s
kube-system kube-flannel-ds-crph6 1/1 Running 0 48s
kube-system kube-flannel-ds-q56dd 1/1 Running 0 115s
kube-system kube-flannel-ds-rcqcr 1/1 Running 0 14m
kube-system kube-proxy-5xnhb 1/1 Running 0 21m
kube-system kube-proxy-k6bm9 1/1 Running 0 48s
kube-system kube-proxy-v4brg 1/1 Running 0 115s
kube-system kube-scheduler-k8s-master1.waluna.top 1/1 Running 1 7m8s
kube-system kube-scheduler-k8s-master2.waluna.top 1/1 Running 0 113s
kube-system kube-scheduler-k8s-master3.waluna.top 1/1 Running 0 47s
[root@k8s-master1 ~]# 1.4.5.1 验证 k8s 集群状态
[root@k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
[root@k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Unhealthy Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused
scheduler Unhealthy Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0 Healthy {"health":"true"}
[root@k8s-master1 ~]# 因为使用haproxy进行调度，所有又报前面的错误，同样需要修改两外两个master节点的配置文件
[root@k8s-master2 ~]# vim /etc/kubernetes/manifests/kube-controller-manager.yaml
26 #- --port=0 # 注释此行，打开10252端口
[root@k8s-master2 ~]# vim /etc/kubernetes/manifests/kube-scheduler.yaml
19 #- --port=0 # 注释此行，打开10251端口
# 同样在10.0.0.29上也进行注释
[root@k8s-master3 ~]# vim /etc/kubernetes/manifests/kube-controller-manager.yaml
#- --port=0
[root@k8s-master3 ~]# vim /etc/kubernetes/manifests/kube-scheduler.yaml
#- --port=0
# 再次验证组件状态
[root@k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
[root@k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
[root@k8s-master1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
[root@k8s-master1 ~]# 1.4.6.2 验证 csr 证书状态
有节点加入不需要同意就会自动接受，不太安全
[root@k8s-master1 ~]# kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-ccxjq 8m23s kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:abcdef Approved,Issued
csr-sjxmw 7m17s kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:abcdef Approved,Issued
[root@k8s-master1 ~]# 证书存放路径
[root@k8s-master1 ~]# ll /etc/kubernetes/pki/
total 68
drwxr-xr-x 3 root root 4096 Oct 27 22:03 ./
drwxr-xr-x 4 root root 4096 Oct 27 22:03 ../
-rw-r--r-- 1 root root 1314 Oct 27 22:03 apiserver.crt
-rw-r--r-- 1 root root 1155 Oct 27 22:03 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Oct 27 22:03 apiserver-etcd-client.key
-rw------- 1 root root 1679 Oct 27 22:03 apiserver.key
-rw-r--r-- 1 root root 1164 Oct 27 22:03 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Oct 27 22:03 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1066 Oct 27 22:03 ca.crt
-rw------- 1 root root 1675 Oct 27 22:03 ca.key
drwxr-xr-x 2 root root 4096 Oct 27 22:03 etcd/
-rw-r--r-- 1 root root 1078 Oct 27 22:03 front-proxy-ca.crt
-rw------- 1 root root 1679 Oct 27 22:03 front-proxy-ca.key
-rw-r--r-- 1 root root 1119 Oct 27 22:03 front-proxy-client.crt
-rw------- 1 root root 1679 Oct 27 22:03 front-proxy-client.key
-rw------- 1 root root 1679 Oct 27 22:03 sa.key
-rw------- 1 root root 451 Oct 27 22:03 sa.pub
[root@k8s-master1 ~]#
Comments | NOTHING