1.4 kubernetes 部署过程

具体步骤：

当前部署版本为当前次新版本， 因为后面需要使用 kubeadm 做 kubernetes 版本升级演示 。目标使用版本 v1.21.5，因此本次先以 v1.21.1 版本 kubernetes 为例进行安装

https://github.com/kubernetes/kubernetes/releases
  1. 基础环境准备
  2. 部署 harbor 及 haproxy 高可用反向代理
  3. 在所有 master 安装指定版本的 kubeadm、kubelet、kubectl、docker
  4. 在所有 node 节点安装指定版本的 kubeadm、kubelet、docker，在 node 节点 kubectl 为可选安，看是否需要在 node 执行 kubectl 命令进行集群管理及 pod 管理等操作 。
  5. master 节点运行 kubeadm init 初始化命令
  6. 验证 master 节点状态
  7. 在 node 节点使用 kubeadm 命令将自己加入 k8s master 需要使用 master 生成 token 认证
  8. 验证 node 节点状态
  9. 创建 pod 并测试网络通信
  10. 部署 web 服务 Dashboard
  11. k8s 集群升级

1.4.1 基础环境准备

服务器环境：

最小化安装系统，关闭防火墙、selinux 和 swap，更新软件软源，时间同步，安装常用命令，重启并验证基础环境。

主机名 IP 软件版本 OS版本
k8s-master1.waluna.top 10.0.0.9 Kubernetes v1.21.5 Ubuntu 18.04.5 LTS
k8s-master2.waluna.top 10.0.0.19 Kubernetes v1.21.5 Ubuntu 18.04.5 LTS
k8s-master3.waluna.top 10.0.0.29 Kubernetes v1.21.5 Ubuntu 18.04.5 LTS
ha1.waluna.top 10.0.0.39 haproxy+keepalived Ubuntu 18.04.5 LTS
ha2.waluna.top 10.0.0.49 haproxy+keepalived Ubuntu 18.04.5 LTS
harbor.waluna.top 10.0.0.59 harbor v1.7.6 Ubuntu 18.04.5 LTS
node1.waluna.top 10.0.0.69 kubelet、kube-proxy Ubuntu 18.04.5 LTS
node2.waluna.top 10.0.0.79 kubelet、kube-proxy Ubuntu 18.04.5 LTS
node3.waluna.top 10.0.0.89 kubelet、kube-proxy Ubuntu 18.04.5 LTS

资源限制及内核参数并且关闭 swap

# 修改主机名
[root@ubuntu1804 ~]# hostnamectl set-hostname k8s-master1.waluna.top
[root@ubuntu1804 ~]# exit

# 设置资源限制，注意ubuntu中root用户需要单独设置
[root@k8s-master1 ~]# vim /etc/security/limits.conf
[root@k8s-master1 ~]# tail /etc/security/limits.conf
*       -       core            unlimited
*       -       nproc           1000000
*       -       nofile          1000000
*       -       memlock         32000
*       -       msgqueue        8192000
root    -       core            unlimited
root    -       nproc           1000000
root    -       nofile          1000000
root    -       memlock         32000
root    -       msgqueue        8192000
[root@k8s-master1 ~]# 

# 修改内核参数
[root@k8s-master1 ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward =1
net.ipv4.ip_nonlocal_bind = 1
[root@k8s-master1 ~]# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
[root@k8s-master1 ~]# 

# 关闭所有交换分区
[root@k8s-master1 ~]# swapoff -a
[root@k8s-master1 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        300M        692M        9.8M        977M        1.5G
Swap:            0B          0B          0B
[root@k8s-master1 ~]# 
# 在/etc/fstab文件中注释swap所在行
[root@k8s-master1 ~]# vim /etc/fstab
# swap was on /dev/sda5 during installation
#UUID=bbd98c83-dc72-4698-b83b-03225a178a48 none            swap    sw              0       0

# 重启机器
[root@k8s-master1 ~]# reboot

# 其他机器同样操作

如果不开启 net.ipv4.ip_forward 功能，就无法进行路由转换，就无法进行源地址转换，无法将容器的地址转换为宿主机的地址，就只能在本地通讯，导致无法跨主机访问。

如果不开启 net.ipv4.ip_nonlocal_bind 功能，将无法监听在一个本地没有的IP的端口上，会导致haproxy无法启动。

1.4.2 harbor 及反向代理

1.4.2.1 安装并配置 keepalived

在 10.0.0.39上配置

# 安装keepalived
[root@ha1 ~]# apt install keepalived -y

# 查找模板配置文件
[root@ha1 ~]# find / -name keepalived*vrrp
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp
[root@ha1 ~]# 

# 复制模板文件
[root@ha1 ~]# cp /usr/share/doc/keepalived/samples/keepalived.conf.vrrp /etc/keepalived/keepalived.conf
[root@ha1 ~]# 

# 修改配置文件
[root@ha1 ~]# vim /etc/keepalived/keepalived.conf
[root@ha1 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    garp_master_delay 10
    smtp_alert
    virtual_router_id 56    # 怕冲突可更改下id
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.0.100 dev eth0 label eth0:1    # 添加VIP
    }
}
[root@ha1 ~]# 

# 重启keepalived
[root@ha1 ~]# systemctl restart keepalived.service
[root@ha1 ~]# systemctl enable keepalived.service 
Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable keepalived
[root@ha1 ~]# 

# 验证VIP
[root@ha1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:cd:c3:8e brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.39/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fecd:c38e/64 scope link 
       valid_lft forever preferred_lft forever
[root@ha1 ~]#

在 10.0.0.49上配置

# 安装keepalived
[root@ha2 ~]# apt install keepalived -y

# 拷贝配置文件
[root@ha2 ~]# scp 10.0.0.39:/etc/keepalived/keepalived.conf /etc/keepalived/

# 修改配置文件
[root@ha2 ~]# vim /etc/keepalived/keepalived.conf
[root@ha2 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP    # 修改为BACKUP
    interface eth0
    garp_master_delay 10
    smtp_alert
    virtual_router_id 56
    priority 80 # 调为80
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.0.100 dev eth0 label eth0:1
    }
}
[root@ha2 ~]# 

[root@ha2 ~]# systemctl restart keepalived.service
[root@ha2 ~]# systemctl enable keepalived.service 
Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable keepalived
[root@ha2 ~]#

1.4.2.2 安装并配置 haproxy

在10.0.0.39上配置

[root@ha1 ~]# apt install haproxy -y
[root@ha1 ~]# vim /etc/haproxy/haproxy.cfg
[root@ha1 ~]# tail -15 /etc/haproxy/haproxy.cfg
listen status
    mode http
    bind :9999
    stats enable
    log global
    stats uri /haproxy-status
    stats auth haadmin:123456
    stats admin if TRUE

listen waluna-k8s-api-6443
    mode tcp
    bind 10.0.0.100:6443
    server 10.0.0.9  10.0.0.9:6443  check inter 3s fall 3 rise 5
    server 10.0.0.19 10.0.0.19:6443 check inter 3s fall 3 rise 5
    server 10.0.0.29 10.0.0.29:6443 check inter 3s fall 3 rise 5
[root@ha1 ~]# 

[root@ha1 ~]# systemctl restart haproxy.service
[root@ha1 ~]# systemctl enable haproxy.service 
Synchronizing state of haproxy.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable haproxy
[root@ha1 ~]# ss -ntl
State   Recv-Q   Send-Q      Local Address:Port       Peer Address:Port   
LISTEN  0        128            10.0.0.100:6443            0.0.0.0:*      
LISTEN  0        128               0.0.0.0:9999            0.0.0.0:*      
LISTEN  0        128         127.0.0.53%lo:53              0.0.0.0:*      
LISTEN  0        128               0.0.0.0:22              0.0.0.0:*      
LISTEN  0        128                  [::]:22                 [::]:*      
[root@ha1 ~]#

在10.0.0.49上配置

[root@ha2 ~]# apt install haproxy -y

[root@ha2 ~]# scp 10.0.0.39:/etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg

[root@ha2 ~]# systemctl restart haproxy.service
[root@ha2 ~]# systemctl enable haproxy.service 
Synchronizing state of haproxy.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable haproxy
[root@ha2 ~]# ss -ntl
State   Recv-Q   Send-Q      Local Address:Port       Peer Address:Port   
LISTEN  0        128               0.0.0.0:9999            0.0.0.0:*      
LISTEN  0        128         127.0.0.53%lo:53              0.0.0.0:*      
LISTEN  0        128               0.0.0.0:22              0.0.0.0:*      
LISTEN  0        128            10.0.0.100:6443            0.0.0.0:*      
LISTEN  0        128                  [::]:22                 [::]:*      
[root@ha2 ~]#

1.4.2.3 安装并配置 harbor

在 10.0.0.59上配置

# docker-compose 和 harbor 可能下载不下载来，需要提前准备，docker-compose可以使用apt方式直接安装
[root@harbor ~]# ll docker-compose-Linux-x86_64 harbor-offline-installer-v1.7.6.tgz
-rw-r--r-- 1 root root  12737304 Jul 23 14:28 docker-compose-Linux-x86_64
-rw-r--r-- 1 root root 595286214 Sep 18  2019 harbor-offline-installer-v1.7.6.tgz
[root@harbor ~]# 

# 使用脚本安装
[root@harbor ~]# vim install_harbor_ubuntu1804.sh
[root@harbor ~]# cat install_harbor_ubuntu1804.sh
#!/bin/bash
#Description: Install harbor on ubuntu1804
#Author: xan_yum

COLOR="echo -e \e[1;31m"
END="\e[0m"
DOCKER_VERSION="5:20.10.9~3-0~ubuntu-bionic"
HARBOR_VERSION=1.7.6
IPADDR=`hostname -I|awk '{print $1}'`
HARBOR_ADMIN_PASSWORD=123456

install_docker(){
${COLOR}"Begin install docker-${DOCKER_VERSION}, Please wait..."${END}
sleep 1

dpkg -s docker-ce &> /dev/null && ${COLOR}"Docker already install,exit"${END} && exit
apt update
apt install apt-transport-https ca-certificates curl software-properties-common -y
curl -fsSL https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
apt update -y

${COLOR}"Docker has the following versions"${END}
sleep 2
apt-cache madison docker-ce
${COLOR}"In five seconds begin install: doecker-"${DOCKER_VERSION}"..."${END}
${COLOR}"If you want to install another version,please press Ctrl+c exit,modify version
"${END}
sleep 5

apt install docker-ce=${DOCKER_VERSION} docker-ce-cli=${DOCKER_VERSION} -y
mkdir -p /etc/docker
tee /etc/docker/daemon.json  &> /dev/null <<-'EOF'
{
  "registry-mirrors": ["https://eph8xfli.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl enable --now docker &> /dev/null
docker version && ${COLOR}"Docker install completion"${END} || ${COLOR}"Docker install failure"${END}
}

install_docker_compose(){
${COLOR}"Begin install docker compose..."${END}
sleep 1

#curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-`uname -s`-`uname -m` -o /usr/bin/docker-compose
cp docker-compose-Linux-x86_64 /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose

docker-compose version && ${COLOR}"Docker compose install completion"${END} || ${COLOR}"Docker compose install failure"${END}
}

install_harbor(){
${COLOR}"Begin install harbor..."${END}
sleep 1

#wget https://storage.googleapis.com/harbor-releases/release-1.7.0/harbor-offline-installer-v${HARBOR_VERSION}.tgz
mkdir /apps
tar xvf harbor-offline-installer-v1.7.6.tgz -C /apps

sed -i.bak -e 's/^hostname =.*/hostname = '''${HOSTNAME}'''/' -e 's/^harbor_admin_password =.*/harbor_admin_password = '''${HARBOR_ADMIN_PASSWORD}'''/' /apps/harbor/harbor.cfg

apt install python -y

/apps/harbor/install.sh && ${COLOR}"Harbor install completion"${END} || ${COLOR}"Harbor install failure"${END}
}

harbor_service(){
cat > /lib/systemd/system/harbor.service <<EOF
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/docker-compose -f /apps/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /apps/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable harbor.service && ${COLOR}"Harbor already set boot up"${END}
}

dpkg -s docker-ce &> /dev/null && ${COLOR}"Docker already install"${END} || install_docker

docker-compose version &> /dev/null && ${COLOR}"Docker compose already install"${END} || install_docker_compose

install_harbor

harbor_service
[root@harbor ~]#

验证 harbor

修改hosts ，浏览器访问：harbor.waluna.top

10.0.0.59 harbor.waluna.top


创建一个新仓库

创建成功