2 Keepalived 初步
2.1 keepalived 介绍
vrrp 协议的软件实现，原生设计目的为了高可用ipvs服务
功能：
- 基于vrrp协议完成地址流动
- 为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)
- 为ipvs集群的各RS做健康状态检测
- 基于脚本调用接口完成脚本中定义的功能，进而影响集群事务，以此支持nginx、haproxy等服务
2.2 Keepalived 架构
官方文档：
https://keepalived.org/doc/
https://keepalived.org/documentation.html
- 用户空间核心组件：
vrrp stack：VIP消息通告
checkers：监测real server
system call：实现 vrrp 协议状态转换时调用脚本的功能
SMTP：邮件组件
IPVS wrapper：生成IPVS规则
Netlink Reflector：网络接口
WatchDog：监控进程 - 控制组件：提供 keepalived.conf 的解析器，完成Keepalived配置
- IO复用器：针对网络目的而优化自己的线程抽象
- 内存管理组件：为某些通用的内存管理功能（例如分配，重新分配，发布等）提供访问权限
Keepalived 进程树
Keepalived <-- Parent process monitoring children
\_ Keepalived <-- VRRP child
\_ Keepalived <-- Healthchecking child2.3 Keepalived 环境准备
- 各节点时间必须同步：ntp, chrony
- 关闭防火墙及SELinux
- 各节点之间可通过主机名互相通信：非必须
- 建议使用/etc/hosts文件实现：非必须
- 各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信：非必须
2.4 Keepalived 相关文件
- 软件包名：keepalived
- 主程序文件：/usr/sbin/keepalived
- 主配置文件：/etc/keepalived/keepalived.conf
- 配置文件示例：/usr/share/doc/keepalived/
- Unit File：/usr/lib/systemd/system/keepalived.service
- Unit File的环境配置文件：
- /etc/sysconfig/keepalived CentOS
- /etc/default/keepalived Ubuntu
注意：centos 7 上有bug，可能有下面情况出现
systemctl restart keepalived # 新配置可能无法生效
systemctl stop keepalived;systemctl start keepalived # 无法停止进程，需要kill停止2.5 Keepalived安装
2.5.1 包安装
# centos
[root@centos8 ~]# yum install keepalived -y
# ubuntu
[root@ubuntu1804 ~]# apt install keepalived -y2.5.1.1 CentOS 安装 keepalived
[root@centos8 ~]# dnf install keepalived -y
[root@centos8 ~]# dnf info keepalived
Last metadata expiration check: 0:04:28 ago on Mon 16 Aug 2021 10:24:28 AM CST.
Installed Packages
Name : keepalived
Version : 2.1.5
Release : 6.el8
Architecture : x86_64
Size : 1.5 M
Source : keepalived-2.1.5-6.el8.src.rpm
Repository : @System
From repo : appstream
Summary : High Availability monitor built upon LVS, VRRP and service pollers
URL : http://www.keepalived.org/
License : GPLv2+
Description : Keepalived provides simple and robust facilities for load balancing
: and high availability to Linux system and Linux based infrastructures.
: The load balancing framework relies on well-known and widely used
: Linux Virtual Server (IPVS) kernel module providing Layer4 load
: balancing. Keepalived implements a set of checkers to dynamically and
: adaptively maintain and manage load-balanced server pool according
: their health. High availability is achieved by VRRP protocol. VRRP is
: a fundamental brick for router failover. In addition, keepalived
: implements a set of hooks to the VRRP finite state machine providing
: low-level and high-speed protocol interactions. Keepalived frameworks
: can be used independently or all together to provide resilient
: infrastructures.
[root@centos8 ~]#
[root@centos8 ~]# systemctl enable --now keepalived.service
Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.
[root@centos8 ~]# ps -auxf|grep keepalived
root 3831 0.0 0.1 12112 1100 pts/0 S+ 10:29 0:00 \_ grep --color=auto keepalived
root 3813 0.0 0.3 89876 2428 ? Ss 10:29 0:00 /usr/sbin/keepalived -D
root 3814 0.0 0.5 89988 4464 ? S 10:29 0:00 \_ /usr/sbin/keepalived -D
root 3815 0.0 0.3 89876 2828 ? S 10:29 0:00 \_ /usr/sbin/keepalived -D
[root@centos8 ~]# pstree -p|grep keepalived
|-keepalived(3813)-+-keepalived(3814)
| `-keepalived(3815)2.5.1.2 Ubuntu 安装 keepalived
[root@ubuntu1804 ~]# apt install keepalived -y
[root@ubuntu1804 ~]# dpkg -s keepalived
Package: keepalived
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 824
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 1:1.3.9-1ubuntu0.18.04.2
Depends: iproute2, libc6 (>= 2.27), libglib2.0-0 (>= 2.26.0), libip4tc0 (>= 1.6.0+snapshot20161117), libip6tc0 (>= 1.6.0+snapshot20161117), libnl-3-200 (>= 3.2.27), libnl-genl-3-200 (>= 3.2.7), libnl-route-3-200 (>= 3.2.7), libsnmp30 (>= 5.7.3+dfsg-1.8ubuntu3.1~dfsg), libssl1.1 (>= 1.1.0), libxtables12 (>= 1.6.0+snapshot20161117)
Recommends: ipvsadm
Conffiles:
/etc/dbus-1/system.d/org.keepalived.Vrrp1.conf 6b020ff46c6425d3a9cfa179814d7253
/etc/default/keepalived 6b2e3432e4ae31b444058ba2b0d1f06a
/etc/init.d/keepalived 0312972e0718331b4c90b3b98e623624
Description: Failover and monitoring daemon for LVS clusters
keepalived is used for monitoring real servers within a Linux
Virtual Server (LVS) cluster. keepalived can be configured to
remove real servers from the cluster pool if it stops responding,
as well as send a notification email to make the admin aware of
the service failure.
.
In addition, keepalived implements an independent Virtual Router
Redundancy Protocol (VRRPv2; see rfc2338 for additional info)
framework for director failover.
.
You need a kernel >= 2.4.28 or >= 2.6.11 for keepalived.
See README.Debian for more information.
Homepage: http://keepalived.org
Original-Maintainer: Alexander Wirt <formorer@debian.org>
[root@ubuntu1804 ~]# dpkg -L keepalived
/.
/etc
/etc/dbus-1
/etc/dbus-1/system.d
/etc/dbus-1/system.d/org.keepalived.Vrrp1.conf
/etc/default
/etc/default/keepalived
/etc/init.d
/etc/init.d/keepalived
/etc/keepalived
/lib
/lib/systemd
/lib/systemd/system
/lib/systemd/system/keepalived.service
/usr
/usr/bin
/usr/bin/genhash
/usr/sbin
/usr/sbin/keepalived
/usr/share
/usr/share/dbus-1
/usr/share/dbus-1/interfaces
/usr/share/dbus-1/interfaces/org.keepalived.Vrrp1.Instance.xml
/usr/share/dbus-1/interfaces/org.keepalived.Vrrp1.Vrrp.xml
/usr/share/doc
/usr/share/doc/keepalived
/usr/share/doc/keepalived/AUTHOR
/usr/share/doc/keepalived/CONTRIBUTORS
/usr/share/doc/keepalived/README
/usr/share/doc/keepalived/TODO
/usr/share/doc/keepalived/changelog.Debian.gz
/usr/share/doc/keepalived/copyright
/usr/share/doc/keepalived/keepalived.conf.SYNOPSIS.gz
/usr/share/doc/keepalived/samples
/usr/share/doc/keepalived/samples/client.pem
/usr/share/doc/keepalived/samples/dh1024.pem
/usr/share/doc/keepalived/samples/keepalived.conf.HTTP_GET.port
/usr/share/doc/keepalived/samples/keepalived.conf.IPv6
/usr/share/doc/keepalived/samples/keepalived.conf.SMTP_CHECK
/usr/share/doc/keepalived/samples/keepalived.conf.SSL_GET
/usr/share/doc/keepalived/samples/keepalived.conf.fwmark
/usr/share/doc/keepalived/samples/keepalived.conf.inhibit
/usr/share/doc/keepalived/samples/keepalived.conf.misc_check
/usr/share/doc/keepalived/samples/keepalived.conf.misc_check_arg
/usr/share/doc/keepalived/samples/keepalived.conf.quorum
/usr/share/doc/keepalived/samples/keepalived.conf.sample
/usr/share/doc/keepalived/samples/keepalived.conf.status_code
/usr/share/doc/keepalived/samples/keepalived.conf.track_interface
/usr/share/doc/keepalived/samples/keepalived.conf.virtual_server_group
/usr/share/doc/keepalived/samples/keepalived.conf.virtualhost
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp.localcheck
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp.lvs_syncd
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp.routes
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp.rules
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp.scripts
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp.static_ipaddress
/usr/share/doc/keepalived/samples/keepalived.conf.vrrp.sync
/usr/share/doc/keepalived/samples/root.pem
/usr/share/doc/keepalived/samples/sample.misccheck.smbcheck.sh
/usr/share/doc/keepalived/samples/sample_notify_fifo.sh
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/genhash.1.gz
/usr/share/man/man5
/usr/share/man/man5/keepalived.conf.5.gz
/usr/share/man/man8
/usr/share/man/man8/keepalived.8.gz
/usr/share/snmp
/usr/share/snmp/mibs
/usr/share/snmp/mibs/KEEPALIVED-MIB.txt
/usr/share/snmp/mibs/VRRP-MIB.txt
/usr/share/snmp/mibs/VRRPv3-MIB.txt
[root@ubuntu1804 ~]# cp /usr/share/doc/keepalived/samples/keepalived.conf.sample /etc/keepalived/keepalived.conf
[root@ubuntu1804 ~]# systemctl enable --now keepalived
Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable keepalived
[root@ubuntu1804 ~]# systemctl status keepalived.service
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-08-16 10:32:29 CST; 10s ago
Process: 3129 ExecStart=/usr/sbin/keepalived $DAEMON_ARGS (code=exited, status=0/SUCCESS)
Main PID: 3139 (keepalived)
Tasks: 3 (limit: 2290)
CGroup: /system.slice/keepalived.service
├─3139 /usr/sbin/keepalived
├─3143 /usr/sbin/keepalived
└─3144 /usr/sbin/keepalived
Aug 16 10:32:29 ubuntu1804 Keepalived_vrrp[3144]: Registering Kernel netlink command channel
Aug 16 10:32:29 ubuntu1804 Keepalived_vrrp[3144]: Registering gratuitous ARP shared channel
Aug 16 10:32:29 ubuntu1804 Keepalived_vrrp[3144]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 16 10:32:29 ubuntu1804 Keepalived_vrrp[3144]: Using LinkWatch kernel netlink reflector...
Aug 16 10:32:29 ubuntu1804 Keepalived_vrrp[3144]: VRRP_Instance(VI_1) Entering BACKUP STATE
Aug 16 10:32:30 ubuntu1804 Keepalived_healthcheckers[3143]: Gained quorum 1+0=1 <= 1 for VS [10.10.10.2
Aug 16 10:32:30 ubuntu1804 Keepalived_healthcheckers[3143]: Activating healthchecker for service [192.1
Aug 16 10:32:33 ubuntu1804 Keepalived_vrrp[3144]: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 16 10:32:34 ubuntu1804 Keepalived_vrrp[3144]: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 16 10:32:39 ubuntu1804 Keepalived_healthcheckers[3143]: Timeout connecting server [192.168.200.2]:t
[root@ubuntu1804 ~]# ps auxf|grep keepalived
root 3175 0.0 0.0 14428 1112 pts/0 S+ 10:32 0:00 \_ grep --color=auto keepalived
root 3139 0.0 0.1 91812 3064 ? Ss 10:32 0:00 /usr/sbin/keepalived
root 3143 0.0 0.2 93932 5192 ? S 10:32 0:00 \_ /usr/sbin/keepalived
root 3144 0.0 0.2 94076 5316 ? S 10:32 0:00 \_ /usr/sbin/keepalived
[root@ubuntu1804 ~]# 2.5.2 编译安装
# 安装相关包
[root@centos7 ~]# yum install gcc make curl openssl-devel libnl3-devel net-snmp-devel -y
# 下载源码文件
[root@centos7 ~]# wget https://keepalived.org/software/keepalived-2.2.3.tar.gz
[root@centos7 ~]# tar xvf keepalived-2.2.3.tar.gz -C /usr/local/src/
[root@centos7 ~]# cd /usr/local/src/keepalived-2.2.3/
# 选项--disable-fwmark 可用于禁止iptables规则，可防止VIP无法访问，无此选项默认会启动
[root@centos7 keepalived-2.2.3]# ./configure --prefix=/usr/local/keepalived #--disable-fwmark
[root@centos7 keepalived-2.2.3]# make -j8 && make install
# 如果报下面错误，原因是因为gcc基于c89标准，换成C99标准就可以在for循环内定义变量了：
parser.c: In function ‘read_decimal_unsigned_long_func’:
parser.c:610:3: error: ‘for’ loop initial declarations are only allowed in C99 mode
for (unsigned d = 0; d < shift; d++)
^
parser.c:610:3: note: use option -std=c99 or -std=gnu99 to compile your code
# 由于兼容性问题 换老版本再次安装
[root@centos7 ~]# wget https://keepalived.org/software/keepalived-2.2.2.tar.gz
[root@centos7 ~]# tar xvf keepalived-2.2.2.tar.gz -C /usr/local/src/
[root@centos7 ~]# cd /usr/local/src/keepalived-2.2.2/
[root@centos7 keepalived-2.2.2]# ./configure --prefix=/usr/local/keepalived
[root@centos7 keepalived-2.2.2]# make -j8 && make install
[root@centos7 keepalived-2.2.2]# cd
[root@centos7 ~]# /usr/local/keepalived/sbin/keepalived -v
Keepalived v2.2.2 (03/05,2021)
Copyright(C) 2001-2021 Alexandre Cassen, <acassen@gmail.com>
Built with kernel headers for Linux 3.10.0
Running on Linux 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020
Distro: CentOS Linux 7 (Core)
configure options: --prefix=/usr/local/keepalived
Config options: LVS VRRP VRRP_AUTH VRRP_VMAC OLD_CHKSUM_COMPAT INIT=systemd
System options: VSYSLOG LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTA_VIA IFA_FLAGS NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION LIBIPVS_NETLINK IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE SO_MARK
# 默认会自动生成unit文件
[root@centos7 ~]# cat /usr/lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target
[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target
[root@centos7 ~]# cat /usr/local/keepalived/etc/sysconfig/keepalived
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp -P Only run with VRRP subsystem.
# --check -C Only run with Health-checker subsystem.
# --dont-release-vrrp -V Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs -I Dont remove IPVS topology on daemon stop.
# --dump-conf -d Dump the configuration data.
# --log-detail -D Detailed log messages.
# --log-facility -S 0-7 Set local syslog facility (default=LOG_DAEMON)
#
KEEPALIVED_OPTIONS="-D"
[root@centos7 ~]#
# 默认无法启动
[root@centos7 ~]# systemctl start keepalived.service
Job for keepalived.service failed because the control process exited with error code. See "systemctl status keepalived.service" and "journalctl -xe" for details.
#原因是缺少配置文件导致无法启动
# 创建配置文件
[root@centos7 ~]# mkdir /etc/keepalived
[root@centos7 ~]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
# 再次启动成功
[root@centos7 ~]# systemctl enable --now keepalived.service
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.
[root@centos7 ~]# systemctl status keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-08-16 13:48:17 CST; 12s ago
Process: 11529 ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 11530 (keepalived)
CGroup: /system.slice/keepalived.service
├─11530 /usr/local/keepalived/sbin/keepalived -D
├─11531 /usr/local/keepalived/sbin/keepalived -D
└─11532 /usr/local/keepalived/sbin/keepalived -D
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: HTTP_CHECK on service [192.168.200.....
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: Removing service [192.168.200.4]:tc...8
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: Lost quorum 1-0=1 > 0 for VS [10.10...8
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: Remote SMTP server [192.168.200.1]:....
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: HTTP_CHECK on service [192.168.200.....
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: Removing service [192.168.200.2]:tc...8
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: Lost quorum 1-0=1 > 0 for VS [10.10...8
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: Adding sorry server [192.168.200.20...8
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: Removing alive servers from the poo...8
Aug 16 13:48:24 centos7.waluna.top Keepalived_healthcheckers[11531]: Remote SMTP server [192.168.200.1]:....
Hint: Some lines were ellipsized, use -l to show in full.
# 禁用严格模式
[root@centos7 ~]# vim /etc/keepalived/keepalived.conf
#vrrp_strict # 注释此行
[root@centos7 ~]# systemctl restart keepalived.service
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:dc:0a:2b brd ff:ff:ff:ff:ff:ff
inet 10.0.0.7/24 brd 10.0.0.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 192.168.200.16/32 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.200.17/32 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.200.18/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fedc:a2b/64 scope link
valid_lft forever preferred_lft forever
[root@centos7 ~]# hostname -I
10.0.0.7 192.168.200.16 192.168.200.17 192.168.200.18
[root@centos7 ~]# ping 192.168.200.16
PING 192.168.200.16 (192.168.200.16) 56(84) bytes of data.
64 bytes from 192.168.200.16: icmp_seq=1 ttl=64 time=0.059 ms
64 bytes from 192.168.200.16: icmp_seq=2 ttl=64 time=0.051 ms
^C
--- 192.168.200.16 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1013ms
rtt min/avg/max/mdev = 0.051/0.055/0.059/0.004 ms
[root@centos7 ~]#
Comments | NOTHING