锘??xml version="1.0" encoding="utf-8" standalone="yes"?>
One of these technologies, keepalived, provides interface failover and the ability to perform application-layer health checks. When these capabilities are combined with the Linux Virtual Server (LVS) project, a fault in an application will be detected by keepalived, and the virtual interfaces that are accessed by clients can be migrated to another available node. This article will provide an introduction to keepalived, and will show how to configure interface failover between two or more nodes. Additionally, the article will show how to debug problems with keepalived and VRRP.
The keepalived project provides a keepalive facility for Linux servers. This keepalive facility consists of a VRRP implementation to manage virtual routers (aka virtual interfaces), and a health check facility to determine if a service (web server, samba server, etc.) is up and operational. If a service fails a configurable number of health checks, keepalived will fail a virtual router over to a secondary node. While useful in its own right, keepalived really shines when combined with the Linux Virtual Server project. This article will focus on keepalived, and a future article will show how to integrate the two to create a fault tolerant load-balancer.
Before we dive into configuring keepalived, we need to install it. Keepalived is distributed as source code, and is available in several package repositories. To install from source code, you can execute wget or curl to retrieve the source, and then run "configure", "make" and "make install" compile and install the software:
$ wget http://www.keepalived.org/software/keepalived-1.1.17.tar.gz $ tar xfvz keepalived-1.1.17.tar.gz $ cd keepalived-1.1.17 $ ./configure --prefix=/usr/local $ make && make install
In the example above, the keepalived daemon will be compiled and installed as /usr/local/sbin/keepalived.
The keepalived daemon is configured through a text configuration file, typically named keepalived.conf. This file contains one or more configuration stanzas, which control notification settings, the virtual interfaces to manage, and the health checks to use to test the services that rely on the virtual interfaces. Here is a sample annotated configuration that defines two virtual IP addresses to manage, and the individuals to contact when a state transition or fault occurs:
# Define global configuration directives global_defs { # Send an e-mail to each of the following # addresses when a failure occurs notification_email { matty@prefetch.net operations@prefetch.net } # The address to use in the From: header notification_email_from root@VRRP-director1.prefetch.net # The SMTP server to route mail through smtp_server mail.prefetch.net # How long to wait for the mail server to respond smtp_connect_timeout 30 # A descriptive name describing the router router_id VRRP-director1 } # Create a VRRP instance VRRP_instance VRRP_ROUTER1 { # The initial state to transition to. This option isn't # really all that valuable, since an election will occur # and the host with the highest priority will become # the master. The priority is controlled with the priority # configuration directive. state MASTER # The interface keepalived will manage interface br0 # The virtual router id number to assign the routers to virtual_router_id 100 # The priority to assign to this device. This controls # who will become the MASTER and BACKUP for a given # VRRP instance. priority 100 # How many seconds to wait until a gratuitous arp is sent garp_master_delay 2 # How often to send out VRRP advertisements advert_int 1 # Execute a notification script when a host transitions to # MASTER or BACKUP, or when a fault occurs. The arguments # passed to the script are: # $1 - "GROUP"|"INSTANCE" # $2 = name of group or instance # $3 = target state of transition # Sample: VRRP-notification.sh VRRP_ROUTER1 BACKUP 100 notify "/usr/local/bin/VRRP-notification.sh" # Send an SMTP alert during a state transition smtp_alert # Authenticate the remote endpoints via a simple # username/password combination authentication { auth_type PASS auth_pass 192837465 } # The virtual IP addresses to float between nodes. The # label statement can be used to bring an interface # online to represent the virtual IP. virtual_ipaddress { 192.168.1.100 label br0:100 192.168.1.101 label br0:101 } }
The configuration file listed above is self explanatory, so I won't go over each directive in detail. I will point out a couple of items:
Keepalived can be executed from an RC script, or started from the command line. The following example will start keepalived using the configuration file /usr/local/etc/keepalived.conf:
$ keepalived -f /usr/local/etc/keepalived.conf
If you need to debug keepalived issues, you can run the daemon with the "--dont-fork", "--log-console" and "--log-detail" options:
$ keepalived -f /usr/local/etc/keepalived.conf --dont-fork --log-console --log-detail
These options will stop keepalived from fork'ing, and will provide additional logging data. Using these options is especially useful when you are testing out new configuration directives, or debugging an issue with an existing configuration file.
To see which director is currently the master for a given virtual interface, you can check the output from the ip utility:
VRRP-director1$ ip addr list br0 5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 00:24:8c:4e:07:f6 brd ff:ff:ff:ff:ff:ff inet 192.168.1.6/24 brd 192.168.1.255 scope global br0 inet 192.168.1.100/32 scope global br0:100 inet 192.168.1.101/32 scope global br0:101 inet6 fe80::224:8cff:fe4e:7f6/64 scope link valid_lft forever preferred_lft forever VRRP-director2$ ip addr list br0 5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 00:24:8c:4e:07:f6 brd ff:ff:ff:ff:ff:ff inet 192.168.1.7/24 brd 192.168.1.255 scope global br0 inet6 fe80::224:8cff:fe4e:7f6/64 scope link valid_lft forever preferred_lft forever
In the output above, we can see that the virtual interfaces 192.168.1.100 and 192.168.1.101 are currently active on VRRP-director1.
The keepalived daemon will log to syslog by default. Log entries will range from entries that show when the keepalive daemon started, to entries that show state transitions. Here are a few sample entries that show keepalived starting up, and the node transitioning a VRRP instance to the MASTER state:
Jul 3 16:29:56 disarm Keepalived: Starting Keepalived v1.1.17 (07/03,2009) Jul 3 16:29:56 disarm Keepalived: Starting VRRP child process, pid=1889 Jul 3 16:29:56 disarm Keepalived_VRRP: Using MII-BMSR NIC polling thread... Jul 3 16:29:56 disarm Keepalived_VRRP: Registering Kernel netlink reflector Jul 3 16:29:56 disarm Keepalived_VRRP: Registering Kernel netlink command channel Jul 3 16:29:56 disarm Keepalived_VRRP: Registering gratutious ARP shared channel Jul 3 16:29:56 disarm Keepalived_VRRP: Opening file '/usr/local/etc/keepalived.conf'. Jul 3 16:29:56 disarm Keepalived_VRRP: Configuration is using : 62990 Bytes Jul 3 16:29:57 disarm Keepalived_VRRP: VRRP_Instance(VRRP_ROUTER1) Transition to MASTER STATE Jul 3 16:29:58 disarm Keepalived_VRRP: VRRP_Instance(VRRP_ROUTER1) Entering MASTER STATE Jul 3 16:29:58 disarm Keepalived_VRRP: Netlink: skipping nl_cmd msg...
If you are unable to determine the source of a problem with the system logs, you can use tcpdump to display the VRRP advertisements that are sent on the local network. Advertisements are sent to a reserved VRRP multicast address (224.0.0.18), so the following filter can be used to display all VRRP traffic that is visible on the interface passed to the "-i" option:
$ tcpdump -vvv -n -i br0 host 224.0.0.18 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on br0, link-type EN10MB (Ethernet), capture size 96 bytes 10:18:23.621512 IP (tos 0x0, ttl 255, id 102, offset 0, flags [none], proto VRRP (112), length 40) \ 192.168.1.6 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20, addrs: 192.168.1.100 auth "19283746" 10:18:25.621977 IP (tos 0x0, ttl 255, id 103, offset 0, flags [none], proto VRRP (112), length 40) \ 192.168.1.6 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20, addrs: 192.168.1.100 auth "19283746" .........
The output contains several pieces of data that be useful for debugging problems:
authtype - the type of authentication in use (authentication configuration directive) vrid - the virtual router id (virtual_router_id configuration directive) prio - the priority of the device (priority configuration directive) intvl - how often to send out advertisements (advert_int configuration directive) auth - the authentication token sent (auth_pass configuration directive)
In this article I described how to set up a host to use the keepalived daemon, and provided a sample configuration file that can be used to failover virtual interfaces between servers. Keepalived has a slew of options not covered here, and I will refer you to the keepalived source code and documentation for additional details
鍦ㄤ竴涓竴涓誨澶囩殑Keepalived闆嗙兢涓紝“priority”鍊兼渶澶х殑灝嗘垚涓洪泦緹や腑鐨凪aster鑺傜偣錛岃屽叾浠栭兘鏄疊ackup鑺傜偣銆傚湪Master鑺傜偣鍙戠敓鏁呴殰鍚庯紝Backup鑺傜偣涔嬮棿灝嗚繘琛?#8220;姘戜富閫変婦”錛岄氳繃瀵硅妭鐐逛紭鍏堢駭鍊?#8220;priority”鍜?#8220;“weight”鐨勮綆楋紝閫夊嚭鏂扮殑Master鑺傜偣鎺ョ闆嗙兢鏈嶅姟銆?/p>
鍦╲rrp_script妯″潡涓紝濡傛灉涓嶈緗?#8220;weight”閫夐」鍊鹼紝閭d箞闆嗙兢浼樺厛綰х殑閫夋嫨灝嗙敱Keepalived閰嶇疆鏂囦歡涓殑“priority”鍊煎喅瀹氾紝鑰屽湪闇瑕佸闆嗙兢涓紭鍏堢駭榪涜鐏墊椿鎺у埗鏃訛紝鍙互閫氳繃鍦╲rrp_script妯″潡涓緗?#8220;weight”鍊兼潵瀹炵幇銆備笅闈㈠垪涓句竴涓疄渚嬫潵鍏蜂綋璇存槑銆?/p>
鍋囧畾鏈堿鍜孊涓よ妭鐐圭粍鎴愮殑Keepalived闆嗙兢錛屽湪A鑺傜偣keepalived.conf鏂囦歡涓紝璁劇疆“priority”鍊間負100錛岃屽湪B鑺傜偣keepalived.conf鏂囦歡涓紝璁劇疆“priority”鍊間負80錛屽茍涓擜銆丅涓や釜鑺傜偣閮戒嬌鐢ㄤ簡“vrrp_script”妯″潡鏉ョ洃鎺ysql鏈嶅姟錛屽悓鏃墮兘璁劇疆“weight”鍊間負10錛岄偅涔堝皢浼氬彂鐢熷涓嬫儏鍐點?/p>
鍦ㄤ袱鑺傜偣閮藉惎鍔↘eepalived鏈嶅姟鍚庯紝姝e父鎯呭喌鏄疉鑺傜偣灝嗘垚涓洪泦緹や腑鐨凪aster鑺傜偣錛岃孊鑷姩鎴愪負Backup鑺傜偣錛屾鏃跺皢A鑺傜偣鐨刴ysql鏈嶅姟鍏抽棴錛岄氳繃鏌ョ湅鏃ュ織鍙戠幇錛屽茍娌℃湁鍑虹幇B鑺傜偣鎺ョA鑺傜偣鐨勬棩蹇楋紝B鑺傜偣浠嶇劧澶勪簬Backup鐘舵侊紝鑰孉鑺傜偣渚濇棫鏄疢aster鐘舵侊紝鍦ㄨ繖縐嶆儏鍐典笅鏁翠釜HA闆嗙兢灝嗗け鍘繪剰涔夈?/p>
涓嬮潰灝卞垎鏋愪竴涓嬩駭鐢熻繖縐嶆儏鍐電殑鍘熷洜錛岃繖涔熷氨鏄疜eepalived闆嗙兢涓富銆佸瑙掕壊閫変婦絳栫暐鐨勯棶棰樸備笅闈㈡葷粨浜嗗湪Keepalived涓嬌鐢╲rrp_script妯″潡鏃舵暣涓泦緹よ鑹茬殑閫変婦綆楁硶錛岀敱浜?#8220;weight”鍊煎彲浠ユ槸姝f暟涔熷彲浠ユ槸璐熸暟錛屽洜姝わ紝瑕佸垎涓ょ鎯呭喌榪涜璇存槑銆?/p>
1. “weight”鍊間負姝f暟鏃?/p>
鍦╲rrp_script涓寚瀹氱殑鑴氭湰濡傛灉媯嫻嬫垚鍔燂紝閭d箞Master鑺傜偣鐨勬潈鍊煎皢鏄?#8220;weight鍊間笌”priority“鍊間箣鍜岋紝濡傛灉鑴氭湰媯嫻嬪け璐ワ紝閭d箞Master鑺傜偣鐨勬潈鍊間繚鎸佷負“priority”鍊鹼紝鍥犳鍒囨崲絳栫暐涓猴細
Master鑺傜偣“vrrp_script”鑴氭湰媯嫻嬪け璐ユ椂錛屽鏋淢aster鑺傜偣“priority”鍊煎皬浜嶣ackup鑺傜偣“weight鍊間笌”priority“鍊間箣鍜岋紝灝嗗彂鐢熶富銆佸鍒囨崲銆?/p>
Master鑺傜偣“vrrp_script”鑴氭湰媯嫻嬫垚鍔熸椂錛屽鏋淢aster鑺傜偣“weight”鍊間笌“priority”鍊間箣鍜屽ぇ浜嶣ackup鑺傜偣“weight”鍊間笌“priority”鍊間箣鍜岋紝涓昏妭鐐逛緷鐒朵負涓昏妭鐐癸紝涓嶅彂鐢熷垏鎹€?/p>
2. “weight”鍊間負璐熸暟鏃?/p>
鍦?#8220;vrrp_script”涓寚瀹氱殑鑴氭湰濡傛灉媯嫻嬫垚鍔燂紝閭d箞Master鑺傜偣鐨勬潈鍊間粛涓?#8220;priority”鍊鹼紝褰撹剼鏈嫻嬪け璐ユ椂錛孧aster鑺傜偣鐨勬潈鍊煎皢鏄?#8220;priority“鍊間笌“weight”鍊間箣宸紝鍥犳鍒囨崲絳栫暐涓猴細
Master鑺傜偣“vrrp_script”鑴氭湰媯嫻嬪け璐ユ椂錛屽鏋淢aster鑺傜偣“priority”鍊間笌“weight”鍊間箣宸皬浜嶣ackup鑺傜偣“priority”鍊鹼紝灝嗗彂鐢熶富銆佸鍒囨崲銆?/p>
Master鑺傜偣“vrrp_script”鑴氭湰媯嫻嬫垚鍔熸椂錛屽鏋淢aster鑺傜偣“priority”鍊煎ぇ浜嶣ackup鑺傜偣“priority”鍊兼椂錛屼富鑺傜偣渚濈劧涓轟富鑺傜偣錛屼笉鍙戠敓鍒囨崲銆?/p>
鍦ㄧ啛鎮変簡Keepalived涓匯佸瑙掕壊鐨勯変婦絳栫暐鍚庯紝鍐嶆潵鍒嗘瀽涓涓嬪垰鎵嶅疄渚嬶紝鐢變簬A銆丅涓や釜鑺傜偣璁劇疆鐨?#8220;weight”鍊奸兘涓?0錛屽洜姝ょ鍚堥変婦絳栫暐鐨勭涓縐嶏紝鍦ˋ鑺傜偣鍋滄Mysql鏈嶅姟鍚庯紝A鑺傜偣鐨勮剼鏈嫻嬪皢澶辮觸錛屾鏃禔鑺傜偣鐨勬潈鍊煎皢淇濇寔涓篈鑺傜偣涓婅緗殑“priority”鍊鹼紝鍗充負100錛岃孊鑺傜偣鐨勬潈鍊煎皢鍙樹負“weight”鍊間笌“priority”鍊間箣鍜岋紝涔熷氨鏄?0錛?0+80錛夛紝榪欐牱灝卞嚭鐜頒簡A鑺傜偣鏉冨間粛鐒跺ぇ浜嶣鑺傜偣鏉冨肩殑鎯呭喌錛屽洜姝や笉浼氬彂鐢熶富銆佸鍒囨崲銆?/p>
瀵逛簬“weight”鍊肩殑璁劇疆錛屾湁涓涓畝鍗曠殑鏍囧噯錛屽嵆“weight”鍊肩殑緇濆鍊艱澶т簬Master鍜孊ackup鑺傜偣“priority”鍊間箣宸傚浜庝笂闈銆丅涓や釜鑺傜偣鐨勪緥瀛愶紝鍙璁劇疆“weight”鍊煎ぇ浜?0鍗沖彲淇濊瘉闆嗙兢姝e父榪愯鍜屽垏鎹€傜敱姝ゅ彲瑙侊紝瀵逛簬“weight鍊肩殑璁劇疆錛岃闈炲父璋ㄦ厧錛屽鏋滆緗笉濂斤紝灝嗗鑷撮泦緹よ鑹查変婦澶辮觸錛屼嬌闆嗙兢闄蜂簬鐦棯鐘舵併?/p>