Translated from 日本語 • View original *Please note that this is an old article from more than a year ago. ※
Hello. This is Komiya from the platform.
This time, ELB is high, so it is a record to the effect that it is a small start so let's build LVS. The first thing I wrote was that we were going to load balance both the WEB and the DB with LVS, but in the end, the WEB side was decided to ELB. (It is difficult to explain to customers that it is a public-private mixed flat NW, and the convenience of SSL, scale, and operation, etc.) ）
Things to be careful about more than usual: ・The environment is AWS (be careful not to forget to recreate the instance and disable the Change Source/Dest Check) ・DSR cannot be done if the client is over global (packets that do not match ENI and IP are destroyed on the edge router) ・DSR cannot be done unless it is a flat NW configuration ( This is the same as usual, but a point to be careful of) ・Special circumstances such as the need for NAT instances in multiple NW (that's why I only made one NW) ・DB is distributed with DSR and WEB is distributed with NAT ・VIP can be used, but it is necessary to communicate with the AWS-like administrative API server and switch ・AWS-like administrative splitbrain practically does not happen, but it is necessary to check which one it is attached to. ・Multicast cannot be used due to AWS restrictions, so HA communication uses unicast. ・Certificates are managed on the web server side with load balancing licenses.
Table of Contents: ・Basic configuration ・Basic settings on the distribution target side ・Unit test of load balancing with ipvsadm ・Introduction settings of heartbeat ・Introduction settings of ldirectord ・EIP ・Creation of PrivateIP moving shell ・Resource settings ・Work when copying instances ・Switching test ・Where I got hooked
・Basic configuration
LVS01 (main system) LVS02 (standby system) 80, 443, and 3306 reads are distributed to 2 web, 2 distribution servers, and 2 db, respectively.
lvs01:172.18.1.23 lvs02:172.18.1.25 db-r-vip:172.18.1.200 web-vip:172.18.1.199 db-client:web(local) web-client:any(global) real-db:172.18.1.220,221 real-web:172.18.1.33,34 Web Distribution Method: NAT db Distribution Method: DSR
In the design document, it was divided into a front segment and a back segment, but due to AWS-like specifications, it had to be flat to combine with LVS. ⇒ The web server where the client is over the world is an edge router, and packets that do not match ENI and IP are destroyed, so DSR is impossible for the distributed method and only NAT can be used. The 17th slide is the argument. ⇒ If the distribution method is NAT, the real server needs to specify the load balancing server as the default GW ⇒NAT instances can only have one NIC, but the default GW cannot be specified unless it is the same segment ⇒ at least the WEB must have one segment LVS+ I can't load balance in VPC, but I want to distribute DB and WEB together on the same LVS, and I don't need to change the specifications of the design document because I want to do it together with the same LVS, and I don't need to change the specifications of the design document because I don't need one NW segment, it's a VPC even if there is no ⇒ back segment, and even if the SecurityGroup settings are decent. The reason why it is not HAproxy is because it is not familiar with the operation and LVS seems to have good performance. I'm happy that I can DSR just by DB.
・Basic settings for the distribution target
WEB Settings
vi /etc/sysconfig/network-script/ifcfg-eth0
*The following is added to the web server.
GATEWAY=172.18.1.199
*The distribution server is added as follows.
GATEWAY=172.18.1.206
service network restart
netstat -rn
*If there is a need to communicate with the AWS API and obtain instance IDs with metadata, please note that you will not be able to obtain it if you change GW.
Disable "Change Source/Dest Check" from the Management Console
Place the check file
echo balance `uname -n` > /var/www/html/chk.html
vi /etc/rc.local
echo balance `uname -n` > /var/www/html/chk.html
Set up security groups correctly and check their operation from the browser
DB side configuration Disable arp on the DB server side and upload the lo:0 interface (create an alias for the loopback IF) (There is also a way to route with iptables, but I am worried about overhead, so I use lo:0)
vi /etc/sysctl.conf
# LVS (DSR) reference does not return arp in case of distributed objects
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.eth0.arp_announce = 2
sysctl -p
vi /etc/sysconfig/network-scripts/ifcfg-lo:0
------------------------------------------------
DEVICE=lo:0
TYPE=Ethernet
BOOTPROTO=none
BROADCAST=172.18.1.200
IPADDR=172.18.1.200
NETMASK=255.255.255.255
NETWORK=172.18.1.200
NAME=loopback
ONPARENT=yes
PEERDNS=no
Enabling the Interface
ifup lo:0
ifconfig -a lo:0
Disable "Change Source/Dest Check" from the Management Console
Testing DB Reference Distribution in WEB to DSR Configurations.
mysql -u root -p -h 172.18.1.200 -e show global variables like 'hostname';
・Unit testing of load balancing with ipvsadm
First, from the AWS console screen, disable the Change Source/Dest.Check for all LVS, WEB, and DB instances.
Install ipvsadm
yum install ipvsadm
Kernel parameter adjustment (allows packet forwarding and has the function of a router)
vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
vm.swappiness = 30
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_max_syn_backlog = 8192
net.core.somaxconn = 8192
net.ipv4.tcp_keepalive_intvl = 3
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 10
sysctl -p
Add a virtual IP
ip addr add 172.18.1.200/24 brd 172.18.1.255 dev eth0
ip addr add 172.18.1.199/24 brd 172.18.1.255 dev eth0
ip addr show eth0
Clear settings
ipvsadm -C
FATAL: Error inserting ip_vs (/lib/modules/2.6.32-358.18.1.el6.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko): Unknown symbol in module, or unknown parameter (see dmesg)
Can't initialize ipvs: Protocol not available
Are you sure that IP Virtual Server is built in the kernel or as module?
If you get an error like the one above, disable ipv6 in the same way as the reference link (remove install ipv6 /bin/true) http://pocketstudio.jp/log3/2013/04/18/howto\_disable\_ipv6\_on\_centos/
vi /etc/modprobe.d/ipv6.conf
options ipv6 disable=1
Manually configure Add DB-side distributed settings
ipvsadm -A -t 172.18.1.200:3306
ipvsadm -a -t 172.18.1.200:3306 -r 172.18.1.220:3306 -g -w 10
Added distributed settings on the web side
ipvsadm -A -t 172.18.1.199:80
ipvsadm -a -t 172.18.1.199:80 -r 172.18.1.33:80 -m -w 10
ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.18.1.199:http wlc
-> 172.18.1.33:http Masq 10 0 0
TCP db-r-vip:mysql wlc
-> 172.18.1.220:mysql Route 10 0 0
If you want to save the settings, please click the following (when starting the ipvsadm service)
service ipvsadm save
This time I will be using ldirectord, so be careful because if you don't disable ipvsadm, you will start two and the settings will be rewritten and you will be confused.
・Heartbeat introduction settings
Check with Linux-HA Japan http://linux-ha.sourceforge.jp/wp/dl/packages
cd /usr/local/src
wget http://ftp.iij.ad.jp/pub/sourceforge.jp/linux-ha/58548/pacemaker-1.0.13-1.1.el6.x86_64.repo.tar.gz
cp -a /usr/local/src/pacemaker*.tar.gz /tmp/
cd /tmp
VIRSION=1.0.13-1.1.el6
tar zxvf pacemaker-$VIRSION.x86_64.repo.tar.gz
cd pacemaker-$VIRSION.x86_64.repo
yum install perl-TimeDate PyXML libtool-ltdl OpenIPMI-libs lm_sensors-libs libesmtp
yum -y -c pacemaker.repo install corosync.x86_64 heartbeat.x86_64 pacemaker.x86_64 \
pm_diskd pm_extras pm_logconv-hb
vi /etc/yum.conf
exclude=kernel* pacemaker pacemaker-libs corosync cluster-glue heartbeat resource-agents
vi /etc/ha.d/ha.cf
========================================
crm on
logfile /var/log/ha-log
#debugfile /var/log/ha-debug
logfacility none
debug 0
udpport 34694
keepalive 2
warntime 15
deadtime 30
initdead 60
ucast eth0 172.18.1.23
auto_failback off
node lvs01
node lvs02
uuidfrom nodename
watchdog /dev/watchdog
respawn root /usr/lib64/heartbeat/pingd -m 100 -a default_ping_set
ping 172.18.1.1
respawn root /usr/lib64/heartbeat/diskd -N /dev/xvde1 -a diskcheck_status_internal -i 10
#respawn root /etc/ha.d/monitoring/heartbeat_logmoni.sh monitor
========================================
vi /etc/ha.d/authkeys
========================================
auth 1
1 crc
========================================
chown root:root /etc/ha.d/ha.cf
chown root:root /etc/ha.d/authkeys
chmod 600 /etc/ha.d/authkeys
scp /etc/ha.d/ha.cf lvs02:/etc/ha.d/.
scp /etc/ha.d/authkeys lvs02:/etc/ha.d/.
/etc/init.d/heartbeat start
crm_mon -1
・lDirectorord Introduction Settings
yum -y install ipvsadm perl-Socket6 perl-MailTools \
perl-Net-SSLeay perl-libwww-perl perl-IO-Socket-INET6 ldirectord
vi /etc/ha.d/ldirectord.cf
========================================
#Global Directives
checktimeout=15
checkinterval=30
autoreload=yes
logfile=/var/log/ldirectord.log
quiescent=no
# Virtual for HTTP
virtual = 172.18.1.199:80
real=172.18.1.33:80 masq 5 # web01
real=172.18.1.34:80 masq 5 # web02
fallback=172.18.1.71:80 masq # sorry
checktype=negotiate
service=http
request=chk.html
receive=balance
scheduler=wrr
protocol=tcp
virtual = 172.18.1.206:80
real=172.18.1.44:80 masq 5 # mov01
real=172.18.1.45:80 masq 5 # mov02
fallback=172.18.1.71:80 masq # sorry
checktype=negotiate
service=http
request=chk.html
receive=balance
scheduler=wrr
protocol=tcp
# Virtual for HTTPS
virtual = 172.18.1.199:443
real=172.18.1.33:443 masq 5 # web01
real=172.18.1.34:443 masq 5 # web02
fallback=172.18.1.71:443 masq # sorry
checktype=negotiate
service=https
request=chk.html
receive=balance
scheduler=wrr
persistent=600
protocol=tcp
# MySQL virtual service.
virtual = 172.18.1.200:3306
real=172.18.1.220:3306 gate 4 # db01
real=172.18.1.221:3306 gate 2 # db01
service=mysql
scheduler=wlc
netmask=255.255.255.255
protocol=tcp
checktype=connect
========================================
scp /etc/ha.d/ldirectord.cf lvs02:/etc/ha.d/.
chkconfig ldirectord off
chkconfig --list ldirectord
・Create an EIP/PrivateIP mobile shell It seems that the recent python tools are lighter and better, so I will do it from the python setup.
wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py
easy_install pip
pip install awscli
complete -C aws_completer aws
vi /root/.ec2/aws.config
-----
[default]
aws_access_key_id=AKIAIBLHWHEQNV*****
aws_secret_access_key=OyvqwLtDiB9HZQO44fJryv86t1vjvvmf*****
region=ap-northeast-1
-----
echo export AWS_CONFIG_FILE=/root/.ec2/aws.config ~/.bash_profile
echo complete -C aws_completer aws ~/.bash_profile
export AWS_CONFIG_FILE=/root/.ec2/aws.config
Creating while researching commands
aws ec2 associate-address help
Creating Subcommands
mkdir /opt/{bin,log}
vi /opt/bin/aws_eip_ctl.sh
cat /opt/bin/aws_eip_ctl.sh
-------------------
#!/bin/bash
#
# aws_eip_ctl.sh: Remove ErasticIP and PrivateIP and grant them to local instances
# Dependencies: AWS Commands
# Updated:20131001 - create komiyay
#
if [ $# -lt 2 ]; then
echo argument is missing.
echo usage: $0 erastic_ip private_ip1,private_ip2
exit 1
else
:
fi
export PATH=$PATH:/usr/local/bin
export AWS_CONFIG_FILE=/root/.ec2/aws.config
## variables
EIP=$1
PIP=`echo $2|sed -e 's/,/ /g'`
PIP1=`echo $2|awk -F, '{print $1}'`
PIP2=`echo $2|awk -F, '{print $2}'`
MYinstanceID=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id`
MY_ENI=`aws ec2 describe-network-interfaces --filters Name=attachment.status,Values=attached,Name=attachment.instance-id,Values=$MYinstanceID|grep NetworkInterfaceId|awk '{ print $2}'|sed -e 's/[,]//g'`
## disassociate erastic-ip.
aws ec2 disassociate-address --public-ip ${EIP}
## unassign and assign private-ip.
for i in $PIP; do
attached_ENI=`aws ec2 describe-network-interfaces --filters Name=addresses.private-ip-address,Values=${i} |grep NetworkInterfaceId|awk '{print $2}'|sed -e 's/[,]//g'`
if [ -n ${attached_ENI} ]; then
if [ ${attached_ENI} != ${MY_ENI} ]; then
sleep 2
aws ec2 unassign-private-ip-addresses \
--network-interface-id ${attached_ENI} \
--private-ip-addresses ${i}
## sleep for aws api server.
sleep 18
aws ec2 assign-private-ip-addresses \
--network-interface-id ${MY_ENI} \
--private-ip-addresses ${i} --allow-reassignment
else
echo PrivateIP ${i} is already attached on this instance.
fi
else
echo PrivateIP ${i} is not bind instance.
aws ec2 assign-private-ip-addresses \
--network-interface-id ${MY_ENI} \
--private-ip-addresses ${i} --allow-reassignment
fi
done
sleep 2
## associate
aws ec2 associate-address --instance-id $MYinstanceID \
--public-ip ${EIP} --private-ip-address ${PIP1} --allow-reassociation
exit $?
-------------------
chmod 700 /opt/bin/aws_eip_ctl.sh
Here we test the stand-alone operation with bash -x.
Create LSB-compliant resource scripts
vi /etc/init.d/eip_web
-------
#!/bin/bash
#
# eip Associate EIP.
#
# chkconfig: 2345 99 10
# description: Associate EIP
# Source function library.
. /etc/init.d/functions
ErasticIP=xxx.xxx.131.136
PrivateIP=172.18.1.199,172.18.1.200
prog=eip_web
lock=/var/lock/subsys/$prog
log=/opt/log/aws_error.log
# Source config
if [ -f /etc/sysconfig/$prog ] ; then
. /etc/sysconfig/$prog
fi
case $1 in
start)
echo `date +%Y-%m-%d %T` begin associate-eip >> $log
touch $lock
/opt/bin/aws_eip_ctl.sh ${ErasticIP} ${PrivateIP} >> $log 2>&1
exit $?
;;
stop)
echo `date +%Y-%m-%d %T` end associate-eip >> $log
rm -f $lock
exit $?
;;
status)
if [ -f $lock ]
then
exit 0
else
exit 3
fi
;;
*)
echo $Usage: $0 {start|stop}
exit 1
esac
-------
vi /etc/init.d/eip_mob
-------
#!/bin/bash
#
# eip Associate EIP.
#
# chkconfig: 2345 99 10
# description: Associate EIP
# Source function library.
. /etc/init.d/functions
ErasticIP=xxx.xxx.155.40
PrivateIP=172.18.1.206
prog=eip_mob
lock=/var/lock/subsys/$prog
log=/opt/log/aws_error.log
# Source config
if [ -f /etc/sysconfig/$prog ] ; then
. /etc/sysconfig/$prog
fi
case $1 in
start)
echo `date +%Y-%m-%d %T` begin associate-eip >> $log
touch $lock
/opt/bin/aws_eip_ctl.sh ${ErasticIP} ${PrivateIP} >> $log 2>&1
exit $?
;;
stop)
echo `date +%Y-%m-%d %T` end associate-eip >> $log
rm -f $lock
exit $?
;;
status)
if [ -f $lock ]
then
exit 0
else
exit 3
fi
;;
*)
echo $Usage: $0 {start|stop}
exit 1
esac
-------
chmod +x /etc/init.d/eip*
Copy the above script to the secondary as well.
・Resource Settings
crm
configure
primitive eip_web lsb:eip_web
primitive eip_mob lsb:eip_mob
edit
Now edit the resource group as follows:
group rg_LVS res_Mailto eip_mob eip_web res_VIPcheck_WEB res_VIP_WEB res_VIPcheck_MOB res_VIP_MOB res_VIPcheck_DB res_VIP_DB res_ldirectord
verify
commit
show
node $id=54ce2c1a-7647-851f-2146-e7f2bde2fce4 lvs02
node $id=b6592742-bfd3-dcdf-63cc-8db63f4ea265 lvs01
primitive eip_mob lsb:eip_mob \
op start interval=0s timeout=120s on-fail=restart \
op stop interval=0 timeout=120s on_fail=block \
op monitor interval=30s timeout=120s on_fail=restart
primitive eip_web lsb:eip_web \
op start interval=0s timeout=120s on-fail=restart \
op stop interval=0 timeout=120s on_fail=block \
op monitor interval=30s timeout=120s on_fail=restart
primitive res_Mailto ocf:heartbeat:MailTo \
params email=unyou-all@isao.net subject=(xxxx) \
op start interval=0 timeout=60s on_fail=restart \
op monitor interval=10s timeout=60s on_fail=restart \
op stop interval=0 timeout=60s on_fail=block
primitive res_VIP_DB ocf:heartbeat:IPaddr2 \
params ip=172.18.1.200 nic=eth0 cidr_netmask=24 \
op start interval=0 timeout=60s on_fail=restart \
op stop interval=0 timeout=60s on_fail=block \
op monitor interval=30s timeout=60s on_fail=restart
primitive res_VIP_MOB ocf:heartbeat:IPaddr2 \
params ip=172.18.1.206 nic=eth0 cidr_netmask=24 \
op start interval=0 timeout=60s on_fail=restart \
op stop interval=0 timeout=60s on_fail=block \
op monitor interval=30s timeout=60s on_fail=restart
primitive res_VIP_WEB ocf:heartbeat:IPaddr2 \
params ip=172.18.1.199 nic=eth0 cidr_netmask=24 \
op start interval=0 timeout=60s on_fail=restart \
op stop interval=0 timeout=60s on_fail=block \
op monitor interval=30s timeout=60s on_fail=restart
primitive res_VIPcheck_DB ocf:heartbeat:VIPcheck \
params target_ip=172.18.1.200 count=1 wait=10 \
op start interval=0 timeout=90s on_fail=block \
op stop interval=0 timeout=90s on_fail=block
primitive res_VIPcheck_MOB ocf:heartbeat:VIPcheck \
params target_ip=172.18.1.206 count=1 wait=10 \
op start interval=0 timeout=90s on_fail=block \
op stop interval=0 timeout=90s on_fail=block
primitive res_VIPcheck_WEB ocf:heartbeat:VIPcheck \
params target_ip=172.18.1.199 count=1 wait=10 \
op start interval=0 timeout=90s on_fail=block \
op stop interval=0 timeout=90s on_fail=block
primitive res_ldirectord ocf:heartbeat:ldirectord \
params configfile=/etc/ha.d/ldirectord.cf \
op start interval=0 timeout=60s on_fail=restart \
op monitor interval=30s timeout=60s on_fail=restart \
op stop interval=0
group rg_LVS res_Mailto eip_mob eip_web res_VIPcheck_WEB res_VIP_WEB res_VIPcheck_MOB res_VIP_MOB res_VIPcheck_DB res_VIP_DB res_ldirectord
location group_lvs-location rg_LVS \
rule $id=group_lvs-location-rule 200: #uname eq lvs01 \
rule $id=group_lvs-location-rule-0 100: #uname eq lvs02 \
rule $id=group_lvs-location-rule-1 -inf: defined default_ping_set and default_ping_set lt 100
property $id=cib-bootstrap-options \
dc-version=1.0.13-30bb726 \
cluster-infrastructure=Heartbeat \
expected-quorum-votes=2 \
no-quorum-policy=ignore \
stonith-enabled=false \
startup-fencing=false \
dc-deadtime=20s \
last-lrm-refresh=1380613160
rsc_defaults $id=rsc-options \
resource-stickiness=INFINITY \
migration-threshold=1
*There was a waiting time for a response to the AWS API server, and it took quite a long time to timeout, so the timeout value was increased.
This is the end of the setup, and the next step is to check the operation.
crm_mon -i1 -Af
・Operations when copying instances
vi /etc/ha.d/ha.cf
echo lvs02 > /proc/sys/kernel/hostname; bash
sed -i 's/lvs01/lvs02/g' /etc/sysconfig/network
service rsyslog restart; tail -f /var/log/messages
service postfix restart; tail -f /var/log/maillog
ucast eth0 172.18.1.23
↓
ucast eth0 172.18.1.25
service heartbeat start
*Implement it first from the master and have resources.
tail -f /var/log/ha-log
・Switching test
While confirming with the confirmation command, drop heartbeat and confirm the switching operation.
ip addr show eth0
ipvsadm -Ln
crm_mon -i1 -Af
Ping the representative IP from the outside
ping 172.18.1.199
ping 172.18.1.200
ping 172.18.1.206
・Where I got hooked
ipvsadm did not work unless ipv6 was disabled. In AWS (VPC), DSR was a flat configuration, and it was impossible if the client was not in the local NW. I forgot to easily recreate the instance and make "Change Source/Dest Check" Disable. If you do not attach a fixed EIP other than the floating EIP to lvs, you will not be able to go globally and communicate with the API server, so be careful. If it is not a fixed EIP, the global communication at the first start will not be in time and the resource will not start. If you are a small instance, you can grant up to 3 PrivateIPs to one IF, so be careful as a limit error will occur if there is anything extra. If you change the default GW to something other than the initial value, the metadata returned from the API will be GW's, and you will not be able to get the local one. Due to circumstances such as communicating with the AWS API server, the timeout value must be increased.
・There are a lot of reference sites, but I found them particularly helpful. LVS Reference: Linux Virtual Server Tutorial Stray Penguin - Linux Memo (Ultra Monkey) heartbeat Reference: suz-lab - blog: Creating "High Availability NAT" (CentOS6) Redundant Asterisk Servers on AWS with Heartbeat+Pacemaker - Serverworks Engineer Blog Testing Redundancy Configurations with HeartBeat and Elastic IP - Serverworks Engineer Blog IP and GW Limits and More Information Amazon EC2 Instance Comparison [Fedora] route command gives the error "SIOCADDRT: there is no such process" - Life with IT Linux - Disable NIC offload function - Qiita [Keeta] I got hooked on building an LVS+keepalivated load balancer with CentOS 6 on a DELL server - Infrastructure Engineer way - Powered by HEARTBEATS 2011-11-14-AWS_System_Design How to Set Up Multiple SSL Certificates in an LVS DSR Environment with SSL | Carpe Diem Persistence Handling in LVS
Now, there may be various things to explore, such as how to deal with zone failures without building it across zones. I looked into VRRP, but it was rejected due to operational experience issues. I think the zone failure will feel like you have to work hard manually with AMI and snapshot. I don't have money, so I guess it's like putting it in one place and prioritizing communication speed.
I posted the story of what kind of structure I made when I separated the WEB side into ELB, so please feel free to do so. The story of building LVS on a VPC
Thank you for watching the above.