Translated from 日本語 • View original Hello. This is Komiya from the platform.
Even if you redundish the others, if you don't redundancy the nat instance,
I thought that the servers that are HA in the private segment would not be able to communicate with the AWS API server and would be stuck.
Referring to the articles of our predecessors, I made the following statements.
**・Consider what to do somehow
**
Since I made it with Amazon Linux, it's probably hard to put Heartbeat or something like that, and I don't think I need to use heartbaet in the first place.
It seems that it would be enough to monitor from the standby system and replace the routing table in AWS.
For the standby NAT instance,
If you don't create it, you will have to create, start, and create a routing table, so the switching time will be long.
There is an instance fee, but it seems that it is better to keep it started.
**・For the time being, start by fiddling with the settings of the existing Amazon Linux NAT instance.
**
sudo
# usermod -G wheel user-op
# id user-op
uid=500(user-op) gid=501(user-op) groups=501(user-op),10(wheel)
# visudo
%wheel ALL=(ALL) ALL
Remove the comment.
Check if you can sudo across from other instances
Also, put the python tool in the NAT instance
Set up email sending as well
It seemed that sendmail was running on Amazon Linux.
vi /etc/mail/submit.cf
D{MTAHost}[172.18.10.22]
Djnat01.hoge.com *nat02 changes that way
service sendmail stop
chkconfig sendmail off
yum install mailx
echo hoge |mail -s testkomi komiyay@xxxxx
Matching locales
mv /etc/localtime /etc/localtime.org
ln -s /usr/share/zoneinfo/Asia/Tokyo /etc/localtime
date
Thu Nov 14 10:25:59 JST 2013
crontab -e
# Time Sync
0 * * * * /usr/sbin/ntpdate -bs 172.18.10.24
service ntpd stop
chkconfig ntpd off
Put in kernel tuning
# cp -p /etc/sysctl.conf{,.org}
# vi /etc/sysctl.conf
vm.swappiness = 30
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_max_syn_backlog = 8192
net.core.somaxconn = 8192
net.ipv4.tcp_keepalive_intvl = 3
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 10
# sysctl -p
・Create a separate NAT instance and a routing table with it as the default gateway.
NAT instances copy AMIs into the same segment
Made
i-005a5e02
If you don't set Source/DestCheck to Disable, it won't come out as an exit instance when routing to RouterTable.
rtb-yyyyyyyy
The target of Destination 0.0.0.0/0 (i.e. defoge) in the routing table is
Specify and add to the previous instance
At this point, there are no subnets that are assosiating in this table.
・For the time being**, try plugging the routing table with AWS commands while ping yahoo on the private host**.
Take a look at the manual
# aws ec2 replace-route-table-association help
SYNOPSIS
replace-route-table-association
[--dry-run | --no-dry-run]
--association-id <value>
--route-table-id <value>
--association-id (string)
The ID representing the current association between the original
route table and the subnet.
--route-table-id (string)
The ID of the new route table to associate with the subnet.
# aws ec2 describe-route-tables help
SYNOPSIS
describe-route-tables
[--dry-run | --no-dry-run]
[--route-table-ids <value>]
[--filters <value>]
--filters Name=string1,Values=string1,string2
association.subnet-id
Checking for failover in commands
Reference: http://d.hatena.ne.jp/j3tm0t0/20120814/1344971491
active_rt=rtb-xxxxxxxx
standby_rt=rtb-yyyyyyyy
subnetid=subnet-zzzzzzzz
aws ec2 describe-route-tables --route-table-ids ${active_rt} --filters Name=association.subnet-id,Values=${subnetid}
"Associations": [
{
"SubnetId": "subnet-zzzzzzzz",
"RouteTableAssociationId": "; rtbassoc-7c535b1e",
"RouteTableId": "rtb-xxxxxxxx"
},
association=`aws ec2 describe-route-tables --route-table-ids ${active_rt} --filters Name=association.subnet-id,Values=${subnetid}|grep -A 1 ${subnetid}|awk '{print $2}'|tail -1| sed -e 's/[",]//g'`
# aws ec2 replace-route-table-association --route-table-id ${standby_rt} --association-id $association
{
"NewAssociationId": "rtbassoc-52767e30"
}
# aws ec2 describe-route-tables --filters Name=association.sub
net-id,Values=${subnetid}|grep -A 1 ${subnetid}|awk '{print $2}'|tail -1|sed -e
's/[",]//g'
rtbassoc-52767e30
I was able to confirm from the management console that it had indeed switched to the standby route.
The switching test with the command alone was successful. The ping to yahoo never stopped.
・Consider the conditions for failover
If you think about it with reference below, if you can't communicate with about 3 hosts on the back of the scenes, you can use a failover.
http://d.hatena.ne.jp/hirose31/20131105/1383623672
I think it's easier to use serf. I'll look it up.
Trying to use the messaging tool Serf on EC2 | Developers.IOI tried using Serf - jedipunkz' bloghttps://dl.bintray.com/mitchellh/serf/[Serf] Version upgrade to v0.2.0, I checked what has changed | Pocketstudio.jp Log3Serf-Munin Monitors Munin-Node Automatically Add/Delete | Pocketstudio.jp Log3Automatic Load Balancer with Serf+HAProxy - Glide Note
NAT redundancy reference:
The Abyss of NAT Instance Redundancy - Memosuz-lab - blog: Creating a "High Availability NAT" (CentOS6) cloudpack Blog: (Linked to ELB) Trying to sort in the Host header with HAProxyI tried making a redundant NAT instance - log4moto
I tried to put a serf and looked it up.
As a result, it seems that there is too little information because it is too new, so I decided to do my best with shell scripts.
(There were very few examples of custom user events, so I felt that it did not meet the demand this time.)
It can be a familiar pattern called F/O when iptables drops in pacemaker.
If you ping over ssh from the standby system and about 3 nodes are not good, it seems that F/O is more in line with the actual situation.
Do you want to run it regularly in cron or mon?
I don't have time to ⇒ and it's easy, so cron is fine.
For the time being, it's ping, so it's okay if it's not root.
Register the NAT public key in /home/user-op/.ssh/authorized_keys on all hosts behind it.
$ ssh user-op@nfs02 ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=49 time=37.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=49 time=60.2 ms
It is possible to check like this, so I think about what kind of script and how to execute it
・Think about how to execute the script that fails over.
Completed script↓
mkdir /opt/{bin,log}
vi /opt/bin/chk_backseg_ping.sh
------------------------
#!/bin/bash
#
# chk_backseg_ping.sh: Ping check from the host in the private segment and perform F/O for NAT after failure
# Dependencies: aws commands, /etc/hosts
# Updated:20131114 - create komiyay
#
## variables
title=hoge
datetime=`/bin/date +%Y/%m/%d.%H:%M:%S`
nochk_time=30
up_time=`uptime|grep min|awk '{print $3}'|sed 's/,//g'`
mailto=xxxxx@xxxx.net
log=/opt/log/aws_error.log
backseg='172.18.20'
hostgroup=`grep $backseg /etc/hosts|egrep -v -a 'vip|sorry|^#'|awk '{print $2}'|perl -pe 's/\n/ /g'`
ssh_chkfile=/home/user-op/.ssh/authorized_keys
dir=`echo $(cd $(dirname $0); pwd)`
lockfile=$dir/nat_fo_complete
user=user-op
target1=8.8.8.8
target2=8.8.4.4
target3=yahoo.co.jp
pin_count=4
pin_int=0.5
pin_wait=3
ng_count=3
active_rt=rtb-xxxxxxxx
standby_rt=rtb-yyyyyyyy
subnetid=subnet-zzzzzzzz
## user defined functions
nat_failover() {
echo `date +"%Y-%m-%d %T"` "start replace-route-table-association" >> $log
eval `aws ec2 describe-route-tables|grep -A 2 ${subnetid}|egrep '(RouteTableAssociationId| RouteTableId)'|sed -e 's/[", ]//g'|awk -F : '{print $1"="$2}'`
if [ ${RouteTableId} = ${active_rt} ]
then
aws ec2 replace-route-table-association --route-table-id ${standby_rt} \
--association-id ${RouteTableAssociationId} >> $log 2>&1
result=`echo $?`
echo ${result} > ${lockfile}
failstat="fail from ${active_rt} to ${standby_rt}, segment:${backseg}, status:${result}"
else
aws ec2 replace-route-table-association --route-table-id ${active_rt} \
--association-id ${RouteTableAssociationId} >> $log 2>&1
result=`echo $?`
echo ${result} > ${lockfile}
failstat="fail from ${standby_rt} to ${active_rt}, segment:${backseg}, status:${result}"
fi
}
mail_fail() {
printf "NAT_failover done. please check site.\n ${failstat}" \
|mail -s "(${title}) nat_f/o_${datetime}" ${mailto}
}
## main processing
### failover complete check
if [ -f ${lockfile} ]; then
exit
else
:
fi
## Check invalid few minutes after startup
if [ -n "$up_time" ]; then
if [ $up_time -lt $nochk_time ]; then
exit
else
:
fi
else
:
fi
### ssh check
for i in $hostgroup
do
ssh ${user}@${i} ls ${ssh_chkfile}
result=`echo $?`
if [ $result -eq 0 ]; then
chkhosts="$chkhosts $i"
fi
done
### ping check
ng_hosts=0
for i in $chkhosts
do
target=`printf "$target1\n$target2\n$target3"|sort -R|head -1`
ssh ${user}@${i} ping -q -c ${pin_count} -i ${pin_int} -w ${pin_wait} ${target} > /dev/null 2>&1
result=`echo $?`
if [ ${result} -ne 0 ]; then
ng_hosts=`expr $ng_hosts + 1`
fi
done
### fail over if the ng_hosts equal to or greater than the ng_count
if [ $ng_hosts -ge $ng_count ]
then
if [ -f ${lockfile} ]; then
exit
else
:
fi
nat_failover
mail_fail
else
:
fi
exit 0
------------------------
chmod +x /opt/bin/chk_backseg_ping.sh
You may understand it when you see it, but if I dare to explain it,
hosts, and
If the ping result is NG for 3 or more hosts, re-associate the route on the nat02 side.
I am checking the connectivity of SSH.
The ping target was also randomly specified from the three, so unless it is wiped out,
I won't f/o when I can't go to Target and can go out to Global. Maybe.
If NAT itself fails to reach the API server, it will fail F/O, but
The completion lock file is created and enters an abnormal termination status, so it is not re-executed.
In the case of an environment that regularly stops/starts, it seems that you have to pay attention to the boot order, such as starting NAT last.
If nat is not started, the server behind the HA will not be able to communicate with the API.
I thought it would be better to put a branch such as so that the check does not start within a few minutes.
・Test results
# time bash -x ./chk_backseg_ping.sh
~Omitted~
+ ssh user-op@nfs01 ping -q -c 4 -i 0.5 -w 3 yahoo.co.jp
++ echo 0
+ result=0
+ '[' 0 -ne 0 ']'
+ for i in '$chkhosts'
++ head -1
++ sort -R
++ printf '8.8.8.8\n8.8.4.4\nyahoo.co.jp'
+ target=8.8.8.8
+ ssh user-op@nfs02 ping -q -c 4 -i 0.5 -w 3 8.8.8.8
++ echo 0
+ result=0
+ '[' 0 -ne 0 ']'
+ '[' 0 -ge 3 ']'
+ :
+ exit 0
real 0m19.274s
user 0m0.124s
sys 0m0.280s
It seems that there was no failover because there was no problem with ping communication.
Next, I will stop the iptables of nat01 and make sure that the global is not pinged and f/o.
[root@nat01 ~]# service iptables status
Table: nat
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 MASQUERADE all -- 172.18.0.0/16 0.0.0.0/0
[root@nat01 ~]# service iptables stop
iptables: Flushing firewall rules: [ OK ]
iptables: Setting chains to policy ACCEPT: nat [ OK ]
iptables: Unloading modules: [ OK ]
[root@nat01 ~]# service iptables status
iptables: Firewall is not running.
Performed while pinging from a server in the appropriate back segment.
# time bash -x ./chk_backseg_ping.sh
+ title=hoge
++ /bin/date +%Y/%m/%d.%H:%M:%S
+ datetime=2013/11/15.15:34:16
+ mailto=xxxxxxx@xxxx.net
+ log=/opt/log/aws_error.log
+ backseg=172.18.20
++ awk '{print $2}'
++ perl -pe 's/\n/ /g'
++ egrep -v -a 'vip|sorry|^#'
++ grep 172.18.20 /etc/hosts
+ hostgroup='lvs01 lvs02 cache01 cache02 db01 db02 web01 mov01 nfs01 nfs02 '
+ ssh_chkfile=/home/user-op/.ssh/authorized_keys
++++ dirname ./chk_backseg_ping.sh
+++ cd .
+++ pwd
++ echo /opt/bin
+ dir=/opt/bin
+ lockfile=/opt/bin/nat_fo_complete
+ user=user-op
+ target1=8.8.8.8
+ target2=8.8.4.4
+ target3=yahoo.co.jp
+ pin_count=4
+ pin_int=0.5
+ pin_wait=3
+ ng_count=3
+ active_rt=rtb-xxxxxxxx
+ standby_rt=rtb-yyyyyyyy
+ subnetid=subnet-zzzzzzzz
+ '[' -f /opt/bin/nat_fo_complete ']'
+ :
+ for i in '$hostgroup'
+ ssh user-op@lvs01 ls /home/user-op/.ssh/authorized_keys
/home/user-op/.ssh/authorized_keys
++ echo 0
+ result=0
+ '[' 0 -eq 0 ']'
+ chkhosts=' lvs01'
~Omitted~
+ for i in '$hostgroup'
+ ssh user-op@nfs02 ls /home/user-op/.ssh/authorized_keys
/home/user-op/.ssh/authorized_keys
++ echo 0
+ result=0
+ '[' 0 -eq 0 ']'
+ chkhosts=' lvs01 lvs02 cache01 cache02 db01 db02 web01 mov01 nfs01 nfs02'
+ ng_hosts=0
+ for i in '$chkhosts'
++ head -1
++ sort -R
++ printf '8.8.8.8\n8.8.4.4\nyahoo.co.jp'
+ target=8.8.4.4
+ ssh user-op@lvs01 ping -q -c 4 -i 0.5 -w 3 8.8.4.4
++ echo 1
+ result=1
+ '[' 1 -ne 0 ']'
++ expr 0 + 1
+ ng_hosts=1
+ for i in '$chkhosts'
++ head -1
++ sort -R
++ printf '8.8.8.8\n8.8.4.4\nyahoo.co.jp'
+ target=yahoo.co.jp
+ ssh user-op@lvs02 ping -q -c 4 -i 0.5 -w 3 yahoo.co.jp
++ echo 1
+ result=1
+ '[' 1 -ne 0 ']'
++ expr 1 + 1
+ ng_hosts=2
~Omitted~
+ for i in '$chkhosts'
++ head -1
++ sort -R
++ printf '8.8.8.8\n8.8.4.4\nyahoo.co.jp'
+ target=8.8.4.4
+ ssh user-op@nfs02 ping -q -c 4 -i 0.5 -w 3 8.8.4.4
++ echo 1
+ result=1
+ '[' 1 -ne 0 ']'
++ expr 9 + 1
+ ng_hosts=10
+ '[' 10 -ge 3 ']'
+ '[' -f /opt/bin/nat_fo_complete ']'
+ :
+ nat_failover
++ date '+%Y-%m-%d %T'
+ echo 2013-11-15 15:34:50 'start replace-route-table-association'
++ egrep '(RouteTableAssociationId| RouteTableId)'
++ sed -e 's/[", ]//g'
++ grep -A 2 subnet-zzzzzzzz
++ awk -F : '{print $1"="$2}'
++ aws ec2 describe-route-tables
+ eval RouteTableAssociationId=rtbassoc-6a212908 RouteTableId=rtb-xxxxxxxx
++ RouteTableAssociationId=rtbassoc-6a212908
++ RouteTableId=rtb-xxxxxxxx
+ '[' rtb-xxxxxxxx = rtb-xxxxxxxx ']'
+ aws ec2 replace-route-table-association --route-table-id rtb-yyyyyyyy --association-id rtbassoc-6a212908
++ echo 0
+ result=0
+ echo 0
+ failstat='fail from rtb-xxxxxxxx to rtb-yyyyyyyy, segment:172.18.20, status:0'
+ mail_fail
+ mail -s '(hoge) nat_f/o_2013/11/15.15:34:16' xxxxxxx@xxxx.net
+ printf 'NAT_failover done. please check site.\n fail from rtb-xxxxxxxx to rtb-yyyyyyyy, segment:172.18.20, status:0'
+ exit 0
real 0m35.361s
user 0m1.392s
sys 0m0.456s
The execution time when failover is 35 seconds, so it is okay to run every minute. (If there are too many hosts,
I saw that it could be run in parallel when running in the background, but I didn't really understand it, so I decided not to worry about it.
(I also thought that running in parallel would probably be a DoS attack, so it wouldn't be good.) ）
If you are concerned about the load or execution time, I think it would be a good idea to check on multiple randomly selected hosts instead of all of them.
If you want to change it, it will look like this.
hostgroup=`grep $backseg /etc/hosts|egrep -v -a 'vip|sorry|^#'|awk '{print $2}'|perl -pe 's/\n/ /g'`
↓
check_num=6
hostgroup=`grep $backseg /etc/hosts|egrep -v -a 'vip|sorry|^#'|awk '{print $2}'|sort -R|head -${check_num}|perl -pe 's/\n/ /g'`
I made a lock file to prevent cron registration without thinking about it being executed multiple times.
You will need to manage it manually when you recover.
If you test with a lock file, it will end immediately after the variable is defined.
# time bash -x ./chk_backseg_ping.sh
~Omitted~
+ pin_wait=3
+ ng_count=3
+ active_rt=rtb-xxxxxxxx
+ standby_rt=rtb-yyyyyyyy
+ subnetid=subnet-zzzzzzzz
+ '[' -f /opt/bin/nat_fo_complete ']'
+ exit
real 0m0.039s
user 0m0.000s
sys 0m0.012s
The script I referenced is listed in the URL below.
I tried making a redundant NAT instance - log4moto
Personally, I'm a little concerned about whether I can ping the yahoo.co.jp so much.
8.~ It seems to be Google, so I'll make it okay.
There seems to be something like this. There is also an opinion that I will try pinging yahoo instead of jp.
ping check site (trial operation) - Academic Information Network (SINET4, SINET4) ・ For the time being, cron registration every 2 minutes (on standby machine)
# crontab -e
# crontab -l
## nat check and failover script
*/2 * * * * /opt/bin/chk_backseg_ping.sh
I will add it because it was corrected at a later date.
# diff chk_backseg_ping.sh old/chk_backseg_ping.sh.20140210
31d30
< current_ids=$dir/current_ids
36,41c35
> aws ec2 describe-route-tables|grep -A 2 ${subnetid}|egrep '(RouteTableAssociationId| RouteTableId)'|sed -e 's/[", ]//g'|awk -F : '{print $1"="$2}' > $current_ids
> source $current_ids
> if [ -z "${RouteTableAssociationId}" ]; then
> printf "Failed to set the value of RouteTableAssociationId and RouteTableId.. \n Nat-failover did not start. Maybe aws-api-server unreachable."| tee -a ${log}|mail -s "(${title}) nat_f/o_${datetime}" ${mailto}
> exit
< fi
---
> eval `aws ec2 describe-route-tables|grep -A 2 ${subnetid}|egrep '(RouteTableAssociationId| RouteTableId)'|sed -e 's/[", ]//g'|awk -F : '{print $1"="$2}'`
Thank you for watching the above.