Verify MHA operation and switchover verification

MHA operation confirmation and switchover verification This is the continuation of the previous article. Test to switch as shown in the following diagram.

　*This verification consists of 1 master, 2 slaves, and 1 manager. (Multi-stage configuration is avoided because recovery in the event of an intermediate node failure will be obscene)
　　If you let the slave live with the manager, it seems that it may not switch well if you put purge_relay_logs on it.
　*It seems that there are cases where there are only two units and it does not switch well. See below.
　　http://heartbeats.jp/hbblog/2013/05/mysql-mha-haproxy.html

(6) Pre-start check

・Check the operation of SSH
[shell] # masterha_check_ssh --conf=/etc/app1.cnf [/shell] If OK, the final output is as follows:
[shell] Tue Oct 23 15:02:22 2012 - [info] All SSH connection tests passed successfully. [/shell]
・Checking the operation of replication
[shell] # masterha_check_repl --conf=/etc/app1.cnf [/shell] If OK, the final output is as follows (*From mysql5.6 onwards, depending on the MHA version, you may fail here if you don't set it to binlog-checksum=NONE).
[shell] MySQL Replication Health is OK. [/shell] *Pattern that fails
Replication filtering rules are not correct
If the LVS and MHA manager are carpooling, the check timing will overlap and will be rejected by the DB server for each host (flush hosts;)

(7) Check the updated VIP IF file
db01/db02
[shell] cat /etc/sysconfig/network-scripts/ifcfg-eth1:0 ===================================================== # Intel Corporation 82576 Gigabit Network Connection DEVICE=eth0 :0 BOOTPROTO=static BROADCAST=192.168.100.255 #HWADDR= IPADDR=192.168.100.5 NETMASK=255.255.255.0 NETWORK=192.168.100.0 ONPARENT=no ===================================================== [/shell] *In the case of virtual IF, ONBOOT does not respond, so ONPARENT=no is specified.

db01
[shell] # ifup eth1:0 [/shell]
(8) Starting, confirming, and stopping MHA Manager

• Start Manager
  [shell] # masterha_manager --conf=/etc/app1.cnf & [/shell] *Pattern that fails
  Replication configuration is broken
  It has already been switched and a status file showing completion is printed
  Be careful not to switch well if you look at the log with tail in the terminal where you started the manager.

• Manager status check
  [shell] # masterha_check_status --conf=/etc/app1.cnf [/shell] OK
  [shell] app1 (pid:9883) is running(0:PING_OK), master:192.168.100.1 [/shell]
  [shell] app1 is stopped(2:NOT_RUNNING). [/shell] In the case of the initialization status immediately after startup, it is output as such.
  *If this state continues for a long time, a kill of -9 may require a forced stop.

・Suspension of managers
[shell] # masterha_stop --conf=/etc/app1.cnf

# masterha_check_status --conf=/etc/app1.cnf [/shell] ・If the manager does not stop when it is in the initialized state,
[shell] # ps -ef|grep master [/shell] Copy the process id
[shell] # kill -9 [/shell] Paste the copied PID to force kill
[shell] # masterha_check_status --conf=/etc/app1.cnf # ls -l /var/log/masterha/app1/ [/shell] Delete the status file if it remains.

■MHA Switching Test and Confirmation and Recovery■

*Data update and switching test are omitted. It is mainly to check whether the master is dropped and switched. I think it is also a good idea to additionally check that it does not switch after dropping slave.

(1) Switching test
・Pre-check
[shell] # tail -f /var/log/masterha/app1/manager.log # masterha_check_status --conf=/etc/app1.cnf # ls -lh /var/log/masterha/app1/ # masterha_check_repl --conf=/etc/app1.cnf \ [/shell] ・Tests to drop mysqld
At the old master
[shell] # service mysqld stop # service mysqld status [/shell] Make sure mysqld is stopped

・Test to drop the interface
At the old master
[shell] #ifdown eth1 #ifdown eth1:1 [/shell] ・ Test to close the port in mysqld
At the old master
[shell] # iptables -P INPUT ACCEPT # iptables -A INPUT -p tcp --dport 3306 -j DROP # netstat -lnpt [/shell] Basically accept all inbound packets
Drop packets destined for TCP port 3306
Verify port 3306 closed

(2) Confirmation

・Check the status of the MHA manager
[shell] # tail -f /var/log/masterha/app1/manager.log # masterha_check_status --conf=/etc/app1.cnf # ls -lh /var/log/masterha/app1/ [/shell] Confirm the existence of a switchover completion file called app1.failover.complete

・Check if the VIP has been switched
At the old and new masters
[shell] # ifconfig -a [/shell] Confirm that the update VIP has been removed from the old master and the update VIP has been transferred to the new master
・Confirm that the slave of the new master is stopped and the slave is looking at the new master.
[sql] > show slave status\G [/sql] Make sure that the slave_*_Running of the new master is set to No
Slave Master_Host: Ensures that the new master is facing
[sql] > show global variables like 'read_only'; [/sql] The new master is turned off and can be updated.

• Verify that the weight of the LVS server is as expected (LVS01)
  [shell] # ipvsadm -L --sort # ls -la /etc/ha.d/ldirectord.cf* [/shell] YYYYMMDD. Ensure that HHMM files are backed up
  [shell] # diff /etc/ha.d/ldirectord.cf{,.failover} [/shell] Make sure there are no diffs (make sure the failover file is successfully overwritten)

(3) Configuration restoration

• Start the old master mysqld
  [shell] # netstat -lnpt # service mysqld start # service mysqld status [/shell] ・Start the interface of the old master (DB01)
  [shell] # ifup eth1 # ifconfig [/shell] ・Initialize the port block of the old master
  [shell] # service iptables restart # iptables -ln [/shell] ・Restore replication configuration (*If there is a data difference, you need to dump and input the difference)
  At the old master
  [sql] # mysql -u root -p`cat /path_to_file` > reset master; > show master status; > show slave status\G [/sql] in the new master and slave
  [sql] # mysql -u root -p`cat /path_to_file` > show global variables like 'read_only'; > set global read_only=1; > stop slave; > reset slave; > CHANGE MASTER TO MASTER_HOST='192.168.100.1', MASTER_USER='repl', MASTER_PASSWORD='*******', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=106; > start slave; > show slave status\G [/sql] ・Revert to the weight of the LVS server
  [shell] # ls -la /etc/ha.d/ldirectord.cf* [/shell] YYYYMMDD. Check the HHMM file
  [shell] # mv /etc/ha.d/ldirectord.cf.YYYYMMDD.HHMM /etc/ha.d/ldirectord.cf [/shell] ・Delete unnecessary files
  [shell] # ls -lh /var/log/masterha/app1/ # rm -f /var/log/masterha/app1/app1.failover.complete # rm -f /var/log/masterha/app1/app1.failover.error # rm -f /var/log/masterha/app1/saved_master_binlog_from_192* [/shell] Delete the switched file
  Delete a switchover termination file
  Delete binary log backups of old masters
  *It is turned off because it is an exam environment.

・Manual replacement of renewal VIP
New Master
[shell] # ifdown eth1:0 # ifconfig eth1:0 [/shell] Make sure the IP is not attached

Old Master
[shell] #ifup eth1:0 #ifconfig eth1:0 [/shell] Make sure the IP is on
If you are on an AWS VPC, you will not be able to access it from the outside without changing your private address in AWS.
ec2-unassign-private-ip-addresses --network-interface eni-7060xxxx --secondary-private-ip-address (PrivateIP)
ec2-assign-private-ip-addresses --network-interface eni-4b64xxxx --secondary-private-ip-address (PrivateIP)

・Restart MHA Manager
Confirm
[shell] # masterha_check_repl --conf=/etc/app1.cnf # masterha_check_status --conf=/etc/app1.cnf [/shell] Start
[shell] # masterha_manager --conf=/etc/app1.cnf & # masterha_check_status --conf=/etc/app1.cnf # tail -f /var/log/masterha/app1/manager.log # ls -lh /var/log/masterha/app1/ [/ shell] ・If the MHA manager has remained in the initialized state for a long time
[shell] # masterha_stop --conf=/etc/app1.cnf [/shell] If this command fails, force kill.
[shell] # masterha_check_status --conf=/etc/app1.cnf # ps -ef|grep master [/shell] Copy the process ID and specify it in the kill command
[shell] # kill -9 # masterha_check_status --conf=/etc/app1.cnf # ls -l /var/log/masterha/app1/ [/shell] Delete the status file if it remains

・If the MHA manager is down and you want to manually switch the master (and VIP for renewal)
[shell] # masterha_master_switch --master_state=alive --conf=/etc/app1.cnf [/shell] *I agree because I will be asked if I can really do it. When the MHA manager is activated, he scolds me to drop it.
(*VIP does not switch unless you add processing)
Please see here for details

If you want to switch VIP as well, modify it as follows (please match the IF information to the environment)
[shell] ========================== 33,40d32 < my $vip = '192.168.0.245/24'; # Write Virtual IP < my $key = "0"; < my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip"; < my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down"; < my $ssh_user = "root"; < my $orig_master_host = "192.168.0.248"; < my $new_master_host = "192.168.0.249"; < 47c39 < 'master_state=s' => \$master_state, --- > 'master_state=s' => \$master_state 50,58d41 < # A simple system call that enable the VIP on the new master < sub start_vip() { < `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; < } < # A simple system call that disable the VIP on the old_master < sub stop_vip() { < `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; < } < 72,76d54 < if ( $exit_code eq 0 ) { < &stop_vip(); < sleep 1; < &start_vip(); < } 80,84d57 < if ( $exit_code eq 0 ) { < &stop_vip(); < sleep 1; < &start_vip(); < } ========================== [/shell] ・secondaryが生きているか確認

[shell] # masterha_secondary_check -s 192.168.0.249 --user=root --master_host=test-db02 --master_ip=192.168.0.245 --master_port=3306 [/shell] ※-sの後ろを追加すれば複数確認できる

■参考資料■
・MHAの動作フェーズ
[shell] # grep Phase manager.log |head -20|grep -v completed * Phase 1: Configuration Check Phase.. * Phase 2: Dead Master Shutdown Phase.. * Phase 3: Master Recovery Phase.. * Phase 3.1: Getting Latest Slaves Phase.. * Phase 3.2: Saving Dead Master's Binlog Phase.. * Phase 3.3: Determining New Master Phase.. * Phase 3.3: New Master Diff Log Generation Phase.. * Phase 3.4: Master Log Apply Phase.. * Phase 4: Slaves Recovery Phase.. * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. * Phase 4.2: Starting Parallel Slave Log Apply Phase.. * Phase 5: New master cleanup phease.. [/shell]
フェイルオーバ時の動作は以下のとおり。（ログから追った動き）

※SQL処理のスレッド実行が終わった後
①config(/etc/app1.cnf)から各ノード情報を読み込む
②newMasterのVIPを停止する
③newMasteのmysqldを停止
④各Slaveリレーログを解析して次マスターの選出と差分位置を特定
⑤oldMasterにアクセス可能であればバイナリーログをローカルに(/var/log/masterha/app1)コピーする
⑥⑤で引き上げた最新のバイナリーログをnewMaster(/var/log/masterha/app1)にコピー
⑦oldMasterとの差分をnewMasterで更新
⑧newMasterにVIPを付与する
⑨newMasterのread-onlyを解除
⑩⑤で引き上げた最新のバイナリーログをnewSlave(/var/log/masterha/app1)にコピー
⑪oldMasterサーバとの差分をnewSlaveで更新
⑫newSlaveで最新のバイナリーログとrelayログとの差分を確認して適用
⑬newSlaveのMasterをoldMasterサーバからnewMasterサーバに変更しreplication再開
⑭managerにてapp1.failover.completeを/var/log/masterha/app1に出力してmasterha_managerを停止する

・エラーメッセージと意味

これはmanagerが2重に起動したときに出るログ。
Wed May 29 16:02:19 2013 - [error][/usr/lib/perl5/vendor_perl/MHA/ServerManager.pm, ln917]
Getting advisory lock failed on 10.0.0.86(10.0.0.86:3306). Maybe failover script or purge_relay_logs script is running on the same slave?
Wed May 29 16:02:19 2013 - [error][/usr/lib/perl5/vendor_perl/MHA/ManagerUtil.pm, ln178] Got ERROR:
at /usr/lib/perl5/vendor_perl/MHA/MasterFailover.pm line 305

これは完了ファイルがあるときのエラー
Fri May 24 11:46:01 2013 - [error][/usr/lib/perl5/vendor_perl/MHA/ManagerUtil.pm, ln178] Got ERROR:
at /usr/bin/masterha_manager line 65

以上。ご覧いただきありがとうございました！
MHA operation confirmation and switchover verification This is the continuation of the previous article. Test to switch as shown in the following diagram.

　*This verification consists of 1 master, 2 slaves, and 1 manager. (Multi-stage configuration is avoided because recovery in the event of an intermediate node failure will be obscene)
　　If you let the slave live with the manager, it seems that it may not switch well if you put purge_relay_logs on it.
　*It seems that there are cases where there are only two units and it does not switch well. See below.
　　http://heartbeats.jp/hbblog/2013/05/mysql-mha-haproxy.html

(6) Pre-start check

・Check the operation of SSH
[shell] # masterha_check_ssh --conf=/etc/app1.cnf [/shell] If OK, the final output is as follows:
[shell] Tue Oct 23 15:02:22 2012 - [info] All SSH connection tests passed successfully. [/shell]
・Checking the operation of replication
[shell] # masterha_check_repl --conf=/etc/app1.cnf [/shell] If OK, the final output is as follows (*From mysql 5.6, it fails here, probably because there are many changes in binary log format and other changes)
[shell] MySQL Replication Health is OK. [/shell] *Pattern that fails
Replication filtering rules are not correct
If the LVS and MHA manager are carpooling, the check timing will overlap and will be rejected by the DB server for each host (flush hosts;)

(7) Check the updated VIP IF file
db01/db02
[shell] cat /etc/sysconfig/network-scripts/ifcfg-eth1:0 ===================================================== # Intel Corporation 82576 Gigabit Network Connection DEVICE=eth0 :0 BOOTPROTO=static BROADCAST=192.168.100.255 #HWADDR= IPADDR=192.168.100.5 NETMASK=255.255.255.0 NETWORK=192.168.100.0 ONPARENT=no ===================================================== [/shell] *In the case of virtual IF, ONBOOT does not respond, so ONPARENT=no is specified.

db01
[shell] # ifup eth1:0 [/shell]
(8) Starting, confirming, and stopping MHA Manager

• Start Manager
  [shell] # masterha_manager --conf=/etc/app1.cnf & [/shell] *Pattern that fails
  Replication configuration is broken
  It has already been switched and a status file showing completion is printed
  Be careful not to switch well if you look at the log with tail in the terminal where you started the manager.

• Manager status check
  [shell] # masterha_check_status --conf=/etc/app1.cnf [/shell] OK
  [shell] app1 (pid:9883) is running(0:PING_OK), master:192.168.100.1 [/shell]
  [shell] app1 is stopped(2:NOT_RUNNING). [/shell] In the case of the initialization status immediately after startup, it is output as such.
  *If this state continues for a long time, a kill of -9 may require a forced stop.

・Suspension of managers
[shell] # masterha_stop --conf=/etc/app1.cnf

# masterha_check_status --conf=/etc/app1.cnf [/shell] ・If the manager does not stop when it is in the initialized state,
[shell] # ps -ef|grep master [/shell] Copy the process id
[shell] # kill -9 [/shell] Paste the copied PID to force kill
[shell] # masterha_check_status --conf=/etc/app1.cnf # ls -l /var/log/masterha/app1/ [/shell] Delete the status file if it remains.

■MHA Switching Test and Confirmation and Recovery■

*Data update and switching test are omitted. It is mainly to check whether the master is dropped and switched. I think it is also a good idea to additionally check that it does not switch after dropping slave.

(1) Switching test
・Pre-check
[shell] # tail -f /var/log/masterha/app1/manager.log # masterha_check_status --conf=/etc/app1.cnf # ls -lh /var/log/masterha/app1/ # masterha_check_repl --conf=/etc/app1.cnf \ [/shell] ・Tests to drop mysqld
At the old master
[shell] # service mysqld stop # service mysqld status [/shell] Make sure mysqld is stopped

・Test to drop the interface
At the old master
[shell] #ifdown eth1 #ifdown eth1:1 [/shell] ・ Test to close the port in mysqld
At the old master
[shell] # iptables -P INPUT ACCEPT # iptables -A INPUT -p tcp --dport 3306 -j DROP # netstat -lnpt [/shell] Basically accept all inbound packets
Drop packets destined for TCP port 3306
Verify port 3306 closed

(2) Confirmation

・Check the status of the MHA manager
[shell] # tail -f /var/log/masterha/app1/manager.log # masterha_check_status --conf=/etc/app1.cnf # ls -lh /var/log/masterha/app1/ [/shell] Confirm the existence of a switchover completion file called app1.failover.complete

・Check if the VIP has been switched
At the old and new masters
[shell] # ifconfig -a [/shell] Confirm that the update VIP has been removed from the old master and the update VIP has been transferred to the new master
・Confirm that the slave of the new master is stopped and the slave is looking at the new master.
[sql] > show slave status\G [/sql] Make sure that the slave_*_Running of the new master is set to No
Slave Master_Host: Ensures that the new master is facing
[sql] > show global variables like 'read_only'; [/sql] The new master is turned off and can be updated.

• Verify that the weight of the LVS server is as expected (LVS01)
  [shell] # ipvsadm -L --sort # ls -la /etc/ha.d/ldirectord.cf* [/shell] YYYYMMDD. Ensure that HHMM files are backed up
  [shell] # diff /etc/ha.d/ldirectord.cf{,.failover} [/shell] Make sure there are no diffs (make sure the failover file is successfully overwritten)

(3) Configuration restoration

• Start the old master mysqld
  [shell] # netstat -lnpt # service mysqld start # service mysqld status [/shell] ・Start the interface of the old master (DB01)
  [shell] # ifup eth1 # ifconfig [/shell] ・Initialize the port block of the old master
  [shell] # service iptables restart # iptables -ln [/shell] ・Restore replication configuration (*If there is a data difference, you need to dump and input the difference)
  At the old master
  [sql] # mysql -u root -p`cat /path_to_file` > reset master; > show master status; > show slave status\G [/sql] in the new master and slave
  [sql] # mysql -u root -p`cat /path_to_file` > show global variables like 'read_only'; > set global read_only=1; > stop slave; > reset slave; > CHANGE MASTER TO MASTER_HOST='192.168.100.1', MASTER_USER='repl', MASTER_PASSWORD='*******', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=106; > start slave; > show slave status\G [/sql] ・Revert to the weight of the LVS server
  [shell] # ls -la /etc/ha.d/ldirectord.cf* [/shell] YYYYMMDD. Check the HHMM file
  [shell] # mv /etc/ha.d/ldirectord.cf.YYYYMMDD.HHMM /etc/ha.d/ldirectord.cf [/shell] ・Delete unnecessary files
  [shell] # ls -lh /var/log/masterha/app1/ # rm -f /var/log/masterha/app1/app1.failover.complete # rm -f /var/log/masterha/app1/app1.failover.error # rm -f /var/log/masterha/app1/saved_master_binlog_from_192* [/shell] Delete the switched file
  Delete a switchover termination file
  Delete binary log backups of old masters
  *It is turned off because it is an exam environment.

・Manual replacement of renewal VIP
New Master
[shell] # ifdown eth1:0 # ifconfig eth1:0 [/shell] Make sure the IP is not attached

Old Master
[shell] #ifup eth1:0 #ifconfig eth1:0 [/shell] Make sure the IP is on
If you are on an AWS VPC, you will not be able to access it from the outside without changing your private address in AWS.
ec2-unassign-private-ip-addresses --network-interface eni-7060xxxx --secondary-private-ip-address (PrivateIP)
ec2-assign-private-ip-addresses --network-interface eni-4b64xxxx --secondary-private-ip-address (PrivateIP)

・Restart MHA Manager
Confirm
[shell] # masterha_check_repl --conf=/etc/app1.cnf # masterha_check_status --conf=/etc/app1.cnf [/shell] Start
[shell] # masterha_manager --conf=/etc/app1.cnf & # masterha_check_status --conf=/etc/app1.cnf # tail -f /var/log/masterha/app1/manager.log # ls -lh /var/log/masterha/app1/ [/ shell] ・If the MHA manager has remained in the initialized state for a long time
[shell] # masterha_stop --conf=/etc/app1.cnf [/shell] If this command fails, force kill.
[shell] # masterha_check_status --conf=/etc/app1.cnf # ps -ef|grep master [/shell] Copy the process ID and specify it in the kill command
[shell] # kill -9 # masterha_check_status --conf=/etc/app1.cnf # ls -l /var/log/masterha/app1/ [/shell] Delete the status file if it remains.

・If the MHA manager is down and you want to manually switch the master (and VIP for renewal)
[shell] # masterha_master_switch --master_state=alive --conf=/etc/app1.cnf [/shell] *I agree because I will be asked if I can really do it. When the MHA manager is activated, he scolds me to drop it.
(*VIP does not switch unless you add processing)
Please see here for details

If you want to switch VIP as well, modify it as follows (please match the IF information to the environment)
[shell] ========================== 33,40d32 < my $vip = '192.168.0.245/24'; # Write Virtual IP < my $key = "0"; < my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip"; < my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down"; < my $ssh_user = "root"; < my $orig_master_host = "192.168.0.248"; < my $new_master_host = "192.168.0.249"; < 47c39 < 'master_state=s' => \$master_state, --- > 'master_state=s' => \$master_state 50,58d41 < # A simple system call that enable the VIP on the new master < sub start_vip() { < `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; < } < # A simple system call that disable the VIP on the old_master < sub stop_vip() { < `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; < } < 72,76d54 < if ( $exit_code eq 0 ) { < &stop_vip(); < sleep 1; < &start_vip(); < } 80,84d57 < if ( $exit_code eq 0 ) { < &stop_vip(); < sleep 1; < &start_vip(); < } ========================== [/shell] ・secondaryが生きているか確認

[shell] # masterha_secondary_check -s 192.168.0.249 --user=root --master_host=test-db02 --master_ip=192.168.0.245 --master_port=3306 [/shell] ※-sの後ろを追加すれば複数確認できる

■参考資料■
・MHAの動作フェーズ
[shell] # grep Phase manager.log |head -20|grep -v completed * Phase 1: Configuration Check Phase.. * Phase 2: Dead Master Shutdown Phase.. * Phase 3: Master Recovery Phase.. * Phase 3.1: Getting Latest Slaves Phase.. * Phase 3.2: Saving Dead Master's Binlog Phase.. * Phase 3.3: Determining New Master Phase.. * Phase 3.3: New Master Diff Log Generation Phase.. * Phase 3.4: Master Log Apply Phase.. * Phase 4: Slaves Recovery Phase.. * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. * Phase 4.2: Starting Parallel Slave Log Apply Phase.. * Phase 5: New master cleanup phease.. [/shell]
フェイルオーバ時の動作は以下のとおり。（ログから追った動き）

※SQL処理のスレッド実行が終わった後
①config(/etc/app1.cnf)から各ノード情報を読み込む
②newMasterのVIPを停止する
③newMasteのmysqldを停止
④各Slaveリレーログを解析して次マスターの選出と差分位置を特定
⑤oldMasterにアクセス可能であればバイナリーログをローカルに(/var/log/masterha/app1)コピーする
⑥⑤で引き上げた最新のバイナリーログをnewMaster(/var/log/masterha/app1)にコピー
⑦oldMasterとの差分をnewMasterで更新
⑧newMasterにVIPを付与する
⑨newMasterのread-onlyを解除
⑩⑤で引き上げた最新のバイナリーログをnewSlave(/var/log/masterha/app1)にコピー
⑪oldMasterサーバとの差分をnewSlaveで更新
⑫newSlaveで最新のバイナリーログとrelayログとの差分を確認して適用
⑬newSlaveのMasterをoldMasterサーバからnewMasterサーバに変更しreplication再開
⑭managerにてapp1.failover.completeを/var/log/masterha/app1に出力してmasterha_managerを停止する

・エラーメッセージと意味

これはmanagerが2重に起動したときに出るログ。
Wed May 29 16:02:19 2013 - [error][/usr/lib/perl5/vendor_perl/MHA/ServerManager.pm, ln917]
Getting advisory lock failed on 10.0.0.86(10.0.0.86:3306). Maybe failover script or purge_relay_logs script is running on the same slave?
Wed May 29 16:02:19 2013 - [error][/usr/lib/perl5/vendor_perl/MHA/ManagerUtil.pm, ln178] Got ERROR:
at /usr/lib/perl5/vendor_perl/MHA/MasterFailover.pm line 305

これは完了ファイルがあるときのエラー
Fri May 24 11:46:01 2013 - [error][/usr/lib/perl5/vendor_perl/MHA/ManagerUtil.pm, ln178] Got ERROR:
at /usr/bin/masterha_manager line 65

以上。ご覧いただきありがとうございました！