Translated from 日本語 • View original Hello. This is Komiya from OPS.
It seemed that there was a demand within the company,
I searched for Python tools faster than Java, but I couldn't find AWSCLI, so I made it.
We do not use tags to automatically judge things that are not taken. Fixed to allow multiple generations to be obtained if you specify the number of generations in bkup_num.
It is a specification that specifies the instance ID in argument 1 and the hostname in argument 2.
--no-reboot, so the running instance will not be restarted.
Even if there was EBS in the data area, the ami command seemed to take multiple commands on its own, so
I tried to determine whether it was an OS device or a data device and tag it in an easy-to-understand manner.
※※Note※※
--no-reboot When the target of retrieving the AMI was the mysql DB server of the MyISAM storage engine, the data was corrupted and stopped.
If you really want to acquire an AMI if you are not an InnoDB, it would be better to stop it with a maintenance recommendation.
*AWS does not guarantee the integrity of data related to non-disruptive AMI acquisition.
Reference URL: http://docs.aws.amazon.com/ja\_jp/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html
This means that you should be careful with storage servers (maintenance recommended).
・Install awscli if it is not included
How to put it in: (If it is not included, use pip install awscli.) pip is put in easy_install)
AWS Command Line Tool for Python | Developers.IOEasy_install in python - Hane @ Diary I tried using AWS Command Line Interface (awscli) - Suitable days for a former RX-7 rider
Create an AWS access key and region configuration file that you will call from the script.
[shell]# cat /root/.ec2/aws.config ------------ [default] aws_access_key_id=< access key id> aws_secret_access_key=
・Command investigation (I'll post it first)
[shell]# aws ec2 copy-image help ------------ copy-image DESCRIPTION Initiates the copy of an AMI from the specified source region to the region in which the request was made. It's an inter-region copy, so I don't need it now. ------------ create-image Creates an Amazon EBS-backed AMI from an Amazon EBS-backed instance that is either running or stopped. create-image [--dry-run | --no-dry-run] --instance-id
aws ec2 create-image --instance-id i-10a64379 --name "My server" --description "An AMI for my server"
Output:
{ "ImageId": "ami-5731123e" } ------------ describe-images If you want to manage generations, it would be a good idea to use the imageid or hostname that came up after searching for tags in the --filter option. ------------ describe-instances This is what you use to get the RootDeviceName. ------------ deregister-image Deregisters the specified AMI. After you deregister an AMI, it can't be used to launch new instances. SYNOPSIS deregister-image [--dry-run | --no-dry-run] --image-id
aws ec2 deregister-image --image-id ami-4fa54026
Output:
{ "return": "true" } ------------ NAME create-tags - SYNOPSIS create-tags [--dry-run | --no-dry-run] --resources
aws ec2 create-tags --resources ami-78a54011 --tags Key=Stack,Value=production
Output:
{ "return": "true" } ------------ # aws ec2 delete-snapshot help NAME delete-snapshot - DESCRIPTION Deletes the specified snapshot. SYNOPSIS delete-snapshot [--dry-run | --no-dry-run] --snapshot-id
# /usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id i-292a442c ------------[/shell]
・Create a script
If you have an argument, you can create that instance, if not, create your own AMI.
Put the return imageid into a file and tag it with it,
At this time, the tag is determined to be OS by whether or not the snapshot of the RootDeviceName is determined, and the device name is also included in the tag name.
Search with the hostname filter with the return imageid to get all the imageids related to the host and sort them in descending order by the date and time field.
The process is to delete the rows of the generation you want to keep, get the old imageid of the target to be erased, and delete the deregist and snapshot.
The number of generations is defined in a variable called bkup_num.
[shell]# vi /opt/bin/aws_image_bkup.sh ------------ #!/bin/bash # #aws_image_bkup.sh: Back up AMI with awscli # Dependencies: aws command # Update history: 20140314 - create komiyay # export PATH=$PATH:/usr/ local/bin export AWS_CONFIG_FILE=/root/.ec2/aws.config
if [ $# -eq 0 ]; then echo "If you don't specify instance-id, do an AMI backup of the local instance" echo "usage: $0 [instance-id] [hostname]" myInstanceID='/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id\` host_name=`uname -n` else myInstanceID=$1 host_name=$2 fi
## default variables base=`dirname "$0"` LOG=/opt/log/bkup.log CurrentImage_LOG=${base}/cimageid_${myInstanceID}.log #OLDimageID=`cat ${CurrentImage_LOG}|grep ImageId| awk '{FS=":"; print $2}'|sed -e 's/[ "]//g'` current_result=/opt/log/current_result.log today=`/bin/date +%Y%m%d` amitime=`/bin/date +%Y%m%d_%H%M` datetime=`/bin/date +%Y/%m/%d_%H :%M:%S' mail_to=
## check_param. if [ -z ${myInstanceID} ]; then echo "error: no instance-id, end script ${datetime}."| tee -a ${LOG} exit 1 fi
## function ### mail_send mail_send(){ echo ${mail_body}|tee -a ${LOG}|mail -s ${mail_title} ${mail_to} }
### create_ami create_ami(){ aws ec2 create-image --instance-id ${myInstanceID} --name "${host_name}_${amitime}" \ --description "${host_name}_${myInstanceID}_${datetime }" --no-reboot |tee ${CurrentImage_LOG} grep ImageId ${CurrentImage_LOG} res_create=$? }
### deregister_ami deregister_ami(){ if [ -z ${OLDimageID} ]; then res_deregister=0 else aws ec2 deregister-image --image-id ${OLDimageID}|tee ${current_result} grep true ${current_result} res_deregister=$? if [ ${res_deregister} -ne 0 ]; then mail_body="NG deregister ami for ${myInstanceID}(${host_name}) at ${datetime}" mail_titile="NG_deregister_ami_${today}" mail_send fi fi }
### delete_snapshot delete_snapshot(){ if [ -z ${OLDimageID} ]; then res_delsnap=0 else snapshot_ids=`aws ec2 describe-snapshots --filters Name=description,Values=*${OLDimageID}* \ |grep SnapshotId|awk -F : '{print $2}'|sed -e 's/[ ",]//g'` for i in ${snapshot_ids} do aws ec2 delete-snapshot --snapshot-id ${i}|tee ${current_result} grep true ${current_result} res_delsnap=$? if [ ${res_delsnap} -ne 0 ]; then mail_body="NG delete ami snapshot(${i}) for ${myInstanceID}(${host_name}) at ${datetime}" mail_titile="NG_delete_ami-snapshot_${today}" mail_send fi done fi }
### create_tag create_tag(){ ### create tag for ami CurImageID=`cat ${CurrentImage_LOG}|grep ImageId|awk '{FS=":"; print $2}'|sed -e 's/[ "]//g'` aws ec2 create-tags --resources ${CurImageID} --tags "Key=Name,Value=${host_name}_${today}" ### create tags for snapshots rootDeviceName=`aws ec2 describe-instances --instance-ids ${myInstanceID} \ |grep RootDeviceName|awk '{print $2}'|sed -e 's/[",]//g'` snapshot_ids=`aws ec2 describe-snapshots --filters Name= description,Values=*${CurImageID}* \ |grep SnapshotId|awk -F : '{print $2}'|sed -e 's/[ ",]//g'` for i in ${snapshot_ids} do volume_id=`aws ec2 describe-snapshots -- snapshot-ids ${i}|grep VolumeId \ |awk '{print $2}'|sed -e 's/[",]//g'` deviceName=`aws ec2 describe-volumes --volume-ids ${volume_id}|grep Device \ |awk '{print $2}'|sed -e 's/"// g'` if [ "${rootDeviceName}" == "${deviceName}" ]; then aws ec2 create-tags --resources ${i} \ --tags "Key=Name,Value=AMI_${host_name}_${today}-os_${deviceName}" else aws ec2 create-tags --resources ${i} \ --tags "Key=Name, Value=AMI_${host_name}_${today}-data_${deviceName}" fi done }
## main exec >> ${LOG} exec 2>&1
echo "start ami backup. ${datetime}" create_ami
## rotate_ami delete_images=`aws ec2 describe-images --filters Name=name,Values=*${host_name}* \ |egrep -a 'ImageId| ImageLocation'|sed -e "N; s/\n//" -e 's/[",]//g'|awk '{print $2" "$4}' \ |sort -r -t _ -k 2,3|sed -e "1,${bkup_num}d"|awk '{print $1}'` for i in ${delete_images} do OLDimageID=${i} deregister_ami delete_ snapshot done
## if get error, then send mail. the other case, creating tags. if [ ${res_create} -ne 0 ]; then mail_body="NG create ami for ${myInstanceID}(${host_name}) at ${datetime}" mail_titile="NG_create_ami_${today}" mail_send elif [ ${res_create} -eq 0 ]; then create_tag fi echo "end ami backup. ${datetime}"
## logrotate DAY=`date +%m%d` if [ $DAY = "0101" ] then OY=`date -d '1 year ago' +%Y` mv $LOG $LOG.$OY find /opt/log -name "$LOG.*" -type f -atime +730 -exec rm -f {} \; fi
exit 0 ------------ # chmod +x /opt/bin/aws_image_bkup.sh[/shell]
Try to use it
[shell]# bash -x /opt/bin/aws_image_bkup.sh i-292a442c komiya-test-mysql01 # cat /opt/bin/cimageid_i-292a442c.log # cat /opt/log/bkup.log [/shell]
After running the test, it started to work without any problems.
As a precaution,
When AMI_NAME hits, both seem to disappear and disappear, so I don't think I will do that, but it's better not to run it twice in the same minute.
If you span the minutes, you can get an AMI normally,
I feel that if the number of runs increases too much, the usage fee will increase, so I recommend using it daily.
If you want to keep the old AMI, define the number of generations in the variable bkup_num.
If you want to take multiple instances of AMIs, you can create a list and loop it as follows.
[shell]vi bkup-hosts.txt ---- i-xxxxx,hoge-web01 i-yyyyy,hoge-web02 ... ---- for i in `cat bkup-hosts.txt` do instance-id=`echo $i|awk -F, '{print $1}'` host-name=`echo $i|awk -F, '{print $2}'` /opt/bin/aws_image_bkup.sh ${instance-id} ${host-name} done[/shell]
There may be a pattern that if you only have data, you can recover it, you can save on the amount of backup, and you only need to take the AMI when you change it.
If you want to get the latest AMI for autoscaling, etc., I feel that automatic AMI backup is certainly necessary.
Is it that backup depends on the recovery plan?
What I personally learned from making a script this time was that
sed -e "N; s/\n//" Remove odd line breaks
Is it a place? I had used other ones, but I felt like I knew that I could do it by searching for keywords.
It seems that he still hasn't practiced.
・Bonus
I will post the awscli chef recipe.
Make a cookbook
[shell]knife cookbook create aws -o site-cookbooks[/shell]
Make a recipe called awscli.rb
[shell]# vi site-cookbooks/aws/recipes/awscli.rb ----------------------------------------------------- if node["platform_version"] >= 6 package "python-setuptools" do action :install end bash "pip-awscli-install" do not_if "which aws" code <-EOC easy_install="" pip="" pip="" install="" awscli="" eoc="" end="" elsif="" node["platform_version"]="">= 5 && node["platform_version"] < 6 bash "python26-pip-awscli-install" do not_if "which aws" code <-EOC yum -y install python26 python26-devel python26-distribute --enablerepo=epel easy_install-2.6 pip pip2.6 install awscli EOC end end
directory "/root/.ec2" do action :create end
awstest1 = Chef::EncryptedDataBagItem.load("awskeys","awstest1") awstest1key_id = awstest1["aws_access_key_id"] awstest1secret_key = awstest1["aws_secret_access_key"] awstest1region = awstest1["region"]
bash "set-aws-config" do not_if "grep access_key /root/.ec2/aws.config" code <-EOC echo="" '[default]'="">> /root/.ec2/aws.config echo "aws_access_key_id=#{awstest1key_id}" >> /root/.ec2/aws.config echo "aws_secret_access_key=#{awstest1secret_key}" >> /root/.ec2/aws.config echo "region=#{awstest1region}" >> /root/.ec2/aws.config EOC end -----------------------------------------------------[/shell] I think the access key ID should be managed with data_bags.
[shell]awstest1 = Chef::EncryptedDataBagItem.load("awskeys","awstest1") awstest1key_id = awstest1["aws_access_key_id"] awstest1secret_key = awstest1["aws_secret_ access_key"] awstest1region = awstest1["region"][/shell]
Register an AWS access key, etc. in data_bags
[shell]knife solo data bag create awskeys awstest1 --secret-file ~/.chef/encrypted_data_bag_secret { "id": "awstest1", "aws_access_key_id": "AKIAID4CBY76********", "aws_secret_access_key": "4+saCCqOQL8+CKnzdHOpc2CKg2oWmKRO********", "region": "ap-northeast-1" }
# knife solo data bag show awskeys awstest1 --secret-file data_bag_key WARNING: The encrypted_data_bag_secret option defined in knife.rb was overriden by the command line. aws_access_key_id: AKIAID4CBY76******** aws_secret_access_key: 4+saCCqOQL8+CKnzdHOpc2CKg2oWmKR******** id: awstest1 region: ap-northeast-1[/shell]
Syntax Test
[shell]# knife cookbook test aws[/shell]
Thank you for reading the above. </-EOC></-EOC>