Put Capistrano tasks to sleep until the new EC2 instance is fully launched

If you run a task to SSH access a newly created EC2 instance in Capistrano without fully upping it, the task will not complete with an error somewhere. Therefore, we created a task that checks the startup status of the EC2 instance to be deployed, and if it is not fully launched, it sleeps and waits for startup.

There are three status information for AWS EC2 instances, and if you don't check all of these statuses, you can't say that the instance is fully up (see figure below).

To check whether an instance is starting, you can determine whether the "Instance State" in the AWS Management Console is 'running', but to check whether you can actually SSH to the instance, you can check if you can SSH to the instance by adding "2/2 checks" in the "Status Checks" field. As you can see, you need to check whether both statuses (INSTANCESTATUS and SYSTEMSTATUS reachability) are 'passed'. Normally, when an instance is launched, the "Instance State" becomes 'running' in a few seconds to ten seconds, but the "Status Checks" are initialized for a few minutes or so. If both of these statuses are not 'passed', SSH access will be used.

In the 'check' task I used so far, I only checked the status of 'Instance State', so the subsequent task was interrupted by SSH access. The task to avoid it will be the 'check' task this time.

task :check do
  run_locally do
    created_instances_list = 'CREATED_INSTANCES'

def check_instance_status(instance_ids=[]) 
      ec2 = AWS::EC2.new
      AWS.memoize do
        ec2info = ec2.client.describe_instance_status({'instance_ids' => instance_ids})
        sys_status = ec2info.instance_status_set.map { |i| i.system_status.details[0].status }
        ins_status = ec2info.instance_status_set.map { |i| i.instance_status.details[0].status }
        status = sys_status + ins_status
        return status.include? ('initializing') ? false : true
      end
    end

ec2 = AWS::EC2.new
      begin
        if test "[ -f ~/#{created_instances_list} ]"
          created_instances = capture("cd ~; cat #{created_instances_list}").chomp
          ci = created_instances.gsub(/(\[|\s|\])/, '').split(',')
          target_instances = ec2.instances.select { |i| i.exists? && i.status == :running && ci.include?( i.id) }.map(&:id)
          raise "Is not still created all instances" if target_instances.length < fetch(:instance_count)
          if target_instances.length == fetch(:instance_count) then
            # 全インスタンス起動（Instance Stateがrunning）
            chk_retry = 0

            while !check_instance_status(target_instances) do
              # インスタンスステータスが全てOKでない場合は15秒待つ（※20回までリトライする）
              info "In preparation of the instance: Status check " + (chk_retry > 0 ? "(retry #{chk_retry + 1} times)" : "")
              sleep 15
              if check_instance_status(target_instances) then
                break
              end
              chk_retry += 1
              raise "Instance is not still ready. Please run the task again after waiting for a while." if chk_retry >= 20
            end

# This is the process after all instances are fully launched (first SSH connection setting)
            target_instance_private_ips = ec2.instances.select { |i| i.exists? && i.status == :running && ci.include?( i.id) }.map(&:private_ip_address)
            pkfn = fetch(:private_key_file)
            target_instance_private_ips.each { |var| 
              server var, user: 'ec2-user', roles: %w{web app}, ssh_options: { keys: %W(/home/deploy-user/#{pkfn}), forward_agent: true }
            }
          end
        end
      rescue => e
        info e
        exit
      end
  end

# Sample follow-up task (SSH to display hostname)
  on roles(:web) do
    info capture "hostname"
  end
end

From the above settings, the task with the sample part of the follow-up task removed is inserted just before the deployment task after creating the EC2 instance in Capistrano.

Reference site

• Wait for the launch after creating EC2 in the AWS SDK for Ruby