Are you impatient? Do you want to get your code deployed immediately and skipped Code Review and Quality Assurance testing? If the answer is yes and you use ElasticBeanStalk on AWS then this little tip will save you loads of time. Let us forcefully abort a deployment to ElasticBeanStalk.
We are currently in the development phase of a new prototype portal where I am designing and building a RESTful API that is housed on EBS, and since I am the sole DevOps, Developer, Cloud Engineer / everything else, I don’t have a lot of time for waiting for eb abort to work its wonders.
If you deploy using eb deploy and notice after the fact that your service has moved from an Ok to Severe state but don’t want to wait the extra couple minutes to be able to deploy a fixed version do the following.
Have access to your ElasticBeanStalk instance
If you don’t have access then you’ll just have to wait for the abort command to finish through; otherwise ssh into the instance.
ssh beanstalk.instance -l ec2-user -i .ssh/SomeKey.pem
Find the deployment process
ps auwxf | grep appdeploy -B3 root 28610 0.0 0.0 158352 3100 pts/0 S 14:37 0:00 \_ su - root 28611 0.0 0.0 115324 3464 pts/0 S 14:37 0:00 \_ -bash root 7807 0.0 0.0 117292 2488 pts/0 R+ 19:03 0:00 \_ ps auwxf root 7808 0.0 0.0 110460 2160 pts/0 S+ 19:03 0:00 \_ grep --color=auto appdeploy -B3 -- root 2566 0.2 0.6 599320 26372 ? Ssl 2015 122:15 python /opt/aws/bin/cfn-hup root 7640 0.0 0.0 115184 3076 ? S 19:02 0:00 \_ /bin/bash -e /opt/elasticbeanstalk/bin/command-processor root 7641 2.4 0.6 230300 26600 ? Sl 19:02 0:00 \_ /opt/elasticbeanstalk/lib/ruby/bin/ruby /opt/elasticbeanstalk/lib/ruby/bin/command-processor root 7790 1.5 0.3 203748 13824 ? S 19:03 0:00 \_ python /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.py
Kill it with fire!
kill -9 7790 # This is the 01flip.py script
Fair warning: There were no negative consequences outside of logging that the process had failed (See Next); but I don’t know what this process is doing. I assume since its called 01flip.py that it is taking /opt/python/current and flipping it w/ the object in S3. But alas I don’t have time to look at the source.
Events on AWS
INFO: Environment update is starting. INFO: Deploying new version to instance(s). INFO: Environment health has transitioned from Ok to Info. Command is executing on all instances. WARN: User initiated abort was received, however the current step of the operation in progress is not cancellable. The current operation will be aborted as soon as the non-cancellable step(s) complete. WARN: Environment health has transitioned from Info to Severe. 100.0 % of the requests are failing with HTTP 5xx. Insufficient request rate (24.0 requests/min) to determine application health. 25.0 % of the requests to the ELB are failing with HTTP 5xx. Insufficient request rate (2.0 requests/min) to determine application health (6 minutes ago). Command is executing on all instances. ELB health is failing or not available for all instances. ERROR: [Instance: i-0f4b4ab9] Command failed on instance. Return code: null Output: httpd: stopped httpd: started httpd RUNNING pid 7444, uptime 0:00:03. Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.py failed. For more detail, check /var/log/eb-activity.log using console or EB CLI. INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1]. WARN: User initiated abort was received. Canceling the current operation. ERROR: Failed to deploy application. ERROR: During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version. -- Events -- (safe to Ctrl+C)
Deploy your code back to EB
$ eb deploy Creating application version archive "app-48b2-160201_131124". Uploading MyAPI/app-48b2-160201_131124.zip to S3. This may take a while. Upload Complete. INFO: Environment update is starting. INFO: Deploying new version to instance(s). INFO: Environment health has transitioned from Ok to Info. Command is executing on all instances. INFO: New application version was deployed to running EC2 instances. INFO: Environment update completed successfully.
Yeah; you just broke your immutable secure infrastructure, plus some compliance rules. But… got your application deployed. W00t.