Forcefully Aborting a ElasticBeanStalk Deployment

Are you impatient? Do you want to get your code deployed immediately and skipped Code Review and Quality Assurance testing? If the answer is yes and you use ElasticBeanStalk on AWS then this little tip will save you loads of time. Let us forcefully abort a deployment to ElasticBeanStalk.

We are currently in the development phase of a new prototype portal where I am designing and building a RESTful API that is housed on EBS, and since I am the sole DevOps, Developer, Cloud Engineer / everything else, I don’t have a lot of time for waiting for eb abort to work its wonders.

If you deploy using eb deploy and notice after the fact that your service has moved from an Ok to Severe state but don’t want to wait the extra couple minutes to be able to deploy a fixed version do the following.

Have access to your ElasticBeanStalk instance

If you don’t have access then you’ll just have to wait for the abort command to finish through; otherwise ssh into the instance.

ssh beanstalk.instance -l ec2-user -i .ssh/SomeKey.pem

Find the deployment process

ps auwxf | grep appdeploy -B3
root     28610  0.0  0.0 158352  3100 pts/0    S    14:37   0:00                  \_ su -
root     28611  0.0  0.0 115324  3464 pts/0    S    14:37   0:00                      \_ -bash
root      7807  0.0  0.0 117292  2488 pts/0    R+   19:03   0:00                          \_ ps auwxf
root      7808  0.0  0.0 110460  2160 pts/0    S+   19:03   0:00                          \_ grep --color=auto appdeploy -B3
--
root      2566  0.2  0.6 599320 26372 ?        Ssl   2015 122:15 python /opt/aws/bin/cfn-hup
root      7640  0.0  0.0 115184  3076 ?        S    19:02   0:00  \_ /bin/bash -e /opt/elasticbeanstalk/bin/command-processor
root      7641  2.4  0.6 230300 26600 ?        Sl   19:02   0:00      \_ /opt/elasticbeanstalk/lib/ruby/bin/ruby /opt/elasticbeanstalk/lib/ruby/bin/command-processor
root      7790  1.5  0.3 203748 13824 ?        S    19:03   0:00          \_ python /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.py

Kill it with fire!

kill -9 7790 # This is the 01flip.py script

Fair warning: There were no negative consequences outside of logging that the process had failed (See Next); but I don’t know what this process is doing. I assume since its called 01flip.py that it is taking /opt/python/current and flipping it w/ the object in S3. But alas I don’t have time to look at the source.

Events on AWS

INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
INFO: Environment health has transitioned from Ok to Info. Command is executing on all instances.
WARN: User initiated abort was received, however the current step of the operation in progress is not cancellable. The current operation will be aborted as soon as the non-cancellable step(s) complete.
WARN: Environment health has transitioned from Info to Severe. 100.0 % of the requests are failing with HTTP 5xx. Insufficient request rate (24.0 requests/min) to determine application health. 25.0 % of the requests to the ELB are failing with HTTP 5xx. Insufficient request rate (2.0 requests/min) to determine application health (6 minutes ago). Command is executing on all instances. ELB health is failing or not available for all instances.
ERROR: [Instance: i-0f4b4ab9] Command failed on instance. Return code: null Output: httpd: stopped
httpd: started
httpd                            RUNNING   pid 7444, uptime 0:00:03.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.py failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
WARN: User initiated abort was received. Canceling the current operation.
ERROR: Failed to deploy application.
ERROR: During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
 -- Events -- (safe to Ctrl+C)

Deploy your code back to EB

$ eb deploy
Creating application version archive "app-48b2-160201_131124".
Uploading MyAPI/app-48b2-160201_131124.zip to S3. This may take a while.
Upload Complete.
INFO: Environment update is starting.
INFO: Deploying new version to instance(s).
INFO: Environment health has transitioned from Ok to Info. Command is executing on all instances.
INFO: New application version was deployed to running EC2 instances.
INFO: Environment update completed successfully.

Yeah; you just broke your immutable secure infrastructure, plus some compliance rules. But… got your application deployed. W00t.

Write a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.