{"id":146,"date":"2013-12-02T16:30:31","date_gmt":"2013-12-02T16:30:31","guid":{"rendered":"http:\/\/blogs.nd.edu\/devops\/?p=146"},"modified":"2013-12-09T19:27:37","modified_gmt":"2013-12-09T19:27:37","slug":"cloudformation-and-the-challenge-of-iaas-governance","status":"publish","type":"post","link":"https:\/\/sites.nd.edu\/devops\/2013\/12\/02\/cloudformation-and-the-challenge-of-iaas-governance\/","title":{"rendered":"Cloudformation and  the Challenge of Governance"},"content":{"rendered":"<p><em>aka a maybe possibly better way to script a stack<\/em><\/p>\n<p>As I demonstrated in a <a href=\"https:\/\/blogs.nd.edu\/devops\/2013\/11\/22\/scripting-aws-with-boto\/\">recent post<\/a>, you can script just about anything in AWS from the console using the Boto package for Python. \u00a0I knew that in order to stand up EC2 instances, I was going to need a few things: a VPC, some subnets, and security groups. \u00a0Turns out there are a few other things I needed, like an internet gateway and a routing table, but that comes later. \u00a0As I wrote those Boto scripts, I found myself going out of my way to do a few things:<\/p>\n<ul>\n<li>provide helper functions to resolve &#8220;Name&#8221; tags on objects to their corresponding ID (or to just fetch an object by its Name tag, depending on which I needed)<\/li>\n<li>check for the existence of an object before creating it<\/li>\n<li>separate details about the infrastructure being created into config files<\/li>\n<\/ul>\n<p>The first one was critical, because it doesn&#8217;t take long for your AWS console to become a nice bowl of GUID soup. \u00a0The first time you see a table like this, you know you&#8217;ve got a problem:<\/p>\n<div id=\"attachment_148\" style=\"width: 310px\" class=\"wp-caption alignnone\"><a href=\"http:\/\/blogs.nd.edu\/devops\/files\/2013\/12\/Screen-Shot-2013-12-01-at-10.32.45-PM.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-148\" class=\"size-medium wp-image-148\" alt=\"what is this i don't even\" src=\"http:\/\/blogs.nd.edu\/devops\/files\/2013\/12\/Screen-Shot-2013-12-01-at-10.32.45-PM-300x66.png\" width=\"300\" height=\"66\" srcset=\"https:\/\/sites.nd.edu\/devops\/files\/2013\/12\/Screen-Shot-2013-12-01-at-10.32.45-PM-300x66.png 300w, https:\/\/sites.nd.edu\/devops\/files\/2013\/12\/Screen-Shot-2013-12-01-at-10.32.45-PM-1024x227.png 1024w, https:\/\/sites.nd.edu\/devops\/files\/2013\/12\/Screen-Shot-2013-12-01-at-10.32.45-PM-500x110.png 500w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><p id=\"caption-attachment-148\" class=\"wp-caption-text\">what is this i don&#8217;t even<\/p><\/div>\n<p>I&#8217;ve obscured IDs to protect&#8230; something. \u00a0Resources that probably won&#8217;t exist tomorrow. \u00a0But believe me, changing those IDs took a long time, which is half my point: \u00a0let&#8217;s get some name tags up in here.<\/p>\n<p>The challenge is that you have to be vigilant about creating tags on objects with the key &#8220;Name,&#8221; and then you have to go out of your way to code support for that, because tagging is entirely optional. \u00a0You want friendly names? \u00a0Implement it yourself. \u00a0This is your little cross to bear.<\/p>\n<p>The second task was aimed at making the scripts idempotent. \u00a0It&#8217;s very useful when building anything like this to be able to run it over and over without breaking something.<\/p>\n<p>The third task was an attempt to decouple the infrastructure from boto (and optimistically, any particular IaaS) and plan for a &#8220;configuration as code&#8221; future. \u00a0How nice would it be to commit a change to the JSON for a routing table and have that trigger an update in AWS?<\/p>\n<h3>Enter Cloudformation<\/h3>\n<p>So all this was working rather well, but before I delved too deep, I knew I needed to check out AWS Cloudformation. \u00a0Bob Winding had described how it can stand up a full stack of resources; in fact, it does many of the things I was already trying to do:<\/p>\n<ul>\n<li>describe your infrastructure as JSON<\/li>\n<li>reference resources within the file by a friendly name<\/li>\n<li>stand up a whole stack with one command<\/li>\n<\/ul>\n<p>In addition, it adds a lot of metadata tags to each object that allow it to easily tear down the whole stack after it&#8217;s created. \u00a0As an added bonus, it provides a link to the AWS price calculator before you execute, giving you an indication of how much this stack is going to cost you on a month-to-month basis.<\/p>\n<p>Nice! This is exactly what we want, right? \u00a0<a href=\"http:\/\/docs.aws.amazon.com\/AWSCloudFormation\/latest\/UserGuide\/aws-template-resource-type-ref.html\">The resource schema already exists and is well documented.<\/a>\u00a0 Most of those things correspond to real-life hardware or configuration concepts! \u00a0I could give this to any sysadmin or network engineer, regardless of AWS experience, and they should be able to read it right away.<\/p>\n<h3>The Catch<\/h3>\n<p>I like where this is going, and indeed, it&#8217;s a lovely solution. \u00a0I have a few concerns &#8212; most of which I believe can be handled with good policy and processes. \u00a0Still, they present some challenges, so let&#8217;s look at them individually:<\/p>\n<p><strong>1. One stack, one file.<\/strong><\/p>\n<p>You define a stack in a single JSON file. \u00a0The console \/ command you use to execute it only runs one at a time, which you must name. \u00a0The stack I was trying to run starts with a fundamental resource, the VPC. \u00a0I can&#8217;t just drop and recreate that willy-nilly. \u00a0It&#8217;s clear that there must exist a line between &#8220;permanent&#8221; infrastructure like VPCs and subnets and the more ephemeral resources at the EC2 layer. \u00a0I need to split these files, and not touch the VPC once it&#8217;s up. \u00a0However&#8230;<\/p>\n<p><strong>2.\u00a0Reference-by-ID only works within a single file<\/strong><\/p>\n<p>As soon as you start splitting things up, you lose the ability to reference resources by their friendly names. \u00a0You&#8217;re back to referencing relationships by ID. \u00a0<strong>This is not just annoying: \u00a0it&#8217;s incredibly brittle.<\/strong>\u00a0 AWS resource IDs are generated fresh when the resource is created, so the only way those references stay meaningful is if the resource you depend on is created once and only once. \u00a0That&#8217;s not always what we want, and it&#8217;s extra problematic because&#8230;<\/p>\n<p><strong>3. Cloudformation is not idempotent (but maybe that&#8217;s good)<\/strong><\/p>\n<p>Run that stack twice, and you&#8217;ll get two of everything (usually). \u00a0Now, this is actually a feature of CloudFormation. \u00a0Define a stack, and then you can create multiple instances of it.\u00a0 If you want to update an existing stack, you can declare that explicitly. \u00a0However, some resources have a default &#8220;update&#8221; behavior of &#8220;drop and recreate.&#8221; \u00a0So if it&#8217;s a resource with dependencies, things get tricky. \u00a0The bottom line here is that we have to be smart about what sorts of resources get bundled into a &#8220;stack,&#8221; so we can use this behavior as intended &#8212; to replicate stacks. \u00a0And finally&#8230;<\/p>\n<p><strong>4. It&#8217;s not particularly fast<\/strong><\/p>\n<p>It just seems a bit slow. \u00a0We&#8217;re talking like 2 minutes to go from VPC to reachable EC2 instance, but still. \u00a0My boto scripts are a good deal faster.<\/p>\n<p>There is a lot to like about CloudFormation. \u00a0You can accomplish quite a bit without much trouble, and the resulting configuration file is easy to read and understand. \u00a0Nothing here is a showstopper, as long as we understand the implications of our tool and process choices. \u00a0We can always return to boto or AWS CLI if we need more control over the build process.<\/p>\n<h3>The Challenge of Governance<\/h3>\n<p>I don&#8217;t believe any of the difficulties outlined above are unique to CloudFormation. \u00a0Keeping track of created resources and their various dependencies, deciding on the relative permanence of stack layers, and implementing a solution where parts of the infrastructure can truly be run as &#8220;configuration as code&#8221; are all issues that we must tackle as we get serious about DevOps practices. \u00a0These are just the sorts of questions I have in mind for the first meeting of the DevOps \/ IaaS Governance Team this week.<\/p>\n<p>We should feel encouraged that we&#8217;re not pioneers here! \u00a0Let&#8217;s reach out to friends and colleagues in higher ed and in industry to see how these issues have played out before, and what solutions we may be able to adopt. \u00a0When we know more, we&#8217;ll be that much more confident to proceed, and I can write the next blog post on what we have learned.<\/p>\n<p><a href=\"https:\/\/oit-svn.cc.nd.edu\/es\/master\/trunk\/devops\/cloudformation\/\">OIT staff can view my CloudFormation templates here<\/a>.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>aka a maybe possibly better way to script a stack As I demonstrated in a recent post, you can script just about anything in AWS from the console using the Boto package for Python. \u00a0I knew that in order to &hellip; <a href=\"https:\/\/sites.nd.edu\/devops\/2013\/12\/02\/cloudformation-and-the-challenge-of-iaas-governance\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1550,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[65098,65099,65126,65100],"class_list":["post-146","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-aws-2","tag-cloudformation","tag-iaas","tag-infrastructure"],"_links":{"self":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/146","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/users\/1550"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/comments?post=146"}],"version-history":[{"count":24,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/146\/revisions"}],"predecessor-version":[{"id":215,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/posts\/146\/revisions\/215"}],"wp:attachment":[{"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/media?parent=146"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/categories?post=146"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sites.nd.edu\/devops\/wp-json\/wp\/v2\/tags?post=146"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}