devopsy.kinsta.cloud

Author: Rich Mogull

  • Building Great Cloud Security Guardrails

    Building Great Cloud Security Guardrails

    Security guardrails are an incredible way to keep our cloud deployments safer without slowing things down. Taking this structured approach will minimize friction while increasing protection.

    When I first started working hands-on in cloud (AWS) around nine years ago, I quickly realized both the power and risks of just how darn fast and agile cloud could be. Being, on occasion, an optimist, I decided this capability was a significant upgrade to our security capabilities. Sure, I could accidentally spin up internet-exposed servers, but in exchange I gained levels of central control and visibility unheard of in traditional infrastructure.

    One of the most powerful new capabilities is the ability to leverage cloud APIs and automation to build automated security guardrails without having to spend a gazillion dollars for a big box that actually delivers as promised, but also adds tremendous friction and slows things down. Guardrails are automations that constantly watch your deployments, find deviations from desired baselines and can even automatically remediate issues. Yet, even cloud-native guardrails can become problematic if they aren’t designed properly. Having both built a fair few myself and worked with many organizations as they built out their cloud security programs, here are some of the lessons I’ve learned along the way and some code to show you how it works.

    Building Awesome Guardrails

    My first guardrails were a bit of a hodgepodge in terms of structure and capabilities. And practically speaking, there are a lot of ways to attack the problems. Even today I’ll take different approaches depending on the particular challenge in front of me, but most of my guardrails (including operational ones) tend to follow a consistent pattern:

    • Define the problem: I like to start by defining the business outcome we want, and then move onto the technical problem. I used to start with a narrow technical problem to solve but found that didn’t always lead me to the right outcome. For example, is the problem that we want to stop all internet-facing port 22 access, or is the real outcome we don’t want to expose internet-facing admin servers to the open internet and we want to use a guardrail to restrict to our known corporate IPs instead? This helps prevent a guardrail from becoming a blocker.
    • Set the scope: Multiple cloud accounts (or subscriptions/projects for you Azure/GCP people) are now the standard. The first step is to determine how broad a guardrail you want, which will also define some of the technical specifications. For example, if you want it to apply to multiple accounts in Amazon Web Services, that affects where you run it from and how you manage the required IAM privileges.
    • Build or pick the deployment model: There are a lot of technical specifics regarding where and how to run your guardrail. Today’s example will run in AWS within a single account, but if you manage a larger, enterprise-class deployment you will need to look at how and when to balance centralized and localized guardrails. More often than not, I prefer a serverless approach (FaaS with other supporting PaaS), but I’ve also found longer-running guardrails often need containers as well. Thanks to API gateways, it isn’t bad to build deployments that can support both.
    • Define filters: It’s awesome that you can automatically close all internet-facing security groups with 20 to 40 lines of code—right until you break a production application. Filters allow you to tune your guardrail for a particular environment or project’s needs. Basic filters might work on a simple resource tag, while more complex filters might evaluate a combination of include and exclude rules for greater flexibility. This can, for example, treat development and production environments differently from the same guardrail.
    • Determine triggers: We have a rich range of trigger options on the major cloud platforms today. The main categories are time-based assessments and event-based triggers. I tend to use both, with event-based for more deterministic events (an S3 bucket policy was changed) and time-based sweeps to catch conditions I might miss purely on an event trigger.
    • Perform the analysis: This is the meat of our guardrail. Based on the trigger, collect the information we need and, in combination with our scope and filters, make a decision. For example, if we find port 22 open to the internet, if it is a Dev environment we might decide to update the security group to restrict to known corporate IP addresses. If it is in Prod we might delete the rule completely unless it was implemented through a CloudFormation template that went through an approval process.
    • Send notifications: Even when you have a completely automated guardrail, you likely want a parallel notification. These can go straight to email/text/Slack or drive into a ticketing system. For security guardrails, I also recommend sending them to your SIEM or other security alert/management system. If you’re fancy, you can turn these into the first step of a chatOps process.
    • Take actions: Actions are what separates guardrails from all the annoying alerts of the past. Based on the analysis and filters, actions can be fully automated remediations or request human intervention. Actions also can follow different paths based on the conditions to provide the best options for that situation. The key is that even if a human is called to make a decision, the decision should be a simple click to run the action, saving the person from having to fix things manually. Unfortunately, we still sometimes need human beings to make a decision. Then again, the alternative is bowing down to our robotic overlords. It’s a trade-off, I suppose.
    • Validate and log: Don’t assume the action actually did anything. Rechecking is vital, as is logging and sending the results as a second notification to the ticketing system/SIEM/etc. I fully recommend sending both incident open and close notifications to the destination system, even if the guardrail fixed everything within seconds. This keeps a consistent record and also allows you to do nifty things such as measure your mean resolution time and compare automated versus manual.

    Running a Demonstration Guardrail

    One of my favorite guardrails to demonstrate is automatically reversing security group changes in AWS. Here’s how to set it up for yourself and run my demonstration code to see what it looks like:

    • The first step is to make sure you don’t start running this in production or anyplace else where you might break things before you actually tune the filters. If you do, it isn’t my fault. The desired outcome in this case is ensuring we don’t have network exposures due to unapproved security group changes on designated resources.
    • The scope of this guardrail is a single local account, with a corresponding deployment model. We will use CloudTrail, CloudWatch Rules, Lambda and SNS.
    • There are three prep steps to configure your account before building the guardrail:
      • Log into the AWS console and set up a destination for notifications. If you haven’t done this before, you want to use SNS (Simple Notification Service). Create a topic for security alerts and a subscription to wherever you want to send the alerts. I recommend starting with a simple email subscription.
      • Make sure you have CloudTrail turned on and streaming to CloudWatch. For you newbies, CloudTrail records (almost) all the API calls on your account and we need this for our trigger. Sending the logs to CloudWatch allows us to build near real-time triggers in CloudWatch based on events such as the API call we will look for.
      • Create an IAM Role for Lambda to use. It will need the rights to AuthorizeSecurityGroupIngress, DescribeSecurityGroups and RevokeSecurityGroupIngress. To save logs, it also needs PutLogEvents, CreateLogStream and CreateLogGroup. These are also in a JSON template in the project repository for cut and paste.
    • With that in place, create a new Lambda function (Python 2.7) and load in our code located at https://github.com/disruptops/RevertSecurityGroups.
    • Now go to CloudWatch Rules and create a new rule. For the trigger, use the following:
      • Service Name: CloudTrail
      • Event Type: AWS API Call via CloudTrail
      • Specific Operations: AuthorizeSecurityGroupIngress
    • Then set two targets: your Lambda Function and your SNS topic for security alerts. This will revet the changes and simultaneously send you a notification.
    Screenshot of configuring the trigger in CloudWatch

    This should now revert any new ingress rules added to a security group. If you play with the code you can also use this for egress rules. This is just demo code and there are edge cases it will miss.

    The code also includes a number of filter options, including:

    • Restricting to a region or VPC.
    • Restricting based on a tag or a specific security group ID.
    • Allowing the change if it comes from a designated IAM user, and reversing if it was originated by anyone else.

    It takes the action automatically; if you want to manually review first you would modify the code to send a notification instead. We have some alternative examples posted at https://github.com/disruptops/demos-tools. These includes some samples that protect S3 and also show how to embed notifications in the Lambda itself for more-flexibility.

    Hopefully this gives you some ideas to refine your own guardrails to dramatically improve security while still moving blazingly fast.

    — Rich Mogull

  • Cloud Security: Software Defined. Event Driven. Awesome.

    Cloud Security: Software Defined. Event Driven. Awesome.

    I get it. Cloud and DevOps are disruptive and create new risks. I’m a security pro; one with decades of hard-earned experience telling me that with great chaos comes great opportunity… for the bad as much as the good. And once again it’s time to circle the wagons, make sure policies are enforced, and our organization’s are protected. Or maybe that’s just all the subliminal vendor FUD leaking past my mental firewalls.

    But maybe it’s time to tell that part of my brain to shut the blank up and start looking for all the ways cloud and DevOps make things better for security. Because they do. And not merely on the surface, but in deeply fundamental changes that upend a lot of our previous conceptions. The best part is that, for once, all of this is real, practical, and achievable today.

    Major cloud providers provide solid security foundations (otherwise no one can use them), typically better than most self-managed data centers. Great baseline security is all well and good, but I’m far more interesting in ways cloud enables us to do security better. Not a little better, but fundamentally improve our existing practices, processes, and capabilities. This isn’t a theoretical concept, but a practical one that some of us are using today in the real world, and I’m going to show you a concrete example.

    These capabilities are thanks to Software Defined Security and Event Driven Security.

    Software Defined Security is the ability to programmatically manage our security technologies, including the security capabilities of non-security technologies. For the most part it’s been an abysmal failure since security companies don’t exactly have the best track record when it comes to APIs and automation (well, outside of marketing materials). Even today the burgeoning on-premise automation tools often find themselves having to remotely log into consoles and issue command lines to manage activity. Security automation has usually been a good way to automate your rapid departure from your current employer.

    Cloud computing changes that in two ways. First, the broad accessibility of robust REST-based APIs to manage security features of cloud services and, on occasion, security tools or services themselves. SOAP APIs are usually too difficult to work with for anyone short of dedicated developers, while REST is often consumable for strong scripters and admins. Second, the non-security APIs allow security professionals to integrate with the infrastructure and platforms directly. We now use the same interfaces as operations and development, except we can use them to achieve security objectives.

    For example, at various conferences I demonstrate an automated incident response workflow that quarantines and instance, locks down the AWS management, images the storage, launches a forensics analysis server and connects the images, and performs a deep analysis of it’s state, connected resources, and potential exposure. An old version is up on GitHub and I have a vastly updated version I’ll be releasing soon. All of this happens in a few seconds.

    Another example, in that same project, nearly instantly identifies any instance in your account not managed by Chef. Identifying unmanaged servers in a traditional network is a common audit function that takes weeks or months. In cloud computing it’s two API calls and a three line comparison function. Last year at the RSA Conference I demonstrated automatically inserting a cloud WAF in front of detected web servers in less than a minute.

    The second piece of the puzzle is newer. With Event Driven Security we leverage the core instrumentation of infrastructure and platforms to detect events and initiate security automation. Until now nearly all security tools relied on polling/scanning or custom agents that have a pesky tendency to kill performance or otherwise break things. For example, instead of scanning the network to detect a change, the network tells us when it changes, and that event triggers actions. It’s a subset of Software Defined Security, yet still distinct.

    In this example I posted earlier this week I leverage Amazon’s new CloudWatch events and Lambda to automatically revert any security group changes, or only changes that fail to meet certain criteria, within 10 seconds. You could use the some template to reverse nearly any change in your environment, or kick off more complex workflows like the automated incident response.

    Software Defined Security allows us to build security operations that are as elastic and agile as the cloud itself. Event Driven Security further reduces the changes of manual error, bringing even greater automation and speed, especially when paired with frameworks and concepts like DevOps and immutable infrastructure. These aren’t new concepts, but unlike in the past they are viable and actively used by real organizations in real production environments.

    The combination fundamentally changes how we can build our security programs. Yes, we can still do all the old things we’ve always done, but with a little creative thinking we gain entirely new, and practical, capabilities. It’s a ridiculously exciting time for our community and profession.

  • Security automation with DevOps: show me the code!

    Security automation with DevOps: show me the code!

    Last week Andrew Storms put up a good post hinting at the promise of security automation in [SecDevOps: Security Automation By Example – The Firewall Change]. He included an example of automating a series of actions when a firewall rule is changed. It’s a good article, although I’m increasingly convinced there’s no such thing as SecDevOps. In my book, it’s all DevOps, but that’s fodder for another post (when I’m not battling a stomach bug). However, what Andrew describes is more of what I consider an automated assist. It isn’t necessarily full automation, since it triggers on a manual firewall rule change. Ideally we rarely manually change a firewall (or Security Group) rule, and rely more on self-configuring based on policies. Yeah, I know, the usual analyst BS, so here’s a bit of process, and a bit of code. Let’s approach this differently. Take Andrew’s process, but let’s have the security group or host firewall self configure based on the asset. My example is going to be Amazon Web Services specific, since I have some code snippets to show off how it works. (Sorry, I don’t have the time or intestinal fortitude to write out all the code today; seriously, where did this bug come from?!?) There are two techniques I generally see forming the core of dynamic Security Group policy enforcement:

    1. Tags
    2. Configuration Management integration

    Some cloud services, like AWS, support tags at the object (in this case, instance) level. These tags can tie internally to policies while also being consumable for external tools. Here is a policy I use that restricts access to an object if the “SecurityStatus” tag is “IR” (for “Incident Response”):

    {  
  "Version": "2012-10-17",  
  "Statement": [  
    {
      "Action": "*",
      "Condition": {  
        "StringEquals": {
          "ec2:ResourceTag/SecurityStatus": "IR"
          }
          },
          "Resource": [
          "*"
                      ],
          "Effect": "Deny"
      }
      ]  
    }

    I could apply that tag manually or automatically, and in either case the instance can no longer be managed by any account this policy is applied to. Now this doesn’t help us change firewall/Security Group rules, since you can’t manage those directly in EC2 based on tags (but I had the policy handy to demo, so there you go). For that to work, you need to write your own code to scan for tags and then take actions. I don’t have exactly the right code examples on hand. but here are some snippets to get you close. Most of these are cribbed from my SecuritySquirrel proof of concept on GitHub. They are all in Ruby, using the AWS 2.0 developer preview SDK. First, to pull a list of all instances with a certain tag, you can use the following:

    def testing
 # testing some tag code
 instancelist = @@ec22.describe_tags(
 filters: [
 {
   name: "key",
   values: ["SecurityStatus"]
 }
 ]
 )
 puts instancelist.to_h
end

    That shows everything with the SecurityStatus tag (instances and values, shown as a hash), but you could also filter on value. From there, you could pull the instance IDs with a map (you might want to further filter only on instances, but if you made it this far you can figure that out):

     instances = instancelist.map(&:resource_id)

    Then you can pump those into a method to change the Security Group. Here’s the one from SecuritySquirrel that places an instance into a “Quarantine” group:

    def quarantine
  #this method moves the provided instance into the Quarantine security group defined in the config file.
  puts ""
  puts "Quarantining #{@instance_id}..."
  quarantine = @@ec22.modify_instance_attribute(instance_id: "#{@instance_id}", groups: ["#{@QuarantineGroup}"])
  puts "#{@instance_id} moved to the Quarantine security group from your configuration settings."
end

    Want to trigger some scanning? Here’s a code snippet to trigger a Qualys scan on an instance from a virtual appliance in your AWS account (keep in mind, I’m skipping a bunch of important pieces, like configuring the service and authenticating):

      instance_IP = instance_details.reservations.first.instances.first.private_ip_address
 timestamp = Time.new
 scan =(HTTParty.post("https://qualysapi.qg2.apps.qualys.com/api/2.0/fo/scan/",
      :basic_auth => @qualysauth,
      :query => { 
        :action => "launch",
        :scan_title => "SecuritySquirrel Scan at #{timestamp}", 
        :ip => "#{instance_IP}",
        :option_title => "Initial Options",
        :iscanner_name =>  "us-west-2" 
      },
      :headers => { "X-Requested-With" => "ruby httparty"}))
      puts "Launching Qualys scan named: SecuritySquirrel Scan at #{timestamp}"

    Or want to change the CloudPassage Halo host firewall rules? Well, that code uses a pre-release of their Ruby SDK, so I’ll save it for another day (I’m not employed by them, but they let me play with the pre-release gem). These examples are all based on tags, but remember that if you use a tool like Chef or Puppet you can do the exact same thing based on what software is configured and running within the instance. For example, you could create a policy to quarantine all SSL when the Heartbleed vulnerability hit by closing off port 443 on any server running the vulnerable OpenSSL package, then remove the rule when it is updated to the patched version. I’m a bit rusty since I don’t code every day, but I could probably pop that code off in an hour or two, it isn’t that hard. Also my tool is designed for demos, and doesn’t run in the background, but clearly you could have it running all the time, adjusting things based on tags or even attributes of software deployed in the instance. Security automation is insanely powerful. It can completely automate complex security tasks, or, as in Andrew’s example, supplement manual activities and policy changes. Hopefully my snippets give you some ideas, and pop me over any good examples if you are willing to share.

  • Hacking Your Auditor

    Hacking Your Auditor

    No, not that kind of hacking, give me a little credit…

    Recently I was having a conversation with a consulting friend about DevOps, and he found himself in a bit of a tough situation. The organization he was working with was a pure cloud/DevOps shop, but one that was stepping into having the meet compliance obligations. Needless to say, things weren’t going well with the auditor.

    For those of you who haven’t had to deal with the miasma of audits, one of the strongest requirements, for nearly any regulation/contract/agreement, is strong change management. This is especially true after those Enron guys screwed everything up for the rest of us.

    Change management makes a ton of sense from both an operational and compliance standpoint. I, myself, have blown things up in production but not being smart about my change management. For compliance, it is absolutely essential. If you are dealing with anything with financial or privacy implications, you sure as heck want to get it into a stable, secure, compliant state and ensure that state is kept over time.

    There is a big misconception out there that DevOps-style organizations don’t have change management. That when you push multiple changes per developer per day into production, there is no accountability or control. Now, while you can certainly build things this way, that sure isn’t how I see successful teams doing it. Not even close.

    The problem isn’t that DevOps necessarily skips change management, it’s that it looks different. Much of change management becomes automated, and the fundamental nature of changes themselves is different. But think about how you describe DevOps to your traditional IT-focused friends. Now imagine trying to explain this to an auditor working off set standards that are still figuring out if cloud includes a co-hosted Mac Mini duct taped to a rack.

    Here are some tips on how to both adjust your process, and explain it to the auditors:

    • First, figure out which pieces of your application and operations are in-scope for the regulation. Odds are only financial and privacy related bits are, and you can then apply more-granular controls or logging to these if you need.
    • Make sure your continuous integration tool (if you have one) includes some security checks and logs all updates/changes. If you push right from your code repository to production, get a CI tool.
    • Keep records of all changes as they flow from dev to test to prod (or whatever you use). Again, this should be trivial to automate and dump the records to a secure log repository someplace.
    • Unless your cloud provider tracks all API calls (like CloudTrail does on AWS, although that only works for some services and regions) use an API or web UI proxy like Netflix’s Asgaard. This really is a good security practice anyway, and can actually speed up infrastructure and platform management processes anyway. Keep the logs from this to make the auditor happy.
    • If you SSH into instances, you need to log that activity and dump the logs to a central repository.
    • Make sure you keep revisions of both all code changes and infrastructure automation changes (e.g. your CloudFormation templates, Chef/Puppet scripts/etc.). This is often built-in, but make sure you configure it correctly.
    • Those suggestions now give you logs of all application changes, infrastructure changes, and direct access to instances. These are the things the auditor wants to know exist.
    • Filter them based on your in-scope compliance requirements. Such as payment systems components only.
    • Explain to your auditor that all changes go through an automated approval system that checks the code, tests it (detail the tests), tracks the change, and then moves it into production. That all server logins and infrastructure changes are logged, as is the complete history of the structure of your entire environment. And that you can provide all the documentation, filtered for the pieces within compliance scope.

    This might look like a lot, but odds are very high you are already doing most of this. You keep code revisions, run changes through test suites before they go into production, and track all infrastructure changes. You might not have your own Ops portal/proxy, but you really should for a lot of reasons that have nothing to do with compliance.

    The trick is explaining this to your auditor in their terms. They expect manual checks and records. You need to show them that you still have these checks and balances, but they are automated. You still might need to add some pieces, such as an alert if certain parts of your code are changed (e.g. the file with payment processing or account reconciliation), but these will hopefully be minimal.

    A good DevOps environment actually has a much better compliance infrastructure in place than traditional infrastructure. The automation and template-driven models give us much better control and insight as compared to manually fixing everything, even when we push more day to day changes. The trick is to document this, add a couple pieces if you need it, and then explain it all to someone with… less than refined technical skills.

     

     

  • Bootstrapping Chef (or Whatever) for Autoscaled EC2 Instances

    Bootstrapping Chef (or Whatever) for Autoscaled EC2 Instances

    I realize it is traditional to start writing a new blog with some background and a deep introspection as to the author’s personal motivation for writing said blog, but I’ve never been one for tradition. Thus, for my first official DevOps post, I think I’ll jump write in with a technical tutorial on a problem had to solve last summer that I haven’t seen well documented anywhere else.

    You can read my bio for more, but here’s what you can expect on HackOps. I’m an industry analyst (in security; I’m the CEO of Securosis), but one with a bad habit of giving technical talks at DEFCON. In other words, a mix of research and analysis at both a technical and executive level.

    The Problem

    With that, let’s start with the technical:

    Last summer I was putting together a demonstration for the Black Hat conference when I ran into a little roadblock. I wanted to launch instances and have them automatically connect to a Chef server and pull down a default set of (security) policies. The problem is there really isn’t a great, built-in way to do this, nor could I find one documented anywhere. Normally, your options to install Chef in an EC2 instance are:

    * Load Chef and the configuration into a custom Amazon Machine Image. I wanted to use default AMIs instead of maintaining my own, so this wouldn’t work.
    * Launch instances using the Knife command-line tool (how you manage Chef, for those of you who don’t know) and the EC2 option. But I want to autoscale my images, which means I can’t launch them from Knife, since Amazon Web Services will launch them for me.
    * Bootstrap instances after they are up and running using the SSH bootstrap. But this is manual, or requires me to embed my SSH key in some automagic code, so this wasn’t an option.
    * Manually install and configure Chef. Not very DevOpsy, is it?

    Now I already had some decent experience configuring instances when they launch using cloud-init. cloud-init is a tool included in some operating systems, most notably Ubuntu and Amazon Linux, that allows you to inject a script when you launch an instance and have it run the commands once it launches, before you get to log into it. You simply paste your script into the User Data field.

    Overview

    Chef isn’t actually well-designed to handle this, since it seems to prefer a hands-on install, but I managed to hack together an effective process. At the high level, here’s how it works:

    1. Place your chef configuration (client.rb), validation certificate (validation.pem), and initial first run Chef file (first_run.json) into an Amazon S3 bucket.
    2. Use AWS IAM Roles to provide access to the S3 bucket. IAM Roles are worth a post all on their own, but the short version is you set the role for an object with Amazon and it is provided a set of rotating, temporary credentials to access other bits of Amazon.
    3. Write a cloud-init script to download Chef and s3cmd, which is a command line tool for accessing Amazon S3.
    4. In your script, set s3cmd to work with IAM Roles by installing a blank configuration file (figuring this part our was a major pain since it wasn’t documented at the time).
    5. In your script, use s3cmd to pull down your Chef configuration files, and then install Chef with the configuration files we pulled from S3, one of which will run it for the first time and set the initial role, which Chef will use to push down the first policy set.
    6. Launch an instance manually or as part of an autoscaling group. Set the IAM Role, and paste your cloud-init script into the *User Data* field.

    Now I am assuming a few things here. First, that you know how to use EC2 and set the correct security groups for everything to talk to each other. Second, that you know how to create an autoscaling group (or manually launch an instance and set the User Data field). Third, that you have a Chef server or Hosted Chef set up already.

    If you want more of a step-by-step you can read up on this in a paper I published — A Practical Example of Software Defined Security. It is ever so slightly out of date, but still works (just adjust it to get a current version of S3 tools and cut the *fix routing silliness* part of the script since that isn’t needed anymore and breaks things).

    The Details

    Here are the technical details for the files and commands:

    First, for your IAM Role policy. Replace “cloudsec” with the name of whatever bucket you create to hold your client.rb, validator.pem, and first_run.json (I’ll give you an example one in a moment):

     

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Action": [
    "s3:Get*",
    "s3:List*"
    ],
    "Resource": "arn:aws:s3:::cloudsec",
    "Resource": "arn:aws:s3:::cloudsec/*"
    }
    ]
    }

    Next comes your cloud-init script. You can either past the entire script in, or use the command:

    #include
    http://url-to-script

    If you use a remote script, you probably want to store it someplace safe in S3 where no one can see it and see how you have things configured. Here is the cloud-init script itself, where the magic happens (our blog platform messes up the formatting a bit, but it is still readable):

    #cloud-config

    apt_update: true

    #apt_upgrade: true

    packages:
    - curl

    configchef:
    - &configchef |
    echo "deb http://apt.opscode.com/ precise-0.10 main" | sudo tee /etc/apt/sources.list.d/opscode.list
    apt-get update
    curl http://apt.opscode.com/packages@opscode.com.gpg.key | sudo apt-key add -
    echo "chef chef/chef_server_url string http://your-chef-server-URL:4000" | sudo debconf-set-selections && sudo apt-get install chef -y --force-yes
    wget http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-beta1/s3cmd-1.5.0-beta1.tar.gz
    tar xvfz s3cmd-1.5.0-beta1.tar.gz
    cd s3cmd-1.5.0-beta1/
    cat >s3cfg <<EOM
    [default]
    access_key =
    secret_key =
    security_token =
    EOM
    ./s3cmd --config /s3cmd-1.5.0-beta1/s3cfg ls s3://cloudsec/
    ./s3cmd --config /s3cmd-1.5.0-beta1/s3cfg --force get s3://cloudsec/client.rb /etc/chef/client.rb
    ./s3cmd --config /s3cmd-1.5.0-beta1/s3cfg --force get s3://cloudsec/validation.pem /etc/chef/validation.pem
    ./s3cmd --config /s3cmd-1.5.0-beta1/s3cfg --force get s3://cloudsec/first_run.json /etc/chef/first_run.json
    chef-client -j /etc/chef/first_run.json

    runcmd:
    - [ sh, -c, *configchef]
    - touch /tmp/done

    You’ll notice that when we launch Chef, we use the command line option to load first_run.json, which establishes out initial Chef role. This is what allows us to push an initial policy to the instance, otherwise we have to do more manual stuff (ick). I kept it simple and merely established a starter role for the instance, which corresponds to a basic run list. first_run.json looks like this:

    {“run_list”:[“role[base]”]}

    Once you get all this set up, as I mentioned earlier you only need to launch an instance with the right IAM Role to access the S3 bucket, then paste in the cloud-init script to the *User Data* field. This can all be set as part of your autoscaling group rules so it runs automatically as new instances launch.

    The rest is magic. The instance takes a little longer to run, and it can take a bit for all the pieces to connect, but we are talking less than 30 minutes. Usually a *lot* less than 30 minutes.

    Now while I used this as a security demo, I know this technique is used in product for at least one major web property you have all heard of for non-security configuration management. Not that I can take credit for it, I found out months later they use a really similar technique they figured out on their own.

    The same basic techniques should also work with Puppet or the configuration management tool of your choice.

    To be honest, I was pretty surprised I couldn’t find this process documented anywhere when I started my project. It seems like one of those basic things nearly anyone working in public cloud with a configuration management tool would need, since building your own AMIs all the time is really inefficient.

    That’s one of the big reasons I’m excited to be writing here at devopsy.kinsta.cloud, since I think we, as a community, need a good meeting place to share this kind of information.