devopsy.kinsta.cloud

Tag: AWS

  • Using AWS CloudWatch for Anomaly Detection

    Using AWS CloudWatch for Anomaly Detection

    Based on my unscientific poll of friends, one of the least used and most overlooked features of AWS is CloudWatch.  Not only can CloudWatch be used to monitor the availability of your AWS services, but it can also be used as anomaly an detection tool. Did I mention that these feature are free?

    Since the holy grail of DevOps is to bring  the security and the operations teams together to meet the same goals in an automated fashion; Cloud Watch is a perfect DevOps tool. In fact, once it’s implemented for your environment  you are one step closer to the exalted state of SecDevOps.

    If you aren’t familiar with AWS CloudWatch then head on over to https://aws.amazon.com/cloudwatch/. As a quick introduction, here’s how AWS website describes the service:

    “Amazon CloudWatch provides monitoring for AWS cloud resources and the applications customers run on AWS. Developers and system administrators can use it to collect and track metrics, gain insight, and react immediately to keep their applications and businesses running smoothly. Amazon CloudWatch monitors AWS resources such as Amazon EC2 and Amazon RDS DB instances, and can also monitor custom metrics generated by a customer’s applications and services. With Amazon CloudWatch, you gain system-wide visibility into resource utilization, application performance, and operational health.”

    Let’s assume that both security and Ops teams need to be aware when weird things are happening with their servers and applications.  Events outside the norm (aka weird things) can be either be related to security or operational events; either way they may merit investigation. Here’s a simple CloudWatch  monitoring process you can use  to give you a heads up on events that could indicate someone is trying to exfiltrate data from your systems.

    Opening Scene

    Think about what kind of information attackers are after.  They want your intellectual property and your customer data because it can be converted to cold, hard cash.  Next, consider the architecture of your systems and where that data is stored. Most likely your treasures are stored in databases or Git repositories.

    Guess what? Attackers know that so that’s exactly what they are looking for.  Take a few minutes and think like an attacker bent on exfiltrating your data.  Obviously, the first thing you should do is carefully monitor your egress traffic for suspicious activity.  In addition, look for anomalies in traffic flows – they can also help determine if suspicious events are happening.  Amazon CloudWatch can help you do  both of these things.(Note: This process assumes you haven’t already been breached.)

     1. Enable CloudWatch

    2. Establish a baseline

    Review the historical data on a daily, weekly and monthly basis.  Soon you will start to see a trend in your egress bandwidth usage.  Start with eyeballing the min, max and average usage over time.  Put these in your favorite tracking tool (text file, spreadsheet, abacus, whatever).

    3. Configure alerts

    When you feel comfortable with your baseline, configure your first alert to send an email if traffic usage goes over the baseline.  Consider how “chatty” you want this event to be.  For example, alerting over the average baseline will often generate more alerts than alerts over maximum usage.  This is a tricky balance and you’ll need to make trade offs between missing problems and the time spent responding to potential false positives.

    4. Respond

    However you choose to respond is up to you.  I suggest you start with automatically opening a ticket in your tracking system so that someone will be sure to review the alert and make notes on their findings.

    5. Tune as needed

    Make a plan to review the alerts and the data on a regular basis.  Architecture and other changes to the system are going to change the baseline and the alerts may need to be reconfigured.

    Building and automating security into DevOps is a requirement that everyone should be working toward. It’s a theme I’m going to be returning to in future posts because it’s really important.

    AWS Cloudwatch is a free tool that you can help you get started automating security right now, so what are you waiting for?

  • Bootstrapping Chef (or Whatever) for Autoscaled EC2 Instances

    Bootstrapping Chef (or Whatever) for Autoscaled EC2 Instances

    I realize it is traditional to start writing a new blog with some background and a deep introspection as to the author’s personal motivation for writing said blog, but I’ve never been one for tradition. Thus, for my first official DevOps post, I think I’ll jump write in with a technical tutorial on a problem had to solve last summer that I haven’t seen well documented anywhere else.

    You can read my bio for more, but here’s what you can expect on HackOps. I’m an industry analyst (in security; I’m the CEO of Securosis), but one with a bad habit of giving technical talks at DEFCON. In other words, a mix of research and analysis at both a technical and executive level.

    The Problem

    With that, let’s start with the technical:

    Last summer I was putting together a demonstration for the Black Hat conference when I ran into a little roadblock. I wanted to launch instances and have them automatically connect to a Chef server and pull down a default set of (security) policies. The problem is there really isn’t a great, built-in way to do this, nor could I find one documented anywhere. Normally, your options to install Chef in an EC2 instance are:

    * Load Chef and the configuration into a custom Amazon Machine Image. I wanted to use default AMIs instead of maintaining my own, so this wouldn’t work.
    * Launch instances using the Knife command-line tool (how you manage Chef, for those of you who don’t know) and the EC2 option. But I want to autoscale my images, which means I can’t launch them from Knife, since Amazon Web Services will launch them for me.
    * Bootstrap instances after they are up and running using the SSH bootstrap. But this is manual, or requires me to embed my SSH key in some automagic code, so this wasn’t an option.
    * Manually install and configure Chef. Not very DevOpsy, is it?

    Now I already had some decent experience configuring instances when they launch using cloud-init. cloud-init is a tool included in some operating systems, most notably Ubuntu and Amazon Linux, that allows you to inject a script when you launch an instance and have it run the commands once it launches, before you get to log into it. You simply paste your script into the User Data field.

    Overview

    Chef isn’t actually well-designed to handle this, since it seems to prefer a hands-on install, but I managed to hack together an effective process. At the high level, here’s how it works:

    1. Place your chef configuration (client.rb), validation certificate (validation.pem), and initial first run Chef file (first_run.json) into an Amazon S3 bucket.
    2. Use AWS IAM Roles to provide access to the S3 bucket. IAM Roles are worth a post all on their own, but the short version is you set the role for an object with Amazon and it is provided a set of rotating, temporary credentials to access other bits of Amazon.
    3. Write a cloud-init script to download Chef and s3cmd, which is a command line tool for accessing Amazon S3.
    4. In your script, set s3cmd to work with IAM Roles by installing a blank configuration file (figuring this part our was a major pain since it wasn’t documented at the time).
    5. In your script, use s3cmd to pull down your Chef configuration files, and then install Chef with the configuration files we pulled from S3, one of which will run it for the first time and set the initial role, which Chef will use to push down the first policy set.
    6. Launch an instance manually or as part of an autoscaling group. Set the IAM Role, and paste your cloud-init script into the *User Data* field.

    Now I am assuming a few things here. First, that you know how to use EC2 and set the correct security groups for everything to talk to each other. Second, that you know how to create an autoscaling group (or manually launch an instance and set the User Data field). Third, that you have a Chef server or Hosted Chef set up already.

    If you want more of a step-by-step you can read up on this in a paper I published — A Practical Example of Software Defined Security. It is ever so slightly out of date, but still works (just adjust it to get a current version of S3 tools and cut the *fix routing silliness* part of the script since that isn’t needed anymore and breaks things).

    The Details

    Here are the technical details for the files and commands:

    First, for your IAM Role policy. Replace “cloudsec” with the name of whatever bucket you create to hold your client.rb, validator.pem, and first_run.json (I’ll give you an example one in a moment):

     

    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Action": [
    "s3:Get*",
    "s3:List*"
    ],
    "Resource": "arn:aws:s3:::cloudsec",
    "Resource": "arn:aws:s3:::cloudsec/*"
    }
    ]
    }

    Next comes your cloud-init script. You can either past the entire script in, or use the command:

    #include
    http://url-to-script

    If you use a remote script, you probably want to store it someplace safe in S3 where no one can see it and see how you have things configured. Here is the cloud-init script itself, where the magic happens (our blog platform messes up the formatting a bit, but it is still readable):

    #cloud-config

    apt_update: true

    #apt_upgrade: true

    packages:
    - curl

    configchef:
    - &configchef |
    echo "deb http://apt.opscode.com/ precise-0.10 main" | sudo tee /etc/apt/sources.list.d/opscode.list
    apt-get update
    curl http://apt.opscode.com/packages@opscode.com.gpg.key | sudo apt-key add -
    echo "chef chef/chef_server_url string http://your-chef-server-URL:4000" | sudo debconf-set-selections && sudo apt-get install chef -y --force-yes
    wget http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-beta1/s3cmd-1.5.0-beta1.tar.gz
    tar xvfz s3cmd-1.5.0-beta1.tar.gz
    cd s3cmd-1.5.0-beta1/
    cat >s3cfg <<EOM
    [default]
    access_key =
    secret_key =
    security_token =
    EOM
    ./s3cmd --config /s3cmd-1.5.0-beta1/s3cfg ls s3://cloudsec/
    ./s3cmd --config /s3cmd-1.5.0-beta1/s3cfg --force get s3://cloudsec/client.rb /etc/chef/client.rb
    ./s3cmd --config /s3cmd-1.5.0-beta1/s3cfg --force get s3://cloudsec/validation.pem /etc/chef/validation.pem
    ./s3cmd --config /s3cmd-1.5.0-beta1/s3cfg --force get s3://cloudsec/first_run.json /etc/chef/first_run.json
    chef-client -j /etc/chef/first_run.json

    runcmd:
    - [ sh, -c, *configchef]
    - touch /tmp/done

    You’ll notice that when we launch Chef, we use the command line option to load first_run.json, which establishes out initial Chef role. This is what allows us to push an initial policy to the instance, otherwise we have to do more manual stuff (ick). I kept it simple and merely established a starter role for the instance, which corresponds to a basic run list. first_run.json looks like this:

    {“run_list”:[“role[base]”]}

    Once you get all this set up, as I mentioned earlier you only need to launch an instance with the right IAM Role to access the S3 bucket, then paste in the cloud-init script to the *User Data* field. This can all be set as part of your autoscaling group rules so it runs automatically as new instances launch.

    The rest is magic. The instance takes a little longer to run, and it can take a bit for all the pieces to connect, but we are talking less than 30 minutes. Usually a *lot* less than 30 minutes.

    Now while I used this as a security demo, I know this technique is used in product for at least one major web property you have all heard of for non-security configuration management. Not that I can take credit for it, I found out months later they use a really similar technique they figured out on their own.

    The same basic techniques should also work with Puppet or the configuration management tool of your choice.

    To be honest, I was pretty surprised I couldn’t find this process documented anywhere when I started my project. It seems like one of those basic things nearly anyone working in public cloud with a configuration management tool would need, since building your own AMIs all the time is really inefficient.

    That’s one of the big reasons I’m excited to be writing here at devopsy.kinsta.cloud, since I think we, as a community, need a good meeting place to share this kind of information.

  • Ops is dead, long live DevOps

    Ops is dead, long live DevOps

    Yes, that’s a bold statement, but if the vibe at the recent AWS re:Invent conference was any indication there are a number of methodologies, technologies, and companies working to make that a reality. This article isn’t focused on whether this is a good or bad thing (my personal belief is that ops will always be a critical part of the IT delivery process). Regardless, it is critical for IT operations personnel to understand the implications of this tectonic shift. As Marc Andreessen, founder of the venture capital firm Andreessen Horowitz, has said, “software is eating the world.” As a result, ops professionals must have the ability to automate tasks that used to be managed manually. Whether or not you think that is the death of ops as we know it I will leave to you, but it does mean we will live in a world where DevOps will offer distinct advantages.

    Before we dive into what this change means to ops personnel, let’s review the underlying assumptions. The fundamental driver at the a high level has been the DevOps movement. Yes, DevOps is quickly becoming the buzzword dejour, but the underlying principles are lasting and changing IT to its core. DevOps is about integrating the various functions that it takes to release software or applications: software development, quality assurance, operations, and security among many others and rapid iteration. The process of integrating these tasks is accomplished through new processes and automation. Different functional areas such as development and operations are brought into the process earlier, and their processes are automated to the extent that they can be. This results in fast cycle times, increased quality, and hopefully increased innovation for the business. All of this is accomplished through software – at each functional level.

    Some technologies that have enabled this massive shift include Infrastructure as a Service (IaaS), Platform as a Service (PaaS), automated configuration management, server management tools, virtualization, and software defined networking. IaaS and PaaS offerings are eliminating the need for organizations to procure, provision, and deploy servers. Further, this class of technology is also dramatically reducing the need to deal with the network layer. This allows organizations to concentrate on their applications and development, and focus less on their infrastructure. The amount of time and effort spent on server and network configuration is minimized. Configuration management solutions are enabling the automation of server setup. An organization’s technology stack can be pushed, pulled, cloned and copied versus manual configuration thereby saving time and adding consistency. Top performing organizations employing DevOps go a step further and automate the scaling of capacity based on utilization. To reduce day-to-day operational activities and costs, organizations are also automating the management of servers, including users access, patches, security, and other areas through both traditional written code and SaaS solutions. Another area changing the game for ops personnel is software defined networking (SDN), which is effectively morphing data center and hardware functions into software programs.

    In short, DevOps and associated innovative new technologies are driving a paradigm shift in the way operations is done. No longer are ops personnel required to manually provision, configure, and manage the network, servers, and application stack. Combining technologies from organizations such as AWS, Rackspace, and SoftLayer along with configuration tools such as Chef and Puppet drives the provisioning and configuration of applications through software automation rather than manual effort. Once applications are deployed, the underlying infrastructure can be managed with tools such as my own company, New Relic, this author’s own JumpCloud, and others. Ops personnel have more companies than ever focusing on helping them automate the tasks that they need to deliver applications for their organizations.

    The result from all of this effort is that ops as we know it won’t exist in the future. The network operations wizard that could manually troubleshoot routing and switching issues, provision servers quickly, and tweak configurations will be tasked with automating those tasks through software. Some of that automation will be through code that ops person writes herself, some delivered by third party providers, and other portions delivered by open source or SaaS-based solutions. But, don’t shed tears for the forced obsolescence of the IT ops people. They will still exist and perhaps be more important and powerful than ever. In fact, because of DevOps ops personnel will in essence be set free from mundane, repetitive “busy work” to perform higher value tasks as well as managing over the software and systems at work. The black magic of enabling different layers of the stack to function together will be less about art and more about science, or rather good software. Solutions will be engineered to work together and be tied together through code. Caveat: yes, service providers and some large enterprises will  still run their own data centers and networks, but the same reasons that DevOps works in IaaS or PaaS will apply to private data centers or private clouds as well.

    If all of this trends toward being true as we see it happening today, what are the opportunities for ops personnel to be at the forefront of this shift? Beyond embracing DevOps – which many operations personnel are doing already – learning to code is the most critical step that ops professionals can take. Many ops professionals already know how to script which is absolutely a minimum, but many will need to go further. Embrace the DevOps movement and push further towards the development side by learning how to code. There are any number of amazing services to teach us how to code or code better. Take the step. It may be the difference between a vestige of the old school versus driving the new world in your organization. The second significant step you can take is to automate your job. Leverage the very best automation tools to code your network, server, and application infrastructure. Not only will you spend less time on these tasks manually, but you will increase availability, reliability, and performance. In most organizations, that’s more than a gold star. Finally, make the mental shift that operations in the future will be less art and more science. DevOps is driving action through data, and the technology stack is responding to this demand with more telemetry than ever. Problems are now solved through understanding performance, availability, and other metrics. And code.

    DevOps is the future. Automation is key. Code is the answer. Developers and ops are in it together for the good of their organizations. Ops pros that embrace this will win! Ops is Dead, Long Live DevOps!

  • Automation Driving DevOps Adoption

    Survey of Attendees at JumpCloud and SoftLayer Hosted DevOps Conference Confirms High Adoption of DevOps Driven by Demand for Automation of IT & Time Constraints
    February 10, 2014 11:00 AM Eastern Standard Time

    BOULDER, Colo.–(BUSINESS WIRE)–JumpCloud™, the leading automated server management tool for DevOps and IT pros, today published survey results from its DevOps Conference hosted in conjunction with SoftLayer. The survey confirms that the rapid adoption of DevOps is driven by the need for increased automation and constant pressure to complete tasks. When asked to describe their greatest challenges and most time consuming tasks, leading responses referenced deployment and patching activities. User management and log file analysis and forensics were also near the top of responses by the DevOps Conference attendees.

    “We decided to hold this DevOps conference with Foundry Group, TechStars, and Bullet Time Ventures companies in order to gain some base level insight and understanding into the state of the DevOps movement”

    Regardless of the specific challenges or tasks identified, attendees of the event unified around an increasing lack of time with expanding responsibilities and accelerated production cycles. Survey results pointed specifically to manual tasks as hurdles to time management. In order to address this issue, DevOps focused organizations are searching for ways to automate tasks and more effectively manage the explosion of online infrastructure.

    “We decided to hold this DevOps conference with Foundry Group, TechStars, and Bullet Time Ventures companies in order to gain some base level insight and understanding into the state of the DevOps movement,” said Rajat Bhargava, CEO, JumpCloud. “The conference aimed to advance the DevOps movement through the sharing of best practices and a compilation of survey data from a highly targeted respondent group of leading voices in the DevOps movement. We could not have been happier with the over 200 leaders in the field from over 100 companies who were able to attend and help us to gain and share some insight into this increasingly important movement.”

    Over three quarters of respondents classified themselves as DevOps organizations. Another 15% were on their way to becoming DevOps organizations.

    When asked which cloud providers respondents utilized:

    AWS – 55%
    Rackspace – 14%
    SoftLayer – 6%
    Digital Ocean – 6%
    Heroku – 6%
    Misc. – 13%
    When asked about steps to increase automation in their organizations:

    Over 75% reported using some form of configuration automation tool such as Chef or Puppet.
    Approximately one-third reported using some form of patching solution.
    Over half are utilizing pre-built images.
    When asked which tasks most commonly required manual management processes

    1. Deployment and patching tied for first
    2. User management
    3. Log file analysis and forensics
    The DevOps Conference was co-founded and sponsored by JumpCloud and SoftLayer. Gold sponsors included Authentic8, DH Capital, and VictorOps. Silver sponsors were Cooley, Pantheon, PivotDesk, SendGrid, and Silicon Valley Bank. For more information about the DevOps conference please contact info@jumpcloud.com. Conference organizers are planning to implement multiple events in 2014 in conjunction with devopsy.kinsta.cloud. The conference videos are available here. Full results of the survey are available here.

    About JumpCloud

    JumpCloud automates manual, tedious server management tasks from the cloud for DevOps and IT pros. Complementary to and integrated with Chef and Puppet, JumpCloud picks up where they leave off by automating management of users, patches, security, and compliance. JumpCloud returns days to DevOps and IT admins’ lives while increasing security. An automated SaaS-based offering, JumpCloud installs in seconds and deploys on physical, virtual, or cloud servers. For more information visit JumpCloud.com.

  • Ravello Systems Powers DevOps with Extreme Continuous Integration on Clones of Production Applications in the Cloud

    Ravello adds new Apache Maven plug-in and announces availability of Ruby and Python bindings for integration with build automation tools.

     

    PALO ALTO, Calif. – February 12, 2014 – Ravello Systems, the industry’s first Cloud Application Hypervisor provider, today announced it has added new plug-ins and RESTful APIs to provision entire multi-virtual machine environments, along with networking and storage, in the cloud. Ravello has added a new Apache Maven plug-in, and programming language bindings for Ruby and Python, with support for Continuous Integration (CI) systems including Jenkins, Bamboo, and Teamcity. With one API call, developers can instantly provision a complete clone of their existing on-premise production application and deploy it in the public cloud.

    Ravello enables CI on production replicas in the cloud

    To achieve successful agile development and test processes, enterprises are quickly adopting DevOps principles, and using CI is a key practice. The goal with CI is to get as reliable feedback as quickly as possible, resulting in higher quality software delivered faster. However, most DevOps practitioners encounter constraints when implementing CI due to (1) difficulties in automating the infrastructure, making it problematic to provision clones of multi-tier production applications, and (2) capacity restraints in the datacenter, making it impossible to deploy these clones for every code commit. Automating CI processes can be very painful, time-consuming and fraught with configuration nightmares.

    With Apache Maven, the vastly popular build automation tool, developers are able to manage a project’s build, deployment, testing, reporting and documentation from a central repository. Recognizing the value of Maven as a build automation tool, the Ravello Maven plug-in is designed to help integrate Maven-based projects with Ravello Systems’ rich APIs. This integration makes it easier to automatically configure the build, testing and deployment processes, resulting in smoother operations.

    Ravello’s Software-as-a-Service (SaaS) offering allows enterprises to spin up clones of their existing multi-tier enterprise applications in the public cloud – without any changes to the virtual machines (VMs), networking or storage. With the addition of a new Maven plug-in and the availability of Ruby and Python bindings for integration with other build automation tools, Ravello now enables extreme CI on production clones in the cloud. Now developers and test engineers can use Ravello’s APIs to:

    • Provision the entire environment;
    • Start and stop applications as needed;
    • Automatically execute the tests and either shut down to release resources once the test has passed, or save the state if an issue needs to be debugged;
    • Easily integrate with CI servers such as Jenkins, Bamboo or Teamcity.

    All this can be done right from the integrated development environment (IDE). Now each code commit can be tested on a clone of production in the cloud for instant and accurate feedback.

    “The public cloud with its infinite capacity is very promising for continuous integration, but it’s an inherently different environment than what you have on premise,” said Gil Hoffer, VP of R&D, Ravello Systems. “Your application might have multiple subnets, multiple network cards per server and specific storage requirements. With Ravello we abstract away the differences between the private and public clouds, and have built-in automation for provisioning and deploying those multi-tier applications in the cloud.”

    Supporting Resources

    Follow Ravello Systems on the Web at:

    About Ravello Systems
    Founded in 2011, Ravello Systems delivers a software-as-a-service that enables developers, devops and IT to use the unlimited resources of the public cloud to develop and test their on-premise applications. Powered by the industry’s first Cloud Application Hypervisor, Ravello enables enterprises to encapsulate their multi-tier applications and run them anywhere – on-premise or in any cloud – without making any changes whatsoever, and thereby implement a true hybrid cloud. Ravello Systems is backed by top tier VCs including Sequoia Capital, Norwest Venture Partners and Bessemer Venture Partners. For more information, visit www.ravellosystems.com.

  • Data Driven Observations on AWS Usage from CloudCheckrs User Survey

    Data Driven Observations on AWS Usage from CloudCheckrs User Survey

    This is a guest post by Aaron Klein from CloudCheckr

    We were heartened when AWS made Trusted Advisor free for the month of March. This was an implicit acknowledgement of what many have long known: AWS is complex and can be challenging for users to provision and control their AWS infrastructure effectively.

    We took the AWS announcement as an opportunity to conduct an internal survey of our customers’ usage. We compared the initial assessments of 400 of our users’ accounts against our 125+ best practice checks for proper configurations and policies. Our best practice checks span 3 key categories: Cost, Availability, and Security.  We limited our survey to users with 10 or more running EC2 instances.  In aggregate, the users were running more than 16,000 EC2 instances.

    We were surprised to discover that nearly every customer (99%) experienced at least one serious exception.  Beyond this top level takeaway, our primary conclusion was that controlling cost may grab the headlines, but users also need to button up a number of availability and security issues.

    When considering availability, there were serious configuration issues that were common across a high percentage of users. Users repeatedly failed to optimally configure Auto Scaling and ELB. The failure to create sufficient EBS snapshots was an almost universal issue.

    Although users passed more of our security checks, the exceptions which did arise were serious. Many of the most commons security issues were found in configurations for S3, where nearly 1 in 5 users allowed unfettered access to their buckets through “Upload /Delete” or “Edit Permissions” set to everyone. As we explained in an earlier whitepaper, anyone using a simple bucket finder tool could locate and access these buckets.

    Beyond the numbers, we also interviewed customers to gather qualitative feedback from users on some of the more interesting data points.

    If the findings of this survey sparks questions about how well your AWS account is configured, CloudCheckr offers a free account that you can set up in minutes.  Simply enter read only credentials from your AWS account and CloudCheckr will assess your configurations and policies in just a few minutes:  https://app.cloudcheckr.com/LogOn/Registration

    Conclusions by Area

    Conclusions based upon Cost Exceptions:

    As noted, our sample was comprised of 16,047 instances. The sample group spent a total of $2,254,987 per month on EC2 (and its associated costs) for average monthly cost per customer of $7516. Of course, we noted the mismatch between quantity and cost – spot instances represent 8% of the quantity but only 1.4% of the cost. This is due to the significantly less expensive price of spot instances compared to on demand.

    When we looked at the Cost Exceptions, we found that 96% of all users experienced at least 1 exception (with many experiencing multiple exceptions). In total, we found that users who adopted our recommended instance sizing and purchasing type were able to save an average of $3974 per month for an aggregate total of $1,192,212 per month.

    This suggested that price optimization remains a large hurdle for AWS users who rely on native AWS tools. Users consistently fail to optimize purchasing and also fail to optimize utilization. These combined issues meant that the average customer pays nearly twice as much as necessary for resources to achieve proper performance for their technology.

    To further examine this behavior, we interviewed a number of customers.  We interviewed customers who exclusively purchased on-demand and customers who used multiple purchasing types.

    Here were their answers (summarized and consolidated):

    • Spot instances worry users – there is a general concern of: “what if the price spikes and my instance is terminated?” This fear exists despite the fact that spikes occur very rarely, warnings are available, and proper configuration can significantly mitigate this “surprise termination” risk.
    • It is difficult and time consuming to map the cost scenarios for purchasing reserved instances. The customers who did make this transition had cobbled together home grown spreadsheets as a way of supporting this business decision.  The ones who didn’t make this effort made a gut estimate that it wasn’t worth the time.  AWS was cost effective enough and the time and effort for modeling the transition was an opportunity cost taken away from building and managing their technology.
    • The intricacies of matching the configurations between on demand instances and reserved instances while taking into consideration auto scaling and other necessary configurations were daunting. Many felt it was not worth the effort.
    • Amazon’s own process for regularly lowering prices is a deterrent to purchasing RIs. This is especially true for RIs with a 3 year commitment.  In fact, within the customers who did purchase RI, none expressed a desire to purchase RIs with a 3 year commitment. All supported their decision by referencing the regular AWS price drops combined with the fact that they could not accurately predict their business requirements 3 years out.

    Conclusions based upon Availability Exceptions:

    We compared our users against our Availability best practices and found that nearly 98% suffered from at least 1 exception. We hypothesized that this was due to the overall complexity of AWS and interviewed some of our users for confirmation. Here is what we found from those interviews:

    • Users were generally surprised with the exceptions. They believed that they “had done everything right” but then realized that they underestimated the complexity of AWS.
    • Users were often unsure of exactly why something needed to be remedied. The underlying architecture of AWS continues to evolve and users have a difficult time keeping up to speed with new services and enhancements.
    • AWS dynamism played a large role in the number of exceptions. Users commented that they often fixed exceptions and, after a week of usage, found new exceptions had arisen.
    • Users remained very happy with the overall level of service from AWS. Despite the exceptions which could diminish overall availability, the users still found that AWS offered tremendous functionality advantages.

    Conclusion bases upon Security Exceptions:

    Finally, we looked at security. Here we found that 44% of our users had at least one serious exception present during the initial scan. The most serious and common exceptions occurred within S3 usage and bucket permissioning. Given the differences in cloud v. data center architecture, this was not entirely surprising. We interviewed our users about this area and here is what we found:

    • The AWS management console offered little functionality for helping with S3 security. It does not provide a use friendly means of monitoring and controlling S3 inventory and usage. In fact, we found that most of our users were surprised when the inventory was reported. They often had 300-500% more buckets, objects and storage than they expected.
    • Price = Importance, S3 is often an afterthought for users. Because it is so inexpensive users do not audit it as closely as EC2 and other more expensive services and rarely create and implement formal policies for S3 usage.  The time and effort required to log into each region one by one to collect S3 information and download data through the Management console was not worth the effort relative to spend.
    • Given the low cost and lack of formal policies, team members throw up high volumes of objects and buckets knowing that they can store huge amounts of data at a minimal cost.  Since users did not audit what they had stored, they could not determine the level of security.

    cloudforestfinal

    aaronkleinAuthor Info: Aaron is the Co-Founder/COO of CloudCheckr Inc. (CCI). With over 20 years of managerial experience and vision, he directs the company’s operations.

    Aaron has held key leadership roles at diverse organizations ranging from small entrepreneurial start-ups to multi-billion dollar enterprises. Aaron graduated from Brandeis University and holds a J.D. from SUNY Buffalo.

    Underlying Data Summary

    Cost:                                                                                                       Any exception 96%

    The total of 16,047 instances was broken in the following categories:

    • On Demand:       78%    (12,517 instances)
    • Reserved:             14%    (2,247 instances)
    • Spot:                        8%      (1,284 instances)

    The instance purchasing was broken down as follows:

    • On Demand:        89.7%  ($2,023,623)
    • Reserved:             8.9%     ($199,803)
    • Spot:                        1.4%     ($31,561)

    Common Cost Exceptions we found:

    • Idle EC2 Instances                                                                                                      36%
    • Underutilized EC2 Instances                                                                               84%
    • EC2 Reserved Instance Possible Matching Mistake                                             17%
    • Unused Elastic IP                                                                                                        59%

     

    Availability:                                                                                              Any exception 98%

    Here, broken out by service, are some highlights of common and serious exceptions that we found:

    Service Type:                                                                                      Customers with Exceptions

    EC2:                                                                                                           Any exception   95%

     

    • EBS Volumes That Need Snapshots                                                                91%
    • Over Utilized EC2 Instances                                                                                                   22%

    Auto Scaling:                                                                                              Any exception   66%

     

    • Auto Scaling Groups Not Being Utilized  For All EC2 Instances                       57%
    • All Auto Scaling Groups Not Utilizing Multiple Availability Zones                34%
    • Auto Scaling Launch Configuration Referencing Invalid Security Group                   22%
    • Auto Scaling Launch Configuration Referencing Invalid AMI                           18%
    • Auto Scaling Launch Configuration Referencing Invalid Key Pair                 16%

    ELB:                                                                                                        Any exception   42%

     

    • Elastic Load Balancers Not Utilizing Multiple Availability Zones                 37%
    • Elastic Load Balancers With Fewer Than Two Healthy Instances               21%

    Security:                                                                                                    Any exception   46%

     

    These were the most common exceptions that we found:

    • EC2 Security Groups Allowing Access To Broad IP Ranges                                 36%
    • S3 Bucket(s) With ‘Upload/Delete’ Permission Set To Everyone                   16%
    • S3 Bucket(s) With ‘View Permissions’ Permission Set To Everyone            24%
    • S3 Bucket(s) With ‘Edit Permissions’ Permission Set To Everyone               14%