Category: Blogs

  • Security automation with DevOps: show me the code!

    Security automation with DevOps: show me the code!

    Last week Andrew Storms put up a good post hinting at the promise of security automation in [SecDevOps: Security Automation By Example – The Firewall Change]. He included an example of automating a series of actions when a firewall rule is changed. It’s a good article, although I’m increasingly convinced there’s no such thing as SecDevOps. In my book, it’s all DevOps, but that’s fodder for another post (when I’m not battling a stomach bug). However, what Andrew describes is more of what I consider an automated assist. It isn’t necessarily full automation, since it triggers on a manual firewall rule change. Ideally we rarely manually change a firewall (or Security Group) rule, and rely more on self-configuring based on policies. Yeah, I know, the usual analyst BS, so here’s a bit of process, and a bit of code. Let’s approach this differently. Take Andrew’s process, but let’s have the security group or host firewall self configure based on the asset. My example is going to be Amazon Web Services specific, since I have some code snippets to show off how it works. (Sorry, I don’t have the time or intestinal fortitude to write out all the code today; seriously, where did this bug come from?!?) There are two techniques I generally see forming the core of dynamic Security Group policy enforcement:

    1. Tags
    2. Configuration Management integration

    Some cloud services, like AWS, support tags at the object (in this case, instance) level. These tags can tie internally to policies while also being consumable for external tools. Here is a policy I use that restricts access to an object if the “SecurityStatus” tag is “IR” (for “Incident Response”):

    {  
      "Version": "2012-10-17",  
      "Statement": [  
        {
          "Action": "*",
          "Condition": {  
            "StringEquals": {
              "ec2:ResourceTag/SecurityStatus": "IR"
              }
              },
              "Resource": [
              "*"
                          ],
              "Effect": "Deny"
          }
          ]  
        }         
    

    I could apply that tag manually or automatically, and in either case the instance can no longer be managed by any account this policy is applied to. Now this doesn’t help us change firewall/Security Group rules, since you can’t manage those directly in EC2 based on tags (but I had the policy handy to demo, so there you go). For that to work, you need to write your own code to scan for tags and then take actions. I don’t have exactly the right code examples on hand. but here are some snippets to get you close. Most of these are cribbed from my SecuritySquirrel proof of concept on GitHub. They are all in Ruby, using the AWS 2.0 developer preview SDK. First, to pull a list of all instances with a certain tag, you can use the following:

    def testing
     # testing some tag code
     instancelist = @@ec22.describe_tags(
     filters: [
     {
       name: "key",
       values: ["SecurityStatus"]
     }
     ]
     )
     puts instancelist.to_h
    end  
    

    That shows everything with the SecurityStatus tag (instances and values, shown as a hash), but you could also filter on value. From there, you could pull the instance IDs with a map (you might want to further filter only on instances, but if you made it this far you can figure that out):

     instances = instancelist.map(&:resource_id)  
    

    Then you can pump those into a method to change the Security Group. Here’s the one from SecuritySquirrel that places an instance into a “Quarantine” group:

    def quarantine
      #this method moves the provided instance into the Quarantine security group defined in the config file.
      puts ""
      puts "Quarantining #{@instance_id}..."
      quarantine = @@ec22.modify_instance_attribute(instance_id: "#{@instance_id}", groups: ["#{@QuarantineGroup}"])
      puts "#{@instance_id} moved to the Quarantine security group from your configuration settings."
    end  
    

    Want to trigger some scanning? Here’s a code snippet to trigger a Qualys scan on an instance from a virtual appliance in your AWS account (keep in mind, I’m skipping a bunch of important pieces, like configuring the service and authenticating):

      instance_IP = instance_details.reservations.first.instances.first.private_ip_address
     timestamp = Time.new
     scan =(HTTParty.post("https://qualysapi.qg2.apps.qualys.com/api/2.0/fo/scan/",
          :basic_auth => @qualysauth,
          :query => { 
            :action => "launch",
            :scan_title => "SecuritySquirrel Scan at #{timestamp}", 
            :ip => "#{instance_IP}",
            :option_title => "Initial Options",
            :iscanner_name =>  "us-west-2" 
          },
          :headers => { "X-Requested-With" => "ruby httparty"}))
          puts "Launching Qualys scan named: SecuritySquirrel Scan at #{timestamp}"  
    

    Or want to change the CloudPassage Halo host firewall rules? Well, that code uses a pre-release of their Ruby SDK, so I’ll save it for another day (I’m not employed by them, but they let me play with the pre-release gem). These examples are all based on tags, but remember that if you use a tool like Chef or Puppet you can do the exact same thing based on what software is configured and running within the instance. For example, you could create a policy to quarantine all SSL when the Heartbleed vulnerability hit by closing off port 443 on any server running the vulnerable OpenSSL package, then remove the rule when it is updated to the patched version. I’m a bit rusty since I don’t code every day, but I could probably pop that code off in an hour or two, it isn’t that hard. Also my tool is designed for demos, and doesn’t run in the background, but clearly you could have it running all the time, adjusting things based on tags or even attributes of software deployed in the instance. Security automation is insanely powerful. It can completely automate complex security tasks, or, as in Andrew’s example, supplement manual activities and policy changes. Hopefully my snippets give you some ideas, and pop me over any good examples if you are willing to share.

  • DevOps resilience: going active-active with an existing application

    DevOps resilience: going active-active with an existing application

    What does it take to run an active-active architecture? It boils down to data, and how you interact with it. The most common disaster recovery strategies I encounter with folks are either “hope and pray” on a single location for running an application, or “we have a backup facility we can switch to in a disaster” with an active-passive configuration. This is a shame, as running an active-active architecture is not science fiction, and while it takes some work, for many business use cases the pros greatly outweigh the cons.

    The Devil’s in the Data

    If you work your architecture from the outside in, an active-active architecture is relatively easy to understand: you have more than physical datacenter or cloud region where you run your application, you have your application running in each, and you have traffic management to load balance users to the closest, fastest and most available location for them. Where folks get stuck is in the database. How do you make sure a change made in one location gets propagated to the others? What happens if a user “bounces” from one facility to the next and attempts to read back data they just wrote? How do you avoid race conditions where user A at location 1 and user B at location 2 make a conflicting change to the state of the application?

    The key to handling the scenarios above is understanding the business requirements of your data, and treating each type of data appropriately for an active-active architecture. If you try to “make all data available everywhere”, you’ll quickly run into either CAP theorem constraints or financial constraints on the high costs of this approach. Instead, let’s break the bigger problem down into smaller, more tractable problems.

    Three Classes of Data

    For each type of data your application accesses or produces, let’s classify it according to one of three types, using a banking metaphor for simplicity (and for easier explanation to your business peers):

    1) Is this data like a bank account balance? It changes frequently, and it’s incredibly important to make sure two conflicting changes don’t “break” the balance. We don’t want users A and users B making a withdrawal from two locations at the same time, thereby potentially overdrawing the account.

    2) Is this data like your address and mailing information your bank keeps for your account? It changes, but not frequently, and during a change, it’s OK if there’s some short period of time where the multiple locations that host your application are out of sync for this data. They’ll eventually gain consistency.

    3) Is this data like your banking statements? Once produced, they don’t change. They can be archived.

    Account balance. Mailing address. Historic statement. Three categories, and for each, we have a data replication strategy that can support active-active. By categorizing the data and treating each with a different replication strategy, we’re accomplishing two very important things:

    1) We’re being thoughtful about what data REALLY needs near realtime replication and race condition protection. By doing so, we’re reducing the amount of overall data that needs high priority replication. The less data to replicate, the better and more efficiently replication works.

    2) Rather than try to replicate all data with the same strategy, we’re willing to use the right tool in the toolbox for each type of data, which means we’ll save costs by using archival and eventually consistent strategies where they make sense.

    For each class, we’ll use an appropriate replication strategy:

    1) For “account balance” information, we’ll replicate changes as quickly as possible, and make sure our application is aware and capable of resolving change conflicts. This is certainly one of the hardest pieces to get right, but by reducing the scope of how much data we need to address with this strategy, the problem is much easier to solve than by trying to apply this strategy to all three categories of data.

    2) For “mailing address” information, we’ll use an eventually consistent replication strategy.

    3) For “historic statements” information, we may choose NOT to replicate this to all of our facilities. We may revisit some old assumptions on what’s important to save and what’s not, and decide to discard some of this information. We may choose to simply regenerate the statements should they be lost. We may choose to bulk replicate this data at certain times of the day.

    The Top Three Benefits

    After we migrate an application to active-active, we can expect three key benefits:

    1) Faster and more reliable failover in the event of a disaster. As opposed to an active-passive disaster recovery strategy, which depends on idle infrastructure coming up to full production speed in the event of a disaster, we avoid the risk of “we thought it was ready to handle production but something broke in between disaster recovery drills”. In an active-active scenario, we’re sending production traffic to each location all of the time, so we don’t have an “idle to full capacity” ramp-up problem; instead, we’ll have a “50% to 100%” (for two locations going to one) or “33% to 50%” (for three locations going to two” capacity challenge. We’ll need to make sure we always have spare capacity in an active-active configuration to handle the additional load during a failure.

    2) More flexibility for making changes to applications and infrastructure. During a site maintenance event, we can shift all traffic away from our facility, perform our maintenance, and then pull the traffic back, without downtime.

    3) Improved user experience. By connecting users with an application instance that is near where they are in the world, they’ll receive a better and faster user experience than if they were to be all sent to a single location.

    There are others, but those are the biggest three.

    The Top Three Drawbacks

    Of course, it’s not all sunshine and kittens, and it’s not without work and investment:

    1) You’re going to need spare capacity available, and be diligent and have conviction about maintaining a safe buffer for spare capacity. If you run two facilities each at 75% load, you’re going to have a big problem when one fails. Know your limits, and stick to them. Don’t fall behind in investing in this capacity.

    2) You’re going to need to change your application. If you were looking for a “drop in solution” to active-active, this is not it. You’re going to need to change your application. The most common scenario is you’re going from a single database instance on a single database technology to three separate data services that treat each type of data with the appropriate data management strategy. The most common problem is “I used to be able to JOIN these two tables to get a result, but one was an ‘account balance’ type, and the other was a “mailing address” type, and now they’re stored separately”. In this case, your application will need awareness to retrieve the data from each appropriate location, and “JOIN” in software.

    This sounds hard and time consuming, but there’s a great strategy for how to implement these changes. Presumably, your application accesses data using a core set of libraries (often an object relational mapping or ORM approach). In these libraries, you’ll want to make the changes for “what type of data is retrieved from where”. This will help minimize the impact to the rest of the application in terms of the interfaces to retrieve data, and keep the “data load balancing” functionality in one location in your codebase. Add in helper functions as appropriate to implement the JOIN functionality you need that previously was done in the database. Keep in mind, if you have a LOT of JOINs across categories of data, this strategy as a whole may not be viable for you.

    It’s highly recommended you take a dark architecture approach to these modifications. For some period of time after migrating data to a new strategy, have your application perform reads and writes to BOTH the legacy and new implementations for a single data interaction use case. Compare the values and results, and log/alert when different, while returning the legacy result. This gives you operational experience with the new data strategy while minimizing risk, since the application is still using the legacy implementation for all functionality. After you have confidence that the two approaches are functionally equivalent, you can start using the values from the new implementation, and turn down the legacy implementation. For more details on dark architecture, see here: http://gigaom.com/2013/06/20/making-it-change-less-scary-using-dark-architecture/

    3) You’re going to be running multiple database and data storage implementations. There’s a definite cost here in training your team, licensing, infrastructure, and having the monitoring and associated operational infrastructure required to run a dedicated strategy for each type of data. You’ll have to understand for your business whether the benefits outweigh the costs.

    There are others, but those are the biggest three.

    Your Next Step

    How important is uptime, infrastructure agility and user experience to your business? Are these investments worthwhile? Only you will be able to tell for your business. Once you quantify the value of uptime (e.g., how much money do you lose and damage to your brand is there during an outage?), the value of agility (e.g., how much faster could your team evolve your application if they had more flexibility to make changes anytime to production without affecting customers?), and the value of user experience (e.g., how much faster could your user experience be by splitting your infrastructure into multiple active-active locations, and how much would your users appreciate that improved experience?). Come up with those figures, and compare them to the technical investments required above, and you’ll have a pretty clear GO/NO-GO for evolving your application to active-active.

  • DevOps culture meets the SLA

    DevOps culture meets the SLA

    The goal of every SaaS provider should be 100% availability – it should be ingrained in their culture. But it’s far more common to see a series of nines. Numerically, these are clearly not the same value and culturally, they are a world apart. When you have an availability target that is less that 100%, you are explicitly tolerating some amount of downtime. It changes the requirements, the perspective of all those involved in delivering the application to your customers. When we think of the nines, I think we invariably fall into thinking about the differences between them – especially the technical differences. But to get to 100%, you need culture. A culture that is fanatically dedicated to achieving that goal.

    A review

    This is tired ground, but let’s take another look at what the most common SLA measurement is and what it means. System availability is measured as a ratio of the number of minutes of actual availability over some period of time – a week, a month, a quarter, a year. We usually talk in terms of the monthly SLA for a 30 day month. 99.9% availability is 43.8 minutes of downtime per month. 99.99% is 4.32 minutes per month, and 99.999% is just 25.9 seconds per month.

    What I find interesting about these numbers is the implication of trying to assure a higher SLA. You can achieve 99.9% with process and dedication. But that won’t get you to 99.99%. To achieve that, you need at least fail-over automation. To get the fifth nine, you need self-healing automation. Each of these numbers represents an order of magnitude increase in availability. Likewise, the effort and expertise required to attain that availability is greater.

    What’s your relationship to the SLA?

    When there is a disruption of service, the first obligation an organization has is to restore that service as quickly as possible with the least amount of risk. In mature organizations, there are typically automated and self-healing processes that provide a first line of defense for the most common types of failures. But there are always unanticipated problems that can manifest at any time. The expertise required to address those varies depending on the nature of the problem. It might require someone who primarily identifies as development, or dba or security. Everyone else attending to that problem is secondary to the key resource required to fix it. I have spent many hours feeling helpless during outage war rooms because I was not person with the specific skills required to solve the problem. When you expand this problem out, you begin to see that in order to prevent the outage from occurring again, you need that key resource to be thinking about outage/problem mitigation all the time, rather than in response to a specific incident.

    Sometime, during times of trouble like this, I’ll be approached and asked what we should do to address this problem. On many occasions, my answer has been, “we need to build a system that doesn’t have outages.” After an awkward silence, the asker begins to realize that the askee is actually quite serious. And I am. The problem is that to truly attain that goal, or something reasonably close to it, you need everyone involved at all times at some level.

    SLAs are meaningless

    About 15 years ago, I experienced a catastrophic failure in a name brand storage array I inherited by way of employment. It was one of those managed, completely redundant, fault-tolerant arrays that consumed a considerable amount of OpEx with a 99.5% availability in the SLA. When it inevitably failed (you knew that was coming, right?) and I looked up the contract, I found that we could not claim a material breach of performance unless the availability of the storage device fell below 85%. I was stunned. My business would be dead long before 85%.

    Likewise, you can have a 99.95% month, but if the tiny amount of downtime happens to impact your biggest and most demanding customer, the SLA is immaterial. They become angry and are at risk as a customer because you have failed them. What I learned as an operations executive is that the commitment behind the SLA is far more important than the SLA itself. Put another way, you can make bad situations into positive ones if you handle them exceptionally well every time. That means ruthless dedication to uptime from everyone in the company, from rank-and-file individual contributors to the CEO, all the time, not just during the incident. If that really is a goal – and it’s infused into the culture of the company – everyone you interact with will see it and understand that commitment.

    There’s another artifact of SLAs that can be detrimental. When the focus is on the availability number, the target is too low. The goal of every SaaS provider should be 100% uptime. When your goal is 99.9%, you effectively have a budget of downtime that you can draw from while still exceeding your goal. Here’s an example from my own experience. I once had a service that required periodic downtime for maintenance. One month, I had incurred 30 minutes of downtime via an unplanned incident against that service – a service with a 99.9% SLA. We also had a software update planned for that month that was going to require 20 minutes of downtime. I had the update moved to the following month so that it wouldn’t blow the SLA for the current month. It didn’t make much sense at the time and didn’t feel very customer-centric either, but as Dr. Eliyahu M. Goldratt wrote in The Goal, “Tell me how you will measure me and I will tell you how I will behave.” I had a chance to hit the SLA for that month, so I did everything I could to hit it. Given that I deferred a software update that would have delivered new features to my customers, the choice I made seemed arbitrary and was counter to being a customer-focused organization.

    It takes a culture

    One truly exciting thing about the DevOps movement is that it advocates the spread of knowledge throughout the organization. From my selfish operations perspective, that means that individuals who formerly would only be contacted in the event of dire emergency now have a much more visceral connection to the availability and performance of their services. By being exposed to the operations experience, those formerly external parties now see challenges and problems. And most people in our industry love a good challenge. What I get from this exchange is the injection of new perspectives and a diverse and healthy debate about subjects that have long since been worn out living only in the realm of operations. To have the attention of that level and breadth of expertise is a fantastic opportunity for me to evangelize availability and performance. More importantly, if all that talent is super motivated to achieve 100% uptime, performance and availability are going to follow. Anything less is a compromise and I don’t want to compromise on behalf of my customers. I want them delighted 100% of the time.

  • The DevOps master process: design

    The DevOps master process: design

    In speaking, counseling and talking to people with regard to DevOps, I often frame the discussion in terms of the DevOps master process. While all of the parts of the master process are important, one aspect very often is overlooked. I call this often overlooked aspect the design stage.  The design stage includes the gathering and visualization of requirements. If we assume that our goal is to deliver higher quality and more consistent and resilient deployments, then we must include expectations for how the software should operate and perform.

    When we are able to streamline and optimize the DevOps master process we ensure maximum velocity and agility for delivery of IT services to the business. The importance of the master process is that it applies for both businesses that develop their own software as well as those that operate using only Commercial-Off-The-Shelf (COTS) and Software-as-a-Service (SaaS) products. Before discussing what is involved with design in more detail, let me take a moment to better define what I mean by the DevOps master process:

    1. Design

    2. Build

    3. Test

    4. Stage

    5. Deploy

    6. Operate

    In future articles I will explore each of these elements in greater depth. For this article I wanted to focus in on the design element specifically.

    To ignore the design aspect of the master process means that we have no linkage for what we are building or testing during the Dev phase and no concern for performance and service levels in the Ops phase. Simply because there are no tools that help to automate this aspect of master process, does not mean that it is not an important aspect to fulfilling our DevOps mission; it simply means that we need human intervention to ensure that our requirements are met at each stage of the process.

    However, this is not all a one-way street with requirements only dictating to operations. In the past, traditional requirements gathering often focused on function with very little thought given to performance or consumption. Through our new DevOps collaboration, we introduce operations into the requirements gathering phase, which means that we can capture critical elements, such as service levels and usage models, ensuring all work across the master process leads to meeting these operational requirements. This will, hopefully, limit the spike in support calls operations needs to respond to immediately after deploying new software into production.

    Perhaps the reason we don’t yet hear about the importance of this is because it’s difficult and the tools for requirements gathering and automation around testing against the requirements traceability matrices just don’t exist in the current popular DevOps tool chains. Dare I say, these have been completely ignored by the open source community? It would be understandable since open source communities grow out of interest more so than need and developers are more inclined to work tools and technologies that are interesting to their own efforts. Perhaps we will see better integration of automation of testing not only features but also service levels, from the enterprise DevOps software providers, such as IBM who are applying their Rational suite of tools toward DevOps initiatives.

    Regardless, to achieve optimal benefits from DevOps, we need to automate around the entire master process. This means that we must translate requirements into automatable tasks by linking them to specific unit tests and load stress testing and ensuring that the requirement is only satisfied once these tests have passed.

  • Meet Infrastructure as Code

    Meet Infrastructure as Code

    Maybe you have already heard the terms “Programmable Infrastructure” and “Infrastructure as code”.  But even if you have not, if you are interested in DevOps it’s something you need to know.   Most new lingo causes me to roll my eyes.  But in this case, it’s not just new lingo, it’s an entirely new concept. Besides making the nerdy feel cool when they say it, these terms describe a new idea of what an application consists of. (more…)

  • Bridget Kromhout’s “DevOps Against Humanity” goes viral

    Bridget Kromhout’s “DevOps Against Humanity” goes viral

    Many of you reading this have probably seen the tweet or one of the thousand or so retweets of Minneapolis based Bridget Kromhout’s “DevOps Against Humanity” twitter post. The tweet and attached pic are below:

    For those not familiar, DevOps against Humanity is a take off on “Cards Against Humanity” a group party game with black and white cards. The game is really simple. One player picks a black card that has a fill in the blank question. The other players than pick from their white cards  with what they think is the funniest answer. One of the white card answers is picked as the winner and the person submitting that white card gets a point.  There are some other options, but that is the basic game.

    The game and cards for Cards Against Humanity are released under a creative commons license, so you can print your own copies. You could support the creators by buying pre-printed sets as well. Also players are encouraged to create their own cards with new questions and answers.

    The DevOps Against Humanity variant as championed by Kromhout is a version with DevOps related questions and answers but still retaining the wicked sharp potential that the original game is known for. Bridget has set up a github site where you can download the pdf and images of what she has created, along with instructions on getting them printed.  There is also a link to a Google Doc with ideas for questions and answers. You can add your own to the list. Over time it would be great to see the list really expand.

    Sounds like a cool game to play at your next DevOps conference or gathering of like minded DevsOps folks. I guess you could play with non-DevOps folks, but what is funny to one group may not be as funny to the other.

    Just a reminder that Cards Against Humanity and by extension the DevOps Against Humanity are issued under a Creative Commons license so you cannot resell these.

  • An open letter to Jeff Knupp

    An open letter to Jeff Knupp

    Dear Jeff,

    First off, nice recovery. I commend your gesture to return to positivity and make what to my eyes, looks like a kind and altruistic offer to an underserved class of engineers. I personally sent your offer to all the female engineers on my team and got very good feedback from several of them.

    Some people will try to tear you down and criticize your offer to make a difference in the lives of female engineers as patronizing or sly self promotion, but I take you at your word and applaud your offer as both poised and graceful.

    Given your offer to one underserved population in IT, I thought you would be OK with some constructive feedback from one dev guy concerned with the welfare of IT workers to another and hope you take this feedback as it is intended, with the same eye towards a desire to be a force for positive change to the plight of all IT workers, everywhere. Please remember that this is just my perspective. I could be wrong, and often am. So, here goes:

    DevOps does not mean everyone has to do everything

    DevOps is not just for startups.

    I work in a big shop and we have all the sorts of disciplines you call out in your posting on how DevOps is killing the developer. We are also embracing the DevOps movement not to force anyone to take on roles they aren’t optimized for, but rather to inspire them to expand the boundaries they have set for themselves and others in an effort to make things better for everyone.

    We want QA professionals to know how to script and automate tests. This is good.

    We want operations staff to know how to script and automate changes. This is good.

    We want release managers who know how to triage problems, automate configuration management and instrument the throughput of the IT machine. This is good.

    We want developers and architects who know how to go deep when required to help solve performance problems. This is good.

    You may disagree and I respect your right to do so. I just am not that interested in hiring an IT professional of any type who says “That ain’t my job”. That’s not a guy or gal who gets things done or makes things better.

    I’m not saying that everyone has to be able to do every role or even that developers must branch out. I am saying that making others wrong for increasing their value prop is not likely to do anyone any good.

    Injustice anywhere is a threat to justice everywhere

    Using narratives of which type of IT worker is more important than another type of IT worker does us all harm. All of us. All men. All women. All IT workers. All Developers. All operations staff. All employees from any department of any company no matter the size.

    Now, I realize that this is a bold statement (even from the guy who wrote “Sharepoint is Crack and Microsoft is the Pusher”), and I have no problem admitting that you probably have no desire to harm any employee anywhere (Except for the guy or girl who invented IVRs. I can understand wanting to do bodily harm to whoever did that). I would ask that you consider a few ideas before writing me off as a troll.

    1) The schism upstream is the enemy to be fought, not the people downstream from you

    You may not be aware of the schism to which I refer, but I would argue that you are probably very familiar with it’s effects. The deep and distrusting divide that exists between enterprise IT and the customers of enterprise IT across the industry is the single most destructive and demoralizing artifact in the lives of IT employees everywhere.

    Have you ever worked in a shop where IT employees are casually referred to as “technical” while marketing, product and design employees are referred to as “creative”. Have you ever been exposed to a shop where the developers were treated and talked about like they were worker insects who’s job it was to keep their mouths shut and code what they’ve been told to?

    If you have somehow managed to have an IT career and avoided exposure to this phenomena that seems to pervade so many (but not all) IT shops around the world, I would ask you to pull back and reflect for minute. Spend some time and have some gratitude for the fact that you have not been either witness to, or been the subject of the message: “we are the so and so team and you are not on our team so you have no say IT boy, go back to your nerd closet and build what I tell you to build”.

    If you are familiar with this idea that robs developers everywhere of the right to claim their creativity and basic need for self realization through recognized craft, then I ask you this: Aren’t you all sick and tired of the completely loathsome ideas that we are not “creative” because we are IT? Doesn’t that make you want to throw up? Don’t you want to go postal when you hear people push developers down the collectively agreed to illusion, referred to by you as the “totem pole”, that creates a veritable corporate caste system?

    I would run out of my word limit if I were to try and cite the publications that point out that silos are the prime roadblock to unlocking business potential, but I will point you to a few modern classics which make the case much better than I ever could:

    • The Advantage: Why Organizational Health Trumps Everything Else In Business by Patrick Lencioni, best known for The Five Dysfunctions of a Team: A Leadership Fable
    • Silos, Politics and Turf Wars: A Leadership Fable About Destroying the Barriers That Turn Colleagues Into Competitors also  by Lencioni
    • Orbiting the Giant Hairball: A Corporate Fool’s Guide to Surviving with Grace by Gordon MacKenzie

    This isn’t just about IT workers. This is about every employee everywhere who has their role demeaned and belittled by another. There are three basic roles inside of most companies in the products and services business. Sell the work, do the work, support the work. One thing in common to them all is that if any of the three roles don’t get done well, then everyone is at risk of not being paid.

    We are all necessary and none of us is sufficient at any real corporate scale. The moment we allow division (i.e. the totem pole) between ourselves, we have ceded the ground that other people upstream from us or up the hierarchy from us have the right to demean whomever they want to.

    You can’t possibly eliminate injustice via injustice

    2) The schism will always exist if we reinforce it

    Being of service to another is an honor and all who claim to want to improve the lives of workers anywhere, should hold themselves accountable to not belittle the contributions of others.

    Every single employee has the potential to be innovative and creative in their work. We constantly create and reinforce the concept of commodity items and jobs and then are paradoxically shocked and surprised by:

    • how the unmotivated and unengaged masses of people in jobs frustrate us with their lack of passion for doing a job well
    • how new people constantly create a “new and improved” version of something perceived to be commoditized forever and then dominate the market because they believed it was possible to reinvent a new level of quality

    Innovation is not a department. Creative is not a department. Aren’t you all sick and tired of narratives of ownership and control from other teams who claim to own these abstract qualities when they are delivered at you?

    When we the development community adopt the same basic ideas, we allow it to continue.

    If you want to world to be different, than you must have the courage to be willing to be one of the first ones who fully live in that new different world.

    What world are you working towards?

    3) How will you know your ideas and contributions are better if you set up barriers that make it harder for others to contribute?

    I am privileged to lead one of the best dev teams in the world. I am privileged to work in a shop that is committed to breaking down the barriers to collaboration and cross-pollinating innovation. To keep us all all at our best, we have to be brave enough to say that our dev team and our creations can more than likely withstand any criticism or contribution from any team up or down stream and if they don’t because some other idea was better, so be it.

    I won’t take the cheap and easy shot and ask “Are you afraid that the outside idea is better and that you will seem weak for having a lesser idea?”, because I doubt that this is true. I honestly think this kind of resistance has roots in the human condition itself. It probably has more to do with fear of the clock running out or the possibility of lesser ideas winning the day than of fear of being shown either as “weak” or “not as smart”. With these dreadful possibilities in mind, I would ask you consider this: The way to maximize the likelihood of these undesirable outcomes is by allowing others to say we are closed off to outside ideas. This is what leads to goat-rodeos, witch hunts and ultimately those with more authority than we have removing the little authority and control we actually have.

    Let the best ideas win, not the ones made by those with the authority backed up by a title.

    If you truly are looking for the best and most game changing ideas to help your enterprise, be aware that the richest source of these ideas will inevitably come from people who’s blinders have not been firmly affixed by the discipline they are embedded in. This is a widely accepted historical truth laid out by Thomas Kuhn in the timeless masterpiece The Structure of Scientific Revolutions and if you doubt me, remember that Albert Einstein was a patent clerk when he gave humanity the theory of relativity.

    I’m not trying to make you out to be a bad guy or anything close to one. In fact I praise you for having the guts to articulate a fairly controversial point of view that you believe in.

    When I write this letter, I’m not fighting for or against DevOps in particular (even though I am DevOps fan). I’m fighting for the end to the schism between Business and IT.

    I’m fighting for the end of narratives of control and authority.

    I’m fighting for the end of the un-empowering messages that authority gives to individuals every day.

    If you want to know why enterprise IT employees everywhere are asked questions from their business partners that display lack of trust over and over and over; This is why…because the schism exists.

    If you want to know why enterprise IT can’t complete things it starts; This is why…because the schism exists.

    If you want a world where people trust IT employees expertise and allow IT shops to complete work without questions of minutia that spawn from lack of trust, we must not embody what we oppose – narratives of authority and ownership combined with an “us vs them” mentality.

    I don’t say any of this to scold you Jeff. I say this as an attempt to call upon you to join us . Help us tear down the walls. We need more help. We need you.

    Respectfully,

    Stephen Fishman – Creative IT Professional

  • The Push to Make Ops More Social

    The Push to Make Ops More Social

    Given that social coding is a thing, what about the other side? Operations has the same image of lone wolves solving problems in the dark. The culture of on-call is built around the concept that when you’re holding the pager, you’re the one on the front lines, so that everyone else can sleep.

    Social operations, the ops in DevOps, is going to be about the same things that made social coding important: knocking down walls between individuals; making the knowledge of a system or infrastructure explicit and available, in context. But social operations is more difficult in two major ways: operations problems are often multidisciplinary, and often temporal – meaning that you have to be social when the situation arises,not on your time!

    When I say that operations problems are multidisciplinary, I mean that solving a problem in an operational context might require a number of people with different domain expertise. The solution is going to be much more complex and may not be portable to other people. Fixing something that’s wrong with a puppet script inside of our own infrastructure is fundamentally different than contributing a bug fix back to an open source project, or otherwise contributing to an open development environment.

    And when I say that operations problems are temporal, I mean that many of the forms of social engagement we’re getting used to in a social coding environment – forums, pull requests, etc. – are eminently useless when you’re in a firefight. When something’s wrong enough that you’re being paged at 2am, waiting for someone to respond to a comment thread on a pull request isn’t an option. You have to be able to get the right eyes on the problem, in a timely manner, if you want to solve infrastructure problems faster.

    So, how can we make the operations side of DevOps social? What parts of “social” are useful in the context of an operations issue?

    Opening up the knowledge of your team is a good place to start. Having the ability to get people involved is another good thing. Basically, what can we do to make it so that you know what you need to know, whether that’s how a problem was solved in the past, who’s available to get help, or what else is going on. That’s what we’re trying to figure out at VictorOps.

  • DevOps and Change Management

    DevOps and Change Management

    In “The Phoenix Project”, the character Erik introduces the underpinning principles of DevOps as “The Three Ways”.  Let’s talk about the First and Third Ways and how change management works with DevOps, but first let’s spend a few minutes talking about baseball caps.

    Last weekend I was looking for my Atlanta Brave’s baseball cap and found three as I expected for a family of three with two adults and one child.  Since they all looked alike, I compared sizes to figure out which one was mine and was surprised to find they were all the same size!  How can one size fit three very different sized people?  The answer was simple: They have an adjustable strap.  Let’s use this idea to explore the relationship between DevOps and change management.

    One size fits all with an adjustable strap

    IT is being asked to iterate faster and keep our systems highly available while doing so.  Those two ideas seem at odds: How do we fail fast and keep our infrastructure highly available?  Change management can help!

    Many people I speak with view change management as an anathema, as being incompatible with the principles DevOps.  The word “onerous” comes up a lot.  Inflexible is also quite popular.  I understand the sentiment and would like to help the community see change management differently.

    I think of change management as a continuum spanning rigorous to mild, heavyweight to lightweight.  Just like the baseball cap with an adjustable strap, change management can be adjusted for as needed.

    The First Way

    When describing the First Way, Erik invokes Deming’s quote, “appreciation for the System”.  In “The Top 11 Things to Know about DevOps” Gene Kim refines the idea as, “seeking to achieve profound understanding of the system”.  In my article on devopsy.kinsta.cloud titled “Systems of Things”, I assert that understanding the system is required before improving the system.  It seems pretty clear the First Way is about understanding the system we want to improve.  It is also about delivering business value to the customer.

    The Third Way

    The Third Way is about encouraging experimentation and building a culture that rewards risk as a way to speed maturation.  Rajat Bhargava expands upon that idea in his article, “DevOps Demands we Fix Twice” where he asserts that we should release quickly to speed product maturation.  It seems pretty clear to me that the Third Way is about speeding maturation of the features and functionality that deliver value.  The Third Way suggests we iterate faster, accepting some amount of risk as a trade-off.  I assert this risk is solely focused on features and functionality and not infrastructure.

    Tighter for infrastructure and looser for features

    After reading “The Goal” I began to think of code, the features and functionality that is ultimately delivered to the customer, as the material that flows through the factory and IT infrastructure as the machinery in the factory.   The machinery factories only exist to process material.  IT infrastructure is analogous to machinery; it only exists to allow features flow through to customers.

    In a well-run manufacturing facility the material flowing through the factory and maintenance on the machinery are both governed by well-defined and well-understood processes.  These processes are the analog to IT change management.

    The machinery should only be down for maintenance at very specific times.  At all other times the machinery should be ready to produce widgets.  The same is true for the Ops part of DevOps; Ops exists to allow feature and functional to flow through to the customers.  Ops should endeavor to keep the systems up and available for Dev to push code.

    Practical Application

    In practical terms, this means that infrastructure changes and code releases should be treated very differently.  The frequency and timing of code releases should be decoupled from the frequency and timing of infrastructure changes.  Infrastructure changes should not impact code releases and vice versa.  Infrastructure changes should come with a high amount of rigor while feature functionality should come with much less.

    A single change management process accomplishes this by:

    • Decoupling code releases and infrastructure changes when possible
    • Allow the frequency and timing of code release to be independent of infrastructure changes
    • De-conflicting infrastructure changes and code releases

    I think my colleague Stephen Fishman summed it up nicely, “The mission of Change Management… is to simultaneously increase the frequency of intentional change executions while reducing the duration and impact of interruptions for services provided….”

    Rent to own

    It can be tough to see change management and DevOps as compatible.  Instead of agreeing or disagreeing, just try the concept on for a while.  Rent the idea (it’s free) that change management rigor is a continuum, an adjustable strap and see how if it works for you.

  • SecDevOps: Security Automation By Example – The Firewall Change

    SecDevOps: Security Automation By Example – The Firewall Change

    Security Automation By Example

    The Firewall Change

    Just when you thought DevOps was the new black, along comes SecDevOps.  Yes folks, like most things in life, the new cool is already here.  Before I move on to trying to explain SecDevOps, please accept my mea culpa because for many people DevOps is yet to be clearly defined.  I can imagine your frustration that I’m going to try and explain a new term based on a loosely defined term.

    I’m not going to try and provide a complete definition of SecDevOps, but instead I’d like to provide what I believe to be important attributes of the SecDevOps movement.  In this post, lets discuss security automation and use a firewall change process as an example.

    One important attribute of SecDevOps is heavy use of security automation.  In particular when I think security automation, I tell people to use IFTTT thinking.  Stolen from the service whose tagline is “put the Internet to work for you”, I explain security automation simply as “if this, then that”.  Its a highly simplistic, but poignant way to imagine ways in which one would automate workflow.

    A simple act of changing a firewall has huge potential for security automation.  The SecDevOps firewall change process goes beyond changing a few lines in a config, but also provides an independent and automated verification process that most organizations don’t perform today.

    Consider what happens when a firewall policy is changed.  First the policy change request is reviewed and approved.  Then the policy change request is assigned and acted upon.  Many times, but not always, the change is verified by another person through configuration review.

    Security automation takes this process one step further.  Imagine when a firewall is changed that a security audit event is automatically logged.  Next, an automation component notices the firewall change and automatically spins up a remote scanning instance in a cloud provider.  Next, that scanning instance is contacted using APIs to perform a remote network scan.  The scan completes, results are exported and compared with prior scan results automatically.  Finally, the automation component automatically updates the firewall change request in a ticketing system showing the scan results.  We have now completed an automatic, independent and autonomous firewall change verification.   That simplified yet powerful workflow is security automation and completes what many people refer to as a closed loop.

    The new SecDevOps method for a firewall change not only provides a closed loop process, but does so by automating the verification steps.  Joining traditional security processes with new API driven cloud services is a true force multiplier that many organizations will soon wish they had.

    In upcoming posts, I’ll also discuss some of the other important components of what I believe make up the new black of SecDevOps.