Tag: devops

  • Security automation with DevOps: show me the code!

    Security automation with DevOps: show me the code!

    Last week Andrew Storms put up a good post hinting at the promise of security automation in [SecDevOps: Security Automation By Example – The Firewall Change]. He included an example of automating a series of actions when a firewall rule is changed. It’s a good article, although I’m increasingly convinced there’s no such thing as SecDevOps. In my book, it’s all DevOps, but that’s fodder for another post (when I’m not battling a stomach bug). However, what Andrew describes is more of what I consider an automated assist. It isn’t necessarily full automation, since it triggers on a manual firewall rule change. Ideally we rarely manually change a firewall (or Security Group) rule, and rely more on self-configuring based on policies. Yeah, I know, the usual analyst BS, so here’s a bit of process, and a bit of code. Let’s approach this differently. Take Andrew’s process, but let’s have the security group or host firewall self configure based on the asset. My example is going to be Amazon Web Services specific, since I have some code snippets to show off how it works. (Sorry, I don’t have the time or intestinal fortitude to write out all the code today; seriously, where did this bug come from?!?) There are two techniques I generally see forming the core of dynamic Security Group policy enforcement:

    1. Tags
    2. Configuration Management integration

    Some cloud services, like AWS, support tags at the object (in this case, instance) level. These tags can tie internally to policies while also being consumable for external tools. Here is a policy I use that restricts access to an object if the “SecurityStatus” tag is “IR” (for “Incident Response”):

    {  
      "Version": "2012-10-17",  
      "Statement": [  
        {
          "Action": "*",
          "Condition": {  
            "StringEquals": {
              "ec2:ResourceTag/SecurityStatus": "IR"
              }
              },
              "Resource": [
              "*"
                          ],
              "Effect": "Deny"
          }
          ]  
        }         
    

    I could apply that tag manually or automatically, and in either case the instance can no longer be managed by any account this policy is applied to. Now this doesn’t help us change firewall/Security Group rules, since you can’t manage those directly in EC2 based on tags (but I had the policy handy to demo, so there you go). For that to work, you need to write your own code to scan for tags and then take actions. I don’t have exactly the right code examples on hand. but here are some snippets to get you close. Most of these are cribbed from my SecuritySquirrel proof of concept on GitHub. They are all in Ruby, using the AWS 2.0 developer preview SDK. First, to pull a list of all instances with a certain tag, you can use the following:

    def testing
     # testing some tag code
     instancelist = @@ec22.describe_tags(
     filters: [
     {
       name: "key",
       values: ["SecurityStatus"]
     }
     ]
     )
     puts instancelist.to_h
    end  
    

    That shows everything with the SecurityStatus tag (instances and values, shown as a hash), but you could also filter on value. From there, you could pull the instance IDs with a map (you might want to further filter only on instances, but if you made it this far you can figure that out):

     instances = instancelist.map(&:resource_id)  
    

    Then you can pump those into a method to change the Security Group. Here’s the one from SecuritySquirrel that places an instance into a “Quarantine” group:

    def quarantine
      #this method moves the provided instance into the Quarantine security group defined in the config file.
      puts ""
      puts "Quarantining #{@instance_id}..."
      quarantine = @@ec22.modify_instance_attribute(instance_id: "#{@instance_id}", groups: ["#{@QuarantineGroup}"])
      puts "#{@instance_id} moved to the Quarantine security group from your configuration settings."
    end  
    

    Want to trigger some scanning? Here’s a code snippet to trigger a Qualys scan on an instance from a virtual appliance in your AWS account (keep in mind, I’m skipping a bunch of important pieces, like configuring the service and authenticating):

      instance_IP = instance_details.reservations.first.instances.first.private_ip_address
     timestamp = Time.new
     scan =(HTTParty.post("https://qualysapi.qg2.apps.qualys.com/api/2.0/fo/scan/",
          :basic_auth => @qualysauth,
          :query => { 
            :action => "launch",
            :scan_title => "SecuritySquirrel Scan at #{timestamp}", 
            :ip => "#{instance_IP}",
            :option_title => "Initial Options",
            :iscanner_name =>  "us-west-2" 
          },
          :headers => { "X-Requested-With" => "ruby httparty"}))
          puts "Launching Qualys scan named: SecuritySquirrel Scan at #{timestamp}"  
    

    Or want to change the CloudPassage Halo host firewall rules? Well, that code uses a pre-release of their Ruby SDK, so I’ll save it for another day (I’m not employed by them, but they let me play with the pre-release gem). These examples are all based on tags, but remember that if you use a tool like Chef or Puppet you can do the exact same thing based on what software is configured and running within the instance. For example, you could create a policy to quarantine all SSL when the Heartbleed vulnerability hit by closing off port 443 on any server running the vulnerable OpenSSL package, then remove the rule when it is updated to the patched version. I’m a bit rusty since I don’t code every day, but I could probably pop that code off in an hour or two, it isn’t that hard. Also my tool is designed for demos, and doesn’t run in the background, but clearly you could have it running all the time, adjusting things based on tags or even attributes of software deployed in the instance. Security automation is insanely powerful. It can completely automate complex security tasks, or, as in Andrew’s example, supplement manual activities and policy changes. Hopefully my snippets give you some ideas, and pop me over any good examples if you are willing to share.

  • Meet Infrastructure as Code

    Meet Infrastructure as Code

    Maybe you have already heard the terms “Programmable Infrastructure” and “Infrastructure as code”.  But even if you have not, if you are interested in DevOps it’s something you need to know.   Most new lingo causes me to roll my eyes.  But in this case, it’s not just new lingo, it’s an entirely new concept. Besides making the nerdy feel cool when they say it, these terms describe a new idea of what an application consists of. (more…)

  • An open letter to Jeff Knupp

    An open letter to Jeff Knupp

    Dear Jeff,

    First off, nice recovery. I commend your gesture to return to positivity and make what to my eyes, looks like a kind and altruistic offer to an underserved class of engineers. I personally sent your offer to all the female engineers on my team and got very good feedback from several of them.

    Some people will try to tear you down and criticize your offer to make a difference in the lives of female engineers as patronizing or sly self promotion, but I take you at your word and applaud your offer as both poised and graceful.

    Given your offer to one underserved population in IT, I thought you would be OK with some constructive feedback from one dev guy concerned with the welfare of IT workers to another and hope you take this feedback as it is intended, with the same eye towards a desire to be a force for positive change to the plight of all IT workers, everywhere. Please remember that this is just my perspective. I could be wrong, and often am. So, here goes:

    DevOps does not mean everyone has to do everything

    DevOps is not just for startups.

    I work in a big shop and we have all the sorts of disciplines you call out in your posting on how DevOps is killing the developer. We are also embracing the DevOps movement not to force anyone to take on roles they aren’t optimized for, but rather to inspire them to expand the boundaries they have set for themselves and others in an effort to make things better for everyone.

    We want QA professionals to know how to script and automate tests. This is good.

    We want operations staff to know how to script and automate changes. This is good.

    We want release managers who know how to triage problems, automate configuration management and instrument the throughput of the IT machine. This is good.

    We want developers and architects who know how to go deep when required to help solve performance problems. This is good.

    You may disagree and I respect your right to do so. I just am not that interested in hiring an IT professional of any type who says “That ain’t my job”. That’s not a guy or gal who gets things done or makes things better.

    I’m not saying that everyone has to be able to do every role or even that developers must branch out. I am saying that making others wrong for increasing their value prop is not likely to do anyone any good.

    Injustice anywhere is a threat to justice everywhere

    Using narratives of which type of IT worker is more important than another type of IT worker does us all harm. All of us. All men. All women. All IT workers. All Developers. All operations staff. All employees from any department of any company no matter the size.

    Now, I realize that this is a bold statement (even from the guy who wrote “Sharepoint is Crack and Microsoft is the Pusher”), and I have no problem admitting that you probably have no desire to harm any employee anywhere (Except for the guy or girl who invented IVRs. I can understand wanting to do bodily harm to whoever did that). I would ask that you consider a few ideas before writing me off as a troll.

    1) The schism upstream is the enemy to be fought, not the people downstream from you

    You may not be aware of the schism to which I refer, but I would argue that you are probably very familiar with it’s effects. The deep and distrusting divide that exists between enterprise IT and the customers of enterprise IT across the industry is the single most destructive and demoralizing artifact in the lives of IT employees everywhere.

    Have you ever worked in a shop where IT employees are casually referred to as “technical” while marketing, product and design employees are referred to as “creative”. Have you ever been exposed to a shop where the developers were treated and talked about like they were worker insects who’s job it was to keep their mouths shut and code what they’ve been told to?

    If you have somehow managed to have an IT career and avoided exposure to this phenomena that seems to pervade so many (but not all) IT shops around the world, I would ask you to pull back and reflect for minute. Spend some time and have some gratitude for the fact that you have not been either witness to, or been the subject of the message: “we are the so and so team and you are not on our team so you have no say IT boy, go back to your nerd closet and build what I tell you to build”.

    If you are familiar with this idea that robs developers everywhere of the right to claim their creativity and basic need for self realization through recognized craft, then I ask you this: Aren’t you all sick and tired of the completely loathsome ideas that we are not “creative” because we are IT? Doesn’t that make you want to throw up? Don’t you want to go postal when you hear people push developers down the collectively agreed to illusion, referred to by you as the “totem pole”, that creates a veritable corporate caste system?

    I would run out of my word limit if I were to try and cite the publications that point out that silos are the prime roadblock to unlocking business potential, but I will point you to a few modern classics which make the case much better than I ever could:

    • The Advantage: Why Organizational Health Trumps Everything Else In Business by Patrick Lencioni, best known for The Five Dysfunctions of a Team: A Leadership Fable
    • Silos, Politics and Turf Wars: A Leadership Fable About Destroying the Barriers That Turn Colleagues Into Competitors also  by Lencioni
    • Orbiting the Giant Hairball: A Corporate Fool’s Guide to Surviving with Grace by Gordon MacKenzie

    This isn’t just about IT workers. This is about every employee everywhere who has their role demeaned and belittled by another. There are three basic roles inside of most companies in the products and services business. Sell the work, do the work, support the work. One thing in common to them all is that if any of the three roles don’t get done well, then everyone is at risk of not being paid.

    We are all necessary and none of us is sufficient at any real corporate scale. The moment we allow division (i.e. the totem pole) between ourselves, we have ceded the ground that other people upstream from us or up the hierarchy from us have the right to demean whomever they want to.

    You can’t possibly eliminate injustice via injustice

    2) The schism will always exist if we reinforce it

    Being of service to another is an honor and all who claim to want to improve the lives of workers anywhere, should hold themselves accountable to not belittle the contributions of others.

    Every single employee has the potential to be innovative and creative in their work. We constantly create and reinforce the concept of commodity items and jobs and then are paradoxically shocked and surprised by:

    • how the unmotivated and unengaged masses of people in jobs frustrate us with their lack of passion for doing a job well
    • how new people constantly create a “new and improved” version of something perceived to be commoditized forever and then dominate the market because they believed it was possible to reinvent a new level of quality

    Innovation is not a department. Creative is not a department. Aren’t you all sick and tired of narratives of ownership and control from other teams who claim to own these abstract qualities when they are delivered at you?

    When we the development community adopt the same basic ideas, we allow it to continue.

    If you want to world to be different, than you must have the courage to be willing to be one of the first ones who fully live in that new different world.

    What world are you working towards?

    3) How will you know your ideas and contributions are better if you set up barriers that make it harder for others to contribute?

    I am privileged to lead one of the best dev teams in the world. I am privileged to work in a shop that is committed to breaking down the barriers to collaboration and cross-pollinating innovation. To keep us all all at our best, we have to be brave enough to say that our dev team and our creations can more than likely withstand any criticism or contribution from any team up or down stream and if they don’t because some other idea was better, so be it.

    I won’t take the cheap and easy shot and ask “Are you afraid that the outside idea is better and that you will seem weak for having a lesser idea?”, because I doubt that this is true. I honestly think this kind of resistance has roots in the human condition itself. It probably has more to do with fear of the clock running out or the possibility of lesser ideas winning the day than of fear of being shown either as “weak” or “not as smart”. With these dreadful possibilities in mind, I would ask you consider this: The way to maximize the likelihood of these undesirable outcomes is by allowing others to say we are closed off to outside ideas. This is what leads to goat-rodeos, witch hunts and ultimately those with more authority than we have removing the little authority and control we actually have.

    Let the best ideas win, not the ones made by those with the authority backed up by a title.

    If you truly are looking for the best and most game changing ideas to help your enterprise, be aware that the richest source of these ideas will inevitably come from people who’s blinders have not been firmly affixed by the discipline they are embedded in. This is a widely accepted historical truth laid out by Thomas Kuhn in the timeless masterpiece The Structure of Scientific Revolutions and if you doubt me, remember that Albert Einstein was a patent clerk when he gave humanity the theory of relativity.

    I’m not trying to make you out to be a bad guy or anything close to one. In fact I praise you for having the guts to articulate a fairly controversial point of view that you believe in.

    When I write this letter, I’m not fighting for or against DevOps in particular (even though I am DevOps fan). I’m fighting for the end to the schism between Business and IT.

    I’m fighting for the end of narratives of control and authority.

    I’m fighting for the end of the un-empowering messages that authority gives to individuals every day.

    If you want to know why enterprise IT employees everywhere are asked questions from their business partners that display lack of trust over and over and over; This is why…because the schism exists.

    If you want to know why enterprise IT can’t complete things it starts; This is why…because the schism exists.

    If you want a world where people trust IT employees expertise and allow IT shops to complete work without questions of minutia that spawn from lack of trust, we must not embody what we oppose – narratives of authority and ownership combined with an “us vs them” mentality.

    I don’t say any of this to scold you Jeff. I say this as an attempt to call upon you to join us . Help us tear down the walls. We need more help. We need you.

    Respectfully,

    Stephen Fishman – Creative IT Professional

  • Does DevOps spell death for Line-of-Business (LOB) Applications?

    Does DevOps spell death for Line-of-Business (LOB) Applications?

    Perhaps wrongly, “DevOps” has been associated with “new” and “that thing for the cool kids” and the “hipster developers”. This is used both as an excuse and as a source of animosity against the movement. But more subtly, I sometimes observe IT departments that acknowledge DevOps as a thing, but claim it wont work for their applications. I always pause and ask: What are the applications they are referring to, and are they correct?

    (more…)

  • DevOps storytime: MTTR vs. Goodheart’s Law

    DevOps storytime: MTTR vs. Goodheart’s Law

    I am not a fan of metrics. Wait. That’s not right. Let me try again. I am not a fan of singular metrics that stand by themselves. Nope. That’s still not right. I loathe singular metrics that stand by themselves as if they themselves are solutions to problems. Ok. That works. Let me explain.

    Last week, I told a tale of a secret passageway into DevOps called Mean Time To Resolution (MTTR) and explained how the introduction of this one tiny metric can magically open the door to all things DevOps. Given that MTTR and cycle time are inextricably linked, any shop that is serious about MTTR will sooner or later find itself using DevOps practices.

    Metrics + Crafty Humans = Doom

    This is very different than two weeks ago when I told a different tale. A tale of how the very origins and roots of DevOps gives the DevOps movement an opportunity to do what ITIL could not; Change the relationship between business and IT. In the telling of this tale, rife with philosophical terms like reductionism and formalism, I quoted a brilliant English economist, Charles Goodheart, who noted that “when a measure becomes a target, it ceases to be a good measure”. Goodheart discovered that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The layman’s explanation is that humans are crafty little buggers and will find ways to artificially hit any target because the quantitative systems can always be rigged.

    So what is a DevOps voyager to do? Introduce the metric, get the change, and then proceed to watch devops die a slow death? No, my fellow DevOps devotees, that shall not be our fate. We can avoid this fate by knowing that halfway down the path of the secret passageway known as MTTR, there is a fork!

    More Metrics to the Rescue!

    Instead of introducing and presenting a flat and un-nuanced view of MTTR by itself, we can get a little crafty ourselves and find an appropriate hedge against Goodheart’s lay and instead present a weighted view of MTTR with a few other key metrics to allow for a more sophisticated view of the health of the IT factory floor.

    Apply Weighted Averages – Weight the MTTR of defects and their associated resolutions by priority to intentionally skew the measurement towards the highest priority defects. Clinging to an absolute view of defect free software doesn’t work for a large code base with limited funding and time (aside from being scientifically impossible to achieve via godels proof)

    Track Inbound Defect Counts – Track and present the inbound defect count, keeping the inbound defect count low is important, but pushing it to zero is more likely to be indicative of a shop that is more concerned with perfection than throughput

    Track Outbound Defect Counts – Track and present the defect close count. keeping a negative inbound-outbound differential is a sign of a shop that is arcing in the right direction. Several weeks or months of positive differential is a warning sign that a quality mindset is slipping into dangerous territory.

    Track Backlog Defect Counts – Track and present the count of defects in the backlog. A backlog of zero can be incredibly hard to maintain in a large enterprise with a large code base. In my experience 20 and 50 have been ominous numbers. Any more than 20 defects in an application and the backlog starts to become a fearsome beast that can only be tamed by stopping the factory. Any more than 50 and the incremental additions of any new defect seem trivial.

    Track The Age Defects – Tracking the average age of defects in the backlog is critical for any presentation of MTTR. A spike in MTTR can actually be a good thing if it represents a wave of old defects being cleared from the back of the backlog.

    Resist The Urge to Simplify

    A five part metric begs the question of an balanced score card approach with a grading formula and despite my belief in the value of this tactic for availability and performance dashboards, I advise you to strongly resist the temptation to apply it here (otherwise, we are right back where we started with Goodheart’s Law. Damn you crafty humans!).

    If you decide to bring the magical metric of MTTR into your shop, this type of reporting may be one area where starting small is quite possibly worse than doing nothing. Without the nuanced view, true visibility into the health of the shop isn’t possible. Worse than that, a flat view of just MTTR has the potential to mask real problems and raise false ones.

    Screen Shot 2014-04-28 at 12.44.25 AM

    When you pull all of the metrics together in a single view, like in the above graphic, the real message becomes clear. The health of the factory floor isn’t tied up in a neat little bow by a single number. It isn’t even made perfectly clear by three, four or five numbers. The real health becomes clear in the trend. Is your shop doing better week over week? Month over month? Getting better every day is the name of the game, and MTTR (with a few other metrics) can be a lever that moves your IT world.

  • Security and DevOps showdown

    Security and DevOps showdown

    caution

    George Hulme’s article on proceeding with caution on DevOps implementation offers some advice from a few well known security experts. Article here

     

     

    vs.

    helpmeobiwankenobi Alan Shimel says today’s speed of business isn’t compatible with :go slow”. Security people have to understand that DevOps is our last best hope. But in order to have DevOps help us to security nirvana we need to change how we interact with the rest of our IT teams. Article here

  • DevOps: Security’s last best hope

    DevOps: Security’s last best hope

    Help us Obi Wan! The fact is that DevOps is security’s last best hope. The sooner the security industry realizes it the better it will be for everyone.

    I read George Hulme’s story today about proceeding with caution and going slow with DevOps. Of course it was no surprise to me that those cautioning to go slow were my fellow security industry professionals. What else do you expect, after all we are the “people who say no”.  At least the three folks George interviewed didn’t flat out say no to DevOps. They just advised caution.  The fact is that DevOps is security’s last best hope. The sooner the security industry realizes it,  the better it will be for everyone.

    Let’s be honest. The security industry for the most part has not been a stunning success in keeping our data, apps and infrastructure safe and secure. Some may even say it has been an abject failure. What do you expect though? The game is stacked against us. Almost by definition the entire security industry is by definition a reactive model. We are always one step behind, reacting to the newest moves of the bad guys.  To be a security professional is to know failure.

    What is worse, we are generally ignored until we fail and then the harsh, bright light of a security breach shines brightly in our eyes. Is it any wonder that so many security folks feel that they are not successful? Our budgets are never enough, we never have enough resources, no one ever does what we ask and not enough people take security seriously enough until after the doo doo hits the fan. Yes it takes a special kind of masochist to be a security person.

    It is not by happenstance that in Gene Kim’s book, The Phoenix Project, the poor bastard of the security guy goes out on a bender and comes back at the end having fallen off the wagon?

    For all of the aggravation of fighting an impossible to win fight how are security people viewed by the rest of the IT folks and others in the organization. I am sorry to say that all too often we are though of as the crazy person in the back of the room yelling no. Again as in Gene’s book, it gets so bad that people stop inviting us to their meetings. They know what we are going to say, NO, so what’s the point.

    There are more enlightened or maybe smarter security folks who don’t say no any longer. Now they say yes do it, but let’s go slow and make sure we understand it.  When asked why, these security folks have a built in scape goat. They blame compliance. Yes, f**king compliance. Yeah, that’s the ticket. We have to go slow because our compliance obligations require it.  I laughed out loud when I saw this raggedy old flag being hoisted in George’s article.

    Compliance is always the fallback of the security guys who say no.  I have seen many examples of this over the years. Two of my favorites are:

    1. The security guy who asks the IT manager or C-level guy “how do you look in stripes”, because if you do this without going slow and making sure we are compliant, we could all wind up in the slammer.

    Hell we didn’t send anyone to jail for bringing this country to the brink of economic ruin. Do you really think we are going to send someone to jail because by adopting DevOps we might blow a PCI requirement? Can we really do any worse than  Target?

    2. Walking around a US Army base talking to the security officer I asked him how he was dealing with the compliance issues around securing wireless networks on base. He replied that was easy, he forbid wireless networks on base, so they were in full compliance. Of course as he said this I was watching people unplug their WAPs and throw them under their desks as we walked by.

    Yes, security folk can fall back on compliance as the reason for our no, but it is tired and old. The same for the security person who are always promising the next magic bullet. If only they had the latest shiny, new budget busting trinket they could finally sleep well at night. Even if they get their new toy, it only turns up that the bad guys have found a new vector and we need the next silver bullet.

    No, I am afraid being the guys who says no or even the people who say OK, but let’s go slow is just not going to cut it.  The speed of business today at the enterprise or the startup is anathema to the “go slow” mantra.

    On the other hand, DevOps offers a new way. A way to change the way the game is played.  For so long I have heard security folk wax poetic about the possibility that security could be built in to the process.  Instead of being bolted on after the fact, wouldn’t it be great if it were built in by design? Wow, what a great thing that could be.

    DevOps has the potential to deliver on the built in dream. DevOps can make the developer more aware of security and even compliance issues while the code is being written. It can give the security team insight and input into the development planning and process.

    DevOps has the potential to break down the silo that separates security from the rest of IT. Just as it brings Dev and Ops together, DevOps can give security a seat at the table. Security can move from being the crazy madman in the back to a respectable member of the team.

    This security transformation won’t be easy though. It will demand a change in security’s attitude.  First and foremost, we have to change from being the people who say no or go slow, to the people who say “yes we can”.  We need to understand that many of these projects and initiatives are going to move forward with or without us. Better to get on the bus than to get run over by the bus.

    Next we have to do a better job of communicating our concerns or issues. Falling back on FUD or compliance is not cutting it. We need to communicate the risks involved, how we can mitigate the risk and understand that a business decision will be made as to what level of risk is acceptable and how that risk is going to be managed.  At that point we have given enough information for the business people to make a business decision. They make the decision, we get behind it, we don’t bitch bite and we move forward.

    Security can’t be the people who say no or go slow. When security starts to say yes we can and this is how we do it, security will be given the seat at the IT table that we deserve.

  • DevOps: Caution Ahead

    DevOps: Caution Ahead

    Despite the continued adoption of enterprise DevOps practices, some organizations, especially those in highly-regulated industries remain cautious about moving forward too quickly.

    “There’s no doubt that DevOps brings benefits for some organizations,” says Martin Fisher, director of information security at Atlanta-based WellStar Health System. “However, many pushing for DevOps underestimate the amount of technological and cultural change that is necessary to overcome before moving ahead to boldly, especially among those in security and regulatory compliance roles that are concerned with securing and auditing processes they see as they’re potentially losing control over,” Fisher says. “You can’t audit and secure what you can’t control,” he says.

    Eric Cowperthwaite, former chief information security officer at Providence Health and Services agrees. “There are some who want to move too fast. In many organizations the culture just isn’t yet there, especially where they are used to very rigid quality assurance and audit controls,” says Cowperthwaite, currently a VP at Core Security.

    It’s certainly not uncommon to find chief security officers in heavily regulated industries who are concerned that the move to DevOps is an excuse to cut corners and move more quickly by cutting out necessary oversight. “You’ll find if you have a developer that’s producing a dozen mistakes a day, if you poke at them about it then you’ll find out that they’re tired of the structure, of the bureaucracy,” he says. “DevOps need to be done in a mindful and thoughtful way,” Fisher says.

    Cowperthwaite agrees, and contends that in certain environments DevOps will be – should be – taken slowly. “You have legacy systems and very highly audited and controlled systems where DevOps is just not going to flow for some time,” he says.

    It’s not just the cultural shift, but also the shift in necessary for good governance capabilities to keep up, they contend. “Not all systems are geared to handle 200 commits a day, nor do they have the ability for reasonable tracking and accountability for that kind of speed. In fact, there are many legacy systems were it’s not even possible at all,” Fisher says.

    However, not that all companies that move toward DevOps are attempting to automate everything – at least not at first. But it is higher levels of automation most certainly seek. A survey recently conducted by automated server management software provider JumpCloud found that many companies are turning to DevOps to automate time consuming activity, such as deployment, patching, user management, log file analysis, and forensics activities.

    Some of the trepidation is warranted, to be sure, and some of it may be due to the media and industry focus on the outlier DevOp implementers. “I think, justifiably, a lot of the fear comes from the fact that a lot of the companies that have gotten the most press regarding DevOps are doing things that are radically different than what is going on within the bulk of IT work at traditional enterprises such as banks, healthcare, and organizations like that,” says David Mortman, chief security architect and distinguished engineer at Dell Software Enstratius.

    When it comes to enterprises with trepidations when moving toward DevOps, all of the interviewees we spoke with agreed the key is to move slow. But still move and develop the culture and the toolsets slowly. “Pick one system that is separate and use it as a concept,” Fisher advises. “The DevOps ‘true believers’ are given a hall pass to fully run their systems and group in a DevOps model. As they learn and improve you slowly assimilate into the broader environment,” he says.

    Mortman sees it in a similar way, but notes that it’s not just about technology and absolute speed, but also collaboration, culture, and relative speed. “How fast is fast for your organization? If you’re only making changes to your big medical records database every 18 months, and you switch to every three months – that’s faster. That’s more continuous deployment than you were doing before. No one has to jump right in,” Mortman says.

    “I started working with those sort of DevOps-y concepts back before I was aware of the term “DevOps,” Mortman adds. “I was calling it “Agile InfoSec” and “Agile Ops.” And with that mindset, even the most regulated, paranoid enterprise can embrace DevOps. Start with small, modestly achievable goals and build from there. Collaborate and more tightly integrate the teams.

  • DevOps developers; don’t be a DevGoof

    DevOps developers; don’t be a DevGoof

    There was a lot of brush back last week over Jeff Knupp’s post about how DevOps is killing the developer. Frankly I wasn’t shocked by Knupp’s opinion that developers “are the dentists in the dental office”. In my 30+ years of involvement in the IT industry my experience is that developers often think of themselves as the only smart person in the room. Often times that is the reason they are are the only person in the room, lol. The fact is many developers relate better to code than to people. For me all this talk about DevOps killing the developer is akin to saying power steering killed the driver. Just because you can do more with new technology, does not mean one dies, one just has to pivot and in this case become business social.

    Now the sociable part may be the problem. Sometimes I feel that DevOps is trying to create a love fest between Ops, Developers, QA, security and the rest of the IT stack. While they may not have to love each other, I do think they all need to be nice to each other as opposed to ignoring  or just going around each other, as was all too often done in the past. A common scenario I have seen for example is for developers to rollover Ops by stating this is mission critical for the business. These tactics will no longer work to let developers have their way.

    Developers need to realize they need to become a normal part of the business social structure. They will be required to talk and work with people, not throw tantrums when they don’t get their way. Left unchecked, certain developer personality types can sabotage your project, or worse, make themselves no fun to work with.  I am sure we have all met and unfortunately had to work with people like that. In my career, I’ve encountered virtually every personality type. In fact, I’ve also knowingly hired all of them. They know who they are; they are the developer that if challenged or worse accurately taken to task for lack of being part of the team; quit cause they can’t manage the confrontation, thus interpersonal stress. In the old days they would not come out of their house for weeks continually playing Leisure Suit Larry, if your not familiar with it Google it

    I remember one of my good friends telling me this story that sums this whole developer persona issue perfectly. This company, we will call DevGoof, was a well funded start up. Every quarter they held a meeting with all the employees to let them know how the company was doing. Most employees had some kind of equity play in the company, so everyone was incented and theoretically interested in company performance. Several quarterly meetings came and went and not one developer showed up. Eventually DevGoof management got concerned and called a mandatory meeting for all developers’.

    DevGoof management put out some pastries, juice and coffee for the brain trust. Then proceeded to tell them why these meetings were important to them and their fellow employees. Notably pointing out how it is not just about them but also about the rest of the DevGoof team. This meeting went on for about one hour and not one developer asked a question. So eventually this meeting was over and everyone went back to developing. The next morning,   about 50% of the developers quit and some in a not so professional fashion.

    The moral of this blog is the DevOps is going to force the development community to become more interpersonal within the business environment. This will lead to personal success within the new world of DevOps. The DevOps Developer/Engineer that can demonstrate leadership and compassion for team will be the one leading the DevOps way within the organization. Don’t be a Devgoof!

    If you would like say something; email parker@devops.com and don’t forget to comment here.

  • Merging creation with operations: facilitated collaboration enables continuous delivery

    Merging creation with operations: facilitated collaboration enables continuous delivery

    So much has been written lately about the challenge of improving IT agility in the enterprise. The best sources of insight on why this challenge is so difficult are the CIOs, application owners, ecommerce and release engineering executives, and VPs of I&O, grappling to change their organizations right now.

    At a conference I attended recently, I met two Fortune 100 IT executives from the same company: one the head of development and the other operations. Their story is emblematic of just how hard this is in the real world. As interesting background, both the development and operations leaders were childhood best friends, participated in each others’ weddings, and spend time together socially on an almost weekly basis – but by their own admission, even they couldn’t get effective collaboration and communication to work between their two organizations.

    The lesson learned from this example is that the DevOps collaboration and communication challenge cannot be solved by sheer will, desire or executive fiat. Instead, you must breakdown the barriers that inhibit collaborative behavior and facilitate new ways of communicating and working together. The old standbys of email, instant messaging, SharePoint sites, and conference calls don’t cut it.

    The challenge of two opposing forces: Dev and Ops

    Imagine yourself helping your children put together a new jigsaw puzzle. Each time you turn your attention to a specific piece, the kids reorganize what you have already completed and they add new pieces, but in the wrong places. For sure, three pairs of hands can be better than one, but they can also create chaos, confusion and significantly elongate the completion of the puzzle.

    The collaboration challenge in the DevOps movement is grounded in this analogy. How do you get multiple people working together across teams, locations, and time zones to build and get things deployed faster without chaos, confusion, and delay? How do you get these teams to speak the same language and collaborate together with a singular purpose when their priorities and motivations are so different?

    Faced with this challenge, it’s easy to see why many organizations have stayed in their comfort zone of ‘waterfall’ releases and keep the number of releases per year small. The issue is this method isn’t meeting the demands of the business, the market and competition. As a result, more and more business leaders are going around their IT organizations. Options like public cloud, SaaS, open source tools, skunk works IT and outsourcing are making it easier for them to control IT decisions and implementations within the business unit or department itself. 

    So let’s dive deeper to understand the two forces at the heart of the issue: development (focused on the creation or modification of applications to support a business need) and operations (delivering defined services with stability and quality). It appears these forces are working in opposition, but both groups are focused on doing what leadership asks of them.

    Developers tend to think their job is done once the application is rapidly created, but it’s not because no actual value has been delivered to the business until the application is operational and in production. Operations is severely disciplined when services experience performance and availability issues and they have come to learn that uncontrolled change is the biggest cause of these issues.  As a result, operations teams often believe their number one job is to minimize change to better control impact on performance and availability. This causes operations to be a barrier to the rapid change required to give the business the speed and agility it needs.

    Critical to enabling DevOps is an explicit recognition of this situation and the ability to link discrete phases of the application development and operations lifecycle to enabling ‘fast, continuous flow’ – from defining requirements, to architecting the design, to building the functionality, to testing the application, to deploying the application to both pre-production and production environments and to managing all the underlying infrastructure change required for the application to operate efficiently and effectively in all environments.

    Why current approaches don’t work

    There are several challenges in achieving this ideal.

    1. Developers hate to document (can you blame them?), and, when they do, their communication is in a context they understand, not necessarily empathetic with the language that operations speaks. The view from operations is that the documentation they receive is incomplete, confusing, and/or misleading. With the rapid pace of development, this challenge is getting worse with documentation becoming more and more transient as developers “reconfigure the puzzle” on the fly.

    2. Today’s operations teams typically take responsibility for production environments and their stability. That means there is usually a group wedged in between the two – the quality assurance (QA) team. QA’s job is to validate the application works as expected and often they require multiple environments for each release. This group is typically juggling multiple releases and is, in essence, working on and reconfiguring multiple puzzles at the same time. The challenge of keeping QA environments in sync with both in-process releases and production can be maddening (just talk to any QA leader and they’ll tell you first-hand). The documentation coming from development is inadequate, and the documentation coming from production is often no better, since most operations teams store much of their most current information about configurations in personal files or simply in their brains.

    3. The ad hoc handoffs from development to operations and QA take time away from development’s primary mission: creating applications that advance the business. Some suggest developers should operate and support what they code in order to reduce handoffs and the risk of information distortion or loss. A fundamental risk with this approach is opportunity cost. Does a developer really understand the latest and greatest technology available for infrastructure and how to flex and scale those technologies to fit the organization need? Do you even want them to or would you rather they be coding instead?

    4. Others have suggested that Operations move upstream and own all environments from dev to QA to production, and treat configuration and deployment scripts as code just like a developer would. This may sound like a good option, but it can create a constraint on your operations team and cause valuable intelligence to become hidden in scripts. A particular application deployment could have one or more software packages required and potentially hundreds of different configuration settings. If all that information is embedded in a script, how will other team members know this if they go to make a change to the underlying infrastructure to apply a security patch, upgrade an OS version, or any of the other changes made in IT every day?

    Real DevOps transformation doesn’t mean that you give everyone new jobs, instead, it’s about creating an environment where teams can collaborate together with a common language and where information is immediately available at the point of execution and in a context unique to each team.

    A better way forward

    In The Phoenix Project, written by DevOps thought leaders Kevin Behr, Gene Kim and George Spafford, the authors promote the need for optimizing the entire end-to-end flow of the discrete processes required for application delivery – applying principles as they achieved agility in the discrete manufacturing process.  

    Manufacturing in the 1980s resembled IT operations today, employing rigid silos of people and automation for efficiency and low cost, but this became a huge barrier to the agility, stability and quality the market demanded. They learned if you optimize each workstation in a plant, you don’t optimize for the end-to-end process. They also learned that if quality and compliance processes were ancillary to the manufacturing process, it slowed things down, drove up costs and actually decreased quality and compliance. 

    Successful manufacturers brought a broader view and optimized end-to-end flow rather than operate in a particular silo. They also brought quality and compliance processes inline with the manufacturing process. By addressing quality and compliance early in the cycle and at the moment that an issue occurred, cycle times decreased significantly, costs plummeted and quality and compliance increased dramatically.

    These same principles can be applied to IT resulting in:

    • faster time to market;
    • greater ability to react to competitive pressures;
    • deployments with fewer errors
    • continuous compliance with policies; and
    • improved productivity.

    DevOps can best be realized when IT operates in a social, collaborative environment that ensures all groups are working with a visual model in their context with the necessary information from downstream and upstream teams, as well as in collaborating with relevant experts at the moment when clarifications are needed or issues arise.

    To merge creation with operation, the core idea behind DevOps, requires a cultural change and new methods in which cross-functional teams are in a state of continuous collaboration as they deliver their piece of the puzzle at the right time, in the right way, in-context with the other teams in other silos. Operating something that never existed before requires documentation so that operations teams have the information they need to manage change with stability and quality.

    With more modern collaboration methods, self-documenting capabilities are now possible as development, release and operations do their respective jobs, including visualization of documentation with analytics and with the perspective and context each team needs to effectively do their job downstream. These types of capabilities will transform organizational culture and break down barriers to collaboration that impede agility, stability and quality.

    Is this simply nirvana, unachievable in the real world?  No. Manufacturing achieved the same results by applying these principles; the fundamental point being made in The Phoenix Project.

    The goal is not to write code or to keep infrastructure up and running, or to release new applications or to maintain quality and compliance. Instead, the goal is for IT to take the discrete silos of people, tools, information and automation, and create a continuous delivery environment through facilitated collaboration and communication. This will drive the cultural and operational transformation necessary to enable IT to respond to business needs with agility while ensuring operational stability and quality.