devopsy.kinsta.cloud

Category: Blogs

  • Improving Communication Through Autotmation

    Improving Communication Through Autotmation

    One of the great traits of an organization that adopts DevOps principles or even just an Agile methodology is that everything, and I mean everything, is more dynamic.  While each company will have a different level of fluidity in their processes, Clip Interactive’s source control and deployment processes are very fluid.  As such, we attempt to over communicate when it comes to these parts of the engineering process.

    The primary methods of communication the engineering team and rest of the company have adopted are chat/instant messaging and email.   There are several chat clients out there, all of which have their pros and cons and there is no one right answer except what works best for your team.  At Clip, we have chosen to use HipChat.  Other options include IRC, Flowdock, GTalk, Skype and the list goes on. HipChat was a good option for us because it is cross-platform, supports custom plugins and integrates with a wide variety of tools. 

    In HipChat, each time an engineer pushes code into source control, a build is started, finishes or deployment runs on a QA system a message gets posted to the “Engineering Feed”.  The messages for code pushes include the commit summary and link to the code diffs.  Messages regarding builds and deployments contain a link to their respective outputs.  More significant events, such as deployments to staging and production, get posted to the general chat room where more conversation happens and other non-engineers are more likely to be.  Those notifications are pulled “out of the noise” of the feed.  Additionally, these activities are typically chatted about more before they happened, so they fit more in the flow of conversation.  Last but not least, system notifications, including warnings, problems, recoveries and acknowledgements from Nagios, are sent into the “Operations” room.  To keep everyone informed and spread the knowledge, we post updates and resolutions to issues in the chat room.  All of these messages enable everyone to keep a pulse on the state of the system without being intrusive and should problems arise, to very quickly put together a chronology of events.  Aside from the automated items, there are additional rooms for in-depth feature discussion, production problems and sometimes just for fun.

    Another key form of communication is good old email.  Our build and monitoring systems sends the standard emails and text messages directly to the engineering team.  Each production deployment sends out the list of commit messages, naturally with links to the code diffs.  Since our deployment process is very fluid, this extra notification has gone a long way toward keeping everyone informed on the features being deployed into production.

    Going forward, we intend to add more automated communication around the product management process and system performance metrics.  These automated communications, when added to our normal cadence of meetings, have had a significant impact on getting us closer to the goal of improving transparency in and out of the engineering team, while not making people go out of their way gain value from the tools.

  • DevOps: It’s All Open Source’s Fault

    DevOps: It’s All Open Source’s Fault

    Blame DevOps on open source. After all, it was the open source movement that gave developers the idea that they were free to code, free to build applications and other software for their companies without needing prior approval. It was open source that crowned developers kingmakers. It was open source that gave developers the freedom to not only write code, but also manage it.

    It was open source that gave us DevOps.

    DevOps: Freedom To Get Stuff Done

    Traditionally, IT operations has been a specialized task, an exclusive one that largely controlled the code allowed to run within the enterprise. Given the fact that all software had to be purchased before it could be run, there was little risk of developers or anyone else introducing rogue software into the hallowed halls of the data center.

    Enterprises were able to enforce policies because all software had to start with Purchasing and end with Operations. There simply was no other viable way to get software into the enterprise.

    That is, until open source.

    Open source ruined the neat and tidy relationship between Purchasing and Operations. Suddenly software didn’t have to be purchased. It could simply be downloaded. As pressures mounted on lines of business to accelerate their pace of innovation, developers stepped up: “I have an open source project for that.”

    Operations fought back, trying to crimp open source adoption. They failed.

    There simply was too much pressure on developers to get stuff done. Once Amazon gave developers a home outside the data center to deploy their code, the battle was over and a new model for building and managing code was born. Dubbed DevOps, the movement gave developers end-to-end control over their creations.

    Since 2011, DevOps adoption has increased 26%, according to a 2013 survey by Puppet Labs, resulting in 50% application failures and the ability to recover up to 12X faster. The rise in DevOps also translates into the ability to ship code 30X faster. All of which is expressed in a separate CA Technologies survey of senior IT decision-makers, which found that improvements to the customer experience is by far the biggest reason enterprises are embracing DevOps. 

    Given these improvements, there’s no way we’re going back to Operations-focused IT.

    Blessing Or Curse?

    All of which is not to suggest that DevOps has been an unalloyed blessing. In my experience, developers are often unprepared or unwilling to take on the burden of managing their applications in production. Trained for years on the idea that they could build an application and dump it on Operations to manage, developers are discovering that the “Ops” in DevOps is real, and sometimes really a pain.

    For example, one big problem is that production environments and dev/test environments are typically very different. An application that works flawlessly in a small-scale test environment may miss a whole class of errors that only occur in the presence of the more complex production environment (e.g. problems that only occur when application changes interact with a complex, zoned network topology).

    The reality is that over-optimizing for Ops (safe, secure and slow) or Development (Fast, opportunistic, iterative) proves sub-optimal. A balance is needed.

    But that balance increasingly skews toward upgrading Ops to meet developer demands. Indeed, developers are writing a host of infrastructure and tools to help other developers better manage their creations. Often such tooling is SaaS-based, and a significant percentage is open source, allowing developers to continue to skirt the controls of Operations while maintaining flexibility. Companies like Boundary, Puppet Labs, SumoLogic and more have arisen to make operations easier for DevOps professionals, among others.

    My own company, MongoDB, develops the most popular database for modern applications. Universally lauded for how easy MongoDB makes development, our tools for facilitating the operation of MongoDB at scale have lagged. Over the past two years we have been building out monitoring, back-up and other management tooling for MongoDB. It’s simply not enough anymore to forget the “Ops” in DevOps.

    No Putting The DevOps Genie Back In The Bottle

    I expect we’ll see this trend continue. The power of developers will increase in large part because the industry will better enable them to manage their applications, or to work more seamlessly with their Ops peers. But the era of Ops dictating to developers is over.

    Blame open source.

  • You Can’t Sell DevOps

    You Can’t Sell DevOps

    A few days ago, I made an introduction for colleague Jason Dixon (@obfuscurity) over email, and described Jason as “a leading supporter of the DevOps movement”. A mere 17 minutes later, Jason replies to me, “Are you trolling me?”.

    I was aghast. Granted, I haven’t known Jason terribly long (measured in months), but I’ve been well aware of his impact and contributions to the technical community (pulling together Monitorama , for starters). I couldn’t fathom the visceral reaction. But it did get me thinking… Why would someone so impactful want to distance themselves from the description?

    For Jason, and many like him, it’s a question of intentions. To explore that, we’ll have to go back to the beginning, and for me the beginning was 5:20pm PST on June 24, 2010, in the last session on the last day of Velocity CA… a “choose your own adventure” talk by Adam Jacob (@adamhjk ), answering the question “what is DevOps?”.

    I happened to have my flip cam handy…

    http://www.youtube.com/watch?v=Fx8OBeNmaWw

    “So here’s the thing: DevOps is a cultural and professional movement. Period. That’s it… There’s no technology, you can’t patent ‘devops’. Can you market a DevOps compliant toolset? You cannot.”Adam went on to describe the evolution of the system operation profession, and the challenges he personally found. Then came the org chart of a traditional silo approach to a technical organization hierarchy: software developers

    Adam proceeded to challenge this approach with an alternative. “So this is everybody in a big pile. What we are is peers, and friends, and mentors, and we are having fun!”. Hysterically live in one silo, reporting into a director who reports into a VP, network engineers live in a separate silo, reporting into a separate director, system administrators in a third silo, and security engineers in a fourth. Proverbial walls erected between technical contributors, isolated from one another through a formal hierarchy. Need to communicate across to a peer? Better go up the stack to go back down.

    Our hypothetical example continues of a conversation in the big pile: “Hey security dude, what’s that you’re doing?!?”, and he’s like “I’m breaking into the web site.” And you’re like “How’d you do that?!?”, and he shows you and the network engineer is like “I can stop you,” and you’re like “How?!?,” and he’s like “Firewall rules.” And you’re like “Whoa! Firewall rules?!?” and the security engineer is like “Why did you need to do that anyway, that’s a protected network”. And then everybody is having a good time. And the Directors. They’re just helping you out. They’re propping you up.” cory2

    While improvised, comical, and at best a generalization of an industry in a hypothetical example, one key point stuck in my mind from Adam’s talk: DevOps was about people, and how they were most effective and happiest working together. That was the point of the cultural and professional movement: to be more effective and happier as a team, embracing a “shared outcome.” Software developers, network engineers, system administrators, security engineers, directors, VPs, etc., we all succeed or fail together.

    That’s not something you can sell. (In this context, I’ve specifically referring to selling a tool or service.) Nor is it something you can pin down to one definition applicable everywhere. Nor is it something that you can transplant wholesale from one company to another and expect the same level of effectiveness and happiness. You can’t “bolt on” DevOps to your existing organization with only tools and services. You can’t buy a toolset and expect your challenges to be alleviated. You have to understand the role improved tooling has played for those that have found success in this field.

    Tools can certainly be enablers and facilitators, particularly when they free up person hours from manual tasks (assuming you can channel the new found time in an enabling direction), and particularly when they increase transparency as to where technical problems exist, helping to avoid the “It’s not my servers, it’s your code. It’s not my code, it’s your servers” blamefest. How we interact with them, and the patterns of behavior they impose on us as people, can lead to drastically different results from one organization to the next, even if those organizations have identical tools.

    When our tools look like the first silo org chart example above, with a different set of tools presenting different perspectives to different teams bound together by a shared outcome, the tools end up dictating and reinforcing the silo organization hierarchy for how people work together.  To break down the proverbial walls, our tools instead must help define how people should work together.

    Tools enabling configuration management, automation, improved monitoring, metrics collection, and source control won’t ensure success in isolation, because the DevOps movement isn’t ultimately about the tools. It’s about the people, and how they will work best together to succeed. The right tools can go a long way toward removing barriers, enabling the right conversations, and setting the stage for success.

    Part of the challenge is that the tools are one of the more tangible and visible aspects of how we work. In can be easy to take a stroll through a peer organization and take note of the different tools, and attribute their success accordingly. You might be sold on the idea that a tool they have has been the missing piece to your success all along. Fight this first impulse to the tangible and visible, and remember that it’s hard to see how folks really interact. It’s hard to see how well their people, processes and tools work together to produce results in sunny-day scenarios. It’s even harder to see when the sh*t hits the proverbial fan. That’s where the real test is though, and that’s where the real lessons are to be learned. 

    So let’s all be a little more focused on the lessons learned in ways to work together supported by tools than on the merits of one tool over another in isolation. No doubt, it’s a harder and deeper conversation to have, but the value is tremendously greater. If we keep it up, maybe we can make Jason proud to call himself a leading supporter of this professional and cultural movement.

    Or maybe I was trolling Jason Dixon all along.

  • Changing Organizational Culture – A Sweaty Use Case

    Changing Organizational Culture – A Sweaty Use Case

    When we think about changing organizational culture our immediate reaction is usually – yeah right, go fight city hall – and we take the Homer Simpson route, and don’t even bother trying.

    Over the course of the last year, I managed to achieve what felt like an impossible change within my organization, and I took some valuable lessons from the experience.

    It dawned on me afterwards that there were a lot of parallels in this process that we also underwent when trying to instill a DevOps culture in our organization, and I thought I’d share what I learned.

    The Epiphany

     I love riding my bikes – mostly mountain biking, and I usually am only able to do this over the weekend.  It’s something I really love to do, but in addition to my full time day job – Head of Product at my company, I’m a full time dad – a proud father of three.  All of this doesn’t leave me a lot of time to do the things I love.

    On a good day I spend more than an hour stuck in traffic, and on bad days more than an hour and a half.  This is basically dead wasted time.

    So I had an epiphany one day. Why shouldn’t I take advantage of this time to my favorite thing?  I could easily bike to work – and use this wasted time to my benefit, doing the thing I love most.  The problem lies in the way I look after an hour bike ride.  Pretty sweaty.  Here’s a pic.

    And, I know what you’re thinking – the answer is – no it wasn’t raining that day, in case you were wondering.

    The challenge though, is that I work with people, so I needed to find a way to clean up after my rides.  But, alas alack, we don’t have a shower in the office.  The plot thickens.

    So you might be wondering at this point, how this all relates to DevOps.

    DevOps is about pushing change in organizations – and not necessarily just technical ones. There are cultural organizational practices that often times need to be changed before you can even start to think about technical changes.

    So I believe this story represents a real parallel case study on the organizational changes often needed, to create a DevOps culture within an organization.  These processes many times involve a change in hardware – in this case, the shower itself.  As well as require a change in culture, actually motivating the management to realize the importance of it, and have people start choosing to do athletics before work (and preferably also ride to work).

    The Method

    And then I ran into Jesse’s Robbins’ rules. For those who don’t know, Jesse is one of the founding fathers of DevOps and web operations, and happens to also be the founder of Chef (OpsCode at the time). In one of the recent CloudExpo conferences in Europe, Jesse gave an awesome keynote about how to hack organizational culture to drive change. So working on the premise of “Jesse’s Rules” – I got to work:

    • #1 – Start small – Build trust

    Until the new shower was built, I’d shower daily with a HelioPressure shower  (you can find it on Amazon for less than $100), and I’d shower every day on Nati Shalom’s [LINK TO DEVOPS BLOG] balcony in the office.  Demonstrating I was serious and dedicated to my biking to work. Yes, it definitely raised a few eyebrows, but then again, better to shower on a balcony than walk around smelling like a sweaty rug all day…

    • #2 – Create Champions

    Doing crazy stuff yourself is the easy part. Finding like-minded individuals that will do the same, well that’s a different story. But after working hard at convincing others, I managed to find at least two people that would occasionally use my improvised spa, and this really helped me drive change in my company.

    • #3 – Use Metrics to Build Confidence

    For me there were actually a few metrics that contributed to the driver for change.  The first was that, I right off the bat, lost quite a few pounds – which was an incentive for a number of people to jump on the bandwagon – or rather bike – in this context.  The next metric was hard research that shows that people that are in shape tend to be absent from work 80% less often than employees that are not in shape. That helped me in convincing my CEO that it’s actually an investment and not just thoughtless spending…

    • #4 – Celebrate Success

    After committing to our makeshift shower for months, budget was officially allocated, and a shower was built for all the employees looking to participate in sport activity on their way to work.  Needless to say that I’m not the only one riding my bikes to work today 🙂

    Lessons Learned

    So the lessons I learned throughout the process, and my key take aways as I see them are:

    •  #1 You’ll run into a lot of skeptics along the way. IGNORE THEM.

    Many people will attempt to discourage you from your goals along the way – don’t listen to them. If I had a dollar for the amount of times I heard “it’s never gonna happen”, I wouldn’t need company budget to build a shower.  But I stuck with the plan, and it ultimately paid off.

    •  #2 Change takes time

    Rome wasn’t built in a day.  It’s a cliche – but cliches are born from common experience.  We’d been discussing building a shower for 2-3 years at the least.  It’s true that until I didn’t demonstrate that I was truly dedicated, rule #1, it wasn’t taken seriously. It took time for the company to come around, and digest the idea, until it gradually happened.   So again, if you stick to your guns, and have faith – a change ‘gone come.

    •  #3 Take it one small step at a time

    Basically start small, and be creative. Until you can achieve the big results, celebrate the small achievements too.  Adding champions to your cause, increasing awareness, possibly a meeting to discuss the changes you propose.

    •  #4 Be prepared to improvise

    I owe a lot to my colleague that came up with the pressure shower idea.  I think until I actually demonstrated that there was a really dire need for such a change, it wasn’t really considered seriously.

    •  #5 Find a management sponsor

    Luckily our CEO is athletic in orientation, and is an avid bike rider as well, and he was committed to help making this happen.  Many times it won’t necessarily be the CEO, but find a management champion that believes in your cause, and is willing to help you bring about the change.

     Bottom Line

    No matter what change you’re trying to push, these lessons and rules have proven to hold true for me in any circumstances, and I often times use this method to push significant changes in work and in life.

    Oh, and here’s a pic of the new shower.

    If you’re ever in Israel, and need a place to work, and want to bike there…you’re welcome.

  • DevOps – A wake up call to security vendors

    DevOps – A wake up call to security vendors

    DevOps is changing the way IT works.  Through a collaboration of development and IT teams, organizations can achieve tremendous scale at increased speed to deploy and maintain critical applications and infrastructure.  The key technologies of virtualization and configuration automation have made this possible.  Without these technologies, development may not slow down, but deployment would slow to a crawl and the benefits of DevOps would be minimal.

    It is interesting to note that in almost no definition of DevOps is the security process discussed as a key element.  In some cases you will hear mention of improved security through consistent configuration management enabled through automation, but security teams are not at the DevOps table.  Why is that?  I would suggest that for most people security is considered a barrier to IT.  Not surprisingly, DevOps, a practice focused on streamlining IT, security has been left out of the conversation so as to not prevent achieving the goal.

    But if you accept that security needs to remain a part of the story, then it must be just as manageable as the infrastructure or it will never have a seat at the DevOps table.  Take a look at Puppet or Chef; they automate the configuration of the systems necessary to run the applications.  Through automation scripts, systems are deployed, configured, updated, and managed.  Development boxes are configured consistently with production boxes limiting impacts related to deploying an application into production.  So why can’t security fit into this picture?

    One reason, is a lack of standardization.  Every security vendor wants to be unique.  They promote their solution as so incredibly unique from their competitors that there is no way to standardize.  So, creating scripts to manage security would require not only a unique script for each vendor, but a unique solution.  In comparison, the command to edit a route in Linux may be slightly different than in Windows, but the parameters are the same.  When you are managing security rules on a firewall as an example, you can’t even get agreement on that.  Zones are required for some firewalls, not used in others.  Applications define access in Palo Alto, but don’t exist in Cisco.  Even firewalls that share the idea of Applications use non-standard dictionaries of Applications.  So, creating a script becomes a non-standard solution.  Worse yet, the script that worked today won’t necessarily work tomorrow when the vendor changes their solution.

    To make it all unmanageable, is the fact that they are so proud of their uniqueness that they have obscured even how they do it.  If you take all the firewall vendors on the market, not one of them publishes an API on how to fully manage their firewall policies.  Palo Alto exposes some capabilities for object management in a cool SDN sort of way.  Check Point has a very powerful, but proprietary API that allows full management, but they do not recommend it be used in that way.  Cisco is straight up command line with some newly available API’s and should be commended for making it at least obvious how to do it, if not easy.

    The result: security is not a big part of the DevOps picture today.  And if the vendors don’t do something, they will be replaced by solutions willing to operate in this new DevOps world.  Open source firewalls or integrated firewalls in the virtual stack for example are ready to take their place.

    The solution for security vendors: publish API’s to fully manage their systems.  They must accept that their UI is no longer what differentiates them from their competitors, it is the security engine and enterprise manageability.  Make it possible for others to manage their solution through open API’s with their full support.  All the security vendors will not come together and agree on a single API, but each should at least publish a usable API for their own product.  Vendors like FireMon or open source contributors to solutions like Puppet and Chef will solve the rest.

    Security vendors better wake up or they will get left behind.

  • The Fourth Seat at the IT Service Velocity Table

    The Fourth Seat at the IT Service Velocity Table

    Agile. Devops. SDN.

    Development. Operations. Network.

    Notice anything missing? How about security? On the one hand, that could be because we’ve finally managed to realize that security is an integral part of every aspect of IT and it’s by default embraced by the other three groups already.

    I’ll pause while you stop laughing and catch your breath.

    On the other hand, it could be because it’s like every other technological movement and advancement and we’ve simply plowed ahead without considering security until, as is always the case, someone remembers that security is important too. If development, operations and the network teams are going to have an answer to improving the speed with which applications get from the IDE to the end-user’s iPhone, then shouldn’t security have a seat at that table, too?

    Absolutely.

    It can be argued, in fact, that some areas of security already have a seat at the service velocity table. Consider automated pen testing, virtual patching and continuous vulnerability assessment services. These are often continuous processes that integrate (sometimes) with existing systems or, at a minimum, with some process through which aberrations can be noted and addressed.

    It’s important to note that this – and devops – isn’t a plea for automation. Automation and orchestration are tools in a toolbox that may or may not be a fit for a particular tasks or process. Most enterprise organizations, for example, refuse to automate changes to data center firewall rules. Not because it can’t be done – it can – but because the ramifications of a change going wrong is too great to take even the small risk that might be incurred.

    But in the interests of speeding up the deployment process (in which security is – or should be – a significant player) there are likely adjustments that can be made to align security with other groups already adopting more agile methodologies.

    For example, consider that along with agile development often comes a “nightly build” of the software, designed to ensure that changes don’t break the complete package. It might be beneficial for application security-focused assessments to be inserted into that process, to test early and often the security profile of the software being developed. This helps in two ways: first, it ensures that developers can be made aware of potential vulnerabilities while they’re actively developing the software. That’s important because one of the precepts of agile is move quickly and by the time you’ve found a vulnerability in the released software the developers are already half way through the next spring and not thinking about what they just tossed over the wall.

    Second, if there are vulnerabilities that turn up that developers can’t, for some reason, address you have a head start in figuring out how to mitigate them. If that requires a security service of some kind, you’ve got time to get the right policies into place before the application hits the Internet and becomes a target.

    There are plenty of ways in which agile methodologies can be adopted by security professionals. One of the precepts of devops is getting outside the silos that exist within the organization and communicating with the other folks seated at the “get the application to the user” table. Being more involved with development and operations and even the network will enable security to integrate more fluidly with the increasingly agile processes that drive the application lifecycle.

  • DOCKER 0.9: INTRODUCING EXECUTION DRIVERS AND LIBCONTAINER

    From the Docker Blog:

    Fellow Dockers,

    Today we are happy to introduce Docker 0.9. With this release we are continuing our focus on quality over features, shrinking and stabilizing the core, and providing first-class support for all major operating systems.

    In addition to dozen of bugfixes, Docker 0.9 includes 2 major improvements: execution drivers andlibcontainer.

    As usual, for a complete list of improvements, you can check out the Changelog.

    Execution drivers

    First, we are introducing an execution driver API which can be used to customize the execution environment surrounding each container. This allows Docker to take advantage of the numerous isolation tools available, each with their particular tradeoffs and install base: OpenVZ, systemd-nspawn, libvirt-lxc, libvirt-sandbox, qemu/kvm, BSD Jails, Solaris Zones, and even good old chroot. This is in addition to LXC, which will continue to be available as a driver of its own.

    There are already several projects underway to develop more drivers. Want to join the fun? Come say hi on #docker-dev on Freenode, and we’ll help you get started.

     New default driver: libcontainer

    docker-execdriver-diagram

    Second, we are introducing a new built-in execution driver which is shipping alongside the LXC driver. This driver is based on libcontainer, a pure Go library which we developed to access the kernel’s container APIs directly, without any other dependencies.

    Thanks to libcontainer, Docker out of the box can now manipulate namespaces, control groups, capabilities, apparmor profiles, network interfaces and firewalling rules – all in a consistent and predictable way, and without depending on LXC or any other userland package. This drastically reduces the number of moving parts, and insulates Docker from the side-effects introduced across versions and distributions of LXC. In fact, libcontainer delivered such a boost to stability that we decided to make it the default. In other words, as of Docker 0.9, LXC is now optional. To switch back to the LXC driver, simply restart the Docker daemon with docker -d -e lxc. Of course we will continue to support the LXC driver going forward.

    Using libcontainer for your Go projects

    We have developed libcontainer in the hope that other projects will reuse it. If you’re interested in playing with the native container features of Linux – namespaces, cgroups, capabilities etc – then we encourage you to start hacking! To get started go get the Go package and check out the API docs:

     

    1
    2
    go get github.com/dotcloud/docker/pkg/libcontainer
    godoc github.com/dotcloud/docker/pkg/libcontainer

     Objective 1.0

    This release is a major step towards a stable, production-ready 1.0 release. We plan on making our next release, 0.10, the first release candidate for 1.0.

     As discussed previously, the goals for Docker 1.0 are:

    • production quality

    • first class support of all major operating systems

    • a shrunken core and a stable plug in architecture

    • well documented

    • able to be commercially supported by Docker and our partners

    • Docker able to offer long term support

    We are already hard at work preparing 0.10, with several exciting improvements that we think you will like. If you would like a sneak peek, or if you feel like contributing – come say hi! We are on #docker on Freenode. We welcome enthusiasts of all levels and can help you get started with your first contribution. As always, thanks go out to our community of contributors, now 352 strong!

    Thanks and happy hacking!

     

    The Docker team

  • DevOps Culture Hacks

    DevOps Culture Hacks

    opscode_logo_final1

    Talk by Jesse Robbins (@jesserobbins), Chairman and CEO of OpsCode

    Blogged live from DevOpsDays Boston 2011 by @martinjlogan

    Tricks for getting DevOps to work in your company, from the technical to the social, taken from experiences at Amazon.

    Jesse Robbins was the “Master of Disaster for Amazon”. As an ops guy, he worked multiple 72 hour sessions, took site up time very seriously, and eventually turned into the stereotypical ops guy who said “no” all the time. What’s worse is he discovered that he was proud of it. Jesse started to notice that he was taking site outages personally. Amazon at the time, 2002-2003, was doing standard ops. They deployed in large monolithic fashion which is an absolutely painful process prone to error. Managing such a process and also finding yourself emotionally involved with the work to a high degree is not a productive situation to be in.

    A turning point for Jesse in terms of moving from an obstacle in the way of change to someone that really knew how to add value with ops practice stemmed from a battle he got into with the “VP of Awesome” at Amazon. This was the nickname of this particular VP because it seemed that pretty much any highly interesting project at Amazon was under this man’s purview. What happened was that Jesse did not want to let out a piece of software because he knew, for sure, that it would bring the site down. The VP overrode him by saying that the site may go down, but the stock price will go up. So, the software went out, and it brought the site down. Two days of firefighting and the site came back up, and so did the stock price, and so did the volume of orders.

    The dev team went on and had a party, they were rewarded for job well done, new and profitable functionality released. At the end of the year, Ops got penalized for the outage! Amazon rewarded development for releasing software and providing value and operations was not a part of that. They were in fact penalized for something that was out of their control.

    This of course did not sit well and as a result of this and other similar situations Jesse actually got famous for saying no. Who in their right mind would want to release software and go through that over and over? When the business would put up a sign advertising new functionality that was to be developed, something they were presumably excited about, Jesse would write the word “No” on it.

    Operations naturally wanted to protect itself and came up with all kinds of artifice in order to do so. Root cause analysis so that blame could be assigned efficiently. Software freezes that prevented software from being delivered to the site during peak times of the year. This seemed like progress, but clearly was not looking back on it.

    (This all sounded fairly familiar to the folks in the room)

    On to DevOps

    Now to talk about DevOps. We have at least the beginnings of an idea of what to do about the situation described above. Prior to addressing this situation correctly, or at least more correctly, Ops looked at their output as a waste. The best thing they could do was to cost the site $0. Instead we need to be looking at this another way, bringing value through the function of ops. DevOps is about creating a competitive advantage around the things Ops does every day.

    Why does the break occur? Historically Ops creates value by reducing change and getting paged when things break. Dev is about value creation and Ops is about protecting that value. This creates a “misalignment of incentives” meaning that different organizations are rewarded for different behaviors. This creates something called local optimization. Knowing these terms will help you talk to MBAs about DevOps!

    We have a fundamental misalignment of incentives and in fact a conflict in incentives. Development is exclusively aligned to releasing software and not at all focused on maintaining it. Ops, is the opposite. Each group optimizes locally around this which creates conflict. Operations is focused on minimizing change because that reduces outage where as Dev is entirely focused on maximizing change.

    Solving this problem is what DevOps is all about.

    The unproductive way of thinking mentioned earlier came about was in an environment with 4000 devs and significantly fewer ops folks. In order to alleviate the problems caused by misaligned incentives and local optimizations were to come up with those punitive changes that are incredibly satisfying to Ops folks but really don’t help solve the problem. Those changes in the form of meetings and review boards that are around to punishing people into releasing the software the way you the Ops person wants. These are the kind of measures that control oriented people gravitate towards, and it feels like progress for them. To be more DevOps don’t try to fight them in this: “Don’t fight stupid, make more awesome”.

    One initial thing that changed, that started the real progress, was to align dev and ops in a way that prevented local optimization; putting devs on call for their own software. This started to shift ops from being the people that just dealt with all the problems to people that became experts on all the services that allow the software to run. Ops started to become tier 2, escalation for devs. The way you got there, was to offer devs deployment options and permissions, if they passed some training and were willing to be on call. Initially this caused a fair amount of chaos. Devs had a load of pagers and got messages that confused the heck out of them. There was pain and frustration. This pain and frustration and the fact that devs were now playing with tools in actual production environments, really changed the culture quickly.

    Through trial and error, top down fiat, audits, and every carrot and stick approach the formula for this class of organizational change was developed. This what Jesse uses even today to accomplish these changes and what we will concern ourselves with for the rest of this talk.

    • Start small and build on trust and safety.
    • Create champions
    • Use metrics to build confidence
    • Celebrate success
    • Exploit compelling events

    Start Small and Build on Trust and Safety

    We tend to want to take on the entire org up front. We want to throw everything out, starting at the top and working down. This simply does not work, Jesse had failed multiple times before realizing that this was a failing strategy. Continuous deployment for example seems like something you want to deploy widely. Instead you should start with a small motivated team and build some success there.

    Another thing to consider when thinking about starting small and building trust is that when introducing disruptive change in an organization you should lead with questions to garner buy in instead of just telling people that you have the solution and such and such is what to do. Don’t even use the word DevOps, just focus on the problems and get permission to start to change it.

    You have to make the experiment of changing things safe. Jesse tells people that he will take 100% of the blame if things go wrong, in exchange for the space needed to make the desired changes. Creating safety is critical in pushing through organizational change. Crucial Conversations by Kerry Patterson is a book that covers this really well and the Jesse thinks is one of the most critical books to read if you want to create organizational change.

    Create Champions

    “You can accomplish anything you want so long as you don’t require credit or compensation”. It is amazing what you can accomplish when you give away that part of your self that requires recognition. You must shine the spotlight on those in your organization that get it. Get those that are recognizing the need and acting upon what you are pushing forward to talk. Make your boss a champion; this is critical. It is really important that your boss can explain what you are doing and why or at least be able to provide air cover for you.

    The second part of this is to give people status; special status. SRE “site reliability engineers” walk around Google with leather bomber jackets. They get hazard pay, they have special parties, they are considered to have elevated status around the organization. When you find your champions do something that makes them stand out. Wikis are quite powerful for this. Write down and explain in very powerful language what these champions do.

    At Amazon Jesse created this thing called the call leader program. They trained people to handle high impact events. After a while there evolved a pressure to join. Eventually you become the person that people have to go to in order to get a certain status which gives you personally more organizational power – not the point but helpful in furthering the change you want to further.

    Use metrics to build trust

    Get as many metrics as you can. Begin to look at them for KPI’s (key performance indicators). These are the things you will use to prove your case. What you are looking for is a story, and a set of metrics to prove you have one. John Allspaw talks about things like MTTR, mean time to recover. These are great for story telling. You want to capture metrics early and use them to tell your story. “Having devs on call will be a great thing for us, and oh, by the way, here is the data that proves it!”

    Make sure that you are good at telling your story with the data that you capture. Here Hans Rosling gives a TED talk about how to tell a story with data that Jesse recommends for everyone. This can be used for inspiration. Ideally have your champions tell your story.

    Celebrating your Successes

    This comes back to “you can accomplish anything you want so long as you don’t require credit or compensation”. Create moments in time where you celebrate the success of the change you created. Have parties when you reduce MTTR by 15 percent. Give people to a moment in time where they recognize the change the created and that the change was good. This gives people a moment in time to look back on and judge progress. This is of critical importance.

    Exploit Compelling Events

    Compelling events are those big company issues, big or small, it does not actually matter. They are the events that create cultural permission to make important change. An example is the executive mandate toward cloud computing. This is a compelling event that allows you to make a whole bunch of procedural change. Big outages are compelling events, they give you cultural permission to make significant change that would be otherwise impossible in normal times.

    When you have a compelling event you don’t encounter resistance, but instead, permission to make large change. If you don’t have such an event, then create it. Jesse had something called “game day” at Amazon. Creating outages to test failure recovery. Non-recovery became a compelling event. Big deployment pushes are examples of compelling events. If you are in the middle of a serious problem with process it is hard to propose chucking it out in flight but if you offer to own the postmortem process you can direct it toward the change you want to make – though start small as indicated by the first point in this list.

    The next time you want to create change in your org particularly DevOps change keep in mind:

    • Start small and build on trust and safety.
    • Create champions
    • Use metrics to build confidence
    • Celebrate success
    • Exploit compelling events

    What are your thoughts on DevOps culture hacks? Anything to add to the list?

  • DevOps – A Valentine’s Day Fairy Tale

    DevOps – A Valentine’s Day Fairy Tale

    This is a guest post by Matt Watson from Stackify

    Once upon a time two people from different sides of the tracks met and fell in love. Never before had the two people found another person who so perfectly complemented them. Society tried to keep them apart – “It’s just not how things are done,” they’d say. But times were changing, and this sort of pairing was becoming more socially acceptable.

    They met at the perfect time.

    Ops had grown tired of the day to day grind of solving other people’s problems. Enough was enough and she needed a change in her life. A perfectionist and taskmaster to the highest degree, she tended to be very controlling and possessive in relationships. It became more about commands than conversation, making life miserable for both parties. She began to realize she hated change, and felt like she spent most of her time saying “No.” It was time to open up and begin to share to make a relationship work.

    Dev, on the other hand, was beginning to mature (a little late in the game, as guys seem to) and trying to find some direction. He had grown tired of communication breakdowns in relationships – angry phone calls in the middle of the night, playing the blame game, and his inability to meet halfway on anything. He began to realize most of those angry phone calls came as a result of making impulsive decisions without considering how they would impact others. His bad decisions commonly led to performance problems and created a mess for his partners. Dev wanted to more actively seek out everything that makes a healthy relationship work.

    The timing was right for a match made in heaven. Dev and Ops openly working and living side by side to make sure both contributed equally to making their relationship work. Ops realized she didn’t have to be so controlling if she and Dev could build trust between one another. Dev realized that he caused fewer fights if he involved Ops in decisions about the future, since those decisions impacted both of them. It was a growing process that caused a lot of rapid and sudden change. Although, like most relationships, they knew it was important to not move too fast, no matter how good it felt.

    Dev and Ops dated for about four years before they decided to get married. Now they will be living together and sharing so much more; will their relationship last? How will it need to change to support the additional closeness? But they aren’t worried, they know it is true love and will do whatever it takes to make it work. Relationships are always hard, and they know they can solve most of their problems with a reboot, hotfix, or patch cable.

    Will you accept their forbidden love?

    7 Reasons the DevOps Relationship is Built to Last

    1. Faster development and deployment cycles (but don’t move too fast!)
    2. Stronger and more flexible automation with deployment task repeatability
    3. Lowers the risk and stress of a product deployment by making development more iterative, so small changes are made all the time instead of large changes every so often
    4. Improves interaction and communication between the two parties to keep both sides in the loop and active
    5. Aids in standardizing all development environments
    6. DevOps dramatically simplifies application support because everyone has a better view of the big picture.
    7. Improves application testing and troubleshooting

    mat

    About the author: Matt Watson is the Founder & CEO of Stackify. He has a lot of experience managing high growth and complex technology projects. He is focused on changing the way developers support their production applications with DevOps.

  • Defining the Dev and the Ops in Devops

    Defining the Dev and the Ops in Devops

    This is a guest post by Matt Watson from Stackify

    So what does DevOps mean exactly? What is the Dev and what is the Ops in DevOps? The role of Operations can mean a lot of things and even different things to different people. DevOps is becoming more and more popular but a lot of people are confused on the topic of who does what. So let’s make a list of the responsibilities operations traditionally has and then figure out what developers should be doing, and which if any responsibilities should be shared.

    Operations responsibilities

    • IT buying
    • Installation of server hardware and OS
    • Configuration of servers, networks, storage, etc…
    • Monitoring of servers
    • Respond to outages
    • IT security
    • Managing phone systems, network
    • Change control
    • Backup and disaster recovery planning
    • Manage active directory
    • Asset tracking

    Shared Development & Operations duties

    • Software deployments
    • Application support

    Some of these traditional responsibilities have changed in the last few years. Virtualization and the cloud have greatly simplified buying decisions, installation, and configuration. For example, nobody cares what kind of server we are going to buy anymore for a specific application or project. We buy great big ones, virtualize them, and just carve out what we need and change it on the fly. Cloud hosting simplifies this even more by eliminating the need to buy servers at all.

    So what part of the “Ops” duties should developers be responsible for?

    • Be involved in selecting the application stack
    • Configure and deploy virtual or cloud servers (potentially)
    • Deploy their applications
    • Monitor application and system health
    • Respond to applications problems as they arise.

    Developers who take ownership of these responsibilities can ultimately deploy and support their applications more rapidly. DevOps processes and tools eliminate the walls between the teams and enables more agility for the business. This philosophy can enable the developers to potentially be responsible for the enter application stack from OS level and up in more a self service mode.

    So what does the operations team do then?

    • Manage the hardware infrastructure
    • Configure and monitor networking
    • Enforce policies around backup, DR, security, compliance, change control, etc
    • Assist in monitoring the systems
    • Manage active directory
    • Asset tracking
    • Other non production application related tasks

    Depending on the company size the workload of these tasks will vary greatly. In large enterprise companies these operations tasks become complex enough to require specialization and dedicated personnel for these responsibilities. For small to midsize companies the IT manager and 1-2 system administrators can typically handle these tasks.

    DevOps is evolving into letting the operations team focus on the infrastructure and IT policies while empowering the developers to exercise tremendous ownership from the OS level and up. With a solid infrastructure developers can own the application stack, build it, deploy it, and cover much if not all of its support. This enables development teams to be more self-service and independent of a busy centralized operations team. DevOps enables more agility, better efficiency, and ultimately a higher level of service to their customers.

    mat

    About the author: Matt Watson is the Founder & CEO of Stackify. He has a lot of experience managing high growth and complex technology projects. He is focused on changing the way developers support their production applications with DevOps.