devopsy.kinsta.cloud

Category: Features

  • A Critical First Step in Bringing Dev and Ops Together

    A Critical First Step in Bringing Dev and Ops Together

    We all want to deliver excellent software to our customers. We now have a greater awareness than ever before of the interconnectedness of all the activities required to do so. The agile movement started this but it was not until the advent of DevOps that Ops was considered part of the full picture. In this article I will explain an effective first step that agile organizations can take to complete the picture and bring ops more completely into the product development process.

    Delivering excellent software starts with product definition and flows from those initial stories and designs, into dev, verification and testing, into production and then back again in the form of customer feedback and bugs. In reality the feedback paths represent a transitive closure of all activities just mentioned as you can see in the diagram below. Agile recognized the importance of feedback loops in good product development. Agile also realized that for the right information to flow, walls had to be broken down. The motto “Done means tested” arose and indicated that product owners were not done with a story, could not consider it closed, until it had been tested and they had verified it. This represented the integration of PM, development and QA functions. A huge step, but one step too short. Operations was still not viewed as a key part of building excellent software and there was not much insight into the fact that the feedback loop did not stop at testing. DevOps has recognized this shortfall and taken it to its logical conclusion – “Done means in production.”

    Operational Excellence Feedback Loop

    Product development is a cycle, a cycle that operations is a critical part of. In describing how all parts of the cycle can be integrated lets start at the most natural beginning; product definition. Product definition activities produce stories, designs, functional acceptance criteria (A.C.) and prioritization. In this phase we don’t produce all we need for the finished product but just enough to get us going. With popular agile methodologies like Scrum this is ideally not done solely by a product owner but instead in conjunction with dev and test at story grooming meetings, design reviews and sprint planning. In these meetings dev, QA and product communicate and negotiate stories, comment on designs, and flesh out acceptance criteria. In this way the stories, designs and acceptance criteria are improved and everyone gains an understanding of what is being built; they also become invested in it. A story may look something like this (abbreviated):

    Story: Blog administrator can change user roles SO THAT users can be managed effectively

    A.C: Given blog administrator is logged in
    And on a user profile
    When he selects a permission scheme from a permissions drop-down menu
    Then the users permissions are updated and stored in the DB

    This is a great start but it is not enough. If we are to be excellent we must modernize this just a bit and add in some DevOps spice.

    A first step towards bringing ops effectively into the cycle is to include someone responsible for actually operating the software in the story and acceptance criteria grooming sessions, design reviews, and sprint planning meetings. The critical aspect is that when the team negotiates the acceptance criteria associated with a story it will be ops responsibility to ensure we for each story or feature the team adds success and failure metrics, initial alarm values and any operational functionality such as a kill switch that will be needed in order to operate the functionality effectively. So a story will now contain things like the following in the A.C. (acceptance criteria).

    Given operation X fails
    Then the metric operation-x:failed will be sent through the metrics feed

    We will also alter our definition of done to make sure that a story will not be closed until we meet the ops driven acceptance criteria. For example: we can see the metrics we have defined in our A.C. show up on an ops dashboard. There are so many benefits to this such as folks that are unlucky enough to be on on-call will have participated in this story grooming and will know exactly what these metrics mean when they encounter an issue with them for the first time in production. Furthermore, they will know what operational levers they can pull or request to be pulled.

    This first step represents a connection from product definition to dev, to QA, to operations. This connection is perfectly natural and intuitive when you think about it. Being a true DevOps organization and developing operationally excellent products requires more but having connected product definition to operations as we have provides a nice toe hold from which to flesh things out further. Best of luck. If you have any questions leave them as comments and I will respond as best I can.

  • DevOps & UX: A Star-Crossed Love Story

    DevOps & UX: A Star-Crossed Love Story

    “Two households, both alike in dignity
    In fair Verona, where we lay our scene
    From ancient grudge break to new mutiny
    Where civil blood makes civil hands unclean.
    From forth the fatal loins of these two foes
    A pair of star-cross’d lovers take their life
    Whose misadventured piteous overthrows
    Do with their death bury their parents’ strife.” ~ Romeo and Juliet: Prologue, William Shakespeare

    For more than 15 years I have watched as they battle. The chasm between the two houses was so wide that I often wondered if it would ever be bridged. I never wavered from my view that we were all one house striving to find common ground for making working solutions for the world around us. Sometimes there were allies beside me. Sometimes I stood alone, For the first time in my professional history, it seems like the blood-feud could be put aside. Is it time for peace? Even more ambitious, is the day of an alliance at hand?

    The Feud

    For many years, corporate IT and finance were the only voices in the room. Ruled by an unquestioned and unmeasured desire for quantification and linear solutions, design considerations were usually an afterthought. All parties were subservient to the almighty ROI which ruled over all with an iron fist. Claims of possible productivity gains through better tooling were always met with doubt and even when taken seriously, you could not claim productivity gains unless you were willing to commit to lowering headcount. Tooling for technical employees to manage the environment and its operating data was so scarce, that there were more applications with no maintenance tooling than applications with operational plans. Operational changes were made by going into databases by hand and making changes in a “hit or miss until you hit” fashion.

    In the late 90’s UX showed up to the party and with a basis in humanism, the equation changed. This was the time that the “great misunderstanding” happened and 15 years were wasted building up a corporate landscape strewn with enmity and false narratives. The “great misunderstanding” goes something like this:

    • “Let’s make something that people can use” – UX practitioner espouses user-centric design and introduces new methods and artifacts into the SDLC processes which take interaction design specifications to a new level in corporate America
    • “That’s going to take more time and I don’t agree that it’s any better than what I could do myself” – IT staff view this incursion as a mechanism to take what little time they are given to complete projects while also limiting their creative outlet
    •  “I don’t care what you think and neither does anyone else” – UX practitioners view the response as a denial of their value proposition and a fundamental lack of care about the people who interact with technology

    Several of the basic assumptions in these archetypal interactions between disciplines were short-sighted. For example:

    • The “more time” fallacy – adding a design process may take more time if you only think about a single iteration in development and that’s only if you are lucky enough to have a complete interaction model worked out in your head that doesn’t have any design holes in it. Designing interactions on paper saves time in the same way that technical design saves time over “jumping straight to coding”
    • The “fighting on the wrong front” mistake – It was never the goal of UX to remove IT budget. The goal was to make things better for the humans that interacted with technology while creating business value. The two different disciplines never saw their goals as aligned. We will never know what the alternate history would have been like had UX and IT teamed up to get larger budgets in an effort to create better and more sustainable experiences for users, developers AND operators.
    • The “reductive labeling” error – Both IT and UX created a narrative where the interaction and visual design tasks were the “creative tasks” for “creative people”, while the development was the “technical tasks” for “technical people”. These labels have stuck around for the last 15 years and we are still paying the price for removing the idea that IT staff is fundamentally non-creative and devoid of craft.
    • The “self-reinforcing paradox” tragedy – It was never the goal of IT to demean the UX discipline. When one neglected child is joined by another neglected child, the two will almost unavoidably fight over the little attention afforded to either of them. The irony here is that UX’s basis in cognitive science could have possibly seen the hostile response from IT for what it was; a cry for help. Instead, UX took the bait offered by IT and thus we are here 15 years later with a wide chasm between two disciplines that should be united by common causes:
      • a desire to make awesome stuff
      • a desire to create sustainable solutions
      • a desire to express one’s self at work
      • a desire to improve the lives of their customers and employees at the same time

    Oh The Humanities

    When IT resources and UX practitioners first came together the familiar plot “boy meets girl, boy loses girl” emerged and we may now, thanks to DevOps finally have arrived at the time when “boy gets girl again”. Gene Kim and the folks at IT Revolution Press have hit the target dead center in their manifesto:

    “Working in most IT organizations is often thankless and frustrating. People feel as if they’re trapped in an ever-repeating horror movie, helpless to change the outcome The organization abdicates their responsibility to ensure that IT is managed well, plunging the department into relentless intertribal warfare between development, IT operations and information security. And of course, the auditors.

    What inevitably results is chronic under-achievement. The life of an IT professional is often demoralizing and frustrating, typically leading to feelings of powerlessness and rife with stress which seeps into every aspect of life. From stress-related health problems, to social issues, to tension at home, it has become clear that the current state of the IT professional is not only unhealthy, but unsustainable.

    As people, we’re wired to contribute and to feel like we’re actively making a difference. Yet, all too often when IT professionals ask their organization for support, they’re met with “you don’t understand,” or worse, a barely masked, “you don’t matter.”

    As a human being, this is the worst response we could receive. Because the opposite of love isn’t hate — it’s apathy.”

    Despite what some IT step-children will tell you, UX is not about making pretty pictures. UX as a discipline was formed to combat apathy and contempt by corporations through a mechanism of introducing empathy. Similarly, DevOps wasn’t formed to “make web pages faster”. DevOps clearly has a tactic of emphasizing performance (as well as other technical tactics), and performance improvements are clearly one of the benefits that come out of a DevOps organization. DevOps was formed to make IT a more sane and human space inside companies for IT employees, it just so happens that DevOps produces other impacts that are financially beneficial for enterprises.

    Avoiding The Fates of Romeo And Juliet

    Knowing that waters of DevOps and UX flow from the same wellspring allows the practitioners of DevOps several opportunities to repair the tragedies that have already befallen our enterprises before they escalate to the Shakespearian proportions. Three broad categories of opportunities would be:

    1. Enroll UX into the DevOps movement – Knowing that we are aligned creates multiple possibilities to bring many industry and enterprise change agents into our cause. How much would the DevOps adoption curve accelerate if people like Don Norman, Jesse James Garret, Alan Cooper or Steve Krug were visible advocates for DevOps? Support from UX at the leadership level in the industry and within your enterprise is an advantage you cannot afford to pass up. The number of consensus driven organizations are just too many to count and the more groups and individuals that believe in advancing our cause, the better.
    2. Learn from UX’s travails – Creating artificial barriers to discipline inclusion is ultimately an adoption limiting factor. When we make things pass-fail or turn collaborations into confrontations or competitions for who is DevOps and who is not, we all lose out on the what a wider community could offer. Hold yourself accountable to pivot confrontations into constructive opportunities where learning can happen for people of different disciplines.
    3. Take some pages from UX’s playbook –  UX’s biggest lever for creating believers is by involving non-UXers in the research methods and activities (think about the last time you went to a usability test behind the 2 way mirror and saw a person struggle to use a system you were part of implementing). One of the best things we the practitioner community can do is to demonstrate what life is like without DevOps in real-life scenarios. When someone really attaches to how ugly a manual process is, or how soul-crushing it is to not be able to self-actualize in their job, they can’t help but want to be a part of making it better. Problem solving is hard-wired into the human condition, let’s leverage it.
  • DevOps: The Last Great Hope for Application Security?

    DevOps: The Last Great Hope for Application Security?

    Once upon a time, there was a great battle between speed and security.  Development wanted to go fast.  But, Security wanted to slow down and be safe.

    “We must protect our gilded apps”, cried the application security team.

    “Speed is cherished by our people”, declared the development team.

    For years, they endured the pain of testing late in the lifecycle, sorting through reams of false positive reports, and dealing with the added cost of pushing bad software out the door.  They knew there had to be a better way…

    And then came, The DevOps Revolution.  The DevOps team had an answer.  “Let’s bring Application Security and Development closer together — and shift its focus further to the left”.  The DevOps team knew that by introducing awareness of security vulnerabilities and policies early in and across the software development lifecycle — without creating a time-consuming tax on development — that both teams could win.

    During an evening event named “Wining Not Whining” at the RSA Conference 2014, we gathered some of the top DevOps experts and influencers and asked them “Why is application security so important to the DevOps revolution?”

    Listen to what they had to say:

  • Q & A with Kevin Behr on DevOps, its impact on security, and the challenges of bringing change to the organization

    Q & A with Kevin Behr on DevOps, its impact on security, and the challenges of bringing change to the organization

    Kevin Behr has been in IT game for 26 years and is the author of seven IT management books, including the bestselling The Visible Ops Handbook, which he coauthored with Gene Kim and George Spafford, and The Definitive Guide to IT Management, published by Hewlett Packard. His most recent book, The Phoenix Project, also coauthored with Kim and Spafford, has helped to spark discussion about DevOps globally. Behr is also well-known for his technology and management keynotes for such organizations as the National Academy of Sciences, Hewlett-Packard, the SANS Institute, AFCOM, and the IT Service Management Forum.

    Behr is also the founder of the Information Technology Process Institute (ITPI) and chief science officer for the CXO and Board Advisory Practice at Praxisflow (formerly Assemblage Pointe), where Behr consults, mentors, and coaches IT organizations and executives on how to increase their business effectiveness and competitive advantage.

    We caught up with Kevin Behr to discuss DevOps, its impact on security, and the challenges of bringing change to the organization.

    devopsy.kinsta.cloud: We’re now hearing a lot about continuous deployment, but what does that mean compared to traditional enterprise thinking about deploys? And what does it mean for IT disciplines like security?
    Behr: With continuous monitoring, you can catch failure quickly, even if the issue that caused the failure makes it into a production state. With today’s SaaS and cloud models, failures don’t matter as much because you are not shipping software as much as you are managing a service. And one fix works for all of your users instantaneously.

    This is the difference between traditional enterprise IT thinking and the thinking around continuous deployment, which really is starting to creep into the enterprise. The new thinking is: how do we do this smaller, faster, and more safe to fail. And organizations aren’t ready to make everything safe to fail, but pushing things in that direction becomes an actual security strategy—versus the notion of, let’s just see what happens.

    That’s what we’re getting into with continuous deployment. You’ve actually made the deployment safe to fail. You almost don’t need very much testing because you can actually deploy, then wait to turn on a new feature; and if it doesn’t work, you can turn that feature off. What we’re starting to see is this kind of radio button style of deployment: they’ll deploy software and it’ll be turned on progressively, versus an old-school Big Bang.

    That’s really kind of neat when you think about it. Software is being treated more like infrastructure. You hear that phrase a lot: Code is infrastructure, and infrastructure is code. You start to see the different meanings of that everywhere. And I think that’s right. I think the security practices have to be a part of the safe to fail approach. You want to have very good monitoring and detecting controls, because they’re going to help you be more resilient.

    devopsy.kinsta.cloud: I think these are concepts that may have challenges when working with some security and IT professionals and others who are used to having control over the process. What are you seeing in regard to the balance between an organization’s need to oversee and protect itself and the benefits of continuous deployment?
    Behr: You’re getting at the fear that security practitioners and others may have, that this breaks down controls that they’ve put in place. They’re realizing that it could actually add more resiliency—consider continuous authorization to do certain things, cross-functional teams, continuous monitoring, and all these things that could very possibly increase resiliency.

    You’ll see more success if security could embrace DevOps, from the standpoint of, “Oh my: This is amazing and so what are the top ten things we must do to make DevOps work? Not “here’s why it can’t work.” Security teams that roadblock like that are going to get steamrolled right now.

    But it’s not good news everywhere. I hate to say that some companies out there are more stupid than others. I’m seeing places use DevOps as a way to excuse all the reduction in force that they’re planning: “We need to get rid of five people because we’re doing DevOps.” What sense does that make? I think mini-backlashes are coming in some organizations. And one of them is going to be where organizations call things DevOps that aren’t, just because it serves their ideas.

    For instance, I’ve been into a couple of banks that have created DevOps departments, which is sacrilege to this whole movement, because the whole idea is to get rid of silos, not to create a new one.

    Some days I get bothered by the notion of organizations calling it DevOps, and then other days I don’t because it’s just the result of the success of an idea.

    devopsy.kinsta.cloud: What steps do you see organizations taking to deal with the change? Are you seeing schisms in organizations where some people are resilient to change?
    Behr: I think that the folks who own these systems that they work in have a long way to go. And I think the people who are having a hard time with DevOps are getting steamrolled right now. People in vice president of operations positions, or the director of operations: They may not love the concept of DevOps because it creates cross-functional challenges for them. The first is: “All of these people who don’t report to me and now are in my data center. What are they doing? Why are they on keyboards? How do I control this?” [Story]

    Think about it. Their whole working role is judged on this notion of reliability. Their heuristics tell them that having lots of unaccountable folks doing stuff isn’t consistent with reliability in the long term. In the shops where DevOps has been successful, the security folks have been able to get in and say: this is awesome. They love the reduced cycle times. They love that the dev guys are coming down and helping with patching. Getting things done whenever they need it. And that they actually know what this stuff means now; it means that their security feedback loops directly into dev.

    I think that’s the other thing that people don’t get. DevOps, when really done right, creates parallel feedback loops, not just serial feedback loops. In the old way, someone asks someone to do something, they do it, and then the provider asks how they liked the service? That’s the feedback loop most operations have today. But when the devs are brought in, all of a sudden the devs are adding another feedback loop. It goes something like, ”Hey, I just watched you do this thing that I don’t do every day. Man, that’s a lot of extra steps. Why are you doing it that way?”

    That’s one feedback loop that comes from just teaching somebody something new, and getting the feedback from the outside. And then there’s the feedback loop of the quality of what you delivered. When you’re getting both of those at the same time, parallel feedback loops are how evolution happens. That’s one of the hallmarks of all evolving systems and adapting systems.

    If security can become one of those parallel feedback loops, and all the documentation of what you’re doing when it happened, and all that related stuff is in one place, then security becomes part of the refactoring.

    I think the other thing is, because the dev and ops people are collaborating more, you’re starting to hear more contextualization happen with the developers. They’ll say things like, “I’ve always wondered why they yell at me whenever I hard-code IP addresses. Now I get it: because you guys have to change these all the time, because we’re now down here changing a bunch of them.” That contextualized understanding goes a long way.

    This is a huge thing. Because there’s a ton of infosec ops: patching, log parsing, and different things that happen at an operation level that are really security driven. And if you can get the devs involved in helping out with some of that stuff, they start to understand why you’re doing it. And in many cases, we’ve got technologies in place that we spend a lot of money on, and security, that are just to prevent devs from doing things that we didn’t think we could change.

    The organizations that get this—and foster it—are the ones that are going to succeed.

  • DevOps Adoption – Startups

    DevOps Adoption – Startups

    This is part 1 of a 2 part article

    DevOps is somewhat unique in that it has applications to both large enterprises and to startups.  The way DevOps is adopted in startups is incredibly different than how it happens in large organizations. The profile of how DevOps is adopted in each of these cases is the focus for this two part article series.

    In this first installment, we will study how DevOps is adopted within startups and in the second, how enterprises adopt DevOps. There are lessons for each in the way that DevOps is adopted. Organizations can leverage this information to enable better, faster, and easier adoption of DevOps.

    Much of the DevOps way is rooted in startup culture. From methodologies such as Agile, lean principles, and just in time manufacturing, startups adopted DevOps as a means to compete, innovate, and move faster. DevOps is inherently a quicker, better way to get product to market – a perfect fit for startups. They could adjust quickly, iterate again and again to get to the point where customers en masse were interested in their technology.

    In order to understand how DevOps is adopted in startups, we need to look at how startups evolve. In the high-tech world, a startup gets going when a founder or group of founders has an epiphany for a new product or service. Generally one of the founders will start to develop the solution by writing some code. The startup may add a couple of developers but it is usually just a few folks cranking out the first version of the product. There is often a business or product focused function where requirements and feedback are gathered from potential users or customers.

    In this first phase, the development team often adopts Agile as a way to quickly iterate on the development process. These short-term sprints build the initial functionality of the product. While initially, there isn’t really much to release to potential customers, the core developers are starting to build their development and release cadence. The short sprint help to keep the functionality requirements small and the team on-track to deliver a small piece of functionality in a short amount of time. This foundation also ends up being an important part of the future development process. And, just as importantly becomes the kernel of DevOps.

    As a small team building the initial releases of the product, the culture and environment of the startup is generally fast, highly communicative with little to no process. Often the development team is sitting side-by-side cranking out code quickly. No specs, just a list of what needs to be done often stored in Pivotal Tracker or some similar tool. Sometimes mockups and screens designed to describe the functionality. There isn’t a rigorous process in the choice of technology platforms – language, database, etc. are chosen often based on what the core team knows or thinks is the right solution rather than on research. This small, agile environment is critical to understand when thinking about how DevOps is adopted.

    There is little formal process at most startups. Things are evolving and moving so quickly that there just isn’t time. The goal, after all, is to get a product into market and start selling. While startups are high adopters of Agile as a development methodology, there is generally little rigor in its implementation. There doesn’t need to be because the team is so small and trying to move quickly. Adding in product management, sales, and marketing functions are often ad hoc. There isn’t a well-developed end-to-end product release process. The movement from development to operations is also less than well defined. Often, the developers building the product are the operations people deploying and managing the product or service. Process gives way to the practicality of just getting things done.

    The first phases of DevOps adoption start to occur at this early stage in a company’s life. Developers as they are building their product often encounter some basic challenges and begin to solve these issues. How they solve them often dictates whether they take a DevOps path or not. One of the first things that developers do is build an environment to test their software. Often, they will find a cloud provider and start spinning up a few instances to host their application. They also may need a database server to round out their testing. They might manually do this a few times, but as they test more and potentially on different platforms they will start to look for some tools to automate this process. Of course, at this stage, there are likely zero operations people, so the tasks fall on the developers. And, developers wanting to spend more time coding pick-up some tools to do this work.

    The tools might include infrastructure providers like AWS, Rackspace, or SoftLayer to configuration automation tools such as Chef or Puppet. There may be a variety of other toolsets chosen as well including project management tracking, libraries, language frameworks, and others. The goal with this first wave of tools is just to save time. That could be by not having to code basic functions, the setup of test environments, or manually tracking tasks. As the organization launches and brings their product or service to market, they may adopt some management and monitoring tools to help them gain visibility into their product or service. Generally at this early stage the decisions are focused on driving short-term problem solving. A developer needs a test environment quickly and repeatedly. A solution to that need is quickly found and implemented. The underlying paradigm is to move fast, solve problems, and get to market.

    While tools are hardly a complete DevOps methodology, they are the catalyst for the next step in a startup’s journey to adopt DevOps. Many of the tools chosen in this first period have APIs, can create automation, and are highly configurable. As a startup keeps moving forward and increasing their users, the flexibility of these tools and the various ways that the developers utilize them starts to drive some conversation around process. A few hiccups in the production environment with some bugs, outages, or failures, and the technical leaders begin to start to say the ‘p’ word. Process. While there may be some Agile components already embedded, the growth of the startup starts to create a desire to add in some control, accountability, and guard rails around the product development process. Many organizations will also start to loop in the business side so that there is a more comprehensive process around releasing a product to market. Agile now starts to extend up and down the organization in a more rigorous way. DevOps is starting to take hold and become more fully formed.

    The early stage DevOps lifecycle wraps up with it becoming a part of the culture. The organization now shifts into a mode where this methodology becomes a fundamental underpinning of the business. Values such as agility, intense customer focus, rapid iteration and others are a part of the fabric of the company and encompass the entire organization. Hiring decisions are made based on whether the candidates match the philosophy and culture of the company. DevOps is viewed as more than just a process that the company follows, but a competitive advantage in fulfilling its mission to its stakeholders. At this point, DevOps is firmly embedded into the organization and becomes its way of life.

    Startups are unique in their approach to adoption of technology and methodologies because of their size, focus, and evolution. DevOps adoption, as we have seen out in the wild, seems to go through some natural stages. While the stages may overlap, it generally follows a tools, process, and then culture sequence. It could be said that some startups just start with the culture of DevOps, and that can be true for some, but many that we see are often so small at that stage that the culture is largely undefined. IT leaders who understand this sequence can become more intentional about the way that DevOps is adopted, perhaps accelerating it or even shaping it earlier in the company’s lifecycle. A little bit from now, our hypothesis is that startups won’t even start without DevOps embedded into their plan.

    In part 2 of this series, we will examine the adoption of DevOps at larger companies.

  • Netflix, the Simian Army, and the culture of freedom and responsibility

    Netflix, the Simian Army, and the culture of freedom and responsibility

    What do you do if you’re entire business relies on delivering streaming video content over the Web to millions of customers, and there are no examples to follow or tools to use? If you’re Netflix, you set the example and build the tools.

    Netflix had tremendous success with its original business model shipping movies on DVD to customers in trademark red envelopes. The precision of the shipping logistics, and the efficiency with which Netflix was able to meet user demand and ensure speedy delivery is a case study of its own. But, Netflix also saw the writing on the wall and became one of the first to embrace delivering movies via streaming video over the Internet.

    Networking protocols are fairly resilient, and able to get data from Point A to Point B eventually, but streaming content like audio or video is less forgiving. The data needs to be transferred at high speed, and the packets need to arrive in order to avoid buffering or glitchy playback.

    Netflix chose Amazon Web Services (AWS) to provide the cloud server infrastructure for delivering its streaming content. Amazon is reliable and resilient, but it is not infallible. When Amazon experiences a massive outage—as it has a few times in the past—it can mean down time for Netflix customers. Netflix, however, works very hard to guard against being affected by issues like that, and—thanks to the Simian Army—it is quite successful.

    Simian Army

    I spoke recently with Ruslan Meshenberg, director of platform engineering and Josh Evans, director of operations engineering for Netflix to learn more about how Netflix has embraced both DevOps and open source to develop tools that help it deliver high quality streaming video content to customers around the world.

    Simian Army is the name Netflix has given its suite of automated tools engineered to stress test the Netflix infrastructure and proactively identify weaknesses so Netflix can resolve them. The Simian Army is comprised of tools like ChaosMonkey, ChaosGorilla, and ChaosKong.

    Josh explained that when you’re dealing with a massively distributed system like the Netflix cloud infrastructure, it’s inevitable that you will encounter bugs. Rather than waiting for issues to surface, Netflix uses the Simian Army to intentionally induce failure on its own terms, so that issues can be identified and fixed before they impact customers.

    ChaosMonkey wreaks havoc on an individual server or cluster level—randomly shutting down servers to make sure that automated resilience works, and that redundant systems pick up the slack so content delivery is not interrupted. ChaosGorilla works on a larger scale by taking out entire AWS zones, and ChaosKong takes the concept to a national level by randomly shutting down the East Coast or West Coast regions of AWS to ensure traffic is automatically redirected to the remaining region.

    There are also other tools in the Simian Army. JanitorMonkey keeps everything tidy by cleaning up unused EC2 resources. As the name implies, its job is to automatically clean up after developers. Then there’s ConformityMonkey, which checks for EC2 instances that are not conforming to predefined rules for best practices.

    Netflix lets the Simian Army roam free causing random chaos, but only Monday through Friday between the hours of 9am and 3pm when managers and developers are present to address any urgent situations that might be caused inadvertently. In some cases the tools take automated corrective action, and in some they simply generate an alert and escalate the issue to the appropriate group or individual.

    Embracing Open Source

    One of the things that makes the Simian Army unique is that Netflix created these tools out of necessity. Netflix was blazing a trail on the cutting edge of delivering content from the cloud, and there simply were no commercial tools available to do the things Netflix needed to do.

    In the early stages of scaling cloud services to meet demand, Netflix frequently stumbled and hit a variety of walls. Netflix was a pioneer in delivering content through the cloud at this scale, but the open source community has experience, and an established understanding of working with a distributed environment.

    By marshaling the resources of the open source community, Netflix was able to get more eyes on the code, and enlist the volunteer support of hundreds or thousands of developers to help refine and improve the tools in the Simian Army, and other tools Netflix has developed. Ruslan and Josh also explained that there is an unintended benefit of cooperating with the open source community: in-house developers are more diligent when they know that the code they write will be exposed to the general public.

    Netflix doesn’t just take advantage of the open source community; it also makes its tools available to the general public through the Netflix Open Source Software Center on Github. There are tools for availability (like the Simian Army), cloud management, persistent systems, infrastructure services, developer productivity, and more—all available for free for anyone who wishes to use them.

    Culture of Freedom and Responsibility

    Lots of organizations talk a good game. It’s easy to come up with a profound mission statement, or an insightful set of company values. Actually living up to them, however, is a whole different story. Netflix prides itself on building a culture of freedom and responsibility.

    The Netflix culture is part of what makes the company so successful, and why Netflix has been able to achieve what it has in the realms of DevOps and open source. Netflix believes in finding highly skilled, seasoned engineers and developers, and allowing them the autonomy to get their jobs done and own responsibility for what they produce.

    Other organizations can learn a lot from Netflix. Netflix has paved new paths for DevOps, set the bar for collaborating with the open source community, and established a culture of success that any company could benefit from adopting.

  • Google Starts Cloud World War I, DevOps Wins

    Google Starts Cloud World War I, DevOps Wins

    No matter the outcome, a cloud pricing war will leave DevOps the winner

    By now you probably saw the news and all of the buzz about how Google dropped its drawers eh, pricing on its cloud hosting. This is of course a direct shot at Amazon’s AWS. It’s about time someone with big enough stones (and a big enough data center) took on Bezos and his band of merry cloud makers.  But the real winner in all of this may in fact turn out to be DevOps.

    Of course this is just the first salvo in what promises to go down in history as Cloud War I. Amazon is not going to stand still.  They are perfectly willing to compete on price and in fact part of the AWS mystique is that it is the most cost effective solution (cheapest) out there.

    This will not be just a tit-for-tat move though. In announcing the price cuts Urs Hölzle, VP of TI at Google said, “”We think cloud pricing should track Moore’s Law, so we’re simplifying and reducing prices for our various on-demand, pay-as-you-go services by 30 to 85 percent.”  That means we are going to see cloud prices drop as the speed and power of chips goes up. That is good news! You can see Hölzle’s complete presentation including a cameo by SnapChat’s Bobby Murphy below (be warned this is about 1 hour and 20 minutes long).

    [youtube=http://www.youtube.com/watch?v=qokEYBNWA_0]

    Of course this begs the question, what about the other cloud providers.  How long can Rackspace sit up on its OpenStack perch before diving into this fray? What about Microsoft’s Azure? Redmond wants its piece of the cloud pie, does new CEO Satya Nadella have the stomach for this fight? Can they afford not to? The fact is that Microsoft has the girth and arsenal to compete here. Maybe Google starting this war is the opening they need to climb out of the tech cellar and be relevant again. Cisco just announced a billion dollar initiative to become a cloud powerhouse as well.  As big as a billion is though, that is penny stakes in this game I am afraid.

    Here is another one to watch, IBM’s SoftLayer. Fresh off of a 2b+ acquisition and pledging another billion in cloud dollars, IBM via SoftLayer has the profile to be a winner in this war as well.  They also have IBM’s minions out pushing the solutions. In fact Microsoft and SoftLayer/IBM may have the channels to win this thing in the long run (OK maybe not).

    So where does this leave DevOps? Think Switzerland.  From a DevOps perspective it really doesn’t make a difference which cloud provider emerges dominant. The cheaper and more attractive cloud becomes the more it will drive better DevOps tools, DevOps culture and philosophy.

    Of course there are those that say that if PaaS becomes the dominant cloud platform (and I think it will), that could spell trouble for DevOps.  As if somehow PaaS doesn’t need DevOps.  I say poppycock to that too. Maybe you won’t configure servers in a PaaS environment (and I am not conceding that either), but there will be plenty of places along path that will call for automation, continuous delivery, continuous monitoring and continuous management.  PaaS does not obliviate DevOps.

    That is not to say that applications are not king.  Applications are driving the cloud, DevOps and everything else tech today. A large reason why Google is so goo-goo over Cloud is because it drives more Android apps.  I think we will see more DevOps tools and practice around application development, configuration, management and monitoring. New Relic is already on it.  Anything related to Docker stands to do as well as munition and tank manufacturers did during WW II.

    So buckle in as it may get rough out there as these tech giants battle for cloud supremacy, there an will be winners and there will be losers.  It won’t be just the cloud providers either.  Wars create opportunity and those willing to take the chance will gain the most.

    But for DevOps it is all good. It is like we have a stash of chocolate and gold while we make watches here in Geneva. The war rages on around us but we need at the end of the day no matter what we win.

  • DevOps Is For Horses: Stop Making Excuses For Starting

    DevOps Is For Horses: Stop Making Excuses For Starting

    While DevOps practices may be spreading throughout organizations both large and small, often times the most visible success stories are also the most exceptional. Organizations like Netflix, Etsy and Google are brought up in conversations about DevOps time and again because they offer the most stark examples of the benefits that the model can bring to businesses. But the scale of their operations can often be intimidating to the everyday IT person.

    “People will tell me, ‘Well, these are unicorns—special companies. We’re not like that. What works there will not work here,’” says Jez Humble, a principal for the consultancy ThoughtWorks and co-author of Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. “I think that’s a fallacy.”

    According to Humble, two phrases that he hates the most as excuses for not adopting DevOps or continuous delivery principles are ‘That can’t work here,’ and ‘But we’ve always done it that way!’ He’s one of a growing contingent of DevOps pundits who argue that it is plenty possible for even the most run-of-the-mill organizations to engage in DevOps.

    “DevOps is also for horses,” says Gene Kim, author of The Phoenix Project. “In fact, every unicorn was once a horse. Amazon in 2001, Google in 2004, Twitter in 2009 and LinkedIn in 2010—they all had huge monolithic code bases and inconsistent deploys.”

    Humble agrees that these so-called unicorns “have the same problems as everyone else.” But no matter how remarkable the scale of their solutions to those problems, other organizations should be able to learn at least one very valuable lesson from their case studies.

    “The critical piece is they’re always working to improve what they do, and that’s something that works everywhere,” Humble says.

    That’s Nick Galbreath’s philosophy. As the former director of engineering at Etsy, Galbreath saw first hand the transformation that the organization was able to make using DevOps patterns. He’ll assure any organization that it didn’t all happen in a day. Which is why when people ask ‘How do you get started with DevOps?’ his answer is simply to start.

    “If you have a sympathetic management ear, ask ‘Why don’t we try to cut deployment time in half?’” says Galbreath, who is now the vice president of engineering at IPONWEB philosophy. “Because the other way is to say ‘Deployment time takes too long, let’s do half as many.’ Wrong direction.”

    Setting up a goal like cutting deployment times in half will get an organization streamlining and weeding out junk processes. It’ll also have the team working to move those processes from inside a select few peoples’ heads and into things like a batch script or even just a simple document so that they’re easily repeatable across the team and that more can be done in parallel.

    “Cutting that work that adds no value in half, you’re really going to empower your group and your team to come up with innovative solutions,” he says.

    In a similar vein, DevOps for normal folks could also be chunked up incrementally on a project-by-project basis or automation tool-by-automation tool basis, says Josh Corman, CTO of Sonatype.

    “You don’t have to do an enterprise-wide shift hard left to DevOps,” he says. “You can do it in little projects or little small automations that free up time so you can achieve more competence.”

    In addition to incrementalism, Humble suggests providing a leadership environment that encourages taking chances and learning from mistakes.

    “Create safe situations so that when people make mistakes we can actually learn from them rather than engage in finger pointing,” he says.

  • DevOps Contrarian

    DevOps Contrarian

    I admire and appreciate the ideas of DevOps as a result of living through two decades of working in IT and experiencing the pain of what I would call the “failed state of IT”. The failed state of IT being something inherently broken, something that couldn’t survive on its own, something that eats its budget and provides no value and something where the people working in such a system don’t even see the worth of the system and stop challenging it all together. In order to correct or prevent these “failed states” I often apply my experiences learned through DevOps and Lean processes and values.

    DevOps to me has been about recognizing and pulling yourself out of this “failed state” concept of IT and doing so with a little bit of empathy, a little bit of common sense and a lot of science. I don’t necessarily mean pure science in trying to come up with an underlying law that describes and predicts your system of work, but the foundations of science – The ability to question everything, the ability to collect data, analyze data, measure data and develop analysis thereof. By thinking “DevOps Science” you choose to develop models that describe your work, define your flow, show the value of your system and tailor it to your business needs and to research the best models to achieve your end goal and strive to continually improve those models. DevOps is about putting this science to work, and doing so with collected intelligence & experimentation behind it.

    Where I find myself being a DevOps contrarian of sorts is in the often over simplified representation of DevOps as an applied model. I don’t have any issue whatsoever with the concepts described below by themselves; they create amazing discussions and speak to very core foundations of DevOps, but they’re often only spoken of superficially.

    DevOps is not just about Culture.

    I often see people speak of DevOps as an applied “Cultural Invention”, an innovation found to be useful to a group of people in their behavior. While that is a great description from a 30 foot view as if you’re looking from the outside, it really doesn’t help one understand culture in any sense to help foster it. DevOps culture isn’t about copying what others do and hoping for success, it’s about developing it within realizing that its affected by both forces encouraging change and forces resisting change and finding the balance there of is that is important. You can’t simply force DevOps on an organization and call that culture. The real power of  DevOps as a cultural invention is being able to develop DevOps values that directly apply to your organization so that it can be passed down generation to generation and survive on its own.  Thinking scientifically, we learn from these shared examples of cultural invention and their successes by experimenting with them to see how they fit our organization as there is no universal law of culture but merely models we can research and identify with. It’s great to build upon the success for others, but the only way to validate those successes is to test them yourself!

    Another danger of thinking applied culture is the development of cultural optimums, I parallel this to what Garret Hardin coined “The Tragedy of the commons”, whereby individuals, acting independently and rationally according to each one’s self-interest, behave contrary to the whole group’s long-term best interests by depleting some common resource. I won’t claim to define what your resource is that you’re depleting, but it’s something to keep in your mind as you think of culture because it’s important to understand culture external of yourself as much as it is within. The value of culture isn’t in how you rationalize it for your own goals, but how it operates on its own and if that culture is appropriate for the good of the entire system (business). The strongest metaphor in DevOps for depleting the wrong resource is wasted resources not working towards your common goal. Are you actively engaged on the right tasks? Right automation? Right flows? Are you merely creating more work or are you adding value? DevOps isn’t something you apply methodologically, its something you experiment with methodologically.

    DevOps is not just about automation.

    Automation is the process for operating with minimal or reduced human intervention.  If you’re focused on automating processes are you still improving that process or did you think to yourself “I automated that, now I’m free to work on other stuff”? Automation can tempt you into doing “more work” if not kept under control and that creates a real danger of losing focus, having too much work in progress and actually hurting your overall performance.

    Often times the best value in the systems we call “Systems Automation“ – the Chef, CFEngine, Puppet, Ansible & Saltstack systems  is not necessarily their amazing capability to automate processes but their ability to document processes. Those cookbooks, manifests, scripts and DSL’s you’re using are absolutely the best documentation of your system you can have. They provide a great resource when done in such a way to easily describe the processes that they’re automating so that they’re well enough documented for improving them. You can’t automate that which you don’t understand and the best way to understand them is to make sure the processes that describe them are easily readable, documented, tested, and validated. They should be part of your experiment, not just testing for failure but testing for improvement.

    Through the lens of science, we can look at automation more as standardizing on a process and applying the best resource to that process.  Sometimes it is compute power instead of human power  but don’t fall into the trap that the process is said and done (Automated all the things!) and time to move on! Science is a never ending process by where you build upon your conclusions!

    One conclusion I and many others have come across is that people are the most flexible resource you have! It may be a better value to your business to have the flexibility of a human resource if the process you’re trying to build requires flexibility. Even when planning for automation, requiring human intervention is a perfectly valid result and a justification you can sell to your business that has been proven through experimentation & data. Don’t ride the automation bandwagon when your data shows otherwise!

    DevOps is not just about measuring.

    What are you doing with the data you measure? What value does measuring give your organization? When you have the flu and you take your own temperature, what are you measuring for? Do you have the cultural aptitude in your organization to accept the ramifications of what you measured? Can your organization stop working when it’s “sick”, take the time to heal and start back up after taking corrective or remediation actions?

    Measuring is extremely important, but measuring isn’t what DevOps is.  Measuring is a part of what makes your organization responsive, culturally adept, and accountable for itself. It’s what allows you to convey value, to compare itself against and strive for improvement. If you’re not measuring, what are you improving against? Inversely, If all you’re doing is measuring, what is the value of your measurements? What are they the basis of? I like to think of biological systems when I start thinking of computing systems where there is a cost of energy to do something. Are you utilizing that cost the best way you can by measuring what you are or should you evolve the system to spend that energy elsewhere? Are you collecting the right metrics to even comprehend your systems? Do your metrics allow for capturing exaptation and evolution thereof? When you experiment, measure, rinse & repeat you get patterns; measuring helps identify those patterns. Creativity, experimentation and the process of science allow you to act upon those patterns and create value from those patterns. Some people call that innovation.

    DevOps is not just about sharing.

    There are people who believe that unless you are open source, have a GitHub repo and publish your code, you’re not really sharing and you’re not DevOps. I certainly hope we’re not going down this path and certainly hope we can appeal to the broader ideas of sharing. Sharing is about the joint use of a resource and sharing is knowledge charity. Sharing goes hand in hand with culture in not just expressing your needs and sharing your knowledge, but having the ability to listen, feel empathy and accept what others share as well.  When it comes to collaboration & sharing it’s much easier to accomplish if everyone sees it through the lens of science, not only do you speak to DevOps values more rationally but you can even accept opposing views and ideas much easier because you have evidence to them and aren’t emotionally tied to their success, you’re tied the process that challenges their success as a way to improve them!

    DevOps through the lens of Science.

    The beauty of DevOps through the lens of science is that science is about challenging ideas,  accepting failure, learning from history/prior experiments and having the strength to not only say those successes and  failures are a valid output of your experiments but also come up with a new experiment or new hypotheses and do the process all over again. This helps define your culture, this helps setup what you should automate, this helps you validate what to measure and this develops what you should share.  Through the lens of science you learn through experimentation, testing & verification what works for you, what creates value, what brings you closer to your goal. When honestly done you also learn what doesn’t work for you, where you failed, and how you can use those lessons in failure to continuously improve.  A failed hypothesis after all is still good data!

    Why so much focus on the sciences? Personally, I enjoy astrophysics not because there is direct monetary value in understanding the cosmos around us but because of the philosophical nature of being, the beauty of knowing and the questions it gets you asking as the more you know, the more you wonder. That wonder, is what makes DevOps so great to me. That wonder is what drives continuous improvement. That wonder is what drives the “what next…”. That wonder is what pushes “question everything”. That wonder is always what pushes me to accept and understand the science we do know and trust it because it’s based upon the values that are respected by thousands of scientists all over the world through peer review and consensus. That foundation is not the assumption that we think we know it all, but that we know enough to describe something and we trust that description good is enough that we challenge you to break it.  I would highly recommend that any DevOps practitioner cross pollinate their technology passion with the sciences as there is nothing more exciting than realizing commonalities of biology, physics, astrophysics and the laws of nature within your own thoughts and patterns as they’re applied to DevOps.

    So be a DevOps contrarian; for its not accepting DevOps as the answer, it’s the challenge there of that will make it aspire to bigger and better things. It’s not that Culture, Automation, Measuring and Sharing is wrong by any means, but they’re just scratching the surface of what you need to understand to make DevOps “Work” for your organization.

    I’m not only a DevOps contrarian, but I’m an OS contrarian, a Developer contrarian, a Systems Contrarian. I’ve made the best of running Windows, Oracle Linux, Oracle Database, eBusiness Suite, Weblogic and much more. Not because I’ve personally identified or I am idealistically aligned with any of these tools & systems, but because they solve the needs of the business; they create value, they operate and perform admirably and we have proven their worth. It’s no utopian system by any stretch of the imagination, but I find great reward in the continuous experimentation of making these systems work and improving upon them in a very DevOps way.

    After practicing DevOps & Lean principals and having  over two decades of IT experience, I’ve realized the “failed states of IT” and people who thrive in them are people who stopped critical thinking, stopped challenging the status quo and stopped experimenting.  They stopped having a contrarian.

  • Config Management & F***ing Shell Scripts

    Config Management & F***ing Shell Scripts

    Joke tech projects pop up all the time. Some, like the lamentable titstare, should never have seen the light of day. Others, like Mark Zuckerberg’s Facemash start out as a joke and grow into something that is anything but. Most come and go without anyone really noticing.

    Whilst their lives are typically short we often learn more from people’s reactions to these projects than we do from their use of them. An excellent example of this popped up recently when a couple of frustrated Chef and Ansible users came up with a simple configuration management alternative, the regrettably named F***ing Shell Scripts (FSS). As they explain it, as in “Why can’t we just use f***ing shell scripts?”.

    Whilst usability issues with configuration management tools – particularly in the Enterprise – is a topic close to my heart, I would not normally have had much more than a passing interest in FSS. What grabbed my attention was the reaction it elicited when posted to Hacker News. With over 150 comments FSS clearly touched a nerve.

    Now to clarify, the authors of the project were quite definitive in assuring the mob that they had created FSS as a joke:

    We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time.

    People either missed this, or chose to ignore it:

    “This is simply brilliant! I’ve dreamt about such tools.”
    – artellectual

    “I’d love to see a tool like yours be brought to full maturity, then I think I’d love to use it.”
    – sdegutis

    It provided a perfect platform for many to voice their frustrations with current tools:

    “Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days.”
    – mattjaynes

    “In my experience, there’s a huge amount of work involved in just getting Ansible (and other CM systems) to work right in the first place”
    – s_kilk

    “We’re using Chef right now, and evaluating other options. They seem to either be between way too over-engineered, or not flexible enough, but nowhere in between.”
    – sdegutis

    “Keep it simple. puppet/chef have horrendous DSL’s that make it really complicated to reason about.”
    – sp1982

    In the interest of balance I should point out that there were some voices of support amongst the crowd. Ansible in particular won praise for the effort its team has put into simplifying things.

    Overall though I was still blown away with the level of frustration being expressed. Automation tools have been around for years now; CFEngine, the grandfather of the modern tools was created in 1993. Despite this, a tongue in cheek project suggesting ditching them all in favor of the technology they set out to replace (or at least abstract away), shell scripts, is considered credible. For me this is akin to people ditching virtualization for physical boxes. It just doesn’t make sense.

    Unless of course something is still fundamentally broken in the approach the current set of tools is taking; that they are still too hard; that they still fail to address key prerequisites like visibility and accountability. I believe that this is the case because I’ve experienced it first hand; I’ve seen it a hundred times in businesses I’ve talked to, and threads like the Hacker News one mentioned above leave little room for alternative interpretations. Heck, we’ve based our product GuardRail on this entire premise; these are not empty words on my part.

    Incremental improvements such as those offered by Ansible are a good thing, but what people seem to be searching for is a step change. As things stand this is less likely to come from the current batch of tools, and far more likely to come in the form of a new approach. Docker anyone?