devopsy.kinsta.cloud

Category: Features

  • DevOps killing outsourcing? Another point of view

    DevOps killing outsourcing? Another point of view

    After reviewing the articles (here and here) on DevOps driving the end to outsourcing I felt it would be constructive to provide an alternative point of view.  The premise of the article is that because DevOps requires close collaboration between development and operations that it would be difficult if not impossible to effectively implement DevOps in a heavily outsourced environment.  Additionally the productivity advantages of DevOps would help eliminate the need for outsourcing.  Therefore, organizations wishing to see the advantages of DevOps should move away from outsourcing.  While I can see the advantages of that approach, I also think that companies that have to live with outsourcing can’t ignore the benefits from implementing DevOps principles with a well-designed deployment pipeline.

    From my perspective, companies looking to implement DevOps first need to start by understanding their architecture because it has such a large influence on how to effectively implement DevOps.  If they have loosely coupled architectures that enable small teams to independently develop, qualify, and release code then they should look to the world of unicorns for how to implement DevOps.  They should look at integrating operational people onto the development team and work to evolve the culture by blurring the lines and responsibilities between the two groups.  This feels like a fairly straightforward process that many unicorns have proved to be very successful.  For these types of organizations, I tend to agree with the articles recommendations that DevOps would work better without outsourcing.

    The problem is that most large traditional organizations looking to implement DevOps don’t have the benefit of an architecture that enables small nimble teams to work independently.  They typically have very tightly coupled legacy architectures that require 100-1000s of engineers to work closely together to develop, qualify, and deploy complex integrated systems.  Ideally, they would re-architect their applications so they could see the benefits of nimble independent teams.  The reality though is that this is just not practical for most traditional organizations in the short-term.  Additionally, they can’t wait for the re-architecture before they start realizing the advantages that DevOps principles can provide with a well-designed deployment pipeline.

    For these organizations implementing DevOps principles (the ability to release code to the customer on a more frequent basis while maintaining or improving stability and quality) is more about creating a well-designed deployment pipeline that builds up a more stable enterprise systems on a regular basis so it is much easier to release the code on a more frequent basis.  This is done by creating a deployment pipeline that integrates the code across the enterprise system on a much more frequent basis with automated testing to ensure that new functionality is not breaking existing code and the code quality is kept much closer to release quality.

    From my perspective this approach to DevOps is rather different from the more unicorn type approach described in this article.  It though does address the biggest opportunity for improvement that does exist in more large traditional organizations which is coordinating the work across teams.  In these cases the working code in the deployment pipeline is the forcing function used to coordinate work and ensure alignment across the organization.  If the code from different teams won’t work together or it won’t work in production, the organization is forced to fix those issues immediately before too much code is written that will not work together in a production environment.  Addressing these issues early and often in a deployment pipeline is one of the most important things large traditional organizations can and should be doing to improve the effectiveness of their development and deployment processes.

    These benefits are even more pronounced when working in organizations that have outsource development across multiple companies around the world.  Ensuring the different vendors don’t go off track and create code that won’t work together in production is critical and a well-designed deployment pipeline is a critical tool for coordinating development across different outsourced organizations in traditional tightly coupled systems.  For these types of outsourced organization, DevOps principles while different are almost more important than they are for companies that do all their development internally.

    About the Author/ Gary Gruver

    gary gruverGary Gruver is an executive with a proven track record of leading Large-Scale Agile transformations and co-author of “A Practical Approach to Large-Scale Agile Development” and the new book, “Leading the Transformation: Applying Agile and DevOps Principles at Scale.” He is currently working to help organizations with their transformations after successfully leading two large transitions at HP and Macy’s.  As the Director of Engineering at HP, he led a large organization (400+ developers) on a journey from waterfall to agile development. At Macy’s.com, as the VP of QE, Release, and Operations, he led the transition to Continuous Delivery.  Reach him on twitter at: @GRUVERGary

  • Will DevOps Kill IT Outsourcing?: Part 2

    Will DevOps Kill IT Outsourcing?: Part 2

    Part 2: Continuous delivery tool chain efficiency > Outsourcing savings

    (Check out Part 1 of the series Will DevOps Kill IT Outsourcing?)

    Experts on digital disruption and DevOps practices are vocal in their belief that DevOps and traditional outsourcing models don’t mix well.

    “Adopting DevOps requires organizations to erase the barriers between development and operations teams,” says Dan Cornell, CTO of Denim Group. “This is hard enough to do within a single organization and nearly impossible to do across organization boundaries.”

    If push came to shove and an organization needed to justify an either-or decision for DevOps over outsourcing, the argument would likely revolve largely around business value. Continuous delivery makes it possible to move to market more quickly, develop products better aligned to customer sentiment and kick the pants out of competitors who don’t do the same. That was the impetus behind the shift at GE Capital. Last year the firm’s CIO Jim Fowler said he was looking to significantly reduce IT outsourcing and shift toward more Agile teams. Earlier this summer Matt Merchant, former global managing director of DevOps for GE Capital gave devopsy.kinsta.cloud a snapshot into how this is driving innovation. (Ed. Note: Merchant suddenly passed on this summer. Our interview was just prior to this tragic event.) He explained how the company was able shift customer-facing mobile app development from a six- to eight-week release cycle to twice weekly.

    But the decision to dump outsourcing in favor of supporting DevOps practices within a more centralized IT group doesn’t just have topline revenue and innovation benefits. The upside presents itself in the bottom line as well.

    In many ways, we can chalk that up to the fact that the automation and cloud technology that facilitates continuous delivery is light years ahead of where it was when belt-tightening outsourcing deals crescendoed following the 2008 financial crisis. As things stand, the cloud has enabled IT to decouple infrastructure outsourcing from operational and process outsourcing.

    “Companies today realize they can achieve many of the cost efficiencies that outsourcing was originally intended to achieve by leveraging cloud technologies,” says Thomas Enochs, vice president of customer success at Chef . “There is a general trend towards leveraging cloud services rather than cheap labor to achieve the scale, speed, and flexibility they desire.”
    The fact is that many an outsourcing deal was born in a day when the tools of development were slow to deploy and expensive.

    “DevOps practices and modern software tools have drastically reduced the time and cost required to develop custom applications,” says Bernard Sanders, CTO of CloudBolt Software. “In the early 2000s, the step of just getting the servers and infrastructure set up for a new app could be a multi-week or -month chore.”

    Virtualization vendors, cloud providers and self-service automation have all made it possible for developers to get easy and affordable access to the infrastructure they need when they need it, Sanders says. And then there’s the advances in deployment pipeline automation that are seriously streamlining hand-offs over the course of the software development lifecycle. It’s all adding up to added efficiency and reduced cost of development and deployment.

    “In the future, the overhead required to build custom apps will be driven to near zero, and enterprises will be able to in-source this development, retaining this function that has become key to enterprises’ brand identity, product offerings, and customer experience,” Sanders says.

    Even now, though, the DevOps toolchain is completely blowing up a lot of the financial models that initially drove IT departments to aggressively outsource. While many outsourcing deals continue to be renewed without reviewing old assumptions, this is starting to change.

    “We have seen cases where outsourced functions like automating test or other processes has been brought back in house because equipped with the right approach and tooling, the internal team could achieve better results, but also because the automation work was a key building-block to a longer-term DevOps plan,” says Alex Henthorn-Iwane of QualiSystems.

  • Part 2: Meeting relentless mobile app demand

    Part 2: Meeting relentless mobile app demand

    As we discussed in  Meeting the Relentless Mobile App Demand of the Enterprise, the enterprise thirst for mobile apps will far outstretch what enterprise internal development teams will be in a position to produce.

    Developers will need every edge provided to them in order to help their enterprise success. And there’s no doubt that IT teams need to be agile and move as quickly as possible and this is as true with mobile development than anywhere. By not being able to meet the increasing demand for enterprise mobile apps, enterprises can’t be as effective and productive as they need to be. That creates substantial business opportunity costs.

    Here are some ways I think enterprises can improve the mobile app development efforts, in addition to what Gartner suggested, detailed in our part 1:

    Adopt agility. Enterprise IT is changing rapidly. The tight collaboration with development and operations teams, drive for agility, continuous development and other approaches that are increasingly mainstream are pushing applications out into production, and updating existing ones, more quickly than ever was before possible. Not only do successful DevOps organizations have short app backlogs, they are more innovative. Our special guest contributor Matt Selheimer wrote am insightful series that culminated with DevOps with purpose: ACT-ing with greater agility that details how to be more agile through DevOps practices.

    The more agile your organization, the more swiftly they can re-adjust and move always be producing the apps that matter and those that will drive the greatest business impact.

    Leverage APIs. Instead of building apps from scratch and creating more data silos, take a page from Gartner’s advice on mode 1 which is building infrastructure and APIs. Most existing large enterprises have enormous data and application and database silos. One effective way, when possible, to unleash that data and reduce silos is through APIs. APIs are the most straightforward way to build the capabilities the business needs through    extending the utility of existing applications.

    Leverage the cloud to the hilt. Whenever possible, it’s time to go cloud first. It’s time to gate legacy on-premises systems and maintain them until they can be displaced. This will include public, private, and hybrid cloud architectures – but the more rapidly legacy on-premises systems can be retired the more agile IT can be.

    Our post, Three tips for maximizing efficiency in the age of the hybrid cloud environment, provides insight into how to manage hybrid cloud environments.

    Choose cloud and development platforms carefully. There are a number of other critical platform questions, too: Can the data be provided wherever the user needs it? Does it help alleviate security, privacy, and regulatory compliance demands? And can the platform help with role-based identity management and provide for auditable access control?  All of these things being provided as utility help expedite application development and clear your app backlog.

    The more people in your organization who can develop the better. This could come in the form of training non-developers to build some of the app capabilities they need in cloud platforms and with very high level coding tools. It’s easier now than ever before for anyone to create business process workflows with hardly any code.

    Be MVP lean. The Minimum Viable Product (MVP) is a viable way to identify mobile apps that are in need and will likely provide the most immediate value. Simply defined, the MVP consists of what is only necessary to get the task done and little more. If the app is successful, it can be improved upon and iterated based on user feedback and business need.

    One way DevOps helps to achieve this is through splicing development time so many apps can be worked on incrementally, and the MVP concept is ideal for this type of workflow. It makes it easy to re-prioritize apps and new incremental work on existing ones based on pressing business priorities.

    I’m sure there are other ways to help make mobile app development efforts both effective and strategic to the enterprise rather than fire-fighting app demand. If you have any ideas, we’d like to hear them in the comments below. What strategies, tools, techniques are your organization using to increase mobile app development efficiency?

  • Will DevOps Kill IT Outsourcing?: Part 1

    Will DevOps Kill IT Outsourcing?: Part 1

    Part 1: DevOps and outsourcing like oil and water

    After nearly two decades of steady IT outsourcing for the sake of savings and efficiency gains, the shine seems to have worn off from this once glittering phenomenon. And according to many experts, DevOps and continuous delivery practices are playing a big part in many enterprises’ plans to reverse course on outsourcing key functions in their technology stack.

    “DevOps transformations emphasize collaboration, which is often at odds with outsourcing arrangements,” says Jeff Gallimore, founding partner at Excella Consulting. “Outsourcing arrangements often reinforce stovepipes and silos rather than break them down. And silos are speed killers and barriers to improvement.”

    According to a report out earlier this spring by outsourcing research firm Information Services Group (ISG), average outsourcing contract values saw their worst first quarter in a decade, dropping by 27 percent year-over-year. Big deals over $30 million were particularly hit hard, declining by 25 percent in both number and value.

    Even without the tidal wave of Agile and DevOps practices seeping into enterprise IT over the last few years, many organizations are already predisposed to take a second look at their outsourcing contracts, says Gallimore, who sees time and again that many businesses aren’t getting the value they thought they would from their outsourcing deals.

    “I hear many more negative experiences organizations have had with outsourcing arrangements than I do positive ones,” he says. “Poor quality. Higher-than-expected management overhead required. Inflexibility. Process challenges. All are reasons cited for why the arrangements didn’t live up to expectations.”

    The anecdotal stories are also piling up of big name enterprises pulling more of their IT resources back in house after years of outsourcing. For example, General Motors made waves three years ago when its CIO Randy Mott said he would completely invert the outsourcing ratio the car company had by 2017. The firm continues to make headway in changing from 90 percent outsourced IT to 90 percent insourced, with many of the positions advertised these days focused on DevOps and innovation-related roles. Similarly, David Smoley, CIO of pharmaceuticals firm AstraZeneca, went public with his goal to shift the company from 70 percent dependence on outsourcing to 30 percent, along with a commitment to Agile and lean development.

    As organizations increasingly witness the business boosts gained through a more collaborative DevOps delivery model, they are seeking ways to adopt continuous delivery practices across their entire software delivery lifecycle. But they’re finding that these practices are often wholly incompatible with the outsourcing arrangements they’ve developed, says says Jonathan Braunhut, chief scientist at KEMP Technologies. As he explains, many organizations will need to “unwind” these arrangements if they want DevOps to work well because in many cases the contracts and SLAs that govern them are inextricably tied with waterfall development.

    “Outsourcing in an enterprise setting is often implemented in a way that advances outdated waterfall-like practices and processes that are antithetical to continuous delivery,” Braunhut says. “The very wall between dev and ops that the DevOps movement seeks to eliminate can become codified in outsourcing agreements, as a reasonable means of achieving an enforceable and accountable contract.”

    More specifically, outsourcing tends to fly in the face of DevOps’ lean ethos, which is to “minimize handoffs and waste and increase quality by thinking about the overall system and flow,” says Gallimore, who believes that traditional contractual arrangements simply can’t operate with the kind of agility necessary to establish speedy and iterative cycles of continuous delivery.

    Perhaps even more fundamentally flawed in the outsourcing relationship is the idea of ownership. As many organizations finally recognize the value that IT brings to the table—more likely now to serve as a revenue generator than a cost center—it doesn’t make as much sense to give up so much ownership over the technology and tech-enabled services that could potentially transform the business.

    “To be most effective, enterprises need to own the transformation, and it has to be a truly collaborative effort across disciplines–business leadership, engineering, system administration, security & compliance,” says Thomas Enochs, vice president of customer success at Chef. “Collaboration and transformation are difficult to achieve with outside third parties or vendors. They need to own and drive the change themselves, and not be dependent on others.”

  • DevOps Dozen – Nominations are now open

    DevOps Dozen – Nominations are now open

    To succeed in todays speed of business, app-centric world the old ways of doing business just don’t work anymore. IT and development teams are being pulled, pushed, stretched and changed into new combinations and configurations. Greater cross-functional cooperation is a must, a culture of cooperation and a new generation of tools that facilitate these are changes are sweeping in like a Tsunami.

    DevOps has appeared like a superhero fighting a villain to combat these issues. Featuring a one-two punch of continuous delivery and continuous integration with a finishing blow of automation, DevOps enables easier and closer collaboration between development and operation teams.

    Whether they be cloud-native “unicorns”, traditional enterprise “horses” or born on the run startups, DevOps is becoming the “way they do things.”

    The field has matured enough where we think it is time to recognize some of the best and brightest.  Our first venture with this is our “DevOps Dozen”  The DevOps Dozen will feature the top 12 vendors in the DevOps space. I know 12 isn’t a lot given the amount of companies in the mix, but we had to start somewhere.

    So how do we get to a dozen from such a large and worthy group? We at devopsy.kinsta.cloud have compiled a list of what we think are the best known DevOps companies. Some of these companies have their product or service listed as well to help identify the face to the name. We have broken them down into several categories to hopefully make it easer to select (tip of the hat to XebiaLabs and their Periodic Table of DevOps for the help with that).

    The DevOps Dozen selections will be made by you our readers in two phases. First we will open the whole list as well as any write ins for those we may have missed. This nomination voting will be open until August 12th. At that point we will take the top 24 company/products who garnered the most nominations. These will be the finalists and we will have a 2nd round of voting for final Dozen.

    Feel free to tweet about the DevOps Dozen using the hashtags: #DevOpsDozen and #DO12

    Please click here to vote for the finalist nominations

    or fill in below

    Create your own user feedback survey

  • The Calculus of DevOps

    The Calculus of DevOps

    This is the second in a series of blogs, partnering with Lee Reid, who wrote the first installment “The Simple Math of DevOps” and Sanjeev Sharma both of who are outstanding DevOps Thought Leaders from IBM.

    I am picking up on Lee’s mathematical DevOps model to apply a calculus perspective, a “calculus of DevOps” if you will, (OK, not quite core Big Bang Theory Calculus but rather approaching it from the aspects of engineering queuing theory).

    One of the cultural challenges that DevOps poses is, that some aspects of it run counter to engineering principles that many of us learned with respect to capacity and response time.   When I worked at Bell Labs, we would apply queuing theory to determine how to best package information to be sent across the network. A basic premise revolved around a tradeoff between throughput and response time. On one hand, the more messages that were packed into the packet, the more the payload would improve capacity because the overhead of processing the packet of information would be spread across the messages.   However, waiting to accumulate enough call processing information meant that the packets would be transmitted less frequently, increasing the response time.

    A simple example of this would be making trips to the grocery store. If one wanted to spend the least amount of resources (time, gas, car wear, etc.) on shopping, then one would make less frequent trips and then buy a larger amount of food. So if the Smith family made one trip a month and spent 2 hours, then their average time per week spent on shopping would be 30 minutes.   If the Jones family went every week and spent an hour, then they would be spending twice the time and four times the gas on grocery shopping. However if the Smith family run out of an item (say cereal), they would have to wait, on average 2 weeks before this was replenished. The Jones family average on any item that they ran out of would be 3.5 days in comparison.

    So when the 2014 DevOps report was published[1] which states that areas that deploy more frequently get customer feedback more quickly, that is to be expected. But what’s not expected is that these organizations also have higher productivity.   So exactly what is happening here that seems to be in a basic conflict to the principles of throughput and response time?

    Keeping up with the JONESES

    If we think of a pre- DevOps state for a medium to large enterprise, it may be typical for applications to only be released (deployed into production) a few times a year to provide business functionality. (I assume there are more frequent releases for “unplanned work” like defects and other changes or operational work). For the sake of this discussion, let’s call this the Smith Application. So from a business perspective, the business wants to pack as much work into the Smith releases as possible given they only have a few cracks a year at getting new business functionality into production.

    SMITH APPLICATION CHARACTERISTICS:

    • Deliver Business Value 3 times a year (every 4 months)
    • Cost is C1
    • Business Value (BV) delivered is V1
    • Average time from concept (e.g. Story) being ready to be pulled from the backlog to the time it is delivered to the customer is 2 months plus any additional wait times (e.g. prioritization)
    • BV delivered per year is 3*V1
    • Cost per year is 3*C1

    Now let’s transport over to the Jones Planet where the application is released into production with new business capabilities at the end of every 2 week iteration.   In this case the business is more concerned with providing a constant demand flow into IT than they are with debating over whether a given feature/story is going to be included in this particular release.   Instead there is a constant prioritization (or continuous planning) process that ensures the highest value work is being fed into the IT pipeline.

    JONES APPLICATION CHARACTERISTICS:

    • Deliver Business Value 26 times a year (every 2 weeks)
    • Cost is C2
    • BV delivered is V2
    • Average time from concept (e.g. Story) being ready to be pulled from the backlog to the time it is delivered to the customer is 1 week plus any additional wait times
    • BV delivered per year is 26*V2
    • Cost per year is 26*C2

    Based on the previous engineering analysis, we would expect the following:

    1. SMITH annual cost (3C1) < JONES Annual Cost (26C2)
    2. SMITH annual BV (3V1) > JONES annual BV (26V2)

    So the tradeoff (we would believe) JONES is making by delivering more frequently is that they are spending more money and are delivering less business value on an annual basis.

    However the data from the previously referenced DevOps report is just the opposite. In fact the findings are that enterprises that deploy smaller units of work more frequently have much higher quality and much higher productivity (Capacity). So what’s going on here?

    A NEW PARADIGM

    If you go up to any IT team (even an agile one) and ask them why they don’t deploy business value more frequently, they will generally refer to the following types of wait states of waste that they encounter:

    • Waiting for work (lack of consistent flow)
    • Waiting for somebody else to do something (dependencies from other teams)
    • Waiting for environments (contention)

    One major root cause of this is the dependencies that arise from large enterprises packing a lot of work into infrequent releases. From a planning perspective, the process is elongated due to prioritization discussions because being on the wrong side of the line can mean delay amounting to months, in delivering a product to a customer.   Ask a business portfolio leader what their highest priority is for IT and I am guessing they can tell you in a matter of seconds. However, most large organizations can spend several months a year planning their annual portfolio of work for IT. Some organizations spend more than 50% of their new project build time prior to a single story entering an agile development team’s backlog!

    From a development perspective, packing a lot of work into releases means many more dependencies that have to be managed across initiatives and impacted applications. From a testing perspective, downstream environments (ST, UAT) have to manage much more information in terms of application versions and especially test data that complicate and slow down the process. The complexity of test data increase is proportional to the number of pairwise relationships that exist within the datasets required for testing.   As such, the testing complexity will increase non-linearly (perhaps proportional to the square of the dependent systems) and from a release and change perspective; there is a lot of coordination necessary given how much is being changed at one time.

    All of this work to manage all of the dependencies found in medium to large enterprises completely eliminates any potential savings one might get from batching work into a single delivery. In addition, the possibility of having to pull some work out of a release (due to quality or business readiness or some other reason) sometimes leads to complicated source code management (feature branches in some cases) that further complicate the process of merging frequently into the trunk. These types of practices also serve to drive up technical debt. So looking at the problem from this angle shows why results we are seeing make perfect sense.

    Productivity is inversely proportional to the number of dependencies in a release

    As Lee related to in the initial blog, a lot of an organization’s success depends on trust. And one factor of trust is clarity which speaks to understanding the changes that each release is bringing to the IT and customer environments. So a factor that is working against trust is the notion of delivering more things less frequently and better controlling change somehow, is a good notion.

    Trust is inversely proportional to the number of dependencies in a release

    There is another hidden component of waste associated with “big bang” initiatives. Go back to our grocery example for a moment. Chances are that the Smith monthly excursion results in buying some stuff they end up not using. Perishable items are not spread out very well when they aren’t purchased more frequently. Likewise in IT planning, we tend to put a bunch of stuff in our IT Delivery basket that we don’t know for sure that we will need and we end up with a lot of wasted effort for features that are hardly used or not needed at all. Some enterprise estimates are that 50-65% of what they put into large releases does not actually result in true value to the business. Doing more frequent releases allows a constant discovery process to better guide the usage of IT resources on what is truly of value.

    Waste is proportional to the size of a release.  

    In other words:

    Yield (Value per IT investment) is proportional to the frequency of releases.

    REVISTING THE MATH OF DELIVERY TIME

    So if we revisit Lee’s model

    Lee Reid Blog equationand apply the “Calculus of Release” as we stated above, there are increased costs associated with having larger and less frequent releases in medium/large enterprises.

    • Planning cost TPLAN
    • Development complexity TDESIGN + TDEVELOP + TBUILD
    • Testing Complexity TTEST
    • Release and Change Management Complexity TRELEASE + TDEPLOY + TEVALUATE

    TRUST % – Even if everything was proportional between the JONES and SMITH models, the component of trust that impacts clarity, consistency and collaboration, is much lower for large (SMITH) releases.  Given that TRUST has a multiplicative impact, this alone can account for the results found in the studies.

    CULTURAL CHALLENGE

    While clearly one needs to be sensitive to the changes being introduced into the business, having the ability for the business stakeholders to pull capability when they want it and having it available more frequently is not only good for customers but also has extreme benefits from an IT perspective in terms of increased productivity and reduction of technical debt. The challenge is to convince leaders in both the business and IT areas on the merits of doing more continuous planning and frequent deliveries.   Unicorns may have led the way in this type of thinking but it is actually the horses that will benefit most from this type of culture shift because it is the horses[2] that have the most to gain in mitigating the dependency factor which can undermine quality, productivity and cycle times.

    Read more about the topic in the next blog of this series The Art of DevOps by Sanjeev Sharma.

    [1] “2014 State of DevOps Reports”, Puppet Labs, IT Revolution, https://puppetlabs.com/sites/default/files/2014-state-of-devops-report.pdf

    [2] “The Phoenix Project”, Gene Kim, Kevin Behr, George Spafford, http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592

     

    About the Author

    CarmenCarmen DeArdo is the Director of Application Development Tools and Technologies at Nationwide Insurance. Carmen is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across Mobile, Distributed and Mainframe and other technologies. This includes recommendations and implementation of technologies used across the development life cycle (e.g. IBM Rational tool suite, open source technologies).

  • Inside Look at State of DevOps Report: Q&A With Puppet Labs

    Inside Look at State of DevOps Report: Q&A With Puppet Labs

    Last week Puppet Labs released the analysis behind its annual study of DevOps and continuous delivery practices in the 2015 State of DevOps Report. Now in its fourth year, the report offers an in-depth look at the findings of the team on what it means to be a high-performance IT organization—the cultural characteristics these organizations share, the lean engineering practices they engage in and the technical architectures they depend upon.  Like last year,  however, the report tends to be a bit of a black box when it comes to actual statistical findings. While some figures do make it through, for example the oft-flogged statistic of high-performance organizations deploying 30 times as fast with 200 times the lead time of low performers stands pat, the report itself is somewhat oblique about the statistical basis upon which the designators of low-, medium-, and high-performance were determined.

    devopsy.kinsta.cloud spoke to Nigel Kersten, CIO for Puppet, and Alanna Brown, senior product marketing manager for Puppet,  about the report to get a little more detail, to have them explain the process behind the analysis, and to offer up some of the most important facets of their findings over the last few years.

    devopsy.kinsta.cloud: Can you tell our readers a little bit about the high points from the report?

    Kersten: Big finding from 2013 was the orgs that used Devops’ practices deployed code 30 times more often and had 50 percent fewer failures.  In 2014 what was quite fascinating was that we actually found a link between these practices and overall organizational performance, not just IT performance. Higher-performing IT organizations are twice as likely to exceed their own profitability, market share and productivity goals.

    If we fast-forward to today, the high-performing organizations using Devops practices are more stable and reliable than they’ve ever been.  They’re recovering 168 times faster from their failures and have 60 times fewer failures due to their changes.  So we’re not actually seeing throughput increase from the year before, but what we are seeing is the quality and speed has definitely increased.  So people are producing changes that are of higher quality and that require fewer rollbacks.

    And so what we’re concluding from the data that we’ve seen this year in the report is that there’s been a really huge focus to the left of the assembly line from production. So if we envisage the developer work flow application delivery, essentially we start with the developer laptop like on the far left, and on the far right is the application actually existing in production.  So from the point between the developer’s laptop and the actual release, we’re seeing organizations focus much more on that stage, putting more testing in the hands of the developers, and all of these practices not just being focused around the release but the pre-production stage of the application life cycle having a really huge impact upon quality.

    devopsy.kinsta.cloud: Why doesn’t Puppet publish the actual data from the survey? You have great analysis here, but there’s very little statistical proof behind it—why don’t you provide that?

    Brown: Well so we have a ton of data, and part of the reason we don’t issue the data is because you wouldn’t be able to actually get to the same results without doing a ton of data cleansing and statistical analysis.  So our analysis is actually fairly rigorous, and some of that analysis doesn’t really lend itself very well to graphs and charts and visuals and things like that because it’s all based on correlations and predictive regressions.  So that’s part of the reason.

    Kersten: I think in many ways it’s analogous to releasing software to the world. We have all of these tools inhouse.  We have all of these Excel spreadsheets we’ve put together.  We have all of these you know pivot tables and different bits of cluster analysis, but they only – they’re not particularly usable for anyone else right now.

    As much as we would love to work out how to polish all of that up and share it so it’s one big data set along with all of the tools, our biggest worry is honestly being that people would actually draw sort of misleading and incorrect conclusions from the data because of the level of analysis that was required.

    devopsy.kinsta.cloud: Can you tell us, then how you came to the determination of what’s a high-, medium- or low-performance organization? Because if you look at some of these numbers, you say they’ve got thirty times more deployment frequency. But of course they do because they’re high performers. So how does that work out statistically?

    Brown: No, no, you’re absolutely right, and I will agree that IT performance metric is a bit circular.  However, those are the actual metrics that are important to an IT organization, so they care about deploy frequency, deployment lead time, their mean time to recover and change failure rate.  So we feel like that’s a pretty good approximation or a proxy for IT performance, and it’s better you add at doing those things that is correlated to higher performance overall.

    Kersten: One thing, too, is that even though there’s a whole spectrum in terms of how fast people are able to actually deliver and ship code, what actually came out really strongly out of the survey this year and in previous years is that the high performance (orgs) definitely cluster in terms of data points. It’s not just a smooth spectrum, but there’s this really obvious clusters of high performance that leap out that all have a very high correlation of the kinds of practices they use.  So it is a little bit circular but it’s also it’s not that we’re seeing this smooth spectrum from zero to 100.  We’re seeing the masses and then the high performers, and they divide into quite clear clusters.

    Brown: Yeah, and they’re not arbitrary clusters; they’re actually statistical.  So the low IT performers share all the same characteristics and the high IT performers share all the same characteristics, so they’re very clear.

    devopsy.kinsta.cloud: OK, so let’s talk about some of those common characteristics between these groups if you’ve got statistical bases for these.  Tell me a little bit about the low performers and we’ll kind of go on up the spectrum and we can talk about the characteristics they share.

    Brown: So for IT performance, deployment pain was actually the number one most highest-correlated construct. And deployment pain consists of a couple of questions.

    Like how painful are your deployments?  Are they very disruptive events, etc?  So there’s three questions that we ask around deployment pain.

    The next thing that’s highly correlated with IT performance is version control. We use the version control for all production artifacts.  Following that is culture, and the model we use for culture comes from a guy named Ron Westrom who is a sociologist.  And he created a typology of cultures, and you can see the table in the report itself.  So he has a pathological culture, bureaucratic, and then generative.  So the more generative your culture is, the more likely you are to have higher IT performance.

    Kersten: And I think this is what was so fascinating for us when we first started adopting Westrum last year We actually adopted Westrum as part of the model in that this pathological versus generative cultures, this is the language our practitioners in this space use to talk about you know what’s so great about working there: “My ideas are valued. I get to create ideas.  I can create impact.” Similarly around what’s horrible about working at this other place.  It maps to the pathological and generative descriptions of organizations perfectly really.

    devopsy.kinsta.cloud: So what would you say is the key characteristic in that first leap from low performance to medium performance, based on your data?

    Brown: So the lower performers in terms of IT performance will be doing less automation, less testing.  They haven’t automated their deployments, for example.

    Kersten: I think one way I at least think about is that there are these characteristics that are necessary but none of them are sufficient on their own, and so you see people who say just adopted version control.  They’re definitely heading ahead of the pack of the people who haven’t adopted any of these tools.

    But there’s this transformative leap once you actually adopt version control and continuous delivery and the highly-automated conflict management system, that those things together are greater than the sum of their parts.  That is at least how the data seems to us, ’cause we don’t actually have this smooth little spectrum between low performers and high performers who just adopted one of those technologies.

    We definitely see improvement from those people, but it’s when they adopt all of them in total that they suddenly take this sort of order-of-magnitude leap.

  • The Myth of DevOps as a Catalyst to improve Security?

    The Myth of DevOps as a Catalyst to improve Security?

    Few topics within DevOps discussions elicit more controversy than the relationship between DevOps and information security. Do DevOps practices help to improve security, or do most organizations simply automate and bring their existing bad internal processes along into DevOps? With this topic in mind, I recently had an engaging conversion with Adam Muntner on this and more, and thought it would also be of interest to devopsy.kinsta.cloud readers.

    Adam Muntner is a Security engineer at Mozilla and the creator of FuzzDB, an open-content licensed attack and discovery pattern database of test cases for software security testing and research. He has 20 years of experience at security governance and management, technical assessment, research, and consulting with a focus on secure software development. The opinions expressed are his, not his employer’s.

    Here is our recent conversation on DevOps and security, edited for brevity and clarity.

    devopsy.kinsta.cloud You’ve taken issue with some of the claims that DevOps can help to improve security. For example, my writing, “several years into enterprise DevOps deployments, what we are actually learning is that unifying development and operations teams through DevOps efforts does not create the wild west environments feared,” in my post DevOps Security Comes of Age. Another was from this Q&A with Gene Kim where he said that high performers are able to conduct more frequent code deployments complete their deployments from code committed to running in production more quickly, that they enjoyed twice the change success rate, and that outages were fixed 12 times more quickly.

    I had used these findings to support DevOps as helping to improve security outcomes. You took issue with these statements. Why?

    Muntner: I’m not opposed to DevOps or packaged deployment, rather I’m for taking a holistic view of benefit and risk from an informed perspective. The thing to keep in mind is that the security of an application environment is inherited – it’s the aggregate result of all its component pieces.

    When system architectures are so complex that using opaque, componentized, pre-built pieces in an application environment becomes the norm, how do you know what’s in them? How much attackable surface area is being inherited?

    Gene Kim is a smart guy and chose his words carefully with regard to security. Gene said,

    “The first one is the high performers were massively outperforming their non-high performing peers. Thirty times more frequent code deployments. They were able to complete the deployments from code committed to running a production 8,000 times faster. In other words they could do a deployment measured in minutes or hours. Whereas low performers it took them weeks, months, or quarters. Furthermore, they actually had better outcomes. Twice the change success rate and when they caused outages they could fix it 12 times faster. There were 4,200 that completely responded and it just showed that you could be more agile and reliable at the same time. I think if we just connect some dots, they’re also going to be more secure, they’re probably going to have a faster find fix cycle time. We didn’t actually test that but, if that were true it would mirror what we found in the previous benchmarking work I had done.”

    The old engineering maxim of “good, cheap, fast” has not been obsoleted by DevOps. Deploying code 8,000 times more quickly is not a measure of risk reduction. It might help get fixes out faster, but that doesn’t tell the whole story.

    Gene said, when they caused outages they could fix it 12 times faster,” but how many of the outages are the result of faster deployment cycles? What does it mean to fix an outage 12 times faster than if the outage had never occurred due to a different development practice? Fixing outages quickly is good, that’s what is seen. It doesn’t take into account the unseen – newly introduced weaknesses that aren’t being tested or measured for.

    How much of what was measured resulted from the ex-Devops security practices of organizations measured and resultant outcomes? What does “better outcome” mean, and is that even objectively quantifiable across different organizations or even situations in the same organization? Are we confusing correlation and causation? Every organization is different. The metrics are being held up as quantitative proof but it’s not certain that they are measuring what they purport to.

    Consider this scenario: Operations staff are incentivized through the salary review/bonus process to maintain uptime. Despite working in the test environment, the latest code push to production broke your application. DevOps reinstalls the Docker and the problem is still there. What now? Go back to the old version that ran without crashing but (possibly unknown to your operations team) contains a known vulnerability in some library that is static-compiled into a binary?

    devopsy.kinsta.cloud: What about the argument that DevOps helps to reduce code bloat from legacy environments goes a long way to reducing both technical and security debt?

    Muntner: Reducing bloat might be one of a number of goal of a devops team, but devops practices are just as likely to increase code bloat, opacity, and attackable surface. Security is not an inherent property of DevOps practices. For example, the perception that software which can only be installed through prebuilt packages increases simplicity only creates the dangerous illusion of simplicity.

    Relying on these opaque components increases code bloat in unpredictable and unknown ways, potentially decreasing security posture. If there even is a threat model (usually not, they’re hard and time-consuming to produce, thus expensive) then a popular software package like Hadoop probably won’t more than a box in a threat model, when in reality it’s an aggregation of many components and n-th order subcomponents, each with their own attackable surface area, depending on configuration. If it’s being installed as part of a prebuilt environment, there are many possible security consequences and none of these are addressed or measured in any of the types of tests or metrics that have been described.

    Performing thousands of tests sounds good, but what if tens of thousands of tests are necessary? Or a completely different testing methodology and toolset? Where are will the false negatives be based on the limitations of the existing set of tests? And most important, do the metrics support the claim?

    There are many categories of attacks that these tests aren’t able to identify: Logic flaws are one. Different components of a system understanding the same piece of (malicious or spurious) data to mean different things, another. Sometimes the problem isn’t a bug, it’s architectural deficiency. If it’s deeply layered inside a component of your system that uses its own non standard build system, you probably won’t find it. Software tests are for known, expected code execution paths and interactions, it’s much harder to identify all possible execution paths and orders of operations.

    DevOps: But all of those flaws can be found later in production, when they exist. It is not as if those manually tests are being done away with.

    Muntner: Typically, security audits and assessments are time-boxed. It’s often hard to assess an entire large application inside the time box, let alone all the many opaque components. And then there is the question of pen testers relying on commercial automated tools where even the tests the tools are performing and the included test cases are opaque. Ultimately, they are a snapshot of the security posture at a point in time, given the limitations of the tester(s), tools, methodology, time, and ability of the organization to efficiently consume and triage testing results and lessons learned. Given the rapid release cycles implied by Devops and Agile development, the results can go stale fast.

    Adding unknown layers of dependencies and libraries stretches those already-thin review hours even thinner. Security resources are scarce and expensive. Organizations with mature application security assurance programs only apply those resources to the most critical applications. Outside consultants reviewing one application could cost $20,000 or more.

    Automation is very critical but automated tests have big limitations and often miss configuration management details. Most important, not all vulnerabilities get found. If the automated tools don’t test for it and the pentester didn’t find it within their testing window, it’s still there. Good penetration testers are always finding bugs that weren’t found during previously time-boxed assessments. And so are malicious actors.

    devopsy.kinsta.cloud: All of these possible conditions exist in enterprises now. And certainly, automating bad practices could more rapidly propagate a mess, or thinking it through and automating the right things could improve real outcomes. So why can’t this prove to produce more resilient code and architecture than current methods?

    Muntner: Thinking security testing through and automating as much as possible will yield results, but that can happen with or without devops. I’m not saying devops is invalid, rather that it alone is not responsible for good outcomes. Thinking that an approach delivers more than it really does is only a false sense of security, arguably worse than awareness of insufficient security.

    Secure systems and software development practices like command-safe APIs, network-layer features in TLS, HTTP layer features like CSP, improvements in application and protocol layer firewalls, developers learning to do proper encoding for the appropriate output context, automated testing with tools like OWASP ZAP or commercial equivalents as appropriate for the type of application are all high-impact but have nothing to do with devops.

    devopsy.kinsta.cloud: Security should be part of the flow, an integral
 part of QA, security and functional testing.

















    Muntner: Security isn’t a state, it’s a process. It’s a verb, not a noun. Security ‘what’ should be part of the workflow? Security activities and tests, personnel, all of the above? Should a security organization report to the business management and governance side of management, or the technical side? And why is DevOps better for security maturity than separation of duties?
    DevOps is the approach we take at Mozilla, but the IT and product organizations at Mozilla are full of exceptionally talented people who would do a good job with security using any development process.
    devopsy.kinsta.cloud: Depends on the organization, I’m sure, but if an organization is going to move to DevOps than DevOps needs to be part of the processes.

    Muntner: DevOps can be good or bad but DevOps doesn’t have much to do with security in and of itself, except for some of its popularly misunderstood practices, and then maybe not always in the ways people expect. Standardizing is good unless you standardize on terrible things.

    devopsy.kinsta.cloud: That’s certainly a risk. For many companies they are implementing DevOps regardless, so it’s a matter for security working toward the best way to integrate themselves and their processes.

    Muntner: IT is a business support function. Security is a business risk analysis function. If you standardize and integrate things without understanding threat, risk and security posture, what have you done? Ultimately the decisions are business decisions. Unfortunately, they are frequently made from the perspective of insufficient knowledge.

    There is nothing inherent to DevOps that solves these problems and DevOps comes with its own potential pitfalls.
 The problem is external to what DevOps can deliver.
 






For organizations that want to better understand the maturity of security in their Software Development Lifecycle, OWASP OpenSAMM is a good place to start. It helps organizations understand security maturity means for them specifically and incrementally advance towards their desired posture.

  • Microservices define the next era of IT

    Microservices define the next era of IT

    There is a lot of focus on DevOps and container technologies these days. Both are very strong trends in IT on their own, but each is also proof of a larger shift in IT culture to a microservices architecture. A new report from IDC titled “The Emergence of Microservices as a New Architectural Approach to Building New Software Systems” takes a closer look at the evolution and impact of microservices.

    Let’s start with the obvious question: What exactly is microservices? “Microservices is an architectural approach that draws on long evolving experience in software engineering and system design, including the SOA efforts of the last two decades,” said Al Hilwa, research director, Application Development Software at IDC. “Microservices architecture is enabled by a spectrum of tools categories but is primarily an architectural approach to system design that also requires considerable organizational and cultural adjustment to execute successfully.”

    Put another way, microservices is an umbrella term that applies to breaking systems and applications down to a more granular, modular level. Containers are an example of taking an application and making it into a self-contained component, and DevOps provides the framework for developing, deploying, and managing the container ecosystem.

    By some interpretations microservices and containers are somewhat synonymous. In other words, microservices is just another way of saying that an organization is using container technologies for app development and deployment. IDC classifies microservices more broadly, saying it is based on an approach to organizing software development—more of a concept and less of an actual product or technology.

    The Wikipedia definition of microservices sums it up like this: “In computing, microservices is a software architecture style, in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. These services are small, highly decoupled and focus on doing a small task.”

    Although the concept of microservices is broader than the underlying platforms or tools it uses, IDC acknowledges that the advancements in software development and deployment methodologies—namely DevOps and container technologies—has played a pivotal role in elevating microservices to where it is today.

    The transition from a traditional hardware server infrastructure in an on-premise datacenter to a cloud-based, software-defined infrastructure that can be created, modified, or removed programmatically offers significant benefits for organizations. The microservices architecture enables companies to be much more agile and cut costs at the same time.

    There are some challenges and concerns as well, though. Hilwa notes, “Microservices architecture, by its very nature, is distributed and operates across networks and can thus exact significant performance penalties that must be accommodated with architectural solutions, like caching layers and API management and optimization techniques.”

    In the IDC report Hilwa talks about the need for strong organizational support and an overall shift in the IT culture in order to succeed with microservices. As Hilwa puts it, “Organizations have to increase their tolerance for risk and learn to fail gracefully, fast and often in order to achieve success.”

    Hilwa provides a comprehensive overview of the current state of microservices in the report along with some prescriptive guidance for organizations that are either just trying to understand what microservices is, or those that have already waded into microservices but want a better appreciation for the benefits and pitfalls.

  • Loggly Derived Fields Add Structure To Unstructured Logs

    Loggly Derived Fields Add Structure To Unstructured Logs

    The Gist of Loggly’s Derived Fields Gesture

    A top log management product vendor, Loggly announced an update to its tooling last month in the form of its new Derived Fields. The vendor has drilled down past the summaries in its Dynamic Field Explorer interface into Derived Fields that use metadata to structure unstructured log data.

    Where traditional log management and analysis products have typically required you to create one-off regular expressions for custom analyses—expressions that you must build anew each time you want to put log data under a microscope—Loggly now enables you to use rules rooted in regular expressions that you create once to structure unstructured data when your systems first enter logs into the tool. You can look at resulting Derived Fields instantly, every time you need to do log analysis, according to Hector Angulo, Head of Products, Loggly.

    The necessity for the Derived Fields capability arises from the fact that logs that escape structured creation contain so much untapped value. Until now, Loggly’s approach has been to automatically apply structure to common log types. “But we find that more than one-third of customer logs are still not structured. Even worse, many of these logs were created by developers for their own use, without the expectation that others would need to rely on them for critical troubleshooting. As such, it can be really hard for everyone but the original developer to decipher what specific logs mean,” says Angulo. Derived Fields attempt to retrieve 100-percent of that value from the log data by making analysis possible for any team member.

    How Loggly Dogs Data Using Derived Fields

    Loggly absorbs data from common log types such as Apache or JSON. By adding metadata that describes unstructured log data and then “injecting” context and structure into logs or parts of a log, Loggly’s Derived Fields reclaim obscure developer logs. Fields that uniformly describe the same kinds of data elements across logs enable users to compare data within logs to see what is happening across a DevOps environment.

    Staff members who can create custom parsing rules based on regular expressions can set up the rules the DevOps teams will need at the start. Then, unless you come up with a new rule you want to add, your team members won’t have to create regular expressions again, and certainly not in order to search log data on the other end of that pipe.

    Using Derived Fields

    Using Loggly’s Derived Fields, you can drill down on a specific value in a field. Click on the field name and you can inspect all log events that have the same type of value. A summary chart above the individual event log will automatically refresh to summarize the events of the data from the given value, says Angulo.

    You can easily see other log events that share similar data or characteristics in order to arrive at insights about what is going wrong, or right with the software.

    How Derived Fields Help DevOps

    “Rather than starting with an empty search box, Loggly starts off with bird’s-eye-view summaries using Dynamic Field Explorer,” says Angulo. From there, you drill down to fine grain details.

    The time that Loggly’s Derived Fields save, time that it would normally take to create one-off queries for every analysis, is critical if you’re trying to solve a problem that is preventing end users from completing tasks and generating revenue for the business, says Angulo.

    According to Angulo, Derived Fields specifically help DevOps teams to:

    • Resolve issues faster because it is easier to spot the data that matters.

    • Perform advanced analytics even with legacy applications that send out unstructured, text-based logs that you cannot update or don’t have the bandwidth to update.

    • Extend advanced log analysis to more team members. You don’t have to be a regex wizard to gain insight from unstructured logs.

    • Finally, the intuitive, navigable summaries generated in Field Explorer equate to less training time, less time with reference guides, and more time on data analysis and problem solving.

    A Cautionary Note

    As with any DevOps tool update, the proof lies in how Derived Fields work in your development process. Loggly touts 5K customers using its existing product. I’m sure they’re not trying to chase anyone away.

    Are you using this tool? Do you have other input on this topic? See that empty comments field below?