devopsy.kinsta.cloud

Category: Features

  • Will DevOps Kill IT Outsourcing?: Part 1

    Will DevOps Kill IT Outsourcing?: Part 1

    Part 1: DevOps and outsourcing like oil and water

    After nearly two decades of steady IT outsourcing for the sake of savings and efficiency gains, the shine seems to have worn off from this once glittering phenomenon. And according to many experts, DevOps and continuous delivery practices are playing a big part in many enterprises’ plans to reverse course on outsourcing key functions in their technology stack.

    “DevOps transformations emphasize collaboration, which is often at odds with outsourcing arrangements,” says Jeff Gallimore, founding partner at Excella Consulting. “Outsourcing arrangements often reinforce stovepipes and silos rather than break them down. And silos are speed killers and barriers to improvement.”

    According to a report out earlier this spring by outsourcing research firm Information Services Group (ISG), average outsourcing contract values saw their worst first quarter in a decade, dropping by 27 percent year-over-year. Big deals over $30 million were particularly hit hard, declining by 25 percent in both number and value.

    Even without the tidal wave of Agile and DevOps practices seeping into enterprise IT over the last few years, many organizations are already predisposed to take a second look at their outsourcing contracts, says Gallimore, who sees time and again that many businesses aren’t getting the value they thought they would from their outsourcing deals.

    “I hear many more negative experiences organizations have had with outsourcing arrangements than I do positive ones,” he says. “Poor quality. Higher-than-expected management overhead required. Inflexibility. Process challenges. All are reasons cited for why the arrangements didn’t live up to expectations.”

    The anecdotal stories are also piling up of big name enterprises pulling more of their IT resources back in house after years of outsourcing. For example, General Motors made waves three years ago when its CIO Randy Mott said he would completely invert the outsourcing ratio the car company had by 2017. The firm continues to make headway in changing from 90 percent outsourced IT to 90 percent insourced, with many of the positions advertised these days focused on DevOps and innovation-related roles. Similarly, David Smoley, CIO of pharmaceuticals firm AstraZeneca, went public with his goal to shift the company from 70 percent dependence on outsourcing to 30 percent, along with a commitment to Agile and lean development.

    As organizations increasingly witness the business boosts gained through a more collaborative DevOps delivery model, they are seeking ways to adopt continuous delivery practices across their entire software delivery lifecycle. But they’re finding that these practices are often wholly incompatible with the outsourcing arrangements they’ve developed, says says Jonathan Braunhut, chief scientist at KEMP Technologies. As he explains, many organizations will need to “unwind” these arrangements if they want DevOps to work well because in many cases the contracts and SLAs that govern them are inextricably tied with waterfall development.

    “Outsourcing in an enterprise setting is often implemented in a way that advances outdated waterfall-like practices and processes that are antithetical to continuous delivery,” Braunhut says. “The very wall between dev and ops that the DevOps movement seeks to eliminate can become codified in outsourcing agreements, as a reasonable means of achieving an enforceable and accountable contract.”

    More specifically, outsourcing tends to fly in the face of DevOps’ lean ethos, which is to “minimize handoffs and waste and increase quality by thinking about the overall system and flow,” says Gallimore, who believes that traditional contractual arrangements simply can’t operate with the kind of agility necessary to establish speedy and iterative cycles of continuous delivery.

    Perhaps even more fundamentally flawed in the outsourcing relationship is the idea of ownership. As many organizations finally recognize the value that IT brings to the table—more likely now to serve as a revenue generator than a cost center—it doesn’t make as much sense to give up so much ownership over the technology and tech-enabled services that could potentially transform the business.

    “To be most effective, enterprises need to own the transformation, and it has to be a truly collaborative effort across disciplines–business leadership, engineering, system administration, security & compliance,” says Thomas Enochs, vice president of customer success at Chef. “Collaboration and transformation are difficult to achieve with outside third parties or vendors. They need to own and drive the change themselves, and not be dependent on others.”

  • DevOps Dozen – Nominations are now open

    DevOps Dozen – Nominations are now open

    To succeed in todays speed of business, app-centric world the old ways of doing business just don’t work anymore. IT and development teams are being pulled, pushed, stretched and changed into new combinations and configurations. Greater cross-functional cooperation is a must, a culture of cooperation and a new generation of tools that facilitate these are changes are sweeping in like a Tsunami.

    DevOps has appeared like a superhero fighting a villain to combat these issues. Featuring a one-two punch of continuous delivery and continuous integration with a finishing blow of automation, DevOps enables easier and closer collaboration between development and operation teams.

    Whether they be cloud-native “unicorns”, traditional enterprise “horses” or born on the run startups, DevOps is becoming the “way they do things.”

    The field has matured enough where we think it is time to recognize some of the best and brightest.  Our first venture with this is our “DevOps Dozen”  The DevOps Dozen will feature the top 12 vendors in the DevOps space. I know 12 isn’t a lot given the amount of companies in the mix, but we had to start somewhere.

    So how do we get to a dozen from such a large and worthy group? We at devopsy.kinsta.cloud have compiled a list of what we think are the best known DevOps companies. Some of these companies have their product or service listed as well to help identify the face to the name. We have broken them down into several categories to hopefully make it easer to select (tip of the hat to XebiaLabs and their Periodic Table of DevOps for the help with that).

    The DevOps Dozen selections will be made by you our readers in two phases. First we will open the whole list as well as any write ins for those we may have missed. This nomination voting will be open until August 12th. At that point we will take the top 24 company/products who garnered the most nominations. These will be the finalists and we will have a 2nd round of voting for final Dozen.

    Feel free to tweet about the DevOps Dozen using the hashtags: #DevOpsDozen and #DO12

    Please click here to vote for the finalist nominations

    or fill in below

    Create your own user feedback survey

  • The Calculus of DevOps

    The Calculus of DevOps

    This is the second in a series of blogs, partnering with Lee Reid, who wrote the first installment “The Simple Math of DevOps” and Sanjeev Sharma both of who are outstanding DevOps Thought Leaders from IBM.

    I am picking up on Lee’s mathematical DevOps model to apply a calculus perspective, a “calculus of DevOps” if you will, (OK, not quite core Big Bang Theory Calculus but rather approaching it from the aspects of engineering queuing theory).

    One of the cultural challenges that DevOps poses is, that some aspects of it run counter to engineering principles that many of us learned with respect to capacity and response time.   When I worked at Bell Labs, we would apply queuing theory to determine how to best package information to be sent across the network. A basic premise revolved around a tradeoff between throughput and response time. On one hand, the more messages that were packed into the packet, the more the payload would improve capacity because the overhead of processing the packet of information would be spread across the messages.   However, waiting to accumulate enough call processing information meant that the packets would be transmitted less frequently, increasing the response time.

    A simple example of this would be making trips to the grocery store. If one wanted to spend the least amount of resources (time, gas, car wear, etc.) on shopping, then one would make less frequent trips and then buy a larger amount of food. So if the Smith family made one trip a month and spent 2 hours, then their average time per week spent on shopping would be 30 minutes.   If the Jones family went every week and spent an hour, then they would be spending twice the time and four times the gas on grocery shopping. However if the Smith family run out of an item (say cereal), they would have to wait, on average 2 weeks before this was replenished. The Jones family average on any item that they ran out of would be 3.5 days in comparison.

    So when the 2014 DevOps report was published[1] which states that areas that deploy more frequently get customer feedback more quickly, that is to be expected. But what’s not expected is that these organizations also have higher productivity.   So exactly what is happening here that seems to be in a basic conflict to the principles of throughput and response time?

    Keeping up with the JONESES

    If we think of a pre- DevOps state for a medium to large enterprise, it may be typical for applications to only be released (deployed into production) a few times a year to provide business functionality. (I assume there are more frequent releases for “unplanned work” like defects and other changes or operational work). For the sake of this discussion, let’s call this the Smith Application. So from a business perspective, the business wants to pack as much work into the Smith releases as possible given they only have a few cracks a year at getting new business functionality into production.

    SMITH APPLICATION CHARACTERISTICS:

    • Deliver Business Value 3 times a year (every 4 months)
    • Cost is C1
    • Business Value (BV) delivered is V1
    • Average time from concept (e.g. Story) being ready to be pulled from the backlog to the time it is delivered to the customer is 2 months plus any additional wait times (e.g. prioritization)
    • BV delivered per year is 3*V1
    • Cost per year is 3*C1

    Now let’s transport over to the Jones Planet where the application is released into production with new business capabilities at the end of every 2 week iteration.   In this case the business is more concerned with providing a constant demand flow into IT than they are with debating over whether a given feature/story is going to be included in this particular release.   Instead there is a constant prioritization (or continuous planning) process that ensures the highest value work is being fed into the IT pipeline.

    JONES APPLICATION CHARACTERISTICS:

    • Deliver Business Value 26 times a year (every 2 weeks)
    • Cost is C2
    • BV delivered is V2
    • Average time from concept (e.g. Story) being ready to be pulled from the backlog to the time it is delivered to the customer is 1 week plus any additional wait times
    • BV delivered per year is 26*V2
    • Cost per year is 26*C2

    Based on the previous engineering analysis, we would expect the following:

    1. SMITH annual cost (3C1) < JONES Annual Cost (26C2)
    2. SMITH annual BV (3V1) > JONES annual BV (26V2)

    So the tradeoff (we would believe) JONES is making by delivering more frequently is that they are spending more money and are delivering less business value on an annual basis.

    However the data from the previously referenced DevOps report is just the opposite. In fact the findings are that enterprises that deploy smaller units of work more frequently have much higher quality and much higher productivity (Capacity). So what’s going on here?

    A NEW PARADIGM

    If you go up to any IT team (even an agile one) and ask them why they don’t deploy business value more frequently, they will generally refer to the following types of wait states of waste that they encounter:

    • Waiting for work (lack of consistent flow)
    • Waiting for somebody else to do something (dependencies from other teams)
    • Waiting for environments (contention)

    One major root cause of this is the dependencies that arise from large enterprises packing a lot of work into infrequent releases. From a planning perspective, the process is elongated due to prioritization discussions because being on the wrong side of the line can mean delay amounting to months, in delivering a product to a customer.   Ask a business portfolio leader what their highest priority is for IT and I am guessing they can tell you in a matter of seconds. However, most large organizations can spend several months a year planning their annual portfolio of work for IT. Some organizations spend more than 50% of their new project build time prior to a single story entering an agile development team’s backlog!

    From a development perspective, packing a lot of work into releases means many more dependencies that have to be managed across initiatives and impacted applications. From a testing perspective, downstream environments (ST, UAT) have to manage much more information in terms of application versions and especially test data that complicate and slow down the process. The complexity of test data increase is proportional to the number of pairwise relationships that exist within the datasets required for testing.   As such, the testing complexity will increase non-linearly (perhaps proportional to the square of the dependent systems) and from a release and change perspective; there is a lot of coordination necessary given how much is being changed at one time.

    All of this work to manage all of the dependencies found in medium to large enterprises completely eliminates any potential savings one might get from batching work into a single delivery. In addition, the possibility of having to pull some work out of a release (due to quality or business readiness or some other reason) sometimes leads to complicated source code management (feature branches in some cases) that further complicate the process of merging frequently into the trunk. These types of practices also serve to drive up technical debt. So looking at the problem from this angle shows why results we are seeing make perfect sense.

    Productivity is inversely proportional to the number of dependencies in a release

    As Lee related to in the initial blog, a lot of an organization’s success depends on trust. And one factor of trust is clarity which speaks to understanding the changes that each release is bringing to the IT and customer environments. So a factor that is working against trust is the notion of delivering more things less frequently and better controlling change somehow, is a good notion.

    Trust is inversely proportional to the number of dependencies in a release

    There is another hidden component of waste associated with “big bang” initiatives. Go back to our grocery example for a moment. Chances are that the Smith monthly excursion results in buying some stuff they end up not using. Perishable items are not spread out very well when they aren’t purchased more frequently. Likewise in IT planning, we tend to put a bunch of stuff in our IT Delivery basket that we don’t know for sure that we will need and we end up with a lot of wasted effort for features that are hardly used or not needed at all. Some enterprise estimates are that 50-65% of what they put into large releases does not actually result in true value to the business. Doing more frequent releases allows a constant discovery process to better guide the usage of IT resources on what is truly of value.

    Waste is proportional to the size of a release.  

    In other words:

    Yield (Value per IT investment) is proportional to the frequency of releases.

    REVISTING THE MATH OF DELIVERY TIME

    So if we revisit Lee’s model

    Lee Reid Blog equationand apply the “Calculus of Release” as we stated above, there are increased costs associated with having larger and less frequent releases in medium/large enterprises.

    • Planning cost TPLAN
    • Development complexity TDESIGN + TDEVELOP + TBUILD
    • Testing Complexity TTEST
    • Release and Change Management Complexity TRELEASE + TDEPLOY + TEVALUATE

    TRUST % – Even if everything was proportional between the JONES and SMITH models, the component of trust that impacts clarity, consistency and collaboration, is much lower for large (SMITH) releases.  Given that TRUST has a multiplicative impact, this alone can account for the results found in the studies.

    CULTURAL CHALLENGE

    While clearly one needs to be sensitive to the changes being introduced into the business, having the ability for the business stakeholders to pull capability when they want it and having it available more frequently is not only good for customers but also has extreme benefits from an IT perspective in terms of increased productivity and reduction of technical debt. The challenge is to convince leaders in both the business and IT areas on the merits of doing more continuous planning and frequent deliveries.   Unicorns may have led the way in this type of thinking but it is actually the horses that will benefit most from this type of culture shift because it is the horses[2] that have the most to gain in mitigating the dependency factor which can undermine quality, productivity and cycle times.

    Read more about the topic in the next blog of this series The Art of DevOps by Sanjeev Sharma.

    [1] “2014 State of DevOps Reports”, Puppet Labs, IT Revolution, https://puppetlabs.com/sites/default/files/2014-state-of-devops-report.pdf

    [2] “The Phoenix Project”, Gene Kim, Kevin Behr, George Spafford, http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592

     

    About the Author

    CarmenCarmen DeArdo is the Director of Application Development Tools and Technologies at Nationwide Insurance. Carmen is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across Mobile, Distributed and Mainframe and other technologies. This includes recommendations and implementation of technologies used across the development life cycle (e.g. IBM Rational tool suite, open source technologies).

  • Inside Look at State of DevOps Report: Q&A With Puppet Labs

    Inside Look at State of DevOps Report: Q&A With Puppet Labs

    Last week Puppet Labs released the analysis behind its annual study of DevOps and continuous delivery practices in the 2015 State of DevOps Report. Now in its fourth year, the report offers an in-depth look at the findings of the team on what it means to be a high-performance IT organization—the cultural characteristics these organizations share, the lean engineering practices they engage in and the technical architectures they depend upon.  Like last year,  however, the report tends to be a bit of a black box when it comes to actual statistical findings. While some figures do make it through, for example the oft-flogged statistic of high-performance organizations deploying 30 times as fast with 200 times the lead time of low performers stands pat, the report itself is somewhat oblique about the statistical basis upon which the designators of low-, medium-, and high-performance were determined.

    devopsy.kinsta.cloud spoke to Nigel Kersten, CIO for Puppet, and Alanna Brown, senior product marketing manager for Puppet,  about the report to get a little more detail, to have them explain the process behind the analysis, and to offer up some of the most important facets of their findings over the last few years.

    devopsy.kinsta.cloud: Can you tell our readers a little bit about the high points from the report?

    Kersten: Big finding from 2013 was the orgs that used Devops’ practices deployed code 30 times more often and had 50 percent fewer failures.  In 2014 what was quite fascinating was that we actually found a link between these practices and overall organizational performance, not just IT performance. Higher-performing IT organizations are twice as likely to exceed their own profitability, market share and productivity goals.

    If we fast-forward to today, the high-performing organizations using Devops practices are more stable and reliable than they’ve ever been.  They’re recovering 168 times faster from their failures and have 60 times fewer failures due to their changes.  So we’re not actually seeing throughput increase from the year before, but what we are seeing is the quality and speed has definitely increased.  So people are producing changes that are of higher quality and that require fewer rollbacks.

    And so what we’re concluding from the data that we’ve seen this year in the report is that there’s been a really huge focus to the left of the assembly line from production. So if we envisage the developer work flow application delivery, essentially we start with the developer laptop like on the far left, and on the far right is the application actually existing in production.  So from the point between the developer’s laptop and the actual release, we’re seeing organizations focus much more on that stage, putting more testing in the hands of the developers, and all of these practices not just being focused around the release but the pre-production stage of the application life cycle having a really huge impact upon quality.

    devopsy.kinsta.cloud: Why doesn’t Puppet publish the actual data from the survey? You have great analysis here, but there’s very little statistical proof behind it—why don’t you provide that?

    Brown: Well so we have a ton of data, and part of the reason we don’t issue the data is because you wouldn’t be able to actually get to the same results without doing a ton of data cleansing and statistical analysis.  So our analysis is actually fairly rigorous, and some of that analysis doesn’t really lend itself very well to graphs and charts and visuals and things like that because it’s all based on correlations and predictive regressions.  So that’s part of the reason.

    Kersten: I think in many ways it’s analogous to releasing software to the world. We have all of these tools inhouse.  We have all of these Excel spreadsheets we’ve put together.  We have all of these you know pivot tables and different bits of cluster analysis, but they only – they’re not particularly usable for anyone else right now.

    As much as we would love to work out how to polish all of that up and share it so it’s one big data set along with all of the tools, our biggest worry is honestly being that people would actually draw sort of misleading and incorrect conclusions from the data because of the level of analysis that was required.

    devopsy.kinsta.cloud: Can you tell us, then how you came to the determination of what’s a high-, medium- or low-performance organization? Because if you look at some of these numbers, you say they’ve got thirty times more deployment frequency. But of course they do because they’re high performers. So how does that work out statistically?

    Brown: No, no, you’re absolutely right, and I will agree that IT performance metric is a bit circular.  However, those are the actual metrics that are important to an IT organization, so they care about deploy frequency, deployment lead time, their mean time to recover and change failure rate.  So we feel like that’s a pretty good approximation or a proxy for IT performance, and it’s better you add at doing those things that is correlated to higher performance overall.

    Kersten: One thing, too, is that even though there’s a whole spectrum in terms of how fast people are able to actually deliver and ship code, what actually came out really strongly out of the survey this year and in previous years is that the high performance (orgs) definitely cluster in terms of data points. It’s not just a smooth spectrum, but there’s this really obvious clusters of high performance that leap out that all have a very high correlation of the kinds of practices they use.  So it is a little bit circular but it’s also it’s not that we’re seeing this smooth spectrum from zero to 100.  We’re seeing the masses and then the high performers, and they divide into quite clear clusters.

    Brown: Yeah, and they’re not arbitrary clusters; they’re actually statistical.  So the low IT performers share all the same characteristics and the high IT performers share all the same characteristics, so they’re very clear.

    devopsy.kinsta.cloud: OK, so let’s talk about some of those common characteristics between these groups if you’ve got statistical bases for these.  Tell me a little bit about the low performers and we’ll kind of go on up the spectrum and we can talk about the characteristics they share.

    Brown: So for IT performance, deployment pain was actually the number one most highest-correlated construct. And deployment pain consists of a couple of questions.

    Like how painful are your deployments?  Are they very disruptive events, etc?  So there’s three questions that we ask around deployment pain.

    The next thing that’s highly correlated with IT performance is version control. We use the version control for all production artifacts.  Following that is culture, and the model we use for culture comes from a guy named Ron Westrom who is a sociologist.  And he created a typology of cultures, and you can see the table in the report itself.  So he has a pathological culture, bureaucratic, and then generative.  So the more generative your culture is, the more likely you are to have higher IT performance.

    Kersten: And I think this is what was so fascinating for us when we first started adopting Westrum last year We actually adopted Westrum as part of the model in that this pathological versus generative cultures, this is the language our practitioners in this space use to talk about you know what’s so great about working there: “My ideas are valued. I get to create ideas.  I can create impact.” Similarly around what’s horrible about working at this other place.  It maps to the pathological and generative descriptions of organizations perfectly really.

    devopsy.kinsta.cloud: So what would you say is the key characteristic in that first leap from low performance to medium performance, based on your data?

    Brown: So the lower performers in terms of IT performance will be doing less automation, less testing.  They haven’t automated their deployments, for example.

    Kersten: I think one way I at least think about is that there are these characteristics that are necessary but none of them are sufficient on their own, and so you see people who say just adopted version control.  They’re definitely heading ahead of the pack of the people who haven’t adopted any of these tools.

    But there’s this transformative leap once you actually adopt version control and continuous delivery and the highly-automated conflict management system, that those things together are greater than the sum of their parts.  That is at least how the data seems to us, ’cause we don’t actually have this smooth little spectrum between low performers and high performers who just adopted one of those technologies.

    We definitely see improvement from those people, but it’s when they adopt all of them in total that they suddenly take this sort of order-of-magnitude leap.

  • The Myth of DevOps as a Catalyst to improve Security?

    The Myth of DevOps as a Catalyst to improve Security?

    Few topics within DevOps discussions elicit more controversy than the relationship between DevOps and information security. Do DevOps practices help to improve security, or do most organizations simply automate and bring their existing bad internal processes along into DevOps? With this topic in mind, I recently had an engaging conversion with Adam Muntner on this and more, and thought it would also be of interest to devopsy.kinsta.cloud readers.

    Adam Muntner is a Security engineer at Mozilla and the creator of FuzzDB, an open-content licensed attack and discovery pattern database of test cases for software security testing and research. He has 20 years of experience at security governance and management, technical assessment, research, and consulting with a focus on secure software development. The opinions expressed are his, not his employer’s.

    Here is our recent conversation on DevOps and security, edited for brevity and clarity.

    devopsy.kinsta.cloud You’ve taken issue with some of the claims that DevOps can help to improve security. For example, my writing, “several years into enterprise DevOps deployments, what we are actually learning is that unifying development and operations teams through DevOps efforts does not create the wild west environments feared,” in my post DevOps Security Comes of Age. Another was from this Q&A with Gene Kim where he said that high performers are able to conduct more frequent code deployments complete their deployments from code committed to running in production more quickly, that they enjoyed twice the change success rate, and that outages were fixed 12 times more quickly.

    I had used these findings to support DevOps as helping to improve security outcomes. You took issue with these statements. Why?

    Muntner: I’m not opposed to DevOps or packaged deployment, rather I’m for taking a holistic view of benefit and risk from an informed perspective. The thing to keep in mind is that the security of an application environment is inherited – it’s the aggregate result of all its component pieces.

    When system architectures are so complex that using opaque, componentized, pre-built pieces in an application environment becomes the norm, how do you know what’s in them? How much attackable surface area is being inherited?

    Gene Kim is a smart guy and chose his words carefully with regard to security. Gene said,

    “The first one is the high performers were massively outperforming their non-high performing peers. Thirty times more frequent code deployments. They were able to complete the deployments from code committed to running a production 8,000 times faster. In other words they could do a deployment measured in minutes or hours. Whereas low performers it took them weeks, months, or quarters. Furthermore, they actually had better outcomes. Twice the change success rate and when they caused outages they could fix it 12 times faster. There were 4,200 that completely responded and it just showed that you could be more agile and reliable at the same time. I think if we just connect some dots, they’re also going to be more secure, they’re probably going to have a faster find fix cycle time. We didn’t actually test that but, if that were true it would mirror what we found in the previous benchmarking work I had done.”

    The old engineering maxim of “good, cheap, fast” has not been obsoleted by DevOps. Deploying code 8,000 times more quickly is not a measure of risk reduction. It might help get fixes out faster, but that doesn’t tell the whole story.

    Gene said, when they caused outages they could fix it 12 times faster,” but how many of the outages are the result of faster deployment cycles? What does it mean to fix an outage 12 times faster than if the outage had never occurred due to a different development practice? Fixing outages quickly is good, that’s what is seen. It doesn’t take into account the unseen – newly introduced weaknesses that aren’t being tested or measured for.

    How much of what was measured resulted from the ex-Devops security practices of organizations measured and resultant outcomes? What does “better outcome” mean, and is that even objectively quantifiable across different organizations or even situations in the same organization? Are we confusing correlation and causation? Every organization is different. The metrics are being held up as quantitative proof but it’s not certain that they are measuring what they purport to.

    Consider this scenario: Operations staff are incentivized through the salary review/bonus process to maintain uptime. Despite working in the test environment, the latest code push to production broke your application. DevOps reinstalls the Docker and the problem is still there. What now? Go back to the old version that ran without crashing but (possibly unknown to your operations team) contains a known vulnerability in some library that is static-compiled into a binary?

    devopsy.kinsta.cloud: What about the argument that DevOps helps to reduce code bloat from legacy environments goes a long way to reducing both technical and security debt?

    Muntner: Reducing bloat might be one of a number of goal of a devops team, but devops practices are just as likely to increase code bloat, opacity, and attackable surface. Security is not an inherent property of DevOps practices. For example, the perception that software which can only be installed through prebuilt packages increases simplicity only creates the dangerous illusion of simplicity.

    Relying on these opaque components increases code bloat in unpredictable and unknown ways, potentially decreasing security posture. If there even is a threat model (usually not, they’re hard and time-consuming to produce, thus expensive) then a popular software package like Hadoop probably won’t more than a box in a threat model, when in reality it’s an aggregation of many components and n-th order subcomponents, each with their own attackable surface area, depending on configuration. If it’s being installed as part of a prebuilt environment, there are many possible security consequences and none of these are addressed or measured in any of the types of tests or metrics that have been described.

    Performing thousands of tests sounds good, but what if tens of thousands of tests are necessary? Or a completely different testing methodology and toolset? Where are will the false negatives be based on the limitations of the existing set of tests? And most important, do the metrics support the claim?

    There are many categories of attacks that these tests aren’t able to identify: Logic flaws are one. Different components of a system understanding the same piece of (malicious or spurious) data to mean different things, another. Sometimes the problem isn’t a bug, it’s architectural deficiency. If it’s deeply layered inside a component of your system that uses its own non standard build system, you probably won’t find it. Software tests are for known, expected code execution paths and interactions, it’s much harder to identify all possible execution paths and orders of operations.

    DevOps: But all of those flaws can be found later in production, when they exist. It is not as if those manually tests are being done away with.

    Muntner: Typically, security audits and assessments are time-boxed. It’s often hard to assess an entire large application inside the time box, let alone all the many opaque components. And then there is the question of pen testers relying on commercial automated tools where even the tests the tools are performing and the included test cases are opaque. Ultimately, they are a snapshot of the security posture at a point in time, given the limitations of the tester(s), tools, methodology, time, and ability of the organization to efficiently consume and triage testing results and lessons learned. Given the rapid release cycles implied by Devops and Agile development, the results can go stale fast.

    Adding unknown layers of dependencies and libraries stretches those already-thin review hours even thinner. Security resources are scarce and expensive. Organizations with mature application security assurance programs only apply those resources to the most critical applications. Outside consultants reviewing one application could cost $20,000 or more.

    Automation is very critical but automated tests have big limitations and often miss configuration management details. Most important, not all vulnerabilities get found. If the automated tools don’t test for it and the pentester didn’t find it within their testing window, it’s still there. Good penetration testers are always finding bugs that weren’t found during previously time-boxed assessments. And so are malicious actors.

    devopsy.kinsta.cloud: All of these possible conditions exist in enterprises now. And certainly, automating bad practices could more rapidly propagate a mess, or thinking it through and automating the right things could improve real outcomes. So why can’t this prove to produce more resilient code and architecture than current methods?

    Muntner: Thinking security testing through and automating as much as possible will yield results, but that can happen with or without devops. I’m not saying devops is invalid, rather that it alone is not responsible for good outcomes. Thinking that an approach delivers more than it really does is only a false sense of security, arguably worse than awareness of insufficient security.

    Secure systems and software development practices like command-safe APIs, network-layer features in TLS, HTTP layer features like CSP, improvements in application and protocol layer firewalls, developers learning to do proper encoding for the appropriate output context, automated testing with tools like OWASP ZAP or commercial equivalents as appropriate for the type of application are all high-impact but have nothing to do with devops.

    devopsy.kinsta.cloud: Security should be part of the flow, an integral
 part of QA, security and functional testing.

















    Muntner: Security isn’t a state, it’s a process. It’s a verb, not a noun. Security ‘what’ should be part of the workflow? Security activities and tests, personnel, all of the above? Should a security organization report to the business management and governance side of management, or the technical side? And why is DevOps better for security maturity than separation of duties?
    DevOps is the approach we take at Mozilla, but the IT and product organizations at Mozilla are full of exceptionally talented people who would do a good job with security using any development process.
    devopsy.kinsta.cloud: Depends on the organization, I’m sure, but if an organization is going to move to DevOps than DevOps needs to be part of the processes.

    Muntner: DevOps can be good or bad but DevOps doesn’t have much to do with security in and of itself, except for some of its popularly misunderstood practices, and then maybe not always in the ways people expect. Standardizing is good unless you standardize on terrible things.

    devopsy.kinsta.cloud: That’s certainly a risk. For many companies they are implementing DevOps regardless, so it’s a matter for security working toward the best way to integrate themselves and their processes.

    Muntner: IT is a business support function. Security is a business risk analysis function. If you standardize and integrate things without understanding threat, risk and security posture, what have you done? Ultimately the decisions are business decisions. Unfortunately, they are frequently made from the perspective of insufficient knowledge.

    There is nothing inherent to DevOps that solves these problems and DevOps comes with its own potential pitfalls.
 The problem is external to what DevOps can deliver.
 






For organizations that want to better understand the maturity of security in their Software Development Lifecycle, OWASP OpenSAMM is a good place to start. It helps organizations understand security maturity means for them specifically and incrementally advance towards their desired posture.

  • Microservices define the next era of IT

    Microservices define the next era of IT

    There is a lot of focus on DevOps and container technologies these days. Both are very strong trends in IT on their own, but each is also proof of a larger shift in IT culture to a microservices architecture. A new report from IDC titled “The Emergence of Microservices as a New Architectural Approach to Building New Software Systems” takes a closer look at the evolution and impact of microservices.

    Let’s start with the obvious question: What exactly is microservices? “Microservices is an architectural approach that draws on long evolving experience in software engineering and system design, including the SOA efforts of the last two decades,” said Al Hilwa, research director, Application Development Software at IDC. “Microservices architecture is enabled by a spectrum of tools categories but is primarily an architectural approach to system design that also requires considerable organizational and cultural adjustment to execute successfully.”

    Put another way, microservices is an umbrella term that applies to breaking systems and applications down to a more granular, modular level. Containers are an example of taking an application and making it into a self-contained component, and DevOps provides the framework for developing, deploying, and managing the container ecosystem.

    By some interpretations microservices and containers are somewhat synonymous. In other words, microservices is just another way of saying that an organization is using container technologies for app development and deployment. IDC classifies microservices more broadly, saying it is based on an approach to organizing software development—more of a concept and less of an actual product or technology.

    The Wikipedia definition of microservices sums it up like this: “In computing, microservices is a software architecture style, in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. These services are small, highly decoupled and focus on doing a small task.”

    Although the concept of microservices is broader than the underlying platforms or tools it uses, IDC acknowledges that the advancements in software development and deployment methodologies—namely DevOps and container technologies—has played a pivotal role in elevating microservices to where it is today.

    The transition from a traditional hardware server infrastructure in an on-premise datacenter to a cloud-based, software-defined infrastructure that can be created, modified, or removed programmatically offers significant benefits for organizations. The microservices architecture enables companies to be much more agile and cut costs at the same time.

    There are some challenges and concerns as well, though. Hilwa notes, “Microservices architecture, by its very nature, is distributed and operates across networks and can thus exact significant performance penalties that must be accommodated with architectural solutions, like caching layers and API management and optimization techniques.”

    In the IDC report Hilwa talks about the need for strong organizational support and an overall shift in the IT culture in order to succeed with microservices. As Hilwa puts it, “Organizations have to increase their tolerance for risk and learn to fail gracefully, fast and often in order to achieve success.”

    Hilwa provides a comprehensive overview of the current state of microservices in the report along with some prescriptive guidance for organizations that are either just trying to understand what microservices is, or those that have already waded into microservices but want a better appreciation for the benefits and pitfalls.

  • Loggly Derived Fields Add Structure To Unstructured Logs

    Loggly Derived Fields Add Structure To Unstructured Logs

    The Gist of Loggly’s Derived Fields Gesture

    A top log management product vendor, Loggly announced an update to its tooling last month in the form of its new Derived Fields. The vendor has drilled down past the summaries in its Dynamic Field Explorer interface into Derived Fields that use metadata to structure unstructured log data.

    Where traditional log management and analysis products have typically required you to create one-off regular expressions for custom analyses—expressions that you must build anew each time you want to put log data under a microscope—Loggly now enables you to use rules rooted in regular expressions that you create once to structure unstructured data when your systems first enter logs into the tool. You can look at resulting Derived Fields instantly, every time you need to do log analysis, according to Hector Angulo, Head of Products, Loggly.

    The necessity for the Derived Fields capability arises from the fact that logs that escape structured creation contain so much untapped value. Until now, Loggly’s approach has been to automatically apply structure to common log types. “But we find that more than one-third of customer logs are still not structured. Even worse, many of these logs were created by developers for their own use, without the expectation that others would need to rely on them for critical troubleshooting. As such, it can be really hard for everyone but the original developer to decipher what specific logs mean,” says Angulo. Derived Fields attempt to retrieve 100-percent of that value from the log data by making analysis possible for any team member.

    How Loggly Dogs Data Using Derived Fields

    Loggly absorbs data from common log types such as Apache or JSON. By adding metadata that describes unstructured log data and then “injecting” context and structure into logs or parts of a log, Loggly’s Derived Fields reclaim obscure developer logs. Fields that uniformly describe the same kinds of data elements across logs enable users to compare data within logs to see what is happening across a DevOps environment.

    Staff members who can create custom parsing rules based on regular expressions can set up the rules the DevOps teams will need at the start. Then, unless you come up with a new rule you want to add, your team members won’t have to create regular expressions again, and certainly not in order to search log data on the other end of that pipe.

    Using Derived Fields

    Using Loggly’s Derived Fields, you can drill down on a specific value in a field. Click on the field name and you can inspect all log events that have the same type of value. A summary chart above the individual event log will automatically refresh to summarize the events of the data from the given value, says Angulo.

    You can easily see other log events that share similar data or characteristics in order to arrive at insights about what is going wrong, or right with the software.

    How Derived Fields Help DevOps

    “Rather than starting with an empty search box, Loggly starts off with bird’s-eye-view summaries using Dynamic Field Explorer,” says Angulo. From there, you drill down to fine grain details.

    The time that Loggly’s Derived Fields save, time that it would normally take to create one-off queries for every analysis, is critical if you’re trying to solve a problem that is preventing end users from completing tasks and generating revenue for the business, says Angulo.

    According to Angulo, Derived Fields specifically help DevOps teams to:

    • Resolve issues faster because it is easier to spot the data that matters.

    • Perform advanced analytics even with legacy applications that send out unstructured, text-based logs that you cannot update or don’t have the bandwidth to update.

    • Extend advanced log analysis to more team members. You don’t have to be a regex wizard to gain insight from unstructured logs.

    • Finally, the intuitive, navigable summaries generated in Field Explorer equate to less training time, less time with reference guides, and more time on data analysis and problem solving.

    A Cautionary Note

    As with any DevOps tool update, the proof lies in how Derived Fields work in your development process. Loggly touts 5K customers using its existing product. I’m sure they’re not trying to chase anyone away.

    Are you using this tool? Do you have other input on this topic? See that empty comments field below?

  • Meeting the Relentless Mobile App Demand of the Enterprise

    Meeting the Relentless Mobile App Demand of the Enterprise

    Talk to any CIO today and one of the most pressing challenges they face is building mobile apps that enterprises want to use as fast as they need them. Most enterprises today have huge mobile app backlogs with no sign of it getting any better.

    According to Gartner, the demand for enterprise mobile apps will be a $40 billion market by next year. And, according to its latest enterprise application trend findings, by the time 2017 comes to an end, market demand for mobile app dev services will grow five times faster, or more, than internal IT organizations’ ability to meet that demand. Gartner forecasts mobile phone sales will reach 2.1 billion units by 2019.

    “Organizations increasingly find it difficult to be proactive against competitive pressures, which is resulting in their mobile apps becoming tactical, rather than strategic,” said Gartner analyst Adrian Leow. “We’re seeing demand for mobile apps outstrip available development capacity, making quick creation of apps even more challenging. Mobile strategists must use tools and techniques that match the increase in mobile app needs within their organizations,” Leow said.

    Interestingly, according to Gartner, employees today use about three different devices daily. Gartner expects that to hit five or six soon. With many firms embracing BYOD, mobile development teams will be pressured to build more apps under more pressure for more form factors than ever before. And they are already pressured. Consider this: A 2014 Gartner survey on mobile app development found that the majority of organizations have developed and released fewer than 10 apps, with a significant number of respondents not having released any mobile apps.

    In its statement, Gartner detailed four practices it feels are necessary to meet mobile enterprise mobile app demand. They’re a good start, but not enough. I share Gartner’s practices in the next paragraph. The practices I also think are necessary to meet enterprise mobile app demand will be included in a follow-up post next week:

     

    1) Prioritize app development

    Mobile development teams are overstretched and have difficulty effectively delivering the growing number of mobile apps in their queues. The result is apps being built on a first-come, first-served basis, with the line of business making the most noise having its needs met first. This lack of value-driven prioritization leads to inefficient use of IT resources and a degradation in the quality of apps delivered. According to Gartner analysts, mobile development teams need to formulate a process of mobile app prioritization that involves understanding the needs of business stakeholders. This will be a key factor in defining common criteria for evaluating mobile app projects.

    2) Adopt a bimodal IT approach

    Integration is often the largest part of the effort of delivering an enterprise mobile app, with many app development teams underestimating the time and resources required for integration during the planning stage of the traditional waterfall methodology they are following. Gartner believes organizations need to replace this traditional IT development approach with a bimodal strategic approach that supports innovation and agility to deliver apps more efficiently and quickly. Bimodal approaches consist of two models: Mode 1 drives the creation of stable infrastructure and APIs to allow apps to retrieve and deliver data to back-end systems without impacting those enterprise applications, while Mode 2 uses high-productivity, agile approaches to quickly deliver front-end app features required by the business.

    3) Use rapid mobile app development (RMAD) tools

    Using development tools that can produce apps more rapidly is crucial for enterprises to help bridge the gap between mobile app demand and supply. Significant innovation is driving this market and replacing traditional coding approaches, such as native development tools, with more effective RMAD tools. There are many approaches to RMAD, including drag-and-drop codeless tools, code generation and orchestration, model-driven development, virtualization, forms construction, and others. These approaches are allowing those with no programming skills or coding ability, such as people in business roles, to rapidly assemble mobile app prototypes and continuously iterate on these designs.

    4) Adopt a mixed-sourcing approach

    Organizations want to have full control over their mobile app development initiatives, however, maintaining a pure in-house development environment is difficult to achieve given mobile is a relatively new competency to many developers. It entails many complexities and specific activities, such as UX design and psychology or cellular coverage testing, which may be more efficiently handled by an outsourced third party experienced in mobile app development. Gartner believes organizations will improve their in-house mobile development skills over time, but currently only 26 percent of organizations are adopting an in-house-only development approach, while 55 percent are successfully delivering apps using mixed sourcing.

  • AppFormix promises to help you manage your cloud infrastructure more efficiently

    AppFormix promises to help you manage your cloud infrastructure more efficiently

    Application Performance Management (APM) tools are great for…well, for managing application performance. AppFormix believes, however, that APM only takes you so far. You also need to manage infrastructure performance.

    The theory behind APM is simple enough: proactively monitor applications to detect and resolve issues in real-time without undue impact on application performance. What if the issues affecting application performance don’t have anything to do with the application itself, though? There are multiple components involved in developing, deploying, and running an application. One very important element of app performance is the infrastructure hosting and delivering that app. AppFormix wants to help organizations bring that same sort of APM visibility and efficiency to proactively addressing issues with the underlying infrastructure.

    In a press release announcing its exit from stealth mode and availability of AppFormix in beta, the company said, “Applications can only perform as well as the infrastructure they run on. Truly reliable performance can only occur when cloud infrastructure is more responsive to the needs of developers and their applications. AppFormix software automatically detects and eliminates resource contention among applications on shared infrastructure. Resource bottlenecks are always moving, and AppFormix is focusing on enabling developers, operations and DevOps teams to use the same toolset to pinpoint them in real-time and establish policies for preventing problems in the future.”

    In the world of DevOps and open source it’s crucial for tools and platforms to play nicely together. It seems that AppFormix delivers on that front. The company claims that AppFormix integrates with OpenStack, Kubernetes, Mesos and other platforms to enable organizations to effectively manage the physical infrastructure while also providing visibility and management for the virtual environment and container technologies.

    “For example, let’s say your application is a flashy, new (and expensive) sports car. Would you drive it on a dirt road? A rickety bridge? A street cratered with potholes? Of course not. But unfortunately cloud infrastructure today IS that dirt road, it IS that rickety bridge, and it IS that street cratered with potholes,” explains Sean Dunnigan in an AppFormix blog post. “If your enterprise does not invest in monitoring and controlling cloud infrastructure, your applications will be in for a bumpy ride. Enterprise cloud infrastructure needs monitoring and control to clearly eliminate bottlenecks and contention to ensure applications have the best infrastructure to “drive” on.”

    What’s interesting about DevOps and the various tools and platforms that have evolved to facilitate and support DevOps is how the line in the sand just keeps getting moved. First application development was the bottleneck so there was Agile. Then infrastructure was the bottleneck so we got DevOps. Then various integration, testing, deployment, and performance automation tools have sprung up to deal with new bottlenecks as they’re discovered. If AppFormix works as advertised, organizations can monitor and deal with infrastructure issues more proactively and move that line in the sand to the next bottleneck.

    AppFormix is available for both Linux and Windows operating systems and is currently available in beta. It is slated for general release in the third quarter of this year.

  • Security at the Speed of Innovation

    Security at the Speed of Innovation

    With the advent of the cloud and platform as a service (PaaS) offerings, developers are innovating faster than ever.  But with this rapid pace comes risk. Companies quickly implement new apps to meet their business demands, but the increasing complexity of the infrastructure being used – both in the cloud, on premise, or a hybrid combination – makes it difficult to ensure vulnerabilities are not introduced or vital corporate information is not exposed. In the quest to ensure that policy and governance are in place and security risks are negated, IT often brings innovation to a screeching halt.

    Is there a happy medium that enables developers to continue accelerating innovation, but makes IT confident there’s adequate security? In this article, we’ll look at the importance of governance and policy, and outline the five fundamental security requirements that will help enterprises bridge that chasm that currently separates development and IT.

    Policy Debt – The Downside of Faster Innovation

    Security flaws are not new. Enterprises have faced serious security problems for decades. As today’s computing landscape becomes increasingly diverse and complex with increasing use of the cloud, new vulnerabilities are introduced, which exacerbate problems with existing security methods, such as simple usernames and passwords and an over-reliance on network perimeter security models.

    The rapid introduction of new developer technologies far outpaces an enterprise’s ability to vet them. As a result, enterprise IT faces a tradeoff: rapidly adopt new technologies to satiate business demands thereby accepting more risk, or slow down new technology adoption with the hope of mitigating risk. Tension between enterprise IT and the rest of the business is the inevitable outcome. More importantly, because new apps have so many dependencies, as soon as one is put into production, an enterprise almost immediately incurs policy debt.

    Policy should enable enterprises to track these dependencies, but in today’s IT environments, each piece of infrastructure or cloud resource has its own unique policy model, operational mode and lifecycle.  As a result, enterprises end up with a layered cake of dependencies, which is difficult to keep track of and ensure no security holes are left unpatched.

    Most of today’s security technologies – like gateways and firewalls – are either point-based or palliative solutions. They treat symptoms rather than root causes. And, they only work for traffic that’s routed through them, which precludes most of the server-to-server interactions that happen within the datacenter. Reliance on gateways is akin to locking the deadbolt on the front door of your house but leaving the back door open.

    These simply are not enough to address the proliferation of attacks that have paralyzed companies and exposed customer and consumer data. Consider the three major security Linux exploits – Ghost, Shellshock and Heartbleed – that came to light in the past year.  To patch their systems against these problems, companies needed to find the vulnerabilities and reset servers to activate the fix, which resulted in a cascading failure of servers, downtime and a host of other performance-related issues – simply because there’s no centralized way to apply policy and determine whether a resource in use is vulnerable.

    The Fundamental Five Security Requirements in the Brave New IT World

    To fully protect their increasingly complex IT environments, there are five fundamental security requirements no enterprise should be without:

    1. Resource Naming and Identity – To secure access of a workload or process, it needs to be named so that it can referenced as logic is applied. There is no notion of identity for the applications running today. They’re simply workloads running on a CPU.  If enterprises want to apply policy, they will need an overlay operating system that gives identity to that workload, so it can be organized into a manageable, policy-driven hierarchy.  This type of naming scheme is being done for Internet domains, but there is nothing comparable for the data center that enables IT to find, maintain and operate workloads residing there, regardless of where they are physically run.
    2. Chain of Trust – In today’s data center topologies, no actual proof is needed to ensure that a holder of a token is exactly who it claims to be.  To guarantee a high level of proof of identity and actual trust of components, enterprises require a central signing authority that issues tokens to all clients and internal components. This ensures that the management backplane is resilient to token and credential theft, and the system offers a high degree of trust.
    3. Context Aware Policy – Most systems rely solely on role membership to make authorization decisions, such as  “who” can do “what” to “what.” But enterprises need to go further than that, with the use of a dynamic scripting language for authorization decisions. Every authorization decision will require information over and above simple role membership (date and time, authentication strength, targeted resource and more). The result is a system that adapts more easily to enterprise authorization requirements.
    4. Control of the Network – The network is a vital part of any enterprise system, and controlling access to network nodes is key for enterprise IT governance. Today, network access control is a hodge-podge combination of network perimeters, firewalls, VLANs, VXLans, proxies and security appliances. Workloads exist in network segments and workload connectivity is a function of network segment configuration. Over time, these configurations become rigid, brittle and difficult to maintain. To prevent this, enterprises need to control the network within the cluster so that any workload running must be given explicit access to the network and other workloads (via policy) in real time and regardless of physical properties of the infrastructure.
    5. Ephemeral Service Credentials – Applications and services require connectivity and sometimes credentials to talk to one another. Once an application gets a credential for a service, the application becomes responsible for safeguarding that credential. However, credentials sometimes leak — they can appear in log files or configuration files, and can even be scraped from memory. What enterprises need is a new approach to service access that binds a credential to an individual network path. This ensures that even if a credential were to leak, it couldn’t be used from another location on the network. Moreover, with this method credentials live only as long as an individual workload in the system, rotating every time the application stops and starts again. By taking this approach, the burden on applications for handling credentials is reduced, strains of NetOps and network perimeter security models at scale decreases, and weak links in the trust chain are removed. The result is a more secure system that you can trust without changing your existing investments.

    About the Author/Derek Collison

    derek-collison

    Derek Collison is founder and CEO of Apcera. An industry veteran and pioneer in large-scale distributed systems and enterprise computing, Derek has held executive positions at VMware, Google and TIBCO Software.  As CTO and chief architect of cloud application platforms at VMware, he co-founded the cloud computing group and designed and architected Cloud Foundry,the industry’s first open PaaS. During his time at Google, he co-founded the AJAX APIs group.