devopsy.kinsta.cloud

Tag: devops process

  • Culture Change: Metrics, Tools Make a Difference

    Culture Change: Metrics, Tools Make a Difference

    Change is in the wind.

    This time of the year, as winter weather gives way to springtime renewal, this concept isn’t a cliché but rather a concrete, visceral experience. You go outside, smell the earth defrosting and know for certain that change is happening all around you. You can feel it.

    Such is the case on the current DevOps landscape. Seemingly every conversation, with everyone from end users to industry analysts, reaffirms that people are making significant progress toward greater adoption. For an increasing number of organizations, DevOps is maturing from a set of budding goals into a tangible reality.

    At the center of this massive shift within the software factory is the issue of cultural change; as we know in all things DevOps, culture trumps all else. So how are organizations affecting, empowering and measuring all this seismic transition, in particular the human elements?

    During the recent DevOps Virtual Summit, leading practitioners representing industries ranging from banking to travel and technology shared their views on the progress of cultural reinvention.

    From management tips, to tools and even metrics useful for driving transformation, here are some of their observations:

    1. What’s the most critical cultural aspect of planning?
      Kazi Whitfield, director, Quality Engineering, MUFG Union Bank, N.A.: Scalability on this is the key; if you’re doing this at a new organization, you need to understand how to scale it, and starting at some smaller level helps you get there. It also helps each individual and those components that are changing to truly understand what’s changing, and why it’s changing; I think that’s a huge part of the conversation that has to happen.
    1. What’s the biggest obstacle to cultural change?
      Eric Warheit, senior manager, Environment Management, Southwest Airlines: The challenges that we’re working through are organizational in nature. It goes back to the question of localization, and people maintaining a mindset that they’ve previously had, and what we do to create the conversation around what’s possible and the key impacts that we want to have. Ultimately, it’s about being productive with quality, and creating those common values.
    1. How can tools enable cultural change?
      Eugene “Bear” Lehenbauer, group manager, Information Technology, Citrix: If I have to pick one that links them all, a ring to rule them all, that’s when you have an orchestration platform that can take what individual teams are already using and plug right in, or you can take what needs to be automated and automate it; that orchestration harness can now become that common thing that links them.
    1. How can you ensure cross-culture collaboration?
      Warheit: Creating that concrete example, that proof case tying analytics to it, so that we have real measurable ways to make good business decisions, and being able to foster that perspective so that we can look across the organization, as opposed to in a localized way, is really critical to be able to foment that collaboration that’s needed.
    1. How do you measure cultural change?
      Daniele Romano, product owner, Continuous Delivery Pipeline, ING Bank: The most important thing as an organization is to able to apply an analytical approach, meaning that for each team or department you should look at the current situation and have in mind the target situation, and then have different means to guide everyone toward that target.
    1. What’s a key cultural change metric?
      Lehenbauer: We’ve a developed an agility rating scale that allows people to earn scores for something as basic as change management. Based on how those key metrics behave, and there are several of them, but they are fact-based and they’re user impact driven for the most part, they reward the right behaviors. Those types of things can draw out some of DevOps principles results, but also have hard data behind them.
    1. How do you recognize advocates of cultural change?
      Whitfield: Being able to phase reward and recognition into the process is key. Organizations are doing a lot of great things and making millions of dollars, and as the individuals that are coding or testing are driving it, you have to understand how you get to those people’s hearts, to make them passionate about what they’re doing. That’s something that becomes key and it’s something that’s different per organization.
    1. What’s the best way to gauge successful cultural transformation?
      Romano: There is a really only one critical human metric, which is happiness. So, if engineers are happy and they can deliver fast, at a high quality and with intense focus—that’s a very unique but important metric.

    So there you have it, some of today’s most critical elements in driving the cultural transformation necessary to succeed in the DevOps world.

    If you missed the DevOps Virtual Summit you can still view recordings of all the sessions including this customer panel, and stay tuned to this space for more excerpts and highlights.

  • Top 5 Reasons to Back Up Your OpenStack Environment

    Top 5 Reasons to Back Up Your OpenStack Environment

    I am a big fan of lists. If you recall the movie “High Fidelity,” John Cusack’s character Rob and his co-workers—armed with an encyclopedic knowledge of all things musical—compile “top five” lists for every conceivable occasion, openly mock the tastes of people, and, every so often, sell a few records. In the spirit of a lists, here are the top five reasons to back up your OpenStack environment. We thought that it would be helpful to many.

    If you’re currently working with OpenStack clouds, or are in the process of embracing it, you need to include backup and recovery for OpenStack on your list of to-dos from the start. You may ask, “Why? I have orchestrated/automated everything, I just need to spin resources up and down.” Cloud environments are fundamentally different from the virtualization we have become used to. Virtualization relies on infrastructure for resiliency. In the cloud, distributed applications are responsible for their own resiliency, and are independent of the underlying infrastructure. Therefore, cloud apps need backup and recovery options, lest you face a complete machine, data or file meltdown.

    We’ve compiled the top five reasons to backup your OpenStack environment. Take a read and it will become clear why you must include backup in your OpenStack plans.

    1. Compliance — Compliance may seem like an issue for only certain industries—finance and health care come to mind—but nearly every organization has to deal with compliance issues. For some, it’s compliance to external regulations, while for others it’s simply complying with internal policies and best practices. Backup for OpenStack is an essential component for compliance; it ensures the availability of the data at a specific point in time within your OpenStack applications. I was recently involved in a dialogue/string on LinkedIn watching experienced OpenStackers go back and forth debating “pet vs. cattle.” “I am getting tired of it,” one person said. “This is cloud. It’s about scaling up and out … With automation and orchestration there is no need for cattle.” This conversation went on for days. Theory is great. Practice is reality. What people failed to discuss is that compliance or the business units will drive many requirements (such as backup) either today or tomorrow. End of story.
    2. DevOps — Backup is often the forgotten tool for DevOps. Containers are great to use, but the ability to test with real data is an issue. The reason to use backup is quite simple: test and development. With a copy of a production environment, you now can test upgrades and new features in your agile development process. Furthermore, if you lose a machine or your data, or you end up with corrupted files while in the middle of testing or development, your life will be much easier if you have an appropriate backup solution in place. Restoring machines, data and files is a better option than re-creating them entirely. Cookbooks and recipes are great as well, but you still need to couple it with the appropriate “ingredients.” It is like playing with a fake kitchen as a child … do you want to make breakfast with plastic pancakes and toast? Nothing beats the real thing. Yes, more resources are required for one over the other, but when something goes wrong—it is unfortunate—people point fingers.
    3. Disaster recovery plans — While the terms “backup” and “disaster recovery” often are used interchangeably, they are actually not the same thing. You might have what appears to be a great recovery environment (ideally one that mirrors your production environment for high availability), but without a comprehensive backup solution, your recovery might not be the painless experience you expect it to be. Perhaps the solution for intelligent disaster recovery is to marry backup with a Swift environment so you can recover from any point in time, anywhere in your organization.
    4. Save time — Often recovery can be a frantic process, driven by a business need to have the machine, data or files recovered as soon as possible. Or, more likely, sooner than possible. If you are trying to re-create something under that type of environment, there’s a likelihood that mistakes could be made. This becomes increasingly true the more complex and mature your cloud becomes (think about the growing number of workloads and applications). A much easier approach—one that provides you with peace of mind—is to have a comprehensive backup solution ready and available for when you need it. There’s also another reason backup can save you time: When you have to re-create the lost information, you are taking your time away from something else—and that means you are taking a big hit to your productivity. Save yourself and your organization a big headache with backup.
    5. Security and forensics — You never know when you are going to be asked to provide data or files related to a security issue or a forensics investigation. The right backup solution not only makes it easier to assist with the activity, but think how good you’ll look by being able to deliver the information the forensics team needs quickly.

     Are you currently looking into backup solutions for OpenStack? Now that you know why you need backup for OpenStack, learn more about how to backup OpenStack.

  • Carving through the hype – take the DevOps maturity assessment

    Carving through the hype – take the DevOps maturity assessment

    As anyone who spends time with their family over the holidays can attest, the notion of maturity is a highly subjective concept.

    Whether it’s sizing up the next generation’s advancement or defaulting to time-honored inter-sibling behaviors, annual get-togethers offer tangible proof of what’s new and those things that likely won’t ever change.

    In the domain of DevOps – an umbrella concept that encompasses a massive array of processes and supporting capabilities – gauging the maturity of your organization’s standing is also a highly interpretive task.

    Given that “DevOps” means something different to nearly every organization, based on its unique makeup, methodologies, tools, and most importantly, business goals, attempting to measure progress against any outside model may seem an effort of limited value.

    That said, as an increasing number of organizations seek to embrace DevOps, and growing recognition that well-worn stories of leading-edge “unicorns” don’t relate closely to most people’s real-world expectations, many teams are left wondering where their own efforts stand, especially in comparison to their peers.

    To help acquit this pursuit, and allow organizations to scope their progress, CA recently launched the second iteration of its online DevOps Assessment.

    Split into three short surveys covering core limbs of the DevOps tree – “Develop and Test”, “Release and Deploy”, and “Manage and Monitor” – this free tool offers detailed insight into how your organization’s maturity stacks up to that of other practitioners.

    For instance, in the domain of development, how well do your teams fare in defining and managing requirements, and then presenting them to others in a structured manner? Related to release cycles, are your teams truly meeting business-driven expectations for more frequent application deployments, shorter lead times, and higher success rates? Or, when it comes to applications monitoring, are you able to distinguish the most critical and recurrent problems and how they truly impact business services?

    These are perhaps some of the most fundamental questions that one could apply to measurement of DevOps adoption in general, but further investigation attempts to take a closer look at more finite, underlying issues.

    Are your development teams accurately replicating realistic conditions from production systems to create virtual services and testing scenarios? Are your application release deployments fully automated across development, test and production environments? Can your teams monitor complete business services that link user experience, transactions and applications to infrastructure to pinpoint any root cause issues?

    The best thing about the survey is that it immediately provides visibility into how your answers compare to those of previous respondents. There’s no fuzzy math or editorial interpretation (though we may highlight trends in a report at some point), merely instant validation, and the option to create a customized report card charting where your efforts stand in relation to those of your peers.

    If you’re wondering precisely just how to achieve your long-term DevOps goals comparative to where you stand today, it might even be worth leveraging some of the involved questions for the purpose of evolving process and methodology.

    The only remaining questions are: Do you think that you’re ahead of the curve? Are you afraid to find out just how far you’re behind the crowd? And most importantly, what do you have to lose?

    Take the survey today and get a real-world fix on your overall level of DevOps maturity. Guaranteed it’s a far more informative activity than debating family and friends over whose turn it is to carve the turkey, or wash the dishes.

  • Native Integration of APM Driving DevOps Maturity

    Native Integration of APM Driving DevOps Maturity

    With applications performance monitoring established as one of the fundamental cornerstones of today’s DevOps methodologies, it’s easy to overlook the reality that effective instrumentation of such tools remains extremely difficult. Specifically, the biggest challenge remains creating performance metrics that address both IT and larger business initiatives.

    There’s no question that APM is one of the most widely adopted and deeply embedded elements in today’s DevOps world, serving as the critical source of measurement that enables more transparent and agile applications. Yet, despite the fact that performance monitoring solutions, and their respective providers, are considered to be among the most mature on the DevOps landscape, customers still struggle to optimize the systems to meet their specific requirements.

    However, as the market continues to advance, specifically via direct integration of widely-adopted APM tools with other strategic applications management systems, such as e-commerce platforms, organizations are finding a faster path to success, and bigger-picture DevOps benefits. One such partnership, announced in late August, saw market leader New Relic expanding integration with e-commerce platform provider Magento, owned by eBay.

    Jon Azrielant, director of marketing at online retailer Jewelry.com, a Magento customer, said that his company initially ran into significant obstacles in adapting New Relic’s APM capabilities to its processes, despite the comparative strength of the tools.

    “We had a really difficult time implementing New Relic initially, it was a painful process,” Azrielant said.

    While the APM tools provided Jewelry.com with the data necessary to generate vital alerts, along with more agile sysadmin and operations monitoring, the more silo-based, fragmented approach that it employed could not empower the retailer to circumvent traditional walls between apps support and service support teams, Azrielant said.

    Leveraging the extension of New Relic’s Software Analytics Cloud within Magento, the company has created more advanced instrumentation to inform critical business decisions. Among the most helpful of these measurements are those relating directly to customer experiences, while correlation of such metrics with Jewelry.com’s bottom line has also advanced executive decision-making, he said.

    “The critical result of this integration is more complete visibility across applications and business performance, importantly tied to the actual business ramifications,” said Doug McIver, director of product for Magento. “In the past, merchants had to do a lot of difficult integration to use these systems together, so obviously this is a hugely valuable capability for us to be able to offer, especially to customers with highly performant site requirements.”

    Moreover, Magento claims that it is growing its own customer base by tying APM more closely to its platform. Likewise, New Relic officials concede that to help advance crucial DevOps workflows, it must partner with such platforms to deliver APM data within the context of business requirements.

    “If we’re talking about how to win in world where we’re iterating the end users’ experience on a regular basis, we have to put performance metrics within context of the UI, and the business results; working from the same data sets and getting people on the same page is incredibly important,” said Abner Germanow, senior director of solutions marketing at New Relic.

    While pure performance stats are “near and dear” to New Relic for obvious reasons, the solutions provider is also attempting to advance a broader array of information that drives closer collaboration through new “iteration and experimentation”. New Relic is also focused on further developing the specific types of information most critical to key constituents such as e-commerce platform users. The need to create a common language for those DevOps efforts is amplified by the fact that development and operations officials are often unsure which metrics are critical to the business, and vice versa.

    “Instrumenting code has always been a challenge, and we’ve worked very hard to ease that pain across a lot of different platforms; there’s always another language where work can be done, and we look to our customers to help lead us into the next phase of instrumentation,” Germanow said. “We’re seeing a lot of pressure around the need for constant change and iteration in industries where it’s truly a matter of survival, such as e-commerce, publishing, digital governance and finance – places where the industry is behaving very differently now than it has over the last 50 years.”

    This work, and the subsequent APM integrations offered via platform providers such as Magento, have allowed end user organizations to remain focused on customer experience and sales, the ultimate goal of DevOps, said Jewelry.com’s Azrielant.

    “In terms of how this relates to the DevOps revolution, this is the advancement that is bringing everything together and allowing everyone to look at the same numbers, to share common APIs, to streamline reporting and event response, and define everything in uniform terms,” Azrielant said. “Ostensibly, the most basic tenet of DevOps centers around the unification of these disciplines, and you can’t support that without the unification of this information.”

  • Code Review – Write your code right

    Code Review – Write your code right

    Writing code is definitely subjected to individual thinking, logics and perspective. How one writes code is completely depend on their technical and logical skills. Two different people can write the same code in an entirely different tactic (May be both logics of the code are right in their own way). Even two persons can write bad code as well as good code as per the same standard.

    In the real world it is most likely possible that one writes the code and same will be read multiple times by many other developers to fix the bugs, improve the features, understand the code. That is why each language has its own style of coding and guideline to make it more meaningful and understandable.

    Most of time the Developers struggle to understand the code written by someone else on the team. I have even heard people saying that it’s difficult to read someone else’s code rather writing same from scratch.

    I would like to highlight a few compiled techniques in once place to make writing code better.

    Classical Method

    Classical method is a tools based code review. Developers write their code and submit it for review in some 3rd party tool like review board, Rietveld, or Collaborator for somebody to review the code.

    The tool will be used to track code changes comments, patches and review. Someone can merge code to the active branch once it is reviewed. Reviewers can have a look at the code and provide comments and inputs to the relevant piece of code.

    The good side is that it can be used for almost all sorts of old source code tools and is very much compatible with most of the version control tools.

    The problem with this mechanism is that it needs another tool other then version control system to maintain code review related information.

    Integrated Method

    To overcome the classical method downside, review systems have been introduced with in version control tools; a very good example of this is GIT. Where there is no (or rarely) 2nd tool needed to perform code review activity. Code review can be done via pull request even before merging. Comments, issues, patches can be tracked within Version control system with same workflow.

     As mentioned earlier everything related to review remains with in the same system and that is the biggest benefit of this system.But this mechanist also needs the same time and effort for someone else to go through the code and sign off.

    Extreme Workflow

    Pair programming falls under this category. In this Technique, two programmers work as a pair together on one workstation. One writes code while the other, the observer, pointer or navigator, reviews each line of code as it is typed in. The two programmers switch roles frequently.

    This way coding and its review happen at the same time by 2 programmers simultaneously.

    Some say that 2 developers working on same code at same time doesn’t make any sense, as they will be end up spending double man hours. It may lead to problems with practical implementation of the technique.

    But the good part is that they don’t need to spend extra time to understand and perform code review. It removes to and fro communication between developers. You don’t even need another tool for additional help to submit patch.

    In conjunction with review techniques I would like to add one more point about who can review code.

    • Leaders review the code: – One of the very common processes is that 2-3 leads will review the code always from the entire team (number depends on team size). They are always whole responsible for performing the review, checking the code before it goes for further consumption.
    • Anybody can review the code: – Another technique is that anybody with in team can review the code. Only restriction is that the one is writing code cannot review his own code because this doesn’t make any sense to review your own written code. There is no leader who is responsible for code review. A junior programmer can also review code written by senior, as there is no reason to allot hard role and not allowing others to review code.

    Whatever practice we use it is very necessary to have a look at code by another pair of eyes and have code reviewed.

     

    References-

    https://en.wikipedia.org/wiki/Code_review

    https://en.wikipedia.org/wiki/Pair_programming

    http://www.scmtechblog.net

  • The Art of DevOps

    The Art of DevOps

    This is the third in a series of posts on DevOps. The first written by my colleague Lee Reid was titled The Simple Math of DevOps. The second The Calculus of DevOps was written by IBM client, and my friend Carmen DeArdo of Nationwide. Lee introduced mathematically improving delivery throughput by leveraging DevOps to improve Trust. And Carmen used Calculus to imply that the business value of IT can be improved by adopting DevOps to increase the frequency of releases. In my post I would like to visit the art of DevOps , or the art of adopting DevOps by picking up right where Carmen left off – by looking at the culture side of adopting DevOps.

    Lee and Carmen have already alluded to some basic premises behind DevOps as an approach – reducing batch sizeand improving collaboration. These result in reducing complexity, increasing throughput, and improving trust. Let’s talk about how to get there.

    As I have worked with customers adopting DevOps, across company sizes, across industries and countries, there are four common threads that make DevOps adoption successful, leveraging all three aspects of DevOps adoption – culture, process and automation. These are:

    1. Reduce batch size: The most effective way to manage risk and quality, while increasing speed is to reduce the batch size in each iteration or ‘sprint’. This is a mind shift to release smaller, more frequent new versions. Where it is not possible to release new versions that frequently to customers, release small batches to a pre-production area. The deliverable in pre-production is tested and made customer-ready, and then released to the customer at formal release dates. As Lee and Carmen have both discussed, reducing batch size is a pre-requisite to improving both trust and business value.
    2. Shift-left Operational engagement: Get operations engaged right from the beginning of a project/initiative. This has been a core principle of DevOps. Do not leave Ops in a silo that one throws code over the wall to, but have them engaged in the development process, right from requirement inception stage. This allows Ops to have visibility into what is coming from Dev; allowing them to deliver the right production-like environments, as and when needed to Dev and Test teams. It also allows Ops to shift operational concerns left, left into the minds of developers, making them more Ops conscious and engaged in what happens after they hand-off to Ops.
    3. Continuous funding: This is a significant change in an organizations funding model to enable continuous availability and engagement of SMEs and business functions, previously not engaged on a continuous basis – like security, legal, marketing, etc. Carmen in his post mentioned several examples of ‘waiting for…’ work, someone else to do something and environments. Another ‘waiting for’ in enterprises is for funding. Funding is typically provided in a waterfall manner, with specific hard dates (Months, quarters, fiscal year) and gates, not suitable for a Continuous Delivery model. Funding too should be continuous.
    4. Create a ‘Product Management’ team: This team includes a Designer, Development lead, operations lead and an Architect at the minimum (4-in-box). They own the ‘product’ through its entire lifecycle, and beyond transient projects. They are responsible for long-term thinking, design thinking, evolutionary architectural thinking, and overall ownership of the product, from concept to end-of-life.
    5. Setting up a DevOps Center of Excellence (CoE): This center of excellence is not an administrative organization or a ‘tools/enablement group’, but a place where DevOps adoptees come to learn from each other and share expertise and lessons learnt. As organizations adopt and scale DevOps adoption, this CoE also becomes a source of DevOps coaches, helping teams and programs adopt DevOps, and own the organizations DevOps framework – their own flavor of DevOps.

    Adopting DevOps is an art. It is not as simple as hiring a consultant who can show how to improve processes; it is not just buying and adopting tools to automate manual tasks; and it is not going through ‘fall-back-into-the-arms-of-the-person-behind-you’ exercises to build trust. It is a combination of all three – process improvement, organizational and cultural change, and automation with tools to replace manual processes. Adopting these in any enterprise, outside of startups, requires overcoming organizational inertia. By changing culture not by edict, but by showing the need and value of change. By building trust, not through meetings, but with improved communication and transparency. By improving business value, not by measuring mandated KPIs, but by improving organizational agility and throughput. Overcoming organizational inertia, by delivering towards a common goal for the business, irrespective of where in the organization one is, and what role one has. By developing a culture where everyone takes responsibility of delivering value to the business.

    Watch the webcast series hosted by devopsy.kinsta.cloud in partnership with IBM or download a copy of DevOps for Dummies book for more information.

    About the Author/ Sanjeev Sharma

    SanjeevSanjeev Sharma, CTO and Distinguished Engineer – DevOps Technical Sales and Adoption, IBM Cloud Unit, is a 20-year veteran of the software industry with expertise in DevOps, Mobile Development and UX, Lean and Agile Transformation, Application Lifecycle Management and Software Supply Chains. He is a DevOps Thought Leader at IBM and speaks regularly at conferences. He has written several papers and is the author of the DevOps for Dummies book.

     

    Contact Sanjeev on linkedintwitter

  • The Calculus of DevOps

    The Calculus of DevOps

    This is the second in a series of blogs, partnering with Lee Reid, who wrote the first installment “The Simple Math of DevOps” and Sanjeev Sharma both of who are outstanding DevOps Thought Leaders from IBM.

    I am picking up on Lee’s mathematical DevOps model to apply a calculus perspective, a “calculus of DevOps” if you will, (OK, not quite core Big Bang Theory Calculus but rather approaching it from the aspects of engineering queuing theory).

    One of the cultural challenges that DevOps poses is, that some aspects of it run counter to engineering principles that many of us learned with respect to capacity and response time.   When I worked at Bell Labs, we would apply queuing theory to determine how to best package information to be sent across the network. A basic premise revolved around a tradeoff between throughput and response time. On one hand, the more messages that were packed into the packet, the more the payload would improve capacity because the overhead of processing the packet of information would be spread across the messages.   However, waiting to accumulate enough call processing information meant that the packets would be transmitted less frequently, increasing the response time.

    A simple example of this would be making trips to the grocery store. If one wanted to spend the least amount of resources (time, gas, car wear, etc.) on shopping, then one would make less frequent trips and then buy a larger amount of food. So if the Smith family made one trip a month and spent 2 hours, then their average time per week spent on shopping would be 30 minutes.   If the Jones family went every week and spent an hour, then they would be spending twice the time and four times the gas on grocery shopping. However if the Smith family run out of an item (say cereal), they would have to wait, on average 2 weeks before this was replenished. The Jones family average on any item that they ran out of would be 3.5 days in comparison.

    So when the 2014 DevOps report was published[1] which states that areas that deploy more frequently get customer feedback more quickly, that is to be expected. But what’s not expected is that these organizations also have higher productivity.   So exactly what is happening here that seems to be in a basic conflict to the principles of throughput and response time?

    Keeping up with the JONESES

    If we think of a pre- DevOps state for a medium to large enterprise, it may be typical for applications to only be released (deployed into production) a few times a year to provide business functionality. (I assume there are more frequent releases for “unplanned work” like defects and other changes or operational work). For the sake of this discussion, let’s call this the Smith Application. So from a business perspective, the business wants to pack as much work into the Smith releases as possible given they only have a few cracks a year at getting new business functionality into production.

    SMITH APPLICATION CHARACTERISTICS:

    • Deliver Business Value 3 times a year (every 4 months)
    • Cost is C1
    • Business Value (BV) delivered is V1
    • Average time from concept (e.g. Story) being ready to be pulled from the backlog to the time it is delivered to the customer is 2 months plus any additional wait times (e.g. prioritization)
    • BV delivered per year is 3*V1
    • Cost per year is 3*C1

    Now let’s transport over to the Jones Planet where the application is released into production with new business capabilities at the end of every 2 week iteration.   In this case the business is more concerned with providing a constant demand flow into IT than they are with debating over whether a given feature/story is going to be included in this particular release.   Instead there is a constant prioritization (or continuous planning) process that ensures the highest value work is being fed into the IT pipeline.

    JONES APPLICATION CHARACTERISTICS:

    • Deliver Business Value 26 times a year (every 2 weeks)
    • Cost is C2
    • BV delivered is V2
    • Average time from concept (e.g. Story) being ready to be pulled from the backlog to the time it is delivered to the customer is 1 week plus any additional wait times
    • BV delivered per year is 26*V2
    • Cost per year is 26*C2

    Based on the previous engineering analysis, we would expect the following:

    1. SMITH annual cost (3C1) < JONES Annual Cost (26C2)
    2. SMITH annual BV (3V1) > JONES annual BV (26V2)

    So the tradeoff (we would believe) JONES is making by delivering more frequently is that they are spending more money and are delivering less business value on an annual basis.

    However the data from the previously referenced DevOps report is just the opposite. In fact the findings are that enterprises that deploy smaller units of work more frequently have much higher quality and much higher productivity (Capacity). So what’s going on here?

    A NEW PARADIGM

    If you go up to any IT team (even an agile one) and ask them why they don’t deploy business value more frequently, they will generally refer to the following types of wait states of waste that they encounter:

    • Waiting for work (lack of consistent flow)
    • Waiting for somebody else to do something (dependencies from other teams)
    • Waiting for environments (contention)

    One major root cause of this is the dependencies that arise from large enterprises packing a lot of work into infrequent releases. From a planning perspective, the process is elongated due to prioritization discussions because being on the wrong side of the line can mean delay amounting to months, in delivering a product to a customer.   Ask a business portfolio leader what their highest priority is for IT and I am guessing they can tell you in a matter of seconds. However, most large organizations can spend several months a year planning their annual portfolio of work for IT. Some organizations spend more than 50% of their new project build time prior to a single story entering an agile development team’s backlog!

    From a development perspective, packing a lot of work into releases means many more dependencies that have to be managed across initiatives and impacted applications. From a testing perspective, downstream environments (ST, UAT) have to manage much more information in terms of application versions and especially test data that complicate and slow down the process. The complexity of test data increase is proportional to the number of pairwise relationships that exist within the datasets required for testing.   As such, the testing complexity will increase non-linearly (perhaps proportional to the square of the dependent systems) and from a release and change perspective; there is a lot of coordination necessary given how much is being changed at one time.

    All of this work to manage all of the dependencies found in medium to large enterprises completely eliminates any potential savings one might get from batching work into a single delivery. In addition, the possibility of having to pull some work out of a release (due to quality or business readiness or some other reason) sometimes leads to complicated source code management (feature branches in some cases) that further complicate the process of merging frequently into the trunk. These types of practices also serve to drive up technical debt. So looking at the problem from this angle shows why results we are seeing make perfect sense.

    Productivity is inversely proportional to the number of dependencies in a release

    As Lee related to in the initial blog, a lot of an organization’s success depends on trust. And one factor of trust is clarity which speaks to understanding the changes that each release is bringing to the IT and customer environments. So a factor that is working against trust is the notion of delivering more things less frequently and better controlling change somehow, is a good notion.

    Trust is inversely proportional to the number of dependencies in a release

    There is another hidden component of waste associated with “big bang” initiatives. Go back to our grocery example for a moment. Chances are that the Smith monthly excursion results in buying some stuff they end up not using. Perishable items are not spread out very well when they aren’t purchased more frequently. Likewise in IT planning, we tend to put a bunch of stuff in our IT Delivery basket that we don’t know for sure that we will need and we end up with a lot of wasted effort for features that are hardly used or not needed at all. Some enterprise estimates are that 50-65% of what they put into large releases does not actually result in true value to the business. Doing more frequent releases allows a constant discovery process to better guide the usage of IT resources on what is truly of value.

    Waste is proportional to the size of a release.  

    In other words:

    Yield (Value per IT investment) is proportional to the frequency of releases.

    REVISTING THE MATH OF DELIVERY TIME

    So if we revisit Lee’s model

    Lee Reid Blog equationand apply the “Calculus of Release” as we stated above, there are increased costs associated with having larger and less frequent releases in medium/large enterprises.

    • Planning cost TPLAN
    • Development complexity TDESIGN + TDEVELOP + TBUILD
    • Testing Complexity TTEST
    • Release and Change Management Complexity TRELEASE + TDEPLOY + TEVALUATE

    TRUST % – Even if everything was proportional between the JONES and SMITH models, the component of trust that impacts clarity, consistency and collaboration, is much lower for large (SMITH) releases.  Given that TRUST has a multiplicative impact, this alone can account for the results found in the studies.

    CULTURAL CHALLENGE

    While clearly one needs to be sensitive to the changes being introduced into the business, having the ability for the business stakeholders to pull capability when they want it and having it available more frequently is not only good for customers but also has extreme benefits from an IT perspective in terms of increased productivity and reduction of technical debt. The challenge is to convince leaders in both the business and IT areas on the merits of doing more continuous planning and frequent deliveries.   Unicorns may have led the way in this type of thinking but it is actually the horses that will benefit most from this type of culture shift because it is the horses[2] that have the most to gain in mitigating the dependency factor which can undermine quality, productivity and cycle times.

    Read more about the topic in the next blog of this series The Art of DevOps by Sanjeev Sharma.

    [1] “2014 State of DevOps Reports”, Puppet Labs, IT Revolution, https://puppetlabs.com/sites/default/files/2014-state-of-devops-report.pdf

    [2] “The Phoenix Project”, Gene Kim, Kevin Behr, George Spafford, http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592

     

    About the Author

    CarmenCarmen DeArdo is the Director of Application Development Tools and Technologies at Nationwide Insurance. Carmen is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across Mobile, Distributed and Mainframe and other technologies. This includes recommendations and implementation of technologies used across the development life cycle (e.g. IBM Rational tool suite, open source technologies).

  • The right answer was “process”

    The right answer was “process”

    The question is “why is there no one-size-fits-all automation and orchestration solution”?

    Automation, a component critical for continuous deployment particularly as it extends beyond app operations and into the broader environment to encompass network and security operations, too, continues to be a custom build kind of thing. There’s no single automation tool, framework or pattern that can be generally applied to a significant number of environments. Mi casa es no su casa, after all. That’s because environments don’t actually look like power point slides; your environment is not the simplistically diagrammed firewall –> load balancer –> web server typically found in decks extolling the virtue of one solution over another. They are far more complex, comprising not just multiple tiers of interdependent and interconnected apps and services, but multiple chains of infrastructure services spanning the gamut of security, availability, mobility, performance and access control. The specific flow through and between these services  is unique. Indeed, the intimate relationship between network latency and app performance is well known, and thus the navigation through the critical yet complex site of services required to deliver an application could easily be considered a competitive advantage if it’s able to keep performance under the 5 second toleration limit on response time apparently now the standard for consumer satisfaction.

    So it’s no surprise that automation and orchestration, responsible for automatically setting up the triggers and routes required for application traffic to correctly traverse the right set of services for optimal performance, would not be a simple, turn-key solution. One size does fit all. It must, like its business process focused counterpart – BPM – be customizable and easily changeable.

    BPM? Are you seriously?

    I’m dead seriously. BPM – business process management – has been around far longer than automation and orchestration of IT systems and its focus is, as its name implies, on process. Automation is included, but it is the orchestration of business processes that often change in the face of variable business conditions that makes it an invaluable tool for optimizing business and a source of competitive advantage. “The ability to change quickly is essential. Businesses change their key processes 4-7 times per year. The driver for change can be internal or external. New opportunities can arise. New partners or customers need you to support a different way of doing business. Federal or international regulations can require you to change your processes.”  (Making the Case for BPM, Jim Rudden)

    Consider for a moment this data from a 2012 Bearing Point survey on the goals and impact on culture:

    Top BPM goals are increased efficiency (93%), increased transparency (90%) and the realization of standardization potentials (85%). A number of BPM goals have an impact on corporate culture, e.g. increased transparency (rank 2), increased customer orientation (rank 5), disclosure of weaknesses/ bottlenecks (rank 6) and the reduction of mistakes/ error rates (rank 9).

    – Bearing Point Business Process Management Survey 2012

    If many of these goals and impacts sound familiar to you, they should. Because what DevOps ultimately needs to focus on is exactly the same as the focus of BPM – process. In an application world, these processes require applications and services. IT must be able to support these process changes in the application and service infrastructure.

    BPM (Business Process Management) has known this for years, and it is the orchestration of the process – the workflow – from task to task, from approver to doer  that generates the efficiency gains achieved with that technology.

    Indeed, adopters of DevOps who focus on the “big” picture would do well to consider the similarities to BPM which touts as benefits such things as:

    • Process quality improvements
    • Continuous process improvement
    • Reduces costs
    • Improves business agility

    It would be difficult if presented with that list of benefits with no attribution to determine whether it was referring to DevOps or BPM. That’s because the operationalization of business through process orchestration is not all that different from the operationalization of IT through process orchestration.

     

    CustomerCentricBPM.pdfAnd processes are not, nor should they be, turn key and one-size fits all. They should be tailored to address the needs of a specific operational or business goal, i.e. deploy an application or secure an order from a customer, based on the systems in use to achieve those goals. BPM shines where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. DevOps will shine where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. It isn’t about racing through a given task with automation. Speed isn’t efficiency and, if you do the math, a focus on automation without similar focus on process can lead to even greater inefficiency.

    And that’s why there is no one size fits all automation and orchestration tooling; because the goal isn’t automating tasks, it’s orchestrating process. Coordinating workflows. Ferreting out inefficiencies.

    Orchestration is the glue that binds automated tasks together.  For example, recently a missed route update caused some unexpected down time during what was considered a routine*  decommissioning of gear. A missed route update. In other words, someone forgot. That task might have been automated, it might have just needed to be kicked off. But it wasn’t.

    That’s the kind of thing that would likely not happen if the process was governed by some form of orchestration; if there was a clear list of what needed to be updated or changed based on the removal of that particular component. If orchestration was involved to provide visibility and oversight of the operational processes needed to perform such a routine task. This does not mean the tasks should have been automated; in fact automation is not strictly required to achieve the visibility and oversight necessary because processes are about workflows and hand offs, not specific tasks completed during the steps that comprise  a process.

    Which leads us right back to Business Process Management. Take a deeper dive into the aforementioned Bearing Point survey and you’ll find more similarities with DevOps:

    • Crucial elements for sustainable BPM are clearly defined roles (95%), top management support (92%) and a clear mandate (92%)
    • Typical implementation barriers such as the lack of resources (53%), sponsorship (32%) and experience (25%) need to be identified early and addressed in a structured way
    • BPM as a continuous (=sustainable) approach is already adopted by 59% of companies, coordinated and implemented either centrally or locally
    • Involvement of stakeholders (90%), process oriented thinking (90%), organizational institutionalization (82%) and a well-communicated concept (75%) are elements of communication and change management which must accompany the establishment of sustainable BPM

    That’s because both are ultimately about process, with automation being only one of the tools in the tool chest that support implementation of a continuous approach to enabling agility in the process layer. Indeed, about 38% of the companies in the survey use software to support their BPM initiatives while another 37% do not. A spreadsheet with the right Six Sigma calculations can be as powerful as any automation framework, toolset or API in identifying the dead space between tasks that introduce inefficiencies and delay application deployments.

    BPM has, like DevOps, brought tangible, measurable results to the table. Bearing Point found that almost three quarters of the surveyed companies (72%) were able to achieve measurable benefits through BPM such as process lead time (72%), process cost (68%) and error rate improvements (67%).

    So rather than focusing too much on automation, adopters and proponents of DevOps would do well to up-level the conversation and start looking at process orchestration and improvement as the goal. The successes BPM has had on the business  side of the equation tell us that doing so will provide a better path to achieving success with DevOps in the operational side of the equation.

     

    * I’d argue that in today’s environment of hyper-sensitivity to availability that nothing that touches a service or gear in the critical path of an application is “routine” but, hey, it is what it is.

  • DevOps & Continuous Change

    DevOps & Continuous Change

    A remark by a colleague while waiting for the coffee machine to complete its cycle started my train of thought. “Should we have multiple minor releases or just do a few major ones in a year?”

    In large organizations, due to many factors, the turnaround time for a single successful release is quite extensive; but if the conceptual change which we are talking about bringing whilst using the #DEVOPS methodology would not only reduce these long cycles but deliver quicker and quality software.

    I have already discussed in my earlier article “DEVOPS: Getting our organizations aligned” on the traditional approach and how the combination of this two (DEV & OPS) has been proven beneficial in various organizations and how many are still hesitant in embarking on that path.

    Continuous Change which encompasses both Continuous Integration and Delivery is paramount to achieving this objective. To attain the objective of delivering rapid and consistent value to the stakeholders regularly, we have to have Continuous Change happening.

    I am sure we have already moved out from the Mainframe era to a more Distributed systems state. But unfortunately we have yet to let go of that culture where we pile up components on a single quasi Distributed Application, instead of creating smaller independent, yet interdependent systems.

    So, back to the same question; shall we go with a few Dinosaur sized, labored releases or make multiple short and swifter runs? In my opinion, we should strive towards the leaner model. Easier said than done you say! Agree. If I have to, then I would go about it this way:

    In an environment having a suite of Distributed Applications, start with Analyzing the components which make up the Major release, List out the changes (fixes and new features) which are planned, asses functional and technological impact, Identify dependencies, upstream/downstream connectivity’s, architectural deficiencies, list out the identified subset of components or group of components by functionality, which can go as an independent release.

    I am resisting the urge to say ‘standalone’ as with the current set of complex intervened collection of applications, it would be an incorrect statement. It would rather be a subset of components which can be independently upgraded. Now the big task is to review and decide if this subset or group would add functional value to the overall suite of applications without the rest of components going in.

    Once identified, the success of this release working as expected would be in the extensive testing (Unit/ Regression/Performance) of these components, and as we gain confidence in this split and deploy procedure, we can start with, prioritizing and scheduling critical must have features or bug fixes which also should be backward compatible with other components lagging behind. We can even come up with scenarios where these components are not just getting updated frequently, but would also have multiple independent tracks. Of course it goes without saying the testing and signoff process is strictly followed albeit in a shorter window.

    Deployments to production have always been a source of heartburn to the OPS teams, we have often seen finger pointing and firefights to contain change fails. In order to maintain stability and maximize availability, stricter controls and reduced deployment windows are put in place. In promoting frequent and smaller component level changes, OPS will also be in control of what’s going in, and smaller component level releases can be rolled back quickly within the Green-Zone window if the change is not working as expected. Though I would categorize this rollback, as a process fail than a deployment fail, and should be reviewed very seriously with right mitigation plan and lessons learnt retrospection, Continuous learning after every successful or failed sprint ensures competent and confident releases.

    Another major factor impacting multiple runs of Production releases is the long Green-Zone windows, by moving to the quicker and leaner deployments cycles, these requests can be substantially reduced, and can eventually aim to reach a stage where changes are deployed online without extensive downtime.

    Similarly, frequent maintenance activities when there are no planned deployments, effectively has the same risk on environment unavailability. Analyzing and moving to a state where these activities are conducted online without a Green-Zone would alleviate the maintenance downtimes and ensure seamless availability, adding stakeholder value.

    I have not touched on the Continuous Integration or the Automation of the deployment process for obvious reasons, we can’t achieve the objective of Continuous Change without having the Integration process in place where incremental changes to code are build into a package swiftly and seamlessly, even a sanity or basic testing might be integrated to ensure the quality of the builds. Same goes for automating the deployments, develop & integrate tools required for rapid deployments. We have seen instances where automating a single activity or revisiting a process has created substantial time savings.

    While it’s easy to propose changes to the Release Process or advocate rapid deployment cycles, we have to ensure adequate checks and balances are in place, compliance adherence and audit trails are created.

    Now to the most important measure of this article, Cost Savings!

    Proponents of the Dinosaur Releases would argue that the common activities which entail a single release like the builds, Testing, Release Management, OPS availability or reduced GZ would cut costs and multiple runs to PROD would logically increase costs.

    But it’s definitely the other way around, we wouldn’t need to feed the Dino anymore, it will be a cost worthy rapid delivery with optimized resources, a leaner process where DEV and OPS are working together to ensure quality value added deliverables are made available to business faster.. This cultural change and collaborative effort also creates new vistas of growth and provides a perfect platform to excel. The cost benefit analysis would eventually be in favor of Continuous Delivery.

    Opportunity cost is another variable which has not been considered; an innovation or a new feature available to customers quickly with reduced time to market cycle will provide the much needed edge in these competitive times. I recall some conversations where a few must have’s were pushed out due to various reasons, frustrating the business.

    DEVOPS was never meant to replace the traditional approach to software development; rather it is an efficient use of the existing resources within a collaborative model to rapidly deliver quality software products. A major cultural shift in our approach is required and to reach that goal we need to embrace the shades of grays, many a times over 50 (sorry, couldn’t resist) when it’s not truly black or white.

    Thank you for the time.

    Visit my blog http://askhurram.com/

    Read On, Live On.

     

  • ChatOps: Communicating at the speed of DevOps

    ChatOps: Communicating at the speed of DevOps

    DevOps is about knocking down barriers between developers and operations teams, and in doing so reducing organizational frictions wherever possible. And that’s certainly the spirit behind ChatOps – the term for the hyper-collaborative way of running DevOps, including operating aspects of systems and infrastructure, through online chat.

    Proponents of ChatOps say it streamlines communication in a centralized and powerful way. Born within Github, ChatOps evolved from its open source chat bot, Hubot, which has changed the way the web-based Git repository runs operations through the automation of deployment, graphing, monitoring, provisioning, and even mitigating security events.

    Think of Hubot, a Node.js app that is commonly run on Heroku, as a robot that helps keep everyone informed in a centralized place and in real-time, or in a near real-time, way. There are many benefits to ChatOps, says David Mortman, David Mortman, contributing analyst at Securosis, including the integration of ops activities with notifications into one place, making it easier for everyone to know what’s going on across the enterprise, as well as everyone throughout the organization having the ability to see error messages on events as they occur, he says.

    Github, as a highly distributed organization, with its several hundred employees spread across 103 cities, can use all of the communication efficiencies it can muster. As Mark Imbriaco, Ops engineer at Github, in his presentation ChatOps, Augmented Reality for Ops, uses Github, email, internal apps, other web systems that they’ve built to help teams communicate asynchronously – and yet they needed an option for the real-time or “semi”-real time communication. “I say semi-real time because, when we talk about chat, the social contract at GitHub around chat doesn’t require you to be immediately responsive to everything you see. Chat is still asynchronous, but it’s more synchronous than GitHub or email or other sorts of notification mechanisms,” said Imbriaco.

    Without enhancement, Hubot doesn’t do a whole lot right away. But when added to chat with one of the dozen or so connectors that include access to Google Talk, IRC, Skype, and Hubot monitors for events that come from within chat or from one’s systems and applications. “The magic happens when you build additional functionality into him to automate, well, everything,” Imbriaco said.

    Hubot will do certain things, such as notify people not present in a chatroom that they were mentioned, phrases can be registered so that notifications to those interested in certain events will be notified. Github has also plugged Hubot into the GitHub service itself. Today, when anyone creates an issue or pull request in GitHub, according to Imbriaco, a link and definition of the event is sent to chat. While Github already has ways to send notifications, by entering events real-time into the chat stream, others can jump in and comment, help, or otherwise advise on events and interact as they’re occurring.

    As for the adoption of ChatOps, Github isn’t the only one: but it’s not yet widespread among DevOps practicing organizations, either, based on our reaching out to numerous organizations. Nor, is the concept especially well known.

    That’s not slowing Owen Zanzal a developer at systems management provider VividCortex down from adopting ChatOps. Zanzal says in his blog post, ChatOps at VividCortex, that, despite ChatOps being off to a slow rise, “ChatOps has really revolutionized how we operate,” he wrote.

    The first Hubot mission at VividCortex was to help improve how code is deployed. The problem Zanzal highlights is that encryption keys and new developers apprehension around code deployment, was slowing down production.

    To streamline deployment, Zanzal moved the execution of Capistrano commands to a Jenkins (a continuous integration server), and created Hubot commands so that jobs could be created using a Jenkins exposed REST API. When done, developers needed only run a chatroom command, such as “hubot stage webapp” or to deploy “hubot release webapp,” Zanzal explained. “Now, no one is blocked by the gatekeeper from pushing code into staging or production, and every developer can push code when they are ready. The unfamiliar Capistrano commands were simplified into much easier to understand Hubot commands,” he wrote.

    VividCortex is also using ChatOps to fetch logs and more quickly remediate issues as they arise, Zanal wrote.

    Chat-as-Ops
    GitHub has also pushed ChatOps much further than simply as a place to stage and deploy code. It’s using ChatOps as a way to help actually run ops. For instance, Imbriaco spoke to how Hubot and ChatOps is used for efforts such as rebalancing shards across its file servers. “Honestly, it took me a little while to get used to some of these. We have some [Hubot actions] that do things like rebalance shards across our file server. And that’s horrifying to me, because the consequences of failure are so bad. But the reality is, I’m either going to run it from a chat room, or I’m going to run a shell script from a terminal. If I ran it here, at least everyone saw it. So at the end of the day, this is repeatable, it’s tested, it’s commonly used,” Imbriaco said. “And if it stopped working we would notice,” he said.

    And one time when Github was fending off a distributed denial of service attack, Imbriaco needed only enter the name of their service provider to direct traffic and the attack was successfully mitigated.

    Moving to ChatOps isn’t without risk, however, Imbriaco explained. A lessoned learned when it became a little too easy to deploy to Amazon Web Services. “We went from zero to way too much spend on Amazon in about two weeks, when I turned up the ability for us to provision EC2 hosts from Chat,” he said. But that issue was resolved by giving users information at the time of provisioning, such as including metadata into the instance that details who created it, as well as informing users how much it’s likely to cost,” Imbriaco said.

    Despite the efficiencies of ChatOps, don’t expect a widespread embracing of ChatOps, but more likely a slow adoption. “GitHub is a flying unicorn. They’re like Netflix,” says Mortman. “People will pick up some of what they are doing but it’s too advanced for most,” he says.