devopsy.kinsta.cloud

Tag: it ops

  • Rugged DevOps: Good, Bad & Ugly

    Rugged DevOps: Good, Bad & Ugly

    The Good: DevOps has been a success for Dev.

    With higher expectations, better processes, more management focus and new tools like advanced PaaS and containers, developers are churning out more and more software faster than ever. Success! Or is it?

    The Bad: DevOps has been a success…for Dev.

    For Dev teams, this may result in pats on the back and high fives. But for Ops, this can look like one thankless tidal wave of work after another. The same software velocity that is praised in Dev is currently overwhelming their counterparts in Operations. And frankly, many DevOps tools released in recent years happen to benefit Dev much more than their counterparts in Ops. (My pet theory is that developers at IT automation firms are acting and thinking like…developers – focusing their creative energies on solving Dev problems.) A similar problem holds true for IT Security teams who want to build quality in by “shifting left” – see  https://devopsy.kinsta.cloud/2015/12/10/getting-rugged-devops-right/ . From the vantage point of IT Operations and Security teams, they are doing more work than ever but without any more human or IT automation resources. To add insult to injury amid their increased workloads, many IT Ops teams have to invest countless hours (away from family and personal time) in training courses and migrations to reorient their processes to fit into shiny, new Dev-centric “DevOps” tools and paradigms. 

    The Ugly.  

    Work. Overwork. Glitch. Search. Find. Rework. Repeat.

    A quick glance at IsItDownRightNow.com will show you which IT Operations teams are in a living hell at the moment. The other thing you will notice is that these IT Ops teams are from some of the most reputable companies on the planet. They are talented teams but stuck in rework cycles of web performance problems, security vulnerabilities and outages.

    These problems get the attention of everyone. Operations…and Dev…and Security…and customers…and Finance…and auditors and the press.

    No one likes rework. And no one likes the panicked fire drills that cause panicked rework. So what causes those rework cycles?

    Drama-free IT. Visibility Cures Chaos.  

    Much of this chaos can be tied to miscommunication. Boring? Maybe. Simple? No. Part of miscommunication is cultural. (Hint: OrcaConfig’s “How to Screw Up DevOps” white paper covers the cultural aspects of miscommunication). But miscommunication is also cured by automation solutions that promote transparency and visibility.

    If we want drama-free IT we need IT automation solutions that eliminate miscommunication and missed assumptions between Dev, Ops and Security.

    What better way to do this than with full transparency and visibility?

    Bird’s Eye Views:

    Many IT automation tools cater to SysAdmins who are interested in managing configurations of individual servers. That’s great if your focus is on individual severs. But IT Operations, eCommerce teams, Application Owners, Compliance & Security teams and IT Managers often care more about the health and performance of their application ecosystem (including related applications, middleware, databases and configurations that tie them together). So there is often a mismatch between the tools that SysAdmins and Developers have versus what IT Ops needs.

    Operating at Enterprise Scale Requires at-a-glance Visibility:

    IT Ops teams need bird’s-eye views of applications, databases, middleware and ecosystems to promote ease of ongoing operation. They need to be able to see, at a glance that a configuration compliance problem exists and they need to know right away the location and nature of the error. Large scale IT Ops teams do not have the luxury of toggling back and forth between multiple reports and multiple screens searching for that non-compliant, broken needle in an ecosystem haystack. If they are faced with a glitch or an outage, they need to find and fix it right away. One size does not fit all. Dev, Ops and Security each have their own interests and their own need to view their application through a lens that is most meaningful for them. Multiple ecosystem views eliminate guesswork between Dev, Ops and Security and save precious time in the find-fix cycle.

    Forrester Research spells it out well in their 7 Habits of Rugged DevOps white paper, Habit 1: Increase Trust And Transparency Between Dev, Sec, And Ops:

    “I&O pros pride themselves on understanding the interconnectedness of the hardware and software environment, and they are unnerved by complexities that poorly governed delivery practices create because the firm rates I&O pros on their ability to improve the performance and stability of the production environment. To maintain the environment, they must reduce the chances of an outage or interruption of service, and they must reduce the MTTR when an issue does occur.“

    Understanding that interconnectedness depends greatly on how much intuitive ecosystem visibility we provide to our colleagues in IT Operations. But it’s not just IT Ops that benefits from visibility, Dev and Security do too.

    Developers are able to write more secure code when they get early, in-context feedback. As David Mortman, distinguished engineer and Dell chief security architect, said, “Security issues are product quality issues, and no one wants to write buggy code.”

    Even better than efficiently finding and fixing errors is of course to prevent them in the first place and to proactively eliminate the damage they cause. Here too, the answer is better visibility. With true application ecosystem visibility, Dev and Ops teams would be able to model relationships between applications, middleware and their configurations so users will know that “changing this, affects that”.

    My teenage son has better visibility via his smartphone as to his driving route and weather conditions than I have for my multibillion dollar IT operation.

    Visibility into the Future:

    Ecosystem visibility is a key to communicating between Dev, Ops and Security and running complex application environments in the present. But what about the future?

    Release automation tools and custom scripts are excellent at deploying changes across large environments. Unfortunately they can be just as efficient at deploying typos and other mistakes across large environments, causing the chaos that we discussed earlier. Here again Operations teams would benefit from improved visibility – the ability to preview and pre-validate proposed changes across the entire ecosystem well before those changes are deployed. This is the real promise and savings of ecosystem visibility. Better visibility, with more use-case lenses cuts out the miscommunication that causes too many performance glitches, security vulnerabilities and outages. This chaos takes a toll on our IT Operations teams, our employers and our customers.

    What Next?

    We are soon entering a new season with new budgets. The temptation to “buy something” will be great. In the spirit of Rugged DevOps my recommendation is to take a full, bird’s eye view of your SDLC and to shift Ops and Security leftwards, building their critical considerations into your processes and into your IT automation tool selection.

  • Video did not kill the radio star – IT Ops Continues on

    Video did not kill the radio star – IT Ops Continues on

    The Buggles classic “Video Killed the Radio Star” may have been a hit song, as well as hotly contested idea in the early 80s, but given the retrospect of the last 40 years, we know that video did not kill the radio star – it simply changed him. Just as MTV failed to kill the radio star, DevOps will fail to kill IT Ops.

    Today’s dynamic virtualized and cloud-filled world has significantly changed IT. With Software Defined “Everything” and virtualized compute, network, storage, etc., everything is fluid.  And to top this off, bandwidth is unlimited and inexpensive; making it easy to offload and move things in ways we couldn’t have dreamed possible just a few years ago. As a result, the infrastructure supporting these technologies is highly distributed, and can no longer be served from one place. All of this is significantly changing the way IT Ops, well, operates. In order to understand how IT Ops has changed, we need to look back at how it was. We don’t need to look back far, however.

    Most organizations have individual business units. Be it a large organization with separate businesses, or a smaller one with separate departments, these business units often find themselves competing over shared IT resources. Five years ago, business units vied for the time of shared application developers that lived in IT. Now, with the explosive growth and maturation of rapid delivery application development technologies, and with more and more revenue generated or influenced online, it makes sense, that these business units lobbied to the CIO to bring the app developer and application support functions within their control.  Application development has become a critical part of most businesses today..

    Once this happened, the distribution of budget also changed, taking full control of the IT budget away from IT Operations. Today, as much as 30% of the IT budget is directed by the individual business units. This happened because the business units were not getting the prioritization they wanted from the shared application developer approach. In the eyes of the business, IT couldn’t move fast enough for them, or didn’t have the same priorities that the business unit had. No one was satisfied.  So, the executive teams moved the application development and support budget to the respective business units, partly to stop the bickering.

    Some of these business units took their funds and used them to outsource development or to purchase software-as-a-service applications. Once cloud platforms and inexpensive, unlimited bandwidth arrived, the business units had alternatives, and they used them.

    This is all logical for business units, but it does not come without risk to the organization. With applications developed, controlled, and delivered outside of the purview of IT Ops, there were concerns about whether these applications would meet HIPAA or PCI compliance standards, IT’s security standards, business continuity guarantees, or back up schedules. A compromise was needed. What most organizations have come up with is to have IT Ops handle the general infrastructure, and allow each business to do what they want with the rest – about 30 percent of the budget. How does this look?

    Most apps today are componentized. If you take a retail app for example, the product browsing, the shopping cart, the payment page, and the confirmation and fulfillment are each separate applications. While the underlying infrastructure – how the app is configured, monitored, and supported – needs to be handled by IT Ops, each component lives on its own schedule of updates, and has its own team that maintains and fine tunes these apps to make sure they’re accountable to the business goals. But who, you might ask, comprises the team if not IT Ops?

    DevOps: The intersection of Development and Operations

    Enter DevOps. DevOps is one of those terms that can mean different things to different people, but in its most simple definition, DevOps is the convergence of development and operations. Forrester analyst Mike Gualtieri hit the nail on the head when he said: “The goal of DevOps to make the process of deploying applications faster and smoother.”

    Without “DevOps” all application issues (outages, etc.) will be routed to IT Ops. Because IT Ops is shared, has other priorities, and does not necessarily know the goals of each business unit, this is not the most practical option. Sure, IT Ops are sharp and talented people, but they specialize in maintaining infrastructure – they are not application experts by trade. Some organizations have hundreds of different applications spread among a dozen business units. There is no way that anyone could expect IT Ops to understand the specific needs and behaviors of each of these specific applications.

    The application developers know the app best, they know the requirements, and how to diagnose issues. They need a deep solution to trace transactions and end user experience. And, they know best what the app needs in a general sense….how much CPU, how much storage, how much RAM, etc. However, what they don’t have is expertise with VMware vSphere, Cisco UCS, VCE vBlock, FlexPod, IBM hardware, NetApp storage, EMC storage, etc. They are developers. They can manage their application operationally, but what happens when they run into an infrastructure problem? This is where IT Operations comes in.

    Gualtieri also said, “DevOps is a loosely defined set of emerging practices to get developers and operations pros to work together. Developers and operations professionals are often at odds. Developers want to release software more frequently; operations professionals want to protect the stability of the infrastructure.”

    This is why the portion of the IT budget that is moving away from IT and into individual business units continues to grow. It’s not necessarily surprising when people start talking about an end to IT Ops, or in some cases, and end to Operations all together (NoOps).

    However, despite the fact that organizations are moving away from IT Operations overseeing 100 percent of the budget, to a chunk of it now being managed by individual business units, without IT Ops each unit would control all of their own IT. Imagine unified communications without IT Ops. Do you really want each business unit in charge of that? No, you want economies of scale.  You want it to be shared – the network, the bandwidth, the storage – IT Ops will remain to manage all of that.

    As Gualtieri noted, “developers and operations need to stick together to best meet the demands of the business.” Recent research drives this point home. A study on the state of DevOps by Puppet Labs found that IT performance strongly correlates with DevOps practices like proactive monitoring, and continuous delivery and integration. In fact, according to the research, DevOps practices can boost IT performance and ultimately improve a business’s bottom line.

    IT teams have a more consultative and strategic approach, working in partnership with developers and other users of IT infrastructure to ensure that they all have access to what they need without bottlenecks or barriers to entry. Despite the fact that there appears to be room for both IT Ops and DevOps within organizations, it is clear that a shift is occurring.

    IT Ops: Making a move to large service providers

    Taking advantage of the latest technology innovation, as well as economies of scale, many organizations are now using service providers instead of owning their own infrastructure. One could say that this is pushing the IT Ops folks out, and they would be correct, but don’t worry they’re not becoming an endangered species. Instead of working for large enterprises, IT Ops professionals are working for the large service providers. They aren’t going away, they’re just moving off site.

    The budget will likely continue to go down for IT Ops on premise, but the cost of hosted infrastructure will go up, so the budget for IT Ops will even out. IT Ops is not disappearing, it’s transforming.

    In a time when Ops have become trendy and heated debates over terms have garnered the attention of the tech community, it comes as no surprise that the majority of us are left wondering what the future holds. One thing is for certain, be it DevOps, IT Ops, or SecDevOps, there will continue to be places for Ops within organizations.