devopsy.kinsta.cloud

Category: Enterprise DevOps

Comparing DevOps to traditional IT: Eight key differences
You can make the argument that in the Software Delivery Life Cycle (SDLC) value is created in two places. First, when the software is created (development) and second, when the software is delivered to the customer and feedback received. The rest is transportation. The faster one can move code from the hands of the developer to the customer and receive feedback, the more value can be realized by both. Don’t get me wrong, there is benefit to various “transportation” activities (testing, QA, staging etc.). But those are not value creating activities, they are risk mitigating processes.

A DevOps organization understands this value equation. They focus on the processes that add value, organize themselves around those actions and do the best to minimize the risk. Traditional IT gets it too – at least conceptually. However they treat all stages in the SDLC with equal importance or worse emphasize the wrong stage(s).

In general, when comparing DevOps organizations to traditional IT, DevOps will have policies and practices that differ from those of traditional IT along the following eight key dimensions,

Planning & Organization
1. Batch Sizes: Go from Big to Mirco
Traditional IT has a bias for going big and for good reason. First, most development shops grew out of the waterfall method which by its inherent nature takes a lot of time. Second, since releases are costly and disruptive, operations allows developers only a few windows a year to release software. And third, its simply not sexy – doing big is how you get promoted. As a result, development organizations maximize productivity, by planning big projects, that involve a lot of code, bundled (hastily at times) into a release, and jammed into production.

A DevOps organization takes the opposite point of view and believe that small is beautiful. They understand that large batch sizes are inherently complex, risky (since there are so many moving parts), and hard to coordinate. Small batches sizes, on the other hand, are simple, easy to understand and test rigorously, and less risky. If things go wrong the impact is minimal and it is much easier and faster to fix. In other words, by going small organizations are able to perform more frequent releases and become more responsive to the customer.
1. Organization: From Skill Centric Silos to Dedicated Cells
Traditional IT is organized around skill centric silos. For the most part silos work, they band like skills together, drive greater utilization, and benefit from economies of scale. However, where these “cost optimized” silos break down is at the hand offs. In a typical IT environment, a new feature has to go through at least 3 – 4 silos before the customer gets it. It is not uncommon for an idea / code to spend 80% of its time waiting or ping-ponging back and forth between silos.

A DevOps organization also operates in a silos but at a different cross section. Here teams are arranged in “cells”, consisting of dedicated cross functional teams, and focused on only one application. By creating this self-sufficient cell that consists of developers, testers, business analysts, and operators, an idea can move from one stage to another without hand offs, promotes cross-training / understanding (key to collaboration), focuses the team on the end goal, and encourages the “shift left” thinking. As for utilization and scale benefits, the rules for response states that for every ¼ reduction in cycle time, productivity will improve 2x, and operating costs by 20%.
1. Scheduling: Centralize to Decentralize & Continuous
Efficient scheduling is at the heart of a Traditional IT organization. Since resources are pooled, projects are usually clamoring for access to SMEs and / or infrastructure. To get around this, enterprises have invested in sophisticated scheduling planning systems. And while these systems are quite sensitive, they are inherently inaccurate, take up to much time to manage, and in some cases become the very bottleneck that they are trying to alleviate.

In a DevOps organization scheduling is pushed to the local cell level. The combination of smaller batch sizes, dedicated teams, and automated processes makes scheduling more simpler to operate. How? First, forecasting is limited to the very near future (2 – 3 weeks) where the teams have better insight. Second, there is no fighting for people’s time since it is a dedicated team. Third, there is no waiting for infrastructure since it has already been defined and automatically provisioned. And fourth, it eliminates time consuming loop backs to management for escalations and decisions.

Performance & Culture
1. Release: Turn a High Risk Event to a Non Event
In a Traditional IT organization, releasing software into production is a high risk proposition. It is fraught with issues, escalations, and constant fire-fighting. The process is tightly managed, governed from the highest levels, and requires participation from all parts of the organization. It is not uncommon for IT to set up war rooms staffed 24×7 for weeks before and after the release.

DevOps organizations, on the other hand, make the release of software into a non-event much as possible. They reduce risk by daily integrating code into the trunk, automating testing, ensuring all environments are in sync, reducing batch sizes (as mentioned above) etc. In other words, they only promote code from one stage to another after they are confident that it will work in production. Thus eliminating all the hoopla about release windows and they are able to move new functionality into production at a much faster clip.
1. Information: Focus is on Dissemination vs. Actionable
Both types of organizations generate and share a ton of data. The difference lies in how the teams uses data. In Traditional IT, information is generated by specialists (e.g. operations team), bundled together with other data into a massive report, goes through management approval, is then shared with other managers, who then send it out their specialist (testers, developers etc.). In most cases the report goes unread, simply because there is too much data, not timely enough, and / or not the right data. In other words, information is shared but poorly consumed, and rarely used to take any actions.

Within a DevOps organization, it’s the team cell that gather and creates the data. Since the data is to be consumed locally they only collect the data (also automated) that the team deems is necessary. And since the data is processed within the team, it eliminates the time lag of creating lengthy reports, manager approvals, and queue time (sitting in some ones email box). As a result, the team is quickly able to read and react to the data – resulting in faster feedback time.
1. Culture: Do Not Fail vs. Fail Early
Traditional IT is fundamentally a risk averse organization. A CIO’s first priority is to do no harm to the business. It is the reason why IT invests in so much red tape, processes, approvals etc. All focused on preventing failure. And yet despite all these investments, IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues.

A DevOps organization is risk averse too but they also understand that failure is inevitable. So instead of trying to eliminate failure they prefer to choose when and how they fail. They prefer to fail small, fail early, and recover fast. And they have built their structure and process around it. Again the building blocks we have referred to in this article – from test driven development, daily integration, done mean deployable, small batch sizes, cell structure, automation etc. all reinforce this mindset.

Measure
1. Metric: Cost & Capacity to Cost, Capacity, & Flow
Silos work for Traditional IT because they are directly tied to its measurement model – cost and capacity. IT is measured on how much can it get done (capacity) for the least amount of money (cost). It is no wonder that cutting cost, while trying to keep capacity constant, has been all the rage in IT over the last two decades. It’s one of the reasons why outsourcing became so popular.

And while this model works great for stable legacy applications, the newer “system of engagement” applications require an additional metric – the element of time. DevOps organization understand this paradigm shift and have added “flow” as an additional metric. Flow forces an organization to take a look at its end to end cycle time, identify areas of waste, calculate true productive time, quantify quality, and focus on activities that add the most value.
1. Definition of Done: “I did my job” vs. “it’s ready to deploy”
Another source of measurement difference is in how the two types of organizations define done. In Traditional IT, done is defined as the specialist doing “just” their part and handing it off. In essence, their done is more focused on meeting the hand off deadline vs. making sure what is done is deployable. This varying definition leads to some bad habits such as sub-par work, quality loss, finger pointing, and an overall sub-optimized process.

To get way from the finger pointing mentality is another reason why a DevOps organization prefers to create a dedicated cross functional team. With every member of the various parts of IT represented in this “cell” structure and all of them being held accountable for one and only one thing, they all end up having just one definition of done – bring quality software to market.

If you would like to learn more about this topic, please join me at IBM InterConnect 2015. I will be hosting several workshops on DevOps.

About the Author/Mustafa Kapadia

Mustafa Kapadia is the Service Line Leader for IBM’s DevOps practice, a business advisory practice focused on helping large enterprises transform their software & application delivery. He has over 17+ years of experience in the tech. space, both as a service provider and a management consultant.

Prior to joining IBM, Mustafa was a management consultant with Deloitte’s Strategy & Operations practice. He lives in San Francisco Bay Area, is an avid blogger (when time permitting), a speaker, and adviser to start ups.

Connect with Mustafa LinkedIn| Twitter
February 17, 2015
Why are we still asking, “Can DevOps work here?”

It’s easy to get tired of buzzwords, especially in the software development and testing world where they’re so prevalent. But while it’s easy to blame the thought leaders and evangelists for creating them, and publishers and powerful marketing teams for inserting them into every content channel imaginable, the buzzwords that truly stick around do so on their own merit.

Agile, big data, and cloud have been written about and philosophized for years—and I hate to break it to you, but they’re not going anywhere. While they may not be the panacea that some make them out to be, their impact on the way we develop, test, deploy, and release software has been documented in every market and industry around the globe. This gives the words their staying power. Not relentless press coverage that’s been covertly sponsored by marketing and PR dollars.

Members of the academy and esteemed buzzwords in the software hall of fame: join me in welcoming our newest member: DevOps.

You may say it’s too early, and that DevOps hasn’t been around long enough to earn such universal acceptance. Many quickly cite John Allspaw and Paul Hammond’s 2009 presentation, “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” as the real inspiration behind Patrick Debois’ coining of the term, and if you’ve never viewed the session, I can’t recommend it enough.

Skip ahead to today and we’ve got multiple conferences that focus on DevOps exclusively. Devops Days, DevOps Summit, DevOps Connect and DevOps Enterprise feature some the most respected speakers and writers alive today.

But they’re not the only ones speaking there. CTOs, CIOs, chief architects, IT heads, and QA directors from every bank, retailer, hospitality group, government agency, and software development shop are there, too, and with a captivatingly similar story behind their employer’s decision to “do” DevOps.

They were stuck. Their software was late. It didn’t address their customers’ real needs, and it didn’t address their own company’s needs. Their application stacks were growing increasingly complex, the bottlenecks in their SDLC were increasing, and their ability to innovate, accelerate, and stay ahead of their competition was slipping further away from them.

What I find especially interesting in re-watching many of these conference sessions is that a lot of them were also leveraging the other buzzwords mentioned above. They’d been practicing agile for years, and they’d started to look to the cloud to lighten their hardware dependence and provisioning headaches.

What they all needed, and thankfully found, was a disruptor. And no matter what definition you attribute to DevOps (and good grief, there are so many) there’s no denying its disruptive nature.

It disrupts the culture of your dev, QA, and ops teams. It requires constant communication, more deploys, earlier testing, shifting left, an embracing of “failing fast and failing often” and what’s often described as “relentless automation.” Some would even argue that the hardest part of DevOps is getting executive buy-in and sponsorship so that you can even begin your initiative. Though to be fair, your ground level troops should be just as on-board as your commanding officers in order to have the best chance for success.

Joachim Bauernberger at InfoQ recently looked into whether DevOps can work in Telecoms, and it turns out, like in any industry looking to resolve the constraints and bottlenecks listed above, it most certainly can—however, it requires support and commitment from “all links in the delivery chain.”

And this is the only reason you should be asking, “Can DevOps work here?” Do you have the right people, and are you the right person to increase the likelihood of this disruptive shift within your company? Stop wondering if (or worse, doubting) your industry will support DevOps, because each year and each conference that passes proves that it can.

And soon enough, if the people at your organization are exposed as the real constraints and bottlenecks, they’ll be removed. Don’t be one of them. Be disruptive.

About the Author/Noel Wurst

Noel Wurst, Managing Editor at Skytap, Noel is responsible for setting and executing Skytap’s content marketing strategy. That includes overseeing the creation and distribution of high-quality, targeted content, and managing the team and external resources associated with those efforts. By leveraging deep online marketing, project management, and editorial experience, Noel plays a crucial role in helping generate and nurture leads with compelling content. You can follow Noel and Skytap at @NoelWurst and LinkedIn and @Skytap, Skytap’s LinkedIn

February 12, 2015
Variable Speed DevOps

I was lucky enough to be the host of a Google Hangout last week with some very smart folks who will be speaking at IBM InterConnect 2015. One of the topics that we discussed was Two-speed IT or as Carmen DeArdo of Nationwide called it, variable speed DevOps.

What is meant by this term is the realization of the fact that different parts of an organization, even different parts of an IT team move at different speeds. Rather than forcing everyone to move at one speed, how can you use DevOps to allow these different size gears to mesh into one smooth functioning machine?

Constraints and bottlenecks pop up when one part of the process is allowed to proceed untethered only to either be constrained from a lack of resources or data from a pre-requisite or to be wasted as the output then has to sit and wait around for another part of the process to proceed. It is almost right out of The Goal by Goldratt.

How do we achieve a “just in time” method of IT? This is a problem that agile has tried to tackle in the development world, but what about the larger canvas of IT? That is variable speed DevOps.

Perhaps nowhere is this variable speed issue as visible then when talking about legacy mainframe systems working with modern cloud architectures. Yes IBM’s z/system team has numerous examples of exactly this taking place. But don’t be fooled into thinking that this is about making old stuff work with new stuff, for it is not.

I have discussed this with Sanjeev Sharma of IBM as well regarding a follow on panel on this subject that both he, Carmen and Rosalind Radcliffe will be participating in this month at the InterConnect 2015 event in Las Vegas (Mobile to Mainframe Experts: DevOps Best Practices for Systems of Engagement and Systems of Record, Session DEM-4399, Monday 2PM).

The real key to a successful DevOps implementation is recognizing the variable speeds of the different parts of the organization operate at. Yes automate, yes do more faster, yes, yes, yes. But at the end of the day some processes move like Cheetahs, others like Sloths. A Sloth is never going to be a Cheetah. The trick is making sure the Cheetah is not wasted behind the Sloth.

To be successful in DevOps, you cannot focus on one process or part of the story without a keen understanding of the big picture, the whole story. That is part of what I like about DevOps. It is a big picture, big goal game. While it is easy for any one of you to focus in on one piece or gear of the machine, it is the machine that counts. Machines have big gears and little gears, slow gears and fast gears. Getting them to function like clockwork is the key.

I think too many people try to get all of these gears to function at the same speed. In most cases this is a strategy destined to fail. Understanding the variable speeds of the people and parts of the process is necessary to cook the perfect dish.

So next time someone tells you that DevOps is about getting things done faster, remind them that we all can’t be the Cheetah. Sometimes speed kills. In real life we all move at variable speeds. So do different parts of the IT stack. Orchestrating them to be most efficient is really where DevOps wins.

February 6, 2015
DevOps and Organizational Debt

I recently published a blog on the Perficient Digital Transformation blog entitled “Is Cloud & DevOps Factored Into Your Digital Transformation?” In that blog I discuss the concept of organizational debt and its impact on your DevOps adoption. Here’s what I had to say in that blog:

If technical debt addresses too little investment in upgrading, poor vendor selection, bad platform selection, etc. then, many businesses have also built up organizational debt. Organizational debt could be over regulation, too many low-value policies, to much middle management attempting to hold onto their area of control, and too much governance. If these are not revisited often to ensure that they are continually adding value for the cost of reducing throughput, then that organizational debt will continue to increase backlog.

Interestingly, there’s a lot written about technical debt, but very little has been written about organizational debt, which can have a significantly deeper impact on your ability to continuously deliver than your technical debt.

In working with clients on their DevOps initiatives, we map the current value stream for a given solution area. What becomes apparent from these value stream maps is that controls and policies have been derived within organizational silos with no attention given to how that will impact IT’s overall ability to be agile and deliver more quickly for the business. Moreover, the owners of these controls and policies seem reticent to remove these bottlenecks even after being shown how it impacts business agility.

Here’s an anecdote from one client when asked why they aren’t using snapshots to speed preparation for QA testing. The suggestion was made in response to the client stating it can take upwards of a week to prepare the QA environment for a single round of testing. Their current approach serializes all testing and believes they need to perform a complete regression test due to the resource contention issues. Hence, they cannot test small changes easily and they cannot test often due to these limitations.

“The main issue is that we don’t control the application or database environment, so we need to prepare a QA release and then have those groups perform their duties in preparation of the QA environment for testing.”

Upon hearing this my mind summoned up past experiences working conferences at major venues, such as the Jacob Javitz Center in NY, where you can’t plug in an extension cord or run an Ethernet cable yourself, but you need to use the Javitz Center’s electrician because of union rules.

Is this what IT has become a set of unofficial unions that hold the business in a death grip? This is clearly a case of organizational debt that needs to be addressed.

Of note, I’m not admonishing the various groups for instituting their controls and policies. I’m sure they have emerged due out of need due to past grievances, failures or as part of IT management practices. Regardless, they are now acting as a bottleneck in implementing proven practices for delivering fewer features more often, which have been proven to increase trust and facilitate better prioritization of work.

The result of this organizational debt is lead times have grown to four months or longer to release functionality that is directly related to key business initiatives by the business for increasing revenue.

In his book, “Kanban,” author David Anderson discusses his recipe for addressing bottlenecks and constraints with existing teams. One of his steps is ‘attack sources of variability to improve predictability’. However, this is his last step because, as he notes, “some types of variability require behavioral changes in order to reduce them. Asking people to change behavior is difficult!” And, so it goes, many businesses never address this key constraint because the hurdles of organizational debt have become so burdensome to overcome. It requires a high degree of trust and political capital along with executive sponsorship. As shown by the anecdote earlier, even if the constraint is directly impeding other business priorities it is not easy to overcome.

I believe DevOps practices can help organizations significantly reduce their technical debt. I have seen and read about organizations that have been able to rapidly reduce their lead times while still refactoring and removing backlog. However, the goal of DevOps is to improve the entire flow, not just development, and a good majority of the release aspects of applications will require organizational debt to be addressed to enable improvement across the entire continuous delivery pipeline.

February 6, 2015
The Last Frontier in the Software-Defined Data Center

Three years ago, technology pioneer and venture capitalist Marc Andreesen famously proclaimed, “Software is eating the world,” and, more recently, industry research firm Gartner – in its top predictions for 2014 – declared, “The business is IT.” Combined, what do these statements mean for today’s enterprises? Simply put, with the continued migration toward a software-based world, a modernized IT infrastructure means the difference between business success and failure.

Over the past 10 years, technology companies have seized this opportunity, delivering massive innovation in IT to help organizations transform business models, processes and infrastructure to succeed in an increasingly online world. Key among them has been the use of software to break the 1:1 relationships between application architecture and physical hardware infrastructure, enabling major technology movements such as cloud, SaaS, and big data.

As each part of the infrastructure gains this abstraction process, commoditized, templatized services emerge, improving availability and performance, simplifying operations, and driving further innovation.

The Last Frontier Ahead
The data center is undergoing a revolution in migrating to a software-defined model. As a result, the move to the abstraction layer approach – the software-defined approach – is underway for all five fundamental infrastructure components of the data center:

Compute
Storage
Networking
Security
Data access

Technology vendors have evolved most of these components to operate in this software-defined model. But data access has lagged – it’s the last frontier in successfully building the software-defined data center. Until the benefits of the abstraction approach extend to the data access layer, data center operations will remain compromised.

The Common Thread: Breaking the 1:1 Chain
The software-defined data center breaks the 1:1 relationship that has existed previously for IT functions. This approach holds true whether that 1:1 relationship tied hardware switches that provide the fabric to the vLAN domain or applications to database servers. By allowing the software to abstract the data from the physical device, and treat the device as just a resource rather than a physically bound container for the data itself, software creates high-performing services that can be infinitely scalable and infinitely available.

What innovation in each part of the software-defined data center enabled, and how can we achieve that benefit in the last frontier of data access? Here’s a look.

Making Computing Capacity Elastic – Virtual Compute
Compute resources were among the first to undergo the migration to virtualization. VM has enabled the fast and easy provisioning of compute resources. With the ability to easily add or remove virtual machines (VMs) based on load requirements, access to computing resources has become magnitudes more simplified. Running many VMs simultaneously, organizations benefit through improved system availability, performance and resource utilization.

Slashing Storage Costs, Automating Provisioning – Software-defined Storage (SDS)
Software-defined storage (SDS) decouples the programming requirements for storage-related tasks from the physical storage hardware. By managing and automating storage through software, SDS has allowed organizations to simplify storage functions, including configuration and provisioning, as administrators no longer have to consider hardware attributes. The end result is that storage resources can be more efficiently and easily allocated as needed within an enterprise.

Simplifying Network Operations – Software-defined Networking (SDN)
As VMs rapidly multiplied, they became deployed in large and disparate environments. With networks scaling across thousands of typical machines and hundreds of thousands of VMs, organizations needed a solution that wasn’t tied to the switch.

The OpenFlow protocol delivered an important stepping-stone in the development of software-defined networking (SDN). Separating the programming of routers and switches from the underlying hardware, OpenFlow lets switches and controllers talk with each other. Other technology vendors have also contributed with solutions to separate the control plane from the physical network, enabling VMs to easily move from one physical server to another without limitations.

With the ability to program the network free of hardware, SDN has simplified network operations and vastly improved the provisioning of large, complex networks.

Simplifying Data Asset Protection – Software-defined Security
Software-defined security abstracts security resources away from dedicated appliances, such as physical firewalls or wire sniffers. This model lets organizations establish security parameters without changing the physical hardware, so administrators can easily extend security policies across a cluster or data center as needed. Software-defined security can make it easier for administrators to spot susceptible behaviors, for example. Unlike traditional security, which depends heavily on manual detection and administration, software-defined security leverages automated processes to alert organizations to possible security breaches.

Delivering Continuous Application Availability – Software-defined Data Access
Companies are just now starting to address the data tier, the key communications layer between the database elements and applications through software-defined data access. Some larger companies, such as Google and Facebook, have built their own proprietary data abstraction layer, while other companies are bringing these capabilities to the mass market.

One growing approach is to leverage SQL load balancing software to create a transparent, intermediary layer between the applications and databases. With database load balancing software, organizations can separate the application from the database, removing application complexity by avoiding having to program database structure into the applications. This approach provides for flexibility in how the database components are deployed and scaled, enabling automatic failover and other load balancing functionality with no application engineering.

Without changing a single thing at the database or application layer, software-defined data access provides dramatic benefits. It enables zero downtime for applications, boosts application performance, and reduces both hardware and license costs.

Conquering the Last Frontier

A software-defined data center ushers in new growth opportunities. It eliminates the technical obstacles that can consume operational energies and streamlines IT to improve customer interactions, driving new revenue streams and innovative social opportunities.

With so much business value wrapped up in easily and always accessing data, getting to the software-defined data access layer is crucial to business agility and success.

About the Author/Michelle McLean

As Vice President of Marketing at ScaleArc (link), Michelle is responsible for overseeing all of the company’s marketing strategy and initiatives. She has more than 20 years of networking and market positioning experience. Prior to ScaleArc, she held director of product marketing positions at Silver Spring Networks, ConSentry Networks, Peribit Networks, and Trapeze Networks, and prior to that, she was director of strategic marketing at Pluris. She previously served as program director at the research firm META Group, providing technology and strategy direction to global 2000 enterprise clients. Before that, she tracked technical developments, networking trends, and vendor strategies as a journalist for two leading networking publications, LAN Times and LAN Magazine. Michelle earned her BA in English from the University of California at Berkeley. ScaleArc social links: Facebook, Linkedin McLean social links: Twitter

February 5, 2015
The Role of a Traditional NOC in the new DevOps World
This new question in the devopsy.kinsta.cloud Enterprise DevOps Q&A series comes from a large business, though not what many would recognize as a traditional or legacy enterprise. This organization, like many high-growth businesses, is in a relatively new market segment that only exists due to the rise of cloud computing and digital media. In their business, the network (especially access to and from their public cloud service) is critical to the value they provide their customers.

So it is no surprise then that, as they contemplate their transformation to an agile environment built on DevOps principles, they are asking me about the role of the Network Operations Center (NOC).
1. What Role Does a NOC Play in the DevOps World?
Despite coming from a ‘new tech’ business, this is another of the questions that is typical in a large enterprise. While smaller startups and even many web-scale businesses may have an Application Support team, not many run their own NOC, and even fewer separate out these roles from the rest of the operations team.

First, some things do not change in a DevOps world. Fundamentally, the NOC is still tasked with maintaining high quality production networks and systems, detecting potential problems as soon as possible, triaging those problems to find root cause, and remediating problems as fast as they can.

However, the NOC team will gain additional activities and support a broader interlock beyond just the production environment, especially as networking becomes more embedded in application delivery through capabilities like software-defined networks and cloud computing. They will work more closely with teams like app dev, SQA, pre-prod and release; they will provide input to design and development; and they will contribute earlier in the software cycle. This is the logical outcome of a DevOps approach that emphasizes the need to work more collaboratively across the entire end-to-end SDLC.

So you should consider bringing these network specialists into:
- development planning, to help architects and engineers to build applications with operable networking by design, including software-defined networks that help make a truly idempotent application
- testing and QA, so they can provide engineers and SQA teams with tests and analysis that accurately reflect the reality of the prod network, rather than leaving network testing to after production deployment
- dev/test support (where they may already be acting), to ensure that the environments that dev/test rely on to create new capabilities are just as stable and reliable as the systems they will eventually be deployed to
- release and deployment, to ensure they have input into determining and defining ‘known good values’ for production monitoring and for problem diagnosis, triage, and escalation – before release, not just after
- expanding interoperability and self-service, by exposing ‘known-good’ deployment, triage, and (re-)configuration capabilities as automated software-defined processes (or APIs) that dev/test can use without opening a new service ticket
- a broader role in ensuring the end-to-end quality of the network, alongside traditional operations/infrastructure support as part of a new role commonly referred to as a ‘Site Reliability Engineer’ (SRE)
Because DevOps is about improving collaboration, remember too that you should also bring dev, test, QA, etc. into the NOC, and even rotate them onto support, so they have an understanding of the core operations of a production network, how their code acts in a real environment, and how their work affects other teams. This will help to build the empathy between teams that is a core tenet of DevOps.

You may be concerned that asking your NOC to work more upstream is adding a load of work to those teams that they didn’t have before. That may be true, but a significant outcome of Enterprise DevOps in practice is to reduce the ‘busy work’ of such teams, free them up to do more ‘smart work’, and to do it earlier in the software lifecycle.

For example, recent research has shown that DevOps is correlated with improving the quality of applications (especially before they hit production), reducing the amount of time and effort the NOC team spends on problem detection, triage, and remediation. In fact, in this research, ‘reduction in time spent fixing and maintaining applications’ is one of the strongest outcomes, with an average 21% reduction in this metric.

Similarly, standardizing on known-good processes, APIs, configurations etc., and enabling other teams to employ self-service and software-defined networking capabilities will allow the NOC team to spend less time on routine responses to things like servicing provisioning and configuration tickets.

I cannot guarantee they will not be a zero-sum game, especially in the short-term, but over time a DevOps approach should actually spare workload from the NOC team, freeing them up to ‘shift left,’ moving their talents earlier in the SDLC, as they communicate, collaborate, and integrate more closely with upstream teams like dev, test and QA.
February 4, 2015
Measure Twice. Automate Once.

“Shipping” code may start with the completion of development but it doesn’t end until after the operational delivery process ends. Just like any fulfillment process, shipping is actually just the kick off of an often arduous journey that ultimately delivers the app to the end user.

Ship It

Back in the old days (when they still let me write code) one of our favorite phrases was “ship it” to indicate completeness of a task. That’s because the ultimate goal was, of course, to “ship” the code off to the users so they could start doing whatever it was they were going to do with it*.

But the reality was, of course, even when we were really ready to “ship it” to users that it didn’t actually get to users for varying amounts of time. Like shipping anything today, it depends on the shipment method. In terms of today’s enterprise IT, “shipping method” is really a metaphor for the operational approach to deploying all the infrastructure – the “middle boxes” – with the services required to actually deliver a secure, fast and available application to its ultimate users.

We know from various studies that this takes time. From days to weeks to even months. And we know we don’t like that and we want to improve it. Which is where DevOps comes in, or NetOps or SDN or SDDC or SDO or whatever you want to call it.

Speeding up the deployment process is certainly one of the goals of operationalizing the network, but without a baseline how do we know if we’ve gotten faster in the first place?

That’s a piece of the DevOps puzzle we’ve been ignoring – the need to measure the deployment process. You might notice that “measure” is bolded and in red, that a visual clue meant to reinforce the importance of the measurement aspect of DevOps. If you aren’t measuring how long it takes to deliver that app from dev to its destination – production use by an end-user – then you have no idea which pieces of the process are holding you back. You can’t effectively determine where automation might be applied to improve the time it takes to get to market. You don’t know exactly where the obstacles are that need to be addressed in the first place.

While we can make some assumptions based on long experience and research and point to handoffs across the IT silos, the reality is that there probably isn’t just one handoff between ops and the network and there may be multiple intra-IT group handoffs. For example, the network ops that handles provisioning network services might not be the same team that handles load balancing and performance services. But they’re both within the “network” silo. There may be multiple handoffs back and forth as various services also need networking services on top of the ones required by the app itself.

Image Credit: Gaping Void

Without some kind of measurement of how long each of the various steps is taking it’s impossible to determine exactly where the bottlenecks in the app deployment process are in your organization. I can not only determine where that Minecraft Lego set I ordered for my son’s birthday is right now, I can see how long it’s taken to get from each point in the process. I know it goes from point A to point B in X days, and from point B to point C in Y days, and I know if there’s a winter storm that it’s going to cause a slowdown in that second step (which is called a movement in the logistics vernacular, in case you were wondering. I knew you were, you’re welcome!)

Measure Twice. Automate Once.

The hidden value in setting out to measure the deployment process is in pulling together an understanding of the process itself. It’s in finding redundancies and discovering sub-processes that are running in serial but could be running in parallel to improve them. It’s not necessarily about automating the whole thing up front, it’s about understanding what the process is and making it more efficient – whether through elimination, reorganization or automation (or all three).

You don’t simply want to be automating things willy nilly without any idea whether it’s going to do any good. As my dad taught me about working with wood, you measure twice and cut once. With DevOps you want to measure twice (at least) and automate once. Otherwise you may be wasting cycles on one-off tasks or tasks so specific to one application that automation has no real value because it’s not the same across applications.

The key is to start with defining what you’re going to measure, do the measurement and then analyze it to determine what you need to improve.

Because automating just for automation’s sake isn’t any more fruitful in the end than adopting technology for technology’s sake.

* I say “whatever” because over the course of my career I’ve worked on tax software, GIS software, communications and transportation and logistics systems. There’s a lot of variance there in what end users might be doing with “the app”.

February 2, 2015
Google Hangout: InterConnect 2015 preview from IBM & Nationwide on #DevOps
Join us for a Google Hangout* with a distinguished panel of guests as we discuss: Adopting DevOps for Two-Speed IT.

With IBM InterCnnect 2015 just a month away, three featured presenters will give us an Interconnect preview

*Google Hangouts are FREE. To participate in a Google Hangouts LIVE, a Gmail or Google Profile is required.

Date: Wednesday, January 28th, 2015

Time: 3pm eastern standard time
```
REGISTRATION LINK:(NO REGISTRATION REQUIRED*) 
```
About Our Guests:

Rosalind Radcliffe, our guest from @IBM

Distinguished Engineer – Chief Architect for CLM and DevOps, Member of the Academy of Technology

Carmen DeArdo, our guest from @Nationwide

Director – Tools and Technology Leader

Sanjeev Sharma, our guest from @IBM

IBM Worldwide Lead – DevOps Technical Sales (Executive IT Specialist),

Author of DevOps for Dummies
January 26, 2015
The Trouble With Enterprise DevOps

The literary world and beyond has been up in arms as of late regarding the use of the word ‘literally’. It seems that many are using the word literally when they mean figuratively. For example, if you say, “I literally split my pants laughing,” it would be plausible for one to look at your pants looking for a tear. Words are funny that way. When used to emphasize the action or outcome, some will take literary privilege. Enterprise is also one of those words that are used to convey emphasis often with disregard to the implied meaning.

When it comes to DevOps there has been a lot of debate about the modifier ‘enterprise’. Modifiers are supposed to add specificity to the noun it is modifying. Agreeably, enterprise is a horrible modifier for DevOps when used without additional modifiers representing scale. I have been, and continue to be, a big advocate of the concept of Enterprise DevOps. That said, I too have been remiss in my use of the word enterprise to further modify with the term ‘large-scale.’

I don’t remember exactly what triggered me to seek out if the word enterprise implied large, but when I did I was surprised to find that there is no implied meaning that enterprise means large and, in fact, can be modified with small-, medium- and large-scale to represent the appropriate size. As further evidence to the ambiguity of the use of the word enterprise, I worked for a company that divided their sales organization into enterprise and commercial accounts. Clearly, this organization believes that enterprise connotes large-scale since this group was responsible for selling to a specific class of customer that spent a significant amount of money with them. In this case, scale was identified by how much money the business spent with that vendor versus annual revenues versus number of employees. So, clearly, not only does the word enterprise not imply scale, but it also has different metrics that will determine the scale of an enterprise.

Merriam-Webster defines Enterprise as follows:

en·ter·prise

noun
\ˈen-tə(r)-ˌprīz\

: a project or activity that involves many people and that is often difficult

: a business organization

: the ability or desire to do dangerous or difficult things or to solve problems in new ways

Obviously, none of these meanings relate to size or scale, but two of the meanings do indicate difficulty. Let’s put that in the parking lot for later.

Why is this important? With regard to IT, I believe there is a tendency to throw around the word enterprise with no modifiers when we mean large-scale. For example, “Enterprise Software” is a term that is often used to describe very expensive and complex software applications that are designed to operate at the scale of the entire business versus the individual. Typically, the software described by enterprise software is found in mid- to large-scale enterprises where they can afford to acquire, configure and operate this type of software. Likewise, “Enterprise Storage” is a way to describe compute data storage environments for large organizations.

Likewise, “Enterprise DevOps” is a term that that is often used to represent adoption of DevOps practices in large, complex organizations with many different stakeholders and sub-organizations making it much more difficult to coordinate efforts around efforts like continuous delivery and continuous improvement.

Interestingly, and I believe accidentally, implementing enterprise software and adopting enterprise DevOps are difficult projects. So, based on our Webster’s definition, enterprise works as a modifier for these nouns. But there’s still the issue that for small-, mid- and large-scale enterprises the challenges and hurdles associated with DevOps adoption vary enough that the constraints and bottlenecks require different approaches to remove. This is especially true of large-scale global enterprises where development exists in four continents, operations data centers exist in multiple countries on four continents and the business operates in six continents with a comprised IT staff of 120,000 people.

I for one will be very careful in the future to define what I mean when I’m using the word enterprise in the future. If I’m talking about large-scale enterprises, which I usually am, I will make sure that early on in my writings and presentations I specify that when you see enterprise going forward I’m talking about large-scale enterprises.

January 23, 2015
DevOps needs infrastructure multi-tenancy

DevOps will, under whatever moniker you want to give it – NetOps, DevOps for Networks, operationalization – continue to make inroads into the network infrastructure because ultimately it’s part of the application deployment lifecycle. While DevOps folks call it “application delivery” the reality is that an application is not ready to deliver to a user (internal or external) until all its requisite services have been provisioned and configured.

Yes, infrastructure and network devices are increasingly API-enabled and supportive of a variety of tools and frameworks most often associated with DevOps – Puppet, Chef, OpenStack, VMware – and those more commonly associated with just the network – Arista, Cisco ACI, and OpenDaylight.

But it takes more than APIs. APIs do not inherently bestow upon devices the ability to support multi-tenancy. That is, the isolation of services unique to a team (or application) required to effectively deploy application-supporting infrastructure services within what is traditionally a shared environment.

According to ThoughtWorks Technology Radar (jan 2015) partitioning infrastructure along team bounds is becoming an imperative to removing roadblocks in the application deployment lifecycle:

Many of our customers have made DevOps a reality in their organization with delivery teams that build, deploy, and support their own applications and services. Unfortunately, a regular roadblock on that journey is allowing teams to have superuser privileges in production environments. In most organizations, the production environment is shared, and therefore risky to provide access widely. It is effective when we can partition infrastructure along team bounds,so that those teams can have safe isolated access to do their work, without risking impact to other systems. Where cloud environments are used, this is much easier to implement, aligning account structures to team boundaries. [emphasis mine]

This has always been an issue where shared services are relied upon. A change to support one application (or team) can be disruptive or cause issues for another application or team. Thus, most infrastructure and network teams staunchly refuse to let anyone in the application or operations side of the house play with their toys.

They are the network ninjas; guarding their configurations with their very lives.

It’s this kind of conflict that needs resolution if we – as a unified IT organization – reach goals of improving the time to market and frequency of deployment. It doesn’t matter if the app team can deliver an app in 3 weeks if it takes 3 months to deliver it to the customer because of overloaded network and infrastructure staff.

There are two good solutions to this problem*.

1. The governance over more application-affine infrastructure services – load balancing, caching, proxies, etc… – is necessarily moving closer to the application and under the control of app and operations teams. This is supported by concepts like Network Service Virtualization, proposed in Lippis Report 217 as well as the increasing adoption of open source proxy-based (virtual and software) services. This model assumes continued shared network and infrastructure services as appropriate while migrating per-application services to the more agile, operational architecture.

2. The introduction of multi-tenancy in infrastructure architecture provides a similar approach while still maintaining shared resources (hardware). A single, high capacity system is able to host multiple, virtual instances of its services, each dedicated to either teams or applications, as is the organization’s need. This approach spreads the cost of the hardware resources across the entire organization while allowing a more per-application approach to provisioning, configuration and ultimately, cost. Isolation addresses the concerns of shared resources and configurations that can cause consternation among network and infrastructure teams because the underlying shared system is managed by them while role-based control over application services is offered to teams or designated responsible individuals.

Both approaches are valid and in many cases both architectural solutions will be used to resolve the issues caused by traditionally shared infrastructure.

The key takeaway here is that there are answers to the need for partitioned infrastructure that are supportive of DevOps-driven workflows and processes and can certainly provide a path toward ensuring a more seamless, time-sensitive end-to-end application deployment process.

* Oh, I’m sure there’s more, but right now these appear to be the best two options.

January 22, 2015