Tag: enterprise

Survey says network agility is still immature

At least that’s the word from SDx Central’s Cloud Management, Orchestration and DevOps 2015 Report in which 76% of respondents said the network was the most immature component in the cloud management stack.

Now to be fair, the network is one of the last components (I’m looking at you, security) to get on board the “automate or die” train, which has already left the station. That’s partly due to pressures from increased DevOps adoption in general and more specifically thanks to greater enterprise penetration which is continuing to up the amperage in the data center with respect to velocity and ease of deployment of all the network bits, bytes, and pipes needed to support each and every application.

It’s also partly due to the complexity of the network environment and the blast radius of potential errors or misconfigurations as well as the broad set of differences in the object models (and therefore APIs) used to configure, manage, and automate provisioning of network services. The more features and capabilities, the more differences between them.

OpenStack manages this difference by forcing standardization; that is, it defines the interface for LBaaS and if you want to be a part of the stack, you have to conform. Cisco ACI manages the difference by taking a declarative policy-based approach that let’s plug-ins define their own interfaces and expose them to ACI in a standardized way. VMware NSX and related technologies take yet another approach, leveraging its ecosystems’ APIs to integrate and orchestrate the network.

All are relatively new to the game (OpenStack may be the exception but even its network implementation has lagged far behind other components of its system) and thus the perception that the network is the most immature in the cloud management stack would seem to be reflective of reality.

But at least it’s heading in the right direction.

The network was going to have to be included at some point in this grand journey to transform the data center through automation and orchestration. At some point Agile and DevOps methodologies being embraced by dev and ops were going to start paying off in the form of more applications with more frequent deploys. Eventually that was going to overwhelm the network (and its staff) and give them cause to start figuring out how they were going to transform the network from the static and very fixed methods of scale and management to a more modern dynamic and fluid method.

Eventually is now. Whether it’s DevOps expanding into the network or SDN or cloud management stacks, the network has to mature in terms of its orchestration and automation capabilities if organizations are going to continue on their journey toward the agile network.

We’re still trying to move out of the first phase in that transformation; toward a culture inclusive of measuring and standardization with templates (or configurable policies, if you prefer). We’ve got a long way to go toward a sustainable, agile network and just as it’s taken years for Agile to truly envelope most projects within the enterprise, it’s going to take years for the network to get from where we are today to where we suddenly want to be yesterday but accept that we won’t get there till tomorrow.

October 21, 2015
Five often expressed obstacles to enterprise IT innovation

Business leaders know, for their business to succeed in the years ahead, that their IT and developer teams need to innovate. But surveys show that too often the IT teams may not be able to deliver the innovative value that the business seeks. Have a look out our recent Q&A, The IT mandate: Be a catalyst of business innovation for some more insight on the IT and business innovation gap. The fact is that technology and developer teams can’t do it on their own, they need to operate in an environment that enables innovation. In speaking with CIOs, developers, and others close to the process of creating value with IT a number of obstacles continuously surface. And these obstacles are holding their development efforts back.

These obstacles include the tight collaboration with development and operations teams, the failure to actually embrace Shadow IT for what it actually is, staying stuck in single mode IT, and oversights in how they are developing mobile applications.

Here they are:

Obstacle Number 1: They don’t effectively unleash the power of their developers. DevOps teams are known to grind through technical debt and when they manage to hit on all cylinders they also have very short application backlogs. One way to keep such app backlogs short is to focus on the Minimum Viable Product (MVP). Because the MVP is an application that is, initially, focused only on what it takes to get the job done, and no less or more, developers don’t get bogged down overdeveloping apps that won’t get used. To be clear, MVP doesn’t mean shoddy app, rather it means the minimum app to be productive at the task. With this Spartan approach, development teams can invest no more than is needed into an app, release it into production and see how enthusiastically (or not) it is embraced by the user base. This enables development teams to push apps forward with as little risk and investment as necessary. And then either dig deeper into that app, or move on to another.

What does that have to do with innovation? Plenty if you consider the amount of time saved from over-developing applications that wont be used and investing that time and effort elsewhere, perhaps apps that will provide value to the organization.

Obstacle Number 2: They don’t embrace shadow IT. Rather than embrace Shadow IT, many organizations shun the notion of users choosing the tools and technology they need to get their job done. But end users and employees are turning to cloud services and bringing their own devices because these are the devices and services that they know they enjoy to use and they can use to effectively do their jobs. That makes for more productive and happier workers. Certainly not just any old rogue service and device can be used, but the enterprise can watch and vet the services that are safe or find ways to make them safe. A good place to look for this is the Cloud Security Alliance’s Security Guidance for Critical Areas of Mobile Computing.

Obstacle Number 3: They don’t bimodal. By not embracing two-speed or bimodal IT, enterprises have a harder time allotting resources to strategic development and IT efforts. In bimodal IT there are two modes, one mode is focused on quickly delivering the applications the enterprise most needs to compete and be effective while the other mode is focused on the infrastructure services and API development and management.

A good overview on bimodal IT can be found here in our post Bimodal IT and Remodeling Traditional IT for Greater Agility.

Obstacle Number 4: There exists an average to poor business to technology interface in the organization. In too many organizations there is a lot of collaboration within the technology groups, such as operations and developers, but not enough interfacing with the business teams. For better outcomes, there needs to be tighter collaboration between DevOps and the business through continuous feedback on current and future objectives. This way DevOps teams know what the business priorities are and can develop to them more effectively.

Obstacle number 5: They aren’t focused on the context within mobile moments. The world of IT has gone mobile and will someday go wearable. This is more than just a mere form factor trade; it is creating a profound change in the way we interact with our apps. When we sit down to work on a laptop, we are usually settling in for a nice stretch of work time. However, when it comes to phones and wearables, we are usually working in short instances. We are sending a text, answering one email, or capturing one or two new data points about a client for entry into the CRM app. Developers need to focus on providing users apps that work in the same way too. That is they need to develop apps that provide users the data they need based on short device interactions and hopefully context of what they’re doing. If the user were at a clients’ office, for instance, and the rep launces the CRM app, it’d be ideal if that customer profile launched by default with the app. If the user is involved in a certain workflow, other items that are commonly taken care of at the same time would also ideally be suggested.

While these obstacles may sound simple, they can be quite transformative to the way your business-technology teams will develop, collaborate with the business, and hopefully provide more innovative services.

October 15, 2015
An Open Letter to DevOps “enthusiast”

Dear DevOps “enthusiast”,

You are confused. But I am confused as well. For approximately two years now this DevOps thing has been both exciting and frustrating. And in ways that are totally different for each of us. At the intersection of all the disparate confusion is the “perfect storm”. What that means for now is that we are all hunkered down in the basement, reading books on “DevOps” to pass the time, and better understand the thing that is attempting to destroy us.

(more…)

July 9, 2015
Imperative or Declarative? The question of DevOps in the network.

A significant amount of bandwidth is spent discussing automation and praising the value of APIs within the broader DevOps umbrella. These concepts are particularly important in the context of CI/CD and the evolution of application architectures toward decoupled, service-oriented systems. Automation is used for build and test, to drive the workflow that moves an app from disconnected sets of services to a working release candidate ready to breach the wall of production and be put into the hands of adoring fans impatient consumers.

But in between the consumers and your app stands the production pipeline; a process-filled wasteland through which an app must travel so that the appropriate services required to deliver it can be provisioned, configured and put into place. It is this pipeline that will meet DevOps head on next, whether we call it DevOps or SDN or SDDC. It stands in the way of an increasingly critical “time to market” metric by which all of IT is being measured. Research shows that the production pipeline is a major pain point for IT; taking weeks or even months to traverse. The network is still in the way, and that must be addressed if C-level concerns about meeting increasing pressure to get apps to market are to be met.

Applying DevOps principles to the infrastructure and network makes sense as it’s as much about optimizing processes (workflow) as it is writing scripts and it is the production pipeline process that would benefit. Automation and orchestration, of course, will help drive a more consistent, efficient, and repeatable production pipeline traversal. Identifying where inefficiencies (latency) lies in the process and eliminating it by establishing more efficient (and perhaps automated) handoffs using automation and orchestration systems can go a long way toward shortening the amount of time between hand off to production and delivery to consumers.

APIs are our “go to” technology today for driving automation, orchestration, and integration with the frameworks and systems we rely on to scale operations. Most every infrastructure and network device today has an API and can be controlled via frameworks like Puppet and Chef as well as directly using .

But most are imperative methods of control. It requires processes to be strictly defined through a sequence of API calls. Unlike simpler resources like virtual machines, one does not simply need “start, stop, or pause.” Configuration that was once (and still is, let’s be honest) accomplished via CLI (command line interfaces) using SSH is simply migrating to an API using HTTP instead.

But otherwise, not much has changed. The same commands used on the CLI are now sent via an API. We’re still in an imperative mode and if we need to tweak or adjust the configuration, we need to tweak and adjust the script that drives the configuration.

Compare that now with a different approach: a declarative approach. Using templates to describe what needs to be configured or accomplished (data) as opposed to how it needs to be accomplished (commands) makes a significant difference in not only the process but in the long-term technical debt acquired (or not acquired, as the case may be). APIs change; methods deprecate, versions matter. Treating application services (load balancing, caching, optimization and acceleration, security) as “code” (templates) instead of a series of API calls can dramatically change the velocity with which such services can be provisioned and configured. This has the added effect of reducing the amount of churn in the automation code which should reduce technical debt as well. Such methods are also more portable, enabling easier migration into cloud environments for applications that might be (or will be) moved into or around different environments.

Ultimately, the question is whether or not you want to tie your processes – and the interfaces between workflow steps in that process – to many API calls or a standardized set of API calls that leverage declarative policies (templates) to provision and configure the services that make up the production pipeline.

June 1, 2015
Enterprise DevOps: Standardize for Security

Etsy is a shining example of the success that can be achieved with continuous delivery (CD). By 2014 Etsy had doubled its deployment rate, deploying more than 50 times a day. Today the site deploys API changes in 18 seconds and launches a new website every 150 seconds.

That’s speed, to be sure. Speed that very few enterprises need or want to achieve. But that doesn’t mean that Etsy and CD don’t have valuable lessons for the enterprise, because one of the reasons CD and Etsy succeeds is because of the underlying principle of standardization.

Whether it’s standardization of tools or of processes – or both – it is standardization that eventually drives such speed. The use of the same tools and the same processes over and over hones them until they’re an optimized, lean delivery machine. That standardization has other, benefits, too that are as beneficial – maybe moreso – to the enterprise than the speed it eventually brings.

Standardization of delivery processes in the enterprise, especially when extended to such concerns as security and the network, result in direct and indirect benefits. Directly, standardization allows security and even network infrastructure to ensure that corporate policies are implemented. If all infrastructure – application and network alike – should only be hosting certain services (like HTTP and SSH) and not others, standardization can help enforce that. Template-based (or policy-based, if you prefer) automation along with an “infrastructure as code” approach can assist in ensuring that basic security and network policies are put into place for every application deployed. Customization can (and should) be allowed and encouraged as extensions to the core templates (or policies) deployed during the app deployment process.

Indirectly, standardization on a common toolset and processes will reduce the stress on network and security operations that comes from trying to implement common policies across a conga line of solutions. There’s no getting around the reality that many enterprises employ sometimes twenty or more “middle boxes” that provide everything from security to monitoring to performance and scalability services. Each one must be provisioned and configured during the app deployment process. This is not a trivial problem. 26% of organizations cited “security policies that are too complex and can’t be enforced with the current network security processes and controls” as a significant challenge in a 2014 ESG survey. Establishing a standardized template that encapsulates common (required) policy implementations can go along way toward addressing the challenges network security professionals face when trying to secure applications across a growing portfolio of both applications and network services.

This level of standardization is not easily achieved. There is no easy button, and that’s one of the reasons DevOps can be a desirable approach in the enterprise to addressing these kinds of problems. It isn’t just about achieving phenomenal speed, but rather achieving consistency through the standardization of processes and methods used to deploy security policies and controls that can improve the stability and the security of the entire application and network environment.

DevOps is, in relation to network and security operations, more about CPR for infrastructure services than it is speed. Consistency through standardization. Predictability through automation. Repeatability through templates and by treating infrastructure as code.

DevOps is an approach to operations and inter-operations that can have just as many benefits in the enterprise as it does for web-scale and startups born in the cloud. The same standardization that benefits Etsy by giving it extreme speed can benefit enterprises by endowing it with extreme security.

April 23, 2015

Is DevOps a Competitive Advantage? A Debate

The idea of DevOps being a competitive advantage is not a new one. Service providers, software sellers, and internal IT frequently use this argument to justify all kinds of investments. But is there any real merit to this argument? Is DevOps truly a source of sustainable competitive advantage? Or is it simply just hype and a new way to sell a product / service?

Here are some of the more popular points of view from both sides of the aisle.

“Yes Absolutely”	“No, Go Take A Hike”
Speed is sexyIn the mobile, social, and web world customers expect speed and responsiveness. Bringing an idea to the market faster than your competitors is definitely an advantage and DevOps allows us to do just that.	DevOps is not the only fish in the seaAgreed that speed is sexy. But at this given moment there are multiple technologies / philosophies out there that claim to do the same – i.e. cloud, big data, analytics etc. What makes DevOps so different?
DevOps is hard to replicateWhile purchasing tools is easy, the real benefit lies in getting the culture and process right. Any organization that can get that right will be able to form a sustainable competitive advantage.	Same has been said about other strategiesRemember when outsourcing started. It too was touted as a sustainable competitive advantage. Now it is the de facto strategy for IT erasing any advantages that the early adopters had. DevOps, if it does take off, too will follow the same path.
New IT operating modelDevOps represents a new way for enterprise IT to operate. It breaks down the silo structures and unlocks the value trapped within the existing organization.	Enterprises are not start-upsJust because it worked in a start-up environment does not mean that it can work in a large enterprise.
Listening to customersSpeed is only one equation of DevOps advantage, the other is faster feedback loops so that you can determine that you are going in the right direction. Combine the two and you have a powerful competitive advantage.	Not an IT responsibility / competencyIsn’t that what most organizations already do? I have my entire business / product / marketing org. focused on just that. The last thing I need now is IT to jump in and muck it all up.
The new Software Delivery LifeCycle (SDLC)DevOps represents a new way for enterprise IT to operate. It breaks down the silo structures and unlocks the value trapped within the existing organization.	Just good marketingThis is just repackaging of an old wine into a new bottle…DevOps is just a new thing. Like all other fads this too will pass.

My Take

Allow me to be the quintessential consultant and speak from both sides of my mouth. DevOps is and is not a competitive advantage. Now before you close the browser and move on, let me explain.

DevOps in the short term is most definitely an advantage. Moving an organization from Traditional to a DevOps approach is not an easy task. Any corporation that can successfully achieve this transformation will be the first to reap its benefits – from faster speed to market to being more responsive to their customers. And the benefits are real! Ask Netflix and Etsy, who have been able to out innovate their competitors and gain a sizable lead in the market.

However, in the long term, as more and more firms invest in DevOps the advantage will continue to shrink. DevOps, if you strip all the marketing jargon and go down to the most basic definition, is about reorganizing the internal plumbing of IT (i.e. how IT performs the various tasks within the software delivery lifecycle). And while in the short term plumbing can allow you to do some neat stuff, at the end it’s only internal plumbing and rarely a source of lasting competitive advantage.

To see this is action, look at the automotive, financial, and then our very own Information Technology industries. In all three examples, the company that was able to transform was able to establish an early sizeable lead, reap the rewards, and the rest of the industry forced to catch up eventually.

In automotive industry, early investment in lean and TQM earned Toyota customer loyalty and a significant lead over its traditional competitors. And while the lead lasted for a decade (short time in an industry where product lifecycles are 4 – 5 years), competitors quickly started to invest and catch up. You can now argue that U.S. manufacturers are at par with Toyota in both quality and speed. Where the competitors have still not caught up is in perception.

In the 1970s it was not unusual for a home buyer to wait for up to 90 days to get a credit decision and a loan, as per “industry standards” in those days. Citibank rearranged its internal processes to give buyers a decision in 30 days. It took the industry by storm. Competitors eventually caught up and now 30 days is the norm.

And as for our very own, Information Technology industry…outsourcing was considered to be a competitive advantage for many of the first movers. Like other strategies, it too required a structural change to make it happen. Many of the initial pioneers were able to cut IT costs by over 30% and establish a cost advantage over their competitors. But within a decade competitors copied, banked the savings, and eroded the cost advantages.

In my opinion, DevOps too will follow a similar path. A few leaders will emerge (already happening), they will set the pace of change for the industry, and the rest will have to follow….whether they like it or not.

About the Author

Mustafa Kapadia is the Service Line Leader for IBM’s DevOps practice, a business advisory practice focused on helping large enterprises transform their software & application delivery. He has over 17+ years of experience in the tech. space, both as a service provider and a management consultant.

Prior to joining IBM, Mustafa was a management consultant with Deloitte’s Strategy & Operations practice. He lives in San Francisco Bay Area, is an avid blogger (when time permitting), a speaker, and adviser to start ups.

Connect with Mustafa LinkedIn| Twitter

April 21, 2015

What IT expects from SDN is DevOps

There was much excitement in the air when the announcement that Docker had acquired SocketPlane. Not the least of which was emanating from my corner of the world, where DevOps for network (the whole network) is something you’ve noticed I pretty much live and breathe.

The announcement was couched as an SDN (Software-Defined Networking) play by containerization-leading Docker. Which might be confusing at first because, well, Docker (and to be more general, containers) are generally associated with DevOps, not SDN.

Ah, but therein lies the rub, doesn’t it? While the original precepts of SDN were most definitely not DevOpsy in nature, it (like every other trendy term in tech) has evolved to focus as much (if not more) on what are certainly DevOps-like principles and goals than on technical specifications and standards.

Consider, for example, this recent Avaya sponsored, Dynamic Markets report (gated) on “SDN Expectations.”

The first data point to examine is the ranking of “problems” experienced by respondents. The number one issue (38%) was downtime caused by human error. The second (tying with a network-failover issue at 37%) was complexity of configuring services and applications across the network.

If these aren’t typically problems that are said to be resolved by DevOps – particularly with attention paid to collaboration and automation – I’ll eat my hat. And yes, I often wear hats, it’s a quirk I have.

And just as interesting was this table illustrating not just problem areas, but how many respondents had the problem and how many expected SDN to solve them:

If you ignore the network management tuple, the top two problems are definitely often associated with DevOps as issues that can be resolved with collaboration and automation and some measurement thrown in to boot.

SDN has evolved, at this point, to focusing more on network automation than it does on automated networks. The difference is actually profound as it relates to DevOps, as the former is really about management, about configuration and provisioning and automation of processes designed to extend CD across the entire application deployment lifecycle into the production environment. It’s about codifying and encapsulating configurations in templates and centralizing state of the network in order to engender better agility in both moving forward and falling back when things go awry.

SDN today really is (or should be) a subset of DevOps, focusing on network infrastructure operations rather than app infrastructure operations. But the same principles and goals are very much aligned between DevOps and SDN, between the network and the application infrastructure. That’s a critical convergence starting to appear that’s necessary, as when you dig into IT and its problems delivering apps on time these days, it’s mostly the network that’s still in the way.

The network is identified often as a key IT pain point and usually the pain is related to the provisioning of services and applications across the network. High costs of networking is also cited, and that is largely due to the complexity of the network and the amount of operational sweat and blood that goes into manually provisioning, configuring and testing the various pieces of network infrastructure required to deploy the services business needs to deliver applications.

SDN may continue to be referred to as SDN, if only to emphasis its focus on network infrastructure, but the reality is that it’s DevOps. DevOps for Networks.

Which brings us right back to why a DevOps-related, containerization company like Docker would swallow up an “SDN”-focused start-up like SocketPlane.

Because what IT expects from SDN is really DevOps. And if DevOps is ultimately about culturally change and breaking down silos, the folks who focus on the network can’t be (and shouldn’t be) isolated from the folks managing the application infrastructure.

Docker’s move brings software-defined networks together with software-defined operations under (what should be) an IT encompassing concept: DevOps.

April 2, 2015
The right answer was “process”
The question is “why is there no one-size-fits-all automation and orchestration solution”?

Automation, a component critical for continuous deployment particularly as it extends beyond app operations and into the broader environment to encompass network and security operations, too, continues to be a custom build kind of thing. There’s no single automation tool, framework or pattern that can be generally applied to a significant number of environments. Mi casa es no su casa, after all. That’s because environments don’t actually look like power point slides; your environment is not the simplistically diagrammed firewall –> load balancer –> web server typically found in decks extolling the virtue of one solution over another. They are far more complex, comprising not just multiple tiers of interdependent and interconnected apps and services, but multiple chains of infrastructure services spanning the gamut of security, availability, mobility, performance and access control. The specific flow through and between these services is unique. Indeed, the intimate relationship between network latency and app performance is well known, and thus the navigation through the critical yet complex site of services required to deliver an application could easily be considered a competitive advantage if it’s able to keep performance under the 5 second toleration limit on response time apparently now the standard for consumer satisfaction.

So it’s no surprise that automation and orchestration, responsible for automatically setting up the triggers and routes required for application traffic to correctly traverse the right set of services for optimal performance, would not be a simple, turn-key solution. One size does fit all. It must, like its business process focused counterpart – BPM – be customizable and easily changeable.

BPM? Are you seriously?

I’m dead seriously. BPM – business process management – has been around far longer than automation and orchestration of IT systems and its focus is, as its name implies, on process. Automation is included, but it is the orchestration of business processes that often change in the face of variable business conditions that makes it an invaluable tool for optimizing business and a source of competitive advantage. “The ability to change quickly is essential. Businesses change their key processes 4-7 times per year. The driver for change can be internal or external. New opportunities can arise. New partners or customers need you to support a different way of doing business. Federal or international regulations can require you to change your processes.” (Making the Case for BPM, Jim Rudden)

Consider for a moment this data from a 2012 Bearing Point survey on the goals and impact on culture:

Top BPM goals are increased efficiency (93%), increased transparency (90%) and the realization of standardization potentials (85%). A number of BPM goals have an impact on corporate culture, e.g. increased transparency (rank 2), increased customer orientation (rank 5), disclosure of weaknesses/ bottlenecks (rank 6) and the reduction of mistakes/ error rates (rank 9).

– Bearing Point Business Process Management Survey 2012

If many of these goals and impacts sound familiar to you, they should. Because what DevOps ultimately needs to focus on is exactly the same as the focus of BPM – process. In an application world, these processes require applications and services. IT must be able to support these process changes in the application and service infrastructure.

BPM (Business Process Management) has known this for years, and it is the orchestration of the process – the workflow – from task to task, from approver to doer that generates the efficiency gains achieved with that technology.

Indeed, adopters of DevOps who focus on the “big” picture would do well to consider the similarities to BPM which touts as benefits such things as:

Process quality improvements

Continuous process improvement

Reduces costs

Improves business agility

It would be difficult if presented with that list of benefits with no attribution to determine whether it was referring to DevOps or BPM. That’s because the operationalization of business through process orchestration is not all that different from the operationalization of IT through process orchestration.

And processes are not, nor should they be, turn key and one-size fits all. They should be tailored to address the needs of a specific operational or business goal, i.e. deploy an application or secure an order from a customer, based on the systems in use to achieve those goals. BPM shines where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. DevOps will shine where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. It isn’t about racing through a given task with automation. Speed isn’t efficiency and, if you do the math, a focus on automation without similar focus on process can lead to even greater inefficiency.

And that’s why there is no one size fits all automation and orchestration tooling; because the goal isn’t automating tasks, it’s orchestrating process. Coordinating workflows. Ferreting out inefficiencies.

Orchestration is the glue that binds automated tasks together. For example, recently a missed route update caused some unexpected down time during what was considered a routine* decommissioning of gear. A missed route update. In other words, someone forgot. That task might have been automated, it might have just needed to be kicked off. But it wasn’t.

That’s the kind of thing that would likely not happen if the process was governed by some form of orchestration; if there was a clear list of what needed to be updated or changed based on the removal of that particular component. If orchestration was involved to provide visibility and oversight of the operational processes needed to perform such a routine task. This does not mean the tasks should have been automated; in fact automation is not strictly required to achieve the visibility and oversight necessary because processes are about workflows and hand offs, not specific tasks completed during the steps that comprise a process.

Which leads us right back to Business Process Management. Take a deeper dive into the aforementioned Bearing Point survey and you’ll find more similarities with DevOps:

Crucial elements for sustainable BPM are clearly defined roles (95%), top management support (92%) and a clear mandate (92%)

Typical implementation barriers such as the lack of resources (53%), sponsorship (32%) and experience (25%) need to be identified early and addressed in a structured way

BPM as a continuous (=sustainable) approach is already adopted by 59% of companies, coordinated and implemented either centrally or locally

Involvement of stakeholders (90%), process oriented thinking (90%), organizational institutionalization (82%) and a well-communicated concept (75%) are elements of communication and change management which must accompany the establishment of sustainable BPM

That’s because both are ultimately about process, with automation being only one of the tools in the tool chest that support implementation of a continuous approach to enabling agility in the process layer. Indeed, about 38% of the companies in the survey use software to support their BPM initiatives while another 37% do not. A spreadsheet with the right Six Sigma calculations can be as powerful as any automation framework, toolset or API in identifying the dead space between tasks that introduce inefficiencies and delay application deployments.

BPM has, like DevOps, brought tangible, measurable results to the table. Bearing Point found that almost three quarters of the surveyed companies (72%) were able to achieve measurable benefits through BPM such as process lead time (72%), process cost (68%) and error rate improvements (67%).

So rather than focusing too much on automation, adopters and proponents of DevOps would do well to up-level the conversation and start looking at process orchestration and improvement as the goal. The successes BPM has had on the business side of the equation tell us that doing so will provide a better path to achieving success with DevOps in the operational side of the equation.

* I’d argue that in today’s environment of hyper-sensitivity to availability that nothing that touches a service or gear in the critical path of an application is “routine” but, hey, it is what it is.
March 12, 2015
How Deploy Frequency Impacts Infrastructure Stability
I recently took some time out from reading the Internet to read a very specific piece of research I found on the Internet from IDC – its “DevOps and the Cost of Downtime” to be exact. Sponsored by AppDynamics, this fascinating read included a variety of statistics that are helpful in understanding the impact of (or a lack of) DevOps on organizations given certain financial and competitive impacts.

I won’t bore you by repeating what you can so easily read yourself, but I have pulled out a few key stats that are worthy of further investigation – that of the failure of stability on the business and the predicted increase in deployment frequency.

The average hourly cost of an infrastructure failure is $100,000.

The average hourly cost of a critical application failure is $500,000 – $1M.

The average number of deployments per month is expected to double in 2 years.

Interestingly, that prediction on deployment frequency is a good thing and carries with it the possibility of making sure the costs associated with the other two rarely (if ever) happen.

We’ve already gotten to a point where “speed” with respect to DevOps is effectively equivalent to “time to market”. But there’s another measure of “speed” that’s just as critical, particularly when we apply it to maintaining stability in the infrastructure. That measure is MTTR (Mean Time to Resolution). This measure is very important when you consider that 60% of production failures are caused by human error or lack of automation according to research from ZeroTurnaround. A goal of DevOps, then, should be to reduce that number as much as possible and introduce more stability into the environment. Frequency of deployment can actually be of assistance in realizing that goal.

So how does frequency of deployments help do that?

First let us consider the number of possible points of failure that might cause downtime – both in the application and core network infrastructure. Then let us consider that each of those possible points of failure also have multiple potential causes of failure. A single service in the logical critical data path within the core service infrastructure, for example, can easily cause “downtime” that, in the case of IDC, would be characterized as “infrastructure failure” and run a tab of $100K per hour.

If that application is a critical business application, the tab runs much higher per hour.

Obviously the speed with which we restore stability (service) in such an event is a factor in the overall impact to the business.

This is where DevOps comes in and where operationalization can play a significant role in mitigating the impact of downtime by aiding in faster time to resolution.

Consider that one of the most difficult tasks in the network is not provisioning or configuration, but troubleshooting. To be fair, troubleshooting in app dev, too, is one of the more time-consuming and frustrating tasks, consuming 75% of a developer’s time (according to research from InitialState). That’s close to the 65% of network engineers who are dedicated to “break and fix” tasks, who spend their time troubleshooting network issues up and down and all along the network stack. Needless to say, if you just deployed ten or fifteen different configuration changes splayed across the core and per-app service infrastructure and suddenly something is “wrong”, it’s going to take some time to figure out exactly which configuration change caused the problem in the first place.

The relationship between number of changes and frequency of deployment is no secret; if you’re deploying changes more frequently then it’s likely there are fewer of them. With an average sprint cycle of 3-4 months, the amount of change is logically not going to exceed that of a cycle that takes 12-18 months. The same holds true for the infrastructure; more frequent deployments should mean fewer changes).

Logic (and math) tells us that the fewer lines of code and configuration that have changed, the less time it should take to identify the culprit should something go wrong, thereby hastening the time to resolution. Assuming that an agile, frequent deploy rate results in fewer changes to the environment it stands to reason that this approach should also reduce both the number of failures and the time to resolve any that do crop up. Frequency is a significant factor in reducing failures by increasing its overall stability.

This is backed up anecdotally by Puppet Labs findings in its “State of DevOps” in which organizations adopting a DevOps approach reported 50% fewer failures and a 12x improvement in time to resolution. That’s stability at work and it’s driven in part by the frequency of deployments and its positive impact on MTTR.

So let’s apply this approach (and logic) to the rest of the services along the critical data path in the data center infrastructure, i.e. apply it to application services like caching, web application security, load balancing, identity management and application access control. Automation to reduce misconfiguration caused by human error and smaller, more frequent changes that enable a faster time to identity the inevitable issues that will be introduced. No methodology or approach is going to completely eliminate errors, after all. There’s a limit to how perfect we can get and it’s pretty much a Zeno’s Paradox kind of problem. While we keep solving current issues, more config and code is being deployed that is statistically likely to contain yet another error, and so and so forth.

But the goal is not necessarily to eliminate all errors (though that’d be nice, it’s simply not practical and it’s the primary reason why our goals for uptime are called “5 9s” and not a “1 with two zeros”) it’s to reduce the number of errors and their impact on downtime. And that is something we can achieve without running afoul of Zeno and his paradox.

Smaller, more frequent deployments can result in a more stable environment in which downtime is reduced to a level consistent with acceptable risk.

DevOps, whether applied directly to infrastructure or not, will play a role in improving its stability which in turn has a direct and very real impact on the business by eliminating (some) downtime thanks to more frequent deployments.
February 26, 2015
Measure Twice. Automate Once.

“Shipping” code may start with the completion of development but it doesn’t end until after the operational delivery process ends. Just like any fulfillment process, shipping is actually just the kick off of an often arduous journey that ultimately delivers the app to the end user.

Ship It

Back in the old days (when they still let me write code) one of our favorite phrases was “ship it” to indicate completeness of a task. That’s because the ultimate goal was, of course, to “ship” the code off to the users so they could start doing whatever it was they were going to do with it*.

But the reality was, of course, even when we were really ready to “ship it” to users that it didn’t actually get to users for varying amounts of time. Like shipping anything today, it depends on the shipment method. In terms of today’s enterprise IT, “shipping method” is really a metaphor for the operational approach to deploying all the infrastructure – the “middle boxes” – with the services required to actually deliver a secure, fast and available application to its ultimate users.

We know from various studies that this takes time. From days to weeks to even months. And we know we don’t like that and we want to improve it. Which is where DevOps comes in, or NetOps or SDN or SDDC or SDO or whatever you want to call it.

Speeding up the deployment process is certainly one of the goals of operationalizing the network, but without a baseline how do we know if we’ve gotten faster in the first place?

That’s a piece of the DevOps puzzle we’ve been ignoring – the need to measure the deployment process. You might notice that “measure” is bolded and in red, that a visual clue meant to reinforce the importance of the measurement aspect of DevOps. If you aren’t measuring how long it takes to deliver that app from dev to its destination – production use by an end-user – then you have no idea which pieces of the process are holding you back. You can’t effectively determine where automation might be applied to improve the time it takes to get to market. You don’t know exactly where the obstacles are that need to be addressed in the first place.

While we can make some assumptions based on long experience and research and point to handoffs across the IT silos, the reality is that there probably isn’t just one handoff between ops and the network and there may be multiple intra-IT group handoffs. For example, the network ops that handles provisioning network services might not be the same team that handles load balancing and performance services. But they’re both within the “network” silo. There may be multiple handoffs back and forth as various services also need networking services on top of the ones required by the app itself.

Image Credit: Gaping Void

Without some kind of measurement of how long each of the various steps is taking it’s impossible to determine exactly where the bottlenecks in the app deployment process are in your organization. I can not only determine where that Minecraft Lego set I ordered for my son’s birthday is right now, I can see how long it’s taken to get from each point in the process. I know it goes from point A to point B in X days, and from point B to point C in Y days, and I know if there’s a winter storm that it’s going to cause a slowdown in that second step (which is called a movement in the logistics vernacular, in case you were wondering. I knew you were, you’re welcome!)

Measure Twice. Automate Once.

The hidden value in setting out to measure the deployment process is in pulling together an understanding of the process itself. It’s in finding redundancies and discovering sub-processes that are running in serial but could be running in parallel to improve them. It’s not necessarily about automating the whole thing up front, it’s about understanding what the process is and making it more efficient – whether through elimination, reorganization or automation (or all three).

You don’t simply want to be automating things willy nilly without any idea whether it’s going to do any good. As my dad taught me about working with wood, you measure twice and cut once. With DevOps you want to measure twice (at least) and automate once. Otherwise you may be wasting cycles on one-off tasks or tasks so specific to one application that automation has no real value because it’s not the same across applications.

The key is to start with defining what you’re going to measure, do the measurement and then analyze it to determine what you need to improve.

Because automating just for automation’s sake isn’t any more fruitful in the end than adopting technology for technology’s sake.

* I say “whatever” because over the course of my career I’ve worked on tax software, GIS software, communications and transportation and logistics systems. There’s a lot of variance there in what end users might be doing with “the app”.

February 2, 2015

Tag: enterprise

About the Author

BPM? Are you seriously?

Ship It

Measure Twice. Automate Once.