devopsy.kinsta.cloud

Category: Enterprise DevOps

Enabling Collaboration Across Diverse Cultures

Welcome back to this series, ‘Enterprise DevOps Q&A.’ In my last post, I talked about breaking down the barriers between dev and ops. Now, I want to address another, very typical question from a global enterprise about managing cultural differences in a globally dispersed enterprise.

Q. Being part of a large org, our biggest issue seems to be a global culture clash. How can we address the issues of geographic location and culture that are a big obstacle for us?

Large multi-national enterprises, like CA Technologies too, often have major data centers in multiple geographies in North America, EMEA, Asia and Latin America. Despite so much data center consolidation and cloud migration, cross-geo cultural issues remain a major issue.

For example, in Israel the holy day is Saturday, while in the US and Europe it is Sunday, so it can be hard to lock down a release window. In Japan it is common to work long, late, and on the weekends, while by contrast some French labor agreements specifically limit the workday, so ‘crunch time’ may not have the same meaning in both locales. In the USA, staff may only get ten days annual vacation, and not even take all of it, while much of Europe takes all of July or August off, so consistent teaming can be a challenge.

To help you address these geo-cultural challenges, here are some approaches I have seen my customers adopt including:

A local approach

A local approach, such as co-locating devs and ops to service business users in a common geographic or cultural area allows teams to adopt DevOps practices within cultural norms. This may mean imposing a difficult structural change, but it can also be very effective. See my last Q&A for more on this approach.

A ‘core team’ approach

Some businesses have assigned one or two team members from each cultural area to represent their teams in a common virtual or ‘core’ DevOps transformation team. Representatives on the core team can translate (sometimes literally) for their ‘home’ teams, and bring unique culture back to the core team. This can raise process/hierarchy barriers, but when done well can also help to break them down.

A vertical approach

I have seen a handful of global businesses implement DevOps vertically, with devs and ops teams, often geo-located, acting as service providers to individual business units, rather than the whole company. This is best suited for organizations that are already organized by business unit, but it ensures teams have common focus on specific and shared business problems.

A ‘known good process’ approach

Speed the flow of software delivery by focusing on refining and documenting specific processes all teams can rely on. Each geographical or cultural team can define and share their own ‘known good process’ (cf. the Visible Ops handbooks), and share the interfaces to those processes, so they can continue to optimize their own activity, but work seamlessly with other teams at handoff points.

A process automation approach

Even better is if each team can encode their processes in automation to make it possible for anyone, anywhere, using any language, to execute those processes remotely. Even small improvements matter, so even task automation can help get closer to a cross-cultural, ‘enterprise-wide,’ DevOps approach.

A ‘pace-layered’ approach

Different business services have differing levels of acceptable risk and tolerance to change. Mobile app dev, for example, is more tolerant of risk than, say, accounts receivable. Rather than changing the world (literally!), instead build a DevOps approach into a specific service delivery team, even in a single geography. This ‘pace-layered’ approach is working very well for some of the world’s largest enterprises.

A tactical approach

Similarly, you do not need to fix all your service delivery problems at once. In a global enterprise, pick one big winner at a time instead. Identify constraints in the test process, a roadblock in QA, a provisioning delay, a problematic release process, a monitoring gap, or a broken feedback loop, and connect cross-cultural staffs from similar functional teams to solve specific shared issues.

A technology-assisted approach

Regardless of the approach, technology can assist. Global tools with multiple language support can enable common experience and terminology. Telecommuting can give regional staff a virtual presence in team meetings. At CA Technologies, we are trialing remote-controlled robotics to give staff a physical avatar in meetings from anywhere across the globe.

Other approaches?

As with most of DevOps, there is no definitive prescription, and I am sure there are many other approaches that can work. If you have ideas or experience to share, please post in the comments below, and let us all know how you have or could address these cultural and geographic barriers.

And remember, if you have questions about Enterprise DevOps, let me know and I will try to answer them for you. If I can’t answer them myself, I will find someone who can! Please leave your questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.

August 11, 2014
A DevOps Approach to Delivering Frequent, High-Performing Releases
We live in an application-driven world. Applications power businesses, provide connectivity, and help fulfill goals and customer needs. In a competitive world, needs evolve quickly as our users demand more features — all the while expecting around-the-clock availability and reliability, whether they use smartphones, tablets, or computers.

Keeping Up With User Demands

As IT folk — irrespective of whether we are in development or operations — we need to deliver new apps and features frequently to satisfy our users’ needs or else we risk losing them to competitors or alternatives. What do we do to stay ahead? We release updates faster by:
- Organizing around small teams and delivering more agile releases
- Automating where possible (test and release processes)
- Deploying updates into production when available
However, the unfortunate truth is that frequent changes often cause errors and instability in production. We need to mitigate risk by taking a close look at our processes and practices before we hit the gas.

Preparing for fast and frequent application releases

Here are two issues (amongst many others!) that hold us back from delivering frequent updates, while ensuring reliability:
- Tedious release processes: Developers have a multitude of collaboration tools, Kanban boards, build tools, and agile practices, and can build features quickly. But once it comes to releasing these changes through QA, staging, and production, the process is more manual, ad hoc, and slow.
- Unreliability in production: Code that runs as expected in development sometimes falls short in production. Production environments are subject to limitations imposed by third-party services, regional ISPs, firewalls, and so on. They are often quite different from development and QA environments.
Collaborate on a singular goal

Development and operations teams come from different schools of thought. Devs push for change and ops drive for stability. While they both aspire to offer superior software that adds value to users, their goals are diametrically opposed! What we need is a culture where development and operations teams draw from each other’s experiences, perspectives, concerns, and continuously improve processes to build and deliver resilient applications rapidly, into the hands of their customers. This requires discipline, a sense of mutual respect, focus on the big picture, and a strong commitment toward delivering on business goals.

Streamline and automate releases

Developers enjoy streamlined processes with their collaboration and build tools to produce releases effectively. However, the operations folk do not share the same good fortune. Quite often, they get artifacts, scripts, and resources from different members of the team, and run them in sequence in order to deploy and test the application. This lack of coordination and automation greatly slows down the release process. Here’s how you can begin to address this issue:
- Critique your processes: Where do the dependencies lie? Are there any redundancies that can be consolidated? Where are the bottlenecks, what’s causing them, and how expensive are they? Start with small opportunities that can bring big improvements.
- Automate: Once you have optimized release processes, look to automating them. Time and effort saved from repetitive tasks can be put toward innovating and adding value.
Once released, monitor for quality

By automating steps within the release pipeline, you set yourself up for increasing release velocity while ensuring optimal quality. However, no amount of preparation can guarantee expected results in production. Elements out of your control — like network traffic, third-party services, local ISPs, or browsers — can potentially impact user experience or disrupt service. Many of these outages have been very public. What we need is a strong feedback loop with every release that measures metrics around performance, exceptions, and, most importantly, end-user experience. Watch out for subtle changes in performance Subtle changes can alter your application’s end-user experience, costing you customer acquisition and brand loyalty. It is death by a million pinpricks. Slow applications, JavaScript errors, and 404s are very effective in turning users away. Here is where application performance monitoring tools — like Riverbed SteelCentral™ AppInternals — help diagnose problems that degrade application performance and behavior. They monitor end-user experience from your users’ devices, trace transactions through the application tiers and infrastructure, and diagnose performance problems down to application code. These insights gives developers the visibility they need to quickly fix problems that occur in production without going through the painful process of recreating and analyzing them.

Monitor end-to-end application performance continuously

Monitor your application for performance and behavior in production. Make this an ongoing activity during peak and off-peak hours alike. A keen insight into your application and infrastructure health can reveal snags that would otherwise go unnoticed. You really want to look for symptoms that compromise end-user experience, such as:
- How are my servers and infrastructure performing at current capacity?
- How are users in one location faring compared to others?
- What parts of my application — internal or third party — impact performance?
Diagnose problems with performance monitoring tools

Diagnose where and why these symptoms occur. Are they because of poor code, network conditions, server capacity, or third-party services? What is the context in which these issues occur? Which code segment, SQL query, or app component is at fault? You will be able to package up everything the development team needs to fix the problem and prevent it from recurring. Many of these tools, like AppInternals, allow developers can view this information from within IDEs like Eclipse and Visual Studio.

In conclusion, aim for continuous improvement

Particularly in the case of rapid release cycles, constant monitoring is a necessity. Identify symptoms in runtime — in QA and production — and quickly fix them using diagnostic data captured by your tools. This will keep the backlog low, your IT team productive and focused delivering new features, and your customers happy.
August 8, 2014
Enterprises, DevOps and Economies of Scale

In infrastructure there are two economies of scale: operational and capital. DevOps should attend to the former, especially in the enterprise.

Soon after cloud burst (sorry, pun not intended) onto the scene we all learned about economy of scale because that, primarily, is what cloud computing’s business model is based on. At great volume, standardization can be used to achieve greater output and, thus, can cost less.

But there’s a second economy of scale at work with cloud; the operational economy of scale. If cloud providers scaled operations in the same way that enterprises traditionally scale operations, well, instance prices would be much, much higher.

What cloud providers recognized is that in order to achieve the economy of scale required to make cloud a success they had to address both the operational and capital economies of scale.

According to research, surveys and studies, operational overhead consumes anywhere from 63% to 80% of IT budgets. That’s just maintenance, management and general “keeping the lights on” kind of things. And each time a new application is pushed into production, that number gets bigger because there are more of everything – from storage to compute to network – to maintain.

To wit, Computer Economics “IT Spending and Staffing Benchmark 2014/2015” indicates that 67% of organizations were increasing their operational expenditures versus only 20% that planned on a decrease. Because IT budgets are generally determined as a percentage of revenue, that means operational budgets are chewing up the business, slowly but surely, and leaving little room for becoming an enabler of business rather than an application assembly line.

The holy grail, as its been described, for IT is to move budget from operational overhead to innovation. To do that means somehow magically reducing the costs associated with operations. Adding more people doesn’t help; that just adds more overhead. The goal is to do more with less; to increase the ratio of systems and services and applications per operator that can be managed. That’s operational economy of scale at work.

To do that, enterprises need to look to DevOps and its leverage of automation tools to enable leaner, more efficient operational processes. That means not just automating, but enabling a self-service model in which common provisioning and deployment processes can be orchestrated at the behest of the app or business owner. It means applying the same “gold image” standard used for provisioning and managing virtual machines to infrastructure; leveraging templates to gather the requisite input from app or business owners and automatically provisioning systems.

Limited variations on an infrastructure deployment template will enable customization where necessary, but otherwise provide standardized (stable) provisioning and deployment processes that can be self-service enabled. By empowering developers, app and line of business owners to provision application required infrastructure services, operations can use technology to scale to meet demand while lowering costs.

The goal of DevOps in the enterprise is not to enable 10 application deployments a day. It’s to achieve operational economies of scale and a more consistent, predictable and repeatable operational environment.

August 5, 2014
Removing the Wall Between Dev and Ops
Welcome to the new devopsy.kinsta.cloud Q&A series on Enterprise DevOps. In the first post in this series, I answered five key concerns for adopting a DevOps approach in a large enterprise. Now, I want to address a more specific question from an admin at a large enterprise about the silos and barriers between dev and ops …

Q. How do you work with customers to remove the imaginary wall between dev and ops to create that DevOps environment?

This is perhaps the core of a DevOps transformation – dev and ops teams working closely together toward common goals, rather than in isolated silos. I have seen several methods have substantive impacts on these entrenched silos, so I will take you through some of the more effective approaches.

Familiarization

In many organizations, dev and ops barely know each other – a huge barrier to the empathy at the centre of devops. Breaking this wall down may be as simple as getting both departments together for some social activities. Some successful approaches I have seen include:
- Sponsor a one-off or regular shared non-work team activity, like paintball, a pool tournament, team picnic, LAN game, or cooking classes. Bring both teams together in a non-workplace environment to work together. Remember to mix up the teams – you do not want to reinforce any us vs them mentality;
- Schedule sessions for team members to present to each other in short sessions (think lightning talks like Ignite or TedX) beyond technology to any areas of interest – like ‘My Sports Team,’ ‘Steampunk in the 21^st Century,’ ‘Restoring Train Carriages,’ or ‘Strategies for TF2 Domination.’ Share common interests to create empathy for each other as people, not just functionaries;
- Initiate informal knowledge sharing, with regular ‘lunch-and-learn’ or ‘brown bag’ sessions, where team members can explain what their role is, why they do it, how they do it. Offer a ‘Day in the Life…’ presentation, a Q&A session, or a whiteboard over pizza and sodas and your technical staff will come out like vultures, ravenous for … knowledge!
Co-Location

Telecommuting has many benefits, but standup meetings are hard to achieve across five time zones, ten locations and six languages. People communicate better in person, so physically co-locate teams as we at CA and others have done, to enable them to connect and talk in person, not just via e-mail or IM.

Geo Alignment

In multi-national enterprises central developers often push code to global operations, reinforcing the silo mentality. Align local devs and ops with local business users instead to facilitate customer-centric agile development, and enable smaller, closer, more agile teams with more immediate shared goals.

Secondment

Consider seconding team members into the other team e.g. for one day a week, one week a month, or one month in six. When ops learn firsthand the basics of their dev counterparts, and vice versa, they will better internalize how to work together, and the impact of each role on the overall process.

Restructuring

Some enterprises have combined dev and ops into one team. While often seen as an ‘anti-pattern,’ for some organizations this is working well. Others have adopted a flexible org structure with hierarchical management for administration (reviews, goals, training, etc.), but loose teaming for functional duties.

Shared Technologies

Beyond cultural and attitudinal shift, technology can also help. A shared toolchain for the end-to-end service delivery lifecycle can bring teams together too. Common tools for planning, testing, release, monitoring, or triage will help break down barriers with a common terminology and shared experience.

Other Approaches?

While there are no silver bullets, these are some ways I have seen enterprises break down these silos. But perhaps you have seen or been a part of other approaches? If so, other readers and I would love to know more, so please take the time to leave a note in the comments below, or drop me a line directly.

And remember, if you have questions about Enterprise DevOps, let me know and I will try to answer them for you. If I can’t answer them myself, I will find someone who can! Please leave your questions in the comments below; hit me up on Twitter at @AndiMann; or send me an e-mail to me at andi.mann@ca.com.
August 4, 2014
Five Top Tips for DevOps At Scale
Since I published my blog explaining the topic of ‘Big DevOps’ late last year, the concept of Enterprise DevOps has exploded, with commentary, experience, and advice (some good, some awful) popping up from many different quarters. Fortunately, many pundits have now come to agree that application delivery in a traditional enterprise is very different from in a typical web-scale/startup environment.

As a result, a growing body of work is filling a significant and important gap with unique insight into Enterprise DevOps, from many excellent commentators and practitioners who understand how large enterprises are different, and how to address those different challenges.

As Enterprise DevOps takes off, I seem to field questions about it on an almost daily basis, so I decided to contribute to the body of work with a webinar explaining my answers to five of the most frequent challenges I hear about DevOps at scale. It was titled ‘Big DevOps: Five Top Tips When Implementing at Scale’ and you can catch the full replay on YouTube.

The top five issues I chose to address are below:

My five ‘Big DevOps’ tips for overcoming these issues center mainly on cultural change, because DevOps is primarily a cultural and attitudinal shift. However, it is also undoubtedly true that certain technologies can play a large part in enabling that change, especially in the more complex world of larger enterprises.

With that in mind, my top five tips to address these challenges are as follows:
- Facilitate a culture of teaming and collaboration – this is at the core of devops in any environment, large or small, but especially for large organizations with bright lines between teams, this is essential to overcoming the tyranny of complex and dispersed silos
- Establish agile development as a shared discipline – agile development is brings code to market faster, but can result in just filling the clogged pipes of PMO, HR, training, security, and other departments faster. ‘Agile’ needs to be common throughout the SDLC – in dev, ops, and beyond
- Automate relentlessly to enable rapid devops response – automation allows faster execution throughout the system, enabling downstream processes to keep up with speed of DevOps. Automation also adds audit and governance, especially important to larger organizations
- Push smaller releases faster, measure and remediate impact – large organizations especially are can avoid big errors by avoiding big releases. Adopt faster, smaller cycles with closer scrutiny to achieve a DevOps approach while managing the risk of faster cycles and fewer controls
- Manage user identities to enable shared access – in smaller organizations it might be acceptable to give out root access like candy, but not for large enterprises. Identity and access management can enable a new DevOps approach that is safe, secure, auditable, and well-governed
While these tips may not be the perfect answer for every organization, I have seen how these approaches have made the journey to DevOps easier, safer, and more effective for big enterprises. I hope they help you on your journey.

If so, you will want to stay tuned to devopsy.kinsta.cloud. Since starting work on Enterprise DevOps concepts, I have collected many questions on how large organizations can best succeed in their DevOps transformation. I will be sharing these questions along with my answers here on devopsy.kinsta.cloud in a regular series, to give everyone the chance to understand how large enterprises can overcome the challenges of adopting DevOps at scale.

So keep your eyes out for my posts over the coming days, weeks, and months in the brand new occasional series, ‘Enterprise DevOps Q&A’. I hope you will keep reading, and maybe even help me out by discussing your experiences and ideas in the comments below.

Finally if you have any questions about Big DevOps, Enterprise DevOps, or anything to do with adopting a DevOps approach at scale, let me know in the comments below too. I would love to hear from you, and answer your questions if I can, or source guest opinions from the DevOps community.

Meanwhile, please stay tuned for more Enterprise DevOps Q&A. I am excited to continue the discussion on what it takes to transform application delivery at scale.
July 24, 2014
Is SDN just DevOps for the network?

The answer depends on your definition of SDN.

(more…)

July 22, 2014
Provisioning versus Configuration

There’s a ton of hype and excitement surrounding containers, a la Docker, today, much in the same way virtualization took the data center by storm. In both cases, the excitement for operations focuses on the ease with which infrastructure might be deployed and managed via such technology.

Whether container or hypervisor, however, there remains a distinct difference between server and network virtualization that must be considered before getting too excited about the possibilities.

There are two distinct use cases for which virtualization and containerization of anything are considered beneficial: scale and deployment.

Scale

In terms of scale, both virtualization and containerization are excellent technologies. Both can fully encapsulate a service – whether application or network – and enable rapid deployment of multiple instances of the same service with very little (if any) post-provisioning requirements. This is the premise upon which cloud computing and NFV is based: clones of services and applications rapidly deployed to enable elastic scalability that matches demand.

For scalability purposes, virtualization or containerization of services is a match made (mostly) in data center heaven.

Deployment

Enabling rapid deployment of new services using virtualization or containers, however, is not necessarily going to net the same benefits. It is important note that the standardization and APIs associated with virtualization and containers that enable scalability so seamlessly is focused solely on the management of the container or virtual machine. The ability to provision a virtual machine or container is not at issue; APIs enable the rapid provisioning of a container or virtual machine with extreme alacrity.

What those APIs do not do is enable the rapid configuration of those services once provisioned.

Configuration != Provisioning

This is a critical differentiation that must be made lest we assume we can leverage virtualization and containers to enable a self-service style data center, a la IT as a Service. While virtualization and containers make provisioning a service instance a relatively painless experience, configuring those services must still be accomplished. When deploying a new service, existing configurations will not be wholly valid. There will always be some amount of customization that must occur, if only to make sure the service is interacting with the right application.

This means that once provisioning of the service is complete, there is still work to be done. One does not simply launch a load balancing service, for example, without instructing it how to distribute load (what algorithm to use) and to what set of servers (configure a pool).

It is important to recognize the difference between these two steps in the deployment process and take into consideration the impact of configuration after provisioning on that process. Depending on the method of configuration, this step can have a serious impact on the speed and efficiency of the deployment process as a whole.

It is also important to note for monitoring purposes, as virtual machine health and status is not the same as the health and status of the service, whether that be an application or a network service. Both must be monitoring and managed in a virtualized infrastructure to meet MTTD (mean time to detection) and MTTR (mean time to resolution) objectives.

In most cases, the impact from configuration is the greater of the two, with its time requirements most significantly impacting the application deployment lifecycle. It is optimization of configuration processes, not provisioning, that will ultimately provide the biggest benefits for DevOps in terms of reducing errors and improving time to market.

July 10, 2014
Building Security and Compliance into the SDLC

As the concept of DevOps has become more popular in recent years, there are some common themes that always seem to come up in discussions about it. For example, how often do we talk about ways to speed up the software development life cycle (SDLC) in order to bring innovative applications to market faster and meet ever-increasing customer demands?

As important as this and other popular DevOps themes may be, though, they often have the side effect of overshadowing some of the less glamorous topics, such as security and compliance, which is often viewed as a sort of “necessary evil.” That said, the risks of putting an unsecure application into production can create massive challenges for organizations – from breaches to loss of company IP to brand damage – making security and compliance a topic that must be tackled as enthusiastically as accelerating innovation.

Three Critical Areas of Focus

When it comes to considering how security and compliance can become more integrated with DevOps practices, there are three critical areas on which all organizations should focus: shifting security left in the SDLC, building security into policies and creating an audit trail throughout development.

Shifting Security Left in the SDLC

Organizations that incorporate security in the earliest stages of the SDLC will ultimately benefit from products and applications that are secure by design. But in order to accomplish this, they can’t wait until the software hits production to test it for vulnerabilities.

Service virtualization can help by simulating the application delivery run-time environment and its data, allowing developers and testers to focus on security immediately, instead of waiting for access to crucial resources. In addition, a release automation tool can help collapse development cycle times from days to minutes, enabling IT teams to spend more time improving the functionality, quality and security of their applications – and less manually promoting them from stage to stage.

Building Security into Policies

It used to be that every element of application development took place between an organization’s four walls, but with the rise of cloud and software-as-a-service (SaaS) technologies, it’s not uncommon for different stages of the SDLC to occur on different infrastructures. So if the environments are always changing, how do we ensure security is consistently addressed? By building it into the policies that comprise our continuous validation methodologies.

Release automation enables development teams to create policy-driven infrastructure provisioning rules that automatically determine what application elements can and cannot be staged on specific infrastructure types. For example, a common policy would be to never let customer or database information move to a public cloud environment, where sensitive information could be lost or stolen.

Creating an Audit Trail through Development

To this point we’ve centered the conversation mostly on security, but being able to maintain – and prove – compliance throughout the SDLC is an equally important challenge that IT development and operations teams must solve.

So when we think about compliance, we’re really asking questions like “What application moved from this stage to that stage?”, “Who released the application into production?” and “If there was a rollback, what was the cause and who rolled it back?” In other words, we’re talking about audit trails and logging all activities within the SDLC – something that can be difficult when numerous manual steps are involved. However, a release automation tool will capture all of this data as a matter of course as it automates release deployments through orchestration and promotion of applications from development through production. This means you’ll always be able to answer those who, what, where and when questions that pop up during audits.

Eliminating Vulnerabilities and Getting Back to the “Good Stuff”

I’ll admit that security and compliance are not the most exciting topics in DevOps. If application development were a meal, they’d be the pile of lima beans you have to eat before you get to the dessert that is innovative feature development. The good news is, service virtualization and release automation tools can help you speed and simplify the process of building security and compliance into the SDLC, which ensures your applications will hit the market without vulnerabilities – and lets you get back to the fun stuff.

July 7, 2014
Infrastructure-as-code, new rules for the old game

There’s been a lot of discussion regarding DevOps and its role in fostering quality and continuous improvement through increased levels of automation. In turn, this automation is fostered through emerging infrastructure and software architectures that are exposing control plane and management programming interfaces, which has been dubbed in many circles as Infrastructure-as-Code.

This movement allows development and operations personnel to have real-time interactions with the underlying environment—the network, servers, operating systems, and storage—that are used to deliver applications and data. It also means that as the environment becomes more programmatic the business must treat this this effort as a first-class application development process and ensure that the corresponding assets follow common coding practices and meet with a consistent and sustainable architecture.

Those that are unfamiliar with the activities of system administrators and data center operations may have an ill-conceived notion after reading emerging literature on DevOps that operators have been running the business on the backbone of purely manual processes. Even as a life-long IT person, until I started working with infrastructure and operations as part of my role, I was unaware of what happens behind the great curtain that keeps the digital bits flowing through the organization.

The truth is that operations and administrators have been developing scripts for years to automate routine tasks. These scripts have typically been used to dynamically generate or update configuration files that were then applied after a restart. This change to Infrastructure-as-Code is significant because now these configuration changes can be made on the fly and are often immediately acted upon without having to take the system offline. If you’re now putting your hand over your mouth upon the realization of the potential disastrous effects this can have, you understand the impact of this power.

Thanks to Software Defined Networking, hypervisors, and even Software Defined Storage, an operator can programmatically change the entire environment underlying the application without having to notify the application of this change. Architecturally, this is a wonderful thing. Pragmatically? Well, there’s a lot of bad code out there that makes assumptions about the underlying infrastructure and is about to be exposed when migrated to these modern infrastructures.

Hence, building, testing and deploying infrastructure control applications must become part of the overall Software Development Life Cycle (SDLC) and begin to be built, tested, deployed and managed using the same controlled techniques that are hopefully being used for any applications that are being developed by application engineering.

You may be asking, “Wait, doesn’t that mean my QA environment must have the same infrastructure components as my production environment?” To wit I respond, “Maybe!” In some cases this may be advantageous for ensuring that production deployment can support continuous deployment processes, which will ensure limited human intervention from the point of user acceptance through deployment, which will, hopefully, lead to limited opportunity for production failures. However, because we’re moving to a more programmatic infrastructure many vendors can also supply virtual instances of their infrastructure code for purposes of testing and this may be satisfactory for your needs.

Of note, there are many businesses out there that do not develop their own software. They primarily use Commercial-Off-The-Shelf (COTS) applications. Thus, these businesses may not have an established SDLC practice. Likewise, these businesses may not believe DevOps is relative for them since they do not do “development”. If your business falls into this category, the move to Infrastructure-as-Code and Software Define Data Centers should be perceived as a driver for DevOps adoption as well as formally adopting one of the well-defined SDLC approaches. Even if these components won’t impact you for a few years until your next refresh cycle, the advantages DevOps brings around improving quality and consistency in delivery is of definite value to your business now.

Infrastructure-as-Code is a significant change in capability that operations has for delivering compute and storage resources to the business. It will allow them to have greater insight into the hardware allowing them to do predictive maintenance, migrate applications to various resources based on demand, and tune the overall environment to meet the business’ needs at a given moment. In short, it will allow IT to deliver compute services with a high degree of availability and security. It will also expose shortcomings in existing applications that have made assumptions about the underlying environment on which it is operating. Thus, if operations and application engineering are not working together and the entire application environment inclusive of the infrastructure control apps is not tested and deployed as a unit, the likelihood for failure will be high.

June 30, 2014
The DevOps value stream is not SOP

#devops #lean Don’t confuse doing something faster with greater efficiency.

Much of the focus on DevOps remains at the implementation level – on gaining efficiencies through automation of tasks and orchestration of processes. While this is not a bad thing, such a micro-focus on the moving parts can result in the loss of efficiencies to be gained at the macro-level.

One of the biggest inhibitors of a timely release cycle is not the moving parts, but rather the process in which those moving part are executed. It’s not necessarily the actual push into production that’s problematic, but the length of time it takes to accomplish certain tasks when they cross internal organizational boundaries.

For example, many years ago we began the process of pushing a new application into production – a web application. This required, perhaps obviously, changes to the corporate firewall. The “process” for accomplishing this change – essentially opening a port on the firewall for the intended public IP address – required no less than five different signatures across networking, security and architecture teams. There were only two security roles allowed to approve such a change, and the process was noted as taking up to 15 days to get a signature on a piece of paper and subsequently a rule change.

This was not a task level problem. It was not implementation issues. It wasn’t even difficulty in making the change on the firewall. It was all about the process. Scheduling a release meant taking into consideration not only the readiness of the application itself, but its supporting services – including networking.

This is why DevOps should start with a review of the processes used to release an application into an environment (whether test, QA or production). Mapping out the process is the first step to finding where the bottlenecks might be that are causing delays in moving that application to the next phase of its lifecycle.

Simply automating an existing process may not provide the value expected or touted by DevOps supporters, because if the process is poor, you just end up doing something poorly much faster.

In other words, the value of DevOps is not necessarily in simply automating SOP (Standard Operating Procedure).

The initial Value Stream Map captures the current process as it exists. This is known as the “As Is.” It is the actual process flow and should be created through observation of the current flow and with participation of those who are in the process. Do not make the mistake of pulling a process map from the procedural manual (SOP). The best technique for mapping is to be the unit (the “thing”) coming through the process. For example, in the insurance industry it might be the policy. In the auto industry, it may be the car. In each instance, the Value Stream Map begins with the request for the unit i.e. insurance policy or car. [emphasis mine]

http://www.goleansixsigma.com/how-to-visualize-a-process-with-a-value-stream-map/

For application deployment, the “thing” is … an application, and the process should be mapped based on what’s actually happening. Only when you have the process mapped can you evaluate it for bottlenecks and understand the steps of the process at which each group adds value. Value is defined as any step that moves the thing (the application” toward the end goal. Efficiency is then measured as a percentage; specifically it’s the value added time as a percentage of the total time it takes to complete the process (the cycle time).

Don’t confuse doing something faster with greater efficiency. If every step of the cycle is automated, the actual process efficiency hasn’t changed. The ultimate goal of DevOps should be about improving the efficiency of these processes, not just doing what’s always been done faster.

Some of the opportunities for improvement in efficiency that will surface is certainly interfaces with other organizational groups. Perhaps it’s that a process step might not take as long if you had relevant data or information up front. If you’re wasting time waiting for some piece of information, that’s not value added time. It’s just cycle time. More cycle time reduces the efficiency of the overall process by overloading it with wasted time. Better information gathering up front can help reduce the wait time and immediately provide greater efficiency of the process.

It is these opportunities for improvement that will be particularly helpful in the enterprise arena, where the issue is not so much attempting to hit ten deploys a day, but rather the predictability and efficiency of the processes that enable teams to hit a specified change window as desired.

DevOps is not about the tools, or the automation of manual tasks. It’s about the big picture, and that means gaining greater efficiencies through process re-engineering and orchestration as well as automation. Focusing on the micro aspects of DevOps will only ensure that if you’re doing it wrong, you’re doing it wrong faster. Step back and look at the big picture, and find those process steps that are really causing inefficiencies in your organization.

June 16, 2014