Many have asked me what I’m looking forward to at DevOps Enterprise Summit 2016 in San Francisco—it is the third year of our conference, and I think it will be our best conference yet! Here are three of the things I’m looking forward to most:
Where Are They Now?
First off, we will have seven speakers back to present for the third time at the conference. Having this pattern of repeat speakers is a bit unusual for a conference, but I love that we’re following these leaders on their continuing journey. For me, it’s almost as if we have front row seats of an unfolding documentary of the ongoing transformations that they are helping drive in their large, complex organization.
Why do I think this is important? My belief is that in the DevOps Enterprise community, many of us are on a common journey, increasingly being asked to help elevate the state of the practice for the entire technology organization, often involving thousands of engineers. To see the journey unfolding for other pioneering leaders is often a glimpse into our own future.
Jason Cox, who is giving the closing keynote about the DevOps journey at Disney. (So stick around!)
Each of these leaders will recap on their DevOps journey so far, whether and how their roles have changed, the new challenges that have emerged since last year, and what they’ve done to overcome those challenges.
We’ll also be hearing from four people who have had a tremendous influence on the entire DevOps community: John Allspaw, CTO at Etsy; Mark Imbriaco, who led the Ops organizations at 37signals, Heroku, GitHub, LivingSocial and DigitalOcean; Adrian Cockcroft, who led the migration of Netflix into the public clouds; and Michael Nygard, author of the seminal book “Release It!”
Transformations from Different Industry Verticals
We always want to hear about DevOps transformations at all stages of the journey—I’m particularly looking forward to three of them that that are outstanding on their own merits, but are also interesting because of the industry they operate in: airline, hospitality and insurance. These are three industry verticals that we don’t typically associate with DevOps.
Susanna Brown and Ben Chan will talk about how DevOps is changing how American Airlines works (and doing it in the middle of its merger and integration with US Airways), Matt O’Keefe will talk about speed becoming the prime directive at Hyatt Hotels and Resorts, and Opal Perry will talk about her efforts to transform culture at Allstate Insurance (which, like Nationwide Insurance, operates in an industry famous for its centuries-old, risk-averse traditions and decision-making styles).
I’m also excited about another conference first for us, which is the joint presentation from Suzette Johnson and Robin Yemen, from Northrop Grumman and Lockheed Martin, respectively. I met Johnson at an Agile conference for the Federal Government community, and I was so inspired by the work that they’ve done within their firms and the broader community at large to improve outcomes for federal programs—both civilian and military.
New Session Format: DevOps Workshops
As adult learners—and this is especially true for leaders—we tend to learn more effectively from observation, experimentation and group exercises and group learning, as opposed to didactic lectures and descriptions of theory.
I’ve observed that the DevOps Enterprise community is truly a group of people who are helping each other. To help further enable the spontaneous and serendipitous interactions that make great communities great, we are adding a set of sessions called DevOps Workshops.
These are 60-minute sessions that run alongside the conference talks, and each workshop will focus on one of the community’s top problem areas:
How can we better lead change in our organizations?
What are the modern architectural and technical practices that every leader in our organization needs to know?
How do we design our organization to best enable DevOps outcomes?
What are ways that we can more effectively bridge the information security and compliance gap?
Each of these workshops will have a 10-minute introduction, which will be followed by 45 minutes of small-group discussions. And to help ensure that each small group achieves its learning objective, as well as to ensure open and healthy dialogue, we have decided to have trained facilitators use the Lean Coffee format.
Lean Coffee helps achieve this through a democratic selection of topics and rigorously enforced time-boxing of each topic. I’ve always been amazed at how effective this technique is at creating good outcomes.
(Why Lean Coffee? In my personal experience, all it takes is a bad facilitator or one person who loves hearing themselves talk to ruin the small-group session for everyone. I can’t think of a better mechanism than Lean Coffee to neutralize these risks.)
In addition to the amazing lineup of repeat speakers, expert talks, and new stories, my hope is that these DevOps Workshops will be of tremendous value and create fantastic learning and networking opportunities for all attendees. The collaboration and relationships that result from each of the DevOps Enterprise Summits never ceases to amaze me, so we’d like to give attendees a variety of opportunities to engage with our speakers and with each other throughout all three days.
I’m looking forward to the best DevOps Enterprise Summit ever! I hope to see you there!
What can data scientists learn from DevOps? A lot, actually. By applying DevOps principles to the world of big data, you get DataOps, which streamlines data analytics.
If you’re reading this site, you’re probably already familiar with DevOps. The DevOps philosophy emphasizes continuous collaboration between software developers (the “dev” people) and system admins (the operations, or “ops,” folks)—as well as everyone in between, such as software testers.
It’s possible to apply similar ideas to the world of big data and data analytics. There, as in the world of software development and delivery, different teams traditionally have worked in isolation from one another. Those teams include storage admins, who keep data storage systems running; data analysts, who write and execute programs for interpreting large volumes of data; data quality specialists, who ensure data consistency; and data security teams, which protect data from attackers.
As DevOps professionals know, keeping different teams isolated from one another is not efficient. It limits collaboration and maximizes confusion when problems occur.
The data world, therefore, has much to gain by ending this isolation between teams and adopting a DataOps workflow. This will keep data operations flowing smoothly and provide continuous insight into the data that organizations collect.
DataOps is not a brand new concept. People like Andy Palmer have been writing about it for a little while now. But the term and the idea have yet to gain nearly as wide a following as DevOps. (There is not yet even a Wikipedia entry for DataOps—and you know an idea is still in its infancy when no one has taken the time to write about it on Wikipedia.)
Making DevOps Bigger
Meanwhile, from the DevOps perspective, the growth of interest in DataOps is significant because it exemplifies the broad applicability of DevOps principles. So far, the DevOps conversation has focused on the relatively narrow context of software development and delivery. But it can extend much further, too. DataOps is a start.
So, if you’re a data analyst, storage admin, data security professional or someone who works with data in any other capacity, it will be worth your while to borrow some ideas from your DevOps brethren. And if you work in software development, testing or delivery, it may be worth your time to educate your data friends on DevOps principles. It will make their workflows more efficient, while introducing DevOps to a wider audience.
Many business leaders are planning to make DevOps the cornerstone of their IT transformation strategy. But, as surprising as it might sound, skepticism abounds regarding how major vendors address the DevOps transformation challenges—they confuse DevOps with continuous delivery (CD) infrastructure.
“We wanted DevOps and we got after a nine-month-long implementation a continuous delivery platform which speeds up IT operations without significant business value. How differently can you help?” the senior business executive of a major financial institution in New York sarcastically asked me during a conversation three months ago.
The business executive is right: DevOps isn’t a technology as I’ve been implementing it, it’s primarily an accelerated application delivery approach implemented in the form of a technology-enabled application delivery operational model. Adam Jacob, CTO of Chef, substantiates that fact in “The Secret of DevOps: It’s Always Been About People, not Technology,” when he says, “DevOps is about bringing together all the people you need to build and run your business effectively, and empowering them to move as quickly as possible towards their goals.”
Agile Admin’s Ernest Mueller in, “DevOps: It’s not Chef and Puppet,” confirms Adam’s point: “Chef and Puppet are individual tools that you can use to implement specific parts of an overall DevOps strategy if you want to use them—but that’s it. They are fine tools, but do not solve DevOps for you.”
How can we implement DevOps in line with business leader expectations? Three Amazon Web Services (AWS) Design Patterns I developed can help to get the best of DevOps on AWS cloud: Software Organization Factory, Digital Organization Factory and CD Infrastructure Factory. This article discusses the Software Organization Factory.
The Foundation of DevOps Value: Consequences to IT Service Delivery
Before delving into these AWS design patterns, a quick reminder of what DevOps is, its value propositions and, more importantly, what changes are needed in how IT services are delivered.
The Ignored DevOps Business Logic We All Must Consider
The combined effect of cloud adoption and the rising digital economy is changing competitive environments; it forces business leaders to focus on innovation and market responsiveness as the competitive differentiators.
The fact of the matter is, by facilitating access to technology, cloud accelerates entrance of several thousands of tech startups that take advantage of the recent technology innovations to substitute traditional products and services into digital. This creates a context where new entrants increasingly compete with established brands, forcing business leaders to seek value paradigms other than IT efficiency.
The business executive argues, “In these new markets, our concern is to increase customer value by delivering the right services to the right segments at the right moment. You guys need to change the IT paradigm; IT alone does not create that kind of value.”
Then, what should be changed in today’s IT approach?
The Two Dimensions of Modern IT: The Operational Model and the Continuous Delivery Infrastructure
The new IT paradigm advocated in my recent book, “Rethink Your IT with DevOps: The Complete Guide to Aligning Your IT to DevOps,” is the idea that business value results from two dimensions: organizational agility and technology efficiency. In other words, it defines modern IT as a two-dimension asset including the operational model and the continuous delivery infrastructure.
The timely delivery of innovation transcends IT issues; it not only cuts across your sales, marketing, post sales, application development and IT operations, but also demands concrete answers as to how to accelerate business prioritization, decision-making and problem solving. The operational model is where business and IT people, processes, practices and tools are combined to create agile and cross-functional environments. The following exhibit shows a typical DevOps operational model:
Typical DevOps Operational Model
Organizational flexibility for the sake of flexibility doesn’t make sense unless technology is leveraged to speed processes. This is where CD infrastructures come into play; throughout the delivery process, they build on IDEs to accelerate coding and testing, on versioning tools to convert software version into releases, on continuous integration (CI) tools to build releases and on delivery pipelines to ensure continuous delivery.
So, what is DevOps?
DevOps is a Technology-Enabled Application Delivery Model
DevOps is a combination of value, principles, processes and practices shared not only by developers and IT operations but also by the business. The goal is to help the organization deliver the right service to the right market segments at the right moment.
DevOps from the Business Perspective
As I’ve been implementing it, it’s a cross-functional application delivery model involving the business and IT and enabled by continuous delivery infrastructures.
By enabling and automating agile practices, the continuous delivery infrastructure makes the operational model agile and provides the business with the flexibility and velocity needed to meet competitive challenges.
Design Patterns as DevOps Implementation Engineering Framework
As suggested by the business executive, today’s DevOps implementations struggle to deliver the promised business benefits. The reason is, the operational model dimension is ignored in favor of tools and infrastructure.
Another reason is, DevOps is young and is still at the early stage of the technology maturity model; the expertise and engineering methodologies to implement it as what it is—a technology-enabled application delivery model—are lacking. That’s where DevOps transformation design patterns help.
What are DevOps Transformation Design Patterns?
In this article, the term DevOps transformation design patterns refers to general reusable architectures addressing specific DevOps strategy. They’re not necessarily implementable solutions, their primary goal is twofold:
Offer a neutral language for designing and implementing technology-enabled operational models
Offer an engineering approach that reduces design and implementation duration and effort
They’re formalized architecture design and implementation best practices for DevOps and platform-as-a-service (PaaS) services.
Design Pattern as DevOps Implementation Engineering Methodology
To meet its goals—technology-enabled application delivery model—the design pattern is structured around four items including purpose, principles, general solution and implementation techniques.
Purpose: Clarifies the purpose of the design pattern along with the DevOps strategy to address.
Principles: Are best practices formulated as rules and beliefs supposed to guarantee expected DevOps strategy benefits? They’re the determinants of the solutions to implement the DevOps strategy.
General solution: Are blueprints resulting from the application of principles? They summarize the functional and technical architecture of the DevOps platform to implement.
Implementation techniques: What are recommended technical approaches, solutions and tools to implement the general solutions?
Why AWS? Why would it be the catalyst for DevOps value?
AWS-ome Design Patterns That Will Change DevOps Implementation
AWS offers such a rich, varied and easy-to-implement services portfolio that it would be unfortunate not to take advantage of it to modernize IT. The rationale underpinning the value businesses can derive from their DevOps transformation experience with AWS is summarized in the following figure:
Rationale of the DevOps Design Pattern
It’s the belief that:
Certain principles determine the AWS services needed to deliver the application delivery model (ADM) architecture that guarantees value.
The implementation of the ADM using these AWS services results in AWS-based DevOps platforms that tangibly deliver expected benefits.
The Four DevOps Principles that Yield AWS Value
Let’s explore the principles I’ve been using to make CIOs happy with their AWS cloud:
Think holistic, not only IT: Reminds that transformation isn’t only about IT but the entire organization’s value stream from marketing and sales to development and IT operations.
Think cross-functional, not only IT: Reminds that the bottom line is to set up a cross-functional application development environments to accelerate business prioritization, problem-solving and decision-making, and meet time-to-value objectives. They’ll be the core value propositions of the DevOps platform.
Make it agile, flexible, and speedy: Reminds that agility, flexibility and speed are the catalysts for competitiveness and business value. They’ll be part of the DevOps platform.
Think PaaS: Reminds that AWS provides varied services for implementing state-of-the-art CD infrastructure.
Let’s illustrate three AWSome design patterns that’ll change DevOps experience with AWS!
Implement a Startup Mindset with the Software Organization Factory
The purpose is to implement DevOps strategies focused on time-to-market and time-to-value. It’s adapted for tech startups.
To implement the startup mindset, the DevOps four principles for AWS are implemented as follows:
Think holistic and cross-functional, not only IT: Relevant business and IT resources are put together in a single and dedicated service development team supported by appropriate Scrum and XP agile practices.
Make it agile, flexible and speedy: PaaS services building on agile practices and AWS infrastructure-as-code are provided to make application delivery processes agile, flexible and speedy.
Think PaaS: AWS infrastructure as code and DevOps tools are combined to implement a CD infrastructure in the form of a PaaS platform.
The DevOps platform resulting from the principles application is illustrated below:
Software Organization Factory
The “Startup Mindset” platform is structured around three components including the cloud-enabled operational model, PaaS and the CD infrastructure.
The operational model builds on a three-stage application delivery life cycle:
Plan & measure: It mobilizes business users, development, QA and operations staff on business prioritization issues using agile practices such as Scrum product backlog planning or XP’s user stories.
Develop & Test: It mobilizes business users, development, QA and operations staff across the development life cycle using agile techniques such as Scrum iterations, XP’s test-driven development (TDD) and behavior-driven development (BDD).
Release & Deploy: It leverages continuous integration practices to build and test releases prior to deploying to production.
Monitor & Improve: It’s outsourced to keep the service development team on its core business: developing software.
The CD infrastructure combines the followings services:
AWS CodePipeline and CodeDeploy to set up the delivery pipeline.
AWS infrastructure as code including CloudFormation, OpsWorks and Elastic Beanstalk to generate the CD infrastructure.
Summing It Up
The DevOps business benefits are real, yet, it’s young and is still at the early stage of the technology maturity model. The expertise and engineering methodologies needed to implement it as what it is—a technology-enabled application delivery environment—are lacking.
DevOps design patterns such as the software organization factory fill the gaps of today’s transformation practices:
They provide the architect team and the executive team a neutral language to discuss the business, technological and technical challenges and solutions.
By pointing out the agile practices to leverage, they simplify and accelerate the design and deployment of the operational model.
By pointing out the AWS services and features to leverage they facilitate the design and implementation of the CD infrastructure.
This article will provide an example delivery pipeline used through continuous integration, as well as the DevOps tools involved in the automation of building, testing and deploying code through your software development life cycle (SDLC).
Background
These days, it’s almost impossible to go anywhere without hearing about DevOps. In short, DevOps is the effort to improve communication and synergies between development organizations and operations organizations—hence the term, “DevOps.” The main focus is to improve the efficiency, quality and speed to market within the software development world.
Today, you also may hear and see a lot about continuous integration and continuous delivery/deployment. There are differences between the two for sure, but for simplicity’s sake, this blog focuses on breaking down the different stages of the process. This way you can get the basics and hit the ground running with a delivery pipeline in your organization. The delivery pipeline can be broken down into a few major buckets of work, or stages, as mentioned below.
In my opinion—and, again, trying to keep things simple for now—the stages can be broken down as follows:
Source Code Control (Management)
Build Automation
Unit Test Automation (could also include Integration Testing here as well)
Deployment Automation
Monitoring – not included in this discussion, and can be added at any time
I have outlined the progression of these stages in Figure 1 below. Let’s review each of the stages in a little more detail.
Source code management, or source code control, certainly is not a new topic. This has been around for decades and has evolved over time. In a nutshell, an organization stores its code in a source code control system or repository so it can be tracked, maintained, versioned and audited. Developers should not store the code on their laptops or virtual machines and trust that will suffice for managing the code.
Possible Tools
Git is probably the most widely used SCM system out there. It is an open-source system.
Subversion (SVN) has been around for quite some time. It is also an open-source system and is heavily used in many organizations, but there has been more of a push toward Git.
Once a source code management system is in place and actively being used by a development team, the team will need to be able to compile and build their code. This is probably the first step in the whole chain of continuous integration events—it’s what gets the ball rolling. The code needs to build cleanly before an organization can even think about deploying out to its environments for testing and production.
Possible Tools
Gradle is an open-source build automation system used by top companies including Netflix, Google and LinkedIn.
Maven is another open-source build automation system.
Gradle and Maven comparison can be found here and here.
Unit Test Automation
Background
Developers unit test their code to ensure the functionality they are building works as expected. In an ideal world, the development team should be saving these unit tests so they can be reused and put into a regression test bed.
Possible Tools
There are multiple tools out there for helping developers unit test their code. Many of these tools are open-source and can be used freely.
JUnit is an open-source unit test framework pretty widely used.
CA DevTest allows for the automation of unit testing, as well as a few other bells and whistles, such as service virtualization.
Deployment Automation
Background
For the last stage in the process, delivery teams need to deploy their code/applications out to various test environments and, of course, production. To reduce errors and overhead in the deployment process, while increasing speed to market, this step can be automated through a variety of tools and methods.
Possible Tools
IBM Urbancode uDeploy allows users to model a process and orchestrate their deployment. This process then can be repeated across all an organization’s environments, and tweaked for each environment as needed.
Ansible is an open-source IT automation tool. It can be used for everything from configuration management to product installation to application deployments.This tool is rapidly gaining acceptance and momentum in the DevOps community.
Conclusion
This was simply a quick overview for companies to get started down the path of continuous integration and the DevOps world. Figure 2 is an example of what the whole flow looks like from committing the code to repo to deploying the code to an environment, to help put things into perspective. It’s easy to see how the tools interact with each other on the journey to production.
Figure 2 – Continuous Integration Process Flow
About the Author / Seth Gagnon
Seth Gagnon is a Middleware / Infrastructure / DevOps architect with more than 12 years’ experience in the field. Please visit sethgagnon.com for additional related articles. Connect with him on LinkedIn and Twitter.
In “The Phoenix Project,” a 2013 novel about an IT manager rescuing a late, over-budget IT initiative, Gene Kim demonstrates how collaboration between IT operations, development and business users can impact the business. In the era of digital transformation, the need to bring development and operations together—in what is often called DevOps—is more important than ever, as IT is now more than an enabler of the business. IT is the business.
What’s at stake today is more than just a project or someone’s position or even the viability of an in-house department versus outsourcing the function. Instead, it may be the very existence of the company. According to Gartner, 25 percent of companies will lose business due to digital incompetence by 2017 while IDC predicts that 75 percent of the S&P 500 will be replaced by 2027. To avoid the fate of becoming an industry has-been, today’s market leaders need to ensure they won’t be overtaken by more agile competitors that respond faster to a rapidly changing marketplace.
Automate Everything
Today’s businesses need to transform into digital enterprises to thrive in today’s digital economy, and this transformation needs to be far more profound than simply investing in the latest digital technology. Because the pace of the digital economy requires companies to roll out new services faster than ever before, businesses are challenged to move from idea to revenue as soon as possible. This need for speed leads to a desire—really a requirement—for agile applications.
For many of today’s enterprises, this means they have to stop thinking that a successful future depends solely on a traditional strategy of product innovation, using data analytics to better understand and serve their customers or building a higher quality, more profitable product portfolio. Instead, like the mythical phoenix, their survival may well depend on a complete rebirth around making the practices of agile application delivery and DevOps a core competency.
DevOps teams need the ability to efficiently develop, test and deploy new releases faster, with high quality and stability. Using automation technologies to design-in this speed, quality and scalability within the development life cycle is critical. The continuous integration and delivery processes at the core of DevOps and agile practices are enabled by many new automation tools and capabilities. Each of these tools typically targets discrete elements or steps in the entire process and DevOps teams use them as such, finding the right tool for the job. In addition to faster and higher-quality development throughput, building automation into the code of the application ensures better application services delivery in the production environment.
The good news is that 60 percent to 70 percent of these enterprise application services or jobs within most organizations are micro-batch, batch or event-driven—and can be managed with a comprehensive job scheduling solution. By automating jobs as part of the application code and subjecting the job artifact to all the other automation processes, companies can substantially reduce time and effort during and after the application delivery process.
According to Enterprise Management Associates, companies with workload automation reduce application outage time by about 70 hours annually, raise service level agreement standards, free 41 percent more IT staff for strategic projects, and defer or eliminate an average of 50 percent of IT staffing requirements.
Traditional vs. Innovative IT
Too many companies intent on being agile and innovative believe that a successful DevOps organization must be free of the trappings of rigorous change management. Business innovation and increased employee productivity, however, can’t exist without a stable IT foundation. The goal is to have a balance of both strategies in an organization. Gartner refers to this as Bimodal IT, which has two modes: “Mode 1 is a traditional, plan-driven approach that emphasizes safety and accuracy in the pursuit of reliability. Mode 2 is iterative, being largely outcome-focused and emphasizing agility and ‘just enough’ with respect to areas such as planning and process.”
By committing to balancing the two strategies, organizations can increase agility without sacrificing stability. More importantly, companies can increase employee satisfaction by different teams successfully doing the jobs they are paid to do, whether it’s to drive innovation or keep the lights on. It’s worth noting that Gartner believes many elements in DevOps apply to both modes and that by 2020, at least 80 percent of practices identified today with DevOps and Mode 2 will be adopted by traditional Mode 1 groups.
While thinking of your company as a phoenix may be essential for survival, the benefits go way beyond that. Just like the revitalized bird, your company can once again be youthful and vibrant, capable of soaring to new heights. This is a strategic goal worth investing in.
About the Author / Gur Steif
Gur Steif is the president of the Workload Automation product line for BMC Software. Under Steif’s leadership, the Workload Automation business has more than doubled in size. Previously, Steif was the general manager and vice president for both the Workload Automation product line and the Performance & Availability business line. During this time, he grew the Workload Automation business at triple the market growth rate while also delivering a double-digit increase in customer satisfaction for Performance & Availability. In his 25-year career in technology, Steif has held leadership roles in strategy, product management, R&D, sales, marketing and M&A in IT operations management, application management and workload automation. Connect with him on LinkedIn/Twitter.
Recent IT disasters suggest gaps in hardware testing, backup system testing and inadequate disaster recovery plans.
Although the summer often is the most profitable season for most airlines, this past summer wasn’t great for several of the largest carriers. In July, Southwest Airlines suffered a 12-hour IT outage, triggered by a router malfunction that quickly took down many of its core operating systems like a house of cards. Worse, backup systems failed to deploy. Flight delays and cancellations continued for a few days.
A similar disaster unfolded at Delta in August, when a power control module at the airline’s central data center malfunctioned, creating a surge to the transformer and loss of power. Delta’s systems didn’t switch over to backups as planned, causing scheduling mayhem as agents couldn’t access key systems. For both companies, the outages resulted in tens of millions of dollars in lost revenues.
Sure, both of these outages affected businesses running more traditional IT infrastructure, but even if your company is running in the cloud, your business isn’t immune from a damaging outage. In September, Microsoft Azure experienced two outages lasting several hours each, one in Europe and the other affecting U.S. customers across several key services, including SQL Database, Virtual Machines, Service Bus and Visual Studio team services.
As software developers and testers, we often focus narrowly on the application and the server it is running on—but there’s a lot more to consider. Hardware and components, backup systems and third-party providers all come into play when testing applications for stability and reliability. Of course, there’s no way to prevent all bad things from happening but there are some steps you can take to cover the bases more thoroughly.
Look Beyond the Software
If there were three main lessons learned from these recent IT disasters, they would be:
Assess risk of the whole environment. The quality of your software is only as good as the components it runs on—the hardware, network, power, storage, and other third-party applications are often required. Understand the full picture and all of the dependencies. Categorize applications based upon business criticality and failure rate to focus attention on the pieces that really matter.
Backup systems and processes are not “set and forget.” Spend more time documenting them, testing them and training staff. Make sure that the processes are up to date as IT adds new components and vendors to the system or if the company goes through a merger or acquisition.
Invest according to your risk profile. If your company can’t operate for more than a few minutes without access to certain systems, plan on budgeting for warm or hot failover systems which enable near 100 percent uptime with always-on, parallel systems continually synchronized and ready to go. It’s too expensive to enable this failover capability for your entire environment—you will need to be picky. Some IT assets, such as an airline’s scheduling and reservation systems, deserve top priority and resources, while other assets such as the HR system, can do with less. IT directors also need to frequently investigate the need for upgrading old, unstable infrastructure technologies. Most companies also don’t spend enough time monitoring and testing backup and recovery systems.
To minimize the risk of a major IT outage bringing down your business and alienating customers, the recipe is as follows: Analyze the points of failure, determine what is reparable and what is not easily fixed (for example, installing software usually can be done much more quickly than building a new server environment) and, finally, install and test failover systems and processes that can maintain basic operations with minimal customer disruption.
Delta & Southwest: A Quick Analysis
In both cases, Delta and Southwest had backup systems that did not deploy properly—or, at least, in a timely fashion. The first point of action would include conducting thorough, regular testing of those backup systems and locations and upgrading capabilities if necessary. Second, since no disaster recovery system is foolproof, create manual workarounds for critical system outages. That would have helped tremendously so that customers weren’t completely stranded. Third, create a fully redundant second data center that switches over automatically when the primary data center goes down. Naturally, this is the most expensive option, but depending upon the outcome of the risk analysis, it may be worth the money.
With Southwest, a router failure caused a cascading effect that brought down the network altogether. It’s not clear if a secondary network was available for failover, although it appears the answer is no. Again, better, more frequent monitoring could have isolated the router problem and alerted IT to fix or replace the hardware before disaster struck.
Any company with technology from many different vendors should work closely with vendors and partners whenever possible to mitigate possible risks. Vendors often are able to provide insight into how they test their own systems and can even share test cases to assist in validating the end-to-end system. Keep in mind that aging technology is not the only problem—sometimes new technology brings untenable risks. A recent example is the Samsung Galaxy Note 7 and its overheating battery problem and recall. As well, it’s smart to understand how the local utility handles its outages and what if any help it provides to customers in the event of service disruption.
Planning Ahead
Large companies with complex IT infrastructures should take a holistic view of critical system availability, considering all of the possible ripple effects from a single point of failure. Today’s systems and networks are increasingly interconnected and interdependent. For both airlines, the effect on customers and employees lasted far beyond the actual outage, as managers struggled to get agents and planes reorganized and rescheduled. Take that risk into any spending calculations for the disaster recovery plan. Companies that have gone through mergers and acquisitions have additional complexities from IT integrations—or lack thereof. Failover strategies should ideally be incorporated upfront as part of any acquisition plans.
Comprehensive testing strategies are paramount, although sometimes overlooked, in preventing outages. Request information from core vendors, such as the networking equipment provider, on testing processes and ask to see their test cases. Though the vendor should be responsible for ensuring thorough testing of their own products, the customer is always the one to suffer the costs of a downed system. Development and testing teams should get clear on how external systems and providers handle down time and equipment failure. Strive to align the company’s disaster recovery processes with those of partners.
Finally, for companies running mission-critical systems in the cloud, there are ways to mitigate the effects of an outage. Multi-cloud management tools such as CliQr (now owned by Cisco) allow IT to quickly redeploy workloads from one cloud service to another whenever needed. Containers are also helping in this regard due to the inherent portability benefits of the technology. Companies are using containers to move applications and workloads from region to region within one cloud provider; from cloud to cloud, such as from AWS to Azure; and even from the cloud to an on-premise data center.
There are many causes of IT outages today, and as IT environments become more multifaceted, supporting more technologies, data and applications all the time, those risks grow. We don’t have an answer to the problem, but there is one strategy: leave no stone unturned. Involving testers and developers in outage prevention and disaster recovery planning is a wise move, indeed.
About the Author / Kevin Dunne
Kevin Dunne is vice president of strategy and business development at QASymphony. Prior to joining QASymphony, Dunne was a business technology analyst at Deloitte. He can be found online at Twitter and LinkedIn.
“Who am I?” It is perhaps the most basic question of consciousness. The answer could be subdivided by aspects of my life story … my career, my family, my beliefs, my education … the list could continue. But since I am unique, the challenge in this digital age is for me to somehow remain unique, and whatever descriptions are used to catalog who I am, I must remain at the center of them all. If my identity is stolen, how does anyone know, that I am who I say I am?
This question is fundamental to every interaction I have with the digital world. And unfortunately, every system I interact with looks for me to redefine who I am, over and over again for each of them. I create a Facebook profile, a Twitter profile, a LinkedIn profile … and the list goes on again. But yet another weakness of our digital age is that there is no cross-reference between these systems to ensure I am who I say I am. This phenomenon remains true within DevOps tooling as well.
The Career Identity Conundrum
Let’s focus on only one aspect of who I am, my career. Who my current employer is dictates what electronic business systems and networks I should have access to. Switching jobs to another employer changes—or should change—the access I have. But getting more specific, what my role is at my current employer should also have an impact on what tools I have access to, and what I can do within them. And as my role changes, my abilities within a tool should change, as well. If I move from practitioner, to manager, to director, my abilities within a given tool should probably diminish, holding practitioners accountable for actions rather than superseding their responsibility by doing things myself.
Then there is the area of focus I work on. Business segmentation differentiates my work from product or service “A” in contrast with product or service “B.” My role might be an application programmer, or database administrator, but then “who” I focus on within a company is yet another level of differentiation. The business may prefer that teams of people focused on product “A” are restricted from making substantive changes on product “B.” So two different database administrators need two different levels of access within the same tooling, segregated by line of business or product line to accomplish it.
The Career Consistency Conundrum
The problem with my corporate identity in the digital age is that it is fluid, like life. In the course of my career I will change employers at one time or another. But beyond this, I will change roles, and/or areas of focus within the same employer over time. I may be assigned to work a “special project” in addition to my normal duties during my time at a given employer. And where DevOps relies upon several types of tooling to facilitate the continuum, this fluidity in my career makes it difficult to keep up with, and can drive administrative costs quite high.
For example, if I am an application programmer (normally), I will then have certain abilities within the build, deploy and release tools—likely high ability in build, moderate in deploy and low in release. If I am temporarily assigned as a release manager for a different product/service (in addition to my normal duties), it should radically alter my abilities within the DevOps tools, at least for a short time. But when I return to normal duties, the only mechanism that gets my abilities reset to what they should be is a strong process control system. The tools do nothing to facilitate this (no reminders, no examination of HR systems for roles, no look at each other to see what I am supposed to be).
An Identity Architecture
It occurs to me that the constants in resolving these kinds of problems are time and who I am. If I could hold an identification construct that identifies myself in career terms—which employer, which role, etc. and for how long—that identity construct could be employed into every system that accepted it. In effect, it would be the ultimate abstraction of user management, transferring the burden from companies and HR departments to the individuals who know, and need to know, the end user. While companies would still need to interact with my personal identity construct for verification, assent and record keeping, having all electronic systems accept my ID construct takes all of the work out of programming those interfaces, makes permissions and roles common, and keeps up with life’s fluidity.
In addition, who I am becomes the same in every system that accepts this construct. I cannot define myself as a release manager in one tool and a test manager in another and an application programmer in yet another to give myself god-like abilities in the DevOps tooling at my company. Each tool would examine the same identity construct I hold and receive from it, my normal duties, roles, permissions, etc. I would have a consistent profile across corporate tooling that would reflect my latest roles and use dates (beginning and end) to validate me against their own records.
An Identity Lockbox
The trick is, how do I keep this identity construct from being stolen? I believe this is where biometrics begins to play a key role. Fingerprint technology is commonplace now, even in cell phones, which are usually chained to us like an additional finger or toe. We rarely go anywhere without them. And when carrying them, we use features such as Touch ID to replace the manual keying-in of passwords. If my identity construct was paired between information likely I alone would have and biometrics tuned only to me (retinal scans, fingerprints, etc.), the pairing could unlock the construct for either use or revision.
Stealing the construct does little good, as without the pairing biometrics, it could not be used or revised. The applications for an identity construct like this are limitless—everything from securing the right to vote, work and collect social security to validating credit card purchases and opening up new social media accounts. If Facebook employed (or accepted) this kind of construct, it would be more difficult to create spoof accounts of me, as the unbeatable pairing could decipher who really owns a photo, or document, etc. And looking forward, a standardized identity construct such as this could interact with the emerging world of IoT (internet of Things), where devices could know a lot about me, and know it was truly me.
Impacts to the Bottom Line
I am not suggesting some wild end-of-the-world scenario where computers control our every move. We already have that in some form or another (see banks – ha ha). But I am suggesting that in the interest of reclaiming digital privacy we should be moving the ownership of identity constructs back to the people who own them: us. We should each own our own singular construct, keep it up to date and as filled out or complete as we want to, sharing only what we want to. The chief benefit of this would be an increased difficulty in stealing them from us. We can all skip the 666 tattoos, and instead try to reclaim a moniker of privacy by holding our own digital identities ourselves. Using them like a master key that will open systems we “should” have access to and abilities we “need” to have to perform the work assigned to us.
The upside to corporations is increased security that requires less on their part to maintain. A single set of records with start and end dates validates IDs used or attempting access. The upside to the software industry as a whole is a common interface for logging into an application with discreet identification. It removes the doubt about “borrowing” an ID if unlocking access requires a biometric pair as well. It would allow every application to abstract user management and make it common. Roles would be abstracted. Permissions could remain unique to each application based on their interpretation of the role.
Moving to a self-maintained identity construct could be taken further for benefits in other areas of our lives. While we have discussed some of how the career portions of our identity might be subdivided and cataloged, imagine for a moment the benefits of having our heath characteristics attached to the same kind of identity construct. Family records, memberships and associations … pick your classification of how a person is described for who they are. Having a way to retain ownership of those things in the digital age is key to keeping our unique position in the world unique. Losing control of those things becomes catastrophic. Where DevOps benefits from a construct such as this is but the tip of an iceberg compared to the benefits society reaps—or at least, our digital society could reap if we pursued it.
To continue the conversation, feel free to contact me.
Rewind 20 years ago when we lived in a business world where big companies with unlimited resources beat out smaller, more strapped companies. The digital era—also known as the “age of the customer”—is changing all that. Where big once beat small, fast now beats slow. The DevOps approaches of startups often make them more adept at continuously delivering the innovations and conveniences that earn customer raves. Bigger, established companies often feel disadvantaged‚ shackled by an outdated culture and inflexible processes that haven’t kept up with the pace of change.
Gartner coined the term “bimodal IT” as a way for older IT establishments to keep pace with agile startups. In the bimodal model, IT teams are separated into two distinct styles of work—one focused on supporting predictable, well-understood workloads with legacy systems and another that is more fast-moving, experimental and adaptable in meeting users’ constantly evolving demands and flourishing digital expectations.
But the bimodal concept is not well-aligned to the spirit of innovation underlying modern application development and delivery. This is especially true for bigger organizations using mainframes. Here’s why.
Successful DevOps Depends on Mainframe Agility
The goal of DevOps is to roll out high-performing software and updates, ahead of the competition and with maximum speed, ease and cost-efficiency. In a DevOps model, software delivery can only happen as quickly as the slowest team. Modern web and mobile applications tend to span multiple platforms in their end-to-end transaction—from front-end web servers (systems of engagement) all the way back to mainframes (systems of record). IT places a lot of emphasis on systems of engagement, often taking back-end systems of record for granted. The reality is, the value in systems of engagement cannot be realized without the systems of record—making it essentially all one system. Developers supporting modern applications must be able to maneuver across and between all platforms, especially within mainframe code and data.
Let’s look at a tactical example of this. According to industry research, 80 percent of the world’s corporate data—including customer data—continues to reside on mainframes. When a mobile web application is used to make a bank deposit, or purchase a product, for example, the transactional component of those applications typically is executed on back-end mainframes.
This means mainframe applications are crucial to development efforts. What happens when mainframe developers and testers are locked into waterfall processes or don’t have the modern tools necessary to understand application logic or access mainframe data for testing? Frustration mounts and the overall innovative effort is unnecessarily stalled.
Stability and Agility are not Mutually Exclusive
The longstanding myth that stability and agile cannot co-exist is simply untrue. Unfortunately, it’s at the core of negative perceptions surrounding the mainframe today—people think the mainframe environment is too slow and cumbersome to support agile development. Development teams may think if they have mainframes, the only approach it can support is waterfall.
The reality is, though, no customer will accept poor availability and performance in exchange for new features delivered quickly. People with hands-on experience painfully understand that slow, waterfall delivery methods produce far greater quality issues and missed expectations, resulting in competitive risk. Additionally, large enterprises taking the “slow and stable” route give startups a head start in beating them to market, with offerings that customers want now.
Alternatively, Agile process and DevOps tool chains enable scrum teams to test quality in terms of usability, function and stability early and often in the development cycle—when the time to remediate is the fastest and the cost the lowest. Agile process and DevOps tool chains build higher quality applications and enable continuous improvement in quality over time.
Re-Platforming is not Necessarily the Answer
Some DevOps proponents believe moving off the mainframe—to a supposedly more modern, commodity server-based distributed architecture—is the best way to circumvent perceived challenges to nimbleness and agility. But this is almost never the best answer. The mainframe retains its distinction as being the most powerful computing platform on the planet. It remains deeply embedded in large organizations, setting the standard for reliability, security and availability for mission-critical data and transactions. The new IBM z13 mainframe, for example, can handle the load of 100 Cyber Mondays every day, 365 days a year. Currently, mainframes process 30 billion business transactions per day. The post-modern mainframe is an engineering marvel unequaled as a system-of-record, business compute platform.
The mainframe also offers cost advantages. Studies have shown that the mainframe is more cost-effective than commodity servers in handling the massive computing volume increases instigated by mobile applications. This is because decreases in commodity server pricing have not kept pace with the increase in computing volumes. In addition, leading vendors have worked together to deliver intelligent cost management on the mainframe, enabling workloads to be distributed in a way that keeps mainframe licensing costs (MLCs) down.
In our view, “biomodal” advocates for the exact problem that must be solved. We think the answer is not replacing the mainframe, although if mainframe development is left unaddressed, this can present obstacles to DevOps. Rather, the answer is to evolve the mainframe to keep up with the pace of DevOps. There are mounting examples from large, high-performing organizations that it is possible to apply modern practices to the fullest extent, including mainframe applications. These organizations do this by giving developers the tools they need to work on mainframe code and data just as they would with any other programming language on any other platform, while meeting the escalating quality expectations of the digital age. For example:
Mainstreaming the Mainframe Environment – Mainframe users need to move away from the antiquated “green screen” mainframe environment, adopting state-of-the-art interfaces and tools. This is the first step enabling developers to work more fluidly on the mainframe, in an environment that’s much more familiar and comfortable to them. Organizations that fail in this adoption will have a very difficult time enabling the next generation of development talent to leverage invaluable and irreplaceable mainframe code and data. This will place a huge hindrance on DevOps success.
Leveraging Tools from the Java Development World – One example is visualization capabilities, which enable developers to quickly and easily identify critical interrelationships between apps and data. This helps them to sidestep unintended, undesirable impacts as they modify applications, increasing confidence and mitigating unnecessary delays.
Make it Easier to Identify Mainframe Code-Level Issues – Developers need tools that enable them to identify problems with mainframe code; for example, when it is the source of a widespread user performance issue, or when it is consuming too many resources and driving up costs unnecessarily.
Conclusion
In our view, “bimodal” is not the solution, but a broken state of disadvantaged affairs for too many organizations. From a competitive vantage point, bimodal will only hurt DevOps teams and hold back the innovative efforts of the organizations they support.
The bimodal IT model is a road to inevitable failure for those enterprises seeking success in the digital economy. This is especially true for organizations that rely on mainframes. A smartly leveraged mainframe is an extremely valuable competitive asset that needs to be removed from its silo, evolved and smoothly integrated into modern, heterogeneous development environments.
It is possible for big, established companies to adopt DevOps, become agile, and move as fast as startups. But the ability to think and act like a startup requires some major fundamental changes, particularly at the level of software development. The good news is, the legacy technologies that are often perceived as impediments can actually be a tremendous competitive advantage, if they are harnessed and evolved in the right manner.
About the Author / Christopher O’Malley
Christopher O’Malley is CEO of Compuware. He has nearly 30 years of IT experience with past positions including CEO of VelociData, CEO of Nimsoft, EVP of CA’s Cloud Products & Solutions and EVP/GM of CA’s Mainframe business unit, where he led the successful transformation of that division. Connect with him on LinkedIn and Twitter.
Scrum is, by far, the most popular way development teams are organized and work. In fact, well over 70 percent of Agile teams around the world use Scrum. But for those unfamiliar with the structure, Scrum may look like a strange collection of rituals and oddly named designations, such as the “Scrum Master.”
However, behind that collection of ceremonies, roles and rituals, Scrum is a huge opportunity for Operations teams to streamline development and delivery, improve productivity and create a strong sense of team within the group. Not only does Scrum encourage DevOps, but because it is empirical, risk and visibility are first-class citizens.
Scrum Basics, and More
For those wondering how to get started, here are eight things that every operations professional needs to know about Scrum:
Scrum is not focused solely on developing software; it’s about delivering software. The Agile Manifesto describes it as “working software” and Scrum describes it as being “Done,” but ultimately, Agile encourages teams to deliver working software to production frequently. With that in mind, it becomes clear that Scrum is not just about software development, but includes everyone involved in delivering working software to customers.
Definition of “Done” is a DevOps artifact. “Done” describes what it means to be finished with a product backlog item (PBI). Ideally, it describes all the steps necessary to release the PBI to production, or highlights the gap between what the Scrum team thinks Done is and Operations’ idea of what it means to be complete with a piece of work. Operations professionals need to get involved in the formation of the definition of Done (DoD) and work with the team to both improve its reach and validate its use.
The increment should be integrated and deployable. In Scum-speak, the increment is the working software the team produces during a Sprint (a set time frame, from start to Done). If multiple Scrum teams are working on the same product, then it is crucial that the increment be integrated across those Scrum teams. And, when I say integrated, I really mean integrated—as in, tested with processes and practices that are consistent with delivery to production. Thus, the increment is “ready to go.” By regularly integrating the increment, many of the problems that historically have plagued Operations are removed, as teams are, by design, forced into falling in sync much earlier in the life cycle.
The Sprint Review is not a phase gate to production. When moving from waterfall to Scrum, many organizations confuse the Sprint Review with a phase gate to move to production. Rather than being a phase gate for production—or the next phase of the process—the Sprint Review is a mechanism for the team and stakeholders to inspect and adapt at the boundary of the sprint. Ideally, the Sprint Review is executed on software that is being used by real users, in production. This increases the value of the review, as the team and stakeholders get to inspect how real people are using the software.
Continuous integration and continuous delivery automate Done. At the heart of Scrum is the idea that teams deliver working software in small batches to better understand the risks, requirements and solution. Adding delivery to continuous integration (CI) ensures they not only are integrating the software, but also formalizing the deployment processes to ensure there are no surprises. This is a great place for Operations to get involved and help ensure that aspect of Done is consistent with how it views delivering software. This might mean additional steps—or at the very least—a better understanding of operational versus development systems.
Scrum is an empirical process which helps reduce risk. The idea of delivering small chunks of functionality to support learning is at the heart of Scrum. Ultimately, Operations cares about risk. By delivering software in small batches, it is possible to surface risks early and mitigate them. By frequently delivering software, Scrum teams demonstrate they understand the processes necessary to deliver software, thus removing friction from traditional release processes. However, all of these benefits are undermined if the team does not use “production ready” deployment processes or include risk as part of their prioritization of what to work on next.
The Scrum Masters are Operations’ friends. The role of the Scrum Master is to both protect the team from outside distractions and help them to identify and remove impediments. To deliver Done, the team needs to work effectively with Operations and thus, the Scrum Master often will be the person who helps build those relationships. By working closely with the Scrum Master, Operations professionals will gain better insight into what the Scrum team will need from them and how they can deliver Done software.
Helping the Product Owner understand risk. The Product Owner (PO) manages the backlog, which is the place where all work exists. This means the PO is prioritizing the backlog and deciding on what should be done next. Normally, this is accomplished by looking at business value and negotiating with the Development team to determine what can be most effectively completed next. Operations has a great understanding of risk and, by working with the product owner, can add that context to the backlog, allowing the PO to make informed decisions and about value and risk.
Ultimately, Scrum provides a great framework to build bridges between Operations and Development and can act as a catalyst to drive fundamental improvements to the way teams collaborate and deliver business value. However, both sides need to stop hiding behind the “mystery of agile” to avoid each other and embrace the desire to break down things into small chunks, incrementally manage risk and get to repeatable Done software as fast as possible.
About the Author / Dave West
Dave West is the product owner at scrum.org. He is a frequent keynote speaker and is a widely published author of articles, along with his acclaimed book, “Head First Object-Oriented Analysis and Design.” He led the development of the Rational Unified Process (RUP) and then worked with Ivar Jacobson running the North American business for IJI. Then managed the software delivery practice at Forrester research where he was VP and research director. Prior to joining Scrum.org he was Chief Product Officer at Tasktop where he was responsible for product management, engineering and architecture. Follow him on Twitter.
By now, surely everyone has seen cooking competition shows such as “Hell’s Kitchen” and “Top Chef,” in which people compete to create and serve the best dishes as quickly as possible. These hit TV programs are like the DevOps world, which also consists of people creating and releasing the best software as fast as they can with whatever they are given. In both instances, the best use of random ingredients brings rewards and failure brings disappointment.
DevOps is becoming evermore common as companies in all industries work to unify their developers and other IT staff while automating as much of the process as possible, creating nimble, fast-moving teams. Think of it in the same way professional kitchens have cooks dedicated to specific stations, such as vegetables or fish, and are able to react swiftly to whatever orders are placed. By redesigning their in-house teams in this fashion, organizations can add speed, agility and resilience to critical functions.
But, often what’s forgotten in the high-pressure, rush-rush-rush world is the critical importance of security and regulations. Governance, risk management and compliance (GRC) always must form the base decisions; otherwise; the final products will either not be able to be used or, worse, be dangerous to the company, its customers and its shareholders. It’s really no different than cooks having to abide by health code regulations or risk sickening and possibly even killing their diners. Don’t think GRC is that serious? It wasn’t long ago that Dwolla was fined $100,000 by the Consumer Fraud Protection Bureau (CFPB) for lying to customers about data risks. That was the CFPB’s first fine and you can bet more are sure to follow.
Of course, a key difference is that in the business world, regulations and compliance standards constantly are being created or changed. That makes DevOps even more challenging, but that challenge cannot mean ignoring GRC. In fact, GRC should be seen as a key DevOps enabler, not an obstacle. An organization’s legal and compliance team can take advantage of highly agile, forward-moving DevOps teams to design and implement changes that satisfy risk and control points (resulting in successful audits and avoiding fines) while, at the same time, staying abreast or even beating the competition.
The key here is to ground DevOps in GRC requirements so they become second nature and security measures are incorporated effectively into all DevOps processes, even automated ones. It’s crucial these measures are seen as integral to the processes and not simply layered on top as an afterthought, just as you wouldn’t try to get raw chicken juice off of lettuce after you’ve made a salad.
Tactically speaking, how is this accomplished? DevOps teams need awareness and training. They need to learn the rules and laws specifically governing their particular industry, and how to implement them. And, companies need to have internal audits and other review policies in place to ensure GRC is enforced.
For all of this to happen, however, a very important role must orchestrate it: management. Just as every kitchen depends upon a head chef to ensure not only that each station is obeying health codes (i.e. the law) but also that overall the meal is being prepared and delivered properly, company management must assume its authoritative role in setting up and defining the organization’s internal communication. This means breaking down silos and coordinating a regular flow of information from Compliance to DevOps team and back again.
If this is done properly, DevOps can enable an innovational Nirvana, with companies boosting speed while maintaining security, creating the perfect consumer experience.
About the Author / Robert Hawk
Robert Hawk serves as Information Security Expert at xMatters, Inc. He has extensive experience in information systems security, computer security, cyber security, information assurance and governance, risk, and compliance (GRC) management. He specializes in frameworks and standards from: ISO/IEC, NIST, IEEE, IETF, ITU-T, Common Criteria, AMI-SEC, NERC, CIS, DoD, ANSI, PCI and ISECOM. Robert is a lifelong researcher, innovator and instructor. Connect with him on LinkedIn.