The IT pro gig usually includes a basic headwind. They’re part of a cost center, and budget for resources and staff are limited by design unless they’re managing a strategic initiative, which—at least for a while—is more generously funded.
Early adopters of DevOps, Agile or site reliability engineering (SRE) teams have been budget beneficiaries when tied to transformation projects—as has always been the case for high-visibility IT initiatives. Fortunately, DevOps has proved more helpful and stickier than some other “next big things,” and for some organizations it’s the engine behind growth and even actual digital transformation success.
IT pros—pragmatic as always—have taken note of DevOps potential, even while keeping lights on, systems up and reliably delivering the critical services businesses have depended on for decades. The question for IT leaders is, how can IT teams realize some of the benefits of DevOps, while operating inside traditional IT organizations?
As DevOps has finally slid off the hype peak and into the shed of useful tools, experience has distilled out a couple of conclusions. First, the tenets of DevOps—breaking down walls, shared feedback loops and learning to love automation—don’t only apply to new or cool apps. They’re even more helpful in traditional IT. For example, the rise of automation isn’t driven by a conspiracy to replace admins with robots, it’s a necessity to shrink the toil of manually wrangling all the new complexity of cloud and cloud-native tech on-premises.
Second, many of the ideas behind DevOps and resulting practices are a direct attempt to fix issues in IT that have frustrated IT teams for years. Ask any IT pro if they’d like to move all deployment to Monday through Thursday, 9 a.m. to 4 p.m., instead of midnight Saturday nights. That team will go nuts for CI/CD. Better, it turns out you don’t have to do all the DevOps things to get many of its advantages.
But even if your business still operates from the C-suite down, entraining quarterly budget spending, waterfall models and lots of walled silos, there’s an opportunity to adapt DevOps processes and culture to deliver progress on truly transformative initiatives to help grow the business. The trick is figuring out which DevOps practices are valuable and which ones are more useful to developers.
Luckily, focusing on just four can get your team most of the benefits without (much) disruption, and development teams are eager to share its growing body of new techniques and successes. We’re working in an interesting epoch of IT history, a potential democratization of DevOps for everyone, DevOps for the rest of ops.
Continuous Delivery
Until recently, some IT pros were skeptical, even suspicious that the development teams’ goal has been to automate them out of a career. Others felt developers were demanding everyone to learn how to program full-stack and invent a way to move fast and break things just as they can (more safely) do. A few admins have even had unpleasant encounters with some DevOps proponents when they showed interest. Nobody likes reflexive replies, such as “You use Oracle and DB2? Switch to MongoDB and come back later” to eager 101 questions, even if they’re just from a handful of community members. It’s especially unpleasant for developers and IT pros alike because that’s not how operations works. We share, we’re open, we’re helpful. We came to tech to help end users get more out of technology.
Instead, successful teams often start by identifying tools supporting fundamental operations goals, without the distraction of any particular technology or vendor, and seek buy-in from both developers and ops. Because if you’re in operations, you’re the team that must:
- Find a way to get software into reliable production.
- Maintain functional, useful governance.
- Maintain apps for years more than anyone thinks they’ll possibly be online.
Operations knows this because it has always been the last mile of delivery and deployment, it cares about availability and SLAs, and defines process for breakfast.
One of the first tools associated with DevOps is CI/CD pipelines, which are literally implementations of automated policy about how code goes from origination to production. But, overlooked in many DevOps blogs is the last step of CD because it’s not for the faint of heart. That’s unfortunate—operations is good at figuring out that last mile.
When operations teams embrace CI/CD/Continuous Deployment, they’re rewarded for investment in automation in the best possible way: less night and weekend work. As a result, the pipelines they build aren’t ephemeral experiments but production-grade CI/CD because it’s part of production itself. And as change velocity increases and error rates decrease, many teams refuse to go back to manual deployment unless they’ve exhausted all other options. Pipelines don’t replace IT pros, they improve work-life balance.
When the operations teams present developers with standardized, battle-tested nozzles through which changes get more quickly deployed with fewer errors, the reaction is usually excitement. It allows developers to mate their build, unit test, packaging and delivery head ends to the typically complex delivery/deployment operations pipeline. Best of all, ops teams tend to include monitoring, security and even cost governance into their processes, resulting in effective controls being automatically and reliably applied. Further, ops metrics create and enrich the feedback loops dev need to improve code over time based on real-world performance, not dev or test data. In environments like that, culture change is more organic based on shared goals, success and the flexibility to let each team do what they enjoy most.
Start Breaking Stuff Fast
Operations teams can have a little envy when it comes to breaking things. They’re technologists just like developers, and like most people attracted to technology, they like to learn by disassembling, recombining and otherwise hacking tech to learn the intricate details of how it works. But unlike a developer testing code, experimental configurations or other normal laptop environment one-offs, operations absolutely can’t experiment. IT is risk-averse by commandment, for good reasons.
“Get a lab!” sounds like an obvious suggestion, and it is—but it’s a great example of easier said than done. Because IT is a cost center, labs are often seen as duplication and waste because their value is harder to justify in a budget presentation. But with a little unconventional use of the monitoring data you already have, that’s not necessarily true. Operations regularly correlates change events with performance histograms in the ongoing search for improved reliability. Consider adding before and after metrics to changes tested in a lab environment first. You’ll likely find you can prove the more realistic the pre-deployment testing, the lower the error rate and higher the performance of that system.
But the mechanism behind improved quality isn’t the lab itself or data it produces. It’s the team’s experimentation—and some blood, sweat and tears shed—impossible in a production environment. IT also allows operations to invite developers to go on deployment and tuning ride-alongs. Developers can use whatever tools they prefer, make changes, probe and otherwise disassemble the lab stack as needed. And from a communications and culture perspective, this can be helpful.
Operations lab environments also allow IT pros to separate hype from reality on their own. They can install the latest shiny new tools for evaluation, try new methodology or even test significant platform architecture redesigns. In short, they can break whatever they want without affecting production and provide both feedback to developers and a place for dev to experiment with changes. Here again operations can be the lead, with new avenues for communication, cooperation and culture change.
Monitor Your Success
Most developers aren’t data driven, they’re functionally driven. They love data-informing design and are good with tools that wrangle it into something useful, but at the end of the day their job is to build code that does things. IT pros have monitoring in the DNA because detailed performance data collected over time is how they initially validate hunches but also break down conclusion bias. But while everyone seems to understand the value of metrics and data, it’s not usually considered a tool for DevOps communication.
As mentioned earlier, feedback loops are a fundamental principle of DevOps. Changes go into production, production generates data sent back to developers to guide future changes to further improve service quality. It’s a virtuous cycle, and both teams value it highly. The trick is in creating a language between dev and ops about what to collect, how it’s used and the stewardship of feedback over time. Step one is to sweep away preconceptions on both sides and start fresh.
A common approach is to bring a small—one-pizza small—team together and start with a blank whiteboard. The goal is to matrix the inventory of what’s already available with what’s helpful, then identify gaps for both teams. Development teams are hungry for the incredible richness of coverage operations can offer, while operations is eager to mine new instrumentation the development team relies on for troubleshooting and internal application performance monitoring (APM).
APM for hybrid applications is a great example of both teams learning how to create ensemble visibility into hybrid on-prem/cloud-native services. High performing operations teams ensure they add metrics dev teams need, and developers discover that the “impossible” information to debug pesky apps has been collected for years and is an API call away. Once again, a little conversation can lead to relief, and earned changes to hearts and minds.
Reward Success With Big Dreams
The final common experience of developers and IT pros both are never really done. Code can always be improved, and service delivery can always get better. But because we’re human, sometimes we get tired, stuck or bored, and it’s easy to take a break, especially after a big win. That’s natural. There are years of deferred work and technical debt lying around, and no time for perfectionism, especially in enterprise IT. This critical aspect of DevOps needs to remain a focus of any team adopting DevOps techniques—remembering DevOps is never done. It’s a culture of continuous, collaborative improvement and evolution. This shared understanding sustains the bonds between dev and ops.
When teams innovate, when they continue forging closer relationships and when they share improvements as the team culture changes, they start asking bigger questions. We’re conditioned in IT to accept that certain things can’t be achieved because, reasons. As a result, sometimes suggestions and questions that could finally resolve the team’s most bedeviling challenges might not always be made or asked.
Track your “impossible” ideas which drive progress, then promote successful ones while sharing details of why they worked. This in turn helps socialize the value of success resulting from openness and encourages great ideas that may have been sitting on the bench for a long time.
Again, you’re investing in more metrics and signals, but not limited to performance data. When you dream big, especially as you untangle the enormous expectations of digital transformation into operations changes, you’ll start to rely on new metrics. For example, business metrics tied to corporate objectives, behavior signals accurately reflecting CSAT, change and error rate data helping teams decide when to add new versus when to fix and new ways to assure leadership operations have everything under control. When managers are confident IT pros can safely transform gear, services and staff without interrupting the business, the most sought-after DevOps goal can come into focus: A new freedom to innovate at speed.
— Patrick Hubbard
