devopsy.kinsta.cloud

Author: Anders Wallgren

Make Metrics Matter: Measure What’s Important

In a data-driven society, organizations rely heavily on sets of metrics to evaluate how they’re doing on a particular task, initiative or resource allocation. Metrics are particularly important to DevOps teams, whose very function is tied to concepts like collaboration and continuous improvement.

But, while metrics can help drive efficiencies, they can also drive DevOps teams down the wrong path. Just because you’re measuring specific things doesn’t mean you’re measuring the right things. And if you add more metrics, it doesn’t necessarily mean you’re creating more value; you may just be spinning your wheels faster and getting nowhere.

The key is choosing the right metrics that drive the right outcomes for your organization. But how do you choose the right metrics? And how do you measure the true value you’re creating? Organizations that successfully navigate this challenge will not only build successful DevOps cultures but also improve their overall competitiveness in an increasingly demanding business climate.

Be Measurable – and Be Actionable

First, choose metrics that are objectively measurable. Eliminate individual subjectivity from your metrics palette. Rather than ask “How beautiful is my UI?” focus on measurable data such as, “Is the UI accessible to people with disabilities?” or “Did it drive more clicks?”

It is important to choose metrics that are actionable, not just “vanity metrics.” Back in the 1970s, McDonald’s used to update the signs outside its restaurants with actual numbers: “3 Billion Served;” “4 Billion Served.” Now the signs read, “Billions and Billions Served.” That is the ultimate vanity metric – it’s not easily auditable and not actionable. How many billions? Should McDonald’s be aiming for trillions? A good example of a DevOps vanity metric is server uptime. In fact, it’s actually a contraindicative metric: with cloud instances playing such an important role, you could make the case that low server uptimes are actually preferable.

Metrics should focus on inter-team and intra-team performance and outcomes. DevOps is all about cross-functional teams — development, test, operations and information security, among others — working together to deliver business value. So, rather than focus on individual metrics like “lines of code,” focus on outcomes like “cost of customer acquisition,” “new customer signups” or “overall revenue.” It may not always be obvious how to move the needle, but that is where experimentation comes in.

Avoid the Noise

Focus on as few metrics as possible and maintain a high signal-to-noise ratio. When everything is a priority, nothing is. If you’re presenting your metrics to the board of directors and you have 50 different graphs on the screen, you’re focusing more on how the presentation looks than on showing real, tactical information. Choose metrics that align with your current goals and be ruthless about ensuring those metrics are accurate, timely and focused on a strategic goal.

Expect goals – and the metrics you track – to evolve over time. You may want to shorten feature delivery time, but to do that, you may start with measuring things like build time, regression testing time, deployment time, etc. You will be impacting the feature delivery metric but doing it in a roundabout way. The key strategy is to pick an outcome you want to achieve. Once you knock down that problem, you can keep that metric around, but you’ll want to push it down. You can put an alert on it so if it ever crosses a threshold where you don’t want it to be, you can act on it.

Engineering organizations should focus on four to five metrics that they actively are attempting to improve at that time, so they can achieve those outcomes. As they achieve those outcomes, they can retire them or put those metrics in the background. Then, they can pick the next one, identify the next outcome to achieve and assign some new metrics around that.

Align Metrics With Your Priorities

Metrics can be situational. If you work in the network ops center, there are going to be some things you want to have up all the time. You’ll want to know the second a denial-of-service hits or the second error rates start going up. But for a software delivery pipeline, you can’t just throw out a bunch of metrics and hope the right ones will find their way to the forefront. You want to focus on metrics that align with your priorities today, next week, next month, this quarter or this year.

So, which metrics are most important for a DevOps organization to pursue? A series of metrics identified by the Google DevOps Research and Assessments (DORA) team has emerged as the industry standard to determine how successful a company is at DevOps. These measurements include deployment frequency (DF); mean lead time for changes (MLT); mean time to recover (MTTR) and change failure rate (CFR). These metrics show how well development teams deliver better software, faster, to their customers.

I would add a fifth metric: mean time to discover. From the time a problem is injected into a project, how long does it take for us to discover it? Did our organization discover it or did the customer? From there, we can strategize about the cheapest possible way we could’ve caught a particular bug ourselves. Could we have written a five-line unit test that takes 100 milliseconds to run that would have caught that? Or was it more complicated? Was the UI misaligned so that it was impossible to catch that automatically?

Different segments of the software delivery process will be concerned with different sets of metrics. While the Dev/CI practice focuses on development lead time, cycle time and work in progress/technical debt, the QA team pays closer attention to idle time, defects discovered and escaped, and the aforementioned mean time to discover. While the deployment group aims to improve deployment frequency and change success rate, operations gets jazzed from studying the cost and frequency of outages.

Metrics are useful tools for DevOps organizations. They can push teams to be more efficient and more accountable. But to truly optimize the value they’re getting out of people and particular initiatives, it’s important to choose metrics carefully. Recognize that metrics aren’t a one-size-fits-all mechanism, and that more metrics doesn’t always equate to better performance. Choosing the right metrics – and reevaluating them on a consistent basis – is the best path to long-term improvement.

April 30, 2021
DevOps World 2020: 5 Sessions to Put on Your Schedule

DevOps World has established a reputation for delivering compelling, high-quality sessions that challenge and inform its audiences year after year, conference after conference. DevOps World 2020 is continuing that pattern, offering what could be the best program yet.

The conference, to be held virtually this year Sept. 22-24, is rolling out more than 100 keynotes, workshops, panels, breakout sessions and “lightning talks.” Topics range from high-level themes such as promoting diversity in the workplace to on-the-ground technical discussions about automated testing in CI/CD pipelines.

If you have unlimited time and boundless energy, you can experience it all, without leaving your home office. Just sign up for the conference and join at the appointed time to ensure that you take advantage of the back-and-forth discussions each session will enable.

This year, you can do it all. Or, if you’re like most of us, you’ll probably want to cherry-pick the best of the best—the sessions that engage you the most and offer you the best perspective on issues that are important to you.

For those who are looking for a great DevOps World experience, here are five sessions that you won’t want to miss.

“Data-Driven DevOps: The Key to Improving Speed & Scale,” Kohsuke Kawaguchi, Launchable

What better place to start than with one of the most respected, most influential programmers in the history of computing? Kohsuke Kawaguchi, founder of Jenkins and former CTO at CloudBees, will deliver a session focusing on how development teams can be more successful.

Kawaguchi, who is now heading a new software testing startup called Launchable, will focus his talk on how a software delivery initiative can rise or fall based on the organization’s ability to use available data. Automation produces tons of data, but most of it gets thrown away. At the same time, leaders don’t have a good handle on how to make their delivery projects successful because they’re not mining the right insights from their data. Kawaguchi will draw on his own experiences and specific user stories to share how organizations can leverage their data to move at a faster speed and make developers feel great.

“Automated Testing as a Part of Pipelines,” Rosalind Radcliffe, IBM

Anybody who’s developed software for a long time knows just how important—and how big a headache—testing is. Without a dependable, effective, automated testing capability, teams end up spending way too much time documenting progress and re-doing work.

Automated testing is critical but organizations often struggle to do it when they have a large number of back end applications running on the mainframe. Rosalind Radcliff, IBM’s chief architect for DevOps, will discuss actual use cases for how unit testing and automated regression testing can be added into software pipelines, even for those COBOL and PL/I applications organizations have running. This session is great for web and application developers and system architect designers. Radcliff will drill into test automation approaches and go over CI/CD and DevOps for classic tools and tech stacks.

“Five Reasons Value Stream Thinking Is the Future of DevOps,” Steve Pereira, Visible

DevOps World attendees come to the conference not just to hear about what teams are doing today. They also want to know what the standards are going to be in the future, so they can help their organizations get a leg up on the competition.

One of those futuristic concepts that’s embedding itself into organizations’ DevOps strategies is “Value Stream Thinking.” The value stream, in and of itself, isn’t brand new. Corporate leaders starting with Henry Ford built processes around delivering value. But the emphasis on thinking, managing and mapping their DevOps strategies around the concept of a value stream is taking hold today at companies including Nationwide, Walmart, Barclays, BMW, Verizon, Liberty Mutual, AT&T, John Deere and Adidas.

Steve Pereira, a consultant who describes himself on Twitter as a “value stream whisperer,” will present real-world transformation stories and results achieved with this new form of management.

“Bad Code Kills: Five Essential Quality Gates You Need in Your CloudBees Core Pipeline,” Eric Mizell, OverOps

You can have the greatest ideas, the greatest team and the most well-designed software in the world. But if your code isn’t up to the required standard, your whole software delivery initiative is going to struggle.

Eric Mizell, vice president of solution engineering at development consulting firm OverOps, will discuss ways organizations can improve the quality of their code while continuing to deliver at a high velocity. The session will delve into common challenges and problems with current QA methods, then define and walk through each of the five quality gates teams can build into their pipelines: Dealing with static analysis, new errors, critical exceptions, resurface errors and unique error counts. Attendees can learn how to fill quality and testing gaps left by existing tools and processes and the open source resources available to build these gates into their own CI/CD tooling.

“Successfully Implementing DevSecOps at Scale At a Large Financial Services Organization,” Nagaraju Kakarla, Fidelity Investments

Many organizations with the size and scope of Fidelity Investments struggle to create a functional DevSecOps practice across the organization. They often get trapped into implementing waves of tools or end up doing it in a piecemeal fashion, leading to inefficiency and frustration.

Nagaraju Kakarla, senior director and chapter leader at Fidelity, will present a session detailing how Fidelity scaled its continuous delivery component and integrated security functions earlier in the delivery process. He will share some challenges he faced as an engineer and as a leader in implementing DevOps and SRE at various large organizations.

To take advantage of these and other sessions at DevOps World 2020, register today!

September 21, 2020
Software Development Management Drives Business Value

How much business value does your software group generate for your organization?

That’s the existential question an international corporation’s software leader poses, kicking off the 2017 book “The Business Value of Software.” It’s also a topic we’ve all likely wrestled with from time to time. It’s the essence of the value we personally bring to the business through our daily actions.

The problem is, value is hard to define and hard to measure. But, it’s easy to see how business value can seep away from software initiatives when software projects start to experience problems.

When production is delayed, and software releases don’t go out on time, sales will fall short of projections. When a release doesn’t satisfy customer needs in a specific market, negative user reviews will spike–and that could impact future sales. When market trends shift suddenly, and business unit leaders have no clear way to push the development team to add new features to the product, that’s a missed market opportunity.

With every misstep, you’ve lost a certain amount of business value.

Conceptually, every one of these situations is fixable. To an extent, some companies have adopted practices that can catch certain problems before they become unfixable. But, to a greater extent, the inability of software-driven companies to anticipate trends, flag problems, share information, iterate on solutions and learn from mistakes is costing them business.

In this blog series, we’ve discussed how software delivery management (SDM) presents a framework for organizations to change the way they think about software and streamline delivery processes. SDM seeks to connect all aspects of the software delivery process to not only become more efficient but also drive toward a common goal of delivering maximum business value.

The SDM model is built around four key pillars: ensuring everybody is working with common data; sharing insights and continuous learning across the organization; connecting processes to bring ideas to market; and enabling closer collaboration across software-related functions and teams to amplify value creation efforts. It emphasizes that software doesn’t have to go it alone–the whole organization is rallying around software and making it a top priority.

How does an interconnected, collaborative, information-driven software organization create business value using SDM? Here are a few ways.

Leaning on Analytics

Rather than just using discrete build-test-deploy processes, make sure management, developers, marketers and engineers are all incorporating key analytical measures in their iteration processes. These stakeholders shouldn’t go on hunches–they should base their decisions, at least in part, on data. The data includes downloads, adoption figures, conversions, support requests and customer experience performance ratings. Armed with information, the organization will do a better job delivering the software customers want and are willing to pay for.

Helping People Ask the Right Questions

Developers are a prideful bunch. They want to build great software that users love to run. Without insight into business needs and user preferences, they’ll do their best with features and functionality. With data sharing and communications channels in place, they can ask questions like “Where is this feature being deployed?” or “Why will customers want a whole new interface?” Consequently, ops pros can ask, “Who pushed this new release to production and when?” VPs can ask whether the latest round of releases will be ready for the big marketing push to launch it. If you’re asking the right questions early enough in the process, you’re going to make better decisions and be doing a better job delivering on business goals.

Being Nimble Enough to Change Quickly

Organizations pride themselves by their ability to act in an agile manner. But, are they really set up to move on a dime? Say a well-researched release flops unexpectedly. Sales and support rapidly gather feedback that European users hate the color and APAC users hate the navigation path. Because all the stakeholders are aligned and ready to move, developers tweak the features, ops leaders juggle the release schedule, management relaunches with regional marketing campaigns, sales follows up with targeted incentives and developers watch for more user feedback that can be incorporated into future releases.

Your company has worked hard to turn itself into a software-driven business. Using the principles of SDM, your business can drive business results to the next level. It can generate more sales. It can make unhappy customers happy. It can make happy customers even happier. It can improve its own responsiveness, up and down the organization. Most of all, it can use software to achieve its ultimate goal: to drive business value.

April 30, 2020
How SDM’s Holistic Approach Connects Organizations and Streamlines Delivery Processes
Since Jez Humble and David Farley popularized the term in 2010, continuous delivery has revolutionized software delivery practices.

Creating mature continuous integration/continuous delivery (CI/CD) pipelines has helped enterprises deliver software that performs better, based on a long list of metrics. Failure rates have dropped from 50% to under 15%, and cycle times are measured in hours, not weeks. Organizations get to market faster and release products into production with fewer bugs.

But, there’s a limit to what CI/CD can provide. CI/CD connects only the parts of the software lifecycle that happen inside DevOps. It doesn’t provide insights into where the delivery cycle is breaking down, whether the software is helping the business meet its internal key performance indicators (KPIs), how customers are responding to it or whether last week’s release is achieving the internal efficiencies the business was projecting.

A More Holistic Approach

This is where a more holistic approach to software delivery can help. As we discussed in a previous blog, there’s a move afoot to recognize software delivery as more of a core business process and connect all the aspects of this process in one new, interconnected discipline.

The new discipline, called software delivery management (SDM), changes the game by going beyond CI/CD and empowering developers and other stakeholders to best collaborate and creatively solve problems. It gives them the data, insights and best practices that allow them to truly continuously improve, keep value flowing and efficiently develop and deliver the right products and features to customers.

To follow SDM principles, a management solution needs to rest upon four key pillars: common data, universal insights, common connected processes and having all functions collaborating.
- Data: All information within and around software delivery activity would need to be captured and stored with a consistent domain model to enable and facilitate connected processes, shared insights and collaboration.
- Insights: Visibility and insights would need to enable a common understanding and continuous learning from data across, up-and-down and throughout all functions in the organization.
- Processes: Processes would need to orchestrate the entirety of the software delivery value stream and connect functions together to efficiently bring ideas to market with maximum value and adoption.
- Collaboration: All functions and teams within and around the software delivery organization would consequently work together to amplify value creation efforts.
This holistic approach helps organizations in a number of ways. Here are three benefits DevOps managers can net in terms of the way they work, collaborate with development teams and connect with peers throughout their organizations.

Faster Development

The current process can be slow and tedious. Developers use tools configured to perform one task, and getting a complete end-to-end view across the toolchain is difficult. An SDM solution stores all the artifacts and data from disparate tools that move software from idea to deployment in a unified common data layer. The information is connected and available for those who need it, when they need it and in a form they can use. Just forging these connections will save hours of time looking for data, checking statuses and tracking issues throughout the delivery process.

Better Software

If stakeholders have no end-to-end view of the development and delivery process, they have no way of knowing if the software is meeting the organization’s business objectives. With SDM, the continuous feedback loop established with CI/CD and DevOps is extended to fully include all stakeholders from the ideation phase through user adoption and back to ideation. Better insights are tracked more closely and shared more widely, helping developers do a better job of delivering what the users want.

Better Collaboration

Developers and business leaders don’t collaborate well for a number of reasons. Developers often don’t have a good feel for how a product is going to be used, and business leaders usually don’t have insight into what’s in development or when it will be delivered. This is complicated by scattershot communications processes–where both sides have to sort through spreadsheets, connect over email threads and wait for meetings to move processes forward. With SDM, the silos between the software organization and other stakeholders are broken down, so both sides can share information, communicate more directly and drive toward solutions.

In an upcoming blog, we’ll explore other examples of SDM in action and dive into the business value it creates for organizations.
April 23, 2020
Software Delivery Management: New Mindset

Innovation and mindset can take a while to evolve. It was the case with social media, wireless communications and the Internet of Things.

Software delivery is a good example of a concept that’s ready to stand on its own. As software becomes more integral to companies and to society at large, the way it’s designed, built and distributed has evolved. Today, the delivery process is far more sophisticated than it used to be, with more resources in the mix and more moving parts. After years of looking at aspects of the software delivery process as separate pieces, people are thinking differently about how all the pieces fit together.

That change in thinking is putting momentum behind the creation of a new IT category called software delivery management (SDM). SDM pulls together under one umbrella all the teams, tools, information sources and processes that relate to the delivery of software. It aims to eliminate inefficiencies that have cropped up along the way and help create a more value-driven process, aligning organizations’ resources toward a common set of goals.

The Road to SDM

For decades people divided IT into two basic camps: hardware and software. The categories were analyzed largely in terms of output. Experts tracked how many computer boxes a big integrated manufacturer such as IBM or Digital Equipment Corp. made, or how many software disks Microsoft spun off its production lines. Companies had their own methods of getting products out the door. Management techniques such as TQM helped guide these processes, but little outside attention was paid to specific steps companies took along the way.

All of this changed as software itself started to evolve. The internet, smartphones and more versatile operating systems opened up the software market to everybody. Software producers (manufacturers and individual companies alike) spent a lot of cycles working to perfect aspects of the software delivery process. Companies created teams to research, scope, develop, engineer, produce, integrate, test, market and sell software apps. They applied a variety of tools to accomplish specific tasks and implemented processes such as continuous integration and continuous delivery (CI/CD) to help streamline the engineering functions.

These advancements brought software to where it is today—companies are delivering apps quickly and updating them often. Operations teams are collaborating with developers, and organizations are doing their best to anticipate customer needs and institute quality control mechanisms. Software is achieving things never thought possible.

Stashed in Silos

But the delivery process isn’t perfect. Too often, pieces are separated from each other. Tools don’t work together. Teams don’t talk to each other enough. Information is stashed in silos that one side of the organization isn’t sharing with another. Data that does get shared hasn’t been updated.

Disconnects like these cause ripple effects throughout the organization. Projects get delayed; apps get delivered with bugs and/or vulnerabilities; developers find themselves overworked, restarting initiatives because they didn’t have the information they needed to do it right the first time. Executives get frustrated that teams aren’t delivering on their goals.

The disconnects can’t really be blamed on a particular team, tool or misplaced plan. They’re the fault of the overall process itself. A development team might be doing a great job with a top-quality tool and a well-thought-out plan. But if the product development group supplied developers with out-of-date information and a testing tool doesn’t integrate with a tracking database, the project will run into problems.

All Processes Connected

What’s needed is a new category that looks at software delivery from a holistic vantage point. Everything connects with everything. SDM provides those connections. It outlines a set of pillars to unify data, insights, processes and functions—aligning organizations around a common set of goals.

In upcoming blogs, we’ll look more closely at how SDM works, what kinds of benefits it can provide and how it can generate true business value for an organization.

— Anders Wallgren

April 16, 2020
Scripts in the Development Pipeline: Best Practices for Avoiding the ‘Scriptocalypse’

The use of scripts as a key tool in the average developer’s bag of tricks has altered the development pipeline in significant ways. It has even led some organizations to outsource their entire team to manage the glut of scripts that have been developed over the course of time.

To help companies move beyond the “scriptocalypse,” we recently participated in a webinar with our friends at Tricentis. Sunil Mavadia, global head of customer journey at Electric Cloud, and Thomas Stocker, director of product management at Tricentis, walked through best practices to follow to avoid the scriptocalypse. Here are some takeaways from their discussion.

Improve Process Robustness and Scalability

Of course, everyone wants to improve their process robustness and scalability, but how? One of the most effective ways of doing this is to employ a tool that helps orchestrate your releases and provides reusable objects. This reduces risk because you no longer have to write scripts for every single application, which creates a cascading effect—not only do you write fewer scripts, but also you don’t have to maintain these scripts across the board. As a result, you are assuming much less risk across many levels of your technology and pipeline. Additionally, system-level visibility across all of your processes will increase—which has many benefits for your entire development life cycle.

Don’t Just Automate, Orchestrate

Many developers make the mistake of thinking that because they have scripts, they’re automated. But to be truly automated, every action needs to be automated and driven by your pipeline. Your approvals, your handoffs … everything. Oftentimes throughout the release process, team members find themselves having to stop in the middle of their tasks to ask for approval. This only creates a process that interrupts the entire pipeline. But what if you could automate all of that?

A great quote from the “2018 Accelerate State of DevOps Report” that speaks to this says, “High performers automate significantly more of their configuration management, testing, deployments and change processes than other teams.” If you look at it purely from a team perspective, the high performers are those that have fully automated all their processes. While this can certainly sound like a daunting task, it’s best to look at it with an agile mindset—this is an iterative process. You have to start somewhere, while keeping the goal of 100 percent automation in view.

Not only should you be automating, you also should orchestrate your pipeline. This is critical. By automating and creating reusable objects, you can get to the point of using your pipeline as a service. Imagine reusing this particular pipeline for every application within your process, so you could have hundreds, even thousands, all using this pipeline and all these objects underneath that are all reusable objects. It’s a pretty picture, isn’t it?

Create a Value Stream Map

When you create a value stream map for a pipeline, you want to be very inclusive about who you invite in the room to create the value stream. Include representatives from release management, development, deployment engineers, team leads, executives and infrastructure engineers. Basically, anyone who has “skin in the game” for your release should be present.

Once you feel you have a solid representation of who is impacted by your releases, sit down and draw out the entire process. This includes figuring out what each script does. While that may be a big task depending on how your pipeline is currently structured, there’s actually a lot of opportunity there. For example, you might find there is one application for which you use one script to do that deployment. Typically, that script is duplicated for another application and so on. Imagine doing that 100 times. You’ll have 100 scripts or variations of that one script for your 100 applications. That’s an obvious and beneficial place to deploy a bit of automation.

This is also a great time to understand what tools you are using for each step. When you’re jumping from toolset to toolset, it can be difficult to get a holistic view of how this is impacting your overall performance without a complete picture of the value stream. When you have that picture, you can start to ask yourself questions such as, “What are all these tools that we interact with that we can integrate into the pipeline? How are we handling approvals?” By including detailed sequencing timelines and identifying bottlenecks, you can realize some impactful efficiencies.

Finally, creating a value stream map gives you an opportunity to organize your scripts. Laying out the entire value stream can give you a picture of the variety and quantity of scripts you’re running—whether they are CI scripts, deployment scripts or test automation scripts—and bucket them in a logical way.

Identify Redundancy Dependencies and Bottlenecks

Of course, identifying (and hopefully rectifying) dependencies and bottlenecks is a good best practice for any aspect of your development pipeline. But it becomes even more important in the context of using scripts. If you have one application dependent on another application, it’s almost impossible to script. For example, imagine you’re trying to do some testing with two dependent apps. If you don’t have the right versions across both, you can’t test one because the other one is not up to speed or it hasn’t been updated in an environment. Something even as simple as that can be really disruptive and make scripting very difficult. By understanding where your dependencies and redundancies exist, you can take the first step toward solving the issues they present.

Scripts are one of the most effective ways of streamlining your pipeline, but they need to be used in a way that will allow your organization’s applications to be scalable and sustainable. By following some of the best practices outlined above, your organization can use scripts like pros—and avoid the dreaded “scriptocalypse.”

For more insights watch the full webinar here.

— Anders Wallgren

October 11, 2018
Continuous Discussions, Episode 90: Gene Kim and DOES18 Speakers

In a recent episode of the Continuous Discussions (#c9d9) podcast, we hosted Gene Kim and several speakers from the upcoming DevOps Enterprise Summit Las Vegas (DOES18) conference. We’re proud Founding Partners of the event and always enjoy having IT thought leaders join us ahead of the show to talk about DevOps, the conference program and more. In this episode, we discussed an emerging conference theme of next-gen operations and infrastructure.

The panel of DOES18 speakers included: Cornelia Davis, senior director of technology at Pivotal; Damon Edwards, co-founder and Chief Product Officer of Rundeck; Mik Kersten, CEO of Tasktop; Tom Limoncelli, SRE at StackOverflow.com; Gene Kim, founder of IT Revolution and co-author of “The DevOps Handbook,” “The Phoenix Project” and “Accelerate”; and Electric Cloud’s Sam Fell and Anders Wallgren.

Continue reading for some of the top takeaways from the discussion!

Next-Generation Operations: Programming and Insights

As we continue to evolve, the best is yet to come for operations teams, per Kim: “If we follow the course of what the development languages and development frameworks have done for development productivity, and extend that to operations, I think the best time for operations and infrastructure and platforms are to come.”

Davis encouraged operations teams to treat their platform like a product instead of infrastructure: “When I speak with the IT operations folks, they’re still largely short order cooks. They’re very reactive, doing things that people need them to do. It’s about being able to apply some of the same principles of continuous delivery to the way that you do ops. At the core of that is this philosophical notion, which is to treat platform just like you treat any other product.”

The way enterprise operations teams do work is unique, per Edwards: “For enterprise operations, you have to fundamentally rethink the conventional wisdom of how you think about, approach and do operations work. DevOps Enterprise Summit started in the early days of the DevOps explosion. This is distinctly an enterprise discussion because it’s all about that history of success that we have to account for.”

Kersten on the different ways development and operations teams measure outcomes: “I so often see the development side of the organization and the infrastructure operations side collecting completely different metrics and telemetry. The development side uses the more familiar telemetry that comes from DevOps. And the operations side is still in a burnup, burndown world. There are no common metrics or languages for what really should be an internal value stream.”

Great operations management advice from Limoncelli: “Get one team to do it well, and let people copy that success. And do it in a way that creates cultural minimums or cultural ways of doing things, so that people are competing to make things better instead of being told you have to make this change.”

Moving away from legacy operations is a dauting challenge especially for the enterprise, explained Wallgren: “The things that we refer to as legacy are the things that keep the lights on and bring the money in. It’s a big leap to move away from that and it’s going to take a lot of work and quite a lot of time. It’s like changing the engine and tires on a car while you’re driving down the highway at speed.”

Functional Programming, SREs and More Musings

Davis on functional programming, something she will be talking in-depth about at DOES18: “Treating function as a first-class citizen is absolutely essential because you start to create these functions very dynamically, and the functions now encapsulate the state. So instead of having state and then having side effects against a state, what we have are these functions encapsulating a sequence of changes that are going to generate a state.”

Kim described an “aha” moment he had during Davis’ talk at DOES18 London about sidecars in Kubernetes: “The ability to liberate developers and address cross-cutting concerns like logging was one of the primary impetuses why people flocked to Spring from orthodox frameworks that were so unsatisfying. And that was, for me, that was one of the biggest aha moments of the conference.”

Edwards discussed the implications of toil work and how to spend more time doing things of value: “Toil work is work that we should have automated, it’s repeatable, and not adding enduring value to what we’re doing. Versus work that requires human ingenuity, the engineering work that adds enduring value to our organization. Set a limit as an organization to not accept more than 50 percent of time being consumed by toil and figure out how to help those organizations where they have too much toil and not enough engineering work.”

Most organizations don’t have $2.5 billion in their back pocket to build a product-oriented value stream, but Kersten said we should take a page from their book: “This is a testament to how invested the tech giants are in their own platform as products. If we don’t at least adopt their thinking around having this connected end to end platform that supports all your products and value streams, you’re just not on the right path.”

Limoncelli on the ability of SRE’s to reach across silos: “In the SRE model, you can reach across silos. I like to think of it as you have the CICD pipeline and the SRE’s stand at the end of the pipeline asking, ‘What are the operational problems that we’re having? What’s the constraint? What’s the bottleneck?’ They fix that by asking how they can collaborate with the other team within our organization. And either sit with them for a week and fix it or collaborate with them to fix the problem.”

Wallgren elaborated on treating pipelines as code: “Treating your pipeline as code is something we’ve been evangelizing. And really, that should be the first thing that you really worry about in some sense, because you’re not going to ship anything unless you have a pipeline.”

The DOES18 Las Vegas agenda is now live! Check out the sessions from Davis, Edwards, Kersten, Limoncelli, Wallgren and dozens more. Tickets to DOES18 are running low, so be sure to register before it’s too late.

Watch the full episode:

— Anders Wallgren

September 6, 2018
Continuous Discussions Video Podcast: DevSecOps, Best Practices and More

In a recent episode of the Continuous Discussions (#c9d9) podcast, a group of industry experts discussed why DevSecOps is officially more than just a buzzword, tips on how to get everyone in the organization to own security and some of their own challenges and experiences baking security into the software delivery pipeline.

The panel included: Alan Shimel, editor in chief at devopsy.kinsta.cloud; Chenxi Wang, managing general partner at Rain Capital; Derek E. Weeks, VP and DevOps Advocate at Sonatype; Paula Thrasher, Chief Architect, National Security Division at GDIT; Geetika Tandon, director of technology at Federal Civilian CTO Group; and Electric Cloud’s Sam Fell and Anders Wallgren.

Continue reading for some of their top takeaways!

Why DevSecOps?

DevSecOps is really more about a cultural shift, said Shimel: “DevOps is DevOps. It’s not just about Dev and Ops, though. DevOps is about the whole organization adapting a common cultural way of interacting including security, and bringing security into that culture, into that software ordination assembly line.”

Integrating security into development practices takes a lot of collaboration and communication, per Wang: “It’s easy to say security should be part of the process. But in practice, it’s difficult to do. When you have a security testing process that takes days to complete from the input to the output, and then you have developer teams releasing 500 releases a day, those are two very different cultures. It’s usually an iteration process but it takes a lot of team communication, a lot of sitting down, figuring things out for security to be part of it, and so it’s a not an easy process.”

Why DevSecOps and not just DevOps? Well, security people weren’t necessarily invited to the party, explained Thrasher: “It’s a cultural challenge that security hasn’t necessarily been an integrative part of a lot of the DevOps transformations. I think ‘DevSecOps’ exists because either we’ve left security out or they’ve stayed outside this transformation that’s happening.”

There really shouldn’t be a trade-off between security and quality, according to Wallgren: “I long for the day when we don’t distinguish between security and quality because, yes, there are different disciplines to them in some extent, but as far as the end user is concerned a bug is a bug.”

Weeks wanted to let nonbelievers know that DevSecOps is real and more than just a buzzword: “DevOps and security can integrate with each other and integrate successfully at scale. There are organizations of a couple of hundred or even a couple of thousand people that have a DevSecOps practice in place that are continuously improving it, adding an automated security, so it’s definitely gone beyond buzzword status.”

Including security in the release process should be a no-brainer, according to Tandon: “I think of it as common sense. I don’t even understand how security has been left out. I don’t understand how we have these huge pipelines, we’re releasing a hundred code releases a day and then, boom, we’re stopped. We’re stopped for scanning.”

Common Barriers to DevSecOps and Patterns for DevSecOps Success

Progress takes time, but Shimel remains hopeful of the future: “We’re not going to boil the ocean or change the world in a day, but we will change the world and we will change the way software is delivered so that software and security is synonymous with quality.”

Wang explained how one company is utilizing automation to successfully bake security into their process: “A company that I have worked with is in the container world and they have what they call a Brown’s registry. So they have code that is assembled by the development team and then goes into a Brown’s registry. No other code can get in to the Brown’s registry unless it’s from their own team. Then from that Brown’s registry the security team automatically goes in and scans it and then only those that pass the scan gets put into what they called a Gold registry. And from there then they could deploy it to the server.”

Everyone in the organization should practice and be prepared to respond to security incidents, advised Thrasher: “The people that need to practice responding to vulnerabilities the most actually aren’t the security people. They’re used to responding to security issues. It’s the ops and the developers who aren’t used to responding to security incidents and they aren’t equipped, they aren’t trained on how to respond. So when your vulnerability happens it’s a bad new story.”

Many security issues have been around for a notorious amount of time, which is where the software delivery industry can improve, said Wallgren: “Where are most of the exploits coming from? The vast majority of them as I understand it are coming from well-known longstanding security issues that have been out there for a long time or, at least long by technology standards. And so a lot of it is just simple stuff that we need to be better at.”

Security as code is the next step in truly integrating security with DevOps, explained Weeks: “I was just talking to someone in Europe at a big telecom company, and they were saying, ‘I’m on the security team, we’re working with development, and we’re getting closer. We have all these rules and things that developers need to follow and we’ve been working with getting them to follow them more.’ And I said, ‘That’s really cool. Any of those rules or policies instrumented as code?’ And he responded cross-eyed like, ‘What? Code?’”

Tandon shared an eye-opening experience with a customer: “I’ve seen a situation where we were about to go to production—it was DevOps with security toward the end—and we got 4,000 hits and we had to go to production, most of them were ignored. So are we really securing our system or are we making our own system even more insecure by doing that?”

Watch the full episode:

Want to dive deeper into DevSecOps? Industry expert John Willis, co-author of “The DevOps Handbook” and “Beyond The Phoenix Project” and VP of DevOps and Digital Practices at SJ Technologies, is scheduled to lead a full day hands-on workshop immediately following DevOps Enterprise Summit in London and Las Vegas. Register soon—space is limited!

Register for the London workshop

Register for the Las Vegas workshop

— Anders Wallgren

May 16, 2018