devopsy.kinsta.cloud

Tag: IT governance

Building DevSecOps With Value Stream Management
As teams develop software, testing for potential security risks and flaws is mission-critical, just as it has always been. Yet, cyberattackers continue to exploit code vulnerabilities as open source code remains the foundation for the majority of commercial applications across all industries. Security firms are churning out solutions at an aggressive rate, and both project managers and their teams hope they can purchase some bolt-on tool or license a service to take care of security, painlessly. In the end, vulnerabilities still occur.

The most effective pathway to secure software code has always been to eliminate vulnerabilities. The question today is, “What is the best way to achieve that goal given such an accelerated, threat-laden landscape?” Before we dig into that answer, let’s take a closer look at the problem.

Software Security Is Everyone’s Issue

Even the largest organizations, from Adobe to Microsoft, have been humbled by the number and scope of code vulnerabilities that have been overlooked—and therefore escaped into public release—due to the frenzied pace of software updates. Here’s just one example:

On May 11, 2021, Adobe issued a security update for Adobe InDesign that addressed multiple critical vulnerabilities. “Successful exploitation could lead to arbitrary code execution in the context of the current user,” the security bulletin read. “Arbitrary code execution in the context of the current user” is cybersecurity speak for “an attacker can execute arbitrary commands or code on a target machine or process.” Arbitrary code execution vulnerabilities are known to be one of the top attack vectors that lead to system penetration resulting in data breaches.

Given the enormous effort companies and their software teams are exerting today, and the importance of avoiding software flaws in the current threat landscape, how is this happening? It doesn’t help to have publicly available lists of software vulnerabilities posted on the dark web and CVE (ironically, a database of vulnerabilities launched to help companies improve their cybersecurity). Hackers — from solopreneurs to nation-state teams — are having a field day with this information.

Rearranging the Puzzle Pieces Doesn’t Guarantee Security

In an attempt to resolve this challenge, industry professionals are coining new terms and calling for new approaches. One I have seen promoted recently is “SecDevOps” rather than “DevSecOps.” With SecDevOps, they assert, security is first—conducted at the start of the project and maintained all the way through.

This approach isn’t new—it’s essentially a variant of shifting left, which for more than a decade has been an established mechanism in software testing for finding flaws and “failing faster.” In my experience, while failing earlier and faster with any software endeavor has considerable value, it is not automatically going to highlight all security challenges and enable their resolution.

It is just as important for DevSecOps teams to collaborate and work together, not only to support development and operations, but also to ensure security, compliance and governance are built into every piece of code.

Insight Is Preferable to Blind Interpretation

Unfortunately, achieving that collaboration productively comes with challenges. DevSecOps teams need rich, real-time visibility into code vulnerabilities, early, so they can act on that information and update code as it moves through the pipeline. To do that, they need actionable insight—not a motley assortment of data that’s open to interpretation.

In value stream management, we have seen firsthand while working with our clients that several techniques can be applied to eliminate vulnerabilities more quickly and effectively. All of these approaches hold value, whether a firm is seeking to deliver a new product to market or secure its current software pipeline.
- Amplify feedback loops. Continuous feedback loops—from DevSecOps tools, for example—give developers more detailed insight into their code’s vulnerability.
- Run security scans. Give testers the ability to run security scans on their code even before it is committed to identify security vulnerabilities as early as possible.
- Leverage reporting and analytics. Gaining early access to as much insight as possible makes teams and management more confident regarding swift action to resolve code deficiencies.
These approaches also support governance and compliance initiatives. As a bonus, they boost software quality, as well.

The Business Case for Built-In Security, Governance and Compliance

As mentioned earlier, these activities are integrally connected with the effort to eliminate code vulnerabilities. In case you are thinking, “This is obvious,” take a moment to reflect on this statistic: 60% of data breaches involved vulnerabilities for which a patch was available but not applied.

The sheer scope of the problem also underscores the lack of attention to security and governance—and any compliance requirements that accompany it.

Half of websites in production have high-risk vulnerabilities, and 82% of these are located in application code.

The high percentage of errors in source code suggests it was not checked for vulnerabilities during development, signaling that developers focused on app functionality and gave short thrift to security. This often means they don’t have a program for governance and compliance, either.

In my experience, a primary reason for these failures is that teams don’t have actionable information from the activities mentioned above, such as real-time feedback from DevSecOps tools. Developers need this insight when they are resolving sprint backlogs. Information relating to high-level security issues must be available so it can be addressed, with issues prioritized and resolved, and not left sitting in siloed DevSecOps tools.

In many ways, DevSecOps is in the same stage of transformation as DevOps was a decade ago. Teams are frustrated by needs they do not know how to meet, and they hope for fast resolution in the form of tools and solutions. As with virtually every other software challenge over time, there is no replacement for getting in the trenches and fixing the problem rather than placing strategic band-aids.

If you are not subject to compliance mandates and therefore dismiss the value of software governance, consider this: If your team misses a software vulnerability that results in a major data breach because you didn’t have the insight to identify it—or the governance to ensure it was addressed—your firm may not recover. The imperative for software security—and the governance that helps ensure it stays that way—should be sufficient for organizations to take vulnerabilities seriously.

Effectively applying the insight gleaned from software code reporting and analytics is the surest way to eliminate vulnerabilities. As an added benefit, the reporting and analytics that informs the software effort can also make the case to upper management for the value of launching future improvement initiatives.
August 27, 2021
The Next Frontier for Enterprises: Moving to a Product IT Operating Model
In the world of technology, there are few things as dramatic and impactful as enterprise IT’s recent ascent from an enabler to a strategic business driver. Tasked with powering next-gen digital business models, IT teams today are fast realizing the harsh truth that they must make a pervasive strategic shift from traditional project-based operating model to a product-centric one.

Traditional project-based operating models focus resources on a specific application or project, leading to fragmented experiences and a disconnect with the rest of the IT organization or company. For instance, in many cases, one team may be working on a back-end system, while entirely different teams work on the front-end application or updating databases. A product-based operating model takes a more holistic approach to ensure projects are integrated and teams work concurrently toward a uniform goal.

The days of plan/build/run are numbered as digitization forces IT leaders to act fast, think outside the norm and embrace the blurring boundaries between the digital and physical worlds. A product-centric operating model, sometimes also referred to as a DevOps operating model, accelerates organization-wide digital transformation and consolidates a business’ competitive standing. Companies must transition to a product-centric model to scale-up their agile programs to support end-to-end innovation and reduce siloed operations and projects.

The companies that have embraced a product-centric operating model are quickly realizing the benefits, but for many IT departments, they have yet to experience or fully understand why the shift is imperative.

What’s Driving the Shift to Product IT Operating Model?

As digitization takes hold and becomes more and more pervasive, CIOs are realizing that intertwining IT and business objectives is quintessential to respond to market changes quickly and remain competitive. Since 2018, volatility has returned to global equity markets in a big way. With the economy in a constant state of flux, coupled with rapid globalization and unprecedented disruption from technology, the average lifespan of a company is 20 years, compared to 60 years in the 1950s, according to Credit Suisse. Overall, the changing business environment makes few things as abundantly clear as this–agility, an appetite for new markets, new products, bias for action and a fail-fast approach will be the new mandates for IT leaders going forward.

To this end, most CIOs are replacing portfolios of discrete, siloed projects with product lines that enable a specific business capability. The key factors driving the shift toward a product IT operating model include:
- Need for Holistic Transformation: Siloed functions have no place in the digital business world that thrives on integration of data and information across structure, culture, processes and technologies. To enable holistic transformation, organizations must draw a balance and establish cross-collaboration between organizational guidelines versus autonomy of teams for process, architecture, technology, tools, etc.
- Lack of Alignment in Traditional IT Operating Models: Lack of alignment across large teams and multiple organizational groups (such as IT, finance and business) prevents organizations from executing an end-to-end digital strategy. Shifting to a product IT operating model can help steer organizational units toward a common goal. It can facilitate greater collaboration between different DevOps teams that support different products and domains. Recognizing the failure of existing IT governance and the pressing need for a shared digital transformation vision, IDC predicts 40% of CIOs will adopt new digital governance models.
- Need for Speed: Most companies cited that their digitization efforts were hindered by inflexible or slow processes with a traditional IT operating model. Adopting a product-oriented management approach for IT can help organizations define and agree on a pace of change congruent with their readiness and appetite. According to IDC, 65% of CIOs will expand Agile/DevOps practices into the wider business to achieve the velocity necessary for innovation, execution and change by 2021.
- Lack of Integration Between Old and New Technologies: Lack of integration between new and existing technologies is a key factor that keeps organizations from realizing the full potential of digitization. Modern companies need to balance innovation, experimentation, adoption of new technologies and tools with the enterprise’s need for business results, cooperation, risk management and compliance. Because a product-centric model takes a holistic, end-to-end approach, it is ideal for integrating technologies, processes and teams at scale.
Embracing a Next-Gen Operating Model

To compete with newer, nimbler companies such as Facebook, Uber, Spotify and others, traditional companies that are serious about continued growth are leaving no stone unturned and embracing a product IT operating model. From being providers of reliable IT services, CIOs have emerged as strategic partners in driving business transformation. Modern day CIOs are driving business innovation and identifying opportunities for competitive differentiation, not delivering ad-hoc, siloed services. As cloud-first and AI-first become the new mandates for organizations, re-engineering the IT operating model is quickly becoming central to creating value, competitive advantage and growth.

— Rohit Nand
December 6, 2019
Rugged DevOps: Less Capture the Flag, More Teamwork

At the recent DEVNEXUS conference in Atlanta, I caught up with Chris Corriere to talk about his experiences in the realm of Rugged DevOps. Chris is a DevOps Engineer at AutoTrader and a contributor here at devopsy.kinsta.cloud. During our conversation, we discussed automation, culture and collaboration, and which thought leaders he is following. Chris also shared insights on the upcoming DevOps Days Atlanta conference where he serves on the organizing committee.

Derek Weeks: What’s your definition of rugged DevOps?

Corriere: Rugged DevOps to me is definitely manifested in the security sector but it’s a shifting quality and awareness left in doing the best we can with the tools we have, the constraints we’re given, and in the environment we’re in. It’s about being more experimental, understanding that we’re going to encounter failure, and learning to adapt accordingly. We’re very much having to make tough decisions in real time. If you’re running anything in production these days, you need to do that as safely and as consistently as you can.

Weeks: Is rugged DevOps only about security?

Corriere: No. Security is really about mitigating risks. I think a lot of this ties into situational awareness about understanding your company’s risk. Is it user data? Is it credit card transactions or do you have a Russian botnet running in your infrastructure that you’re not aware of? What’s the asset you need to protect? Are you trying to protect an asset that is not there?

You can’t have security come in and become suffocating where it’s not needed. In other situations you can’t leave things wide open that really do need to be protected because you’re more concerned with moving quickly than moving safely.

https://youtu.be/fgMPKkM1NdY

At the end of the day when you’re mitigating risk, it doesn’t matter if it’s customer data or a broken build or a bad user experience, these are all risks we need to mitigate. That’s really what rugged DevOps is. It is about is tying that value back to the user. It’s knowing what problems we trying to solve and understanding what we doing effectively (or not). It can be locked down as tight as you please, but if you cannot use the tool to solve the problem you picked it up for, it might as well not be running.

Weeks: Security teams have been one of the last parts of the organization to come into DevOps practices. What’s been your experience with engaging security teams at Auto Trader?

Corriere: That’s a good question. I’ve had experience with more than just security teams at Auto Trader. I’ve been involved in security a few places and they’ve approached me first when they found out I was into the DevOps stuff. They often come looking for assistance. They want to know: How do we get this baked in? How do we make this a priority?

It really ties back into multi-factor authentication. It’s a good way to frame it up where you want developers and operations (those people who aren’t permanently rooted in security) to be aware of these things, to have automated scans, to be aware of cross-site scripting, and to understand how to remediate those kinds of issues.

We’ve been implementing open source tools earlier in the development lifecycle to give us quicker feedback loops in order to be able to triage these things. Security applauds that but they can’t trust us too much with it. They’ve got to verify that we’re doing a good job with this, right? It’s a give and take where I’ve seen security come and offer automated solutions to DevOps teams and those solutions get adopted. The DevOps teams then take that and really start to run with it. Then security wanted to slow down a little bit because they were concerned about the quality of the vulnerability remediation. There is some back and forth there that has to be anticipated.

________________________________

“We’ve been implementing open source tools earlier in the development lifecycle to give us quicker feedback loops.”

________________________________

For security to trust you you’ve got to trust security to a large extent. If you’re putting more automation into the left side of your process and doing things before they hit production should be making your job easier. One of the side effects of that security may start going through some systems with a finer tooth comb because they’re not doing as much fire fighting. That may result in another ask from security. Understanding that this isn’t a declaration of war—we are simply continuing with an incremental progression toward further improvements. Interactions are cyclic and will happen again.

We need to help security understand our pipeline, determine what’s going to be a good fit, and keep that communication level high so the trust is there. In these symbiotic relationships, cooperation is dependent on that trust. If that trust goes away you end up in something more competitive.

There’s always room to drop back to where it’s more commensal and knowing when the kitchen’s gotten too hot. At those times, everybody needs to take a cool down lap and come back to it at the beginning of the next month. It goes back to that situation awareness, knowing where you’re at right now before you try to get to where you’re going.

Weeks: We talked earlier today about red teams, blue teams, purple teams. What are they and can you describe some of the practices you’ve used to end up with more purple teams?

Corriere: Yeah. Red teams take an attacking, reactive posture. Red security teams are worried about internal compromise, and when they find gray areas—for example, things that haven’t been fully signed off and authorized—it can result in unplanned work cropping up for the DevOps team. Generally that kind of unplanned work is introduced because the “Red” team felt they were not able to accomplish their job another way.

Then the blue team side of the equation is really trying to stomp all that out. Blue teams keep things standard and locked down. They make sure everything’s approved in triplicate before it gets implemented, which can be rigid and very frustrating for developers.

Purple teams allow that experimentation in earlier in the development lifecycle where they’ve got a little bit more leash. They’re allowed to experiment and stand up new things and make those things work before they getting it approved. Again, it’s about being transparent. Teams first want to understand what change was introduced and what the advantages of it are. They may also acknowledge that you need more than one solution to some problems, and that one tool might not fit everybody. If they’re using tools to automate that’s good. We can’t box them all into one necessarily. That’s where you see this go from blue and red to more of a purple. There’s more communication. There’s more conversation. There’s less capture the flag, more teamwork.

________________________________

“There’s more communication. There’s more conversation. There’s less capture the flag, more teamwork.”

________________________________

Weeks: You’ve been involved in the conversations around rugged DevOps for awhile. Who are some of the other thought leaders you learn from?

Corriere: Jamesha Fisher out of San Francisco is one. She’s done some neat security stuff and is starting to look more into the vulnerabilities around some of the automation tools. Where are those vulnerable and where can they be leveraged against us? Automation is great until it automatically starts doing things we didn’t want. Another aspect of that is really trying to think more like an attacker; we need to think about: how would I break into this thing instead of just thinking how do I protect it.

Georgia Weidman is popular in a lot of security circles and I’ve missed her workshop once. I’d like to catch it. She’s got a book out on pen testing which covers a lot of operations and concepts from an attack perspective. I would say that can be a fault of some rugged DevOps practices; you’re so quick to get the thing working and provide business value that you don’t realize where you’ve left something vulnerable. Sometimes you’ve got questions about why that thing you work with frequently is always locked down. Perhaps you find you can’t use it the way you want to. At those moments, it’s time to read through Georgia’s book. You’ll get some information as to why that thing is not open the way you want and where someone might use it for reasons might not expect.

________________________________

“You’re so quick to get the thing working to provide business value that you don’t realize where you’ve left something vulnerable.”

________________________________

Weeks: As we wrap up I know you’re on the organizing committee for DevOpsDays Atlanta. Do you want to give us an insight on the dates of when that’s coming up and call for papers, et cetera or we can help you share that?

Corriere: Our CFP is open through the first of March. Our event is going to be in April. It’s the 26th and 27th which is a Tuesday and Wednesday. Monday, the 25th of April, we have Jeff Sussna coming in to do a full day workshop on continuous design. He’s going to be giving the keynote on our first day. We’re excited to have Jeff in town and excited to have him as a keynote. John Willis is going to be giving our keynote on the second day.

We’re looking for proposals from anything in progress, people learning in the field, or trying to drop that barrier of entry. We’ve got a lot of bright people in the space. We’ve got a lot of people learning as they go. We need to be more vocal about that learning process and how we find our way through it. There’s not always this instantaneous vision that strikes us. We understand how to convert a shop into continuous delivery every night. There are definitely steps in between that. We need to have more conversations about those step in our industry. Yeah, anywhere in the stack, any interesting projects going on, we’d love to hear from you if you’ve got something.

If you are interested in learning more about this subject, I invite you to download Amy DeMartine’s Forrester research paper, “The 7 Habits of Rugged DevOps.”

As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps.”

February 24, 2016
IT Governance at DevOps Speed: Time to Shift Gears
Technology plays an incredibly important role in the idea economy, where business innovation is dependent on the software that powers the business. Speed and agility in IT are key to a competitive advantage in this modern economy, in which small startups with disruptive ideas are competing with well-established enterprises and industries. Consider how Uber and Airbnb are challenging taxis, rental car companies and hotels, all based on the power of an idea and software. Yet, traditional enterprise IT struggles to deliver at the speed business demands.

Have you ever heard a business leader say, “Our IT teams are extremely responsive and very fast”? The typical response is quite the opposite. IT is commonly viewed as a bottleneck to business innovation and, in many organizations, the business simply avoids IT and instead creates some form of shadow IT.

IT Governance, the Old Way

The principles and practices of DevOps creates the potential for faster delivery and higher quality. However, one of the largest sources of friction is the traditional focus on IT projects and project governance. Consider this definition of a project from the Project Management Institute:

“A project is temporary in that it has a defined beginning and end in time, and therefore defined scope and resources. And a project is unique in that it is not a routine operation, but a specific set of operations designed to accomplish a singular goal. So a project team often includes people who don’t usually work together – sometimes from different organizations and across multiple geographies.”

Given that projects can be expensive investments, and because of limited resources, enterprises justifiably carefully prioritize and plan which ideas and initiatives should be approved.

Faced with limited resources, enterprises expend significant effort to prioritize, optimize and fund only the most important projects. But no matter how well-intended, this governance approach also struggles to respond to the speed and velocity required in the idea economy. Indeed, the typical portfolio-planning and project-management approach is often an underlying cause of dysfunctional IT behaviors such as excessive requirements, throw-it-over-the-wall deployments and questionable quality—none of which are pleasant.

Consider how the business approaches user stories and requirements management. In a traditional, project-driven world in which the project is a one-time opportunity, businesses often are given an exhaustive list of ‘must have’ features in their user stories. This behavior is understandable—given the ‘feast or famine’ nature project, funding businesses need to make sure their future needs are included, even if the need isn’t immediate.

Sadly, this introduces both technical debt and excess complexity from the very beginning. Especially, when you consider that only a fraction of those ‘must have’ requirements will ever be actually used. In fact, a recent IEEE paper noted that 28 percent to 45 percent of software features often go unused.

Consider how “project success” is typically measured. Project teams typically are evaluated on their ability to deliver on time and on budget, along with other various quality and satisfaction measures. The emphasis on schedule and cost can drive sub-optimal decisions, as deadlines and cost pressures loom. As a result, testing is often abbreviated, as the schedule compresses. Sometimes the scope is reduced to conform to the deadline. This routinely leads to challenged releases in which the service desk is left holding the bag, which ultimately degrades trust and teamwork.

After the project is completed, the project team often redeploys to other projects, depending on demand for specific skills. The accumulated knowledge and wisdom of the team is lost and there is no further incentive to invest in automation or process improvements because the team has disbanded.

The DevOps Shift

DevOps is a very different approach that requires rethinking how work is funded and governed. Fundamentally, a DevOps team is charged with delivering small, high-quality releases, very quickly. The team embraces fast feedback to remove bottlenecks and waste and utilizes relentless automation to make tasks repeatable and efficient. Given that the team will deliver dozens or more releases per year, it is incentivized very differently from the traditional “project” team. The DevOps team often “owns” the application in Development, Test and Production, and has a shared accountability for the value it delivers. The team reduces handoffs and delays and implement small, frequent changes in response to business demand, customer feedback and system performance.

The friction of traditional governance needs to be addressed using DevOps to enable the enterprise to thrive. One of the ways to remove the governance friction is a paradigm shift from projects as described above to products. It is vital to focus on value and business results along with employee and customer satisfaction. The key difference is that a product is funded for the long haul:
- A product team delivers ongoing value and the business can count on improvements to meet market demand.
- A product team does not form and disband depending on funding.
- A product team is focused on delivering value through the product and will naturally enhance and improve their processes. CIOs and senior leaders need to consider using the Product model as an alternative way to fund and govern their initiatives.
I’ve talked with several IT leaders who are exploring how to evolve the concept of the traditional project toward a longer-term product orientation. Some organizations explicitly make the change to products, and others are taking a more evolutionary approach. Regardless of the path, the need to evolve the funding model is clear.

Consider how product governance is different from projects. The product team, in partnership with the business, is funded for a sufficient period of time (year, half-year, perhaps quarter) depending on the business needs. The product team then delivers fast releases of business-driven innovation and rapidly updates the system based on feedback from continuously assessing the product. As the team remains intact, it is able to learn and improve while also accelerating delivery. Additionally, because the product team is able to deliver quickly, the incentive to overload releases with “nice to have” features is greatly reduced.

Here’s how the mindset is different:

Project                                                                         Product

Short time frame                                                   Long term commitment

Limited funding                                                     Predictable funding over time

Team disbands                                                      Team owns the product and results

No incentive to automate                                   Automation improves quality and speed

Incentives based on schedule/budget            Incentives based business value

This does not mean there won’t be projects in the future, but there will be a combination of products and projects in most large enterprises. Most likely, we will see projects will be the way new products are proposed, evaluated and conceived. However, once the product is launched, the product team then will operate under a product model and mindset.

About the Author/John Jeremiah

John Jeremiah leads the Digital Research Team with the HP Software Applications Delivery Management team. He is a software professional with over 20 years of experience including a variety of leadership roles with the U.S. Navy, IT consulting and Fortune 500 IT organizations. His previous roles span from application developer, project and program manager, Director of Testing Services, and Delivery Process and Methodology Director. Prior to joining HP, he was director of Testing Services and Project Delivery Services for Thrivent Financial for Lutherans, where he led the adoption of CMMI Maturity Level 3 process framework.
February 4, 2016