devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

Codenotary Adds SLSA Framework Support to Advance App Security

Codenotary this week announced it has integrated support for the Supply-Chain Levels for Software Artifacts (SLSA) framework in its free notarization and verification service for ensuring the integrity of code.

Moshe Bar, Code Notary CEO, said as the first application security platform to attain SLSA compliance, the company is making it easier for organizations to secure their software supply chains.

SLSA is a framework defined by Google that it uses to attach binary authorization to code. In its current form, SLSA is a set of guidelines that identify how to attain four different levels of application security. The long-term goal is to create a framework that will automatically generate auditable metadata that can be fed into policy engines to create a SLSA certification.

Codenotary provides tools for cataloging and verifying the authenticity of components used with a software development life cycle. At the core of that platform is an immutable open source immudb database that cryptographically attaches an identity to each software artifact. That capability can also be used to dynamically generate a software bill of materials (SBOM).

The Codenotary Attestation Service can also be integrated with the continuous integration/continuous delivery (CI/CD) platform that DevOps teams use to build and deploy applications using those artifacts. Ideally, DevOps teams should construct a security pipeline that runs alongside their existing DevOps pipelines, noted Bar.

In the wake of recent high-profile software supply chain breaches, the level of scrutiny being applied to how software is constructed has increased significantly. In theory, at least, developers have assumed more responsibility for application security as part of an overall shift left that provided them with more programmatic control over IT environments. Cybercriminals, however, are now becoming more adept at compromising developers’ credentials and insert malware as an app is developed.

DevSecOps best practices are, of course, being adopted to teach developers how to better secure their application environments. The issue is that not enough developers are learning how to secure applications fast enough. The volume of attacks against the software supply chain continues to grow at a rate that is now much faster than organizations can thwart simply by educating developers.

There is a now clear need to embed more cybersecurity guardrails within DevOps workflows, noted Bar. Rather than attempting to achieve that goal in a piecemeal fashion, Bar said Codenotary is making a case for an integrated platform that can be invoked as a natural extension of a DevOps workflow.

In general, Bar said there’s now a greater sense of urgency when it comes to software supply chain security. Many of the reviews of those processes are, arguably, long overdue. One way or another, there will be a lot more questions asked about how software is actually developed. The organizations that consume this software are starting to appreciate how vulnerable they are as the overall dependency on software continues to increase. The only real question now, unfortunately, is how many more breaches there might be before those software security issues are thoroughly addressed.

June 17, 2022
Armory Aims to Turn Continuous Delivery Into a Service

Armory this week made generally available a continuous delivery-as-a-service (CDaaS) offering that promises to make it simpler for a much wider range of organizations to programmatically deploy applications.

The Armory Continuous Deployment-as-a-Service platform has been available in beta since March. It differs from Armory’s existing open source Spinnaker continuous delivery (CD) platform in that it is based on a proprietary declarative platform that is designed to integrate with third-party continuous integration platforms.

Armory CEO Jim Douglas said the goal is to meet organizations that have invested in CI where they are versus requiring them to rip and replace their entire DevOps platform. The company is still committed to providing a curated instance of Spinnaker, but it’s become apparent there are many organizations that want to experience the benefits of a more opinionated CD platform that is more accessible via a cloud service, he added.

The Armory Continuous Deployment-as-a-Service offering applies many of the best practices that have emerged for the Spinnaker CD platform without requiring organizations to have a dedicated team in place to just manage the CD platform, noted Douglas. IT teams can start using the service to deploy applications in less than 30 minutes, he noted.

The first edition of Armory Continuous Deployment-as-a-Service is optimized for deploying microservices-based applications on Kubernetes clusters, but Armory will soon extend the reach of the service to platforms running monolithic applications, said Douglas. The company also plans to make available an on-premises edition, he added.

In either scenario, IT teams will be able to engage in everything from canary testing of deployments to automatic rollbacks in the event an application fails, said Douglas.

The number of organizations that routinely make use of a CD platform is still relatively small. Historically, each platform used to run applications has been unique, so automating delivery has been problematic. CI platforms, in some cases, have been extended using custom scripts to automate application delivery, but those efforts don’t typically scale across an extended enterprise.

Douglas said that as more IT teams start to build and deploy microservices-based applications, it’s only a matter of time before CD becomes a more distinct IT discipline. The challenge is enabling IT organizations to achieve that goal without necessarily having to hire a small army of site reliability engineers (SREs). There will still be plenty of DevOps teams that will prefer to deploy Spinnaker, but Douglas said there will also be plenty of instances where a less complex approach to CD will suffice.

It’s hard to say whether there might soon be varying levels of DevOps platform services that organizations might invoke depending on the complexity of the project and their own internal cultural preferences. However, as the pace of application development continues to accelerate, the need to automate CD will become a more pressing issue. There may be many ways to achieve that goal, but if history is any indication, the path of least resistance is likely to be via the cloud.

June 17, 2022
Threat Modeling as a DevSecOps Practice

Software engineers are always under pressure to build more software, faster. At the same time, there is increasing regulatory and market pressure for secure software that meets users’ and regulators’ requirements for data privacy. This dynamic often puts software engineers at odds with application security or product security teams.

In fact, 81% of developer teams at large enterprises have launched applications that feature insecure code, according to one researcher, with half doing this on a regular basis. The firm argued that developers and security architects need a better relationship, and I couldn’t agree more.

These technical teams are working towards the same goal, so it’s in everyone’s interests—particularly senior leadership—to create a more cohesive working environment by breaking down barriers between teams. If the security of a system is designed and enforced by an external security team—disconnected from the rest of the engineering team—then this only reinforces security as a separate silo. But companies can ease tensions by embedding the responsibility for security in the engineering teams themselves: This is the thinking behind DevSecOps. Thinking about security starts when engineers think about solving the problem: At design time.

While the shift left movement was born two decades ago with the goal of ensuring product testing happened early in the development stage, I believe we need to start left to make real change. In adopting this culture, security and privacy will be less of a bolt-on afterthought and a built-in fixture within the development process, ensuring that engineers and security teams alike are aligned and have visibility of each other’s objectives.

By reimagining the build, test, fix, build, test, fix cycle through building an architecturally secure design in real-time, technical teams can detect software vulnerabilities sooner rather than later, which means more efficient product deployment. This is where the process of threat modeling as a business tool can be of great benefit.

Historically, threat modeling was a static, slow and manual process conducted on whiteboards, as part of a workshop led by a third-party security expert who lacked institutional knowledge of the inner workings of the company. However, this has evolved dramatically into an easily implemented, automated security practice that can be baked into the development and security cycle from the outset, empowering organizations to start left.

Through automated threat modeling, the security workload burden is relieved for both security architects and engineers. Companies are, in turn, better positioned to keep pace with the cadence of ongoing software rollouts, as an automated platform suggests security improvements as an application evolves. The launch phase bottleneck that’s typically created during the testing stage is therefore removed, as security requirements are identified prior to development—which means that risk is managed before a line of code is even written. Based on this, it should quickly become apparent what a time- and cost-saving exercise threat modeling is.

No developer or security specialist wants a flawed product to hit the market, so it makes good business sense to adopt a new approach that will lead to better outcomes. Enterprises that introduce threat modeling as part of their development and cybersecurity processes will be on course to embed a true DevSecOps system within the company, transforming the relationship between these two departments.

In my experience, culture and growth are the key factors for enterprises that decide to adjust their cybersecurity diet to include automated threat modeling. Convincing the app dev teams to invest time into the process is, by far, the most significant cultural hurdle to overcome, because they need to witness the benefits—and up until this point they’re disconnected from the AppSec process.

Once equipped with the tools and knowledge, delivering secure design will be second nature to developers and a new collaborative culture will be created. In essence, the more teams are given oversight of and accountability for software design flaws, the easier it is to recognize how vital these preliminary checks are.

Likewise, there comes a stage of business growth when the ‘security first’ and ‘start left’ mindsets are essential to keep enterprise project management on track—and this ties into the culture. As the rate of scale within a company begins to increase and the pace of software deployment must quicken in line with customer demand and market competitiveness, discovering flaws at the end of the design process just isn’t an option.

With developers and security teams responsible for vast amounts of code, product resilience on delivery is crucial and ensuring a robust DevSecOps culture is the route to success.

June 17, 2022
How to Adopt an SRE Practice (When You’re not Google)
Site reliability engineering (SRE) isn’t a new term or practice. The practice of applying software engineering skills and principles to operations problems and tasks happened even before site reliability engineer was a defined job title. But organizing a proactive approach to building and maintaining software drives long-term success in improving operational efficiency, data-driven roadmap planning and general uptime and reliability. All these advantages account for the widespread adoption of SRE.

In this article, I’ll dive into what it takes to get into site reliability engineering, how to adopt it within your own organization and some of the core principles and best practices you’ll need to keep in mind as you move forward in your SRE maturity journey.

What is SRE?

SRE, or site reliability engineering, is the practice of applying software engineering expertise to DevOps and operations problems. Often, this means proactively writing code and developing internal applications or services to combat reliability and performance concerns. SRE has been a practice for many years but has been more recently popularized by Google when they first published Site Reliability Engineering: How Google Runs Production Systems in March 2016.

SRE is a complicated acronym because it both relates to a job title, site reliability engineer, as well as a general practice adopted by development and IT teams, site reliability engineering. SRE teams are often organized differently depending on the organization and the services they support. Sometimes site reliability engineers (SREs) are sprinkled among development teams. This means they can vocalize concerns during roadmap planning and work closely with QA teams, collaborating on everything from shipping features to staging to managing production environments.

Another approach organizes an SRE team as a standalone group. The SRE team as a whole is focused on responsibilities including monitoring applications and infrastructure, establishing reliability metrics, tracking issues with new releases and handling on-call duties. No matter how they are organized, what remains the same for all these SREs is their initiative—reliability and performance.

DevOps Vs. SRE

The rapid adoption of DevOps practices may have you wondering, “How is SRE different from DevOps?” Commonly, the two terms go hand-in-hand. SRE is a practice that doesn’t necessarily have to be part of DevOps culture and adoption but is often implemented by organizations that are further along in their DevOps journey. The further along on your DevOps maturity model, the more likely you are to have bought into SRE practices.

The conversation isn’t ‘SRE versus DevOps; the conversation is about establishing a proactive SRE team and operating model to help strengthen your DevOps practice. Whether you choose an SRE framework integrated into your development and IT teams, or you want to make SRE a standalone business unit, you need to understand their general responsibilities.

SRE Team Responsibilities

Site reliability engineers are responsible for defining what performant and reliable really mean when you talk about your applications, services and infrastructure. The day-to-day tasks can range anywhere from instrumenting new monitoring solutions to building custom apps for technical support teams. They might ship new code to production to fix bugs or deliver new features, or they might respond to incidents in real-time and work closely with support teams to deliver positive customer experiences.

At the end of the day, site reliability engineers are the key to understanding exactly how customers experience your product and how to track system performance and reliability through your customers’ eyes. SRE teams need to figure out exactly where service boundaries exist between what development teams ship and what customers experience. From there, they need to find ways to monitor reliability and performance concerns in a way that helps your internal teams proactively identify risks and deliver better software.

SRE teams spread knowledge across the product and development teams to consistently define reliability across the entire organization. With everyone on the same page, the engineering teams can make data-driven decisions when it comes to releasing new features or improving current experiences in production.

Site Reliability Engineering Principles and Practices

At a very high level, Google defines the core of SRE principles and practices as an ability to ’embrace risk.’ Site reliability engineers balance the organizational need for constant innovation and delivery of new software with the reliability and performance of production environments.

The practice of SRE grows as the adoption of DevOps grows because they both help balance the sometimes opposing needs of the development and operations teams. Site reliability engineers inject processes into the CI/CD and software delivery workflows to improve performance and reliability but they will know when to sacrifice stability for speed. By working closely with DevOps teams to understand critical components of their applications and infrastructure, SREs can also learn the non-critical components.

Creating transparency across all teams about the health of their applications and systems can help site reliability engineers determine a level of risk they can feel comfortable with. The level of desired service availability and acceptable performance issues that you can reasonably allow will depend on the type of service you support as well. SRE principles and practices embrace experimentation and require a dedication to proactively understanding the health of the services they support.

An SRE Operating and Maturity Model

You can perform a lot of the responsibilities asked of a site reliability engineer and still have a software engineer job title. So, how do you know how mature your site reliability engineering practice is? Luckily for you, we’ll lay out a quick way to build an effective SRE operating model and track your maturity against it. An SRE operating model typically includes three elements, which you might achieve in phases:
1. A team (or at least one person) dedicated to the practice of SRE.
2. Deep integration and influence across product, development and operations teams.
3. Autonomy to automate workflows and write code for nearly any part of your application or system.
Your SRE maturity depends on where your organization falls along these three elements of an SRE operating model. If you’ve taken the steps to make an SRE team or hire your first site reliability engineer, you’re at the beginning of your journey. If you have a team and they are a valued part of roadmap discussions, QA, deployment workflows, incident management processes, then you have a somewhat mature SRE practice.

An organization only reaches complete SRE maturity when the SRE business unit has the autonomy to automate workflows, build applications, own monitoring and alerting solutions, or interject themselves into nearly any conversation. Vocalizing performance and reliability concerns upfront and having a proactive discussion about those concerns is always better than simply ignoring them until it’s too late.

Monitoring, CI/CD and Organizational Automation

Site reliability engineers can and will automate just about anything and everything. If it can proactively detect, remediate or resolve an issue, it needs to be automated. From continuous integration and delivery practices to production environment monitoring, SREs should have some visibility into all of it. If they can identify ways to proactively uncover performance and reliability issues, then they need to have the authority to implement those changes.

Today’s DevOps and IT capabilities around automation, monitoring, artificial intelligence and machine learning give SRE teams a huge advantage when identifying issues, responding to them and fixing them. Organizations with mature DevOps and SRE practices can catch problems in staging and they can also build automated incident management workflows and self-correcting systems. By determining critical components in your applications and infrastructure, SREs can narrow down the scope of things that can cause major problems.

The Practice of Service Levels (SLIs, SLOs and SLAs)

Service levels come into play to help SRE teams communicate the true health of digital products and services to all stakeholders. This is done by identifying and measuring critical components that are key to delivering positive customer experiences. In particular, they need to know when one or more components expose functionality to external customers. We call these intersection points system boundaries. System boundaries are the place where site reliability engineers need to apply service-level indicators and objectives to their metrics in order to tell the real story of system performance and reliability.
- Service-level indicators (SLIs) are the key measurements to determine the availability of a system.
- Service-level objectives (SLOs) are goals you set for how much availability you expect out of a system.
- Service-level agreements (SLAs) are the legal contracts that explain what happens if the system doesn’t meet its SLO.
While SREs aren’t always responsible for managing service levels, it often falls within their purview. By tracking SLIs and tying them to SLOs, you can set goals around the performance of a system. Google’s SRE book defines the four golden signals of service levels as latency, traffic, errors and saturation. So, for example, you could look at an API call and track its number of successful/failed requests (the SLI) against a general percentage of requests that need to be successful for customers to have a good experience (the SLO).

SRE teams will often set strict SLOs on critical components within their applications and services to better understand how strict of an SLA they can agree to with customers. From here, the team can apply error budgets as a way to understand how quickly they must resolve issues in order to stay compliant with their SLOs. Service levels allow teams to aggregate metrics and create a transparent view of uptime, performance, and reliability across the entire organization. At a glance, business leaders can use service levels to monitor compliance across multiple teams, applications, services, etc. to gain a comprehensive understanding of their system’s health.

Adopting SRE Best Practices

Adopting SRE best practices and principles won’t happen overnight. It takes time and effort to proactively monitor your teams and systems for performance and reliability concerns. But, in the end, your DevOps teams and especially your customers will thank you for deciding to take advantage of site reliability engineering.
June 14, 2022
CDEvents Aims To Standardize CI/CD Interoperability
Companies are under mounting pressure to continuously release new applications and features. And as software development life cycles become more iterative, continuous integration and continuous delivery (CI/CD) has become a necessary ingredient to enable more rapid deployments.

Many development teams now embrace DevOps, but fully automating a CI/CD pipeline from source code to production deployment is no easy task. For one, the number of tools in use continues to balloon, meaning CI/CD poses an ongoing maintenance hurdle. Furthermore, CI/CD tools often adopt varying formats to describe their events. A CI/CD pipeline, a form of event-driven architecture, may require a lot of special “glue” to patch events together if the output from each tool is not in a standardized format.

Interestingly, CDEvents, an emerging open source standard from the Continous Delivery Foundation (CDF), is working to standardize the root semantics of how CI/CD tools describe the payloads of their events with more granularity than CloudEvents. The project aims to establish schemas for how common CI/CD events are described, easing integration and providing more actionable data surrounding the CI/CD pipeline.

I recently met with Andrea Frittoli, open source developer advocate at IBM, to understand why the CDEvents project exists, how it works and what they’re hoping to achieve. According to Frittoli, who is also co-creator of the CDEvents project and a member of the CDF Technical Oversight Committee, CDEvents has the potential to bring much-needed interoperability to the sprawling CI/CD tooling ecosystem. Below, we’ll investigate where the project is and consider where it’s heading.

The Problem With CI/CD

At its core, the problem CDEvents is trying to solve is a lack of interoperability in the CI/CD field, explained Frittoli. As it stands today, DevOps engineers must routinely make point-to-point integrations between CI/CD tools. “No single tool will take you entirely from managing source code to production,” he said. A single development team could be using up to 15 unique components in one pipeline, and tooling preferences can vary dramatically across teams, he adds, further complicating CI/CD interoperability.

Although CloudEvents has made inroads as a specification to describe event data, nothing has yet existed specifically for describing CI/CD events. According to Frittoli, CloudEvents falls short here, as it’s agnostic about what is inserted into the payload.

Understanding CDEvents

CDEvents is “a common specification for Continuous Delivery events.” Frittoli describes it as a shared dictionary or shared semantics for common events. These events could be a change in source code, running a test, or a code execution or deploy. CDEvents uses JSON object schemas to describe the event types and can work alongside CloudEvents as an attribute. “CDEvents is a standard format for the payload as well, not only for the envelope,” said Frittoli.

As Frittoli explains, the CDF special interest group is still drafting the CDEvent specification’s exact semantics and common names. Currently, CDEvent organizes these objects into four main buckets: Core events, source code version control, CD events and CI events.

While the group is still finalizing the format of events as they work toward the initial release, the example below is an approximation of what the final format will look like:
```
"meta": {
  "version" : "draft",
  "event_id" : "A234-1234-1234",
  "source" : "/staging/tekton/",
  "type" : "dev.cdevents.taskrun.started",
  "timestamp" : "2018-04-05T17:31:00Z"
},
"subject": {
  "id": "/namespace/taskrun-123",
  "type": "taskrun",
  "content": {
    "URL": ...
    "pipelinerun": ...
  }
},
"extensions": [{
  "name": ,
  "content": ...}
],
"links": [{
...
}]
```
Benefits of CI/CD Event Standardization

So, why is it necessary to define the format of CI/CD event data? What benefits does this bring to developers and the business as a whole? Well, Frittoli sees two potential primary use cases. First off, CDEvents could reduce the effort spent writing and maintaining glue code between common DevOps tools like Spinnaker, Argo or Flux. “This will allow a smoother integration between all the parts,” said Frittoli. Alleviating this could help scale CI/CD initiatives and increase overall agility.

Another positive side effect is collecting more detailed information about workloads and their performances. If all these components generate similarly-formatted events, Frittoli explained, then you can begin to create a true end-to-end visualization of the entire CI/CD pipeline. Then, you can start to measure things such as deployment frequency, failure rate, time to change and other metrics.

If events begin to use a common language, you could do many interesting things with that data, says Frittoli. Event data could help ascertain the root cause of incidents and aid debugging. One could even foresee some AIOps potential to further automate operations and increase observability around CI/CD.

Final Thoughts and Ways to Participate

CDEvents emerged at a time when iterative software development has become commonplace. A slew of helpful CI/CD tools is at our disposal to enable niche operations. Yet, the space still requires a standard way to operate between them. CDEvents is an exciting prospect, yet the project is still in its early stages. It has yet to reach a full release or any production use and proofs-of-concept are still being developed.

Of course, encouraging a vendor-neutral specification requires the tooling community to get on board. “It’s vital for the different tools to implement CDEvents for it to be successful,” said Frittoli. He encourages the tooling community to join the working group to collaborate in building proofs-of-concept. The SIG plans to continue working on finalizing the first version of the specification. In the meantime, the group has produced a proof-of-concept using Tekton and keptn and the project has its future sights set on Knative and Jenkins.

The CDEvents project is in incubating mode and maintained by the Events Special Interest Group (SIG) at the CDF. For more information, you can peruse the CDEvents documentation here or become involved with the community here. The Events SIG also holds biweekly meetings and a CDEvents Community Summit will be part of the upcoming CDcon.
June 14, 2022
AppSmith Adds Git Support to Low-Code App Dev Framework

AppSmith has added support for Git repositories to an open source framework for building custom applications using a low-code platform based on the JavaScript programming language.

Rishabh Kaul, head of marketing for AppSmith, said the Git support will make it simpler for developers to manage version control as they iteratively build applications using a set of graphical widgets provided by AppSmith.

AppSmith is used primarily by developers to build internal applications that automate a specific task or workflow. Support for Git makes it possible for multiple developers to more easily collaborate as they add work to a Git branch and make requests within the context of a continuous integration/continuous delivery (CI/CD) pipeline.

That also makes it simpler to test applications and roll back to a previous version in the event something should go wrong, said Kaul.

The Community Edition of AppSmith will make it possible to connect to unlimited public repositories and up to three private ones before requiring developers to upgrade.

In recent years, use of low-code tools has increased dramatically as organizations embrace tools that enable them to build applications faster. Not every application requires developers to employ tools for working with procedural code. There are typically hundreds of use cases where an organization could build and deploy an application that would eliminate the need for an internal team to license yet another packaged application.

In general, AppSmith and other open source tool and framework providers are betting that, as the economy continues to shrink, the adoption of these tools will increase as organizations continue to become more sensitive to costs. That approach will allow them to continue to provide applications tailored to automate specific tasks.

The challenge, of course, is that as the number of applications being built and deployed steadily increases, the DevOps workflows that regularly update, secure and maintain those applications become much more challenging.

It’s not clear what percentage of applications are now being built using some type of low-code tool. However, the overall size of organizations’ application portfolio is generally much higher than it was prior to the pandemic. Many of those applications are driving digital business processes intended to make the workforce more efficient at a time when many employees continue to work from home.

Less clear, however, is how many new applications are being built by so-called citizen developers. One way or another, however, it’s becoming easier for end users that have some basic software understanding to build applications. Inevitably, those application development efforts will also need to be integrated within the context of a DevOps workflow.

Regardless of how an individual DevOps team may view low-code tools, they have permanently altered the application development landscape. The issue is no longer whether low-code tools will be used by developers but rather to what extent. In the meantime, DevOps teams would be well advised to review just how extensible their existing pipelines need to become.

June 13, 2022
Why is Security Still in the Way? A Look at DevSecOps Right Now
According to industry trend reports for 2022, DevSecOps is now considered to be one of the most effective approaches to building software quickly and securely. This effort, of course, means development, security and operations teams commit to addressing security as early as possible in the software development life cycle (SDLC). The goal of the shift into DevSecOps revolves around the need for more automation, streamlined CI/CD pipelines, less friction among teams and plenty of time for productive innovation. Incorporating DevSecOps practices into an organization unites security, compliance and development issues into a collaborative concern, one that allows DevOps to move fast without jeopardizing code quality.

As we approach the middle of 2022, it’s worth looking back and considering why security is still standing in the way of smooth software development. Security vulnerabilities have always been present in the process, and yet these pesky bugs continue to be the biggest threat to software performance. The question is, why?

The Truth About Security

Software has always been deployed and shipped with some existing bugs. But as software and services continue to dominate our everyday lives, the reality of this statement continues to plague both businesses and the end user. On some level, software development has yet to evolve into the next phase of security. Code is still rolled out regularly, only to discover security vulnerabilities in production. Some say the way software is developed today actually lends itself to defects, mostly because the pace of work required supersedes best practices around AppSec.

While the larger cybersecurity industry—and every DevSecOps team in the world—is striving to make software more secure, many organizations still prioritize speed and productivity over digital risk. This means the state of security (and, to some extent, the DevSecOps world) has become somewhat lopsided, with some enterprises making great efforts while others embrace a more lackadaisical approach. To make matters worse, many professionals today believe there has been a general under-investment in cybersecurity over the past several decades.

The Next Phase of DevSecOps

The shift to remote and hybrid work models has led to mass adoption of cloud technologies and digital transformation, across all sectors and industries. As a result of this evolution, organizations today need to bear in mind that the more services they use, the more digital threats and overall risk they face. As this exposure to vulnerabilities increases, so does the need for new approaches to security.

The majority of modern enterprises do, in fact, have a security process in place, as well as other risk management solutions. But with the cybersecurity reckoning of 2021 and the stakes of cloud computing being higher than ever, businesses right now are feeling the pressure to think long and hard about why security still isn’t a guarantee. The trend of slowing down software development to address security more thoroughly, also known as the secret sauce of DevSecOps, is not free from roadblocks—but it is also far from optional. For many organizations, the implementation of DevSecOps practices feels less like security and more like speed bumps that just slow down the shipping of software.

Even so, the effort to improve security is happening now. Organizations do not need several solution platforms or tools to get started. They can begin right away; evolving their security tools and practices as they grow the business. With this growth comes one of security’s biggest challenges—scalability. Because toolchains across the cloud have become more and more complicated, setting policies and/or workflows in one tool is challenging. While these issues can be largely addressed by the adoption of an end-to-end security solution, organizations must continue to focus on strengthening supply chains, improving security oversight, and bolstering their approach to DevSecOps. Some ways to make this happen include:
- Making sure CI/CD pipelines are integrated with DAST and SAST testing in development
- Ensuring developers and DevOps teams complete vulnerability scanning before commit or merge, as they write code
- Minimize manual security as much as possible with automated AI/ML tools to support more effective scanning and monitoring
- Invest in management solutions that improve security in a multi-cloud environment
Addressing the Supply Chain

According to an industry survey, 60% of companies have decided securing the software supply chain should be a top priority for 2022. When we think about software supply chain attacks, we need to think about better controls to validate its integrity through all stages of the SDLC. Obviously, DevOps and automation have provided considerable help in this area; however, they cannot pull all the weight of security without proper governance and guardrails.

With a secure supply chain comes better security, as well as compliance, privacy and transparency. When a supply chain is secure, it can automatically enforce policy requirements for things like instrumentation, testing and architecture. As a result, software development and deployment become a repeatable process that is also reliable. Teams across the organization can rest assured that the products and services they are building comply with security expectations, such as validating code before production with static analysis and scanning tools.
June 10, 2022
Technical Debt: When It Works and When It Doesn’t
Every software team has probably faced technical debt—the accumulation of little development deficiencies that will require rework down the line. Just like financial debt, they accrue interest as they build up, in the form of difficult software modifications or challenges in adding new features. Time is the usual culprit; your development team has one timeframe for creation but your leadership team wants to bring the software to market faster. And when it comes to choosing between building a clean and polished piece of software or hitting that delivery date, you usually need to bow to pressures from on high. It’s at that point that your collaboration might go off track, with little pieces of incomplete code jumping into the mix.

Without careful monitoring, technical debt will eventually slow down your development and deployment processes, degrade your product quality and limit your ability to innovate in a changing market. But is technical debt unilaterally bad? Not always. Sometimes it’s in service to a greater strategy. Let’s look at the differences.

When is Debt Part of a Good Strategy?

Pankaj Gupta, senior director, product marketing at Citrix, said when technical debt impedes an organization’s ability to function, that’s a clear sign they need to wipe out the debt and start with a clean slate.

“But there are a few scenarios under which it makes sense to let debt accumulate,” he said. “For example, if the cost of addressing a technical debt is significantly higher now than in the future; if the debt is not impacting your immediate and short-term business needs or if you have an emergency release like a major security vulnerability fix.”

Daniel Viveiros, chief technology officer at CI&T, recommended that teams keep the big picture in mind when making the right tradeoffs. “For me, it’s about putting the business first and understanding that constantly activating value is imperative for success,” he said. “Well-managed technical debt can be an effective tool to shorten the lead time. Deferring refactorings, modernization efforts and other common debts in order to prioritize deployments is a constant tradeoff the engineering teams deal with on a daily basis.”

What are the Signs of Too Much Technical Debt?

Gupta provided some warning signs for technical debt overload. “You know you’re accumulating too much technical debt when the cost and/or time to fix the technical debt is higher than starting fresh,” he said. “And the time taken for each release and deployment is consistently increasing.”

A less obvious result: a DevOps exodus. The development team will be the first to feel the impact of too much technical debt, such as having a harder time completing tasks and working long hours to fix frequent breakages. “Employees are leaving the company out of frustration from working on legacy systems, spending most of their time fixing ad-hoc issues, routinely missing and being on-call all the time,” Gupta said. What debt reaches that level, resignations begin to mount.

Viveiros pointed out that compiled debt will impact system resilience: “Bugs are originated more often and the change failure rate goes up. Speed is perceived as risky instead of something beneficial to the digital delivery process.”

The ultimate sign of too much debt, he said, is poor business performance and lost revenue. “If downtime equals lost opportunities, the technical debt will become a financial debt,” Viveiros said. “Too much technical debt may impact the perceived quality of the product. As a result, the product and brand’s image will suffer.”

What’s the Best Way to Manage Debt?

Failure to deal with technical debt means kicking it down the road to another developer to tackle—and no one wants to be that person. So what’s the best way to take control and pay it off? To start, approach it as a collaborative effort and implement these four practices.
1. Identify the types of debt. Not all technical debt is created equal. Distinguish between the debt you accept in the moment as something to fix later on and the inadvertent debt you discover—and categorize it into code debt, design debt, build debt and other forms to identify origin patterns.
2. Analyze and automate. Analyze the origins of your debt and look for ways to tighten workflows or automate certain tests and processes. By reducing common errors and hidden bugs, you can stop them from snowballing into technical debt.
3. Develop new policies and standards. These should clarify when debt makes sense and when it causes unacceptable damage. For instance, an immediate release of a security patch would be deemed acceptable while errors that will eventually cause considerable downtime would not.
4. Communicate the cost to IT and business leaders. When another hyper-speed deadline comes your way, make sure your leadership understands the risks to product quality and developer retention. With the price tag made clear, they’re more likely to change delivery dates or provide funding for additional developers.
Like most out-of-control spending, harmful technical debt begins with poor decisions and a certain level of denial. By approaching decisions with greater clarity and foresight on both the Dev and Ops sides of the house, teams can give themselves more breathing room and the space to produce exceptional software.
June 9, 2022
CDF Report Surfaces DevOps Workflow Gains

At its cdCon 2022 conference today, the Continuous Delivery Foundation shared a State of Continuous Delivery in 2022 report that found nearly half of developers (47%) use either continuous integration or continuous deployment platforms—but only about 20% report using both to automate all application build, testing and deployment.

Overall, only a little less than a quarter (23%) of the 18,585 developers surveyed by SlashData on behalf of the CDF said they are not involved in any DevOps-related activities.

Fatih Degirmenci, general manager for the CDF, said that the relatively low percentage suggested that DevOps workflows are now more widely employed than ever. Monitoring software and infrastructure performance and using continuous integration to automatically build and test code changes remain the most popular DevOps-related activities cited in the survey, at 36% each.

The survey also found that, among large enterprises, the proportion of low performers when it comes to lead time for code changes has dropped from 34% to 24% in the last six months, while the proportion of top performers that need less than one day for code changes has also increased from 13% to 21%.

Larger enterprises are also seeing an increase in developers that report they can restore a service in less than an hour (22%) versus those that still require one week (18%), according to the report.

The survey also suggested there is a strong correlation between the adoption of modern backend technologies such as microservices, containers and Kubernetes and improved lead times for code changes and time to restore. Survey respondents using containers are twice as likely to have lead times of less than one day, while respondents that have adopted Kubernetes are one-third more likely to be top performers.

However, the survey found no impact on deployment frequency. The percentage of top performers capable of deploying code multiple times per day is 11%, the report found. Developers who don’t use these technologies are, in fact, about twice as likely to deploy multiple times per day. Overall, only 53% of developers that work in backend development are using microservices.

Degirmenci said it is clear there is still much work to be done in terms of simplifying DevOps workflows. The average number of technologies that DevOps practitioners use now stands at 4.6 in the first quarter of 2022. The CDF last month launched a CDEvents project to create a vendor-neutral specification for defining a format for sharing event data across multiple services, platforms and systems.

In the meantime, the CDF survey also noted more than a quarter of respondents (26%) now rely on managed continuous integration/continuous delivery (CI/CD) services rather than building and maintaining a DevOps platform themselves.

It’s too early to say how much complexity is driving organizations to rely on managed services, but as the pace at which applications are being developed continues to increase, more organizations are starting to focus on developer productivity. As a result, more than a few organizations are reviewing their existing approach to DevOps and whether it is actually a bottleneck rather than an accelerant for building and deploying applications faster.

June 7, 2022
When DevOps-as-a-Service (DaaS) Meets Security

One of the most recent IT methodologies to be offered as a service is DevOps, a cultural and practical approach that brings development and operations teams together under one umbrella of work. DevOps-as-a-Service (DaaS) ensures that everything related to the selection, management and maintenance of DevOps tools and infrastructure, including all policies and procedures, are addressed through a centralized team who are able to provide this service to development teams across the organization.

From a 30,000-foot view, DaaS represents DevOps capabilities made accessible to practitioners through portals and APIs. Because DevOps capabilities vary among organizations, there is no standardization of these services. For the business, DaaS enhances competition. Application users and developers find DaaS improves overall software development, while specialized DaaS teams say it improves the management of DevOps capabilities.

What Are the Other Benefits of DaaS?

DaaS offers many benefits, especially to development teams who no longer want to spend all their time configuring and deploying the necessary security tools to run DevOps pipelines. This means it’s no longer necessary to understand all the differences between DevOps tools and the underlying infrastructure, as the appropriate tools and technologies are automatically deployed for development teams. As a result, developers are able to focus their efforts on creating innovative software at the speed of business.

Further, Daas delivers clear business and operational benefits in the form of a more streamlined process for facilitating DevOps across the enterprise. Daas supports governance while also enabling centralized oversight and visibility into development workloads and productivity. As we know, this shift leads to increased accountability among teams and an overall optimization of business processes. Add on resource efficiency and cost savings through tool selection, and it’s easy to see why DaaS has found a home in the world of software development.

But despite the benefits, Daas also presents new security challenges that can’t be fully addressed by traditional management solutions. These approaches are often sluggish, expensive, and/or incredibly complex to manage.

What About Security and DevSecOps?

Because the principles of DevSecOps proactively address the risks and threats associated with DevOps processes, it has seen rapid adoption in recent years. By integrating AppSec tools and strategies within a DevOps pipeline, developers are able to find and fix security vulnerabilities earlier within the software life cycle—not after the fact. This ability is what allows developers to generate secure and high quality software from the beginning, while also preventing any deployment delays stemming from last-minute security checks in production. With DevSecOps processes in place, AppSec testing becomes seamless and transparent for developers, who need a frictionless experience to facilitate steady and reliable software development and deployment.

DevSecOps is a natural fit for the as-a-service delivery model. Similar to DevOps, AppSec involves various tools, policies and processes that demand a certain degree of professional expertise—not to mention the time and resources it takes to to select, maintain, and deploy them to drive DevSecOps practices. Security and development teams looking to implement and manage DevSecOps can reap major benefits from a centrally defined and managed process, to be administered by a specialist team and delivered “as a service” to those in charge of DaaS.

Standardizing AppSec through a DevSecOps-as-a-Service offering would guarantee security is applied evenly across all DevOps activity, thereby allowing businesses to produce secure and high-quality software consistently while also supporting AppSec compliance demands. This shift also establishes a foundation where a holistic and homogenous view of AppSec can be found, one that is critical to the assessment and management of application and business risk.

Developing DaaS Metrics

First, don’t try to boil the ocean by measuring all applications at the same time. Instead, choose an application that’s supported by DaaS processes and appoint a specialized team to develop metrics for the application. This team can look many ways—but would likely include stakeholders, developers and other DaaS-supporting roles. Service-level indicators (SLIs) and service-level objectives (SLOs) are also baked into this model, which are chosen to best suit the requirements of an individual application.

Once the most valued SLOs of DaaS are determined, the service can be tailored to identify the most critical SLIs. With this knowledge, things like instrumentation, telemetry and analysis requirements can be identified and used to design alerts, dashboards, analyzers and more. Next, setting up policies and procedures around accountability establishes a complete solution. And once the solution for the chosen application is tested, confidence kicks in and creates an easier process for adapting and expanding the process around other applications.

This means DaaS can provide many key capabilities to both the organization and the teams who rely on them. A specialized DaaS team can then figure out the best metrics for larger implementation on any application. As a result, SLOs can be evolved for other future applications, to be used as a baseline for future improvements.

June 7, 2022