devopsy.kinsta.cloud

Tag: devsecops

AppSec Marketing in the Age of DevSecOps
A while back, I had a conversation with a friend I went to school with (currently a senior member of the engineering team at a large retail chain) who was tasked with the job of identifying potential application security partners (he addressed vendors as partners, which I personally liked) to collaborate with on various areas as part of their product security initiative. The following piece emerged as an extension of my immediate thoughts when he shared his views of what could have made his experience of interacting with front-line sales and marketing folks better.

In the context of DevSecOps, much has been said about the need for engineering to speak security, security to speak code, DevOps to speak security, etc. But as a technology service provider (TSP) riding the current wave of application security, it’s almost mandatory for the sales and marketing teams to also speak relevant tech!

Application Security Is Not One-Size-Fits-All

Application security, as a practice area, is dynamic. No two applications are the same, even if they belong in the same market domain, presumably operating on identical business use cases. Some (of the many) factors that cause this variance include technology stack of choice, the programming style of developers, the culture of the product engineering team, priority of the business, platforms used etc. This, consequently, results in a wide spectrum of unique customer needs.

Take penetration testing as an example. This is a practice area that is, presumably, well-entrenched both as a need and as an offering in the application security market. However, in today’s age, even a singular requirement such as this could make or break an initial conversation. While for one prospect, the need could be to conduct the test from a compliance-only perspective, another’s need could stem from a proactive software security initiative. There are many others who have internal assessment teams and often look outside for a third-party view. Others who are further along the maturity curve could be looking to up their game through a hybrid approach of tool automation with a complementary method of manual assessments. I’m not even considering the added complexity involved in the nomenclature of such a service—is it called penetration testing, security testing, vulnerability testing, VAPT (which actually is a combination of two independent practices) etc.?

Each of these unique needs emerges from buyer personas which emerge from varying degrees of informed decision making. TSP’s would need to retrofit their positioning accordingly. They often run the risk of underwhelming a mature buyer or overwhelming an early practitioner, especially in high variance offerings such as security tooling, security regression and threat modeling, for example. While some might argue that losing an overwhelmed prospect could be the result of their accurate customer segmentation, there are other reasons for this.

Scoping questions, such as the ones below, can significantly help technology marketers strike the right chord with their prospects and elevate the experience of the initial interaction.
- What is the motivation for the penetration test? Is it compliance regulation, internal validation, business drivers, their customers’ needs, etc.?
- What are they specifically looking from a third-party partner? Is it external certification, a specialized approach, uncovering logic flaws, etc.?
- What is the current appetite (measured in resource bandwidth, commerce) to take on your advanced offering? (Such as automation, regression etc.?)
- How security aware are the developers? Can they take the findings to their logical conclusion through successful remediation?
DevSecOps Is About the Journey, Not the Destination

Ever since the surge of DevSecOps, marketers and practitioners have been vocal about the possibilities and advantages that smart automation brings with it. Think tanks, too, have statistically alluded to its benefits in terms of cost savings and bandwidth efficiency, among others. Though recent marketing campaigns have rather effectively communicated the ‘what of DevSecOps and AppSec automation, more awareness of the ‘how’ of DevSecOps is needed. TSP marketers need to design and propagate content on use cases focusing on the implementation challenges and suggested how-tos. Such content not only helps build trust and credibility but also allows segues for tech marketing to collaborate with security engineering. Some of these include practical guides on open source tool automation and sample automation scripts and libraries, data sheets on resource optimization through automation and handbooks on vulnerability remediation, just to name a few.

This collaboration between marketing and sales and security would provide an opportunity for the former to get themselves well entrenched in the practical workings of the service or solution that they are responsible for positioning. Ironically, this also allows them to appreciate constraints that would prevent them from over- or under-committing business value to their prospects through their messaging. This is especially more relevant with the numerous myths that surround application automation and DevSecOps in general.

The DevSecOps Rolodex

In 2012, I had the opportunity to meet with one of the senior CISOs in the industry. After some initial small talk, I was introducing what we did and how we thought we could help his team. I was 70 seconds into what was my planned 180-second pitch when he respectfully stopped me, pulled out a rather impressive Rolodex from his desk and said, “I could point you to 15 companies in a 20-mile radius who could help me with exactly what you’ve offered me. I wonder if there’s any secret sauce?”

Startled, embarrassed, I managed to gather my thoughts and moved to the final 30 seconds of the pitch, though with much hesitation. As luck would have it, he found little bits of the secret sauce. The meeting obviously did not go exactly as I had planned, but it didn’t go badly, either. However, it made me realize that our customers and prospects often know just as much—or even more—about our competition than we do. Everybody is selling, all the time. So, someone else has already made their 70-second pitch. Not everyone gets lucky enough to be given an opportunity for a pause! It’s up to us to find the secret sauce and make that close in the final 30 seconds.

We are riding a very healthy wave of application security focus right now, and it’s paramount that technologists and marketers collaborate to identify and nurture their own secret sauce in their messaging and positioning. After all, this is what is going to keep the person at the other end of the table from reaching out to others in their DevSecOps Rolodex!
June 23, 2021
Google Proposes SLSA Framework to Secure Software Supply Chains

Google is proposing organizations adopt a framework for securing the integrity of software artifacts across a software supply chain.

Kim Lewandowski, a product manager for open source software security at Google, said the Supply Chain Levels for Software Artifacts (SLSA) is based on an internal framework, known as binary authorization for Borg, that the company has been employing now for more than eight years to secure its software. Borg is the name Google attached to the platform for building and deploying applications upon which Kubernetes is based.

In its current form, SLSA is a set of guidelines that identifies how to attain four different levels of application security. The goal, however, is to create a framework that will automatically generate auditable metadata that can be fed into policy engines to create a SLSA certification.

The lowest level of SLSA requires that the build process be fully scripted or automated and generates metadata that enables provenance to be created. At the highest level, SLSA requires a two-person review of all changes made during a hermetic, reproducible build process. At the highest SLSA security level, organizations should be able to assure others that their software supply chain has not been compromised, said Lewandowski.

In the wake of a recent series of high-profile breaches of software supply chains, the level of scrutiny being applied to how software is constructed has increased significantly. In theory, at least, developers have assumed more responsibility for application security as part of an overall shift left that provided them with more programmatic control over IT environments. As a result, cybercriminals are now focused on compromising either the credentials of developers as part of an effort to insert malware into application as it is developed, or they are trying to exploit vulnerabilities that developers have inadvertently created by programmatically misconfiguring infrastructure.

DevSecOps best practices are, of course, being adopted to teach developers how to better secure their application environments. The issue is that not enough developers are learning how to secure applications fast enough. The volume of attacks against the software supply chain continues to grow at a rate that is now much faster than organizations can thwart simply by educating developers. In some cases, organizations are once again empowering cybersecurity teams to make sure software supply chains are secure regardless of the potential impact on the rate at which applications are being developed.

Google is trying to promote adoption of SLSA as a framework for securing both open source and proprietary application code, said Lewandowski. There’s no silver bullet for securing software, but a framework that ultimately automates as many of the security processes as possible within the context of a DevOps workflow would go a long way toward hardening the software supply chain, noted Lewandowski.

SLSA is being put forward as a response to a cybersecurity executive order issued by President Biden that, among other things, tasks the National Institute for Standards and Technology (NIST), an arm of the U.S. Department of Commerce, to consult with federal agencies, the private sector, academia and other stakeholders in identifying standards, tools, best practices and other guidelines to enhance software supply chain security. Given the level of dependencies that exist across those software supply chains, that executive order, to varying degrees, now applies to almost any entity that builds software for an organization that, on some level, interacts with the Federal government.

June 21, 2021
Accurics Aligns DevSecOps Platform With GitLab

Accurics today announced it has integrated its tool for discovering violations of security policies that occur when developers provision infrastructure as code with both the continuous integration and continuous delivery (CI/CD) platform and the static application security assessment testing (SAST) tools from GitLab.

Om Moolchandani, chief information and security officer (CISO) and CTO for Accurics, said both integrations make it easier for developers to discover security issues earlier as part of a DevSecOps workflow using the company’s Terrascan tools.

Many of the issues organizations are having with cloud security these days can be traced back to misconfigurations created by developers when configuring infrastructure using tools such as Terraform. Accurics created Terrascan to identify those misconfigurations.

The integration with GitLab makes it easier to incorporate Terrascan into a DevOps workflow in a way that also aggregates data collected from both SAST and dynamic application security testing (DAST) tools, said Moolchandani. That approach effectively unifies what today are two separate cloud infrastructure and application development pipelines by enabling DevOps teams to employ threat scores to enforce security policies as code that are deemed too risky to deploy with block builds, he added.

At the same time, the integration with SAST and DAST tools provides the context developers need to prioritize remediation efforts before applications are deployed in a production environment, noted Moolchandani.

Organizations of all sizes are now trying to strike a balance between two conflicting agendas. On the one hand, infrastructure-as-code (IaC) tools such as Terraform have played a critical role in enabling developers to build and deploy applications faster. The issue is that developers lack the security expertise required to ensure infrastructure is secured properly at a time when cybercriminals are more aggressively seeking to compromise software supply chains. Organizations most likely won’t slow down the rate at which applications are being deployed to make sure software supply chains are not compromised. However, in the absence of best DevSecOps practices—which still are not widely implemented—there may be a backlash against shifting application responsibility left toward developers.

The challenge that creates is most organizations don’t have enough security expertise available to review applications in a timely manner before they are deployed, which results in them hoping security issues will be discovered and remediated during the application update cycle before cybercriminals find a way to exploit a vulnerability.

Of course, hope does not make for an application security strategy. Organizations will need to find ways to enable developers to better secure applications while simultaneously making it easier for cybersecurity teams to maintain a zero-trust IT environment that reduces the chances organizations will be breached via, for example, a phishing attack to steal developer credentials.

Regardless of how DevSecOps workflows and zero-trust IT architectures are implemented, it’s clear organizations have run out of time to resolve longstanding security issues that are now making the kinds of headlines no one wants to see.

June 14, 2021
Majority of Orgs Lack Visibility Into Container Vulnerabilities

Today’s blend of third-party application dependencies and polyglot software development often makes assessing risk difficult. With many new cloud-native deployment models, it can be tricky to discover potential vulnerabilities. These threats take the form of insecure default settings in Kubernetes, over-permissive states, CVEs that threaten container integrity, and other vulnerable conditions.

Plugging gaps throughout the cloud-native strata is now crucial to avoid exposing data and breaking privacy regulations. Yet, gaining visibility into these holdings is challenging, and traditional application security practices may not cut it within a cloud-native context.

A new 2021 CISO Report released by Dynatrace, “Precise, automatic risk and impact assessment is key for DevSecOps,” demonstrates increased false positives and a general lack of real-time container runtime vulnerability scanning among enterprise IT departments worldwide. Of the 700 CISOs surveyed, most report that microservices, containers and Kubernetes have caused security blind spots across their organizations.

Below, we’ll cover the major takeaways from the study to see why new cloud-native development models are leaving security gaps. We’ll consider how CISOs can increase security throughout the cloud-native IT stack while still reaping the rewards of container-based delivery models.

Cloud-Native Security Blindspots

Adopting new container-based technologies may introduce multiple nuanced security implications. Of the tech leaders surveyed, 89% report that microservices, containers and Kubernetes have created new vulnerabilities within their organizations.

Engineers are shipping code to maintain release velocity and to meet the demand for digital innovation. In the process, 28% of CISOs say application teams sometimes bypass vulnerability scans to speed up software delivery. Due to these realities, it’s no surprise that 71% of CISOs admit they are not fully confident code is free of vulnerabilities before going live in production.

As some teams bypass security pre-production, it follows that more aren’t applying holistic security scanning during production. As a result, nearly all organizations (97%) lack real-time visibility into runtime vulnerabilities, found the report.

Adding to the number of cloud-native blindspots is information overload. The report calculated that, on average, an organization receives a staggering 2,169 new alerts of potential application security vulnerabilities each month. Out of this high volume of alerts, 77% of CISOs say they are mostly false positives and not actual exposures.

Due to minimal testing and a high number of false positives, organizations are likely to leave many vulnerabilities unpatched. To make matters worse, developers, pressed for time, may lack the bandwidth to resolve known issues.

Faster Delivery, Harder Detection

Much of the setbacks in vulnerability detection may lie in a general shift toward new ways of working and faster delivery cycles. For example, 63% of CISOs reported that DevOps and agile methodologies are making it harder to detect and manage vulnerabilities.

In addition, 74% of CISOs say traditional security controls such as vulnerability scanners no longer fit today’s cloud-native world, and 77% of CISOs agree that automation could help. This could involve replacing manual deployment, configuration and management with more automated approaches. Others agree that security-as-code is required to ensure high container security best practices.

Another result of more frequent delivery is increased over-reporting. Of the total security vulnerability alerts received per month, CISOs report that only 42% of these alerts require action. More rapid release cycles could thus exacerbate already crowded vulnerability detection systems.

Who Owns Vulnerability Management?

Only 3% of organizations have real-time visibility into runtime vulnerabilities in containerized production environments, the study found. In reality, vulnerability scanning in production is only happening once a month or less within 68% of organizations.

To compensate for this gap, most CISOs believe the onus is not on the security team but the application and DevOps teams to manage and respond to vulnerabilities—85% of CISOs believe these groups should hold responsibility for vulnerability management. These findings come to light as other stakeholders advocate for increased full-cycle developer control and encourage the DevOps mindset across roles.

Handing more control to developers sounds good, yet, 64% of respondents say developers don’t always have time to resolve vulnerabilities before code goes into production. As a result, over one-quarter of application teams bypass vulnerability scanning entirely. Often, engineering teams may not frequently sync with security groups, and DevSecOps principles are not yet pervasive throughout all organizations.

The report also highlighted the fact that existing security tools are often ineffective; 69% of CISOs say security tools slow down DevOps because they only focus on one state of the software delivery cycle.

Cloud Use Requires Security Repositioning

As cloud architectures rise in prominence, organizations are adopting cloud-native tools like microservices, containers and Kubernetes. There is also the chance of multi-tool conditions further complicating the vulnerability scanning process. With more systems to protect than ever before, teams may have to reposition their approach to vulnerability detection.

“The increased use of cloud-native architectures has fundamentally broken traditional approaches to application security,” said Bernd Greifeneder, founder and chief technology officer at Dynatrace. “This research confirms what we’ve long anticipated: manual vulnerability scans and impact assessments are no longer able to keep up with the pace of change in today’s dynamic cloud environments and rapid innovation cycles.”

The Global CISO report, “Precise, automatic risk and impact assessment is key for DevSecOps,” from Dynatrace, queried 700 chief information security officers (CISOs) on their monitoring practices concerning vulnerabilities. To review the report’s full findings, you can download it here.

June 14, 2021
Get the Benefits of Low-Code Without the Risks

Today, more than 300 vendors and various platforms offer different flavors of low-code solutions. However, the majority of these low-code tools are really no-code tools that benefit individuals or groups trying to solve a specific program. The list of benefits of low-code tools in your environment is long and compelling; however, there are security risks that businesses should be aware of when working with these powerful tools.

When evaluating or using a low-code tool, it’s important to remember that while low-code tools abstract the process of writing code, code is still being written. Applications created with low-code tools are susceptible to the same attacks as any other web application. However, companies don’t have control over the code generated by a low-code tool, so the burden of developing secure applications is largely shifted to the low-code tool vendor. Therefore, the best way to minimize risk when using a low-code vendor is to properly evaluate that vendor’s security practices and test applications created using these tools as vigorously as you would any application developed using a traditional/pro-code approach.

Risks to consider include access control and authentication (permissions, multifactor authentication), data security (encryption in transit and at rest) and application security from vulnerabilities and attacks (network security and intrusion detection). Other risks to consider are risk mitigation covering areas such as DDOS (distributed denial of service) attacks, backup and disaster recovery, and security incident handling. Responsibility for management of these risks belongs to the low-code vendor, the hosting provider and the company using the low-code tool.

The following tips should be considered when bringing new low-code tools into your environment to minimize security risk. Remember, it’s critical to perform due diligence of the vendor. Items to consider include:

Hosting Platform: Be sure the vendor is using a reputable hosting provider, ideally Amazon Web Services (AWS), Google Cloud Platform, Salesforce or Microsoft Azure. Leading players will offer secure, highly available platforms that address the aforementioned risks and will have completed third-party audits and certifications demonstrating their commitment to security.

Vendor Infrastructure Management: Low-code vendors typically have some level of responsibility for management of the upper tiers of the overall stack; this is generally referred to as the shared security model. Review the vendor to ensure they have solid controls and processes in place to properly manage and secure the infrastructure. Ideally, the vendor regularly submits to some form of recognized audit, such as a SOC 2 Type 2 or ISO-27001. This helps ensure controls and processes are in place in areas such as security, availability, confidentiality, privacy and processing integrity to address the aforementioned risks in the area of the hosting platform the vendor is responsible for.

Formal SDLC: The vendor should follow a documented software development life cycle (SDLC) and change management process used for the development of their low-code tool. A good SDLC process includes using a source code management system, dependency checking process, static code analysis, automated unit and regression tests, peer code review, secure coding practices training, application vulnerability scanning, quality assurance testing process and third-party penetration testing.

Policies and Processes: Documented policies and processes covering information security, business continuity, disaster recovery, security incident handling, infrastructure hardening and more should be in place at the vendor. This helps demonstrate that the vendor is committed to providing a secure, highly available and high-quality offering.

The company adopting the low-code tool itself should also consider internal processes and how it’s using the low-code tools. Items to consider include:

Formal SDLC: Even though the company uses a low-code tool, there’s still the need to follow a formalized SDLC. It’s important for the company to emphasize and embrace secure programming practices, even for low-code developers. Low-code tools have the ability to implement customer-developed code snippets or components and it is recommended companies follow programming standards and secure programming practices as outlined by the Open Web Application Security Project (OWASP) foundation. Another area of emphasis that can sometimes be overlooked is the need for testing, including functional and security-oriented testing.

Security Testing: Consider performing a penetration test of the completed low-code application. While ideally, the vendor has performed these tests on the tools itself, it’s a good idea to also perform one on the completed low-code application.

Access Control and Authentication: Configure access and permissions according to the least privilege model. Leverage advanced authentication mechanisms like multifactor authentication and single sign-on where possible. Ensure an access removal process is in place to remove access for terminated employees.

Data Security: Verify the application is configured to use HTTPS and that all databases are encrypted. Low-code vendors may or may not provide a database, and either way, the company likely has some level of responsibility for securing the data, regardless of who is hosting it.

Policies and Processes: Documented policies, processes and related training for team members helps mitigate risk. Some level of formality is recommended for documentation and training for the SDLC, business continuity planning, disaster recovery and security incident handling. For example, the team responsible for the low-code application needs to know how to handle a security incident if one occurs, whether it’s due to an issue in the low-code application, the low-code tool or the hosting infrastructure.

The benefits of low-code tools far outweigh the risks of introducing these tools into your environment, but with a little effort up front, you can save a lot of headaches down the road.

June 10, 2021
Why API Quality Is Top Priority for Developers

It is no secret that web APIs have become increasingly important to the operation of modern businesses. According to RapidAPI’s Developer Survey and Insights report, 61% of developers used more APIs in 2020 than 2019, with 71% planning to use even more in 2021. We are no longer tied to a small number of monolithic service providers. Businesses tend to gravitate toward specialized managed services, such as billing and authentication, which are backed by APIs. Business models for new products and services are constructed based on APIs. Trust has become a necessity for APIs. APIs that perform consistently and with high quality earn trust, and those that do not are abandoned for a competitor.

Why Does Quality Matter?

If a market exists for an API, it’s a given that there will be competition. To remain competitive, APIs must build and maintain trust with their consumers; sales and marketing can only go so far. The highest quality APIs will inevitably capture the largest slice of market share. Quality isn’t a destination, either—it is a collection of goals that require continuous investment.

What Does Quality Mean for an API?

Just like the code that drives an API, the quality of an API itself is quantifiable. The following facets describe some broad categories used to measure API quality. The weight of each facet will vary depending on how business-critical the API will be when in production.

Resilient. An API is resilient when it continues to operate in the face of adverse conditions. Can an API continue to operate (even at reduced capacity) if the database is down? How does it handle network traffic spikes?

Robust. A robust API is liberal in what it accepts, but conservative in what it sends. Will it crash when presented with a poorly formatted request, or respond with instructions on how the caller can fix the problem themselves? Can the code be restructured without breaking the API contract?

Secure. API security is vital in most deployments. Are efforts made to check for OWASP Top 10 Web App security risks? Are open source vulnerabilities known and addressed in a timely manner? Are secure endpoints gated with authentication?

Discoverable. APIs that are well documented and well structured are much easier to consume and integrate with other services.

Consistent. A consistent API is one that changes without negatively impacting existing consumers. Changes are backward compatible or versioned as new features are introduced. API consumers should know far in advance if a breaking change will be coming if it cannot be avoided.

How Do I Get There?

Processes, tools and, most importantly, culture, are the key to high-quality APIs. Let’s explore how we can approach each API quality facet.

Resilient. Processes such as chaos engineering, load testing and manual quality assurance can uncover situations where an API is failing to handle unexpected situations. Deploying your API to a cloud provider with a compelling SLA instead of your own hardware and network shifts the burden of infrastructure resiliency to a service, freeing your time to build features for your customers.

Robust. A comprehensive suite of automated tests isn’t always sufficient to provide a robust API. Edge cases, unexpected code branches and other unplanned behavior may be triggered by requests that were not considered when writing the test suite. Traditional automated tests should be complimented by fuzz testing to help uncover hidden execution paths. Canary and blue-green deployments can further expose your API to the real world in a limited manner and prepare for when even more unexpected requests will hit.

Secure. API security is a moving target. It is expected that most APIs are built on layers of open source libraries and frameworks. Software composition analysis is a necessity to stay on top of zero-day vulnerabilities by identifying vulnerable dependencies as soon as they are discovered. OWASP guidance is a must-have—directing API developers to implement attack mitigation strategies such as CORS and CSRF protection. Application logic must be well tested for authorization and authentication.

Discoverable. For REST APIs, the OpenAPI initiative provides a consistent language for describing the API. Many frameworks, such as Spring Boot, can even generate OpenAPI documentation directly from your code. gRPC-based APIs offer similar benefits by providing both instructions for accessing the API and building the clients to communicate with it. Adding support for GraphQL allows developers to connect multiple APIs together with consistent tooling. Whichever direction you choose, it is vital to keep API documentation up to date and accessible.

Consistent. To be consistent, the API development process should make incompatible changes obvious. Code reviews can catch many breaking changes right at the source. API contract testing and integration testing can be used to discover API changes as part of the build. When working in a microservices environment, staging environments may alert engineers if an API change will cause problems before production.

Quality Builds Trust

According to the RapidAPI survey, 58% of executives said participating in the API economy was a top priority for their organization. The figure was even higher in certain industries such as telecommunications, where the number is 89%. Whether you are building APIs for internal use or for public consumption, your goal should be to earn the trust of API consumers. Investing in the culture, tools and processes to build quality APIs will pave the way to building and maintaining that trust.

June 10, 2021
HashiCorp Increases Terraform’s Enterprise Appeal

During the online HashiConf Europe conference today, HashiCorp debuted the general availability of a 1.0 release of HashiCorp Terraform along with updates to HashiCorp Terraform Cloud service.

Meghan Liese, senior director of product marketing for HashiCorp, said in both cases HashiCorp is moving to enable organizations to better secure and govern IT infrastructure-as-code provisioned using open source Terraform tools.

Version 1.0 of HashiCorp Terraform provides access to a stable application programming interface (API) that will foster interoperability and simplify future upgrades.

The Terraform Cloud service, which eliminates the need for IT teams to set up and manage a Terraform platform, adds workflows enhancements to the Terraform Cloud that include the ability to publish curated modules to a private registry, enhancements that make it easier to visualize elements of a workspace and a preview of an ability to check with third-party tools related to a Terraform run.

Overall, the goal is to make it simpler for organizations to compose, collaborate and reuse infrastructure-as-code using modules and the public and private registry options, said Liese. The public registry already has more than 5,000 modules. Terraform Cloud now offers a native workflow to publish modules from the public registry directly into an organization’s private registry.

Terraform, of course, has been widely employed since it first debuted in 2014. There have been 15 major updates prior to the release of Terraform 1.0, which is intended to provide a foundation that ensures backward compatibility between releases going forward.

In some instances, Terraform can be too much of good thing. It has considerably accelerated the rate at which IT infrastructure can be provisioned within the context of a DevOps workflow. The challenge, though, is developers often lack the security expertise required to prevent infrastructure from being misconfigured at a time when cybercriminals have become more adept at scanning for those vulnerabilities. To enable IT teams to prevent those misconfigurations, HashiCorp makes available both an enterprise edition of Terraform as well as its Terraform Cloud service. Terraform Enterprise is used by more than 1,200 enterprises, while Terraform Cloud has more than 120,000 users that execute more than 30,000 runs per day.

It’s not clear if security concerns might one day lead to a backlash against the concept of shifting more responsibility left toward developers to improve productivity. In some cases, developers are simply deploying more insecure applications faster. HashiCorp is making a case for Terraform platforms that allow centralized IT teams to exercise more control over how infrastructure is provisioned as code. The need for that approach is becoming more pronounced, as organizations not only employ Terraform to provision cloud infrastructure but also on-premises IT environments.

Regardless of how infrastructure is provisioned by developers, security of infrastructure, both inside and outside of the cloud, remains a priority as more workloads than ever are deployed. In effect, the attack surface that needs to be defended has dramatically expanded. Of course, the best security issue of all is the one that never happened because infrastructure is being provisioned right the first time and every time thereafter.

June 8, 2021
What Biden’s Cybersecurity EO Means for DevOps Teams

On May 12, 2021 President Biden issued Executive Order 14028, also known as the Executive Order on Improving the Nation’s Cybersecurity. This EO covers a lot of ground, and like all executive orders, it instructs agencies of the U.S. Federal Government to perform specific actions. What it doesn’t do is appropriate funding or create industry mandates. That’ll come later, once some of the plenary stages of the EO are complete.

Despite this, I’m confident that EO 14028 could be as impactful to cybersecurity as GDPR was to data privacy. I’m also confident that the EO has the attention of business leaders who are trying to figure out what it means to them and their teams—DevOps professionals. While the EO has no underlying mandate to private businesses or those located outside the U.S., it does outline a reasonable set of cybersecurity best practices that can be reasonably interpreted as a call to action for greater transparency.

In this article I’m going to use the terms supplier and buyer, but I don’t mean to imply that money needs to change hands; only that there are producers of software (suppliers) and consumers of software (buyers). If there is one supplier and one buyer, the software supply chain has one link. If that supplier is also a buyer, the chain has two links. For most software, there are many suppliers; in these cases, the supply chain isn’t so much a length of links but more of a mesh of interconnected links.

The 2021 Synopsys OSSRA report showed that the average audited application had over 500 dependencies, so that mesh is quite complex—to the point where the security practices used anywhere in the mesh are often obscure, at best. It’s this obscurity that attackers can exploit, which is why one of the key tenants of the EO is transparency.

As anyone charged with operating a piece of software knows, the security testing performed by a supplier isn’t really obvious, but, as a buyer, you assume the supplier has done the right things when it comes to security. But how do you know? Do you know where the code originated from? Do you know what security targets were used by the supplier? If an attacker is actively compromising the application, how would you know?

These are all very good questions that are reflected in the EO, to varying degrees. Those questions also belie the reality that suppliers test software. Suppliers do, of course, test their software, but what testing techniques did they use and how effective are/were their tools? And, more importantly, how can you confirm that the mitigations and fixes applied to address identified defects were effective for your deployment scenario? After all, it’s not like every supplier is going to provide you with full access to their source code.

This is where things become quite actionable for DevOps teams that have embraced the principles of operational feedback mechanisms. For those needing a bit of a refresher, that’s the part of the DevOps mobius where lessons learned operating software are fed back into the next iteration of the development flow. Let’s look at two pieces of operational feedback that are mentioned in the EO: least privilege and log management.

Log management features prominently in the EO in part because logs enable an understanding of how an application is operating. If a supplier provides detailed information about the log structure and flows, it then becomes possible to build alerts in log aggregation services and SIEMs as indications of unexpected operation. That’s the proactive model the EO embraces—as opposed to forensic postmortem log analysis following a breach—but this requires the supplier to provide insights into the log data indicating that the system became compromised. Of course, to provide that type of data, the supplier must have tested their system software in a manner that allows such a log template to be created.

We’ve heard that software should be developed using principles of “least privilege” for years, but if you’ve ever been asked to run an application as root or had a supplier not detail the precise network ports required by the application, you know there is still work to be done there. The EO specifically calls out cybersecurity techniques like multifactor authentication and zero-trust network access as ways to achieve granular control over how applications operate.

It is, however, the coverage of Software Bill of Materials (SBOM) that could have the greatest impact for DevOps teams. In effect, the EO stated that suppliers need to tell buyers the origin of all code within an application. While the EO doesn’t specify the elements of an appropriate SBOM, it does instruct NIST to publish the minimum elements of an acceptable SBOM within 45 days of the EO. When combined with the sections on how build environments are created and managed and the security transparency elements elsewhere in the EO, it’s clear that the EO intends to raise greater awareness of how software is created, patched and maintained.

In the end, Executive Order 14028 sets a high bar for cybersecurity in general—one where everyone involved in the creation and operation of software will need to up their game. I’ve said for years that it’s impossible to patch something you don’t know you’re running, and that remains true. What must be added is that you also can’t reliably secure something when you don’t know how well each component was tested. The EO is prompting us all to take a step toward solving that problem.

June 8, 2021
Fixing Risk Sharing With Observability
Incentives are mismatched among SREs, SecOps, and application developers. These mismatches create challenges around how and what information is shared across siloed teams. This asymmetrical information creates a moral hazard where one team can shift deployment risk to another team, with no accountability back to the originating team. Risk shifting results in unstable applications, inefficient infrastructure, security issues and poor customer experience. All of that impacts your company’s bottom line.

Closing the Information Gap

Observability is positioned as a way to fix the information gap between these disparate groups. Observable systems allow users to ask questions about their behavior in an open-ended way, unlike more rigid monitoring systems. SREs and SecOps teams are already reaping benefits from observability, but mismatched incentives limit its effectiveness. Meeting the observability expectations of IT leaders requires pervasive instrumentation across applications, infrastructure and third-party software. Delivering that level of instrumentation has remained out of reach due to incentive mismatches, as well as human and infrastructure costs.

The main incentive for developers is shipping code quickly at a reasonable level of quality, meaning a low bug count. This contrasts with SREs, who are incentivized around uptime, performance and efficiency, while the incentives for SecOps teams are around risk reduction and breach mitigation.

Since SREs and SecOps teams don’t have insight into what developers have changed, it’s easy for performance or security problems to get thrown over the fence to operations. A fresh code push might only include minor changes and be harmless to existing operations. Or, it could replace large chunks of logic across the entire codebase, including adding calls to external and third-party applications. To the intrepid SRE and SecOps teams, one change looks like any other. They don’t get a peek behind the curtain because of the mismatched incentives. DevOps wants to deploy quickly. Waiting on approval from other teams slows down deployments, so comprehensive reviews don’t happen. This doesn’t mean DevOps teams are intentionally trying to sabotage partner teams. They’re simply acting in their own interests based on their incentives.

The challenge is that one party, the developers, has more information than other parties. That information asymmetry is what creates unbalanced risk sharing. Coping with information asymmetry has led to all kinds of new collaborative models, starting with DevOps and evolving into DevSecOps and other permutations like BizDevSecOps.

True collaboration has been hard to come by. Early DevOps efforts are often successful, but scaling beyond five to seven teams is difficult because teams lack the breadth of experience in IT operations or the SRE capacity to staff multiple product teams. The change velocity DevOps teams can achieve is often far greater than SREs and SecOps can absorb, making information asymmetry worse.

If teams can’t maintain high levels of collaboration and communication, another option must be developed. Observability practices, like collecting all events, metrics, traces and logs, allow SREs and SecOps teams to interrogate applications about their behavior without knowing which questions they want to ask ahead of time. However, observability only works if applications, and the infrastructure they rely on, are instrumented. This creates another problem: who does the instrumentation?

The expectation is DevOps teams embed instrumentation into their code as part of the development process. While that’s a nice idea, there are four reasons this falls short. First, the quality of instrumentation varies. Many log statements are terse and only understandable by the developer that wrote them. The message “In function xyx123!” isn’t helpful to an SRE digging into a performance problem that cropped up in the latest release.

Second, instrumentation libraries vary by implementation, giving inconsistent results across language bindings. OpenTelemetry is trying to improve this, but its progress is slow and still requires developers to do more work that, if we’re honest, doesn’t benefit them; it benefits SREs and SecOps. Now we’re back to those pesky mismatched incentives and information asymmetry problems again.

The third problem with instrumentation is the volume of data. Each instrumented application can produce terabytes of data each day. When you have robust instrumentation, the amount of data can be overwhelming, and extremely costly to analyze and store.

Lastly, instrumentation is isolated to the code your team wrote. That represents a fraction of the code you rely on. Vendor-provided services and APIs remain a black box, limiting your observability into those components.

Resolving information asymmetry across teams requires two things:
1. Pervasive, pluggable instrumentation capabilities for all code that doesn’t require developer involvement, and
2. An observability pipeline to filter, redact and enrich data, then route it to your analytics platform of choice.
Pervasive Instrumentation

Operations teams need instrumentation without having to go back to developers and beg them to add it to existing code. They need to turn it on and off as needed, and they need readily consumable data. They also need every piece of data they can get, including packet payloads, insight into encrypted data and so on. This goes well beyond what’s possible with today’s instrumentation options.

AppScope, a newly released open source project, is a new take on instrumentation. AppScope interposes itself between application threads and system libraries, tracking things like file system access, network and HTTP activity, as well as CPU and process activity. It also provides payload data, and because it sits between the application and encryption libraries, it also gives access to users’ cleartext data. Because it works with any Linux binary, SREs and operations teams can instrument anything, even code they didn’t write.

Observability Pipeline

The challenge with pervasive instrumentation is dealing with all the data generated by applications. Network, file system, and other system data can easily swamp destination APM and log analytics platforms, driving up licensing and infrastructure costs. While instrumentation data is vital to rebalancing risk sharing in organizations, you need a way to manage that data intelligently to get value out of it. This is where the observability pipeline comes in.

An observability pipeline sits between the sources of observability data and its target destinations. By acting as a strategic control point, the pipeline gives users control over how data is formatted, filtered, enriched, and redacted before it is routed to one or more destinations.

Observability pipelines help SREs and operations teams deal with the flood of instrumentation data by routing low value data to low-cost storage, like S3, while higher value information lands in APM and log analytics tools. Later, if you decide you need the data stored in S3 to add more context to your analysis, you can replay it back through the pipeline and enrich your data set.

Shifting business priorities are accelerating the pace of technology deployment across organizations, and teams are pressured to deliver faster. This faster pace of delivery pushes deployment risk to operations and security teams that lack visibility into changes developers have made across complex distributed systems. This information asymmetry also increases risk to the business as applications become less predictable and reliable over time.

Traditional methods of resolving the information mismatch haven’t worked because incentives across teams aren’t aligned. Adopting pervasive instrumentation and observability practices will give SREs and operations teams critical visibility into rapidly changing application and infrastructure environments, without disrupting the developer experience and process.
June 7, 2021
GitLab Acquires UnReview to Further AI Ambitions

GitLab announced this week it has acquired UnReview, a provider of a tool that employs machine learning algorithms to identify which expert code reviewers to assign to a project based on both the quality of their previous efforts and current workloads.

David DeSanto, senior director for product management at GitLab, said the acquisition of UnReview is the latest step in an AI strategy that, in addition to optimizing DevOps processes, will eventually unify machine learning operations (MLOps) and DevOps workflows.

Accessed via the Dev section of the GitLab platform, UnReview will also be employed to manage the overall code review process.

DeSanto said GitLab is committed to employing AI technologies to automate workflows and compressing cycle times across all stages of the DevSecOps life cycle. The goal is to not eliminate the need for DevOps teams, but rather eliminate low-level tasks that conspire to hamper productivity while at the same time improving application security, noted DeSanto.

Providers of DevOps platforms are now locked into an AI arms race on two separate fronts. Each of them clearly sees the opportunity to leverage machine learning algorithms and other forms of AI to automate a wide range of processes. At the same time, it’s apparent providers of MLOps platforms are borrowing heavily from DevOps best practices to manage the development of AI models. However, from the perspective of a DevOps team, an AI model is just another type of software artifact. Rather than deploying two distinct platforms, the goal should be to bring the data scientists that build AI models into the larger DevOps community, said DeSanto.

The current challenge, of course, is the pace at which DevOps and data science teams work. It may take six months or more for a data science team to construct and validate an AI model that needs to be embedded within an application. Aligning the timetable for an AI model with the application development life cycle is difficult to accomplish if neither team has any meaningful visibility into each other’s workflows. As the rate at which AI models can be developed steadily improves, that alignment issue will only become all the more pressing.

Going forward, it’s hard to envision any application that won’t have some sort of AI model embedded within it. The issue is that, just like any other software artifact, an AI model needs to be updated. As new data sources become available, an AI model is likely to drift beyond the parameters intended. A rapid change in business conditions may also require the replacement of an entire AI model. The same processes that DevOps teams apply to update any software artifact can also be applied to an AI model.

It’s unclear how long it will take before some type of convergence of DevOps and MLOps inevitably occurs, as organizations look for more efficient ways to manage what will become thousands of AI models. The one certain thing is the longer it takes to make the effort, the more difficult it will become to overcome the organizational inertia that has been allowed to set in.

June 3, 2021