devopsy.kinsta.cloud

Tag: security

Catch Emerging Security Risks Earlier by Leveraging Kubernetes Audit Logs
When a Kubernetes deployment suffers a security breach, it can be difficult for security teams to diagnose and identify the source of the intrusion and its impact. Each Kubernetes cluster may host multiple applications, and each application may be composed of dozens or even hundreds of interacting microservices deployed as pods.

The complex and dynamic communication between the applications’ components, as well as the automated Kubernetes orchestration activities, makes it difficult to understand the structure of each application and differentiate between normal behavior and malicious behavior from the observed network activity. The ephemeral nature of pods, which may exist only for moments to perform an activity before they are disposed of, makes this task even more difficult. A pod that was vulnerable may be gone by the time security teams are investigating a breach, or resume normal operation as the attacker pauses its abuse or pivots or escalates the attack to another Kubernetes resource.

K8s is by nature dynamic, flexible and scalable, and it is this same advantage for cloud native application developers that becomes a challenge for security professionals. How can one understand the source of a breach, let alone how it propagated through the system, without deep understanding of the architecture of the application?

One solution to the problem of tracing breaches is to examine the Kubernetes audit logs since they record all cluster-administration activity, even for pods that may have existed for seconds and are now gone. But K8s audit logs are complex and often convoluted, requiring deep expertise to understand. Therefore, security teams need to not only increase their proficiency with K8s audit logs, but also to apply automation and software tools to the K8s logs to detect suspicious behavior after the fact and even during their occurrence.

In fact, there are many potential threats that can be discovered when observing audit logs for suspicious behavior. They include:
- Stolen or misused credentials, enabling hackers to gain access to K8s-based clusters or pods.
- Misconfigured Rules Based Access Control (RBAC), enabling lateral attack propagation, privilege escalation and unauthorized data access or manipulation.
- Exploitation of vulnerabilities in the Kubernetes API Server, enabling bypassing of authentication, authorization, admission control or validation of cluster administration requests. This lets users gain access to privileged and sensitive resources.
- Violations of security policies diverge from compliance requirements and best practices.
Detect Breaches in Near Real Time by Watching K8s Audit Logs

There may be a vast time span between an initial breach and the detection of the compromised application or the unauthorized access of secure data and private information. Security teams must seek to close the gap between a breach attempt and its detection as much as possible, while still being able to diagnose the causes of a breach after it has been detected and assess its scope and impact.

Since K8s deployments can be vast and intricate, machine learning and artificial intelligence are preferred tools to monitor audit logs and flag suspicious behavior. An automated tool could continuously scan logs to learn what normal behavior looks like in the company’s clusters, and then flag network behavior that is unusual or suspicious for security teams to qualify, prioritize and investigate. This methodology would enable security teams to investigate possible security violations in near real time to protect and secure the applications.

In addition to detecting security risks, security tools that leverage K8s audit logs should provide reporting that is easy to understand and consume by security teams. By providing the correct user experience, companies can make their K8s logs accessible to more than the hard-core experts, effectively opening a new vista of security possibilities for more security professionals. The difficult learning curve of Kubernetes will no longer be a wall between K8s pros and newcomers with less experience, relieving pressure from understaffed security teams.

Step Up to Security and Compliance Needs

As Kubernetes moves toward the mainstream, with roughly 71% of Fortune 100 companies using Kubernetes as their main container orchestration tool, companies must also look toward compliance with standards and government regulations designed to protect financial transactions and private or personal information such as PCI, GDPR and HIPPA. The penalties for violating these standards and regulations can be severe, so companies should proactively adopt K8s audit log monitoring as a way to identify breaches earlier, to limit their impact on customers and data, and to document the behavior of the application and would-be hackers for possible legal inquiries. An effective audit log strategy will be vital in determining what happened in the case of a breach.

Leverage Audit Logs for Real-Time Security Observability in K8s

Many tools exist to secure the perimeter of the data center and to identify possible security risks during application development. The benefits of analyzing and leveraging Kubernetes audit logs at runtime have been relatively overlooked. K8s audit logs are a prime way to comprehend the behavior of any cloud-native application orchestrated by Kubernetes, as well as the best way to comprehend how the Kubernetes infrastructure itself can be abused by an attacker to compromise the applications it is hosting. As such, security teams have a responsibility to deploy tools and gain a proactive, real-time view into K8s audit logs in order to minimize the time between breach and remediation and to validate compliance conformance at runtime. Such a thoughtful approach to Kubernetes security will go a long way to protect your deployments from harm.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or Aug. 2020.

— Nitzan Niv
March 13, 2020
The DevOps Sweet Spot: Inserting Security at Pull Requests (Part 1)

Today’s organizations are increasingly benefiting from the modernization of the software development lifecycle (SDLC), including the adoption of cloud, DevOps, Agile methodologies, containers and more. Thanks to this modernization, organizations are innovating and releasing updates faster than ever before. But these benefits aren’t without challenges.

Application security arguably tops the list of challenges. Security has inherently manual bottlenecks, and is unable to keep up with the pace of automated DevOps pipelines. The result is that security is often left behind, overlooked or simply ignored in the development process. Developers aren’t willing to slow innovation down today for the potential of security breaches tomorrow.

As a result, more and more applications are being shipped into production with vulnerabilities and security flaws. Today, most of the vulnerabilities are found in post-production either via red teams, bug bounty programs or siloed and irregular code analysis scans. At best, these vulnerabilities are fixed in the next version that’s shipped. At worst, your organization is making headlines for millions of personal records being breached.

However, security in the modern SDLC is becoming more symbiotic thanks to two key advancements—speed and accuracy. Overcoming these two challenges enables security to be inserted into developer workflows earlier than ever before. By automating security with every pull request, development organizations can finally embrace security without slowing down innovation.

Automation — Security at the Speed of DevOps

These days, many vendors claim to insert security into DevOps pipelines, often during the build. While it’s true that security scans may be automatically started, they still finish in siloed processes that don’t keep up with the pace of releases. We call this “automation for automation’s sake.” It does not measurably improve defect density, slow the accrual of technical debt or reduce mean time to repair (MTTR). The only value that automating this way provides is checking a box on a compliance form.

So, how can organizations more effectively automate security quality decisions?

First, security tests and scans need to be dramatically faster. Legacy code analysis takes such a long time that many vendors recommend sacrificing comprehensiveness by scanning only differentials. Even then, they still aren’t fast enough to keep up with CI/CD. It goes without saying that organizations cannot meaningfully automate security into processes such as builds or pull requests, that take just a fraction of the time.

Second, the results need to be more reliable. The goal of security automation should be to fail insecure code. However, historically, code analysis tools are plagued by so many false positives that no code would ever pass any test. Only with reliable results can insecure code be confidently failed.

Third, security scans must be comprehensive as false negatives lead to vulnerabilities going undetected into production. This is, in part, because developers are not asking the right security questions when scanning their code. Traditional code analysis tools leverage pattern matching techniques—they look for symptoms of a vulnerability that are common across all code bases. For example, lack of input validation from sources is a classic cause of injection vulnerabilities. However, there are many vulnerabilities that are unique to each code base, such as business logic flaws, data leakage, hard-coded literals and insider threats.

For example, all applications are purpose-built to meet specific business needs, meaning vulnerabilities may go beyond coding errors and present themselves as business logic flaws in the design of an application. Take the First American Financial breach. Thanks to an insecure direct object reference (IDOR), an attacker has the potential to manipulate an internal implementation object. In the case of First American Financial, this was a URL or form parameter. Given the asymmetrical advantage that attackers have, defenders must protect from all possible vulnerabilities, but attackers need to only find one vulnerable route. As such, partial security can be as problematic as no security.

Only when security is completely comprehensive—reliable enough to fail with confidence and fast enough to keep up—can it be meaningfully automated into developer workflows without slowing them down.

Yet, comprehensive security scanning is only part of the equation. In the second article of this series, I’ll dig into where these security scans should be taking place in the development cycle to make them most effective.

— Chetan Conikee

March 11, 2020
Better Apps and Better Security When You Shift Left

Tens of thousands of people and hundreds of cybersecurity vendors descended on San Francisco at the end of April. While the RSA Conference was the primary draw, there are a number of peripheral events that go on throughout the week as well, such as the Cloud Security Alliance Summit, the CIO/CISO Interchange, BSides San Francisco and the Qualys Security Conference.

I was able to attend some of the Qualys Security Conference and I had the opportunity to see Alex Mandernack, security solution architect for product management at Qualys, presenting on the need to shift left when it comes to cybersecurity.

Mandernack outlined some of the issues organizations face today when it comes to security. Migrating to the cloud and embracing DevOps culture means more ephemeral, elastic environments and compressed development time frames. He pointed out that these trends make security more challenging, but also described the opportunity they present to change our approach to streamline and improve security.

The Traditional Pipeline is Broken

The traditional model of application development is much more linear. Each app team builds their own image—often in a silo or vacuum disconnected from other teams. Penetration testing is conducted on a sporadic or infrequent basis, and eventually the app is deployed. Vulnerability scans are run on the app in production, and issues or deficiencies are reported back to the developers to address or incorporate into the next build.

The process is slow and cumbersome. It also has the potential to introduce redundancies that waste resources and make the organization less efficient. As companies embrace DevOps and move to the cloud, the traditional development pipeline simply doesn’t work—especially not for security.

DevOps, and Cloud, and Containers — Oh My!

The cloud and DevOps fundamentally change the development lifecycle. Development time frames are compressed, and the process is a self-feeding, iterative circle rather than a repeating linear approach. The traditional approach of identifying and resolving security issues after an application is deployed won’t work in a DevOps environment because by the time issues are identified and fixed, the app itself may have already moved on to the next version.

Cybersecurity and the development pipeline have enough challenges with the cloud and DevOps, but there is also a newer technology trend that exacerbates the problem—containers. Containers are designed to get code quickly and to operate fast. They are also built for scalability—spawning and spinning down hundreds or thousands of containers constantly to keep pace with demand.

There is a diverse array of tools and platforms associated with cloud, DevOps and containers. Organizations often employ a complex collection comprised of some combination of AWS, Azure, Docker, Kubernetes, OpenShift and others. The result can quickly become a black hole when it comes to security—which is why it’s important to push security left into the CI/CD pipeline.

Shift Left to Help Security Teams Do Better

In a DevOps culture, developers and security teams need to think about security sooner. It’s important to get the security tools into the process earlier. Thankfully, with the DevOps lifecycle and the tools commonly used within a DevOps environment, it can be done.

For starters, companies should build a golden image. Developers shouldn’t have to patch and update the base operating system before they even start developing. The golden image takes a baseline operating system such as Ubuntu or CentOS, and completely patches and configures it so everyone has a standard, consistent image to develop on.

With the golden image in place, the next step in shifting security left is to implement testing and scanning in the CI/CD pipeline. Put vulnerability information at the fingertips of developers as they’re developing and implement vulnerability gates in the pipeline to ensure that applications that don’t comply with policies and meet minimum requirements are not allowed to be deployed.

New Role of the Security Team

Part of shifting security left includes providing tools that make security self-service for the DevOps team. Automating actions with scripting, APIs and CI plugins ensures that security is simplified and streamlined to provide value for developers.

In this shift left scenario, the security team has to redefine its role in the process. The security team is still vital, but rather than being the police or a roadblock in the development process, the role of security focuses on verifying and auditing the process. The security framework should have dashboards and live data that enable the security team to monitor activity and identify trends that may need further attention.

Better Security, Faster

There is often conflict between developers and security teams, but there doesn’t need to be. Mandernack stressed that developers don’t want to create insecure apps either.

Shifting security to the left into the CI/CD pipeline streamlines the whole process. Ultimately, the result is higher quality, more secure applications that are developed faster.

— Tony Bradley

March 10, 2020
How 5G Mobile Networks Will Change IoT Security

Depending on your location, you might have noticed a new 5G icon on your cell phone recently. The roll-out of the new 5G mobile network is largely a positive change for consumers and businesses. The fifth generation of cellular networks (hence 5G) purports to be one of the fastest wireless networks ever created.

This technology claims to provide more efficient interconnectivity, and faster data transfers between people, objects and devices. All good things, right? But what security risks does 5G represent for the modern age?

Some cybersecurity experts suggest that new vulnerabilities are versions of problems not entirely flushed out from 4G and even 3G networks. This article will attempt to explore some problems and opportunities that IoT security experts need to be aware of, and how savvy enterprises can be proactive about security in the age of 5G.

Riding the 5G Wave

A business’s potential is only as great as the technology they elect, as an organization, to adopt and use. One of the first benefits that 5G provides comes from the additional speed that the network provides. 4G posted average download speeds around 20Mbps. With a fully deployed and optimized 5G network, users should expect those download speeds to reach anywhere from 500 to 1,500Mbps.

Raw speed is one thing, being able to harness this type of efficiency is another. A Barclays’ poll recently reported that only around 28% of businesses understand 5G or the practical business applications the network offers.

Along with the challenges of leveraging the increased speed of the 5G network for business purposes, another important aspect of 5G education involves the security risks associated with faster technology. We suspect that if only 28% of businesses understand the practical applications of a 5G network, the awareness of potential security risks may be a similarly low figure.

Getting Up to Speed on G Security

While 5G touts a tighter grip on security, experts in the space will remind us that some of the concerning holes in the security of 4G, and even 3G, networks have been carried over into the rollout of 5G. Give credit where it’s due, the 5G security wins are associated with the encryption process which improves anti-tracking and spoofing features. This means that would-be criminals have a much harder time tracking and manipulating device connections.

5G is also more software and cloud-focused than previous versions, which allows for better monitoring so threats can be spotted and mitigated more quickly.

While this is good for consumers, it’s not a perfect system by any means. “Stingray” devices can still be used to deploy fake base station attacks, which allow attackers to intercept mobile traffic and potentially manipulate data.

Downgrade Attacks

Purdue University and the University of Iowa outlined findings from design issues in the 5G protocol that exposed 11 vulnerabilities that may be exploited by bad actors. Five of these discovered vulnerabilities were carried over from 3G and 4G networks. Notably, a “downgrade attack” which essentially reverts the devices to use old mobile data networks. On top of potentially leading to higher wireless bills, this may allow attackers to track calls, texts or browsing habits on devices.

If a device is forced to operate in a limited-service mode, an article in WIRED explains how IMSI numbers may also become exposed because of a downgrade attack.

“One purported benefit of 5G is that it protects phone identifiers, like your device’s ‘international mobile subscriber identity,’ to help prevent tracking or targeted attacks,” stated WIRED. “But downgrade attacks like the ones the researchers found can bump your device down to 4G or put it into limited service mode, then force it to send its IMSI number unencrypted. Increasingly, networks use an alternative ID called a Temporary Mobile Subscriber Identity that refreshes periodically to stymie tracking.”

Tips to Avoid Security Pitfalls in the Age of 5G

In the age of the fastest speeds ever seen and nearly ubiquitous connectivity because of the rapid adoption of IoT devices, it’s becoming apparent that a zero trust model could help address security concerns with 5G and beyond.

This may require additional education to be available for employees, but an increase in security best practices for endpoint users is something that will benefit an organization tremendously. Zero-trust security models continually check the user’s presence and behavior within a network regardless of whether the user is human or machine.

It might seem like a great idea, but adopting the model isn’t a change to be taken lightly. A survey conducted by AT&T about 5G security indicated that just 33% of respondents reported they are currently using multi-factor authentication—the road to zero trust will take buy-in from executives and managers.

Sharing the Burden of Security

The 5G rollout does provide built-in security features, but it should not be viewed as the Holy Grail. An organization will benefit from having a comprehensive security model, but that doesn’t mean they need to single-handedly oversee every aspect of that model.

5G is a combined effort that involves network operators and enterprises, and shared responsibility is essential for long-term success. Managed service providers should be considered by companies that may be short-staffed, or don’t have the in-house resources to cover other important areas of security, such as ransomware protection.

Security within a business can’t exist in a silo, it’s a team sport—make sure you’re on the winning side.

— Marty Puranik

February 26, 2020
The DevSecOps Landscape is Maturing — We Want to Hear About Your Journey

The race to out-innovate one’s competition has led to high performing organizations chasing increased deployment velocities but often ignoring the quality of parts being used to manufacture their applications.

It was 2003 when Bruce Schneier penned, “Today there are no real consequences for having bad security, or having low-quality software of any kind. Even worse, the marketplace often rewards low quality. More precisely, it rewards additional features and timely release dates, even if they come at the expense of quality.” Seventeen years later and it can sometimes feel like we haven’t grown enough.

As nimble organizations aim to deliver new innovations faster, using DevOps principles, the question of how and where security fits into the equation has spawned bigger conversations around what DevSecOps really means. And, in some circles where DevSecOps is even a different principle—security should already be a part of DevOps.

This is why we do the DevSecOps Community Survey every year—to better understand how organizations are adapting, what previous challenges were overcome, what new challenges may have popped up and to examine what approaches are being prioritized within teams to better identify risks. These questions, and many others, are extremely important and it’s why we’re embarking on the seventh annual DevSecOps Community survey. And, we’re looking for you to help us understand that state of DevSecOps.

Sonatype, devopsy.kinsta.cloud, Security Boulevard, Cloudbees, Carnegie Mellon SEI, DevOps Institute, NowSecure, Verica, All Day DevOps and DevSecOps Days, launched the annual DevSecOps Community Survey earlier this month. Since then, we’ve already received over 5,000 responses. But, we have high ambitions and want this to be the most in-depth and comprehensive studies to date—and we need your help to do that.

Since we first started this survey, seven years ago, we’ve seen a consistent maturation of DevOps practices and the combination of automated security. More and more respondents highlight that their practices have evolved and adapted to a DevSecOps mentality—or, that they have the ambition to do so.

That said, we know we’re still in the early stages of so many DevSecOps transformations and while people may recognize the need for governance policies with DevOps, the theory doesn’t always make it into practice. The insights we gain from this survey allow us to provide concrete information back to the community on what those who have successfully transformed themselves are doing well—and what their journey looked like.

For instance, from over 5,500 respondents who took last year’s survey, we saw that more than 47% of them were deploying changes into production multiple times per week. This meant that as adversaries are getting faster at exploiting vulnerabilities, DevOps organizations that can identify cybersecurity risks and remediate them sooner can better defend themselves.

While we’ve learned that security is difficult to ignore when it’s embedded where developers already are, there is a lot more to understand about current practices. The voice of the community these past seven years has been invaluable, and we recognize that the experiences of those in the community can help us learn what resources are needed in order to support this ongoing cultural shift.

In this year’s survey, for the first time, we’re also aiming to understand how responders feel about their jobs and the environments they work in. DevSecOps is a culture and we want to better understand the cultural attributes the elite DevSecOps practices employ.

Please take a few minutes to fill out the 2020 DevSecOps Community survey today; help us help the industry by better comprehending how the DevSecOps community has matured over the past year. And, you may even win one of our prizes—a Macbook, AirPods Pro or an Oculus Quest. Everyone who takes the survey will also receive first look at the results.

— DJ Schleen

February 24, 2020
Multi-Cloud Adoption: Time to Rethink Your Identity and Access Management

This may well be the decade that the data center’s reign falls, and multi-clouds prevail. A multi-cloud strategy, which incorporates cloud services from multiple vendors, promises plenty of benefits, including improved resilience and flexibility that makes it easier for enterprises to meet various application and data needs. But, there will also be operational challenges. Expect security, specifically identity and access management (IAM), to be one of them.

Gartner states that “by 2025, 80% of enterprises will have shut down their traditional data center, versus 10% today.” Except for a few mission-critical processes that will remain on-premises because they require intense oversight and control, much of an enterprise’s workloads and data will be spread across a multi-cloud environment, with users accessing applications from a range of devices from multiple locations–making it more difficult to secure, control, track and manage access to applications and services. Without a comprehensive IAM plan, an organization could be more vulnerable to attacks and data breaches.

Instituting the right IAM strategy now, in concert with an evolving multi-cloud architecture, will help protect threats and ensure that the right users have access to the right information when they need it. To get started, we recommend following these key recommendations.

Get Your ID Store in Order

IAM solutions are designed to make sure only people appropriately identified and cleared can access an enterprise’s systems and services. In today’s increasingly complex IT environment, an IAM solution must include an ID store that supports both on-premise systems and cloud services. Unfortunately, too many companies still rely on Microsoft Active Directory as the single source of truth for the IT services they provide. This needs to change.

As companies begin to adopt multi-cloud architectures, they’ll soon discover that Active Directories are not flexible enough to support IAM, nor can they keep up with the growing number of potential threats. Integrating an identity software product that can manage users’ digital identities, credentials and groupings with Active Directory may work as temporary solution, but as user passwords and access rights are dispersed across on-premise and cloud-based systems, things can get complicated quickly, taxing even the well-resourced IT team.

If you haven’t already done so, it may be time to extend Active Directory (AD) to Azure Active Directory (Azure AD), Microsoft’s multi-tenant cloud-based directory and identity management service that enables Single Sign-On (SSO) access to on-premise and cloud applications, putting users through an authentication process to prove they are who they say they are. It uses multi-factor authentication, a two-step identity verification system that requires two or more of the following methods: something you know (password), something you have (trusted device) or something you are (biometric screening).

Embrace Zero Trust

The premise of zero trust has been around for several years. According to CSO, the first model was defined by John Kindervag (then principal analyst at Forrester Research Inc.) in 2010. The idea is that when it comes to securing enterprise systems and data, no user–internal or external–can be trusted. Security threats are much more sophisticated today and are impacting organizations at greater rates than ever before.

Malware such as phishing, email phishing and drive-by downloads–which can occur when visiting a website, viewing an e-mail message or by clicking on a deceptive pop-up window–put an organization’s identity and authentication at increased risk. Once an attacker gets a working identity and authentication method, he or she can do almost anything, including system damage, data leaks and destruction, and more. A zero trust network that rethinks approaches to resource access is quickly becoming a necessity.

Part of the move to zero trust should include a decision to move away from passwords, which are still widely used. Passwords are typically too weak or often re-used, making them vulnerable and difficult to manage. As such, they don’t deliver robust enough security for sensitive systems and confidential information. A passwordless approach eliminates this problem by removing the need for users to remember passwords and for organizations to store them. There are several ways to go passwordless, including adopting biometric authentication such as touch ID and face recognition, and token-based methods such as mobile app authenticators or secured USB keys.

Control Your Multi-Cloud Strategy

Of course, distributing an enterprise’s workloads and data across larger numbers of clouds operated by different providers and located in various geographic regions can increase the complexity of IAM. IT staff will have to learn how to manage, provision, control, track and synchronize the systems in each environment, and such a highly distributed operation can easily become an operational challenge. Things can get even more messy, and risky, if an enterprise’s multi-cloud adoption develops in an ad-hoc manner.

Multi-cloud best practices include working with existing cloud partners, adopting orchestration tools, controlling the risks of shadow IT and keeping the number of cloud environments to a minimum by adding only those clouds that support and elevate an enterprise’s business goals.

With multi-cloud adoption on the rise, IAM is becoming increasingly important in order to protect against insider and outsider cyber threats and to ensure that the only the appropriate users can access the resources they need, when they need them. 2020 is the year to get identity and authentication right. It’s time to develop a comprehensive IAM strategy that includes an advanced ID store, zero trust and controlled multi-cloud adoption.

— Richard Diver

February 24, 2020
DevSecOps: A Renewed Commitment to Secure Delivery, Part 2

If done right, DevSecOps eliminates the cultural roadblocks that often prevent organizations from getting IT security proactively involved with development and operations. Barriers are the last thing any organization should contend with when comprehensive security is an absolute priority at a time when data breaches, intellectual property theft and other cybercrimes can almost immediately cripple a business.

Still, companies are far from perfect. Misunderstandings, ill-defined policies and turf wars can prevent development and operations from collaborating in a way that naturally bakes-in security into their work. To effectively transform a DevOps pipeline into a DevSecOps one, organizations need to change the way people work and collaborate; make processes inclusive of quality, security and compliance activities; and integrate the right tools.

Those three steps are the focus of this two-part series on becoming a true DevSecOps enterprise. We reviewed processes in part one and now we’ll cover people and tools in this article.

For enterprises with deep-rooted DevOps, a renewed focus on security might entail modernizing the release pipeline and involving IT security people even more. For enterprises that are modernizing heritage-monolithic application portfolios and moving to cloud native architectures, pipeline modernization should include a DevSecOps approach that considers containers, microservices and APIs.

DevSecOps done right gives organizations a level of visibility and accountability that helps accelerate coding, testing, production deployments, problem determination and defect resolution–all accomplished because of an organization-wide acceptance and commitment to security that should shatter any and all roadblocks.

Don’t Forget About the Humans in DevSecOps

When was the last time you heard a developer say, “Hey, let’s get the security guys in here, to hear what they have to say?” Reality is far harsher. Collaboration with IT security often triggers visceral reactions that are not exactly positive. IT security is often stereotyped as the organization of “no” and “you can’t do that.”

DevSecOps aims to tear down those cultural barriers. DevSecOps is all about embracing your IT sec organization and even inviting auditors to collaborate early on in the release process.

Ultimately, security is everyone’s responsibility. Being a developer, there was nothing like the mention of security to make me feel super uncomfortable. I don’t believe that developers set out to purposely create insecure code. However, when IT sec is not involved in a collaborative, cooperative relationship with dev and ops, there is an uncertainty that creeps in, and trust and confidence are shattered.

We know developers develop and release code, while operators control and manage the runtime environment. What is the expectation of IT security in this process? What are they looking for? What risks do they need to mitigate when a digital product is delivered? Understanding the objectives of IT security in the delivery process will clarify the steps that must take place in the process and reveal which tools should be used to create a product that’s secure and delivered with velocity and quality.

Understanding what stakeholders do along the pipeline reconciles personal and organizational accountability with tangible value-add in the release process. DevSecOps can be a catalyst to re-examine and provide clarity on who is doing what, when it is being done and why the effort is being exerted.

Clarity might reveal that personas need to be updated to more modern roles. For example, an operations system administrator might become more of a site reliability engineer (SRE). The SRE role involves more proactive activities, earlier in the release pipeline, to infuse greater fault tolerance in code and greater resilience in the environment configurations–and to solve problems by engineering solutions with developers and IT security.

Determine who does what and where. Develop personas that outline responsibilities along the pipeline, from the sourcing of code to production, to create necessary transparency. Value-stream mapping is a good way to capture the current state of a release pipeline and an excellent way to construct a desired state pipeline with the necessary people performing value-add activities to releasing a product.

Use Tools that Provide Necessary Visibility

Tools are the enablers. These are the cogs in the machine that will be running test cases, static analysis, vulnerability scans and integration tests. Tools will automate DevSecOps processes and provide the most critical function: visibility. The ability to observe all DevOps processes and determine their origins in the pipeline is critical. Visibility lessens risk and supports compliance and audits. The process that defines how code is governed through the promotion path should drive the tools that will be integrated into the release pipeline. Most tools serve to automate build, testing, scanning, movement through the promotion path and integration with other tools. It only makes sense that end-to-end visibility in the release pipeline should be engineered in.

Tools, though, must integrate and communicate to achieve end-to-end visibility. In the heritage era of DevOps (basically, the time before cloud native), it was tough to gain visibility because tools came from different vendors and often couldn’t talk with one another. Fortunately, cloud native platforms offer more integrated capabilities. Build and CI tools naturally share information because these capabilities are being refactored into the cloud platforms themselves.

An example of this is the cloud native CI/CD capabilities of Tekton, which leverage and extend the operator framework in Kubernetes. Translation: CI/CD tools are starting to look like extensions of the cloud native platform, in stark contrast to heritage CI/CD tools which were standalone third-party software products. The result of this tool trend is tighter integration and better visibility built into the platform–you don’t have to engineer it.

Companies need to review how effectively their tools communicate and close any visibility gaps. When possible, consider some of the next-gen automation tools and technologies such as Tekton, Kabanero.io and Jenkins-x that bring greater visibility, integration and consideration for cloud native apps which heritage tools have struggled with.

DevSecOps Is a Catalyst for Pipeline Modernization

A modern security framework will let companies take full advantage of the cloud native technologies that are changing DevOps. Granted, most companies won’t completely shift their entire infrastructure to the cloud, nor should they. Most organizations need to identify a partner to help them assess which processes and tools are worth moving to the cloud, and which ones–like a mainframe that runs core business functions–should be left alone.

For those enterprises that have a large heritage application portfolio, DevSecOps could mean retrofitting automation into systems and applications that were often designed and developed without the scaffolding of a secure DevOps pipeline. DevSecOps applies to heritage application, too. After all, what pipeline wouldn’t benefit from the visibility and traceability of tracking a user story to its production release? By automating the delivery of these heritage apps, you’re optimizing the effort of manual activities such as build, test and deploy, and retrofitting the capability of pipeline visibility.

Cloud has changed the DevOps game, newly defining the fundamental technical underpinnings of what an application really is. And it is cloud that’s driving companies to look at the security of DevOps in an entirely new way. With much on the line–the purity of products, compliance, brand reputation–companies need to rebuild their security frameworks so they can take full advantage of cloud and their on-premises programs. They don’t have to do DevSecOps alone.

— Andrea C. Crawford

February 20, 2020
Does the Empire Strike Back When We Learn What Happens to the $10 Billion JEDI Contract?

Microsoft scored a huge victory by sweeping in and taking the $10 billion dollar, multi-year JEDI contract for military IT services from AWS (owned and operated by amazon.com, its parent company). The backstory on this is that James “Mad Dog” Mattis, then Secretary of Defense, met with Jeff Bezos who tweeted about it on August 10, 2017 and it appears that Mattis decided to structure a request for IT services (now known as the JEDI contract) that all but led to AWS being crowned the winner by the way the bid was structured.

However, this was not to be. Not only was Mattis gone by January of 2019, but President Trump and Jeff Bezos were having public squabbling matches regarding coverage of the Trump administration in the Washington Post (which Jeff Bezos had purchased in 2013). So tenuous is their relationship that Trump mocked Bezos on Twitter regarding his divorce which was precipitated by the leaked private messages and pictures between Mr. Bezos and his mistress.

What does this all have to do with JEDI? Quite a bit, actually. Amazon has won a perhaps Pyrrhic victory by getting an injunction and getting any work to stop on the process of implementing the JEDI contract on Microsoft’s cloud. Amazon is posting a $42 million bond to cover any costs the government or Microsoft might incur if it turns out that this stoppage is unjustified—not small beans, but not a lot of money to either of these tech titans.

Which way does this jump ball go now? It seems like Microsoft would still have the upper hand, because they as well as the U.S. government would have contemplated this outcome when they decided to switch to a Microsoft solution at what seems like the last minute.

Amazon makes a compelling case as well: Because of the public animosity between two feuding billionaires, one a president, and one the richest man in America, government officials would be afraid to award a contract to Amazon’s AWS because they would suffer the wrath of Trump and his officials. The Trump administration has made it no secret that there is a revolving door for people who don’t get with the Trump administration and its policies. Sounds like a win for Amazon, right?

At the same time, Jeff Bezos’s leaked text and pictures between his mistress bring up a larger question: How can the U.S. government trust some of its most precious data when the CEO not only leaks such personal information, but it is so public that Mr. Bezos himself discussed what he considered ransom demands from a tabloid that threatened to cover it on a blog post on Twitter. Even worse, supposedly the pictures and texts were obtained by the leader of Saudi Arabia, a foreign country, through a bug in WhatsApp that allowed installation of malware on Jeff’s phone that allowed all the data from his phone to be siphoned off to a foreign entity. It’s difficult for AWS to say they are the best solution when their own CEO admits that his most personal data (a secret affair with his mistress) was stolen. What would happen to real military secrets if Amazon’s CEO can’t protect his own?

The fallout from all this is what you’d expect. More arm waving, conjecture and confusion. The reality is this opens up the field to other players such as Google and Oracle. At the same time, there are hundreds of other clouds that may be able to make a case that they are a best fit for certain applications since they may specialize in specific compliances, such as SOX, PCI or HIPAA.

Ultimately, this calls into question whether the government should be putting all its cloud eggs in one basket. The longer-term solution is that the government needs to dictate a series of APIs that its developers and contractors use, and that all compliant clouds would be required to support. This would alleviate the problem of centralizing resources on a single vendor’s solution.

In the medium term, lots of fireworks, lots of drama and most likely Microsoft, the original software empire, striking back and keeping the JEDI contract—one that was Amazon’s to lose. But the story isn’t written so we all need to keep watching for the return of the JEDI to see where this finally ends up.

— Marty Puranik

February 20, 2020
Six Routing Challenges When Managing Microservices in Production and How to Avoid Them

Even experienced DevOps teams can struggle with the nuances of managing microservices in production–where a greater spotlight on network communications can introduce new operational challenges beyond those common to traditional architectures. But, by being smart about your software-based routing components, it’s possible to avoid or reduce the impact of roadblocks in the following areas.

Implementing Tracing and Monitoring

Even applications that are rigorously tested during development will exhibit anomalies in production. To address these, DevOps teams require effective tracing and monitoring tools to gain visibility into what’s happening within clusters at runtime.

As a strategy, developers should select routing solutions that integrate well with their existing monitoring and tracing backends. Remember that while a microservice may be developed by a certain internal team, once deployed it becomes available for others. If the application is instrumented to be traceable, compatible tracing tools can dynamically identify the source of service invocations and their call flows–enabling other teams to utilize the microservice. When the application is also designed to provide metrics, monitoring tools can usefully track the microservice’s key resource consumption metrics to assess the scalability of its subcomponents under load, and recognize any performance limitations that require attention.

Because all inter-service communications travel through software routing elements, you can strategically design your application to enable microservices tracing and monitoring data to be collected. Teams can then put this data to work to more easily overcome challenges when building out new capabilities.

Mitigating Communication Failures

Considering the myriad communications taking place within production microservices-based environments, it only makes sense that this architecture will be prone to communication failures in which microservices simply cannot be reached. Whether these failures occur as the result of an error by the container, host, a network partition, or a short-term interruption in the availability of the service itself, resilient safeguards must be in place to mitigate any impact to an application’s user base.

For example: When an instance fails, the load balancer can be used to automatically reroute requests to healthy instances, and then return traffic to the failed instance upon detecting that it’s again available. In scenarios where an entire microservice becomes unavailable, client services should leverage circuit breaking to stop requests, return an error and initiate a fallback response. Doing so avoids dangerous repetition of retry requests, which only eat up resources and can lead to cascading failures.

Each of these mitigation strategies could be accomplished by placing the necessary logic with each client. However, this technique would be both cumbersome to implement and prone to errors. The better approach to these challenges is to leverage routing solutions to perform instance health checks and circuit breaking, allowing DevOps teams to control how their applications respond to faults, rather than relying on logic at the network layer.

Handling Unanticipated Load Spikes

In production environments, loads can spike unexpectedly and for any number of reasons–ranging from simple sudden popularity with users to malicious distributed denial-of-service (DDoS) attacks. Whatever the root cause, applications must be well-prepared to withstand these events without failing.

To ready applications to handle load spikes, rate limiting is a simple (but powerful) technique that allows operators to specify the request rates to front-end services. Implementing this safeguard through a routing solution answers the challenge of avoiding downtime due to surprise changes in load intensity.

Securing Microservices Communications and Permissions

Security is always a constant concern when it comes to operating production deployments. However, microservices environments often have absolutely zero enforcement of internal network communications permissions. They also face the challenge of implementing proper encryption to secure sensitive data within both internal and external traffic.

While allowing any service to access any other offers flexibility, the risks are substantial. In order to safeguard microservices that access and expose sensitive data, it’s wise to limit access to only those client services with a vetted business justification. Routing technologies make it possible to enforce these secure policies using network segmentation.

At the same time, routing solutions can help businesses encrypt the in-flight transport of critical data, which is not only a best practice, but in many industries a necessity from a regulatory compliance perspective. Service meshes can offer the benefit of providing secure TLS encryption for all internal east-west traffic, while edge routers offer to encrypt external north-south using a provided certificate. Also of note for north-south traffic: combining routing technology with services such as Let’s Encrypt can serve to fully automate lifecycle management of trusted certificates, eliminating the need for human intervention entirely.

Achieving High Availability

Leveraging software routing technology to achieve high availability is more cost-effective and efficient than traditional hardware failover solutions. It offers the opportunity to build production microservices environments that are quite resilient. For example: Routing implementation should utilize a highly-available architecture that includes a horizontally scalable data plane and a separate fault tolerant control plane. While the data plane allows for adding instances as necessary to meet capacity and resilience needs, the control plane is prepared to tolerate failures and ensure that users experience seamless and uninterrupted uptime.

Deploying Across Multi-Cloud and Heterogeneous Environments

Larger enterprises often utilize deployments across multiple cloud and/or on-prem environments, each of which may rely on disparate container orchestration technologies. Highly portable routing technologies capable of supporting a range of orchestration tools can vastly reduce the challenges developers face in supporting multiple environments. This allows them to deploy the same familiar solution and a common routing layer model across deployments.

Routing technology is a key variable in determining the capabilities and ease with which DevOps teams are able to overcome challenges to the success of their production microservices deployments. By making a carefully considered choice in selecting a routing solution, teams can better prepare for these challenges–or even avoid them altogether.

— Manuel Zapf

February 13, 2020
What Is DevSecOps and How to Enable It on Your SDLC?
For the past three to four years, all the companies around the IT world have adopted agile and different application development methodologies that leverage the work for different departments or areas and helps them to develop new products and release new features to improve their processes and infrastructure.

In this new Agile and DevOps world where everybody on a team is involved in the rapid-changing and evolution of their application, we are promoting accountability for everybody in terms of security—this is when DevSecOps joins the party.

What is DevSecOps?

DevSecOps is a new model that provides accountability for the security implementation in the application; from the planning, design, development, QA/testing, to release and when operating on a production environment.

When implementing DevSecOps on the Software Development Lifecycle (SDLC), an organization will experience the continuous integration and will notice that the costs for compliance are reduced, code is constantly being analyzed, tested, delivered and released properly.

DevSecOps enables the process of implementing security to everybody and makes them accountable.

Why Is It important?

As I stated previously on this blog, on this rapid-changing era, everything is evolving at a very accelerated pace. We continue to discover vulnerabilities and breaches across platforms and operating systems, patches are released constantly but we—as part of the operating team of a company—cannot afford the risk of having a vulnerability on any side of our IT system/application.

Main Benefits
- Reduces vulnerabilities present on your code.
- Reduces vulnerabilities present on your IaC technologies.
- Reduces the number of ways to exploit your application
- Reduces downtime.
- Improves your application stability, availability and security.
How Can I Enable DevSecOps on My Current DevOps Pipeline or SDLC?

DevSecOps is a must-have methodology that needs to be integrated into your DevOps process/pipeline to help you improve your security on your SDLC.

There are five important phases that need to be followed in order to enable DevSecOps on a current DevOps pipeline or in the SDLC. Here are the crucial phases to enable it:

Phase 1: Secure Local Development. Start by implementing secure working-environments. When you are developing an application, in most cases you will use open source technologies. Docker is a great helper at this phase since it automates the infrastructure and services deployments on local machines. So when you are using this ready-to-go docker environment, make sure that you are using the most recent/updated versions of the Docker Images and scan them for vulnerabilities. Even the images from official providers have vulnerabilities that need to be patched.

Phase 2: Version Control and Security Analysis. Enable Vulnerability while uploading your source code. Having multiple hands or people working at a piece of code can lead to vulnerabilities, especially when they are remote. Git systems have been a great improvement for collaboration between team members and code. When a team member uploads a piece of code, I strongly suggest that you enable automated testing for security on your code dependencies and core. Some good alternatives to do it are Snyk or Sonatype’s Nexus.

Phase 3: Continuous Integration and Build. When creating the development image/package, you’ll need to make sure that your build tool or system has the proper security in place. It uses https:// protocol, it’s properly hardened and secure, it’s available and protected for attack mitigation or even not accessible via the internet. The tools that you can use here are Jenkins, Circle CI, AWS CodeBuild, Google Cloud Functions, Azure DevOps.

Phase 4: Promotion and Deployment. When deploying to an environment, insert the environment variables through your CI/CD tool and try to manage them as secrets. Proper encryption and management of these are recommended in order to enhance your security protocols.

Phase 5: Infrastructure Security. When your app is deployed, make sure that you have an IDS (Intrusion Detection System). Tools such as OSSEC or Wazuh will help on this matter to protect all your hosts.

Once your code gets to the production, it doesn’t mean that it will be 100% secure. New vulnerabilities are disclosed every day but this cycle will help you and your team to test your code against all the repository of known vulnerabilities, at the time of monitor, configure, reconfigure, adapt and deploy solutions to the environment.

Tools and Processes

These are the tools you will need to enable in your DevSecOps process:
- CI/CD Tool.
- Secrets Manager.
- Version Control System.
- Docker Orchestration tools.
- Security Analysis tools.
And these are the processes:
- Automated regression testing.
- Dynamic testing.
- Automated Vulnerability Assessment.
- Automated Integration.
- Promotion.
- Automated Deployment.
What Is the Ideal Workflow for DevSecOps?
1. A developer creates a new code and integrates it into the VCS.
2. Members from the QA team retrieves the code to perform the static code analysis to identify security flaws or functional tasks.
3. A test environment is created automatically using IaC such as Terraform, CloudFormation, Chef or Puppet and the security configurations are added to the system.
4. The test automation suite is performed to the application with a tool usually Selenium or any other tool that performs backend, UI, integration, security and API tests.
5. After the test suite is performed and successful the new changes are sent to the production environment.
6. The new version of the code is now going to be monitored in the production environment using an APM or Cloud-native monitoring tool.
Following these points you are ensuring that your application is following TDD practices improving the code quality, compliance, increasing the number of releases of code to production and reducing the time to market which is essential for any organization.

What Are the Challenges When Enabling DevSecOps?
- Enabling too Many Tools: Enabling too many tools can become a problem on your SDLC, especially when your team is not used to work and relate with DevOps/security tasks. The main recommendation here is to start slow. Start by enabling only the necessary tools to get your team familiar with the process and add more when you feel your team is prepared for it.
- Getting Used to the Methodology: It will take some time for all the team to get used to the DevSecOps methodology/culture and also to keep following it in order to be compliant with the normative that your business demands. Try to always stay up-to-date and coaching your team for the newest technologies.
- Chasing Perfection on the Process: Keep in mind that all the DevSecOps process won’t be perfect, but it will get mature over time. Teams always try to chase perfection and this only leads to more problems with even more integrations or dependencies.
Conclusion

In the end, I think every organization must make the effort to shift to a DevSecOps methodology or process and come up with a multidisciplinary team with a focus on security. That’s how an organization will transition from doing DevOps to DevSecOps. Allowing all their collaborators to have accountability on the part they are actively developing.

— Guillermo Vélez
February 10, 2020