devopsy.kinsta.cloud

Category: DevSecOps

DevSecOps: A Renewed Commitment to Secure Delivery, Part 1
Security has never been as high a priority than it is today, as companies fear they’ll be the next headline, the next victim of a data breach. Executives also worry about applications meeting the high standards of compliance–either with global regulations such as GDPR, state-oriented privacy laws or the many specific ones covering finance, health care, energy and other industries.

It’s a bit of a challenge introducing new tools and processes into a longstanding software development life cycle, even when it’s for the sake of increasing security. That’s why companies need to first carefully review their development and production processes to fully understand how they create and release software before they can effectively integrate security into DevOps.

A reconstruction, so to speak, of application delivery allows companies to step back and smartly modernize their pipelines so there’s minimal human interaction or machine disconnect in newly secured DevOps processes. Involving IT security as a stakeholder in defining governance in a DevOps release pipeline is also crucial. Only then does the “Sec” fit nicely into DevSecOps. In this two-part series, we’ll look at the three critical components that shape a sound DevSecOps pipeline. They are easy to remember: processes, people and tools.

Vulnerabilities Call for a Maniacal Focus on Security

Although microservices, containers and cloud-native architectures enable companies to create and deliver applications more nimbly and faster than ever before, many of these innovative technologies also increase the probability of security flaws and vulnerabilities in products. Exposure to risk is much greater with applications being released at higher frequency and in higher quantity as smaller containerized deployment units, as opposed to development happening on a gated monolithic Java application.

For example, the provenance of images from public container registries can be compromised and thus a potential exposure point in development. Likewise for the proliferation of open source software that is often used as libraries and dependencies. Perennial tools, such as GitHub, have fostered sharing and open communities, sometimes with grave consequences to those contributors that publish sensitive data to public repositories. How many times have we heard about passwords being pushed to public repos for all the world to see?

With all those considerations in play and with so much at risk, companies now have a maniacal focus on securing DevOps. But before they can even start formally considering security, they should first review employee responsibilities, organizational accountability, delivery practices, governance and the other aspects that go into the design, development and delivery of digital products. For enterprises that are modernizing to cloud native platforms and technologies, it might be an opportune time to reexamine and freshen up approaches.

DevSecOps is indeed about people, processes and tools. Fully understand those three and you can build a secure delivery process that engenders trust and confidence in DevOps. Trust and confidence foster accountability and the notion that digital reputations are everyone’s responsibility.

Determine the Process for the Release Pipeline

Here, it’s about cementing a formal governance of processes, all the steps behind an application moving from one point to another. How does code graduate from local development and testing to the production station–and what happens in between? This is a fundamental tenet of the plain ol’ DevOps of yesteryear.

There are notable considerations in a cloud native pipeline. For instance, the promotion path might be described with Kubernetes namespaces and clusters (not the baremetal servers or VM environments of yesteryear). Another modern DevOps approach is to apply DevOps principles to the application and environment configuration artifacts. Also known as GitOps, it’s applying the use of version control, access management and lifecycle processes to yaml files, helm charts, monitoring scripts and other artifacts that are primarily a concern of operations. Extending DevOps to operations (or GitOps) mitigates risk in environment misconfigurations and configuration drift, an added benefit. Security, in this respect, is addressed through environmental compliance.

Aim to justify everything that happens in the release pipeline. Examine the value-add of steps and align them with the security risks they are trying to mitigate. Here are some examples:
- Unit tests must pass in the build phase of the release pipeline to mitigate the risk of non-working code being pushed to the Development environment and entering the promotion path.
- Vulnerability scanning must occur on all Docker images prior to go live to mitigate the risk of a security flaw or vulnerability being introduced in production.
- All API interfaces must have a Swagger definition and a set of corresponding Pact/contract tests run in build phase to mitigate the risk of integration defects and corresponding outages.
Seek Answers, Document Everything and Include Security

By questioning everything, the answers will reveal what falls under the security lens and what eludes it. Ultimately, you want to document practices and processes so that they’re codified and held with reverence by everyone along the pipeline. IT security should have a seat at the table (alongside development, operations, product owner, QA) and be an equally important voice in examining the release pipeline. A core agile value is to have a product-focused approach to the release pipeline. Involving IT security will bring a healthy measure of risk mitigation into the pipeline, while still considering velocity in delivery.

But that’s just the process. As we’ll explore in the second part of this series, people and tools also make a difference.

— Andrea C. Crawford
February 5, 2020
The Risks and Potential Impacts Associated with Open Source

Open source software (OSS) is built by communities of developers who contribute their knowledge and time to OSS projects they find appealing. That code can then be used by individuals, communities and organizations in their software products—the only obligation they have is to play under the rules of the license with which the OSS project was published.

This type of knowledge sharing brings many benefits to OSS users as it speeds up software development time and can help companies become more competitive in the market. Unfortunately, there is also a catch. Those benefits also come with certain risks which every OSS user needs to be aware of and take necessary actions to mitigate.

The OSS License

One specific risk to consider involves the OSS license. Not knowing what your obligations are under the license (or not abiding by those obligations) can cause an OSS user to, for example, lose intellectual property or experience a monetary loss.

Operational Impact on the Product

Another risk is the operational impact that the OSS in use will have on the product. It’s important to understand whether the community is consistently managing the OSS project and its issues in a timely manner, and that your organization has implemented the latest version and deployed patches have been applied. Using out-of-date, unpatched or unmaintained OSS could impact your release capabilities, product quality and customer trust.

Security Vulnerabilities

The third risk to consider when using OSS is that of security risk. As open source is software—and in many cases very complex software—it brings with it the possibility of introducing software security vulnerabilities into your source code. Some vulnerabilities can be introduced into your projects through dependencies that the OSS your organization is using has on other OSS projects. Additionally, vulnerabilities could be unintentionally introduced by developers as they code a functionality using open source. The reason for this is that software developers, whether professionals or hobbyists, oftentimes do not consider security while coding. This has a substantial impact on their work and their contributions to open source projects.

It’s not necessarily that developers intentionally skip the implementation of security procedures. Secure coding is a specific discipline in software development that guards against the accidental introduction of vulnerabilities—a discipline that must be learned.

Understand that many developers are self-taught or have been trained in schools and universities (where we are only recently seeing courses in software and application security emerging). Many use their software development skills as a hobby, to contribute to OSS projects or to pursue their own personal projects. Unfortunately, training programs, programming courses in schools and online tutorials often neglect to convey this knowledge or to stress the importance of building secure software.

It’s also important to note that many organizations find it difficult to introduce a secure coding discipline into their software development process. In other words, this struggle persists beyond open source projects. If you’re with an organization experiencing this pain point and unsure where to begin, you’ve come to the right place. There are several strategies that can be used to begin your secure coding journey.

Training

By giving developers the education required for them to become security aware, you’re setting them up for consistent, secure development success. There are a variety of in-person and computer-based training options on the market.

Software Composition Analysis (SCA) Tooling

Implementing an SCA tool will uncover incorrectly coded strings or vulnerabilities that have been introduced in code or OSS components. Not only can SCA tools assist developers in identifying potential software risks, some SCA solutions can help to train developers by offering remediation guidance or short training videos on how to avoid such vulnerabilities in the future.

It’s essential for software issues to be discovered early because the deeper into the process they’re identified, the more time-consuming and costly they are to remediate. As such, in recent years there has been a push to shift left, introducing security into the product development life cycle from the very beginning.

Companies and individuals need to also understand that this security mindset doesn’t start and end in the coding phase of the development process. A security mindset needs to be a part of every phase: from design to deployment to maintenance, all the way through to the product’s end of life.

Emphasizing a security mindset also has a long-term, broad reaching positive effects on the whole software development ecosystem, including the OSS world. As developers are trained to code securely (through university courses and other training institutions, coding portals or professional development through their employer) they pass this knowledge on to the projects with which they’re involved. This then leads to the production of higher quality proprietary and open source software, which in turn minimizes risk across the board.

— Boris Cipot

January 27, 2020
Software Supply Chain Attacks: How to Disrupt Attackers
Supply chain attacks—compromising an organization via insecure components in its software supply chain—are a growing concern for organizations. Throughout the past three years, an increasing number of open source software package repositories have been found to contain malware, making it clear that all installation and update pathways for software and library code must have security controls applied to prevent and mitigate supply chain attacks.

Growing Risks of Open Source Software to the Business

Digital transformation has made every company a software company. As a result, open source software has grown 75% over the past two years. Despite this growth, few companies accurately track the components used in their code. Most lack the policies, processes and tools to keep up with the choices made by their developers, which introduces risk. In fact, 60% of the codebases audited in 2018 contained at least one vulnerability.

Bad actors have turned to exploiting these weaknesses to compromise organizations. If production code is infected with malware or vulnerabilities, inadvertently sourced from the repository, it may contaminate all organizations that come in contact with it—whether by using the code already in their SDLC or by launching presumed trusted applications from third parties who failed to validate their own code.

Attackers are also shifting their techniques to mask malware among large packaged objects using obscure formats we refer to as destructive objects. They infiltrate at various points in the development processes opportunistically, knowing they can infiltrate systems outside a managed operational environment. They take advantage of third-party dependencies and use payloads that can sit and wait until an unsuspecting developer implements their software. Hence, software supply chain attacks are difficult to detect, and often achieve dwell-time of hundreds of days.

Assessing Supply Chain Surface and Sources

In order to assess weaknesses and identify what warrants continuous monitoring, we need to understand both the surface and sources.

When evaluating surface risks, consider the following questions regarding the development environment:

What technologies, repositories and packages are in use?
- Inventory technologies used in development: Python, Ruby, JavaScript, Node.js, .NET, etc. This can also refer to operating systems.
- Third-party repositories of code: PyPI, RubyGems, NPM, NuGet, PEAR or PECL, and OS repositories such as DebianRepository, CentOS, Ubuntu, AUR and more.
- Packages: There can be multiple packages from a single repository. How often are packages updated? How often are new packages added?
Are there update procedures in place?
- What are the dependencies for every package? How many other packages are pulled from a single package update, and how often are they updated and/or reviewed?
Who’s managing the verification processes across all these dependencies? What update procedures are in place if there is a personnel change?

Once these questions have been answered, they should be addressed regularly. The process must be iterative to be successful.

When considering attack sources, it’s important to understand the characteristics of repositories teams draw from in order to understand potential areas of weakness. Third-party repositories rarely infect a big project as they are typically found elsewhere (e.g. GitHub) and have thorough code reviews. Official repositories, unofficial mirrors and internal code repositories represent increasing risk potential.

Random malicious packages, typosquatting, bad code review and phishing developer credentials to hijack accounts are all less technical ploys used by attackers, but equally dangerous.

The Software Supply Chain Challenge

The size, breadth of formats, trusted third-party and open source risks, and misused certificates have made it nearly impossible to stop malware-infected files and objects with today’s signature based, sandbox and vulnerability-based solutions.

Threats evade dynamic analysis remaining unknown to defenders or lacking details when detected. The explicitness of signature blacklists misses new or never-seen-before code, and sandboxes or dynamic analysis solutions are resource intensive and unable to scale to the breadth and volumes of formats required. Without a way to inspect large or cross platform archives, installers and source code, it’s impossible to extract indicators of compromise. These destructive objects can also impact SOC triage, response and continuous improvement.

Rising to the Challenge

How can you address these risks, not only in the software development life cycle (SDLC), but as SOCs address vulnerabilities that may come through other applications and repositories? How do you evaluate and inspect wrappers, digital certificates, executables, scripts, files and libraries, resources, documents and icons?

It’s vital to establish security practices within SDLC, from training developers on secure coding practices using open source libraries to factoring in detection capabilities including static analysis, dynamic analysis, software composition analysis and manual penetration testing. Implementing a secure SDLC process ensures that the development effort is protected by these activities, augmenting code reviews and infrastructure analysis.

Teams can extend security controls into the SDLC with technology that provides deep visibility into all aspects of the supply chain, from open source dependencies through continuous integration and delivery of packaged applications. Capabilities that provide greater application security across code development and deployment activities include:
- File reputation identifies and classifies “known bad” and “known good” files by applying a file’s unique hash against a database of goodware and malware.
- Automated static analysis detects and classifies unknown malware in code repositories by deobfuscating and decomposing objects and files without executing, and extracting threat indicators.
- Dynamic analysis executes files in a safe environment and captures the behaviors and results of malware, allowing the security team to understand what is going to happen.
- Software composition analysis analyzes applications to detect vulnerabilities, dependencies and licensing requirements in software.
- Threat intelligence is served up at various stages of the analysis cycle, and represents evidence-based knowledge including context, mechanisms, indicators, implications and action-oriented advice about existing or emerging threats.
By incorporating these technologies into the SDLC, development teams can obtain continuous validation across the software supply chain. Further automation, implementing new security guidelines and advancing the activities within the existing framework will allow the enterprise to progress its security measures and close gaps in the SDLC and supply chain operations.

At the same time, it’s important to recognize that threats can still make their way into the software supply chain, and it’s equally important that the SOC remain vigilant about maintaining a clear triage and escalation process. Continuous monitoring of software repositories should be an integral part of good hygiene processes, and incorporated into best practices applied to operational infrastructure, including the SOC. The SOC serves as a centralized management infrastructure to respond, contain and remediate attacks. This includes aggregating and consolidating threat data, building a case for managing a response which can include actions to formulate triage and incident response plans, and moving beyond SIEM.

— Tomislav Pericin
January 17, 2020
DevSecOps: 10 Best Practices to Embed Security into DevOps
For companies that employ the agile approach, DevOps seems like a natural extension. Traditionally, enterprises started with integration, development and test automation early in the product lifecycle. Gradually, the agile delivery team took care of iterative development and monitoring practices that increase code quality.

Today, organizations work in unison to bridge the gap between development and operations (DevOps). They want to offer a faster market delivery with the least human interaction.

But what about security integrations? Is there a way to mitigate vulnerabilities early in the development lifecycle?

The answer is DevSecOps.

Exploring the Concept of DevSecOps

DevSecOps is short for development, security and operations. It brings together people, processes and technology to pursue a shared objective.

The objective of DevSecOps is to implement security decisions on the same scale as development and operations and make everyone in the product lifecycle accountable for security.

Why Should You Adopt DevSecOps?

People adopt DevSecOps because they are seeking:
- Modern alternative to traditional security engagement.
- Transparent collaboration and workflows during development.
- Security that’s built into the product, not applied at the final stage.
- Reduced expenses and faster delivery rate.
- Faster recovery speed in case a threat is detected.
Steps to a Typical DevSecOps Workflow
1. A developer starts by writing code within a version control system.
2. Any required change is committed to the version control system.
3. Another developer analyzes the code to identify any security defect that may weaken code quality.
4. An environment is created to deploy and apply security configurations to the system.
5. Next, a test automation suite is executed to evaluate the newly deployed application.
6. After it passes the automation test, the application is deployed to a production environment.
7. This new production environment is actively monitored for security threats.
While there is no right way to transform organizational culture, below are a few components necessary to sustain a DevSecOps environment:

Let Developers Get Security Right: Developers are responsible for security. Therefore, you must keep them on top of cybersecurity best practices through continuous training and learning activities.

Promote an Open Culture: Openness in communication within the enterprise environment can drastically improve development and security. One way to keep information transparent is by using metrics and dashboards wherever possible.

Get Experts on Board: It is extremely difficult to transition from DevOps to DevSecOps without the supervision of expert security professionals. Hire people who understand security within the development and operations environment and let them train your DevSecOps team for the big transition.

Tempted to embed security into DevOps? How do you ensure that the best practices are followed? We have answers.

The Best Way to Implement the DevSecOps Process

Gather a single group of professionals (admins, developers, security engineers and testers) that are aware of your product from start to end. They should know your requirements and should be experts in deploying, monitoring and implementing new changes.

Once you have your team ready, here’s what you need to do next.

Plan

Planning is crucial. Do not just stick to feature descriptions. Instead, go for detailed user stories that include:
- Functional and nonfunctional requirements (e.g., security and performance).
- UI and UX designs.
- Acceptance test criteria.
- Threat models.
Develop

Start by evaluating your existing practices. Choose the best resources to build a development model in coherence with security guidelines.

Build

Automated build tools can do a lot more than compile codes. Use them to conduct test-driven development, enforce quality standards and ensure that the best security practices are implemented through static code analysis.

Test

When it comes to a DevSecOps environment, test automation is not limited to UI-focused Selenium tests. Optimally, your security practice should include the following:
- Unit testing.
- Front-end testing.
- Back-end testing.
- API testing.
- Database testing.
- Passive security testing.
Secure

Because development, operations and security go hand in hand, only a few issues are left unattended toward the end of the development process.

When vulnerabilities are identified, there is a better chance of determining if they are potential exploitations or false positives.

Deploy

Automated provisioning and deployments can be utilized to accelerate product delivery and add consistency in the development process. Using an infrastructure-as-code tool, one can audit properties across the IT infrastructure and enforce secure configurations in a system.

Operate

Routine maintenance and upgrades should be an indispensable component of your operations team. Leverage infrastructure-as-code tools to patch zero-day vulnerabilities and apply updates to the entire organization’s infrastructure.

Monitor

A continuous monitoring plan should be in action to generate real-time stats of how your system is performing. In case any exploitation is recorded, it can be addressed immediately.

Scale

Traditional data-center operations cannot entirely replace a compromised environment. Today’s ability to scale infrastructure through virtualization and the cloud, while addressing the demands of modern-day IT user base, should go a long way.

Adapt

When it is about sustaining an agile practice, continuous improvement is key. This is also true for DevSecOps practices, as you improve and adapt throughout the software development lifecycle.

Conclusion

DevOps isn’t going anywhere, anytime soon. It is the new phase of developing, releasing and updating products in a software lifecycle.

That’s why high time security professionals let go of the traditional security stack and embrace security solutions at the speed of DevOps.

— Deepak Gupta
January 14, 2020
API Security in DevOps: Are We Too Comfortable?

Postman performed its annual survey of developers, and as this survey and many other surveys have shown, developers are generally comfortable with the level of API security that their organization has implemented. This has been a consistent trend, with most developers being comfortable that they are protected.

The problem is most organizations also put API security and management in the hands of developers.

Knowing the rate of change going on in Agile/DevOps shops, and knowing the complexity of environment, I think this mentality is akin to the pre-WW2 French belief in the Maginot Line.

While API security is certainly a necessity, it is not managed in a vacuum. You really are only as secure as your weakest link, and there are a lot of links in an API-focused environment. User credentials, tokens, SSL, requesting apps and their authentication, (still) code injection concerns…The list goes on and on.

I’m never comfortable with the level of API security currently implemented in DevOps environments. And I think it’s dangerous for you to be. It only takes one bad code iteration for one of the client instances to be vulnerable. That vulnerability exposes every API the client in question has access to, no matter what is being used as access control.

In the end, APIs must trust someone, or they’re useless. And that someone is most often a server app running on an instance that your users connect to. That app most often has its own credentials. How are they stored? How are they protected? What code changes could impact that instance’s security?

API gateways and management tools, when properly configured, are a big help here, but they are not a panacea. Since incoming connections must be trusted at some point, and outgoing connections must be assumed clean at some point, when one of those two things is a wrong assumption, what do you have in place?

Here’s the scenario: User A has rights to call the ShowMeCriticalData API. This API is called from a server app running on instance B. The server app authenticates user A, then calls the API for him–either with his credentials (best case) or its own (bad for tracking problems). If instance B receives an update that makes it vulnerable to compromise, user credentials are still needed to call ShowMeCriticalData, which is relatively secure, as long as User A is protecting their credentials adequately. However, if the app running on instance B is proxying requests and using its own credentials, Instance B becomes a weak point and attackers could use the vulnerability to call APIs without authenticating users. Worse, if User A is careless with their authentication info, there is little an organization can do to stop at least some misuse.

API logging and review can be put in place to notice odd behavior by username, and offer early warning that a user has been compromised. But will only catch most issues.

Of course, there are smart people working on the whole array of API security threats from cross site scripting to authentication spoofing even into TLS weaknesses. But that work is only useful if we all remember that our security could be better. With knowledge that there are weaknesses, we can look for things to mitigate the risk. As you all know, if you have public APIs the risk will never be eliminated–the purpose of a public API is for people to use it, and that exposes it to hackers that seem to be more ingenious by the week–but minimizing the risk is worth the effort.

Being secure in your current situation means you’re not focused on making it better. With API security, particularly in the type of dynamic environment DevOps creates, this is a dangerous road to be on. Assume there is a fatal flaw, and work to reduce vulnerabilities at each access point into the overall system.

This also goes for APIs you don’t own. We rate limited an API that called out to Google Places to match up with our plan/budget for the project. Our app was never going to come anywhere near Google’s caps on usage, but should a portion of our infrastructure be compromised, we could potentially end up with a massive bill from Google because we called the API a zillion times in a month. Likely? Not really. Possible? Not after a few lines of rate-limiting code.

You all have made some astounding apps. Make sure they are astoundingly protected. Because both business and users are counting on it.

— Don Macvittie

January 8, 2020
Service Meshes: Improving Security, Delivery and Availability
Containerized microservices are taking over the software world. These services provide developers a way to quickly create and deploy cloud-native applications. However, containers aren’t perfect. Orchestrating and managing container deployments can be a serious challenge.

To help ease this challenge, service meshes were developed. In this article, you’ll learn about service meshes components and architecture options. You’ll also learn how services meshes can improve your container deployments.

What Is a Service Mesh?

A service mesh is a dedicated communication layer that sits on top of your request/response layer. It works similarly to a Container Networking Interface (CNI), which controls the connection of your services to your network. A service mesh sits on top of your CNI, adding additional functionality, such as service discovery or security. A service mesh is not a replacement for a CNI or a service orchestration platform.

Services meshes are designed for use with containerized microservices. You can use this technology with any container or container orchestration tools. However, the most active projects seem to favor developing service meshes for Kubernetes.

Service Mesh Architectures

When deploying a service mesh, there are three architectures you can choose from.

Library

The library architecture employs a library that is imported in an application’s code and runs from inside the application. Using a library eliminates the need to call out to a remote service, making it more secure.

Libraries make it easier to allocate resources since all work is contained in the microservice running the library. A library architecture works well for in-house applications that are written in one language. Third-party applications are difficult or impossible to add the library to.

Node Agent

A node agent architecture uses a node agent or daemon that serves all containers on a particular node. A node agent architecture is language-agnostic since it doesn’t run from inside your microservice. It can be used with in-house and third-party applications. However, node agents must be coordinated with infrastructure processes, making this architecture more complex to deploy.

Node agents require less memory than sidecars since information can be shared across a node. This architecture employs work resource sharing, with dynamically shared resources, making it easier to scale. The downside of sharing, however, is that it can be abused, with one service hogging all available memory.

Sidecar Proxy

A sidecar proxy architecture uses sidecars that run alongside your containers. The sidecar handles all traffic into and out of the container it is attached to. Sidecars do not require coordination with your infrastructure since one proxy is assigned to each container. This lack of coordination makes sidecars easier to set up than node agents.

Sidecars only serve one pod and are not influenced by other pods. This isolation makes this architecture better for zero-trust deployments as each container is accountable for its own actions. Sidecars can be added to containers at any time, enabling you to easily scale processes.

Use Cases For Service Meshes

Service meshes are an emerging technology, with new use cases being actively developed. The three use cases introduced here are just the ones that are currently most common.

Improving Security

Containers are isolated environments and generally considered to be more secure than traditional applications. However, containers still contain vulnerabilities, such as possible abuse of root access.

Service meshes can help address security concerns by providing:
- Zero-trust security—any communication, either from inside or outside requires verification. Verification prevents compromised containers from interfering with other microservices or your broader system. It restricts a container’s actions and communications to that container alone.
- Communication tracing—enables you to collect communications information and isolate errors or issues. If a container is communicating in an unexpected way or against protocol, you can identify it. Once identified, you can easily replace the container with an uncorrupted image or remedy the issue. Tracing enables you to pinpoint possible attacks, including where attackers came from and what they’re trying to accomplish.
Improving Delivery

Containers already improve on application and service delivery by making it easier to release and patch services. Service meshes expand upon these capabilities by improving application and performance monitoring functions.

Service meshes can also help improve service delivery by enabling:
- A/B testing—provides advanced routing, enabling you to divide traffic between A and B versions. This routing makes it simpler to set up A/B testing. Additionally, tracing features enable you to gather and analyze user traffic and interactions.
- Rapid versioning—new versions can be released with zero downtime and reduced risk. Containers can be started and verified healthy before traffic is gradually shifted between versions. If issues arise, changes can be rolled back without system-wide consequences. Easier corrections enable you to relax delivery requirements with minimal risk.
Improving Availability

Containers can provide improved availability, depending on your hosting configuration and resources. Service meshes can expand upon these abilities by improving load balancing, traffic management and health monitoring functionalities.

Service meshes can also help improve availability by enabling:
- Resilience testing—enables you to test how services respond with fault injections. Service meshes enable you to test your services’ fault tolerance. Increased understanding of how faults affect resilience helps you identify and correct vulnerabilities. Greater service resilience leads to greater availability.
- Deployment and request shadowing—enables you to simulate anticipated traffic flow and see how services respond. Shadowing enables you to test the traffic and workload capacities of your services. With this information, you can determine whether you need to scale resources and connections. Proper scaling ensures that your traffic needs are met and prevents traffic bottlenecking and decreased availability.
Conclusion

As microservices become standard, teams are placing an increased focus on improving container performance and security. The development of service meshes is a result of this increased focus.

Hopefully, this article helped you understand what service meshes are and how this technology can help improve your container deployments. If you’re interested in seeing how a service mesh might help you, check out this tutorial explaining how to build a service mesh.

— Gilad David Maayan
January 7, 2020
Certificates a Fly in the DevOps Ointment

The way cybersecurity teams have managed certificates is turning out to be a major impedance that could best be addressed by more organizations adopting best DevSecOps practices in 2020.

A survey of 108 attendees at the recent DevOps Enterprise Summit 2019 conference conducted by Venafi, a provider of tools for assigning identities to machines, finds 75% of respondents said they are concerned that corporate certificate issuance policies slow down development.

In addition, 39% of respondents believe developers should be able to circumvent corporate certificate issuance policies to meet service level agreements (SLAs) and about half (48%) said they are confident that developers always request certificates through authorized channels.

Kevin Bocek, vice president of security strategy and threat intelligence at Venafi, said the root cause of the friction between developers and cybersecurity teams can be traced to reliance on ticket-based systems for managing IT. DevOps teams in the age of the software engineer are building and deploying applications faster than ticket-based systems based on manual requests can keep up with, he said.

Rather than wait on IT Operations teams, DevOps teams are doing an end-run as business leaders push for applications to be deployed faster. In many cases, DevOps teams are attaining certificates directly from a cloud service provider such as Amazon Web Services (AWS), Bocek noted. Unfortunately, that can lead to other issues: More than half of respondents(55%) said their organization experienced a certificate-related outage in the past 12 months. Not all those outages are likely to have resulted from DevOps teams working without internal IT, but it does confirm how deeply flawed the existing process of issuing certifications already is.

To address that issue, Bocek said organizations will need to shift to an approach for issuing trusted certificates that is based on a standard set of REST application programming interfaces (APIs) and is a natural extension of the DevOps processes baked into a continuous integration/continuous delivery (CI/CD) pipeline.

Unfortunately, most organizations are not especially aware of best practices for issuing trusted certificates. The Venafi survey found three-quarters of respondents (75%) are unfamiliar with the “Securing Web Transactions 1800-16 Practice Guide,” available in draft form from the National Institute of Standards and Technology (NIST).

Bocek said the days when cybersecurity teams could employ certificates as a means for governing when applications are deployed are all but over. Cybersecurity teams need to collaborate with DevOps teams to define a set of best DevSecOps processes through which trusted certificates still play a role in helping secure the IT environment without slowing down the application deployment process, he said.

It may take a while for DevOps teams and cybersecurity professionals to get on the same page in terms of issuing certificates. However, as pressure to deploy applications faster continues to mount, cybersecurity teams will need to adjust their policies and processes. How certificates are issued may be as good a place to start their DevSecOps transition as any.

— Mike Vizard

January 6, 2020
Breaking Down the OWASP API Security Top 10, Part 2

Due to the widespread usage of APIs, and the fact that attackers realize APIs are a new attack frontier, the OWASP API Security Top 10 Project was launched. From the beginning, the project was designed to help organizations, developers and application security teams become increasingly aware of the risks associated with APIs. This past September, the OWASP API Security Top 10 release candidate (RC) was finalized and published on OWASP.

In my first article on this topic, I provided a high-level view of the interaction between API endpoints, modern apps and backend servers, in addition to how they’re different from their traditional browser-based counterparts. I also discussed why this project was so important to the contributors and industry overall.

In my second article, I focused on the first five risks and emphasized some of the respective possible attack scenarios. In this article, I will attempt to clarify the last five risks to help organizations understand the dangers associated with deficient API implementations. The following discussion follows the same order as found in the OWASP API Security Top 10.

API6:2019 – Mass Assignment

Modern frameworks encourage developers to use functions that automatically bind input from the client into code variables and internal objects. Which means users should have the ability to update their user name, contact details, etc. (within their profiles for example), but they should not be able to change their user-level permissions, adjust account balances and other administrative-like functions. An API endpoint is considered vulnerable if it automatically converts the client input into internal object properties, without considering the sensitivity and exposure level of these properties. This could allow an attacker to update things they should not have access to.

Example Attack Scenario:

To illustrate this further, imagine a ride sharing application provides the user the option to edit basic information about themselves in their user profile. For example, they can adjust their user name, age, etc. In this case, the API request would look like this: PUT /api/v1/users/me with the following legitimate information {“user_name”:”john”,”age”:24}

However, the attacker determines the request GET /api/v1/users/me includes an additional credit_balance property (field) as shown below.

{“user_name”:”john”,”age”:24,”credit_balance”:10}.

The attacker desires to increase their credit balance on their own and replays the first request with the following payload:

{“user_name”:”john”,”age”:24,”credit_balance”:99999}

Since the endpoint is vulnerable to mass assignment, the attacker can easily adjust their own credit_balance at will, for example changing it from 10 credits to 99999 as shown above.

API7:2019 – Security Misconfiguration

Attackers will often attempt to find unpatched flaws, common endpoints or unprotected files and directories to gain unauthorized access or knowledge of the system they want to attack. Security misconfigurations can not only expose sensitive user data, but also system details that may lead to full server compromise.

Example Attack Scenario:

Say for instance that an attacker uses a popular search engine such as Shodan to search for computers and devices directly accessible from the internet. The attacker found a server running a popular database management system, listening on the default TCP port. The database management system was using the default configuration, which has authentication disabled by default, and the attacker gained access to millions of records with PII, personal preferences and authentication data.

API8:2019 – Injection

Injection vulnerabilities cause computing systems to potentially process malicious data that attackers introduce. To put it in its simplest terms, attackers inject code into a vulnerable piece of software and change the way the software is intended to be executed. As a result, injection attacks can be somewhat disastrous, since they normally lead to data theft, data loss, data corruption, denial of service, etc.

Example Attack Scenario:

Suppose an attacker starts inspecting the network traffic of their web browser and identifies the following API request designed to help a user recover their password. The attacker identifies the request responsible to start the recovery-password process as follows:

POST /api/accounts/recovery {“username”: “john@somehost.com”}

Then the attacker replays the request with a different payload:

POST /api/account/recovery {“email”: “john@somehost.com’;WAITFOR DELAY ‘0:0:5’–“}

By adding the ;WAITFOR DELAY ‘0:0:5’–” the attacker observes that the response from the API took five seconds longer, which helps confirm the API is vulnerable to SQL injection. Exploiting the injection vulnerability, the attacker was able to gain unauthorized access to the system.

API9:2019 – Improper Assets Management

Old API versions are often unpatched and can become an easy way to compromise systems without having to fight state-of-the-art security systems, which might be in place to protect the most recent API versions. Attackers may gain access to sensitive data, or even takeover the server through old, unpatched API versions connected to the same database.

Example Attack Scenario:

Say for instance that an organization redesigning their applications forgot about an old API version (api.someservice.com/v1) and left it unprotected, and with access to the user database. While targeting one of the latest released applications, an attacker found the API address (api.someservice.com/v2). Replacing v2 with v1 in the URL gave the attacker access to the old, unprotected API, exposing the personal identifiable information (PII) of millions of users.

API10:2019 – Insufficient Logging and Monitoring

Without logging and monitoring, or with insufficient logging and monitoring, it is almost impossible to track suspicious activities targeting APIs and respond to them in a timely fashion. Without visibility over ongoing malicious activities, attackers have plenty of time to potentially compromise systems and steal data.

Example Attack Scenario:

Imagine that a video-sharing platform was hit by a large-scale credential stuffing attack. Despite failed logins being logged, no alerts were triggered during the timespan of the attack, and it proceeded without being noticed. As a reaction to user complaints about a possible breach, API logs were analyzed and the attack was detected, way after the fact. The company had to make a public announcement asking users to reset their passwords, and report the incident to their regulatory authorities.

Pertaining to the five risks above, one could easily imagine similar attack scenarios. Those provided were just examples of the nearly unlimited possibilities when attacking vulnerable API implementations. Hopefully, you can see the risks above are primarily caused by errors or oversights that could be easily remedied, especially when it comes to the way organizations utilize APIs today.

— Erez Yalon

January 3, 2020
Secure Your Network: How to Integrate EDR and DevOps

DevOps and Endpoint Detection and Response (EDR) solutions converge when no human element is required while deploying applications or detecting the threat and executing preventive measures. In this article, I am evaluating how the DevOps process integrates EDR solutions for securing services operations at various endpoints in the network.

What is DevOps?

DevOps has an impact on the mindset and technology landscape of software development companies to build cloud compatible services in a highly competitive market environment. Unification of development and operations is streamlined in DevOps that results in more innovation in technology domain driven by software. With DevOps, automation has become a more critical element than software development cycles. Now, updates, fixes and patches to software application services become easy with DevOps.

DevOps Challenges

Over time, it is realized that security is overlooked while moving to DevOps culture. Here are some of the challenges the DevOps approach might have:

Absence of Security Knowledge: It is a known fact that a lack of knowledge of secure coding and ignorance of robust security practices by developers can cause a long development cycle. It can happen due to training to developers for following security methods and guidelines while writing code and committing for the build. It has to be addressed at the development level by companies.

Open-Source Software Security: Use of open source frameworks has increased over time in a DevOps approach. With open-source projects, DevOps teams get pre-made code snippets to enhance the functionality of applications. But a striking fact is that 41% of cyber-security applications found high-risk open source vulnerabilities. To encounter this challenge, the DevOps team should be literate to track updates to open source frameworks used in DevOps toolchain, apply fixes to vulnerabilities as it becomes available. Also, to integrate open source, only use the trusted repositories.

Inverse Speed for Security Testing: DevOps introduces automation with agility in developing and releasing software applications. The speed at which developers commit code and build an executable does not allow the testing team to deep dive into code efficiently. In the traditional model, testers used to get a fair amount of time before applications go live. With the new agile approach, the duration for testing becomes much lower. To tackle this, automation in testing needs to be in line with development.

What is EDR?

EDR solutions offer a new approach to continuously monitor network endpoint devices for malicious activity and prevention of the attack and trigger a further course of action based on the pattern of attack. These solutions analyze endpoint data actively to provide detection and prevention in real-time. EDR solutions analyze the behavior of endpoints users and/or devices for suspicious activity and send alerts to the system.

EDR replaces the use of digital signatures to detect threats and provide behavior-based solutions for proactive security threat intelligence. All the processes of EDR solutions are based on activities, events and interactions on endpoints or with endpoints.

In the realm of CyberSecurity, EDR security offered to enterprises has the topmost priority due to its nature of being proactive and automated.

How Does EDR Solve DevOps Challenges?

DevSecOps is all about introducing and utilizing tools/solutions to secure continuous integration, development and delivery of applications. These applications made for services run at either server-side or endpoint devices. EDR solutions work actively in systems where endpoints have participated. EDR solutions can be integrated within the DevOps cycle so malicious activity can be tracked and hunted down by developers. This can be the quickest and automated way to tackle security threats.

The public cloud provides an environment for developers for the development, testing and execution of applications. The DevOps approach is mostly consumed by public cloud vendors with their set of supporting tools. But the cloud has its security concerns—such as any misconfiguration or code glitch in the application resulting in a huge attack. Also, cloud resources are mostly accessed by various types of devices which may or may not be secure. Some developers use their own devices for the development and monitoring of the application. EDR solutions can help in both cases by applying itself at endpoints as well as at public cloud infrastructure end.

Conclusion

EDR solutions are based on machine learning technologies that generate actions based on analytics information provided by sources/endpoints. EDR tools integration with the DevOps toolchain takes automation in process execution at much further level to track security breaches in run time and hunt down vulnerabilities within code before the application goes live. Additionally, it has seen that endpoint devices are more significantly emerged in generating security incidences. It has become imperative for DevOps teams to look after endpoint protection.

— Sagar Nangare

January 2, 2020
Best of 2019: Breaking Down the OWASP API Security Top 10, Part 1

As we close out 2019, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the fifth in our weeklong series of the Best of 2019.

As a result of a broadening threat landscape and the ever-increasing usage of APIs, the OWASP API Security Top 10 Project was launched. From the start, the project was designed to help organizations, developers and application security teams become more aware of the risks associated with APIs. This past September, the OWASP API Security Top 10 release candidate (RC) was finalized and published on OWASP.

In my previous article, I provided a high-level view of the interaction between API endpoints, modern apps and backend servers, in addition to how they’re different from their traditional browser-based counterparts. I also discussed why this project was so important to the contributors and the industry overall.

In this article, I aim to clarify the first five risks by highlighting some of the possible attack scenarios to help organizations and end-users understand the dangers associated with deficient API implementations. The following discussion follows the same order as found in the OWASP API Security Top 10.

API1:2019 – Broken Object Level Authorization

Attackers can exploit API endpoints vulnerable to broken object level authorization by manipulating the ID of an object sent within the client request. What this means is the client can request information from an API endpoint that they are not supposed to have access to. This attack normally leads to unauthorized information disclosure, modification or destruction of data.

Example attack scenario:

Say for instance there is an e-commerce platform that provides financial and hosted services to a group of different online stores (shops). The platform provides an API used to gain access to revenue charts for each of their hosted stores, and each store should only have access to their own revenue charts. However, while inspecting the client request from a single store who wants to gain access to their own revenue charts, an attacker can identify (find) the API endpoint for those revenue charts and identify the URL in use, for example /shops/{shop name}/revenue_data.json. Using the names of other stores being hosted on the e-commerce platform, an attacker can create a simple script to modify the {shop name} ID object in subsequent requests, and gain access to the revenue charts of every other store.

API2:2019 – Broken Authentication

Being different than authorization discussed above, authentication on the other hand is a complex and confusing mechanism concerning APIs. Since authentication endpoints are exposed to anyone by design, the endpoints responsible for user-authentication must be treated differently from regular API endpoints and implement extra layers of protection for credential stuffing attempts, in addition to brute force password and token guessing attacks.

Example attack scenario:

Suppose an attacker obtained a list of leaked username/password combinations as the result of a data breach from another organizations. If the API endpoint handling authentication does not implement brute force or credential stuffing protections such as CAPTCHA, rate-limiting, account lockout, etc., an attacker can repeatedly attempt to gain access using the list of username/password combinations to determine what combination(s) work.

API3:2019 – Excessive Data Exposure

By design, API endpoints often expose sensitive data since they frequently rely on the client app to perform data filtering. Attackers exploit this issue by sniffing the traffic to analyze the responses, looking for sensitive data that should not be exposed. This data is supposed to be filtered on the client app, before being presented to the user.

Example attack scenario:

Image an IoT-based camera surveillance system allows administrators to add a newly-hired security guard as a system user, and the administrator wants to ensure the new user should only have access to certain cameras. These cameras are accessible via a mobile app the security guard uses while at work. The newly hired security guard’s mobile app makes an API request to an endpoint in order to receive data about the cameras and relies on the mobile app to filter which cameras the guard has access to. Although the mobile app only shows the cameras the guard can access, the actual API response contains a full list of all the cameras. Using the sniffed traffic, an attacker can manipulate the API request to show all cameras, bypassing the filtering on the mobile app.

API4:2019 – Lack of Resources and Rate Limiting

It is common to find API endpoints that do not implement any sort of rate limiting on the number of API requests, or they do not limit the type of requests that can consume considerable network, CPU, memory and storage resources. The amount of resources required to satisfy a request greatly depends on the user input and endpoint business logic. Attackers exploit these issues causing denial-of-service attacks and associated endpoint outages.

Example attack scenario:

Let’s say an attacker wants to cause a denial-of-service outage to a certain API that contains a very large list of users. The users’ list can be queried, but the application limits the number of users that can be displayed to 100 users. A normal request to the application would look like this: /api/users?page=1&size=100. In this case, the request would return with the first page and the first 100 users. If the attacker changed the size parameter from 100 to 200,000, it could cause a performance issue on the backend database, since the size parameter in use is so large. As a result, the API becomes unresponsive and is unable to handle further requests from this or any other client.

API5:2019 – Broken Function Level Authorization

Although different than API1 above, exploitation of this issue requires the attacker to send API requests to endpoints that they should not have access to, yet are exposed to anonymous users or regular, non-privileged users. These types of flaws are often easy to find and can allow attackers to access unauthorized functionality. For example, administrative functions are prime targets for this type of attack.

Example attack scenario:

To illustrate this further, imagine during the registration process to a certain application that only allows invited users to join, the mobile app triggers an API request to GET /api/invites/{invite_guid}. GET is a standard HTTP method used to request information from a particular resource. In this case, the response to the GET contains details about the invite, including the user’s role and email address.

Now, say an attacker duplicated the request and manipulated the HTTP method by changing GET to POST. POST is an HTTP method used to send information to create or update a resource. The URL would look like this: POST /api/invites/new/{“email”:”hugo@malicious.com”,”role”:”admin”}. In this case, the attacker easily exploits this issue and sends himself an email invite to create an admin account.

In the context of the five risks above, one could easily image many similar attack scenarios. Those provided were just examples of the nearly unlimited possibilities when attacking vulnerable API implementations. As you can see, the risks above are primarily caused by errors or oversights. I believe these risks could easily be managed or nearly eliminated when organizations improve their secure coding practices, especially when it comes to the way they’re utilizing APIs.

— Erez Yalon

January 1, 2020