devopsy.kinsta.cloud

Tag: pipelines

  • Using LLMs to Automate Pipeline Conversions From Legacy to Tekton

    Using LLMs to Automate Pipeline Conversions From Legacy to Tekton

    There have been many technological advancements in the market today, but the main hindrance is the cumbersome process of switching from the existing technology/framework to the new one. This is where teams and organizations are left behind in the tech curve. In the DevOps landscape, CI/CD pipelines are the backbone of DevOps, and how you work with them largely dictates your DevOps efficiency.

    CI/CD pipelines, too, like other major technologies and frameworks in the past, have seen various evolutions; the latest among them is being made reusable in an attempt to standardize CI/CD. However, here too, there are several cons:

    1. The pipelines are not proprietary to end-user enterprises and belong to the DevOps platforms which are not open-source
    2. Migrating to these platforms and pipelines is a huge task
    3. They may not be as efficient as depicted

    Speaking of new advancements in technology, most might be familiar with Tekton for CI/CD pipelines: It is Google’s open source framework that is now a graduated CDF project. It redefines how pipelines are built (through reusable tasks) and these tasks are open source and community-vetted. They allow for pipeline authorship, and enterprises can re-use them across platforms that are compatible with Tekton, thus preventing vendor lock-in.

    While this mitigates the first challenge of pipeline ownership and migrations, however, the same problem plagues DevOps teams – transitioning from existing legacy technologies/frameworks to newer, much more capable frameworks like Tekton.

    However, what if there was a way to seamlessly automate your transition from legacy pipelines to cutting-edge, reusable Tekton pipelines without toil? This is precisely what the article aims to explore – automating the conversion of Yaml scripts of existing pipelines to Tekton Yaml scripts.

    Let us consider the three most popular CI/CD platforms for this use case – Jenkins, GitLab and Azure DevOps. Before we proceed to the proposed approach to achieve this (which is detailed in the second half of this article), let us first look at the limitations of Jenkins, GitLab and Azure Pipelines and why one should convert.

    Common Limitations of Jenkins, Azure and GitLab Pipelines

    Resource Intensive: Legacy pipelines can be resource-intensive, especially for larger organizations or projects. Managing server resources, optimizing configurations and ensuring sufficient hardware can be a challenge.
    Complex Setup: Setting up and configuring Jenkins, GitLab and Azure DevOps can be complex, particularly for self-hosted instances. This complexity may require experienced administrators and substantial time investment.
    Plugin and Integration Compatibility: All three platforms may face compatibility issues with certain plugins or integrations.
    Scaling Challenges: Scaling in organizations, especially when dealing with multiple microservices, necessitates a shift from individualized scaling to a template-based approach. Failing to do so leads to a chaotic increase of pipelines, undermining standards and organizational processes.
    Costs: While both Jenkins and GitLab offer free and open-source versions, some advanced features or capabilities may only be available in higher-tier paid plans. This can lead to cost considerations at scale.
    Customization Complexity: These platforms can be complex to customize workflows and integrate custom scripts as they require advanced scripting knowledge.
    User Interface Complexity: Though they have feature-rich interfaces, it can sometimes be overwhelming for new users to navigate the platforms and find specific settings or options.
    Limited Native Integration & Plugins: The native integrations of Jenkins, GitLab and Azure may not cover the full spectrum of tools and services a team may need. Additional integration efforts or third-party tools may be required to bridge gaps in the development workflow.

    As can be seen in the above limitations, many are common including scalability challenges, being resource intensive, limited integrations etc. All the challenges boil down to one common denominator: Complex pipeline management at scale.

    This is where a framework like Tekton can help tackle all the challenges at once, by just easing pipeline management: with its K8s-native pipelines that are reusable, open source and tasks that are community-vetted.

    Tekton Pipelines: Advancing CI/CD with Kubernetes-Native Automation

    Tekton, an open-source project, is rapidly gaining traction as a game-changer in the world of CI/CD pipelines. Its Kubernetes-native design is a standout feature, seamlessly integrating with Kubernetes clusters to harness its scalability and resource efficiency.

    Furthermore, Tekton’s cloud-native roots ensure it can execute tasks in lightweight, ephemeral containers, optimizing resource utilization. The ability to declare CI/CD pipelines as code, coupled with GitOps practices, enhances automation and reproducibility.

    How are Tekton Pipelines Different?

    Kubernetes-Native Automation: Tekton pipelines are deeply integrated with Kubernetes, utilizing Custom Resources and Operators to define and execute CI/CD workflows.

    Declarative and Modular: These pipelines are defined declaratively using YAML manifests, promoting version control, collaboration and code reviews. The modular design of tasks and pipelines enables teams to create reusable components for building, testing and deploying applications.

    Containerization and Isolation: Each task in a Tekton pipeline runs within a dedicated container, ensuring isolation and reproducibility of the build and deployment environment.

    Event-Driven Architecture: Tekton pipelines can be triggered by various events, such as code commits, pull requests, or external events, providing a responsive and event-driven CI/CD process.

    Scalability and Efficiency: Tekton pipelines leverage Kubernetes’ inherent scalability, enabling efficient execution of tasks across a dynamic pool of worker nodes.

    Visibility and Observability: They offer built-in monitoring and logging, allowing teams to gain insights into pipeline execution, track progress and diagnose issues.

    Integration and Extensibility: Tekton pipelines can be easily extended with custom tasks and integrations, enabling teams to connect with various tools, services and cloud platforms.

    Cloud-Native Best Practices: By embracing Tekton pipelines, organizations align with cloud-native best practices, promoting the use of Kubernetes and containerization for streamlined application delivery.

    Why Transition From Jenkins, GitLab, and Azure Pipelines to Tekton Pipelines?

    Migrating from Jenkins, GitLab, or Azure pipelines to Tekton pipelines can be a strategic choice, especially for organizations deeply entrenched in Kubernetes and cloud-native methodologies.

    Optimize Kubernetes-Centric Workflows: Projects tightly integrated with Kubernetes may find Tekton pipelines more aligned with their ecosystem.

    Accelerate Cloud-Native DevOps: Organizations aiming to embrace a cloud-native approach and leverage Kubernetes infrastructure.

    Performance and Scalability: Tekton pipelines run on clusters that can dynamically be created and destroyed according to demand, thus taking resource optimizations to the next level.

    Cloud-Native Best Practices: Best practices, such as containerization, declarative configuration and infrastructure-as-code are a natural fit due to their Kubernetes-native design.

    Proposed Workflow for Automating Pipeline Conversions at Scale

    The overarching objective of this workflow is to facilitate a seamless conversion process using large language models (LLM) without delving into the intricacies of fine-tuning but rather using the benefits of prompt engineering. Central to this approach is the use of the chat context and the strategic application of roles for LLM models to establish context. Along with this, sample pipelines are leveraged to train and fine-tune the model to define how the corresponding YAMLs of all these platforms look for any given pipeline.

    The use of prompts and instructions tailored to fit the use case of generating contextual pipeline YAML and conversion of Jenkins/Gitlab pipelines to Tekton pipelines makes this workflow possible. We maintain a context and a contextual mapping of real-world examples and the most appropriate prompt designs to produce the most accurate responses. The role of users in this context adds weight and further refines the response received.

    However, before proceeding into details and examples, let’s set the base for what exactly generative AI means and its context in our use case:

    Generative AI and LLM for Automating YAML Script Conversions

    Generative AI, such as LLMs, create content like text, benefiting from patterns in large datasets. Large language models (LLMs), with billions of parameters, excel in generating human-like text by understanding language structures. They have broad applications, including natural language understanding, summarization, translation and content generation, promising automation and human-like interactions with AI.

    Adapting the Model to Solve Real-world Use Cases

    Fine-tuning and prompt engineering are two common methods used to train large language models (LLMs) like GPT-3, BERT and others. These methods help adapt pre-trained models for specific tasks or generate contextually relevant responses. Both fine-tuning and prompt engineering play crucial roles. Fine-tuning helps the model comprehend complex use cases, while prompt engineering provides detailed requirements, extracting the best results from the model.

    Why Prompt Engineering?

    Prompt engineering stands out as a highly advantageous approach in specific use cases where precision, control and customization over AI model behavior are paramount. This method empowers developers and users to meticulously design input prompts, effectively guiding AI models to produce contextually relevant and desired responses.

    Prompt engineering is both an AI engineering technique for refining large language models (LLMs) with specific prompts and recommended outputs and the term for the process of refining input to various generative AI services to generate text or images.

    The process of prompt engineering to fit our use case includes :

    Defining the goal: The first step in the process of AI prompt engineering involves setting a clear objective.

    Crafting the initial prompt: With the goal in mind, it’s time to draft an initial prompt. This could take the form of a question, a command or even a scenario, depending on the goal.

    Testing the prompt: The initial prompt is then input into the language model, and the response is analyzed.

    Analyzing the response.

    Refining the prompt: With the insights gathered from testing and analysis, it’s time to revise the prompt. This could involve making it more specific, adding more context or changing the phrasing.

    Iterating the process: The testing, analyzing and refining steps are repeated until we are satisfied that the prompt consistently guides the model toward generating the desired response

    To conclude, here’s an example of typical YAMLs from Jenkins, GitLab and Azure DevOps, and how the same YAML looks when converted to a Tekton YAML. This goes a long way in facilitating teams to finally make a jump from their existing pipelines to cutting-edge reusable pipeline frameworks:

    Jenkins YAML

    GitLab YAML

    Azure YAML

    Converted Tekton YAML

  • 3 Steps to Secure Your CI/CD Pipelines

    3 Steps to Secure Your CI/CD Pipelines

    More and more software teams are waking up to the fact that their CI/CD pipelines are prone to risk. These pipelines often interface with various toolsets and automation, which may be prone to misconfigurations and abuse. This is not to mention the potential for insecure code being pushed through this funnel and external factors that threaten to circumvent the pipeline.

    In recent years, these risks produced some CI/CD tooling breaches—concerns that gained high-level attention recently after a joint CISA and the NSA alert about the importance of securing CI/CD pipelines. “This is something that moves the tectonic plates of the industry,” said Daniel Krivelevich, former founder of Cider Security and current CTO of AppSec at Palo Alto Networks.

    I reconnected with Krivelevich, the ‘godfather of CI/CD security,’ to reevaluate the CI/CD threat landscape and understand how IT should respond. According to Krivelevich, we’re now seeing more maturity around CI/CD security, both in terms of recognizing that these risks exist as well as emerging frameworks to help engineering teams respond. He shared a general three-step framework organizations can use to secure the CI/CD pipeline and surrounding areas, which we’ll explore below.

    The Evolving Threat Landscape

    The traditional approach to application security (AppSec) used to be simpler. Defenders could implement more standard controls to make their code more secure, said Krivelevich. However, throughout the years, the software engineering process has transformed dramatically. It’s become faster, dynamic and reliant upon third-party frameworks. “The entropy of the engineering ecosystem is very high,” he said.

    This paradigm shift has introduced a slew of new risks, which requires rethinking since most traditional security solutions deal solely with AppSec. “Software engineering is a big part of what makes the business tick,” said Daniel. Therefore, having a common language and understanding of this area is crucial to respond to evolving risks that threaten business operations.

    A Three-Step Framework For CI/CD Security

    Krivelevich is involved with creating the OWASP Top 10 CI/CD Security Risks; therefore, he has excellent insight into the types of vulnerabilities that most engineering practices and their CI/CD pipelines are facing. His three-step framework for CI/CD security, which Palo Alto Networks has applied in practice, looks to holistically secure not only the pipeline itself but also internal and external factors.

    1. Security in the Pipeline (SIP)

    SIP can be thought of as the traditional AppSec problem space. This involves ensuring developers are not shipping anything inherently insecure into production. Therefore, whatever code flows through the CI/CD pipeline must be vetted to ensure it has no security flaws or misconfigurations. Of course, this is easier said than done, given that teams are juggling more technologies and frameworks than ever, each of which has different knobs and requires unique security scanning capabilities.

    2. Security of the Pipeline (SOP)

    This area has to do with the security of the software delivery chain itself. Attackers may attempt to breach these pipelines by running malicious code within the pipeline or stealing credentials to CI/CD tools to harm the system. Poisoned dependencies, access management holes or insufficient credential hygiene could all be doorways to disrupting a CI/CD process. Therefore, SOP focuses on the security posture of the actual systems and tools that compromise the software delivery chain.

    3. Security Around the Pipeline (SAP)

    However, even if SIP and SOP are secure, what if hackers can bypass the pipeline altogether? That would render even a perfect set of controls null and void. Therefore, SAP is necessary to ensure that attackers can’t push code directly into production. It’s good to see the bigger picture so that everything that reaches production actually originates from the pipeline.

    To deploy SAP effectively, Krivelevich recommended leveraging both detective and preventative measures. You need detective measures to issue alerts when anomalies occur, such as an artifact deployed by a human account instead of a service account. You also need preventative measures that set the proper branch protection rules across all repositories. For instance, this could be a rule that no single developer can publish to production unless approved by two reviewers.

    The Role of AI in Software Security and CI/CD

    By using both detective and preventative measures, you can detect when settings are disabled or abused and have better configurations across all stages of the software delivery chain. But what other tactics can be used to secure CI/CD? For example, does machine learning have any merit in this area?

    Well, Krivelevich described AI as “both a blessing and a curse.” Attackers are already using AI to operate much faster and in a much more agile way. And unfortunately, he viewed the engineering ecosystem as the new path of least resistance. “I expect the power of AI will be used by sophisticated attackers who are motivated to get access to the ‘crown jewels’ by abusing the engineering ecosystem,” he said.

    On the defensive side, Daniel doesn’t see a specific opportunity for deploying AI to protect CI/CD. That being said, there’s a lot of potential for security builders to utilize AI to advance their efforts and accelerate the pace at which they detect anomalies, which could help respond more quickly to emerging threats.

    Final Thoughts on CI/CD Security

    The recent chatter around CI/CD threats appears to be garnering more attention around securing this niche area of agile software development. To thwart CI/CD attacks, it will be necessary to assess the settings of systems, understand their technical natures and analyze them against an attacker’s perspective, Krivelevich explained. Therefore, understanding what realistic attacks can be carried out will help build measures to correct these gaps.

    In doing so, the SIP, SOP and SAP descriptions serve as a solid reminder of what needs to be addressed when attempting to secure the software delivery process and the supply chain at large. Code that goes through the pipeline needs to be free from vulnerabilities, and the channel itself must be vetted for gaps that could be abused. We also need to consider the possible existential threats if the entire process can be undermined.

  • Tips For Securing CI/CD Pipelines

    Tips For Securing CI/CD Pipelines

    Most development teams want to increase the pace of their software delivery. As such, continuous integration and delivery (CI/CD) has grown in importance, helping push code from build to production as seamlessly as possible. CI/CD pipelines often loop in many elements and may comprise a diverse stack of tools, automations and various languages. But because of this, the attack surface can be large. And as gaps in the CI/CD toolchain emerge, so do security concerns.

    In recent years, we’ve seen several high-profile cases where CI/CD environments were compromised. For example, take the Circle CI breach, in which a bad actor accessed Circle CI production systems. Others include the compromise of the SolarWinds build system and the Codecov breach. CI/CD security issues are alarming because so much control and automation are given to a pipeline. Unauthorized CI/CD access could be leveraged to publish malicious code or steal sensitive information.

    I recently met with Daniel Krivelevich, co-founder and CTO of Cider Security (acquired by Palo Alto Networks), to better understand the risks facing modern CI/CD. Krivelevich is a seasoned CI/CD security expert and co-author of the Top 10 CI/CD Security Risks framework, which OWASP adopted in late 2022. According to Krivelevich, there is a general lack of knowledge and visibility into these risks. “The CI/CD ecosystem as a whole is full of holes and opportunities for attackers,” he said.

    Below, we’ll review some of the risks that CI/CD pipelines face and offer general ways to mitigate some of these concerns. Hopefully, with greater security awareness around CI/CD threats, both tooling providers and users can begin to diffuse the potential of attacks against their systems.

    The Top Concerns Facing CI/CD Security

    The OWASP Top 10 CI/CD Security Risks framework covers the most impactful risks within modern engineering systems and gaps in CI/CD. It was informed by analyzing many attacks in recent years and was compiled with insights from numerous experts. Krivelevich hopes the framework will spark discussions around the mindset defenders should have when they’re looking to protect the modern engineering ecosystem. The top 10 risks are as follows:

    • CICD-SEC-1: Insufficient Flow Control Mechanisms
    • CICD-SEC-2: Inadequate Identity and Access Management
    • CICD-SEC-3: Dependency Chain Abuse
    • CICD-SEC-4: Poisoned Pipeline Execution (PPE)
    • CICD-SEC-5: Insufficient Pipeline-Based Access Controls (PBAC)
    • CICD-SEC-6: Insufficient Credential Hygiene
    • CICD-SEC-7: Insecure System Configuration
    • CICD-SEC-8: Ungoverned Usage of Third-Party Services
    • CICD-SEC-9: Improper Artifact Integrity Validation
    • CICD-SEC-10: Insufficient Logging and Visibility

    Before an organization can dissect these individual risks, however, they must overcome an even bigger problem—a critical lack of understanding regarding the CI/CD ecosystem and the many elements under the hood. Since CI/CD is not traditionally viewed as a security risk, this can result in insecure default configurations, insecure processes and systems not adopting a secure-by-design approach.

    “There’s a big gap between the level of understanding and opportunities attackers have and the knowledge defenders have over what attackers can do,” said Krivelevich. “Organizations traditionally have not been proactive enough in understanding this, unlike other elements that are more traditionally recognized as part of the attack surface. The main gap is a mindset shift.”

    Tips to Secure CI/CD

    So, knowing the above risks, what are some best practices to secure CI/CD pipelines? Krivelevich shared some tips to keep in mind.

    Bring CI/CD under the security umbrella. First, realize that CI/CD should be scrutinized. This will require an awakening in the community and more proactive effort than before. Krivelevich encouraged development teams to place it on their roadmaps and think about it.

    Map the attack surface. Before implementing security measures, you need to know what you’re protecting. Therefore, it’s a good idea to map your CI/CD environment and produce an inventory of assets. Cataloging the total surface area of dependencies and integrations is necessary to understand risks and helps mitigate CICD-SEC-10: Insufficient Logging and Visibility.

    Keep this updated regularly. Since the cadence of change is fast, teams will want constant visibility into their ecosystem and what’s under the hood. This may even lead to shadow IT, as per CICD-SEC-8: Ungoverned Usage of Third-Party Services. Therefore, Krivelevich recommended keeping a regularly updated inventory. This means continually mapping control systems, container registry artifacts, plugins and all other assets used as part of CI/CD.

    Protect the software supply chain. Many building blocks make up the modern software delivery process. But, some dependencies may be abused or contain malicious code, as per risk CICD-SEC-3: Dependency Chain Abuse. Dependencies within the CI/CD process may be prone to a host of open source vulnerabilities.

    Safeguard CI/CD credentials. CI/CD tools “possess secrets to most sensitive crown jewels in an organization,” Krivelevich explained. Therefore, access tokens and secrets related to CI/CD tools should be protected to avoid costly credential leaks. A zero-trust approach to CI/CD could help avoid CICD-SEC-2: Inadequate Identity and Access Management.

    By creating a virtual heat map of areas to address, organizations will be more informed and better equipped to assess risk. And according to Krivelevich, by implementing these best practices, defenders have a lot of power to raise security expectations in the CI/CD space. “When defenders and users become more diligent about security, the requirements from vendors will increase.”

    CI/CD is the Beating Heart, Don’t Let it Break

    “The engineering ecosystem and pipelines themselves are the beating heart of engineering,” said Krivelevich. And as this system becomes a larger part of what makes a business tick, it will be more heavily relied upon for mission-critical tasks and retaining agility.

    Yet, many building blocks comprise the contemporary software delivery process, and CI/CD pipelines can be a risky area due to the number of systems they touch. Further complicating things is the fact that the delivery ecosystem wasn’t traditionally viewed in security terms. This combination of many opportunities and limited security oversight makes CI/CD lucrative for attackers.

    Rectifying the top 10 risks is a relatively easy gap to address, said Krivelevich—what’s not so easy is the mindset shift required to take CI/CD security seriously. Organizations that invested deeply into microservices and rapid engineering processes will most likely be more susceptible to attacks, he added, since they typically use a diverse array of tools and a complex, integrated delivery process.

  • What You’re Getting Wrong About DevOps (and How To Make it Right)

    What You’re Getting Wrong About DevOps (and How To Make it Right)

    DevOps is now the top software deployment strategy by a mile — 77% of organizations say they use the approach to roll out new software. This is good news because a DevOps approach is an important marker of business maturity, and its benefits (more on that later) are so tangible that most organizations don’t need to be convinced that DevOps is the future. 

    The bad news is most organizations are getting DevOps wrong, which is probably why only 10% report that their organization is “very successful” at achieving rapid software development and deployment. My company started its DevOps journey in 2019 as we began to build out the next generation of our ERP software. We were committed to the strategy but made mistakes along the way. 

    Our experience taught me the typical approach to DevOps (the one we took at first and the one many organizations follow now) is an improvement, but that a mindset shift can truly unlock the DevOps strategy’s potential and enable companies to maximize benefits. Here’s a closer look at where companies can go wrong—and what they stand to gain when they get it right. 

    What Happens When Organizations Get DevOps Wrong

    At its core, the DevOps concept is simple: Operational considerations should be considered at the development stage. As SaaS solutions gained popularity, the idea that issues like downtime should be everyone’s problem gained currency. Many organizations interpreted this to mean there should be a DevOps team that acts as a bridge between the development and operations teams.

    It actually makes more sense to merge the teams entirely. If a DevOps team sits in the center, the silos on the development and operations sides still exist. There can be even more communication breakdowns if there are three teams instead of two, and user needs can still be overlooked. A merged operation addresses this by creating a “you build it, you run it” mindset, which eliminates the disconnect. 

    We’ve found that a merged DevOps group handles operational issues as part of the engineering process from the earliest stages. This approach simplifies the process of identifying and addressing operational issues so that the software is more resilient and does a better job of meeting user expectations. 

    What Happens When Companies Get DevOps Right

    Simply put, making this mindset shift to get DevOps right lets your organization build better products, expedite updates, create greater alignment between teams and execute tasks more efficiently. This improves the experience for everyone, including the organization, its customers and end users. 

    One of the key benefits is better uptime. If you incorporate operational elements from the beginning, you reduce the risk of failure. When the groups are separated, operations teams often find it difficult to pinpoint issues, and the larger the system, the longer it can take to locate the problem, which can result in unhappy customers and unacceptably long outages. 

    By merging engineering and operations, you eliminate the tendency of some engineers to regard operations as someone else’s problem. A DevOps strategy that merges the teams makes operations everyone’s problem. And when everyone is subject to late-night calls for help if a problem occurs, it brings home the need to be proactive about preventing issues in the first place. The result is greater stability and reliability—and happier customers. 

    Maximizing DevOps Benefits

    Our experience taught us that merging engineering and operations was the best way to maximize DevOps benefits. It was a mindset shift that didn’t occur immediately; once it did, we realized how powerful it could be. One key component of the shift is to enhance team effectiveness by hiring engineers who are operations experts and honing existing staff skills as necessary. 

    Another important element is to get software engineers on board with the new approach and make it clear that the change is a priority. It’s easier to coast along as if the present approach is working perfectly, but that prevents your DevOps strategy from reaching its full potential. If you prioritize change and acknowledge the challenges involved, you can overcome them and reap the full benefits. 

    The consensus is clear: DevOps is the logical next step toward business maturity. But how you implement your DevOps strategy determines the magnitude of the improvement. Merging your engineering and operations teams is a challenge, but the reward is better products, quicker updates, tighter team alignment and an improved experience across the board. Getting DevOps right is worth it. 

  • How to Secure CI/CD Pipelines With DevSecOps

    How to Secure CI/CD Pipelines With DevSecOps

    Many companies are adopting a DevOps approach in their workflows as IT moves toward a more automated and cloud-native world—but for some industries, this migration isn’t easy. Many of these companies—in finance, health care, government—are obligated to meet compliance requirements. For these organizations, DevSecOps adds necessary security focus to the DevOps methodology.

    Compliance means adhering to certain standards and controls to ensure that applications are secure. But compliance only goes so far. How many applications contain third-party dependencies inside? How expensive would it be for a company in the financial industry, for example, to fail to meet compliance standards?

    Initially, containerizing an application while complying with requirements of whichever framework or standards apply to your organization could seem close to impossible. Within the cloud-native landscape, however, technologies are also rapidly evolving to enable healthy migration and ensure that applications can meet compliance standards without being rewritten from scratch.

    First, let’s look at the standards organizations must adhere to in terms of compliance.

    CIS Benchmarks for Kubernetes and Docker

    The Center for Internet Security (CIS) benchmarks offer guidelines for the correct use of containers in applications. The CIS benchmarks specifically discuss Docker and Kubernetes.

    NIST SP 800-190

    The U.S. government’s National Institute of Standards and Technology (NIST) defines containers as a form of virtualized operating systems within its application container security guide.

    PCI

    The payment card industry (PCI) standards are a framework that helps protect data in the e-commerce and retail areas.

    HIPAA

    The Health Insurance Portability and Accountability Act (HIPAA) sets the compliance standard for patient data protection in health care.

    Securing Infrastructure via CI/CD Pipelines

    All these standards guide organizations to keep their infrastructure secure and compliant with their specific industry best practices, but security doesn’t end (or even start) there.

    Secure systems start from the design stage and from the code itself; you should aim to have security controls and enforcement built in as early as possible in the development life cycle. The practice of moving security checks earlier in the development pipeline (and, thus, nearer the source code) is known as shift left.

    You likely already have CI/CD pipelines in place for your application(s), and this is a great place to shift left your security checks. There are some great tools in the CNCF portfolio to help you achieve this.

    Enter Containers

    In the cloud-native world, containers are first-class citizens. From the DevSecOps point of view, containers represent a new attack surface to consider, starting with how the container image is defined and what ends up inside the container (the Dockerfile) to how the container evolves over time in response to new vulnerabilities that might be found.

    The first thing we can do is to convert our written policies to something that we can automate (such as code); for example, using a tool like OPA/conftest and a Dockerfile linter like hadolint, we can validate that the Dockerfile is compliant with our security policies.

    For instance, we could have a policy in place to limit the base images used to build new containers (limiting the FROM directive in the Dockerfile, for example).

    The second thing to consider is that, depending on how it has been built, a container image can have not only our application but also additional software included, like dependencies and operating system binaries.

    This additional software could have new vulnerabilities discovered in the future, so we should monitor the images that we have already built and pushed to our registry by periodically scanning them to find if they are open to any vulnerabilities that weren’t known at the time of building the container image.

    Just as before, we could sign our application artifacts. Now, our deliverable artifact is a container image, so it is preferable to be able to sign container images. In this way, we can trust that the container image we run comes from us and that it has been through our security process. There are projects like Notary, Grafeas and sigstore/cosign that can help with the container signature process.

    Running Containers in Kubernetes

    If we are running containers, the next logical step is to consider how we run them. Kubernetes use is on the rise as a solution to orchestrating containers across several machines. Introducing Kubernetes into the equation requires considering yet another attack surface in our DevSecOps workflow.

    Manifests

    Alongside our application’s source code, we will have the Kubernetes manifests that define how to deploy our application into a Kubernetes cluster.

    We can take the same approach as the container definition: Define some policies-as-code to enforce early in the process so our manifests are aligned with our security policies (not running untagged images, running images only from our trusted sources, ingress must have TLS, not using node port services, for example) and lint the YAML files to fail early if there are syntax problems, for example.

    We could also enforce most of the same policies and some additional ones at runtime when the manifests get applied to the cluster via a Kubernetes Admission Controller.

    Some tools that can help us with these enforcements are OPA/conftest, kubeval or kubelinter for linting and validation and, for the admission controls, we can use OPA/Gatekeeper or Kyverno, in addition to the API server admission plugins.

    Secrets

    Secrets require special attention since they contain sensitive information.

    If you’re using a GitOps approach, application secrets are often stored in repositories. Here are some resources that discuss sealed secrets in GitOps and Kubernetes secrets. 

    RBAC

    Do not use default service accounts: Define explicit service accounts associated with their permissions so you know exactly who can do what.

    Avoid the use of wildcards in roles and cluster roles; with RBAC you can define fine-grained permissions for users and for service accounts, as well.

    Know Your Dependencies

    Another attack surface that we need to consider is the application’s dependencies. It is important to first know what they are and if they have any known vulnerabilities. To help us with that, we can create a software bill of materials for the application and run an analysis on it.

    Finally, just like we periodically check for the images stored in the registry, we can continuously analyze what’s running in our Kubernetes cluster. This will ensure that there is no suspicious activity like abnormal network requests, images with critical CVEs running, etc.

    Conclusion

    By now, you should have a better idea of what it takes to secure your pipeline. Keep in mind, however, that this is mostly about process and cultural change.

    First of all, you need to know your attack surface—that is not only a technological choice, it’s also about what you think is important.

    These concepts are hard to apply from the beginning, so a better approach is to introduce changes, get used to them and think about how to improve the existing process step by step.

    Join us for KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain (and virtual) from May 16-20—the first in-person European event in three years!