devopsy.kinsta.cloud

Tag: CI/CD

  • Cycode Discloses GitHub Actions Vulnerability in Google Bazel Project

    Cycode Discloses GitHub Actions Vulnerability in Google Bazel Project

    Cycode today revealed it discovered a command injection vulnerability in the way the GitHub Actions continuous integration/continuous delivery (CI/CD) platform was used to update the open source Bazel project that many organizations employ to build and test software.

    The vulnerability, now remediated, could have been used to create a backdoor through which malicious code could have been inserted into a codebase that is managed by Google.

    Cycode researchers shared their discovery with Google, which has since updated the way it employs GitHub Actions to manage the Bazel codebase. Specifically, Google updated the base permissions to embrace least privilege principles to resolve the vulnerability discovered by Cycode.

    The vulnerability was discovered using RAVEN, an open source tool for scanning CI/CD platforms for vulnerabilities that Cycode makes available.

    Elad Pticha, senior security researcher for Cycode, said the vulnerability is the latest example of an instance of an open repository for building software that has been shown to be potentially vulnerable to an attack on a software supply chain that could theoretically impact any number of downstream applications and projects, including Kubernetes, Angular and platforms and services provided by Uber and LinkedIn.

    A Custom Actions capability that enables application development teams to create their own workflows presents any cybercriminal attempting to compromise of software supply chain with a tempting target, noted Pticha. A few changes to lines of code in a workflow can translate into thousands or even millions of lines of code, many of which an organization may never detect. Out of 3.4 million workflows examined by Cycode, nearly all (99%) make use of one or more Custom Action.

    Cycode has previously identified dozens of vulnerabilities in open source projects by mainly focusing on vulnerabilities within workflows. This latest research shows those vulnerabilities extend to indirect dependencies, such as Custom Actions, noted Pticha.

    Each DevOps team should review its own workflows for similar vulnerabilities at a time when more cybercriminals than ever are intent on compromising software supply chains. While many organizations have embraced DevSecOps workflows to improve application security, there is a natural tendency to overlook vulnerabilities that might exist in the workflows used to build and deploy software. The challenge is that DevSecOps workflows are still immature. A recent survey of 500 U.S. CISOs, AppSec directors and DevSecOps team members conducted by Cycode found 78% admitted application security attack surfaces are unmanageable, with 90% acknowledging relationships between their security and development teams need to improve.

    It’s not apparent if the vulnerability discovered by Cycode was actually ever exploited, but as most cybersecurity professionals know, if it can be imagined, someone has tried it. As such, any organization that uses Bazel might be well-advised to conduct a code review.

    In the meantime, it’s already apparent that regulations pertaining to securing software supply chains are only going to become more stringent in the months ahead. In the wake of the executive order from the Biden administration requiring federal agencies to lock down their software supply chains, it’s now only a matter of time before similar requirements are baked into legislation and various other regulations are implemented. The challenge and the opportunity now is to get ahead of this issue before a mandate with a hard deadline that can’t be ignored winds up disrupting the entire software development life cycle.

  • Using LLMs to Automate Pipeline Conversions From Legacy to Tekton

    Using LLMs to Automate Pipeline Conversions From Legacy to Tekton

    There have been many technological advancements in the market today, but the main hindrance is the cumbersome process of switching from the existing technology/framework to the new one. This is where teams and organizations are left behind in the tech curve. In the DevOps landscape, CI/CD pipelines are the backbone of DevOps, and how you work with them largely dictates your DevOps efficiency.

    CI/CD pipelines, too, like other major technologies and frameworks in the past, have seen various evolutions; the latest among them is being made reusable in an attempt to standardize CI/CD. However, here too, there are several cons:

    1. The pipelines are not proprietary to end-user enterprises and belong to the DevOps platforms which are not open-source
    2. Migrating to these platforms and pipelines is a huge task
    3. They may not be as efficient as depicted

    Speaking of new advancements in technology, most might be familiar with Tekton for CI/CD pipelines: It is Google’s open source framework that is now a graduated CDF project. It redefines how pipelines are built (through reusable tasks) and these tasks are open source and community-vetted. They allow for pipeline authorship, and enterprises can re-use them across platforms that are compatible with Tekton, thus preventing vendor lock-in.

    While this mitigates the first challenge of pipeline ownership and migrations, however, the same problem plagues DevOps teams – transitioning from existing legacy technologies/frameworks to newer, much more capable frameworks like Tekton.

    However, what if there was a way to seamlessly automate your transition from legacy pipelines to cutting-edge, reusable Tekton pipelines without toil? This is precisely what the article aims to explore – automating the conversion of Yaml scripts of existing pipelines to Tekton Yaml scripts.

    Let us consider the three most popular CI/CD platforms for this use case – Jenkins, GitLab and Azure DevOps. Before we proceed to the proposed approach to achieve this (which is detailed in the second half of this article), let us first look at the limitations of Jenkins, GitLab and Azure Pipelines and why one should convert.

    Common Limitations of Jenkins, Azure and GitLab Pipelines

    Resource Intensive: Legacy pipelines can be resource-intensive, especially for larger organizations or projects. Managing server resources, optimizing configurations and ensuring sufficient hardware can be a challenge.
    Complex Setup: Setting up and configuring Jenkins, GitLab and Azure DevOps can be complex, particularly for self-hosted instances. This complexity may require experienced administrators and substantial time investment.
    Plugin and Integration Compatibility: All three platforms may face compatibility issues with certain plugins or integrations.
    Scaling Challenges: Scaling in organizations, especially when dealing with multiple microservices, necessitates a shift from individualized scaling to a template-based approach. Failing to do so leads to a chaotic increase of pipelines, undermining standards and organizational processes.
    Costs: While both Jenkins and GitLab offer free and open-source versions, some advanced features or capabilities may only be available in higher-tier paid plans. This can lead to cost considerations at scale.
    Customization Complexity: These platforms can be complex to customize workflows and integrate custom scripts as they require advanced scripting knowledge.
    User Interface Complexity: Though they have feature-rich interfaces, it can sometimes be overwhelming for new users to navigate the platforms and find specific settings or options.
    Limited Native Integration & Plugins: The native integrations of Jenkins, GitLab and Azure may not cover the full spectrum of tools and services a team may need. Additional integration efforts or third-party tools may be required to bridge gaps in the development workflow.

    As can be seen in the above limitations, many are common including scalability challenges, being resource intensive, limited integrations etc. All the challenges boil down to one common denominator: Complex pipeline management at scale.

    This is where a framework like Tekton can help tackle all the challenges at once, by just easing pipeline management: with its K8s-native pipelines that are reusable, open source and tasks that are community-vetted.

    Tekton Pipelines: Advancing CI/CD with Kubernetes-Native Automation

    Tekton, an open-source project, is rapidly gaining traction as a game-changer in the world of CI/CD pipelines. Its Kubernetes-native design is a standout feature, seamlessly integrating with Kubernetes clusters to harness its scalability and resource efficiency.

    Furthermore, Tekton’s cloud-native roots ensure it can execute tasks in lightweight, ephemeral containers, optimizing resource utilization. The ability to declare CI/CD pipelines as code, coupled with GitOps practices, enhances automation and reproducibility.

    How are Tekton Pipelines Different?

    Kubernetes-Native Automation: Tekton pipelines are deeply integrated with Kubernetes, utilizing Custom Resources and Operators to define and execute CI/CD workflows.

    Declarative and Modular: These pipelines are defined declaratively using YAML manifests, promoting version control, collaboration and code reviews. The modular design of tasks and pipelines enables teams to create reusable components for building, testing and deploying applications.

    Containerization and Isolation: Each task in a Tekton pipeline runs within a dedicated container, ensuring isolation and reproducibility of the build and deployment environment.

    Event-Driven Architecture: Tekton pipelines can be triggered by various events, such as code commits, pull requests, or external events, providing a responsive and event-driven CI/CD process.

    Scalability and Efficiency: Tekton pipelines leverage Kubernetes’ inherent scalability, enabling efficient execution of tasks across a dynamic pool of worker nodes.

    Visibility and Observability: They offer built-in monitoring and logging, allowing teams to gain insights into pipeline execution, track progress and diagnose issues.

    Integration and Extensibility: Tekton pipelines can be easily extended with custom tasks and integrations, enabling teams to connect with various tools, services and cloud platforms.

    Cloud-Native Best Practices: By embracing Tekton pipelines, organizations align with cloud-native best practices, promoting the use of Kubernetes and containerization for streamlined application delivery.

    Why Transition From Jenkins, GitLab, and Azure Pipelines to Tekton Pipelines?

    Migrating from Jenkins, GitLab, or Azure pipelines to Tekton pipelines can be a strategic choice, especially for organizations deeply entrenched in Kubernetes and cloud-native methodologies.

    Optimize Kubernetes-Centric Workflows: Projects tightly integrated with Kubernetes may find Tekton pipelines more aligned with their ecosystem.

    Accelerate Cloud-Native DevOps: Organizations aiming to embrace a cloud-native approach and leverage Kubernetes infrastructure.

    Performance and Scalability: Tekton pipelines run on clusters that can dynamically be created and destroyed according to demand, thus taking resource optimizations to the next level.

    Cloud-Native Best Practices: Best practices, such as containerization, declarative configuration and infrastructure-as-code are a natural fit due to their Kubernetes-native design.

    Proposed Workflow for Automating Pipeline Conversions at Scale

    The overarching objective of this workflow is to facilitate a seamless conversion process using large language models (LLM) without delving into the intricacies of fine-tuning but rather using the benefits of prompt engineering. Central to this approach is the use of the chat context and the strategic application of roles for LLM models to establish context. Along with this, sample pipelines are leveraged to train and fine-tune the model to define how the corresponding YAMLs of all these platforms look for any given pipeline.

    The use of prompts and instructions tailored to fit the use case of generating contextual pipeline YAML and conversion of Jenkins/Gitlab pipelines to Tekton pipelines makes this workflow possible. We maintain a context and a contextual mapping of real-world examples and the most appropriate prompt designs to produce the most accurate responses. The role of users in this context adds weight and further refines the response received.

    However, before proceeding into details and examples, let’s set the base for what exactly generative AI means and its context in our use case:

    Generative AI and LLM for Automating YAML Script Conversions

    Generative AI, such as LLMs, create content like text, benefiting from patterns in large datasets. Large language models (LLMs), with billions of parameters, excel in generating human-like text by understanding language structures. They have broad applications, including natural language understanding, summarization, translation and content generation, promising automation and human-like interactions with AI.

    Adapting the Model to Solve Real-world Use Cases

    Fine-tuning and prompt engineering are two common methods used to train large language models (LLMs) like GPT-3, BERT and others. These methods help adapt pre-trained models for specific tasks or generate contextually relevant responses. Both fine-tuning and prompt engineering play crucial roles. Fine-tuning helps the model comprehend complex use cases, while prompt engineering provides detailed requirements, extracting the best results from the model.

    Why Prompt Engineering?

    Prompt engineering stands out as a highly advantageous approach in specific use cases where precision, control and customization over AI model behavior are paramount. This method empowers developers and users to meticulously design input prompts, effectively guiding AI models to produce contextually relevant and desired responses.

    Prompt engineering is both an AI engineering technique for refining large language models (LLMs) with specific prompts and recommended outputs and the term for the process of refining input to various generative AI services to generate text or images.

    The process of prompt engineering to fit our use case includes :

    Defining the goal: The first step in the process of AI prompt engineering involves setting a clear objective.

    Crafting the initial prompt: With the goal in mind, it’s time to draft an initial prompt. This could take the form of a question, a command or even a scenario, depending on the goal.

    Testing the prompt: The initial prompt is then input into the language model, and the response is analyzed.

    Analyzing the response.

    Refining the prompt: With the insights gathered from testing and analysis, it’s time to revise the prompt. This could involve making it more specific, adding more context or changing the phrasing.

    Iterating the process: The testing, analyzing and refining steps are repeated until we are satisfied that the prompt consistently guides the model toward generating the desired response

    To conclude, here’s an example of typical YAMLs from Jenkins, GitLab and Azure DevOps, and how the same YAML looks when converted to a Tekton YAML. This goes a long way in facilitating teams to finally make a jump from their existing pipelines to cutting-edge reusable pipeline frameworks:

    Jenkins YAML

    GitLab YAML

    Azure YAML

    Converted Tekton YAML

  • The Future of Jenkins in 2024

    The Future of Jenkins in 2024

    As software development practices evolve, the tools that support them must also adapt to stay relevant. Jenkins, the venerable automation server used by many DevOps teams, is at such a crossroads as it faces the changing landscape of cloud-native applications and Kubernetes adoption. 

    This article examines Jenkins’ current utility, particularly its features that have made it a staple in CI/CD practices, and addresses the question of whether it will maintain its relevance in 2024 and beyond.

    How Jenkins is Used by DevOps Teams

    CI and CD

    Jenkins has become an integral part of many DevOps teams due to its CI/CD capabilities. Continuous integration (CI) is a software development practice where developers regularly merge their code changes into a central repository. Continuous delivery (CD) is the practice of delivering the software in short cycles to ensure that it can be reliably released at any time.

    Jenkins supports the CI/CD approach in a number of ways. First, it allows developers to automate code integration. This means that whenever code changes are pushed to the repository, Jenkins can automatically build and test the project. Second, Jenkins allows for the continuous delivery of software by automating simple deployment processes (although it lacks full deployment orchestration capabilities).

    Build Pipeline Visualization

    Another key feature of Jenkins is its build pipeline visualization. This feature allows DevOps teams to visually represent the process of software delivery from code integration to deployment. The pipeline visualization provides a clear and intuitive representation of the software delivery process, making it easier for teams to understand and manage the process.

    Build pipeline visualization in Jenkins is customizable, allowing teams to tailor it to their specific needs. For example, it can be customized to show the status of each stage of the delivery process, from code integration to testing to deployment. This allows teams to quickly identify any issues or bottlenecks in the process and address them.

    Automated Testing

    Automated testing is another area where Jenkins shines. With Jenkins, DevOps teams can automate the process of testing their code by running scripts and integrating with automated testing tools as part of the build process.

    Jenkins supports a wide range of testing frameworks and tools, allowing teams to choose the ones that best fit their needs. Moreover, Jenkins can be configured to automatically run tests whenever changes are made to the code. This ensures that any issues or defects in the code are detected early and can be addressed promptly.

    Workflow Scripting

    Workflow scripting is another key feature of Jenkins in DevOps teams. With Jenkins, teams can script their workflows, automating the process of software delivery. This allows teams to standardize their workflows, ensuring that each delivery process is consistent and predictable.

    Jenkins supports the Groovy scripting language for workflow scripting. This means that teams can use the power and flexibility of Groovy to script their workflows. Moreover, Jenkins provides a user-friendly interface for scripting workflows, making the process easy even for those who are not familiar with Groovy.

    Current Usage and Adoption Statistics

    As of 2023, Jenkins remains a leader in the CI/CD space. With its extensive plugin architecture and large user base, it is used by thousands of organizations worldwide. According to the 2022 DevOps Survey by the Cloud Native Computing Foundation (CNCF), Jenkins was used by over 50% of respondents for automation and CI/CD, maintaining a significant presence in the market.

    According to the Jenkins website, there are over 1 million active Jenkins users and more than 200,000 active installations worldwide, with the platform handling 1 million+ jobs per day. It had about 44% of the global market share for CI/CD in 2023.

    The growth trajectory of Jenkins, however, has shown signs of plateauing when compared to newer, more cloud-native-focused CI/CD tools. This observation is backed by Google Trends data, reflecting a steady but not sharply increasing interest in Jenkins over recent years.

    Despite facing competition from emerging tools designed for cloud-native development, such as GitHub Actions, GitLab CI and CircleCI, Jenkins has responded by evolving its ecosystem. The creation of Jenkins X is a strategic move to cater to Kubernetes users and address some of the criticisms related to cloud-native development. However, as we’ll discuss, Jenkins X has not been readily adopted by the industry as of yet.

    Limitations of Jenkins in Cloud-Native Environments

    While Jenkins is a powerful tool for DevOps teams, it has serious limitations in cloud-native environments. These limitations stem from the fact that Jenkins was originally designed for traditional, monolithic applications, not for the dynamic and distributed nature of cloud-native applications.

    Statefulness

    One of the critical limitations of Jenkins in cloud-native environments is its inherent statefulness. Cloud-native architectures typically favor stateless applications, which are easier to scale and manage in dynamic environments like Kubernetes. 

    Jenkins, however, relies on maintaining state—it stores configuration, build history and user data, making it challenging to scale and manage efficiently in a cloud-native setup. This statefulness requires persistent storage, which complicates deployments in distributed systems where resiliency and horizontal scaling are key. Moreover, the risk of data loss or inconsistency increases in cloud-native environments, particularly during scaling operations or when recovering from failures.

    Configuration Complexity

    Jenkins is often criticized for its complex configuration requirements. Setting up Jenkins in a cloud environment involves intricate configurations that are not only time-consuming but also prone to errors. This complexity is exacerbated when integrating with modern cloud-native tools and services, requiring extensive customization and maintenance of plugins. 

    In addition, Jenkins’ traditional master-slave architecture was not designed for dynamic cloud environments, where resources need to be provisioned and deprovisioned rapidly. This leads to a significant operational overhead for DevOps teams.

    Immutable Infrastructure

    Adopting an immutable infrastructure approach is a cornerstone of cloud-native environments, promoting the idea of replacing or rebuilding servers rather than modifying them. 

    However, Jenkins’ traditional model does not align well with this principle. It typically requires modifications to its runtime environment for updates or changes, contrasting with the cloud-native approach of immutable deployments. This can make it challenging to maintain consistency and reliability in cloud-native systems where automation and minimal manual intervention are paramount.

    Microservices Limitations

    Jenkins was originally designed for monolithic architectures and doesn’t naturally fit into microservice-based applications, which are prevalent in cloud-native environments. In microservices architectures, applications are broken down into smaller, independently deployable services. 

    Jenkins, however, tends to treat applications as monolithic units, making it less efficient for handling multiple, independent microservices. This limitation affects its ability to provide fine-grained control over individual services’ build, test and deployment processes. Consequently, Jenkins may struggle with the high degree of automation, flexibility and scalability required in microservices environments.

    What is Jenkins X?

    Jenkins X is an open source project that provides pipeline automation, built-in GitOps, and preview environments to help teams collaborate and accelerate their software delivery at any scale. It is an improved version of the original project, designed specifically for Kubernetes, a popular container orchestration platform.

    Jenkins X streamlines the technical aspects of building, testing, and deploying software. It automates the installation of popular tools like Helm and Draft, making it easier for developers to get started. It also provides pre-configured pipelines so teams can start delivering software right out of the box.

    Jenkins X also introduces a new Jenkinsfile format, Jenkinsfile.yaml, which provides a more straightforward and human-readable way to describe pipelines. This new format, combined with the automation provided by Jenkins X, helps reduce the complexity of pipeline configuration and management in Kubernetes.

    Limitations and Criticisms

    Jenkins X has not been widely adopted by the software industry and has not achieved the respect and prominence of its predecessor, Jenkins. Let’s review the main limitations and criticisms of the tool.

    Lack of Maturity

    Jenkins X’s relatively recent entry into the market means it has not had the time to mature like its predecessor. This lack of maturity can manifest in several ways, including fewer features compared to established tools, fewer community-tested plugins, and potentially more bugs or unresolved issues. Due to the fast-paced evolution of cloud-native technologies, tools need time to adapt and stabilize; Jenkins X is still in this phase of its lifecycle.

    Moreover, a mature tool typically has extensive documentation and a large community of users who can offer support and share best practices. While Jenkins X has seen growth in its user base, it still lags behind more mature CI/CD solutions in these areas. This can lead to challenges when teams are looking to solve complex problems or when they require advanced support.

    Limited to Git Projects

    Another criticism of Jenkins X is that it’s limited to Git projects. While Git is the most popular version control system, it’s not the only one. Some teams may use other systems like Mercurial or SVN, and these teams would be unable to use Jenkins X.

    Even within the world of Git, Jenkins X has limitations. For example, it doesn’t support all Git hosting platforms. While it works with popular platforms like GitHub and Bitbucket, it doesn’t support others like GitLab. This can be a significant limitation for teams that use unsupported platforms.

    Community Support

    Finally, there are concerns about community support for Jenkins X. While the project is open source and theoretically has the potential for wide community involvement, some critics argue that it’s primary source of support is CloudBees, the company behind Jenkins.

    These critics worry that if CloudBees were to withdraw its support, the project could falter. They also argue that the project’s focus on certain tools and platforms (like Kubernetes and Git) reflects CloudBees’ business interests more than the needs of the broader DevOps community.

    Conclusion

    In conclusion, Jenkins’ role in the CI/CD landscape has been pivotal, and its continued evolution with projects like Jenkins X demonstrates an effort to adapt to the needs of modern, cloud-native environments. While Jenkins X has yet to reach the widespread adoption and maturity of its predecessor, its specialized focus on Kubernetes and GitOps reflects the shifting priorities in software development.

    Jenkins, despite showing signs of plateauing in growth, still possesses a strong user base and an established presence in the market, supported by a vast plugin ecosystem and a robust community. Its limitations in cloud-native contexts—such as managing statefulness, configuration complexity, and supporting immutable infrastructure—pose challenges that Jenkins X seeks to address.

    As we approach 2024, the relevance of Jenkins will likely hinge on its ability to integrate more seamlessly with cloud-native technologies and the willingness of the community to evolve its ecosystem. The breadth of its adoption and the depth of its integration in existing development pipelines mean Jenkins will continue to be a significant player in the DevOps toolchain. However, its long-term relevance will depend on the project’s ability to innovate and remain adaptable in a landscape that is rapidly embracing cloud-native solutions.

  • DevOps and Databases: The Key to Faster Deployment

    DevOps and Databases: The Key to Faster Deployment

    With DevOps, we learned that the biggest factor in streamlining deployments is to make things automated and minimize communication. However, we didn’t apply that principle to other areas. While security gets some traction with DevSecOps and machine learning has MLOps, we lack a solution for database management. In this article, we propose a new approach that unlocks the power of DevOps engineers to own their databases and lets them self-serve deployments, maintenance and troubleshooting.

    Why Database Deployment and Maintenance is Slow

    In the last few years, we moved from deployment being a big and challenging event to deploying many times daily. We moved from manual deployments to automated CI/CD pipelines. We built tools for deploying infrastructure and configuration changes. This led to a tremendous increase in the complexity of our systems. We somewhat learned how to handle that in deployments, infrastructure, configuration, security or even machine learning.

    However, database complexity increased significantly as well. We now use many database engines with each application. We use SQL and NoSQL databases as well as databases tuned for specific scenarios like machine learning or vector search. Since we deploy changes often and asynchronously, modifications to database schema or background tasks may happen at any time and affect the performance. When this happens, the business is directly impacted as the database issues quickly spread throughout the ecosystem. However, developers and DevOps engineers can’t fix the issues on their own and need help from other teams, specifically operations and database administrators. This makes the process slow and error-prone as things are not automated but require manual human intervention. Recommended reading: Why DevOps should own their databases and how to do it.

    The problem with database maintenance is the lack of solutions that would present the full and actionable story. We have many tools that can alert us about spiking CPU or increased latency, but we don’t know what’s the reason behind these issues. To find out, developers need to ask database administrators to check the logs and metrics to identify slow queries and whether the CPU spikes are caused by changes in the environment (like updating the operating system), slow background tasks that can be postponed (like vacuuming or index rebuilding) or due to inefficient application code that doesn’t use an index and performs a full table scan.

    Since we don’t know the reason, we need to involve multiple teams to investigate. We need to first identify the points of contact in each team, assign tickets or work items, sometimes page people at night, and coordinate the troubleshooting process. To make things faster, we need to fix two aspects: reduce the amount of manual work and reduce the number of people involved in the investigation. Let’s see how.

    Reducing the Manual Work with Observability

    Many things can go wrong when dealing with databases. Let’s see some of the areas that can break and are not automatically fixed today.

    Observability for the Code Changes

    Every time developers change their codebase, there is a risk of breaking the correctness and performance. We can verify the correctness with automated tests (unit tests, integration tests, end-to-end tests, etc.) and do it regularly in our CI/CD pipelines. However, we can’t verify the performance impact. Our tests focus on what we extract from the database and not how we do it. We may use N+1 queries, we may scan tables instead of using indexes, and we may have slow queries with CTEs or not use searchable arguments (a query that is not sargable). All these things may result in CPU spikes or latency degradation and we can’t automatically explain what happens. We need to have observability that gives the story like “the query execution within a given REST endpoint doesn’t use an index and leads to slow performance.”

    Performance of Schema Changes

    When we modify a schema, we risk that tables will be locked for rewriting. This may take minutes or hours and effectively make our systems unavailable. However, our CI/CD pipelines do not test the performance of schema migrations. Tests run against the latest schema and migrations are fast because databases in pre-production environments are small. We don’t check how long these migrations will take in production and if they’ll lock tables. We need to have tools that can warn us if a slow migration is being deployed and will indicate how to make it faster.

    Understanding of Execution Plans

    Databases grow over time. Queries change their execution plans because statistics get outdated or because indexes are modified. However, automated tests won’t capture these issues as they focus on the correctness of the data and they don’t run on the production database. We don’t see how things evolve and that the query degrades gradually over time. We need to have observability systems that provide an understanding of all the moving pieces around. They need to give us a story explaining that “the query we deployed six months ago stopped using an index because the statistics are out of date as we injected a big batch of data last night”.

    We need a story explaining what happened, like SQL stopped using an index or SQL query is not sargable. Other reasons can be schema changes, mismatched query fields, index fragmentation or significant changes in data volume affecting the query optimizer’s decision-making.

    How to Reduce the Work

    Taking all we saw above, we need to have systems that can understand how our environment evolves and how these changes affect performance. They need to alert us when some slow changes are being deployed to production, understand how the deployments affect the performance and automatically explain the issues. Troubleshooting should be as simple as just getting the notification with links to the places that we need to fix (like configuration, a particular line of code in our application, or the cloud console) with an explanation of how to do it. Fixing should be merely approving a pull request or confirming the configuration change.

    Reducing the Number of People Involved With Self-Servicing

    Having database observability is the first step. The second step is the ownership. Developers and DevOps engineers need to own their databases the same way they own their code and deployments. They need to be able to self-serve the issues and do that without any help from operations or database administrators. This will reduce communication and make things much smoother. There are three things that we need to introduce to let developers and DevOps engineers build their ownership.

    First, we need the tools. Just like we explained in the previous section, we need tools that can explain the story. We can’t just put more responsibilities on developers and DevOps engineers without making the work simple and automated.

    Second, we need new processes. We need to have playbooks and operational procedures explaining how to use the tools, how to self-serve the maintenance, and when to escalate. Developers must be familiarized with the procedures and must know how to follow them even outside of working hours. This way we can replace some work (communication, coordination of teams, manual investigation) with some other work (self-servicing) that we can do much faster.

    Finally, we need to change the mindset. Developers and DevOps engineers need to take on new responsibilities and become true owners of their databases. Even though this sounds like more work, we can make it much less time-consuming by building good observability tools and replacing slow human communication with automated and well-defined self-servicing. It’s not about shifting the work from one place to another. It’s about making the work much smaller and much more automated so the developers and DevOps teams can do it without involving other teams. Since we reduce the cross-team collaboration, the increase in the scope of work of developers and DevOps engineers will lead to doing the work faster.

    Summary

    There are three aspects that we need to work on now. First, tools like Metis must be created. Second, Platform Engineers must use these tools to define processes. Third, developers and DevOps engineers need to take ownership. While it may seem challenging at first, we already saw with the DevOps movement that minimizing communication and owning the full scope of the system is beneficial for the business and for the engineers. We need to restructure our teams and reassign responsibilities to unlock the potential and make organizations work faster.

  • KubeCon 2023: What’s Next for Managed Deployment

    KubeCon 2023: What’s Next for Managed Deployment

    Mitch Ashley: Hi everybody, welcome back to KubeCon here in Chicago, 2023. Lots of great conversations, and here comes another one with Adam Frank Armory, welcome, Adam.

    Adam Frank: Thanks for having me, pleasure to be here.

    Mitch Ashley: Always, we love having you on. So what’s happening in continuous delivery in this, I mean, it’s evolved so much just in the last even year or so, about how many environmists we’re deploying to and the mix of all that. Most of us don’t live in a greenfield, so we’ve got to deal with lots of variety of things.

    Adam Frank: That’s right, that’s right. And I think one of the biggest things that people are really starting to come across now is the fact that they are looking at multi deployments, multi cluster, multi environments, really trying to orchestrate all of that and coming around to manage delivery. Understanding that everybody from a seed stage to large, large enterprises, they all have at least three dev stage prod, right? So how are you going to orchestrate across that when that just grows to more and more environments, talking about global companies and things of that nature.

    So, I think that’s one of the big conversations that’s been happening around here is about the orchestration of that deployment. So it’s been exciting. It’s been exciting for us. That’s one of the things that we went into from the very beginning was understanding that the deployments have to be orchestrated. There’s a lot of things that have to happen, security scanning, integration, tests, you name it, throughout that deployment. So it’s been quite nice.

    Mitch Ashley: When you mentioned managed deployment, right, it’s not just press the button, everything happens. There’s a lot of variables, things have to be sequenced. Some environments require different things than other environments, some you can do is declarative, you can really build it from the ground up through automation and others. Maybe you can’t do that, you’ve got to work with infrastructure or apps or code that’s already there.

    Adam Frank: Yeah, whether it’s infrastructure, apps, or code, or whether it’s just policies that you have in place that you need manual approval before you go out to production. Or something has to happen in staging, like data. Data needs to be transferred from production into staging to do actual validated tests.

    We had a talk on Monday with Portworx actually, really fantastic use case that we’ve come up with them. There’s a use case with their largest customer where they are actually transferring live production data into staging environments to do real tests, because I know a lot of people today, the code’s the same, the configuration’s the same, the infrastructure’s all the same. Is the actual data the same? Well, not really. Okay, well, how do you know that what you’re actually about to change is going to inflict what you want to change? So the fact that we’ve worked a lot with Portworx to do the actual data transfer, to have real live production data in the staging environment to run those tests and things against, has been pretty cool for our customers.

    Mitch Ashley: That was always one of my biggest challenges, how do you get data, especially earlier in the development of an app or a product, is you don’t have a lot of access to a lot of production data unless you’re in a SaaS, and maybe you can then, but you have to worry about things of anonymizing. But you have data, but not necessarily the right data to test what you’re going to deploy so you’ve got to kind of match that all up. Is there anything in that demo that talked about that, sort of that part of the problem, how do you get the right data in the right environment?

    Adam Frank: Yeah, exactly. One of the big things that the security of data, the management of data, moving data around, and and Portworx has a great solution for it. So it’s nice to partner with them, given that we can do the orchestration of those deployments and have that as a check that’s within that deployment. It’s not going out to production before that data is moved into staging, and tests are run against that and validation has happened.

    So like I said, you can have the manual approval at that point if you want, if that’s a policy within your organization, or you can have that continue on automatically and go straight out to production, which is great. That’s the dream right there, is to have that end-to-end automation and have that deployment fully automated, which a lot of people work towards, which is great.

    Mitch Ashley: Well, let’s talk about that. Is there kind of a life cycle that people go through from, we’re not doing it at all now everything’s manual, and maybe it’s scripted, but it’s not automated in the same way they do through Armory. Is there sort of a step by step phases people usually go to get to a point where they might do some really automated through this?

    Adam Frank: There absolutely is. I mean, we talked to a couple of people yesterday from, I can’t remember what company now, but one of the first questions that we start to ask them is, what do you do for your organization? Okay, great. You’re a DevOps engineer, you’re a platform engineer, you’re an SRE, there’s loads them around. These two particular gentlemen said that they were just automation ninjas they didn’t really have specific titles or roles, and they just helped the organization automate everywhere they could. So the next question is then, “okay, well what are you doing to deploy your software? What are you you doing for deployment?” And they said, “Well, it’s manual right now.” Okay, that’s where most people are starting, it’s going to be manual.

    So, I think that one of the biggest things that we see is people start with CI. They’re going to automate the build and the test of that, and they’re going to have an artifact that they want to be deployed. The next thing they’ll start to do is exactly what you said. They’ll start writing some scripts and they’ll start extending some of that CI until that starts to become a little bit fragile and it doesn’t necessarily provide the developer experience that they’re looking for and enabling blue-green and your canary and things like that to protect your customer experience. So at that point, then they go out and they look for a continuous deployment solution that is going to actually meet that, and that’s where the evolution comes from.

    From that point, they’ll start to have some of the manual checks in place. They might not have integration tests all automated, that’s okay. You can have the deployment go out to dev, you can have deployment, go to staging, you can go run some manual integration tests if you need to, and then you can carry that deployment on. That would be the next piece of it, and making sure that all those pieces are automated that are being orchestrated. And then the final piece is, are you comfortable with all of this to remove that gate, to remove that manual approval to go after production and once people are, then you start to see that end-to-end automated deployment. And it’s really, really fantastic to see.

    Mitch Ashley: Really Cool. So, I’ll make a premise supposition you can tell me if it’s true or not, it seems to me that last step of getting into the, okay, yeah, we’re going to pull the pin and let it go and see and let it automatically deploy. A couple things I would want to have experience with one is, how often do we say no when we’re doing a manual decision about, okay, it’s time to release, and when we do, what are the issues? The other is, well, if something does happen, I want to be able to quickly redeploy a fix to whatever it is, and we might’ve put in production, so in minutes, an hour, hour, whatever, very quickly. If I can’t deliver code that fast then maybe I’m not ready to quite go to the automation fully onto production. Is that a good characterization?

    Adam Frank: That is, yeah. I mean, the fact that we’re talking about that as well, the rollback of that as well, are you comfortable rolling that back? Do you have the mechanisms in place that are going to allow you to roll that back? And that’s again, where a great CD solution is going to enable that both automatically or by a single click. You actually see that the baseline is off and this isn’t behaving the way that it’s supposed to be or the way that it’s expected to be, you can one click and roll that back to the last known working state, or you can have that automatically rolled back based on looking at some of that observability data that might be outside of the expected threshold.

    Mitch Ashley: Interesting. So we’re at KubeCon, let’s talk about Kubernetes, that holds another layer of complexity and technology and challenges. How does that roll into it? What are the things you need to consider about automated delivery deployment with Kubernetes?

    Adam Frank: Yeah, I think one of the big things is people have their repos structured in a way that makes sense for their business. People also have a couple of different choices in the way that they’re architecting Kubernetes, so they’re going single big clusters, are they going multi clusters, depending on what they’re doing there. And one of the mantras that we have at Armory is we’ll meet the customer where they are. So we don’t tell people to structure their repos in a certain way, we don’t tell people to architect Kubernetes in a certain way, we don’t tell people that you have to have a CRD written in a certain format. However you have your Kubernetes manifest today, we’ll deploy that.

    And I think that’s one of the beautiful things that alleviates a lot of that complexity of Kubernetes is giving people that flexibility that allows them to meet their organizational needs. We’ll deploy that, that manifest, we’ll deploy those containers any way that you need.

    Mitch Ashley: Yeah, it could be multi-cloud, could be out to the Edge and whatever regional center. I mean, it’s-

    Adam Frank: The Edge stuff is getting really interesting.

    Mitch Ashley: And multiple Applications. It is, it’s picking up. I hear a lot more about the Edge.

    Adam Frank: Yeah, absolutely. I mean, the fact that people are running pretty small clusters, but still Kubernetes clusters out in cars is a great example. That’s one of the big ones.

    Mitch Ashley: One of the people of the big manufacturers in Germany speak about that.

    Adam Frank: There you go. There you go. I know there’s a big American one that does that as well. And again, that’s one of those use cases, having that data on board, shifting that data back to a staging environment, testing with that, and then deploying back out.

    Mitch Ashley: Maybe that’ll be a light on our dashboard. You’ll see the little captain’s wheel. Kubernetes, come on you’re cluster running somewhere in your automobile.

    Adam Frank: That’s it, that’s it.

    Mitch Ashley: It needs a [inaudible 00:09:33].

    Adam Frank: Yeah, yeah. I was actually really surprised when I first started hearing about that, thinking like well that’s got to be pretty resource intensive in a car, we’ve got to figure out a way to do that without having clusters in every single one of these cars and Edge and things like that. But it seems to be working so far, and we’re talking thermostats and things of that nature as well with some of our customers. So, it’s pretty cool.

    Mitch Ashley: I heard someone describe automobiles now are becoming software platforms with wheels.

    Adam Frank: Oh, yeah.

    Mitch Ashley: I mean, essentially-

    Adam Frank: That’s exactly what they are.

    Mitch Ashley: How much of it is, it’s all software, right? With some electric motors or whatever kind of things. So what’s coming up next, the next six months where we’re going to be at re:Invent?

    Adam Frank: We are, we are.

    Mitch Ashley: Hopefully we’ll see you there. What’s on your mind about the next set of things you want to work on?

    Adam Frank: Yeah, so we’re at KubeCon, we’re talking a lot about Kubernetes, but every single person that we talk to also has something else. And whether they’ve found that Kubernetes is a little bit too complex for them and they don’t want to invest the resources in it, and they move to another container service like ECS or they’ve got some serverless. Everybody seems to have some level of serverless, whether it be Lambda or whatnot.

    So one of the things that we’ve really been working on that we’re going to be revealing pretty big at re:Invent is not only deploying to Kubernetes, but also being able to deploy to Lambda in a declarative way. Kubernetes brought us the world of declarative and gave us that abstraction layer above, the imperative nature of things. So we’re now bringing that to Lambda and full blue green, full canary, and being able to just share that declarative config between your Kubernetes or Lambda deployment, so you can actually deploy to both within the exact same deployment, which absolutely nobody’s doing right now. So we are very-

    Mitch Ashley: To both, but also in parallel, right?

    Adam Frank: Exactly.

    Mitch Ashley: Because apps maybe running in both environments, different parts of it.

    Adam Frank: Precisely. Precisely.

    Mitch Ashley: Well, sounds like you get a pretty good peek into what’s coming.

    Adam Frank: Yeah, we’ll have to get back together at re:Invent and sit down and showcase and discuss that a little bit.

    Mitch Ashley: Yeah, we’d love to hear a lot more about that. That sounds exciting. Well, good.

    Adam Frank: Yeah.

    Mitch Ashley: Well, congratulations. I said you and I have talked before the show seems really great. I mean, the buzz and the traffic and conversations-

    Adam Frank: Yeah, its always good.

    Mitch Ashley: And I’ve been to a session, I never get to go to sessions anymore we’re always talking-

    Adam Frank: Likewise.

    Mitch Ashley: Which is good, I’m good. But I’ve heard that they’re really good as well.

    Adam Frank: Yeah, most definitely.

    Mitch Ashley: Great. Have a good rest of the show and we’ll see you-

    Adam Frank: Appreciate it, thank you much.

    Mitch Ashley: In Las Vegas.

    Adam Frank: See you there.

    Mitch Ashley: Okay, re:Invent.

    Adam Frank: All right.

    Mitch Ashley: All right, man.

    Adam Frank: Cheers.

    Mitch Ashley: Say hi to Adam, stop by the booth, go visit Armory. Check him out online Armory dot… Is it io or .com?

    Adam Frank: .io, Armory.io.

    Mitch Ashley: I thought io.

    Adam Frank: You got it.

    Mitch Ashley: Okay, good. Definitely check them out and lots of good things coming too, as well as where we are today with Armory. Okay, talk to you again.

    We’ll be right back with our next great guest.

  • OpsMX Adds Deployment Firewall to Ensure Compliance

    OpsMX Adds Deployment Firewall to Ensure Compliance

    OpsMx today added a deployment firewall to its Deploy Shield portfolio for securing continuous integration/continuous delivery (CI/CD) platforms.

    The deployment firewall automatically applies rules to ensure DevOps teams comply with mandates such as the payment card industry data security standard (PCI DSS) or the Health Insurance Portability and Accountability Act (HIPAA). It then evaluates a release against a checklist of policies and blocks any deployments into a production environment unless a specific exception has been defined. In those instances, DevOps teams can also set an alert to address that issue at a later date.

    In addition, DevOps teams can extend those rules defined by OpsMX to address specific compliance requirements.

    The checklist includes making sure vulnerability scams have been run, tracking when the manifest was last updated, confirming that operational controls are being followed, checking whether images have been modified and evaluating the infrastructure being employed and performing an analysis of the performance impact.

    A simulation feature also allows developers to check their release for compliance before it is time to deploy.

    OpsMx CEO Gopal Dommety said the goal is to streamline the effort that is required to meet multiple compliance mandates as the pace at which applications are being developed and deployed continues to increase.

    Today there are simply too many data silos for DevOps teams to manage, with each one presenting yet another opportunity to run afoul of a compliance mandate, he added. The overall goal is to reduce the cognitive load being placed on developers as more responsibility for compliance is shifted left, noted Dommety.

    The deployment firewall currently supports DevOps platforms such as Jenkins, Argo and Spinnaker, with support for GitHub Actions and GitLab to follow.

    OpsMX claims its overall customer base has increased by more than 60% in the first half of this year, with customers including Google, Cisco and Western Union. In addition to Deploy Shield, the company provides a Delivery Bill of Materials (DeliveryBOM) tool to capture a consolidated record of every step in the application software delivery and deployment process. Security checks, approvals, policy enforcement and audits are tracked via one tool. In addition, the company provides tools for managing the deployments of software via either the Argo or Spinnaker CD platforms.

    It’s now only a matter of time before various regulations that apply to the building and deploying applications become more stringent. Tolerance for sloppy application development processes is dropping as legislative bodies increasingly determine that software development practices are a big cybersecurity problem that threatens national security and the global economy.

    Organizations that fail to comply with those regulations will be subject to even heftier fines than the ones that already exist. In many cases, that may slow down the rate at which applications are deployed unless DevOps teams find ways to manage compliance-as-code more effectively as part of a larger DevSecOps workflow.

    When it comes to DevSecOps, progress tends to be slow in most organizations, but as more tools become available to automate various tasks, it’s now only a matter of time before software supply chains are more tightly secured. Hopefully, the pace at which those advances are being made will occur faster than cybercriminals can exploit the vulnerabilities that today are too numerous to manually track.

  • 3 Steps to Secure Your CI/CD Pipelines

    3 Steps to Secure Your CI/CD Pipelines

    More and more software teams are waking up to the fact that their CI/CD pipelines are prone to risk. These pipelines often interface with various toolsets and automation, which may be prone to misconfigurations and abuse. This is not to mention the potential for insecure code being pushed through this funnel and external factors that threaten to circumvent the pipeline.

    In recent years, these risks produced some CI/CD tooling breaches—concerns that gained high-level attention recently after a joint CISA and the NSA alert about the importance of securing CI/CD pipelines. “This is something that moves the tectonic plates of the industry,” said Daniel Krivelevich, former founder of Cider Security and current CTO of AppSec at Palo Alto Networks.

    I reconnected with Krivelevich, the ‘godfather of CI/CD security,’ to reevaluate the CI/CD threat landscape and understand how IT should respond. According to Krivelevich, we’re now seeing more maturity around CI/CD security, both in terms of recognizing that these risks exist as well as emerging frameworks to help engineering teams respond. He shared a general three-step framework organizations can use to secure the CI/CD pipeline and surrounding areas, which we’ll explore below.

    The Evolving Threat Landscape

    The traditional approach to application security (AppSec) used to be simpler. Defenders could implement more standard controls to make their code more secure, said Krivelevich. However, throughout the years, the software engineering process has transformed dramatically. It’s become faster, dynamic and reliant upon third-party frameworks. “The entropy of the engineering ecosystem is very high,” he said.

    This paradigm shift has introduced a slew of new risks, which requires rethinking since most traditional security solutions deal solely with AppSec. “Software engineering is a big part of what makes the business tick,” said Daniel. Therefore, having a common language and understanding of this area is crucial to respond to evolving risks that threaten business operations.

    A Three-Step Framework For CI/CD Security

    Krivelevich is involved with creating the OWASP Top 10 CI/CD Security Risks; therefore, he has excellent insight into the types of vulnerabilities that most engineering practices and their CI/CD pipelines are facing. His three-step framework for CI/CD security, which Palo Alto Networks has applied in practice, looks to holistically secure not only the pipeline itself but also internal and external factors.

    1. Security in the Pipeline (SIP)

    SIP can be thought of as the traditional AppSec problem space. This involves ensuring developers are not shipping anything inherently insecure into production. Therefore, whatever code flows through the CI/CD pipeline must be vetted to ensure it has no security flaws or misconfigurations. Of course, this is easier said than done, given that teams are juggling more technologies and frameworks than ever, each of which has different knobs and requires unique security scanning capabilities.

    2. Security of the Pipeline (SOP)

    This area has to do with the security of the software delivery chain itself. Attackers may attempt to breach these pipelines by running malicious code within the pipeline or stealing credentials to CI/CD tools to harm the system. Poisoned dependencies, access management holes or insufficient credential hygiene could all be doorways to disrupting a CI/CD process. Therefore, SOP focuses on the security posture of the actual systems and tools that compromise the software delivery chain.

    3. Security Around the Pipeline (SAP)

    However, even if SIP and SOP are secure, what if hackers can bypass the pipeline altogether? That would render even a perfect set of controls null and void. Therefore, SAP is necessary to ensure that attackers can’t push code directly into production. It’s good to see the bigger picture so that everything that reaches production actually originates from the pipeline.

    To deploy SAP effectively, Krivelevich recommended leveraging both detective and preventative measures. You need detective measures to issue alerts when anomalies occur, such as an artifact deployed by a human account instead of a service account. You also need preventative measures that set the proper branch protection rules across all repositories. For instance, this could be a rule that no single developer can publish to production unless approved by two reviewers.

    The Role of AI in Software Security and CI/CD

    By using both detective and preventative measures, you can detect when settings are disabled or abused and have better configurations across all stages of the software delivery chain. But what other tactics can be used to secure CI/CD? For example, does machine learning have any merit in this area?

    Well, Krivelevich described AI as “both a blessing and a curse.” Attackers are already using AI to operate much faster and in a much more agile way. And unfortunately, he viewed the engineering ecosystem as the new path of least resistance. “I expect the power of AI will be used by sophisticated attackers who are motivated to get access to the ‘crown jewels’ by abusing the engineering ecosystem,” he said.

    On the defensive side, Daniel doesn’t see a specific opportunity for deploying AI to protect CI/CD. That being said, there’s a lot of potential for security builders to utilize AI to advance their efforts and accelerate the pace at which they detect anomalies, which could help respond more quickly to emerging threats.

    Final Thoughts on CI/CD Security

    The recent chatter around CI/CD threats appears to be garnering more attention around securing this niche area of agile software development. To thwart CI/CD attacks, it will be necessary to assess the settings of systems, understand their technical natures and analyze them against an attacker’s perspective, Krivelevich explained. Therefore, understanding what realistic attacks can be carried out will help build measures to correct these gaps.

    In doing so, the SIP, SOP and SAP descriptions serve as a solid reminder of what needs to be addressed when attempting to secure the software delivery process and the supply chain at large. Code that goes through the pipeline needs to be free from vulnerabilities, and the channel itself must be vetted for gaps that could be abused. We also need to consider the possible existential threats if the entire process can be undermined.