devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • An Overview of Continuous Security Testing Processes for DevSecOps

    An Overview of Continuous Security Testing Processes for DevSecOps

    As you likely know by now, DevSecOps is a software development methodology that blends Development (Dev), Security (Sec) and Operations (Ops) at all phases of the software development lifecycle (SDLC). It incorporates security checks into the development process and bridges gaps between development, security, and operations teams.

    With DevSecOps practices, you can build a more secure environment, make continuous integration and delivery pipelines more secure, and produce high-quality software. By incorporating security early in the pipeline, organizations can achieve higher productivity. DevSecOps isn’t an option; given the cyberattack surge, it has become necessary.

    What is Continuous Security Testing?

    As the term suggests, continuous security testing ensures that security testing happens at each phase of the SDLC. Its purpose to protect applications from potential threats and vulnerabilities by discovering its risks, threats and security hazards before the software goes to production.

    Traditional security testing checks whether an algorithm works correctly at a particular point in time. In contrast, continuous security testing detects and fixes security weaknesses and loopholes continuously throughout the application’s lifecycle.

    Continuous security testing evaluates the infrastructure, applications and endpoints for possible flaws attackers can leverage. It is an extension of continuous testing, constantly inspecting code and third-party libraries against known or newly discovered security issues.

    Tools can help personnel automate security checks in an application. Several options are available to serve different phases of the development, integration, and deployment processes.

    Regularly running security tests ensures adherence to industry standards. More importantly, it compels developers to implement top-notch security safeguards in their code to thwart potential security threats in the long term.

    Common Practices for Integrating Security Testing in DevSecOps

    The SLDC has several phases, and it can take a while to weave together the traditional development methodology with newer processes. Anyone who wants to integrate security testing in DevSecOps should start with these practices.

    Running Automated Security Checks

    A key element in DevSecOps is automating security checks during the SDLC process. Doing so detects vulnerabilities at an early stage, when it’s easier, faster, and less frustrating to fix problems. Automated security checks work in the background, so developers can concentrate on the other elements of their applications. This is analogous to having an extra set of vigilant eyes that point out potential hazards.

    Whichever software you choose, automated tools can scan code for real-time vulnerabilities during the build process or at other suitable points. The instant feedback allows teams to address issues much before they become major problems, thereby ensuring the security of the code base.

    Integrating Code Review

    Code review is a time-honored tradition among Agile development teams, for reasons that go far beyond security. It’s also a good way to inject continuous security testing in the SDLC process, in part because it’s already familiar to so many teams. Integrating code review with vulnerability scanning tools provides immediate feedback on the source code’s security weaknesses.

    Improving Security Awareness

    Ideally, your organization has already invested in training to help both end users and developers recognize the ways that applications and business practices can make them vulnerable. Regular training keeps developers updated about new threats and encourages information sharing between teams. The more they know, the better they can identify potential security threats and respond quickly and efficiently.

    Continuous Monitoring

    DevSecOps doesn’t stop when the software is launched. What happens once the application goes into production? Continuous monitoring involves monitoring an application against potential real-time security threats by paying attention to logs, events and network traffic. It helps organizations detect possible threats immediately, thus preventing – or at least responding quickly to – security breaches.

    Threat Modeling

    Sometimes you have to think about what could happen, ahead of time. Threat modeling is the practice of identifying potential security threats early – not just what is in the code, but what someone mght try to do to gain access or to take advantage of weaknesses. This is often accomplished by examining an application’s design and architecture to detect potential attack vectors – often with the help of sophisticated tools.

    Don’t Hesitate: Deploy Continuous Security Testing Now

    The success of DevSecOps—the integration of security into DevOps—requires a change in perspective, as well as new resources and methods. It would be wise to embrace DevOps’s collaborative and agile mindset make the development process smooth and transparent and ensuring that security is as smooth and unobtrusive as possible.

  • The Role of AI in Securing Software and Data Supply Chains

    The Role of AI in Securing Software and Data Supply Chains

    Supply-chain vulnerabilities loom large on the cybersecurity landscape, with threats and attacks such as SolarWinds, 3CX, Log4Shell and now XZ Utils underscoring the potentially devastating impact of these security breaches. The latter examples of Open Source Software (OSS) attacks are a growing attack vector. Capterra’s research reveals that software supply-chain attacks impacted almost two-thirds (61%) of all U.S. businesses from April 2022 to April 2023.

    Expect attacks on the open source software supply chain to accelerate, with attackers automating attacks in common open source software projects and package managers. Many CISOs and DevSecOps teams are unprepared to implement controls in their existing build systems to mitigate these threats. In 2024, DevSecOps teams will migrate away from shift-left security models in favor of “shifting down” by using AI to automate security out of the developers’ workflows.

    Here are the factors fueling the increase in software supply-chain attacks and the role of AI in helping developers work more efficiently while creating more secure code.

    Attacks on the Open Source Software Supply Chain Will Accelerate

    Open source libraries and languages are the foundation of over 90% of the world’s software. In a U.S. survey of nearly 300 IT and IT security professionals, 94% reported their companies use open source software, and 57% employ multiple open source platforms. Exactly half of the respondents say the threat level is “high” or “extreme,” while another 41% view it as “moderate.” At the time of writing, details of the backdoor implanted in the XZ library and several other OSS packages have just been published. The ubiquity of open source worldwide is one key factor fueling the rise of supply chain attacks.

    Data Governance and the Data Supply Chain Will Become Critical Issues

    Security pros must also consider how security vulnerabilities extend to their data supply chains. Although organizations typically integrate externally developed software through their software supply chains, their data supply chains often need clearer mechanisms for understanding or contextualizing data. In contrast to software’s structured systems or functions, data is unstructured or semi-structured and faces a wide array of regulatory standards.

    Many companies are building AI or ML systems on top of enormous data pools with heterogeneous sources. ML models on model zoos are published with minimal understanding of the code and content used to produce the models. Software engineers need to handle these models and data just as carefully as they do the code going into the software they’re creating, with attention to its provenance.

    DevSecOps teams must assess the liabilities of using data, especially when building LLMs to train AI tools. That demands careful data management within models to prevent the accidental transmission of sensitive data to third parties like OpenAI.

    Organizations should adopt strict policies outlining the approved usage of AI-generated code, and when incorporating third-party platforms for AI, conduct a thorough due diligence assessment ensuring that their data will not be used for AI/ML model training and fine-tuning

    AI Security Automation Will Help Organizations Transition From ‘Shift-Left’ to ‘Shift-Down’

    The industry adopted the shift-left concept a decade ago to address security flaws early in the software development lifecycle and to enhance developer workflows. Defenders of systems have long been at a disadvantage. AI has the potential to level the playing field. As DevSecOps teams navigate the intricacies of data governance, they must also assess the impact of the evolving shift-left paradigm on their organizations’ security postures.

    Companies will begin moving beyond shift-left to embrace AI to fully automate security processes and remove them from the developer’s workflow. This is called “shifting-down,” because it pushes security into automated and lower-level functions in the tech stack instead of burdening developers with complicated and often difficult decisions.

    GitLab’s Global DevSecOps Report: The State of AI in Software Development found that developers only spend 25% of their time on code generation. AI can elevate their output by optimizing the remaining 75% of their workload. That’s one way to leverage AI’s capacity to solve specific technical issues and improve the efficiency and productivity of the entire software development life cycle.

    Expect to look back on 2024 as the year that the escalating threats OSS ecosystems adversely affecting global software supply chains, catalyzed substantial changes in cybersecurity strategies, including a heightened dependence on AI to safeguard digital infrastructures. The cybersecurity landscape is already transforming, with a growing focus on mitigating supply chain vulnerabilities, enforcing data governance and incorporating AI into security measures. This transformation promises to steer DevSecOps teams toward software development processes with efficiency and security at the forefront.

  • How to Migrate an Observability Platform to Open Source

    How to Migrate an Observability Platform to Open Source

    Achieving full observability across enterprise applications is challenging. While third-party services like New Relic or Datadog make end-to-end observability easier, as your application grows in complexity, so does the telemetry data, leading to increased costs. Migrating to an open source stack gives you control over telemetry data and reduces observability costs, despite challenges with existing service provider commitments.

    Migrating to open source solutions is straightforward with simple architecture and tech stacks. However, transitioning the entire observability stack to open source demands meticulous planning, tools/frameworks assessments, testing, risk analysis, and cross-team communication, especially given the number of microservices, heavy cloud reliance, and diverse languages and frameworks to take into account.

    A cloud observability platform usually absorbs 20%-30% of overall infrastructure spending, but in some cases, it can reach 50%-60%. If your observability expenses exceed 50%, then transitioning to open source solutions would be a sensible option to mitigate tech infrastructure costs.

    For the past two years, I have been working on migrating the observability platform from a proprietary solution provider to an open source stack. Let’s dive into the key steps necessary for such a migration.

    Finalize the key telemetry data and systems

    Managed observability platforms offer comprehensive insights into system health, necessitating a significant volume of high-quality telemetry data. By default, the agents tasked with capturing and transmitting telemetry data are configured to gather as much information as possible, facilitating the creation of extensive dashboards and reports. However, this extensive data collection contributes to the overall cost.

    Select which systems you actually need to monitor. A typical enterprise application includes databases, caching databases, container orchestrators, and numerous cloud services. Engineers often monitor more services than necessary, which leads to unnecessary complexities and costs. It’s often sufficient to streamline the monitoring scope to essential systems.

    In my experience, the quantity of telemetry data required is significantly lower than what third-party services typically retrieve. You can easily divide those data collections into two buckets: must-have and nice-to-have. The “must-have” data aligns with the well-established four golden signals:

    • Latency: the amount of time it takes to serve a request
    • Traffic: the volume of requests that a system is currently handling
    • The error rate: the number of requests that fail or return unexpected responses
    • Resource saturation: the percentage of available resources being consumed

    These fundamental signals serve as the foundation for key dashboards and alerts. Observability platforms built on these signals cover 80% of typical use cases.

    Understanding each signal’s metadata requirements is crucial. The extent of metadata captured directly impacts the observability platform’s complexity and cost. For example, when monitoring service-to-service latency, consider the necessity for service IP addresses, instance IDs, or header information.

    Select the relevant stack

    Managed observability platforms such as New Relic and Datadog provide comprehensive monitoring for various enterprise components. However, when you migrate to an open source stack, you must assess and integrate the tool stacks to meet diverse monitoring requirements.

    One important aspect to consider is scaling: how to manage the huge amount of data generated every minute across all those systems. Focus on two fronts: selecting the stack for storing and processing telemetry data (logs, metrics and traces) and devising methods for capturing and pushing telemetry data from diverse systems.

    Telemetry signal stack selections
    Telemetry signal stack selections

    Stack for logs

    You require a stack that can efficiently and economically process and store the substantial volume of logs generated by the system. In the past, I used the ELK stack for storing and searching logs, but it is a generic solution and not tailored specifically for logs.

    I recommend Grafana’s Loki for its effective management of large log volumes and LogQL, a language akin to PromQL. If you are familiar with PromQL, navigating through extensive log data with LogQL is easy and intuitive.

    Stack for Metrics

    Prometheus is a popular database for storing time-series metrics data. However, it has scaling-related limitations; it cannot scale horizontally. Alternatives like Thanos, Grafana’s Mimir, and VictoriaMetrics offer better out-of-the-box solutions for horizontal scaling.

    I conducted some research and ultimately chose Grafana’s Mimir to store metrics in a similar migration project. This decision was based on Mimir’s remote storage capability, as well as its scalable and highly available architecture.

    Stack of Tracing

    Distributed tracing is a must in a microservices architecture, for easily identifying bottlenecks in latency and for resolving performance-related bugs. Grafana’s Tempo could be a good option for several reasons:

    • It can seamlessly integrate as a backend for Grafana’s dashboard. Consolidating dashboards in Grafana helps avoid navigating between multiple applications. Switching platforms just for tracing while using Grafana for other aspects of observability would not be practical.
    • Tempo provides a similar experience to TraceQL when interacting with the database, streamlining the learning process and significantly reducing the learning curve.
    • Tempo is compatible with popular open source tracing protocols such as Zipkin and Jaeger. If any team already uses these protocols, transitioning to Grafana’s Tempo would be smoother.
    Capturing and pushing telemetry data
    Capturing and pushing telemetry data

    You have to install an agent on the system to capture and push the telemetry data. Those steps vary based on the system being monitored. Typical enterprise application setups involve monitoring cloud integrations, process monitoring, infrastructure hosts, Kubernetes clusters and application monitoring. Among the possibilities:

    • Cloud integrations: This includes monitoring for cloud services such as SQS, SNS, EMR and EC2.
    • Process monitoring: This involves monitoring processes running on bare-metal machines. With the advent of dockerization and Kubernetes, there is now a standardized way of starting the application. In the past, there was no fixed mechanism. For example, to run a Java application, you could use a Java application, the ‘java -jar’ command, Tomcat, or OS systemctl. In the case of node, it could be npm or PM2. Each team or service may have had its own way of starting the process.
    • Infrastructure host: This entails monitoring the machine itself for metrics such as CPU usage, memory, disk I/O and Network IO or checking if the machine is offline.
    • Kubernetes monitoring: You regularly need to monitor a Kubernetes cluster, such as instances where Kubernetes is unable to schedule a pod due to insufficient resources.
    • Application monitoring: This monitoring focuses on overseeing the services the team created. Each service may differ in its development approach and choice of tech stack; but from an observability perspective, they usually are treated the same.
    • Browser and mobile monitoring: These metrics ensure optimal performance and user experience across different platforms. Browser monitoring includes tracking page load times, rendering performance, JavaScript errors and resource usage. For mobile, it includes monitoring app crashes, latency, battery usage, network requests and device-specific metrics.

    Before the advent of OpenTelemetry, there was no standardized method for monitoring applications. OpenTelemetry emerged from the merger of two earlier projects, OpenTracing and OpenCensus. It stands as a vendor- and tool-agnostic framework to instrument applications or systems irrespective of language, infrastructure or runtime environment. OpenTelemetry represents a significant community effort, and its popularity and stability are steadily growing.

    Validate observability stack on application architecture

    Applications vary in architecture and development stages. To address this diversity, you need multiple proofs-of-concept (POCs) across different systems, such as infrastructure, cloud service and backend and frontend applications.

    Migrating an observability platform from one tech stack to another is simple when the application architecture is a monolith. However, when the underlying architecture is microservices, it becomes difficult.

    While the microservices architecture offers flexibility in choosing different tech stacks for building services, devising a standard solution for capturing telemetry data becomes challenging due to the diverse range of tech stacks involved.

    Conducting POCs for various tech combinations is necessary to build a self-service tool that individual teams can follow to migrate their services easily. For instance, if your service uses Java 11, Spring Boot 3.x.x and PostgreSQL, these POCs can provide standardized steps to enable application monitoring.

    Migrate core components

    Setting up the observability platform involves configuring the stack for metrics, logs and traces, as well as installing the necessary agents. One advantage of migrating the observability platform is that there’s no need to transfer old telemetry data. During the testing phase, you will have an overlapping period to accumulate enough useful data to create specific dashboards and alerts.

    Migrating alerts and dashboards to the new system is essential, even when data migration is not necessary. This is because changes in the underlying techniques for capturing metrics may modify the names of the metrics and subsequently change all query expressions used in alerts and dashboards. Some metrics may not be available in certain scenarios, especially derived metrics that can be recreated from underlying metrics.

    While migrating query expressions manually is an option, a manual process is error-prone and time-consuming for a large number of alerts. I had experienced a similar situation with alert numbers ranging from 100 to 400.

    To streamline the migration, we developed a Node.js script that programmatically converted New Relic’s query expressions to Prometheus expressions. The script performed the following high-level steps:

    • Connected to the New Relic API server and fetched all the alerts configured for cloud integration
    • Converted New Relic alert expressions into PromQL alert expressions
    • Wrote all the Prometheus alert expressions into the YAML file

    Implementing this technique allowed us to successfully migrate all alerts and dashboards in four or five days, a process that would normally demand a month’s manual labor. Additionally, scripting significantly minimized the risk of human error.

    Test the migration

    Verify each alert and dashboard. Run both old and new platforms during testing. Parallel operation permits thorough testing of the metrics.

    However, running both systems concurrently may lead to a performance hit due to services sending telemetry data to two locations. Additionally, there may be a slight increase in costs until the complete transition.

    Running both systems in parallel provides an ideal setup to ensure the migrated dashboards and alerts are functioning correctly.

    Testing alerts that trigger during this process is straightforward, but evaluating those that have not yet triggered poses a challenge. In one project, I reduced the threshold value of each alert and tested its functionalities. Approximately 90% of the alerts generated through custom scripts worked seamlessly, with only 10% requiring some manual tweaking.

    Migrate other related components

    Other systems depend on the data generated by the observability platform. For example:

    • Notification systems like email, Slack channel, etc., that alert in case of an incident
    • Incident management tools like PagerDuty provide a streamlined way to handle incidents.

    These systems rely on the alert’s payload to work properly. Their assets should be updated, as a change in the observability platform leads to a change in the alert payload. Update the templates of notification systems to ensure they seamlessly integrate with the new alert payload.

    Similarly, incident management tools like PagerDuty require modifications to routing rules, escalation policies, and scheduling. Fortunately, readymade open-source migration tools are available to facilitate easy migration from PagerDuty to Grafana’s OnCall. However, in other solutions, you may not have access to out-of-the-box migration tools. In such cases, manual migration or script writing may be required.

    Timeframe for the Migration

    Planning a migration involves finalizing key telemetry data and systems, selecting relevant stacks. and validating the observability stack on the application architecture. It typically takes around three weeks.

    The project duration depends on factors such as the number of teams involved and the quantity of active services. For instance, a migration to open source with 100 microservices and involvement from 10 different teams may take four months to complete.

    Migration Complete!

    Transitioning your observability platform to an open-source stack offers a promising avenue for cost reduction and increased control over telemetry data. However, accomplishing this migration demands meticulous planning and execution, encompassing essential steps like feature prioritization, stack selection, POCs, core component migration, testing, and the migration of related systems. Despite the challenges posed by diverse architectures and technologies, a systematic approach, comprehensive assessment, and collaboration among teams can facilitate a smooth migration process and guarantee its success.

  • The Rise of Low-Code/No-Code in DevOps

    The Rise of Low-Code/No-Code in DevOps

    The ever-evolving landscape of technology has ushered in a paradigm shift in software development and deployment. One of the prominent trends gaining momentum is the integration of low-code/no-code platforms within the DevOps ecosystem. This article explores how these platforms are revolutionizing the way applications are developed, deployed and managed, fostering collaboration between developers and operations teams.

    1. Democratizing Development:

    Low-code/no-code platforms empower individuals with limited coding experience to actively participate in the application development process. This democratization of development accelerates the delivery of software by allowing business analysts, designers and other non-developers—sometimes referred to as citizen developers—to contribute directly to the creation of applications.

    2. Accelerating Time-to-Market:

    The traditional software development life cycle often involves lengthy coding processes, leading to delayed releases. Low-code/no-code platforms enable rapid prototyping and development through visual interfaces, significantly reducing the time required to bring applications from concept to production. This acceleration aligns with the principles of DevOps, promoting continuous delivery and faster time-to-market.

    3. Enhancing Collaboration:

    DevOps is all about breaking down silos and fostering collaboration between development and operations teams. Low-code/no-code platforms bridge the gap between these traditionally distinct roles by providing a common ground where both technical and non-technical stakeholders can collaborate seamlessly. This collaborative environment promotes better communication, understanding, and alignment of objectives.

    4. Flexibility in Deployment:

    Low-code/no-code solutions are designed to be platform-agnostic, allowing for the deployment of applications across various environments, including on-premises and cloud. This flexibility aligns with the DevOps principle of infrastructure-as-code (IaC), enabling teams to deploy and manage applications consistently across diverse infrastructures.

    5. Reducing Dependency on Specialized Skills:

    In a DevOps environment, reducing bottlenecks and dependencies on specific individuals or roles is crucial. Low-code/no-code platforms enable teams to be more self-sufficient, reducing the reliance on specialized coding skills or DevOps Training. This, in turn, contributes to a more resilient and adaptable DevOps culture.

    6. Challenges and Considerations:

    While the adoption of low-code/no-code in DevOps brings numerous benefits, it also introduces challenges. This section discusses considerations such as security, scalability and the potential need for customization beyond the platform’s capabilities.

    7. Real-World Use Cases:

    Explore real-world examples of organizations successfully incorporating low-code/no-code in their DevOps workflows. Highlight the positive outcomes, lessons learned, and the impact on their overall software development life cycle.

    Conclusion:

    The rise of low-code/no-code platforms in DevOps is reshaping the way software is developed and deployed. By fostering collaboration, accelerating development cycles, and reducing dependencies on specialized skills, these platforms align seamlessly with the principles of DevOps. As organizations continue to embrace this trend, the synergy between low-code/no-code and DevOps is set to redefine the future of application delivery.

  • Survey Surfaces Substantial Platform Engineering Gains

    Survey Surfaces Substantial Platform Engineering Gains

    A global survey of 500 application development and deployment professionals finds that while 43% work for organizations that have had a platform team in place for three to five years, there doesn’t appear to be much consistency regarding where that team reports within an organization.

    The survey, conducted by Puppet by Perforce, a provider of an automation framework, found that among organizations that have platform teams, well over half (58%) operate within the context of a larger DevOps or infrastructure management organization, compared to 40% that have their own dedicated leadership.

    David Sandilands, a principal solutions architect for Puppet by Perforce, said the survey makes it clear that platform engineering, in general, is not a new concept, but it’s clear more organizations are trying to reduce the level of toil currently associated with building and deploying applications.

    Regardless of where platform teams reside in an organization, nearly two-thirds of respondents (65%) said they are important to their organizations and would receive continued funding. The primary purpose for forming these teams is to increase productivity (58%) and automate standardized processes (51%), the survey finds. Top goals for platform engineering teams are to solve problems (30%), enforce security processes (27%) and accelerate transitions to cloud-native computing environments (26%). A full 70% said security was built into their platforms from the start.

    That suggests that, in addition to assuming responsibility for traditional DevOps workflows, many platform teams are now also being asked to ensure DevSecOps best practices are followed at a time when application security requirements are becoming more stringent, noted Sandilands.

    More than three quarters (76%) also reported that they had deployed two or more self-service portals, with 27% having deployed five or more.

    Only 22%, however, said they have deployed applications on Kubernetes in a production environment, with just under half (46%) reporting they have no current plans to deploy a cloud-native computing platform, which are widely seen to be challenging to deploy and maintain.

    Overall, the survey makes it clear the primary focus is to ensure platforms continue to evolve as developer requirements continue to evolve, said Sandilands. Platform teams are not necessarily trying to dictate what tools should be used; instead, they are finding ways to automate processes that are increasing the level of friction developers encounter, he added.

    In the longer term, advances in artificial intelligence (AI) will make it simpler to achieve that goal using natural language interfaces that DevOps teams can more easily invoke, noted Sandilands.

    Each organization will need to decide for itself whether platform engineering makes sense as a methodology for managing DevOps at scale, but as application development and deployment continues to evolve, more nuanced approaches will be required. In general, organizations are trying to strike a balance between empowering developers and the need to streamline backend processes to increase productivity while also simplifying compliance and improving security.

    The challenge, of course, is enticing developers to buy into that concept rather than using their expertise to resist platform teams that might, from their perspective, limit their prerogatives in ways that ultimately stifle innovation.