devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • Mastering DevOps Automation for Modern Software Delivery

    Mastering DevOps Automation for Modern Software Delivery

    DevOps automation services are a crucial aspect of modern software development that helps organizations to improve their software delivery processes and keep up with the pace of innovation. With DevOps automation, developers can automate repetitive and time-consuming tasks, enabling them to focus on writing code and delivering features faster and with higher quality. By automating processes such as continuous integration, continuous delivery and infrastructure-as-code, organizations can reduce costs, improve collaboration between teams and respond quickly to changing business needs and customer requirements.

    In this article, we will discuss the basics of getting started with DevOps automation services, including popular tools and best practices that organizations can use to implement a successful automation strategy.

    Agility and Efficiency for Smart Digitalization

    Emerging technologies like multi-cloud, IaC, AI/ML and IoT have only increased the importance of DevOps automation, as they bring new challenges demanding agility, flexibility and efficiency in software delivery processes. Multi-cloud and hybrid cloud environments, for example, can benefit from automating the deployment and management of software across multiple cloud environments. Similarly, AI/ML and IoT require organizations to collect and process large amounts of data in real-time, which can be challenging without automation. DevOps automation can help organizations automate data pipelines and analytics, enabling them to process data faster and more accurately.

    Here are some benefits of DevOps automation:

    ● Speed and Efficiency: DevOps automation enables organizations to deliver software faster and with higher quality. By automating repetitive and time-consuming tasks, developers can focus on writing code and delivering features.
    ● Agility and Flexibility: DevOps automation provides organizations with the agility and flexibility to adapt to changing business needs and customer requirements. Automated processes make it easier to release software updates and respond quickly to market demands.
    ● Cost-Effectiveness: DevOps automation reduces the costs associated with software development by reducing manual labor and minimizing errors. It also enables organizations to use cloud resources more efficiently, which can result in cost savings.
    ● Improved Collaboration: Bringing the different stakeholders along with devs and operations, DevOps fosters a collaborative approach toward enhanced CI/CD. By automating processes, teams can work together more effectively and share knowledge and feedback in real-time.
    ● Scalability: DevOps automation enables organizations to scale their software delivery processes quickly and efficiently. Automated processes make it easier to deploy software across multiple environments and platforms, which is essential for organizations that need to scale rapidly.

    Getting Started With Efficient DevOps Automation

    Implementing DevOps automation requires a well-planned strategy, best practices and the right set of tools. In this section, we will discuss how to successfully implement DevOps automation in your organization.

    Understanding DevOps
    Before you begin with automation, it is essential to understand the principles and practices of DevOps. DevOps is a culture that emphasizes collaboration and communication between software development and IT operations teams to build, test and release software more quickly and reliably. There are some common tools that DevOps teams use for their work, collaboration and communication:
    ● GitHub: The Git repository hosting service used for version control and source code management functionality, especially for distributed projects.
    ● JIRA: A project management tool that helps teams plan, track and manage software development projects.
    ● Slack: A communication platform that enables real-time messaging, file sharing and collaboration.

    Choose the Right Tools
    Automation is at the heart of DevOps, and selecting the right tools is crucial. Choose tools that fit your requirements, budget and infrastructure. Some popular DevOps automation tools include:

    ● Jenkins: An open-source automation server that enables continuous integration and continuous delivery (CI/CD) pipelines.
    ● GitLab: A web-based Git repository manager that provides CI/CD capabilities, project management, and collaboration tools.
    ● Puppet: An open-source configuration management tool that automates the deployment and management of software across multiple environments.
    ● Chef: A configuration management tool that automates the deployment and management of software across multiple environments.
    ● Ansible: Open source tools for automating software provisioning, deployment and configuration management

    Identifying Processes for Automation

    First, list all the critical processes within your organization that can be enhanced with automation. Start with the repetitive and time-consuming tasks that can be automated, such as code testing, deployment and monitoring. There are a number of popular tools that can help with such automation:

    ● Selenium: An open source testing framework that enables automated testing of web applications.
    ● Docker: A containerization platform that enables the creation, deployment and management of containerized applications.
    ● Kubernetes: The open source container orchestration tool is essential in the DevOps pipeline for orchestration and automation of the deployment, scaling and management of containerized applications.

    Creating a Plan

    Once you have identified targets for automation and selected tools, create a plan for automation that includes a roadmap and milestones. Start with small projects and gradually expand the automation to cover all aspects of the software development life cycle. Some popular tools for strategizing these life cycles are:

    ● Trello: A project management tool that enables teams to organize and prioritize tasks using Kanban-style boards.
    ● Asana: A project management tool that enables teams to track and manage tasks using to-do lists, calendars and timelines.

    Training The Team

    Automation requires new skills, so it is essential to train your team to use the tools and processes effectively. Provide training and resources to help your team learn new skills and techniques. Some popular education and training resources include:

    ● Udemy: An online learning platform that provides courses on DevOps automation tools and techniques.
    ● Coursera: An online learning platform that provides courses on DevOps and related topics.
    ● Pluralsight: An online learning platform that provides courses on DevOps automation tools and techniques.

    Implementing Continuous Improvement

    A continuous feedback loop is essential for DevOps’ iterative process and continuous improvement. Integrate mechanisms for evaluating your automation processes, identify areas for improvement and make necessary changes. Here are two common tools used to do this:

    ● New Relic: An application performance monitoring tool that provides real-time visibility into the performance of software applications.
    ● Splunk: An operational intelligence tool that enables organizations to analyze and visualize machine data in real-time.

  • DigiCert Allies With ReversingLabs to Secure Software Supply Chains

    DigiCert Allies With ReversingLabs to Secure Software Supply Chains

    DigiCert today announced it has allied with ReversingLabs to integrate binary analysis and threat detection capabilities with a code signing service it provides.

    Deepika Chauhan, chief product officer for DigiCert, said the addition of these capabilities to the DigiCert Software Trust Manager service would make it simpler for organizations to operationalize a methodology for securing their software supply chains. The capabilities are designed to be embedded within continuous integration/continuous delivery (CI/CD) pipelines.

    In addition to being a certificate authority, DigiCert’s Software Trust Manager service automates code signing workflows. ReversingLabs, meanwhile, just added the ability to detect secrets exposed in application binaries to its Software Supply Chain Security (SSCS) platform. Those capabilities can identify secrets inadvertently left in applications as plain text, can identify secrets that can be discovered because of weak cryptography and also includes scripts that scan for malware and can generate software bills of materials (SBOMs).

    Chauhan said when combined with DigiCert Software Trust Manager, it becomes possible for the two companies to provide a comprehensive approach to detecting malware, software tampering, inclusion of secrets and certificate misconfigurations in any type of software.

    A recent ReversingLabs survey found 87% of respondents detected significant risks in their software supply chain in the last year, with software containing vulnerabilities (82%) followed by secrets leaked through source code (55%), malicious code (52%) and suspicious code (46%) posing the most serious risk to the business.

    A full 88% said software supply chain security is an enterprise-wide risk, but only 60% said their software supply chain defenses were up to the task. Nearly two-thirds (65%) of respondents, however, acknowledged their organization’s software supply chain security program wasn’t as mature as it should be. As a result, 80% are currently focused on improving security for the software supply chain, with 96% noting a more comprehensive approach to software supply chain security that detects more than vulnerabilities is needed.

    It’s not entirely clear just yet whether development teams or cybersecurity teams will be taking the lead in terms of driving the adoption of DevSecOps best practices to secure software supply chains. It is clear that securing software supply chains in the wake of a series of high-profile breaches is a much higher priority.

    Erik Thoen, director of product management for ReversingLabs, said as a result, it’s only a matter of time before development teams and cybersecurity professionals are forced to work more closely together than they have in the past.

    In the longer term, a wave of legislation in the U.S. and Europe will soon require organizations to embrace DevOps best practices to address increased potential liability that might stem from an application breach. A proposed Cyber Resilience Act currently being negotiated by the member states of the European Union seeks to require organizations that sell internet-connected hardware platforms to ensure both their devices and their software comply with cybersecurity best practices. In the U.S., meanwhile, a National Cybersecurity Strategy proposal put forward by the Biden administration seeks to hold organizations that collect data or build software more accountable for breaches.

    While both proposals are a long way from becoming law, it’s clear governments around the world are concluding the only way to ensure better cybersecurity is to require it.

  • The Future of Continuous Testing in CI/CD

    The Future of Continuous Testing in CI/CD

    If you ever wonder why continuous testing is so important, think about the following: In the past, software testing was typically done after the code was written and sent to the QA department for independent testing. When bugs were discovered, the code was returned to the developers for correction. While this testing method worked, it was time-consuming. Nowadays, companies want to produce high-quality products quickly, so they need a faster and more efficient testing process. This is when continuous testing comes in.

    In this article, we will explore continuous testing from its current state and its future in CI/CD to the emerging technologies and trends that will shape its development. We will also cover the importance of continuous testing for businesses and some of its challenges.

    What is Continuous Testing?

    Continuous testing is a software testing approach in which tests are automated and run throughout the software development life cycle. It is a part of the continuous delivery (CD) process, in which software changes are delivered to customers continuously and iteratively.

    Continuous testing aims to identify defects and issues as early as possible, reducing the cost and time of fixing them later. Continuous testing provides immediate feedback to the development team, helping them to detect and address potential problems quickly and efficiently.

    According to the Reports and Data source survey, 21% of QA professionals have incorporated continuous testing, source code management and continuous integration into their DevOps process to speed up code release.

    Advantages of Continuous Testing

    Here are several advantages of implementing continuous testing as a part of the software development process:
    ● Continuous testing enables teams to identify bugs earlier in the software development process;
    ● Continuous testing provides immediate feedback to developers on the quality of their code, enabling them to address any issues immediately;
    ● You can minimize business risks by assessing and identifying potential issues before they become real problems;
    ● Automating your test cases can reduce the time spent on testing;
    ● Accelerating the process of software delivery to production helps release the product faster;
    ● You can run performance tests in similitude, which can increase the speed of test execution.

    Disadvantages of Continuous Testing

    While continuous testing has become an integral part of the software development process, it’s not without its challenges. As software development teams work to incorporate testing into their CI/CD pipelines, they may encounter obstacles that can hinder their progress.
    ● Continuous testing requires a different skillset than traditional testing, including expertise in DevOps, automation and CI/CD pipelines;
    ● Different tools and technologies are available for continuous testing, and choosing the right ones can be tough. Integrating these tools with other parts of the development process can also be challenging;
    ● To fully realize the benefits of continuous testing, you need to update your testing strategy by moving away from relying solely on traditional testing methods and adopting a well-defined approach to test data management;
    ● Failing to integrate code regularly can result in issues such as duplicated coding efforts and non-compatible code, leading to defects.

    Emerging Technologies in Continuous Testing

    With the increasing demand for faster and more reliable software releases, emerging technologies such as AI/ML, automation and DevOps appeared in continuous testing. These technologies help teams to automate and streamline their testing processes, identify defects more quickly and improve overall software quality. Let’s examine some of them:

    Artificial intelligence/machine learning (AI/ML). AI and ML refer to the field of computer science that deals with developing intelligent machines that can simulate human intelligence and learn from data.

    One of the main benefits of AI and ML in continuous testing is the ability to automate test case generation, which reduces the need for manual testing and frees up time for other tasks. This enables testers to test applications more comprehensively and frequently, ensuring defects or issues are identified and addressed quickly. AI and ML increase testing accuracy by identifying patterns and anomalies in the testing data.

    AI and ML techniques can efficiently and rapidly analyze extensive amounts of data, allowing them to recognize patterns and irregularities in testing data that human testers might overlook.

    Automation: Automation in continuous testing refers to using automated tools and processes to execute software tests repeatedly and consistently. It involves automating various stages of the software testing life cycle, such as test planning, test case design, test execution and test reporting.

    Automation can efficiently perform regression testing, ensuring that changes do not introduce new defects or issues in the software. Tests can be executed 24/7 without human intervention, reducing the time and cost required for testing.

    DevOps: DevOps in continuous testing is the application of DevOps principles to continuous testing, focusing on integrating testing into the software development life cycle and enabling faster and more reliable delivery of software.

    In the context of continuous testing, DevOps emphasizes testing integration into the development process, enabling developers to detect and fix defects early in the cycle. DevOps also promotes automation, collaboration and communication between development and operations teams, allowing faster and more efficient testing and deployment.

    Blockchain: Blockchain technology has been gaining increasing attention in the software development and testing industry due to its ability to provide secure and tamper-proof data storage.

    In continuous testing, blockchain stores testing results and data in a decentralized and transparent way, making it virtually impossible for anyone to modify or manipulate the data. This enhances the credibility of the testing process and outcomes and provides an added layer of security to ensure the integrity of the data.

    Blockchain technology enables efficient and seamless data sharing between stakeholders involved in the testing process, enabling faster and more accurate decision-making.

    Internet-of-Things (IoT): In continuous testing, IoT simulates many IoT devices and platforms, enabling testers to test their applications across multiple devices and platforms. This ensures that applications are compatible with various IoT devices and platforms and perform reliably and consistently in various environments.

    For instance, when creating an application that needs to run on multiple devices, it is possible to use an emulator to test the app on your laptop without setting up a development environment for each type of device.

    IoT-enabled continuous testing provides real-time feedback on the performance and functionality of applications, enabling organizations to identify and address issues quickly and effectively.

    Trends in Continuous Testing

    Continuous testing is rapidly evolving to meet the needs of modern software development practices, with new trends emerging to address the challenges development teams face. Three key trends currently gaining traction in continuous testing are cloud-based testing, shift-left testing and security testing. These trends are driven by the need to increase efficiency and speed in software development while ensuring the highest quality and security levels. Let’s take a closer look at these trends.

    Cloud-Based Testing: Continuous testing is deployed through cloud-based computing, which provides multiple benefits like ease of deployment, mobile accessibility and quick setup time. Businesses are now adopting cloud-based services due to their availability, flexibility and cost-effectiveness. Cloud-based testing doesn’t require coding skills or setup time, which makes it a popular choice for businesses.

    AI-Driven Testing: AI-driven continuous testing is often integrated into cloud-based testing platforms, providing end-to-end automation for web, mobile and API testing. AI and cloud-based testing allows for access to testing environments anywhere, anytime, and provides for regular automated testing using platforms like Selenium.

    Shift-Left Testing: Shift-left testing is software testing that involves testing earlier in the development cycle rather than waiting until later stages, such as system or acceptance testing. The goal of shift-left testing is to identify and address defects as early as possible.

    Delaying testing until the end of development often results in more complicated bug fixes since the software is already fully built. This can cause an increase in cost and time to market. Identifying and fixing bugs earlier in the software development life cycle is generally more cost-effective.

    As IBM’s Systems Sciences Institute claimed, repairing a bug discovered during implementation costs roughly six times more than one found during design. If a mistake is detected after product release, the cost of fixing it is four to five times greater than one found during design and up to 100 times greater than one found during the maintenance phase. This suggests that a bug’s cost increases exponentially as the software progresses through the SDLC.

    Low-Code/No-Code: Low- or no-code in continuous testing refers to using visual interfaces and pre-built elements or modules to create and execute automated tests instead of writing complex code. Low-code/no-code testing tools often provide drag-and-drop interfaces and pre-built components that can be combined to create tests, reducing the time and effort required to develop and maintain test scripts.

    This type of testing provides several advantages, such as allowing non-technical business experts to run testing, speeding up test production and execution and enhancing product quality by enabling more thorough and efficient testing.

    Generally, according to statistics, by 2025, around 70% of new business applications are predicted to use low-code or no-code technologies.

    Security Testing: Due to the rising incidence of cybersecurity threats and data breaches, security testing has become a crucial aspect of software quality assurance. In the year 2023, the tech world anticipates that there will be a stronger focus on security testing, with the emergence of more advanced tools and methodologies aimed at detecting and resolving security weaknesses in software applications.

  • VMware Streamlines IT Management via Cloud Foundation Update

    VMware Streamlines IT Management via Cloud Foundation Update

    VMware this week updated its Cloud Foundation platform and rolled up recent updates to the company’s virtualization portfolio all at once.

    Version 5.0 of VMware Cloud Foundation (VCF) encompasses the latest versions of VMware vSphere 8.0u1 for workload management, VMware vSAN 8.0u1 for scale-out storage, VMware NSX 4.1 for networking and VMware vRealize LifeCycle Manager 8.10 (Aria) for cloud management.

    In addition, VMware has added a single sign-on (SSO) capability, dubbed Isolated SSO Workload Domains, that enables administrators to configure new workload domains using SSO instances from different providers or platforms via NSX.

    Rick Walsworth, director of product marketing for the Cloud Infrastructure Business Group at VMware, said that as the management of IT continues to evolve in VMware environments, more organizations are opting to update their entire VMware software stack simultaneously rather than attempting to update components individually.

    The primary reason for that shift is that the company ensures that all the latest VCF component releases have been thoroughly stress-tested from an integration perspective, he noted.

    During uncertain economic times, VMware is seeing an increased focus on simplifying IT environments as IT teams are asked to do more with less, said Walsworth. As such, more organizations are looking to standardize on a stack of infrastructure software that can be managed more consistently, he added.

    More organizations are also looking to reduce the total cost of IT by employing the same stack of infrastructure software across hybrid cloud computing environments where workloads are deployed both in on-premises IT environments and public cloud, said Walsworth.

    The company is also seeing increased adoption of DevOps best practices to programmatically manage IT environments that need to respond to rapidly changing requirements, he noted. IT teams are looking to work more collaboratively rather than relying solely on specialists to manage specific functions, said Walsworth.

    Some of that shift is also being driven by Tanzu, a distribution of Kubernetes that the company makes available as part of VCF, he noted. Most application developers don’t have a preference for a specific Kubernetes distribution, so many organizations view Tanzu as an opportunity to extend their existing management framework to a de facto standard for building the next generation of cloud-native applications, said Walsworth.

    VMware is in the midst of being acquired by Broadcom. Nevertheless, VMware this week reported revenues for the first quarter 2023 of $3.28 billion, a 6% year-over-year increase. Subscription and software-as-a-service (SaaS) revenue for the first quarter was $1.22 billion, an increase of 35% year-over-year. That increase in revenue suggests any concerns about the future direction of the company are not impacting the willingness of IT organizations to continue to rely on VMware.

    It may be a while before all the dust settles around Broadcom’s acquisition of VMware. In the meantime, however, it’s clear that the way IT is managed in the cloud era is continuing to rapidly evolve regardless of who owns VMware.

  • CDF Marries Emporous Repository to Ortelius Management Platform

    CDF Marries Emporous Repository to Ortelius Management Platform

    The Continuous Delivery Foundation (CDF) has added the open source Emporous repository for software artifacts to an Ortelius platform for managing the software artifacts that make up a microservice.

    Ortelius was originally created by DeployHub and subsequently contributed to the CDF, which also oversees the development of the open source Jenkins continuous integration/continuous delivery (CI/CD) platform.

    Emporous is based on a Universal Object Reference (UOR) conceptual model that enables dynamic schema registration and decentralized content-addressable reference of a wide spectrum of software artifacts. Originally developed by Red Hat, Emporous will now be managed as a sub-project within the larger Ortelius incubation-level project by the CDF.

    DeployHub CTO Steve Taylor said this integration would make it simpler to store, organize and search metadata related to software artifacts, along with the artifacts themselves, within the context of the Ortelius platform for managing microservices.

    Ortelius provides insights into the relationships and dependencies between different software components and applications. It does this by collecting and aggregating pipeline data from multiple sources within a software supply chain. Emporous will be incorporated within Ortelius to enhance metadata search capabilities and provide a single repository to store any type of artifact, from containers to jar files. It will now also be able to store metadata and also the artifacts themselves within the same Emporous catalog.

    Those capabilities will significantly improve security and compliance reporting across a software supply chain, added Taylor. DevOps and cybersecurity teams will be able to automatically apply and enforce supply chain policies and procedures, he said.

    DevOps teams need a way to aggregate all the data from the various tools they use and the data must be easily searchable. Armed with those insights, it should become simpler to surface insights into, for example, which components of an application contain known vulnerabilities.

    In the longer term, IT organizations will be able to apply machine learning algorithms and other forms of artificial intelligence (AI) to automatically surface insights that will enable DevOps teams to automate workflows, said Taylor.

    Microservices-based applications are considerably harder to build and manage than monolithic applications, but they are also easier to maintain because individual microservices can be ripped and replaced.

    Most microservices today are built using containers, but there are also microservices that are written in Java, for example. In some cases, developers view monolithic applications as being a large, less flexible microservice that they invoke via an application programming interface (API).

    There’s no doubt that the rise of microservices made application environments a lot more complex. The tools needed to simplify the management of the next generation of applications are finally starting to become more widely accessible. In theory, the number of organizations that can successfully build and deploy these types of applications should steadily increase in the months and years ahead.

    In the meantime, there’s clearly still plenty of work that needs to be done to improve the overall state of software delivery. A recent CDF survey found only 22% of respondents were using continuous integration and continuous delivery (CD) to automate the building, testing and deployment of code in a production environment. As CD becomes more distinct from CI, however, the number of organizations that can achieve that goal when deploying microservices-based applications should steadily increase.

  • GitLab Adds More AI and Cybersecurity Capabilities to CI/CD Platform

    GitLab Adds More AI and Cybersecurity Capabilities to CI/CD Platform

    GitLab this week delivered an update to its continuous integration/continuous delivery (CI/CD) platform that adds additional generative artificial intelligence (AI) and cybersecurity capabilities.

    The GitLab 16 release included cybersecurity capabilities such as centralized policy management, expanded compliance reports and controls, compliance dashboards and default Level 3 attestation for the supply-chain levels for software artifacts (SLSA) framework.

    At the same time, GitLab this week also made available an emergency patch for two modules that have a path traversal flaw, tracked as CVE-2023-2825, that could allow unauthenticated users to read arbitrary files on the server under certain circumstances.

    In terms of generative AI, GitLab 16 added a value stream forecasting capability that automatically generates workflow summaries for senior managers. GitLab is also adding a code refactoring capability and the ability to automatically resolve vulnerabilities to an update to GitLab 16. Previously, GitLab made available a ‘suggested reviewers’ features, an ‘explain this code’ and an ‘explain this vulnerability’ capability that are all enabled by large language models (LLMs) developed by Google. GitLab has also partnered with Oracle to provide additional AI capabilities enabled by machine learning algorithms. In 2021, GitLab acquired UnReview, a provider of a tool that uses AI to identify which expert code reviewers to assign to a project based on the quality of their previous efforts and current workloads.

    David DeSanto, chief product officer at GitLab, said with 10 AI capabilities GitLab is providing more of these types of capabilities than any other DevOps platform provider. In addition, those capabilities are distributed across the entire software development life cycle rather than being focused solely on making developers more productive, he noted.

    That’s critical, because if AI capabilities are only focused on developer productivity, an imbalance is created; DevOps workflows will not be able to absorb the increased amount of code that will soon be moving simultaneously through pipelines, DeSanto added.

    AI will also further fuel adoption of platform engineering as a means for centralizing DevOps management. It’s becoming more apparent that infusing these types of capabilities into DevOps platforms built and maintained by an internal IT team is going to be a major challenge, he added.

    There is no doubt that generative AI capabilities will soon become widely integrated across DevOps workflows. That will enable developers to build more secure code faster while at the same time making it simpler for software engineers to maintain large codebases. Just about every job function imaginable will be impacted to varying degrees. In the case of DevOps teams, the ultimate impact should involve less drudgery as many of the manual tasks that conspire to make managing DevOps workflows tedious are eliminated. In fact, AI may ultimately make DevOps, as a methodology for managing IT, more accessible to a much wider range of organizations.

    As AI advances, there should still be plenty of demand for human expertise. For better or worse, the nature of the expertise required, however, will undoubtedly be at a much higher level than it has previously been as more low-level tasks are automated. The challenge is determining how and where to refocus the efforts of the DevOps teams to add higher value.

  • How to Choose an Observability Data Pipeline

    How to Choose an Observability Data Pipeline

    A complete full-stack observability platform combines data from multiple sources to gain holistic insights into system behavior. In fact, organizations without a full observability solution could find themselves facing departmental silos since teams with different data sources cannot share insights.

    While a key step in observability is the ability to combine logging data with metrics, companies that have developers viewing logging data and their metrics elsewhere are not collaborating on their data in the way observability requires. The solution is an observability data pipeline that best fits your business needs.

    What is an Observability Data Pipeline?

    An observability data pipeline does two things—centralizes data from a number of different sources and provides teams with the right tools to turn data into business insights. Observability data pipelines are typically user-friendly so that you can easily set up and manage your data.

    Observability data pipelines integrate data from a range of sources, including AWS, FluentD and Kubernetes. They have capabilities for searching and viewing log data and notifying engineers through email and even Slack.

    An observability pipeline ingests logs, so they can be viewed in a log viewer. Pipelines can also analyze data in real-time, allowing engineers to quickly spot critical security issues before they affect critical systems.

    There are a number of full-stack observability platforms, including Coralogix, Splunk, Elastic and even options like Azure Log Analytics and Google Cloud Logging. Coralogix, for example, is a popular observability data pipeline that can analyze data in real-time using Streama and use Kafka Connect to integrate data from a variety of sources and extract trends.

    Business Benefits of Observability Data Pipelines

    Using observability data pipelines has the potential to transform the way a business does DevOps. Let’s go through a few key benefits of this technology:

    ● Reducing Cost of Observability Data
    Many organizations use systems like Cloudwatch to analyze logging data. However, these often charge per GB of logging data analyzed, raising the question of whether Cloudwatch is really cost-efficient. And it’s not just Cloudwatch. For many systems, logging costs can be increased due to slow query times, noisy logs and the wrong type of storage. Observability data pipelines help organizations manage high volumes of logging data and move away from expensive legacy platforms. This enables enterprises to increase their observability for a modest price.
    ● Reduced Onboarding, Increased Problem-Solving
    Pipelines are straightforward to learn. Unlike options such as ELK, which can be quite specialized and difficult to maintain, observability data pipelines are easy to use and easy to set up. This reduces the time needed to onboard new engineers. Less time onboarding means more time solving problems and serving clients, which increases business revenue.
    ● Better Data Shaping and Merging
    Data from different sources is often stored in multiple formats and structures. Because observability depends so much on well-integrated data, it’s vital to transform this data into standardized formats. Observability data pipelines use log parsing to store key features of logging data for search and analysis.
    ● Reducing MTTD/MTTR
    Having an optimized, user-friendly observability data pipeline reduces downtime by helping engineers solve problems faster.
    ● Enabling DevSecOps With Observability
    We’ve written about the rising prominence of DevSecOps, a paradigm that involves shifting security left in the development life cycle. Key to DevSecOps is the fruitful collaboration between DevOps and Security. This can only be achieved by high levels of observability. Observability data pipelines facilitate this with their ability to integrate data from multiple different sources and present it intuitively. This means security teams can apply their expertise to log analytics that would otherwise have remained siloed.

    Wrapping Up

    Observability, the ability to use logs, traces and metrics to “picture” the internal state of a system, is vital to any DevOps team. Observability data pipelines can help by integrating and presenting data from different sources. This mitigates the effects of organizational silos and allows DevOps teams to solve problems faster.

  • Red Hat Unveils Developer Hub to Boost Dev Productivity

    Red Hat Unveils Developer Hub to Boost Dev Productivity

    At its Red Hat Summit conference today, Red Hat unveiled its Red Hat Developer Hub, a developer portal that provides access to an opinionated set of development tools based on the open source Backstage project.

    The portal pulls together a set of curated tools and resources so that developers encounter less friction and avoid the complexity of numerous optional tools that span disparate vendors. The goal is to increase developer productivity while reducing the volume of optional tools and information that must be sifted through to actually deliver production code.

    Red Hat Developer Hub aims to streamline the development process and allow organizations to maximize existing resources and developer skills while reducing sprawl, said Chris Wright, CTO and SVP for global engineering at Red Hat. This will help organizations maintain application development velocity and drive a consistent user experience for developers based on tools and skills they’re already familiar with, Wright told attendees.

    Rather than building and maintaining their own internal developer portal, Red Hat is making a case for using Developer Hub to deliver a platform approach to addressing the complexity and sprawl DevOps teams face. Red Hat Developer Hub is, in essence, an opinionated and optimized version of Backstage that aims to deliver improved developer experiences across environments, including Kubernetes and container application platforms such as Red Hat OpenShift.

    Developer Hub will offer a single pane of glass to view all available developer tools and resources, developer self-service capabilities and security and compliance guardrails for cloud-native application development that comply with enterprise-wide best practices, according to the company.

    Additionally, Red Hat Developer Hub includes a set of ‘golden path’ templates—pre-defined, pre-architected and supported templates for further accelerating application development.

    The developer hub is based on the Cloud Native Computing Foundation (CNCF) incubating open source project Backstage, originally developed by Spotify and donated to the CNCF in 2020. Backstage is now being advanced by contributions from multiple vendors, including Red Hat.

    Backstage has been gaining traction in cloud-native application environments that are especially challenging to develop using multiple tools. There’s a lot more focus on developer productivity, which Backstage addresses by creating a catalog of blueprints that developers can readily consume rather than requiring them to build these capabilities themselves multiple times over. The goal is to create scaffolds that developers can consistently reuse across multiple application development projects.

    Red Hat has also developed Red Hat Plug-ins for Backstage, a bundle of six plug-ins that integrate various key systems into Backstage, helping to extend the project’s functionality, ensure a consistent user experience and improve workflow efficiency. The plugins can be used across all Backstage installations, providing even more flexibility for developers across platforms and Kubernetes distributions and include:

    • Application Topology for Kubernetes, which enables developers to visualize real-time status of application and infrastructure workloads deployed to any Kubernetes target, including Red Hat OpenShift, with greater consistency.
    • Multi-Cluster View with Open Cluster Manager (OCM), which provides a view into clusters from Open Cluster Manager’s MultiClusterHub and MultiCluster Engine in Backstage.
    • Container Image Registry for Quay, which improves the integration and speed of interactions with Quay registries by providing a view into container image details. This includes security vulnerabilities (CVEs) associated with deployed images.
    • Pipelines with Tekton, which gives users a view into the details of all Tekton pipeline runs and their respective statuses across all services.
    • Authentication and Authorization with Keycloak, which enables platform engineers to load users and groups from Keycloak into Backstage, allowing the use of multiple authentication providers to be applied to Backstage entities.
    • GitOps with Argo CD, which helps track the health and status of the Argo CD status for services within Backstage.

    Red Hat said additional plugins are in the works. Red Hat Developer Hub will be generally available later this year, but Red Hat is offering a ‘test drive’ through the upstream IDP project Janus.

     

  • Red Hat Adds Generative AI and Event-Driven Capabilities to Ansible

    Red Hat Adds Generative AI and Event-Driven Capabilities to Ansible

    Red Hat today unveiled a technical preview of Ansible Lightspeed with IBM Watson Code Assistant, a generative artificial intelligence (AI) service specifically trained on domain-specific data collected via the Ansible automation platform.

    At the same time, Red Hat is also making available an event-driven capability within version 2.4 of the Ansible platform that enables runbooks and other tasks to automatically execute when received from a third-party tool or application.

    Both capabilities were announced at the Red Hat Summit and are part of an ongoing Red Hat effort to democratize IT automation.

    First previewed within the Project Wisdom initiative launched by Red Hat and IBM Research last year, Ansible Lightspeed IBM Watson Code Assistant is based on a large language model (LLM) that was developed using Ansible data. Watson Code Assistant, currently in technical preview, is a service that automatically generates suggestions to improve code as developers create an application. IBM promised to make IBM Watson Code Assistant available later this year.

    Red Hat is now applying that generative AI capability to the YAML code that DevOps teams need to create Ansible playbooks. IT teams can create a simple natural language prompt that will translate their domain expertise into YAML code for creating or editing Ansible Playbooks. Red Hat and IBM, however, will not automatically use the code from an organization to fine-tune their AI model without express consent.

    Rich Henshall, director of product management for Red Hat Ansible, said as IT environments become more complex, it’s clear many are hitting a ceiling in terms of what can be automated using legacy custom scripts. Ansible Lightspeed IBM Watson Code Assistant, along with an event-driven capability, will make it much simpler to automate the management of IT environments at scale using a declarative framework, he added.

    Ansible, for example, can now receive events from third-party tools, determine next steps and consistently initiate an automated response using Ansible playbooks. Ansible can also consume events from third-party monitoring, observability and IT analytics tools from providers such as Cisco ThousandEyes, CyberArk, Dynatrace, F5, IBM Instana, IBM Turbonomic, Palo Alto Networks and Zabbix with more to come, noted Henshall. Red Hat is also making available supplementary content for Red Hat OpenShift, Red Hat Insights, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform and ServiceNow, in addition to enabling IT teams to create their own custom integrations.

    At this juncture, it’s not clear why more IT processes have not been automated using Ansible or other frameworks. Many DevOps teams continue to rely on custom scripts that typically don’t scale and are generally poorly documented. Any time the individual who wrote those scripts leaves an organization, it’s incredibly difficult—if not impossible—for the original scripts to be maintained, noted Henshall.

    One way or another, it’s clear that a more scalable approach to IT automation is required, especially with the rise of more complex cloud-native application environments. It’s no longer economically viable for most organizations to rely on legacy processes that require a small army of IT professionals to maintain.

  • 8 Hot Takes: Will We See a Monolithic Renaissance?

    8 Hot Takes: Will We See a Monolithic Renaissance?

    Is the DevOps world slowly backing away from microservices and to a monolithic renaissance? After years of proselytization about the benefits (Flexibility! Resilience! Support faster release speeds!) of microservices and serverless architecture, there are signs of a potential backlash against the singularly fragmented microservices architectural approach.

    The argument in favor of rolling back to monolithic architectures feels similar to the cloud repatriation debate. Just as with a lot of cloud-first transformations—often in lockstep with them, actually—many organizations moved headlong toward microservices without a case-by-case business or technical rationalization for the change. As the shiny newness of microservices has dulled under that approach’s daily exposure to real-world deployment and business challenges, the monolith isn’t looking so bad in a lot of situations.

    The monolithic versus microservices debate boiled over the last few weeks in the wake of the publication of a case study earlier this month from the Amazon Prime Video team. The analysis from Marcin Kolny, senior software development engineer for Prime Video, reported that his team was experiencing scaling bottlenecks in its audio/video quality inspection tooling, which was architected as a distributed system using serverless components. By rearchitecting the tool as a monolith, they were not only able to increase scaling capabilities but drastically reduced the infrastructure costs around that tool.

    “Moving our service to a monolith reduced our infrastructure cost by over 90%,” Kolny wrote.

    The announcement elicited a fair degree of hyperbole on the topic, but it also spurred a very organic airing of the pros and cons of monolithic, microservices and serverless architectures.

    The following are 10 different perspectives spurred on by this online debate from a range of CTO and engineering experts. Some of them are diametrically opposed to one another, many are overlapping in their opinions. But as a body, they point to the general (and very pragmatic) consensus that there simply will never be a one-size-fits-all approach to software architectures.

    Microservices a Siren Song for Complication

    “It’s clear that in practice, microservices pose perhaps the biggest siren song for needlessly complicating your system. And serverless only makes it worse. Replacing method calls and module separations with network invocations and service partitioning within a single, coherent team and application is madness in almost all cases.”

    —David Heinemeier Hansson, creator of Ruby on Rails and CTO at 37signals (maker of Basecamp and HEY)

    Architectures Aren’t Good or Bad

    “Application architectures are neither good nor bad, nor are they appropriate for every use case. Organizations need to step back and consider carefully what it is they’re trying to achieve and which architecture might best serve that need instead of choosing the most “modern” architecture because, well, it’s in vogue.”

    —Lori MacVittie, distinguished engineer in office of CTO at F5Networks

    Distributed Architectures Will Suit AI-Driven Future

    “In the ongoing debate between monolithic and serverless architectures, it is important to acknowledge that both approaches have their merits and considerations. Monolithic architectures have been prevalent in large human-centric organizations due to their management advantages and historical use cases. However, as the world continues its rapid progression towards a more AI-driven future, the landscape is shifting towards architectures that embrace fluidity, autonomy, and distribution.”

    — Reuven Cohen, veteran CTO and entrepreneur, mentor for TechStars

    Monolithic and Serverless Each Have Pros And Cons

    “Over the last 15 years, everyone has wanted to move to serverless microservices-based horizontally-scaled architecture – but it is not ideal for every application’s specific needs.  As with any choice in life and in business, there are always tradeoffs;  you must evaluate and choose the appropriate architecture based on your business and technology needs. Another important dimension to this is workforce skills for application architecture.

    —Hemang Davé, client technical leader and CTO at Kyndryl

    The Amazon Rearchitect Wasn’t Even Truly a Monolith

    “The problem is that they called this refactoring a microservice to monolith transition when it’s clearly a microservice refactoring step, and it is exactly what I recommend people do in my talks about serverless first. I don’t advocate “serverless only”, and I recommended that if you need sustained high traffic, low latency and higher efficiency, then you should re-implement your rapid prototype as a continuously running autoscaled container as part of a larger serverless event-driven architecture, which is what they did. If you built it as a microservice to start with, it would probably take longer (especially as you have to make lots of decisions about how to build and run it) and be less able to iterate as you figure out exactly what you are trying to build.”

    —Adrian Cockcroft, longtime DevOps advocate and veteran architect with stints at eBay, Netflix, Amazon, and Battery Ventures, among others

    One Tool as a Monolith, Maybe, But Not All of Amazon

    “Microservices, while duplicative and inefficient from a compute standpoint, allow you to release stuff really quickly, which is a tradeoff that most organizations at scale are willing to make.

    Microservices exist to reduce communication pathways—not to solve technical scaling challenges. At a certain point, teams stop functioning due to the communication overhead. A seven-person team has 21 communication pathways, whereas a 100-person team has 4,950. I work with large retailers selling billions of dollars a year online, and putting everything into a single monolith just doesn’t work. Could the entirety of Amazon.com be deployed as a single monolith? No way.”

    —Kelly Goetsch, chief strategy officer at commercetools

    Architectural Tension Always Exists Between Speed and Maintainability

    “Speed to market and speed to develop does not always equate to long-term scalability and maintainability. You must actively balance your investments across these two critical pillars to build a viable product. Active rebalancing and reconsideration of the stack and technology mix for your FinOps and BizOps context is both art and science.

    Cloud-native distributed systems paradigms are here to stay. The Prime Video folks have mistakenly labeled a properly factored, highly coupled data-intensive processing step a monolith—it’s at best a rocky outcropping in their distributed microservices forest. “

    —Ajish George, managing director of cyber data & analytics for State Street

    The Shiny Thing is Not Always the Best Thing

    “The news that AWS discovered that serverless and microservices aren’t for everything is surprising. Not because it didn’t work out for that use case; but because the wrong use cases have been so well-known for the serverless/microservices pattern and should have been obvious to Prime Video. It’s worth remembering: That shiny new thing isn’t necessarily a better way of trapping mice. Right tool for the right job.”

    —Doug Vanderweide, director of cloud architecture for Avanade