devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

  • The Power of Observability: Performance and Reliability

    The Power of Observability: Performance and Reliability

    Observability has become an essential aspect of modern IT, and its importance is only expected to increase in the years to come. The ability to monitor and understand the behavior of systems in real-time provides organizations with a wealth of information that can help them improve the performance, reliability, and overall health of their networks and applications.

    By collecting and analyzing data from a variety of sources, observability enables organizations to proactively address issues before they escalate into major problems. This proactive approach to problem-solving can help businesses avoid costly downtime and lost productivity, ultimately leading to increased customer satisfaction and business growth.

    Current Challenges Facing Today’s Organizations

    Despite its many benefits, achieving proper observability can be challenging, particularly in today’s complex and highly dispersed technology landscape. The sheer volume of data generated by siloed networks and applications can overwhelm even the most sophisticated monitoring tools. However, with the right approach, tools and processes, organizations can overcome these challenges and fully realize the potential that observability has to offer.

    One of the biggest obstacles to achieving comprehensive network observability is the fragmentation of data sources. Logs, metrics, events and other types of data all require different tools to collect and store them, making it difficult to get a complete picture of network performance. To overcome this challenge, organizations must adopt a holistic approach that integrates the right tools and processes to collect and analyze all relevant data.

    This holistic approach involves the use of advanced monitoring tools that can collect and analyze data from multiple sources, as well as the implementation of best practices for data management and analysis. By taking a comprehensive approach to observability, organizations can gain a deeper understanding of their networks and applications, identify potential issues before they become major problems, and ultimately drive business growth and success.

    Best Practices for Performance and Reliability

    First, proper tooling is crucial for organizations to perform a comprehensive analysis of data. This enables cross-domain correlation to achieve meaningful insights. A data-centric approach relying on contextual information can help organizations “connect the dots” and make informed decisions. In addition, organizations must also prioritize data quality to ensure that they are analyzing accurate and reliable data. Poor data quality can lead to incorrect conclusions and poor decision-making, which can have negative impacts on network performance and business outcomes. To ensure data quality, organizations must invest in proper tooling and processes that can validate and verify data accuracy, consistency and completeness.

    Second, having a network data source of truth is crucial for organizations to effectively analyze and make informed decisions based on their data. The ability to extract metadata from existing sources to provide contextual information is also vital to “connecting the dots” across multiple domains to uncover valuable insights. The metadata serves as the ultimate enabler in this process, allowing organizations to gain a comprehensive understanding of their data to make informed decisions. By establishing a network data source of truth and extracting the necessary metadata, organizations can effectively analyze and use their data for more meaningful insights.

    Third, it is important to ensure that observability is incorporated into the entire development lifecycle to support end-to-end visibility. This means accounting for instrumentation as part of the design and planning phases, as well as testing, deployment and ongoing maintenance. By making observability a central part of the development process, organizations can identify and resolve issues before they become major problems. Additionally, incorporating observability into the development lifecycle can also improve the overall performance of the system by identifying bottlenecks and areas for optimization. It can also help to improve the overall security of the system by detecting anomalous behavior and potential vulnerabilities. Moreover, having observability throughout the development process can enable teams to make informed decisions based on real-time data and analytics, leading to more efficient and effective development practices.

    The fourth best practice is to use a centralized observability platform that allows for the collection and analysis of data from all systems and infrastructure. This platform should be able to collect data from all parts of the organization’s technology stack, including databases, servers and applications. Additionally, the platform should be able to correlate data from multiple sources to provide a comprehensive view of the organization’s operations. This centralized approach ensures that all stakeholders have access to the same information, reducing the likelihood of miscommunications and ensuring that everyone is working towards the same goals.

    Last is the democratization of data. This is crucial for organizations as it empowers teams from multiple functions to access and use data effectively. This leads to increased transparency and collaboration, allowing organizations to make data-driven decisions and drive innovation. Democratization also helps to break down silos, increase visibility and facilitate faster problem-solving for improved decision-making. The key to achieving comprehensive network observability is collaboration between teams. Siloed teams and processes can lead to gaps in data collection and analysis, as well as delays in issue resolution. Therefore, organizations must encourage cross-functional collaboration between teams responsible for network infrastructure, application development and operations to promote a shared understanding of network performance and enable faster problem resolution.

    Observability is a Key Discipline

    In conclusion, observability is a key discipline that can help organizations achieve superior performance and reliability in their systems. By following the best practices suggested above and embracing a comprehensive approach, organizations can overcome the challenges of data fragmentation and unlock the full potential of observability. As technology continues to evolve and become more complex, observability will only become more important, making it essential for businesses to prioritize this aspect of their IT strategy.

  • How to Supercharge Your Engineering Teams

    How to Supercharge Your Engineering Teams

    With the economy still bleak, data and engineering teams that were already struggling to keep pace with business needs are now facing a potential hiring freeze or worse in 2023. That’s why there’s never been a more important time to maximize the productivity and impact of your teams, but the question is how to identify the big changes that move the needle in performance. Incremental adjustments that save a sliver of time here and there are not hard to come by, but what teams need are the breakthroughs that yield giant leaps in productivity and efficiency. Rather than a 10% improvement, it’s the 10x improvements that drive lasting gains.

    As an engineer and CEO of a software company, I’m constantly looking for ways to boost the productivity of our teams. Our guiding principle is simple: Maximize the impact per hour invested for every individual. To do that, we must ensure that individuals maximize the time spent flexing their superpowers, minimize busy work and reduce dependencies on others that delay work or tie them up in meetings. While counterintuitive at first, we consistently find that collections of well-aligned, independently executing individuals produce a far greater impact than a large team swarming on a project. Put another way; the individual impact goes up when the number of people you depend on to deliver the result goes down.

    I’ve developed an approach over the years that has allowed me to identify several opportunities for step-change advancements in productivity and efficiency, and I’d like to share it here, along with two concrete examples. I will also share an exercise we run with our teams to help identify and execute these opportunities.

    First, here are my two main criteria for identifying the projects and tasks where we can effect changes that will have an outsized impact on the company:

    ● Big resource-consumers: Work that’s essential to the business but consumes a large amount of engineering and other resources is a top candidate for optimization.
    ● Big impact, easy win: Areas that will have a significant impact on company performance if we can do them faster and with higher frequency are also great candidates—even if they’re not consuming a ton of resources.

    Here are two examples that illustrate each of those criteria and where we were able to see massive gains in productivity, quality of service, or both.

    Tackling Big Resource-Consumers

    At Ascend.io, we provide a platform for building and automating data pipelines, which requires us to build connectors into a huge variety of data sources. At one time, building a new connector required multiple engineers to spend two to three weeks writing more than 5,000 lines of code that spanned four microservices in three different languages. As we added more customers and needed to support more data sources, this quickly became unsustainable.

    When we looked at this process, we saw two issues: Engineers were writing custom code for each connector in a way that would be better served by a common framework. Moreover, the process required engineers from multiple teams and disciplines to work together and coordinate their priorities. As soon as you get dependencies that require engineers to coordinate with others and wait on the outcomes of other people’s work, you’re wasting time.

    We addressed this by designing a new development framework that exploits commonalities across the connectors we build. This now requires far less unique code and removes the need for multiple engineers to work together. In most cases, one person can build a new connector in days with 100 lines of code. This made us far more nimble and freed up engineering time to do other valuable work.

    Achieving High Impact

    Like most companies building software, we need to convey the capabilities of our product to new and existing users to ensure they receive maximum benefit from their use of Ascend. With new features and enhancements being released on a weekly basis, there is a lot to keep up with. This is high-impact work, however, and if we can do it in a vastly better way, it drives an outsized impact for Ascend and our customers.

    For us, building these automated product tours typically required a product marketer or product manager to design the customer journey, and one or more engineers on a sustained basis to build the actions into the product. This again creates dependencies that slow progress and require significant manual coding.

    The breakthrough for us was a tool called Intercom, which allows you to build sophisticated product tours without needing to write code. Our product team can now build more interesting and complex product tours for customers, and our engineers can stay focused on building new features instead of the code needed to showcase them. What’s more, they are able to design, test and deploy these tours independent of our product and code deployment cycle, further freeing them to iterate as fast as possible.

    A Thought Experiment

    So how do you actually find these opportunities for 10x improvement? I run the following thought experiment with our teams several times a year to uncover these opportunities:

    Imagine your workload scales 10x overnight and you absolutely must meet this increased need, but without hiring more engineers or working more hours. How do you do it?

    It’s amazing what this exercise can uncover—but there are ground rules and it can take time to see results. Coming up with these big ideas is a skill like any other, and most people are not great at it the first time they try it. Your first iteration may produce no options to pursue—some may have high-impact ideas but with even higher investment requirements, while others may reject the premise outright. That’s normal. So, don’t insist they sit in a room until they dream up that 10x breakthrough, because it will be demoralizing. Instead, plant the seeds by establishing the above framework and then let the ideas start to grow.

    More likely, a few days later, they’ll be out for a hike or chatting with an old co-worker and they’ll suddenly have the germ of an idea. That may still not be the winning solution, but someone else will build on it, and then someone else and, finally, you iterate your way to a breakthrough. Pulling the team back together and embracing the “yes, and” approach can be both energizing and highly productive.

    Brainstorming like this is fast and free, and you only need one great idea to have a material impact. We run this exercise a few times a year and have come up with several big ideas that were game-changers for the company. Besides the examples above, we also moved key parts of our software development from Scala and Go to Python a few years ago, which is a much more efficient language (as measured by developer time) and allows our engineers to build things much faster.

    Software engineering is a complex beast that often spans many teams working on multiple processes in parallel. But that complexity is also your opportunity: Every complex system has room for optimization, especially at companies that moved fast in the beginning to build a product and find market fit. Now is the time to lift your gaze, look around and ask where significant gains can be made. They almost certainly exist, and a 10x improvement could be what allows you to not only survive the turbulent year ahead but emerge with a thriving team and business on the other side.

  • Tabnine Extends Generative AI Alliance With Google

    Tabnine Extends Generative AI Alliance With Google

    Tabnine has extended its alliance with Google Cloud to advance the adoption of generative artificial intelligence (AI) to automate the writing and testing of code.

    The generative AI platform provider has already developed its own large language model that is hosted on Google Cloud. Tabnine is now also committing to leveraging the large language model that Google is developing to automate software development life cycle processes.

    Brandon Jung, vice president of ecosystems for Tabnine, said it’s apparent that application development and deployment teams will be making use of multiple large language models across SDLC workflows. Not all those large language models will be developed by one single platform provider, he noted.

    Rather than being dogmatic about large language models, Tabnine plans to make it possible to invoke multiple models via application programming interfaces (APIs) that are integrated with development environments, he added.

    The Tabnine platform supports multiple programming languages, including Python, Java and JavaScript and is designed to be integrated with integrated development environments (IDEs) such as Visual Studio Code and Jetbrains. The company also previously integrated its code completion tool with the GitLab continuous integration/continuous delivery (CI/CD) platform. The overall goal is to make it easier for developers to automatically write code based on custom models using approved source code hosted in a secure private repository.

    Generative AI creates a large language model that assesses the probability of what the next line of code will be based on what has preceded it. It’s not likely DevOps teams will be replaced as generative AI is extended across workflows, but the overall size of those teams might shrink as it becomes possible to do more with fewer people. At the same time, the barrier to DevOps adoption will also fall as AI platforms make it simpler for more organizations to embrace DevOps best practices.

    The challenge will be making sure the data collected to train AI models is of a high enough quality to ensure the desired outcome. Organizations will need to find a way to apply generative AI to data models that have been validated by DevOps teams.

    One way or another, it’s only a matter of time before generative AI capabilities are applied more broadly. Platforms such as OpenAI’s ChatGPT are only the tip of an iceberg that impacts almost every manual process, including software development and deployment. The issue will be determining how quickly those innovations will become practical enough to employ.

    In the meantime, it is already apparent generative AI platforms are having a significant impact on the rate at which code can be developed. Inevitably, that means the amount of code moving through DevOps pipelines at any one time should increase significantly. DevOps teams should expect generative AI technologies to be applied both before and after application code is built and deployed, noted Jung.

    In the meantime, the biggest issue is, arguably, keeping up with a rate of generative AI innovation that is occurring faster than many organizations can absorb.

  • Software Supply Chain Risk Management: A 2023 Guide

    Software Supply Chain Risk Management: A 2023 Guide

    Software supply chain risk management (SSCRM) refers to the process of identifying, assessing and mitigating risks associated with third-party software components and services that are integrated into software products. SSCRM involves understanding the potential vulnerabilities that may arise from these components and taking measures to reduce the risk of exploitation or compromise to the software system or end-users.

    Why Software Supply Chain Attacks are Becoming More Common

    Software supply chain attacks have become common due to several factors: 

    • Larger attack surface: The increased use of third-party software components and services in software development has expanded the attack surface for cybercriminals. This means that attackers can exploit vulnerabilities in these third-party components to gain access to the larger software system. 
    • Rise of open source: The proliferation of open source software has also contributed to the rise of supply chain attacks as attackers can inject malicious code into open source repositories, which can then be used by developers in their software products.
    • Difficulty to detect: Detecting supply chain attacks can be challenging since attackers often use legitimate-looking code that may be difficult to distinguish from normal code. This can make it difficult for traditional security measures to detect the attack. Additionally, the attack may not manifest itself until much later, making it even harder to trace back to its source.
    • Automation: Supply chain attacks are also becoming more sophisticated through the use of automation. Attackers can use automated tools to scan for vulnerabilities in third-party components and inject malicious code into them. This means that they can quickly and efficiently compromise a large number of software products.
    • Evasion techniques: Attackers can evade traditional security measures through techniques such as code obfuscation and encryption. This makes it difficult for security measures to detect malicious code, which can easily slip through undetected.
    • Sophistication: Supply chain attacks are becoming more challenging to combat due to the use of advanced malware techniques to change shape. Attackers can use polymorphic malware that can change its code and behavior to avoid detection. This means that even if the malware is detected, it can quickly change its form to evade detection in the future.

    Supply Chain Security and DevSecOps

    DevSecOps is an approach that integrates security into all stages of the software development process. Adopting DevSecOps practices can help businesses manage supply chain risks by building security into their software development and delivery pipelines. 

    Continuous integration/continuous deployment (CI/CD) pipelines are a key component of DevSecOps, as they allow developers to rapidly test and deploy code changes while ensuring that security and compliance requirements are met.

    Here are some ways DevOps tools in the CI/CD pipeline can help manage supply chain risks:

    • Automating security testing: DevSecOps enables businesses to automate security testing throughout the software development life cycle, including during the development, testing and deployment phases. This can help identify and mitigate security vulnerabilities early in the process, reducing the likelihood of supply chain risks.
    • Continuous monitoring: DevSecOps enables continuous monitoring of software and infrastructure, providing real-time visibility into the security posture of the supply chain. This can help identify and respond to security threats quickly and effectively.
    • Secure coding practices: DevSecOps encourages the use of secure coding practices, such as input validation, access control and error handling, reducing the likelihood of supply chain risks.
    • Risk management: DevSecOps incorporates risk management into the software development process, ensuring that supply chain risks are identified, assessed and managed effectively.

    Tools and Solutions to Help Manage Supply Chain Risks

    Supply Chain Risk Management (SCRM)

    SCRM solutions are software tools and services that help organizations manage and mitigate supply chain risks. These solutions provide visibility into the supply chain operations, identify potential risks and enable organizations to take proactive measures to mitigate these risks. 

    They may include features such as supplier risk assessments, real-time monitoring of supply chain operations, analytics and reporting and incident response planning. SCRM solutions are often available as a service (SaaS) and can help organizations to enhance supply chain resilience, reduce disruption and ensure business continuity.

    Software Composition Analysis (SCA)

    Software Composition Analysis (SCA) is a software security practice that involves identifying and managing open source and third-party components used in the development of software applications. SCA tools scan the software codebase, detect and report the presence of third-party and open-source components and identify known vulnerabilities in these components. 

    SCA helps organizations to identify and manage risks associated with third-party components, including license compliance, intellectual property infringement and security vulnerabilities. SCA is an essential component of SSCRM and can help organizations to ensure the security and integrity of the software products they develop and distribute.

    Third-Party and Supplier Risk Management

    Third-party and supplier risk management software is designed to help businesses manage the risks associated with their third-party vendors and suppliers. This software provides tools and resources to identify, assess and manage the risks associated with third-party relationships.

    Here are some features of third-party and supplier risk management software:

    • Supplier management: The software provides tools to manage supplier relationships, including supplier performance monitoring, supplier compliance and supplier risk assessment.
    • Risk assessment: The software enables businesses to assess the risks associated with their third-party relationships, including financial risks, operational risks and reputational risks.
    • Compliance management: The software helps businesses comply with regulations and standards, such as GDPR, HIPAA and ISO 9001.
    • Data analytics: The software uses data analytics tools to help businesses identify trends and patterns in third-party data, enabling them to anticipate and mitigate risks.
    • Incident management: The software provides tools to help businesses respond to and manage incidents related to third-party relationships, such as data breaches or compliance violations.

    Best Practices for Software Supply Chain Risk Management

    The following are some best practices for managing supply chain risks.

    Identifying Supply Chain Threats and Potential Weaknesses

    The first step in managing supply chain risks is to identify potential threats and vulnerabilities in the supply chain. This involves conducting a comprehensive risk assessment that considers all potential sources of risk, including geopolitical factors, natural disasters, cyber threats and other disruptions. The risk assessment should consider the entire supply chain, including all vendors, suppliers and partners.

    During the risk assessment, organizations should identify the critical components and services that are essential to their operations. They should also evaluate the potential impact of a disruption to these critical components and services, including the financial, reputational and operational impact.

    Establishing a Supply Chain Risk Management Framework

    After identifying potential threats and vulnerabilities, organizations should establish a supply chain risk management framework that outlines the policies, procedures and controls required to mitigate these risks. The risk management framework should be aligned with the organization’s overall risk management strategy and should include the following components:

    • Risk management policy: This policy should outline the organization’s approach to supply chain risk management and define the roles and responsibilities of key stakeholders.
    • Risk assessment procedures: These procedures should describe how the organization conducts risk assessments, including the criteria used to evaluate risks and the frequency of assessments.
    • Risk mitigation controls: These controls should outline the specific measures the organization will take to mitigate identified risks.
    • Incident response plan: This plan should describe the steps the organization will take in the event of a supply chain disruption.

    Continuously Monitoring Risks

    Once the supply chain risk management framework is in place, organizations should continuously monitor supply chain and third-party risks and take proactive measures to mitigate them. This includes establishing a patching cadence to ensure that all software components are up to date and protected against known vulnerabilities.

    Organizations should also implement endpoint security controls, such as antivirus software and intrusion detection systems, to protect against malware and other cyber threats. Network security controls, such as firewalls and access controls, should be implemented to protect against unauthorized access to systems and data.

    Web application security controls should also be implemented to protect against web-based attacks, such as SQL injection and cross-site scripting. This includes conducting regular vulnerability assessments and penetration testing of web applications.

    Promoting a Culture of Awareness

    Finally, organizations should promote a culture of awareness among all stakeholders, including employees, vendors, and suppliers. This includes providing risk awareness training to employees, so they understand the risks associated with the supply chain and the steps they can take to mitigate these risks.

    Organizations should also communicate their supply chain risk management policies clearly to vendors and suppliers so they understand their obligations and responsibilities in managing supply chain risks. This includes conducting due diligence on vendors and suppliers to ensure they meet security standards and policies.

    Conclusion

    Managing supply chain risks is a critical task for organizations that rely on third-party vendors and suppliers to deliver goods and services. These risks can arise from a range of sources, including geopolitical factors, natural disasters, cyber threats and other disruptions.

     To effectively manage these risks, organizations need to adopt best practices such as identifying threats and potential weaknesses, establishing a supply chain risk management framework, continuously monitoring risks and promoting a culture of awareness.

    As we look ahead to 2023, it is clear that software supply chain management will continue to be a priority for organizations across industries. With the increasing complexity of software supply chains and the evolving threat landscape, organizations must remain vigilant in their efforts to identify and mitigate potential risks.

  • Survey Surfaces Application Modernization Challenges

    Survey Surfaces Application Modernization Challenges

    A survey of senior IT leaders suggested the level of confidence in application modernization projects that many organizations initially had is now starting to wane.

    The survey polled 200 senior IT leaders at organizations with more 1,000 employees and was conducted by Global Surveyz Research on behalf of EvolveWare, a provider of platform for automating application modernization processes. The survey found 70% of respondents reported they were confident or very confident in their understanding of their applications when they prepared for modernization. However, that level of confidence dropped to 41% at the planning stage and even lower (28%) at the point the project began, the survey found.

    More than half of respondents (51%) are already experiencing or anticipating critical issues that need to be addressed immediately, the survey found. The top areas of importance are understanding the parts of the code that contain business policies (47%), database access details (43%) and understanding the complexity (40%).

    EvolveWare CEO Miten Marfatia said the survey makes it clear organizations are encountering issues associated with capturing tribal knowledge that has been lost mainly because the teams that initially built an application are no longer with an organization. In fact, the survey found a full 81% of respondents reported they currently have or anticipate challenges hiring or retaining talent to support legacy applications. Nearly half (46%) currently don’t have enough talent available to manage legacy applications.

    The survey found the primary reason these initiatives are launched is to boost employee productivity (40%), followed by retiring mainframe infrastructure (37%), reducing the dependency on legacy code (36%), reducing maintenance costs (30%), improving customer experience (29%) and cloud migration (22%). Most of these projects are generally focused on critical applications (55%) as well as applications that have few or no dependencies (55%), the survey finds.

    The primary success metrics for assessing these projects, however, are improving customer experience (46%), the readiness to migrate to the cloud (46%) and meeting the performance requirements of the modernized application (45%).

    Less than a third of respondents (31%), however, reported having access to modern tooling. More than half would like to automate code transformation and business rules extraction (BRE) while 40% would like to automate documentation.

    Nearly two-thirds (64%) said freezing code during the modernization process would have significant business and financial consequences, with 59% noting that finding a way to modernize applications without freezing code is at the top of their technology wish list.

    Application modernization efforts usually have multiple goals. IT teams are usually motivated by a desire to reduce the amount of time and effort required to support legacy applications that are generally less efficient than modern applications. In addition, many IT teams are also looking for ways to reduce the total cost of running those applications, especially during uncertain economic times. Business leaders, meanwhile, are funding digital business transformation initiatives that typically require some level of modernization of legacy applications.

    Regardless of motivation, it’s apparent many IT teams initially underestimate the amount of effort required to achieve those goals.

  • Dylibso Releases Tool for Tracking and Validating Wasm Modules

    Dylibso Releases Tool for Tracking and Validating Wasm Modules

    Dylibso today made generally available a system of record for tracking, investigating, searching and validating modules built using the WebAssembly (Wasm) binary instruction format.

    Fresh from raising $6.6 million in additional funding, Dylibso CEO Steve Manuel said Modsurfer exposes the internal contents to enable developers to debug Wasm modules and ensure compatibility with runtime environments.

    Modsurfer provides development teams with a console for accessing a fully auditable database of modules indexed for search and debugging. In addition, development teams have the ability to extract critical information from binary formats in a way that can be integrated with DevOps pipelines. In effect, Modsurfer provides application development teams with X-ray vision into previously opaque Wasm modules, said Manuel.

    Dylibso previously developed Extism, an open source universal plug-in system that supports multiple programming languages used to build Wasm modules. That reduces the level of effort required to integrate Wasm modules and provided the foundation upon which Modsurfer was created, noted Manuel.

    Wasm, at its core, is a portable binary instruction format for building software that describes a memory-safe, sandboxed execution environment. The World Wide Web Consortium (W3C) drove the development of Wasm as part of an effort to create a common format for browsers executing JavaScript code. Wasm is now being extended beyond browsers and JavaScript to enable developers to create a set of universal binaries that could work on any platform without modification.

    That approach replaces the current predominant method for building software that relies on the aggregation of software components that tend to lack distinct boundaries between them. One of the issues of that approach is that it becomes relatively simple for malware to infect all the components of an application.

    It may be some time before WASM becomes the preferred artifact for building applications, but it’s clear the walls that have separated one IT platform from another are finally starting to crumble. Once applications become much more portable, the longstanding goal of writing an application once and then deploying it anywhere may finally be achieved. That’s not only critical in terms of overall application developer productivity; it also has major implications in terms of streamlining the management of DevOps workflows as it becomes easier to both initially deploy applications and continuously update them.

    As organizations continue to review their application development practices to better secure software supply chains, it’s also become apparent that legacy approaches to building applications are fundamentally flawed. Rather than attempting to reengineer processes to encourage developers to better secure applications, it may be more effective to change the way applications are built. After all, considering all the processes required to secure legacy development approaches, it may be cheaper and easier to adopt a new method for building applications that lowers overall risk, said Manuel.

    Of course, the amount of Wasm expertise that exists in the developer community is still limited. However, DevOps teams should expect to see more Wasm modules moving through DevOps pipelines as Wasm tooling continues to improve.

  • HPE to Acquire OpsRamp to Gain AIOps Platform

    HPE to Acquire OpsRamp to Gain AIOps Platform

    Hewlett-Packard Enterprise (HPE) this week revealed it is acquiring OpsRamp to gain access to a platform for embedding artificial intelligence in IT operations (AIOps).

    Latha Vishnubhotla, chief platform officer in the office of the CTO for HPE, said the company would embed the machine learning technology developed by OpsRamp across its entire portfolio in addition to leveraging it to automate IT processes the company manages via its GreenLake service for HPE customers.

    Ultimately, HPE will be applying machine learning across the entire IT stack, including applications, to reduce the total cost of IT, she said.

    The acquisition of OpsRamp is expected to close in the third quarter of this year. Terms of the deal were not announced, but it comes at a time when more organizations are getting comfortable with AIOps.

    A recent OpsRamp survey polled 138 IT leaders working for enterprise IT organizations and 127 representing managed service providers (MSPs) that generate at least $25 million in revenue and have more than 500 employees. The survey found the primary reason organizations are embracing AIOps is to improve service and application availability and performance (60%), followed closely by automation of operations (58%) and processes (54%).

    The survey found that the greatest IT operations challenge for enterprises in 2023 was automating as many operations as possible (66%), followed by driving greater efficiency and productivity of IT operations teams and processes (61%). Application-to-infrastructure dependency mapping is the top incident management challenge (64%), the survey found.

    The primary operational benefit cited was reducing open incident tickets (65%), followed by reducing mean-time-to-discovery (MTTD) and mean-time-to-remediation (MTTR), tied at 56%, and automation of tedious tasks (52%).

    In general, organizations are choosing one of two AIOps paths, said Vishnubhotla. Some are waiting to evaluate the recommendations made by the AIOps platform before implementing, while others are looking to completely automate processes so they can focus more of their time and effort on application development and deployment, she said. In much the same way a driver of an autonomous vehicle learns to trust the AI over time, there is a similar transition occurring with AIOps, noted Vishnubhotla.

    The challenge is that it takes an AIOps platform a few months to learn an IT environment. The more dynamic that IT environment is, the longer that process can take. At the very least, however, AIOps platforms have the ability to automate many low-level IT tasks today that conspire to increase the amount of toil DevOps teams would otherwise encounter.

    It’s difficult to determine how pervasive AIOps may be within IT environments, but there is no doubt as organizations look for ways to manage IT at scale, there will be increased reliance on automation. Most organizations can’t afford to continue to hire armies of IT personnel, especially as the global economic outlook remains uncertain.

    In fact, one of the benefits of an AIOps platform is that, unlike an employee, it does not take any days off or suddenly decide to take another job. It also, for better or worse, doesn’t forget anything once it’s been learned. As such, it’s now only a question the degree to which AIOps platforms will be trusted over time to manage increasingly more complicated IT tasks.

  • How to Maximize Telemetry Data Value With Observability Pipelines

    How to Maximize Telemetry Data Value With Observability Pipelines

    The software landscape has shifted significantly in recent years as companies digitize their operations and adopt cloud and microservices technologies. The complexity of modern software systems has led to an increasing need for real-time visibility into their performance—resulting in an explosion of telemetry data that is stretching IT budgets and outpacing engineering teams’ ability to find value in that data. In fact, data volume is growing at 23% yearly, while IT budgets are only growing at 5%.

    Managing all this data at increasing cost is further compounded by unpredictable software failures and ever-evolving security threats. As a result, companies are forced to pick which parts of their software to instrument and monitor. To overcome this challenge, a different approach to managing data is required.

    Observability pipelines are emerging as a way to address this problem. They centralize telemetry data (i.e., logs, metrics and traces) from many sources, transform that data to fit downstream needs, and route it to various destinations for analysis. With complete control of the data, companies can sort through large amounts and prioritize what is essential, allowing them to act swiftly to avoid disruptions while reducing costs by only storing the data they need.

    Research indicates 62% of organizations currently have an observability practice. This means 38% operate with limited awareness of what’s happening in their software. Even those with an observability practice can only afford to store some data and have to choose which applications to instrument. This leads to blind spots and, ultimately, issues that are time-consuming to fix due to a lack of data.

    Managing Limited Resources With Observability Pipelines

    Engineers face more demands in today’s fast-paced business environment than ever before. Despite an increase in their responsibilities, the size of engineering teams has yet to grow accordingly. Given this, there has been a rise in full-stack engineers responsible for technologies from the front end to the back end. Short on time and resources, it’s difficult for these engineers to instrument and monitor software.

    Observability pipelines help control the amount of telemetry data using various processors such as sampling, throttling, filtering and parsing, and only forward valuable data to the downstream systems. The rest of the data can either be discarded or stored in a low-cost system, such as Amazon S3. This reduces costs and lets engineers focus on analyzing relevant data to identify issues.

    Engineers can also use observability pipelines to consolidate data from various tools. Many teams use different platforms or open source tools, like Prometheus or Jaeger. The fragmented data makes it hard to solve issues and understand application health. Observability pipelines combine all this data to make it easier to act on.

    It’s also important to remember that real people work on these technical challenges. Engineers face burnout working long hours to keep up with software development demands. Constantly switching between tools and manually moving data adds to the pressure. Moreover, the number of data sources, observability tools, and platforms that need that data are constantly increasing. Embracing open standards such as OpenTelemetry can ease this integration burden. Engineers can swap out tools without extensive testing and integration. The increased control and easy integration enable engineers to focus on delivering business value.

    Making Data More Usable

    Teams across organizations want access to data but struggle to use it effectively. The difficulty stems from the lack of data usability: Telemetry data is unstructured; varying formats make it hard to use; data preparation is time-consuming and sensitive data in logs may lead to compliance violations.

    By most estimates, 80%-90% of data is unstructured, which makes it hard to store and analyze because it doesn’t adhere to traditional data models. Observability pipelines make sense of unstructured data before it reaches its final destination. This is achieved using data processors that shape and transform data to make it more actionable. These processors include advanced parsers that identify and extract relevant information, transforming data into consumable formats that make it easier to work with and analyze.

    The advantage of performing these operations within the pipeline is that the same data can be prepared to fit different use cases downstream. For instance, while one team may require data optimized for visualization and trend analysis, another may require complete data for threat hunting. These transformations are managed from a single control point, eliminating the need to maintain separate data streams.

    Observability pipelines can also help companies meet data compliance requirements. For example, teams can scrub, mask or redact PII data based on a defined key structure before the data reaches a SIEM or audit platform.

    Taking Immediate Steps to Make the Most of Telemetry Data

    Organizations can take immediate action to make the most of their telemetry data. First, they need to adopt a visibility-first rather than a cost-first approach. They should assess what software they’ve instrumented. This will identify gaps in telemetry, which can be used to develop a plan to instrument all software. Next, they need to create a strategy for managing their telemetry data, being particularly thoughtful about how and why the data is stored and analyzed.

    Once a company has taken these two steps, it can decide which tools to use. One of the great things about OpenTelemetry is that it provides flexibility for future decisions about what tools to use.

    As companies invest more resources into building observability practices, they’ll encounter problems processing an extraordinary amount of telemetry data for any meaningful action. Observability pipelines provide a competitive advantage of prioritizing the essential data that will enable companies to make better decisions faster.

  • Awareness of Software Supply Chain Security Issues Improves

    Awareness of Software Supply Chain Security Issues Improves

    A global survey of 167 software professionals suggested that, while there is a lot more awareness of application security issues, the adoption of DevSecOps best practices is still not pervasive.

    The survey, conducted by Chainguard, the Eclipse Foundation, the Rust Foundation and the Open Source Security Foundation (OpenSSF), found that just over half (51%) of respondents are either concerned or are extremely concerned about software supply chain security.

    In addition, only a third of respondents (33%) reported they are familiar with the Supply-chain Levels for Software Artifacts (SLSA) framework for building more secure applications. Well over a third (36%), however, said they had never heard of it.

    Only a quarter of respondents (25%) reported that their team always signed built artifacts, the survey found.

    The survey identified the most challenging issues developers are encountering and found they are reproducing builds (64%) followed by creating hermetic builds that are isolated from other software libraries (62%).

    Mikaël Barbero, head of cybersecurity for the Eclipse Foundation, said while awareness of software supply chain issues is higher, it’s apparent that other priorities are still getting in the way of adoption of DevSecOps best practices. For example, software bills of materials (SBOMs) may generate a lot of headlines, but their adoption remains mixed, he added.

    John Speed Meyers, a security data scientist for Chainguard, noted that incentives still need to change; too many managers that hire developers still emphasize performance and delivery times over application security. Many developers, as a result, still tend to view cybersecurity as an obstacle to be overcome or worked around rather than a core capability that they will be graded on, he added.

    In theory, as responsibility for application security shifts left toward developers, overall cybersecurity will improve. Most developers, however, have little to no cybersecurity training, so the odds they will be able to resolve these issues are low. Instead, many organizations are either trying to embed cybersecurity professionals within DevOps workflows or, at the very least, deputize at least one developer to advocate for cybersecurity within those teams.

    Regardless of commitment level, it’s apparent that organizations need to address everything from locking down developer environments to thwart phishing attacks to ensuring application builds are not inadvertently infected with malware. The challenge organizations face is both technical and cultural, so the amount of effort required should not be underestimated.

    In the meantime, organizations should expect to see cybercriminals increase their attempts to compromise software supply chains. Infecting malware into software components that will be used by multiple downstream applications is, from their perspective, one of the most efficient means of wreaking havoc. Right now, organizations make it too easy for cybercriminals to compromise application environments.

    Organizations should also recognize that tolerance for insecure applications is also declining, as evidenced by the National Cybersecurity Strategy put forward by the Biden administration. It’s not so much a question of whether organizations that create software will be held liable for application security issues as much as it is to what degree.

  • Why Observability is Important for Development Teams

    Why Observability is Important for Development Teams

    Observability, which allows development teams to understand better how their systems behave in production, is critical to successful software delivery. According to Gartner, by 2026, 70% of organizations that successfully applied this practice will achieve shorter latency for decision-making, enabling competitive advantage for target business or IT processes.

    Observability allows development teams to identify and diagnose issues quickly to make decisions that improve overall software delivery performance. Let’s take a closer look at why this practice is critical for development teams.

    Differentiating Observability and Visibility

    To better understand observability, it’s essential to clear up a common misconception – that observability and visibility are the same. Two critical differences exist between the two. Visibility allows development teams to look at the surface-level aspects of an application or system. In contrast, observability provides deeper insight into what an application is doing in production. Being equipped with a deep knowledge of observability allows developers to be more productive within their respective organizations.

    Impact on Developer Productivity

    With observability, developers can identify and diagnose issues within applications quickly, giving them more time to fix bugs and time for resolution, which are both critical success factors for software delivery. Observability allows organizations to quickly identify and respond to potential problems, such as operational issues or security breaches, before they become issues that impact customers or users. While there is no such thing as perfect software, observability helps developers quickly find and fix issues, bottlenecks, inefficiencies, etc., so they can get back to doing what they love: Creating new software.

    With the benefits of observability, developers naturally gravitate toward tooling and technology. There are many tools available for developers to take advantage of, and development teams must learn how to use these tools to integrate them into their applications and services. Let’s explore some of the more popular options.

    The Shift to Cloud-Native Tools

    The shift to cloud-native tools is driven by the increasing popularity of cloud computing and its benefits, such as scalability, cost-effectiveness and flexibility. The move to cloud-native has made an enormous impact on observability efforts. Having good tooling and processes available positively impacts observability efforts because it’s a critical aspect of the developer communities’ systems. Overall, the shift to cloud-native tools is driven by the need for organizations to take advantage of the benefits of cloud computing and the need for more efficient and flexible deployment and scaling of applications.

    Some of the most common tools in observability include:

    These tools offer developers greater insight and knowledge into how applications behave. The cloud-native approach necessitates a new way to aggregate all of these operations.

    Conclusion

    With observability, developers can understand how their systems work in production. These systems act in different ways depending on the environments they run in. Therefore, organizations must do more to provide developers with sufficient observability because of the limited visibility into the overall system performance.

    Observability allows developers to act quickly and remedy issues before they go into production and organizations can better measure success relative to the application’s performance. Hours of developers’ time are spent manually debugging and troubleshooting when organizations could have a system in place to do this automatically and allow developers to get back to what they love doing most: Creating software. Observability helps organizations make better-informed decisions, improve operations and performance and achieve their objectives more effectively.