devopsy.kinsta.cloud

Category: Blogs

The Role of Observability Engineers in Managing Complex IT Systems

The rise of microservices, the adoption of cloud-native architectures and the increasing shift to digital have led to the growing volume of telemetry data (logs, metrics, traces, events) contributing to the complexity of observing modern IT systems effectively and at the right cost. These challenges with processing the telemetry data on time and at the right cost make it difficult for organizations to maximize their observable surface area and ensure they perform at their best.

Enter observability engineers. These professionals are tasked with collecting, processing, analyzing and visualizing data from complex IT systems. They use this data to identify patterns, detect anomalies and understand system behavior. This awareness allows observability engineers to discover and address potential issues before they impact users.

Observability engineers play a critical role in ensuring the reliability, performance and security of complex IT systems, and are essential for organizations that want to stay ahead of the curve in today’s rapidly evolving technology landscape.

Let’s examine the essential characteristics and responsibilities of observability engineers to understand their role.

Characteristics of Observability Engineers

Observability engineers are problem solvers who specialize in optimizing system performance, ensuring reliability and driving actionable insights from telemetry data. They are the driving force behind data-driven solutions and operations. Their specialized skillset and unique focus on observability set them apart from other roles within IT operations. They may have previous backgrounds as system administrators, site reliability engineers (SREs) and IT operations members, who often react to incidents, but observability engineers are proactive and have adopted a shift-left approach to observability. They are adept at understanding the telemetry data and its formats, creating telemetry strategies, implementing comprehensive monitoring systems, and using advanced observability platforms to gain real-time insights.

Another important aspect of observability engineers’ expertise is their in-depth knowledge of telemetry data. They have the necessary skills to collect, analyze and process data such as metrics, events, logs and traces from various sources. They understand the transformations that the data needs for easy consumption by downstream SIEM, analytics, or visualization systems. They are well versed in the challenges posed by microservices architectures, cloud-native environments, and hybrid infrastructure setups and how to extract meaningful information from the system exhaust. This understanding enables them to design telemetry pipelines to move the right data, in the right format, to the right system; build monitoring systems; and implement observability practices that effectively capture, transform and analyze these complex systems.

Collaboration and cross-functional skills are other essential aspects of the observability engineers’ role. They break down the barriers between observability domains, including infrastructure, applications and networking. By coordinating and aligning the efforts of these domains, observability engineers create a more comprehensive understanding of the IT system’s behavior and performance. This approach allows for a holistic analysis beyond the limitations of individual domains and identifies cross-domain optimization and improvement opportunities.

Responsibilities

Observability engineers have many responsibilities focused on addressing problems within IT operations. Some of their key responsibilities are detecting anomalies, troubleshooting incidents, monitoring system health, optimizing resource allocation, enhancing user experiences, enabling data-driven decision-making and supporting compliance and security efforts. Through their skills and expertise, observability engineers empower organizations to maintain highly performant, reliable and secure IT systems.

They are needed throughout the lifecycle of IT systems. They play a vital role in system design and implementation, ongoing maintenance and monitoring, incident response and troubleshooting, optimization, performance enhancement and new feature development and releases. Their expertise ensures that observability is built into the system from the ground up, maintains optimal system performance, minimizes downtime and enhances the user experience.

How Observability Engineers Perform Their Tasks

Observability engineers use telemetry data to learn trends and patterns. Analyzing this data allows them to plan capacity, predict potential issues, and implement preventive measures. This approach enables organizations to continuously improve their IT systems, enhancing performance, reliability and user satisfaction.

To achieve their objectives, observability engineers use various tools for collecting and analyzing data. Telemetry pipelines are emerging as their primary tool for collecting data from various sources within an IT system, transforming it, and routing it to various destinations. These pipelines enable observability engineers to gather data from metrics, logs, events and traces. They process and transform this data to extract meaningful insights and send it to downstream analytics or visualization platforms or for long-term storage.

Various analytics and visualization APM and observability platforms allow observability engineers to analyze and visualize the data. Custom dashboards and visual representations offer them a comprehensive view of system behavior, performance trends and user experiences. With this information, observability engineers can troubleshoot and optimize system performance, finding root causes of issues and implementing effective solutions.

They also contribute to compliance and security efforts. By monitoring and analyzing telemetry data, observability engineers identify possible vulnerabilities and risks within the system. They help organizations maintain robust security practices and ensure compliance with regulations.

Investing in the Promise of Observability Engineering

Observability engineers are increasingly becoming essential for managing the complexity and unpredictability of modern IT systems. Their specialized skills and expertise enable them to address potential issues, optimize system performance and ensure reliability. As technology advances, the demand for observability engineers will only increase. Organizations striving to succeed in their IT operations must invest in these professionals.

October 2, 2023
Five Great DevOps Job Opportunities

DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

SimplyHired.com

Wolfe
Remote
Senior Staff DevOps Engineer
$162,000 to $172,000

LinkedIn

Justworks Labs
New York, New York
DevOps Engineer
$155,000 to $165,000

Dice.com

KBS Solutions
Annapolis Junction, Maryland
DevOps Engineer
$150,000

CareerBuilder.com

Fusion Global Solutions
Dallas, Texas
Azure Architect with DevOps Experience
$140,000 to $190,000

Indeed.com

Kaseware
Denver. Colorado
Senior DevOps Engineer (Kubernetes)
$140,000 to $180,000

October 2, 2023
System Failure

October 2, 2023
Raspberry Pi 5: Faster, Better, Stronger — Spendier

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

In a cheeky extra post this week: Everyone’s favorite single-board ARM computer, the Raspberry Pi, has a new generation coming soon. Compared to the ’4, RPi5 has double the performance, quadruple the base RAM and far more capable I/O. (more…)

September 29, 2023
Senser Unveils AIOps Platform Using eBPF to Collect Data

Senser emerged from stealth this week to launch an artificial intelligence for IT operations (AIOps) platform that leverages extended Berkeley Packet Filter (eBPF) running in the microkernel of Linux operating systems to collect data from IT environments.

Fresh from raising $9.5 million in funding, Senser CEO Amir Krayden said the company’s namesake platform then applies machine learning algorithms to that data to identify issues that could lead to outages. Those insights are surfaced using graph technology to make it simpler to both observe IT environments and triage issues at scale because the AIOps platform is running processes at the microkernel level rather than in user space.

The approach provides IT teams with a more efficient and holistic approach to observability at a level of scale legacy platforms can’t achieve, said Krayden.

The use of machine learning algorithms also reduces the cognitive load on DevOps teams because issues involving, for example, performance degradations are automatically surfaced, he added. In addition, the company is working toward adding generative AI capabilities to provide summaries that explain what IT events have occurred, noted Krayden.

In effect, eBPF changes the way operating systems are designed because it enables networking, storage and observability software to scale at much higher levels of throughput since they are no longer running in user space. That’s especially critical for observability and AIOps platforms that need to dynamically process massive amounts of data in near-real-time.

As the number of organizations running the latest versions of Linux continues to increase, more hands-on experience with eBPF will be gained. IT teams may not need to concern themselves with what is occurring in the microkernel of the operating systems, but they do need to understand how eBPF ultimately reduces the total cost of running IT at scale.

AI and graph technology, in combination with eBPF, will fundamentally change how IT is implemented and managed. The current complexity of application environments is already exceeding the ability of IT teams to cost-effectively manage them at scale, so the need for a different approach is already apparent. Many IT environments are already too complex for IT personnel to manage without the help of some form of AI.

It’s not clear precisely how much AI will automate the management of IT, but it’s not likely the need for humans to manage and supervise these environments will happen any time soon. However, the level of scale at which an IT environment can be effectively managed is changing as AI makes it easier to identify issues and understand their impact. Too often today, there are simply too many dependencies within an IT environment to keep track of using legacy monitoring tools that only track a set of pre-defined metrics.

It may be a while before AI is pervasively employed across IT environments, but it’s now more a question of when rather than if. The issue now is determining where the interface between the humans and the machines that are jointly managing IT environments lies.

September 29, 2023
What is a Shadow Deployment?

Shadow deployment, at its core, is a software deployment practice where any changes to a software application are deployed in a parallel environment that mimics the production environment. The deployed changes are not visible to the end-users, hence the term “shadow.” This hidden deployment enables us to observe the behavior and impact of the changes without causing any disruption to the live application.

The primary purpose of shadow deployment is to test real-world application behavior under load, identify performance bottlenecks, and ensure that the application changes don’t affect the user experience negatively. It allows developers to test changes in an environment that closely replicates the live production environment, thereby reducing the risk of unforeseen issues cropping up post-deployment.

In essence, shadow deployment is akin to having a rehearsal before the actual performance. It provides a safety net to developers and aids in ensuring a smooth transition when the changes are finally deployed to the live environment.

Benefits of Shadow Deployment

Risk Mitigation

One of the most significant benefits of shadow deployment is risk mitigation. In traditional deployment methodologies, code changes are often deployed directly to production, which may lead to unforeseen issues and disruptions.

With shadow deployment, the risk of such disruptions is significantly reduced as the changes are first deployed in a parallel environment. This allows for thorough testing and validation, ensuring any bugs or performance issues are identified and rectified before the changes hit the production environment. This is also useful for DevSecOps teams, as it can help them understand the security impact of a new deployment without exposing it to attackers.

Shadow deployment also offers the benefit of rollback. If any critical issue is identified during the shadow deployment, the changes can be rolled back without affecting the live environment. This provides a safety net to developers and businesses, ensuring that the user experience is not compromised.

Real-World Testing

Another significant advantage of shadow deployment is that it allows for real-world testing. In traditional testing methodologies, simulated data is used, which may not accurately represent the live environment. However, in shadow deployment, the parallel environment mirrors the live environment, ensuring testing is done under real-world conditions.

This offers a more accurate representation of how the changes will perform when deployed to production. It helps identify issues that might not be apparent in a simulated testing environment, ensuring a more robust and reliable deployment.

Efficient Scaling

Finally, shadow deployment aids in efficient scaling. In today’s digital landscape, applications need to be scalable to handle varying loads. Shadow deployment enables developers to test the scalability of their changes in a real-world environment, thereby ensuring efficient scaling.

It allows for the testing of various aspects of scaling, such as database scaling, server load balancing and more. This ensures that when the changes are finally deployed to production, the application can efficiently scale based on user demand.

Use Cases for Shadow Deployment

Performance Testing

One of the primary use cases of shadow deployment is performance testing. It allows developers to test the performance of their changes under real-world conditions. This includes testing aspects such as response times, server loads, database queries and more. The insights gained from this testing can be used to optimize the changes, ensuring optimal performance when deployed to production.

Application Migration

Shadow deployment is also widely used during application migration. When migrating an application from one platform or technology stack to another, it is crucial to ensure that the migration does not negatively impact the user experience. Shadow deployment allows for the testing of the migrated application in a parallel environment, ensuring that any issues are identified and rectified before the migration is made live.

Service Dependency Validation

In today’s microservices architecture, applications often depend on various services to function correctly. Shadow deployment can be used to validate these service dependencies. It allows developers to test if their changes are compatible with the dependent services, ensuring smooth interoperability when deployed to production.

Testing New Plugins or Extensions

Finally, shadow deployment can be used to test new plugins or extensions. Plugins and extensions often add new functionalities to an application, and it is crucial to ensure that these additions do not impact the application negatively. Shadow deployment allows for the testing of these plugins and extensions in a parallel environment, ensuring that they work as expected when deployed to production.

Shadow Deployment Vs. Other Deployment Strategies

Let’s compare the practice of shadow deployment with some of the other common deployment strategies used in the industry.

Blue-Green Deployment

Blue-green deployment is a strategy that reduces downtime and risk by running two identical production environments. The blue environment is live, while the green environment is idle. When a new version of the application is ready, it’s deployed to the idle environment, tested and, once everything is confirmed to be working, traffic is switched from the blue to the green environment. This approach allows for instant rollback if issues arise.

Shadow deployment, on the other hand, takes a slightly different approach. It involves deploying a new version of the application alongside the live version and sending the same input to both. The new version doesn’t affect the live system, and its output is discarded. This strategy allows for extensive testing and performance comparisons without any risk to the live system.

Canary Deployment

Canary deployment is another strategy that involves rolling out changes to a small subset of users before deploying it to the entire infrastructure. This strategy allows teams to test the performance of new features on a small scale before a full-scale rollout.

In contrast, shadow deployment does not expose the new version to any users. Instead, it uses real production traffic to test the new version, discarding its output to ensure it doesn’t affect the live system. This allows for full-scale testing without risking user experience.

Feature Flags

Feature flags, also known as feature toggles, allow developers to switch on or off certain features in their applications. This helps manage the rollout of new features and enables easy rollback if things go wrong.

Shadow deployment, however, focuses more on system performance rather than individual features. It uses duplicate traffic to test the entire system, providing valuable insights into how the new version will perform under real-world conditions.

Best Practices for Shadow Deployment

Now that we understand how shadow deployment fits into the broader landscape of deployment strategies, let’s delve into the best practices to follow when implementing this strategy.

Isolate Real and Shadow Systems

One of the foundational best practices for shadow deployment is the isolation of the real and shadow systems. This ensures that any problems in the shadow system do not affect the real one. The goal is to test how the new version of the software performs under real-world conditions without impacting the live system.

Set Clear Duration and End Criteria

When setting up shadow deployment, it’s crucial to define clear duration and end criteria. This involves determining how long the shadow system will run alongside the real system and what conditions will mark the end of the testing phase. This helps in maintaining control over the process and ensures that you gather the necessary data to make informed decisions.

Set Up Traffic Duplication Efficiently

Efficient traffic duplication is another critical aspect of shadow deployment. This involves mirroring the traffic from the real system to the shadow system in real-time. It’s essential to ensure that this process is efficient to get accurate performance data without overloading your network resources.

Prioritize Feedback Loops

Last but not least, prioritizing feedback loops is crucial in shadow deployment. This involves setting up mechanisms to gather and analyze data from the shadow system. The insights gleaned from this data can provide valuable feedback, helping you refine your deployment process and improve your system performance.

Conclusion

In conclusion, shadow deployment offers a unique approach to software deployment. By allowing for extensive testing under real-world conditions without affecting the live system, it provides invaluable insights into system performance. By understanding its benefits and following best practices, you can leverage this strategy to streamline your deployment process and improve your software systems.

September 29, 2023
Dynatrace Survey Surfaces DevOps Automation Gains and Challenges

A global survey of 450 DevOps and security practitioners published this week by Dynatrace found investments in automation have improved software quality (61%), reduced deployment failures (57%) and decreased IT costs (55%).

However, only 38% of respondents said their organization had a clearly defined DevOps automation strategy. On average, organizations have succeeded in automating just over half (56%) of their end-to-end DevOps life cycle, the survey found.

Saif Gunja, director of product marketing for Dynatrace, said the survey makes it clear that automation is still being applied to isolated workflows rather than centrally managed. That is changing, however, as more organizations embrace platform engineering to manage DevOps workflows at scale, he added.

Overall, organizations are investing in DevOps automation over the next 12 months to improve security and compliance management (55%), infrastructure provisioning and management (52%) and performance optimization (51%).

The biggest barriers preventing organizations from automating new DevOps use cases are security concerns (54%), difficulty operationalizing data (54%) and toolchain complexity (53%). The average organization relies on more than seven different tools for DevOps automation, the survey finds.

As a result, the automation that has been put in place is often too brittle to effectively scale beyond a small DevOps team, noted Gunja.

A full 71% of respondents also used observability data and insights to drive automation decisions and improvements in DevOps workflows. However, 85% also admitted they faced challenges using observability and security data to drive DevOps automation. The top three challenges facing organizations included inaccessible data (51%), siloed data (43%) and the need for data to flow through many systems (41%).

Well over half (54%) said their organization is investing in platforms to enable easier integration of tools and collaboration between teams involved in automation projects.

Finally, 59% also anticipated large language models (LLMs) such as ChatGPT and Bard would have a significant impact on DevOps, with the top three benefits expected including increased productivity and reduced manual effort (57%), improved development, security and operations collaboration (56%) and enabling teams to generate code automatically (48%).

Many organizations that have embraced DevOps are now looking to strike a balance between the concept of developers owning every application or service they developed and the need to reduce the amount of toil developers now encounter, noted Gunja. Developers typically only spend about a quarter of their time writing code so there is a clear need to reduce the current level of cognitive load developers encounter as they maintain application development environments, he added.

The challenge is finding a way to reduce that load in a way that doesn’t overly restrict the ability of developers to embrace new tools and platforms that will enable them to build applications in a more innovative fashion. One of the primary reasons many development teams embraced DevOps in the first place was to not have the way they build applications dictated to them by a centralized IT team.

One way or another, DevOps workflows will continue to be increasingly automated. Less clear at the moment, however, is exactly who within an organization will be driving those initiatives.

September 28, 2023
What DevOps Teams Need to Know About Responsive Web Design

The role of DevOps in modern software development extends beyond just coding, testing and deployment. One of the significant areas where DevOps teams can bring value is delivering a positive user experience, a critical competitive advantage in the digital era. Responsive web design (RWD) is a cornerstone of a friendly user experience, allowing for seamless website functionality across various screen sizes and devices.

This article explores RWD and its fundamental principles and explains how RWD impacts web application development and deployment. We will review critical considerations for DevOps teams such as performance optimization across devices, the integration of responsive design into CI/CD pipelines, automated testing, the importance of a user-centric approach and consistent monitoring and troubleshooting. The objective is to equip DevOps teams with the necessary knowledge to effectively implement and support RWD, enhancing the user experience and significantly improving overall user satisfaction and engagement.

What is Responsive Web Design?

Responsive Web Design is a web development approach that makes your website layout adjust according to the size of the viewer’s screen. This means that your website looks and functions optimally on a wide range of devices, from desktop computers to tablets and smartphones. The primary goal of RWD is to ensure that users have an efficient and enjoyable experience, regardless of the device they are using.

With the increasing use of mobile devices for internet access, it is essential that your website is accessible and user-friendly on all platforms. By implementing RWD, your organization can ensure that your website offers a seamless user experience, no matter the device.

RWD removes the need for creating different versions of your website for different devices. Instead, you can have a single website that adjusts automatically to fit any screen size. This not only improves user experience but also saves you time and resources in website development and maintenance.

Understanding the Principles of Responsive Web Design

Fluid Grid Layouts

A fluid grid layout is a key principle of RWD. It involves designing your website layout in relative units, such as percentages, instead of absolute units like pixels. This allows your website layout to resize dynamically to fit the screen on which it is viewed.

The advantage of fluid grid layouts is that they offer more flexibility than fixed layouts. They allow your website to adjust smoothly to different screen sizes, ensuring that the user experience is not compromised. This is particularly important in today’s digital landscape, where users access websites from a wide range of devices with different screen sizes.

Flexible Images and Media

The second principle of RWD is responsive images and media. This means that your images and media files should be able to resize automatically to fit the screen size. This is essential to ensure that your images and media do not distort or overflow when viewed on smaller screens.

Flexible images and media enhance the visual appeal of your website on different devices. They also improve the loading speed of your website, which is a crucial factor for user experience and search engine ranking.

Mobile-First Design Philosophy

Mobile-first design philosophy is another key principle of RWD. As the name suggests, this approach involves designing your website primarily for mobile devices and scaling it up for larger screens.

The mobile-first design philosophy is based on the reality that a growing number of users are accessing the internet through their mobile devices. By designing your website with a mobile-first approach, you can ensure an optimal user experience on mobile devices.

Progressive Enhancement

Progressive enhancement is the fourth principle of RWD. It involves building your website in layers, starting with a basic layer that works on all devices and then adding more features and functionalities for devices with advanced capabilities.

Progressive enhancement allows you to provide a basic, functional website for all users regardless of their device or browser. At the same time, it enables you to offer an enhanced website experience for users with modern devices and browsers.

How RWD Affects Application Development and Deployment

Responsive web design has a significant impact on the development and deployment of web applications. It affects various aspects of the process, including the codebase, development and testing process, user interface design and deployment and updates.

Unified Codebase

One of the main benefits of RWD is that it allows you to maintain a unified codebase. Instead of creating separate versions of your application for different devices, you can have a single codebase that works across all platforms.

A unified codebase simplifies the development and maintenance process. It eliminates the need for duplicate coding and testing, saving you time and resources. It also ensures consistency in your application, as all users are accessing the same codebase, regardless of their device.

Development and Testing Process

RWD also influences the development and testing process. Since your application needs to work optimally on various devices and screen sizes, you need to incorporate responsive design considerations into your development and testing process.

Responsive design testing involves checking your application on different devices and screen sizes to ensure that it functions and displays correctly. This can be done manually, using device emulators or through automated testing tools.

User Interface Design

The user interface (UI) design of your application is also affected by RWD. A responsive UI design ensures your application is easy to use and visually appealing on all devices.

Responsive UI design involves creating a flexible layout that adjusts automatically to the screen size. It also requires you to design intuitive navigation and interaction elements that work well on touchscreens and desktop interfaces.

Deployment and Updates

Finally, RWD has implications for the deployment and updates of your application. With a responsive application, you can deploy and update a single version of your application for all devices.

This simplifies the deployment and update process and ensures that all users have access to the latest version of your application. It also makes it easier to fix bugs and issues, as you only need to update a single codebase.

Key Considerations for DevOps Teams

Implementing RWD is not a straightforward task. It requires careful planning and consideration. Here are some key considerations for DevOps teams when implementing RWD.

Ensuring Website Performance Across Different Devices

One of the main challenges in implementing RWD is ensuring that your website performs well on all devices. This requires careful optimization of your website elements, including images, CSS and JavaScript.

You also need to consider factors like loading speed and responsiveness, as these can significantly affect user experience. There are a number of available tools to help you analyze and optimize your website performance.

Incorporating Responsive Design into CI/CD

Continuous integration and continuous delivery (CI/CD) are key practices in DevOps. CI/CD involves integrating code changes regularly and delivering updates to users quickly and reliably. Incorporating responsive design in your CI/CD process can help you ensure that your application is responsive and user-friendly at all times.

Automation in Testing

Testing is a crucial part of the development process. With RWD, testing becomes more complex as your application needs to work on a wide range of devices and screen sizes. Automation can help streamline the testing process and ensure that your application is tested thoroughly and efficiently.

User-Centric Approach

A user-centric approach is essential in implementing RWD. This means considering the needs and preferences of your users at every stage of the development process. From the design phase to testing and deployment, the user experience should be at the forefront of your considerations.

Monitoring and Troubleshooting Responsive Design Issues

Finally, monitoring and troubleshooting are important aspects of implementing RWD. You need to monitor your website performance on different devices and troubleshoot any responsive design issues that arise.

Analytics tools can provide valuable insights into how your website is performing on different devices. They can also help you identify and fix any issues that are affecting your website’s responsiveness.

Conclusion

In conclusion, responsive web design is an essential aspect of modern web development. It ensures your website is accessible and user-friendly on all devices, improving user experience and engagement. By understanding the principles of RWD and considering the key aspects of implementing it, you can unlock the power of responsive design and take your website to the next level.

September 28, 2023
Post service upgrade pods are taking too long to serve traffic

After a service upgrade the pods are taking too long to serve traffic. What would you need to find out to debug?
1. The validating webhook for pods has failed
2. The cluster doesn’t have enough CPU/memory to schedule the pod to a node
3. The new code added an init container and the init container is taking a long time to finish
4. The new deployment changed the readiness probe
5. There is a bug in the code

Let’s break down why each of the provided choices is indeed a correct answer in the context of troubleshooting slow pod response times after a service upgrade:

The validating webhook for pods has failed: Validating webhooks in Kubernetes are responsible for ensuring that pods conform to certain policies before being created. If the validating webhook fails, it can prevent pod creation altogether, which would certainly impact the ability of new pods to serve traffic. Therefore, this is a correct choice to consider when debugging service issues.

The cluster doesn’t have enough CPU/memory to schedule the pod to a node: When a service is upgraded, its resource requirements may change. If the cluster lacks the necessary CPU or memory resources to schedule the new pods, it can lead to delayed pod scheduling and slow response times. This is a common issue that SREs and DevOps need to investigate when traffic-serving problems arise after an upgrade.

The new code added an init container, and the init container is taking a long time to finish: Init containers are executed before the main application containers in a pod. If a new init container is introduced during an upgrade and it takes an extended time to complete its tasks, it can indeed cause delays in the pod becoming ready to serve traffic. This is a relevant consideration when diagnosing slow response times post-upgrade.

The new deployment changed the readiness probe: The readiness probe determines when a pod is ready to accept traffic. If the readiness probe is modified during an upgrade and is now misconfigured, it can delay the point at which the pod is considered ready to serve traffic.
Therefore, this is an accurate choice for troubleshooting slow response times.
One of our earlier quiz questions dealt with a similar issue which we discussed in the following blog: https://webb.ai/blog/quiz-1-aug-1-23-healthcheck-port-mismatch/

There is a bug in the code: While the broadest option, it’s also valid. Code bugs can absolutely result in slow response times. However, it’s crucial to note that this choice doesn’t specify what type of bug or where in the code it might be. It highlights the importance of code review and debugging as part of the troubleshooting process.

In summary, all the provided choices are correct because they represent various potential causes of slow pod response times after a service upgrade. Troubleshooting such issues often requires considering a combination of factors, from resource constraints to configuration changes and code-related problems. SREs and DevOps professionals must approach the problem with a holistic perspective to identify and address the root causes effectively.

There are additional reasons, not included in the options provided, for slow pod response times after a service upgrade:
Network Latency/Issues: Connectivity problems, network misconfigurations, or high network latency between pods and services can cause delays in serving traffic.
Storage Latency: Slow storage I/O or inadequate storage resources can impact data retrieval and processing, affecting pod performance.
Resource Contentions: Other resource-intensive workloads on the same nodes can lead to resource contentions, resulting in slow pod response times due to competition for CPU, memory, or disk resources.

Analyzing each potential root cause for each failure is not possible in today’s highly dynamic environments. Leveraging software and the recent advancements in AI to automate root cause analysis is one way to reduce toil and increase productivity allowing SRE and DevOps to become proactive and focus on addressing more complex failure scenarios.

September 28, 2023
Yes, Software Development is an Assembly Line, but not Like That

I grew up in a manufacturing town. Nearly everyone worked for (or previously worked for) the local textile factory. My father was no exception. He had a few jobs but eventually landed a management job at the factory and started working closely with the engineers and other plant operators. At the time, I was in high school and about to start my education in software development. At the time, I felt–maybe hoped–that my world and the world my father was steeped in were vastly different. I was working on cutting-edge technology in the software space while he was helping design and run decades-old manufacturing facilities. It took years to learn that wasn’t the case.

Agile Software Development

I was lucky to be able to learn about software engineering while also hearing my father learn about lean manufacturing at the same time. In the beginning, I failed to see the connection between them, but as I learned and read more about this relatively new “agile” thing, I started to see the big picture. It should come as no surprise to folks that agile software development grew out of manufacturing systems developed for exactly the same reasons–to build things more efficiently.

That same desire to deliver value faster at a lower cost drives much of the innovation and growth in every business, including software companies. It’s not surprising that key leaders in the industry found those impressive innovations in industrial engineering and began adapting them to software. We know that waterfall and agile were adopted from industrial engineering, and the next big software development innovation will likely be as well.

Improve Business Efficiency

We can see many of the agile processes directly map back to either lean philosophy or even older industrial processes. Just-in-time manufacturing, or the idea that you keep an extremely limited number of raw or intermediate materials, is a philosophy that focuses on narrowing your manufacturing process to the essentials. Most agile practitioners argue similarly that your stories and shipped features should be as small as possible to deliver only the value required by the user. Both systems aim to improve business efficiency and agility by reducing potential waste (material breakage, for one, and over-engineering for the other).

All of these parallels make perfect sense to me. However, there was one component that never really clicked. A ticket and a widget are incredibly different things. Let’s explore this difference in two ways.

Traditional manufactured goods are usually tangible things. Tangible things often provide value for a fixed amount of time to a single user. For example, a delicious baked snack I pick up on a road trip provides maybe 30 seconds of value when I get back in the car; however, that car may provide 30 years’ worth of value to multiple users. Features in a software platform similarly provide value to users that vary in scale and duration. Here’s where we find an interesting divergence. A good provides a set amount of value and is often produced in bulk; a feature in a SaaS platform is often produced once and can provide an unbounded amount of value.

Compare and Contrast

Another way to see the contrast between the types of products is to look at the differences between engineers in each discipline. In a traditional manufacturing environment, the engineers are not the people actually operating the machines to produce the widgets. The engineers design the manufacturing lines and the products that will be produced on them. In software engineering, the engineers are the ones producing the features. Curious.

However, if we look closely at the two roles, we see that they are actually producing similar things. Manufacturing engineers produce assembly lines and manufacturing processes that can produce those units of value. Software engineers are largely the same, also producing systems and processes that deliver units of value. The manufactured widget of software is actually the discrete user interactions with those features and pieces of software, not the features themselves. The assembly line in software engineering isn’t, as many think, the engineers producing features. Rather, it’s software producing value-creating experiences for users.

While this is interesting, how does it help us? I’ll propose two key ways.

Manufacturing processes are extremely adaptable to managing the technology and platform. Systems like Total Quality Management, which are focused on driving a cultural mindset of continuous improvement and an entire company focused on providing very low defect rates, easily translate to customer satisfaction in software organizations. Just to pick on TQM a bit, if we were to adapt it to software, we would focus on the number of times users are impacted by a defect more than the number of open bugs. Instead of tracking the number of defects and searching for more, we would be tracking the number of users who either failed to receive the promised value from the product or had severely diminished value. That would drive investment not only at the engineering level but would create buy-in from the whole company. Everyone in the company is interested in a conversation about how many users aren’t able to gain value from a feature, but many fewer are interested in how we go from 20 open bugs to 10 open bugs.

Secondly, this mindset shift can help bridge the expectation gap between engineering and the business. I’ve been a part of many conversations where engineering has discovered a potentially serious flaw, and the whole team is in agreement that this is potentially one of the most serious bugs we’ve found in a while. However, when they bring this issue to the business side, that passion gets quickly overruled if engineering can’t show that that bug is “impacting customers.” Those debates often fall into conversations of projected odds better left to the sportsbook room in Las Vegas. While there are obvious exceptions for security exploits, compliance issues, and other business risk-driven defects, the business is justified in not investing in fixing a defect that users aren’t impacted by.

Reporting Performance Issues

Conversely, this line of thinking can help engineering convince the business that something is a critical problem and needs to be addressed. While previously, I would have reported performance issues in terms of p99 times or average load times, that doesn’t describe performance in terms of our unit of value. If our unit of value is a customer using our app or a feature, then each negative interaction is a broken widget. As an example, I’ve since taken to reporting performance issues in terms of the number of sessions with a frustratingly long load time and the number of clients and users that are experiencing those.

Manufacturing is perhaps the most varied industry in existence, adapting itself to produce nearly everything from food to raw materials to consumer electronics to clothing. That long history, many contributions from a massive number of brilliant engineers, and significant innovation have led to dozens of amazing systems for delivering value. While I proposed two examples of how we should give these processes a second look as engineering and business leaders, it’s worth dusting off some manufacturing engineering books and giving them a read yourself. You may find the key to outpacing your competitors was drawn up by an industrialist decades ago.

My first college internship was at one of the factories my father worked at in the engineering department. I was able to see firsthand how deeply some of today’s engineers think about these types of production processes. They really know how to ship stuff, and software engineers may have a lot to gain by learning from them.

Now, go build great things!

September 28, 2023