devopsy.kinsta.cloud

Category: Blogs

Data Reliability Needs to Shift Left

Data pipelines feed and transform data for consumption and are increasingly growing in complexity. As enterprises continue to add data sources, they run the risk of breaking their data pipelines due to data errors, flawed logic or insufficient resources for processing. The challenge for every modern data team is to establish data reliability as early as possible in the data journey and to create optimized data pipelines capable of performing and scaling to meet the business and technical needs of the enterprise. In the context of data observability, “shift left” refers to a proactive approach that incorporates observability practices during the early stages of the data lifecycle. This concept is borrowed from software development methodologies and emphasizes addressing potential issues and ensuring quality from the outset.

Shifting left involves integrating observability practices and tools into the data pipeline and infrastructure from the beginning. This approach avoids treating observability as an afterthought or applying it only in later stages to identify and resolve data reliability, integrity, and performance issues as early in the development process as possible, minimizing the likelihood of problems cascading downstream.

The Economic Impact of Poor Data

To effectively manage data and optimize data pipelines, it is crucial to detect and address data incidents as early as possible within the supply chain.

The “1 x 10 x 100 Rule” is applicable in a variety of processes, including software development. It states that the cost of fixing a problem increases exponentially, depending on the stage at which it is detected. The rule can be extended to data pipelines and supply chains, where it suggests that detecting and rectifying a problem in the data landing zone (i.e., where source data is fed) incurs a cost of $1, whereas identifying and addressing the issue in the transformation zone (i.e., where data is transformed into its final format) increases the cost to $10. If the problem is only detected in the consumption zone (where data is in its final format and is accessed by users), the cost rises significantly to $100.

Shift Left Data Reliability

The process of identifying data pipeline issues has become more challenging for data teams as data supply chains continue to evolve and grow in complexity, mainly resulting from the following factors:

● Expanding sources: The number of data sources (such as Databricks, Snowflake, Redshift, Teradata and many others) being incorporated into the supply chain has significantly increased. Organizations now integrate data from a wider range of internal and external sources, contributing to the complexity of data management and processing.
● Advanced data transformation logic: The logic and algorithms used to transform the data within the supply chain have become more sophisticated. Complex transformations and calculations are applied to raw data to derive meaningful insights, necessitating robust systems and processes to handle the intricacies involved.
● Resource-intensive processing: The processing requirements for data within the supply chain have significantly escalated. With larger volumes of data and more complex operations, organizations must allocate substantial computing resources, such as servers, storage and processing power, to handle the data processing workload effectively.

By proactively addressing data incidents at an early stage, organizations minimize the potential impact and cost associated with data issues. This not only ensures the reliability and accuracy of data consumed by users but also safeguards the integrity of downstream processes and decision-making. Ultimately, the shift left approach to data reliability promotes efficiency, reduces costs, and enhances overall data quality and trustworthiness.

How Can Data Teams Shift Left?

To effectively shift left in your data reliability solution, certain capabilities are crucial; these include:

● Early Data Reliability Checks: Conduct data reliability tests before data enters the data warehouse and data lakehouse. This ensures that bad data is identified and filtered out early in the data pipelines, preventing its propagation to the transformation and consumption zones.
● Support for Data-in-Motion Platforms: Enable support for data platforms like Kafka and monitor data pipelines within Spark jobs or Airflow orchestrations, which enables real-time monitoring and metering of data pipelines.
● File Support: Perform checks on different file types and capture file events to determine when incremental checks should be performed.
● Circuit-Breakers: Integrate APIs that incorporate data reliability test results into your data pipelines, which can halt data flow when bad data is detected. By preventing the spread of bad data, circuit breakers protect downstream processes from being affected.
● Data Isolation: Identify and isolate bad data rows, preventing their continued processing.
● Data Reconciliation: Implement data reconciliation capabilities to ensure consistency and synchronization of data across multiple locations.

Shifting left your data reliability practices empowers data teams to focus on innovation, reduces costs, enhances data trust and enables agile business processes. Embracing this approach makes data more usable and resilient and delivers a more robust and reliable data ecosystem that supports informed decision-making and drives organizational success.

October 23, 2023
Five Great DevOps Job Opportunities

DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

CareerBuilder.com

Johns Hopkins University Applied Physics Laboratory
Laurel, Maryland
Automation/DevOps Engineer
$160,000

Dice.com

SVMT, Inc.
Fort Meade, Maryland
Systems Engineer -DevOps
$140,000 to $170,000

Indeed.com

Hadron Industries
Remote
Infrastructure Engineer
$120,000 to $200,000

SimplyHired.com

Tesla
Fremont, California
DevOps Engineer – PowerShell
$120,000 to $152,000

LinkedIn

Discover Financial Services
Riverwoods, Illinois
Principal Infrastructure Engineer
$101,000 to $172,000

October 23, 2023
Survey Sees Spike in Untested Code Leading to a DevOps Crisis

A survey of 500 developers conducted by the market research firm OnePoll on behalf of Sauce Labs found more than two-thirds (67%) admitted they pushed code into a production environment without testing, with more than a quarter (28%) acknowledging they do so on a regular basis.

More troubling still, 60% also admitted using untested code generated by ChatGPT, with more than a quarter (26%) doing so regularly.

More than two-thirds of developers have also merged their own pull requests without a review, with 28% confessing they do so often or very often.

Finally, the survey found three-quarters of developers admitted to circumventing security protocols, with 39% of developers doing so routinely. A full 70% also acknowledged using a coworker’s credentials to circumvent restrictions for access to data and/or internal systems, with 41% doing so regularly.

Jason Baum, director for community for Sauce Labs, said that while the survey makes it clear there are a lot of instances where best practices are being ignored, much of this behavior can also be attributed to the amount of extra work developers now routinely take on. The survey found more than three-quarters of developers (77%) have, for example, assumed more responsibility for testing in the last year.

The issue organizations need to evaluate is how much of this is due to developers being lazy versus looking for shortcuts in an era where too many tasks have been shifted left to developers and which they are not able to absorb, said Baum. Many organizations might have better outcomes if more tasks were automated within the context of a DevOps workflow to reduce the amount of cognitive load being placed on developers, he added.

The simple fact of the matter is there is a fundamental skills gap, said Baum. The number of so-called full stack developers capable of managing the entire software development life cycle is relatively small, he noted.

In the meantime, the amount of code generated is steadily increasing as developers take advantage of generative AI platforms to improve productivity. The challenge is the large language models (LLMs) being used to create that code were trained using examples of code pulled from across the web. In addition to having known vulnerabilities, it is of varying quality. In fact, those LLM platforms will frequently generate different snippets of code that vary in quality at different times.

DevOps teams that are responsible for managing the overall codebase are now finding their pipelines are becoming overwhelmed. That’s becoming especially problematic at a time when governments around the world are crafting legislation that will hold organizations more accountable for the security of the applications they build and deploy.

Eventually, AI will also be used to increasingly automate DevOps workflows and should enable software engineers to keep pace with the accelerated rate at which code is being written. In the longer term, LLMs that have been trained using vetted code will also generate higher quality code than a general-purpose LLM such as ChatGPT. In the meantime, however, massive amounts of untested code will likely create a series of downstream cascading events that will come back to haunt DevOps teams in the months and years ahead.

October 20, 2023
DevOps and ITIL Integration: Driving Collaborative Agility in ITSM

Businesses face continuous pressure to deliver high-quality IT services efficiently. One emerging integration to meet these goals is between DevOps and IT service management (ITSM), more commonly known as DevOps+.

DevOps+ is becoming more than a trend and has emerged as a powerful collaborative approach to meet these challenges. By combining the strengths of DevOps’ agile software development practices and ITSM’s structured service management approach, organizations can foster better collaboration, enhance agility and improve overall service delivery efficiency.

DevOps and ITSM Symbiosis

DevOps and ITSM traditionally operated as separate frameworks with distinct focuses. DevOps emphasizes rapid software delivery and collaboration between development and operations teams. On the other hand, ITSM-based ITIL (information technology infrastructure library) provides best practices for ensuring service stability, reliability and adherence to governance and compliance.

Integrating DevOps and ITIL creates a symbiotic relationship, allowing organizations to leverage the best of both worlds. DevOps+ encourages a cultural shift, breaking down silos and promoting cross-functional communication. This collaboration enhances the entire software development lifecycle, leading to faster feedback loops, accelerated innovation, and increased responsiveness to business needs.

Enhancing Agility and Speed

DevOps+ integration brings greater agility and speed to IT service management. Organizations can streamline the release process by automating deployment pipelines and implementing continuous integration and continuous delivery (CI/CD) practices, reducing time-to-market for new features and updates. Agile development methodologies enhance the adaptability of ITSM processes to meet evolving business requirements effectively.

Ensuring Stability and Reliability

While agility is crucial, it must not compromise service stability and reliability. ITIL’s best practices provide a strong foundation for maintaining service quality. By incorporating ITIL processes such as incident management, change management and problem management, organizations can effectively manage risks, reduce service disruptions and ensure the stability of critical IT services.

Leveraging Automation and Orchestration

Automation is central to both DevOps and ITIL. Through integration, organizations can leverage automation and orchestration tools effectively. Automated testing, infrastructure provisioning and configuration management improve efficiency and consistency across the ITSM lifecycle. This enables teams to focus on higher-value tasks, reducing manual errors and operational overhead.

Emphasizing Customer-Centricity

DevOps+ puts the end-user experience at the forefront of service delivery. By continuously seeking customer feedback and measuring service performance, organizations can better align their offerings with user expectations. A customer-centric approach fosters higher satisfaction, loyalty, and competitive advantage.

Breaking Down Barriers to Change

Change management is vital in ITIL, ensuring that modifications are planned and controlled. In the context of DevOps+, change management becomes more adaptive and responsive, facilitating a culture that embraces change as an integral part of the development process. This promotes experimentation, reduces resistance to change, and accelerates innovation.

Challenges of DevOps and ITIL Integration

Despite the numerous benefits of DevOps+ integration, organizations may encounter challenges during implementation:

Cultural Shift: Merging DevOps and ITIL requires a cultural shift. Overcoming resistance to change and fostering collaboration between traditionally separate teams can be challenging but is crucial for success.

Process Alignment: DevOps and ITIL have different processes and terminologies. Aligning these methodologies while preserving their individual strengths can be complex.

Skill Sets: Integrating DevOps and ITIL necessitates upskilling teams in both domains to operate effectively in the integrated environment.

Tooling and Automation: Adopting suitable tools and technologies that support DevOps+ practices and integration is vital but may require significant investments.

Metrics and Performance Measurement

Measuring the success of DevOps+ integration is essential to understand its impact and effectiveness. Key performance indicators (KPIs) can help monitor progress and identify areas for improvement.

Deployment Frequency: The frequency of software deployments indicates the efficiency of the integrated approach. Higher deployment frequency implies faster and more reliable releases.

Mean Time to Resolve Incidents (MTTR): MTTR measures how quickly teams can resolve incidents and restore services after disruptions occur. A reduced MTTR demonstrates improved service reliability and faster incident resolution.

Customer Satisfaction Scores: Measuring customer satisfaction provides insights into how well the integrated approach meets end-users’ needs and expectations.

Continuous Improvement and Feedback Loops

Continuous improvement is at the core of DevOps+. Establishing feedback loops and conducting regular retrospectives empowers teams to identify areas for improvement and implement iterative enhancements to processes and practices.

Implementing DevOps+ integration requires a well-structured roadmap. For example, assessment and planning help leaders evaluate the organization’s current ITSM practices to understand how DevOps and ITIL principles align and define integration vision and goals.

Additionally, your continuous optimization efforts allow you to continuously monitor performance metrics, gather feedback, and conduct retrospectives to identify improvement areas, leading to better team collaboration and communication. Fostering cross-functional collaboration between development, operations and other relevant teams is easily accomplished. To meet these goals, encourage regular meetings, schedule joint planning sessions and enable shared documentation to ensure everyone is aligned on goals and progress.

When this happens, leverage automation tools to streamline manual processes and increase efficiency. Implement continuous integration and continuous delivery pipelines to automate software deployment and testing. Automation enhances consistency, reduces the risk of human errors and adds a new layer for measuring and evaluating to continuously measure the performance of the integrated approach using defined KPIs. In this vein, you can regularly review progress against the initial vision and goals to ensure the integration aligns with organizational objectives.

Next, as the DevOps+ integration proves successful in pilot projects, gradually scale and expand it to other teams and departments. Share the success stories and best practices to encourage broader organizational adoption.

Conclusion

In conclusion, the integration of DevOps and ITIL, known as DevOps+, marks a transformative approach to IT service management. By combining the agility and rapid deployment of DevOps with the stability and reliability of ITIL, organizations can drive collaborative agility and achieve efficient service delivery.

DevOps+ integration requires overcoming cultural barriers, aligning processes and upskilling teams, but the rewards are immense. With clear metrics to measure success, organizations can continually optimize their DevOps+ approach to meet customer needs and adapt to evolving market demands.

As technology evolves, DevOps+ integration will remain at the forefront of cutting-edge IT service management, empowering organizations to embark on a journey of innovation, efficiency and excellence.

By embracing DevOps+ as a guiding philosophy, organizations can cultivate a culture of collaboration, continuous improvement and customer-centricity, laying the groundwork for a prosperous future in the dynamic world of IT service management.

October 20, 2023
Survey Surfaces Benefits of Applying AI to FinOps

A survey of 200 enterprise IT decision-makers published this week found organizations that have infused artificial intelligence (AI) into financial operations (FinOps) workflows to reduce IT costs are 53% more likely to report cost savings of more than 20%.

Conducted by the market research firm Foundry on behalf of Tangoe, a provider of tools for managing IT and telecommunications expenses, the survey found organizations that embraced FinOps without any AI capabilities averaged less than 10% in cost savings.

The top three drivers for adopting FinOps/cloud cost management programs are the need to increase cloud resource production and performance (70%), reduce budgets (60%) and rising costs (58%), and simpler overall program management (50%), the survey found. Major benefits included productivity savings (46%), cost savings (43%) and reduced security risks (43%). Nearly two-thirds of respondents cited service utilization and right-sizing of services as another reason to embrace FinOps.

FinOps describes a methodology for embedding programmatic controls within DevOps workflows to reduce costs. In the face of increased economic headwinds, IT leaders are looking to reduce cloud computing costs, but it’s turning out to be more challenging than many of them anticipated. Cloud infrastructure is typically provisioned by developers using infrastructure-as-code (IaC) tools with little to no supervision. The reason for this is developers have long argued that waiting for an IT team to provision cloud infrastructure took too long. Developers would be more productive if they just provisioned cloud infrastructure themselves.

However, after ten years of cloud computing, it’s become apparent there are a lot of wasted cloud infrastructure resources. Developers who don’t pay the monthly bills for cloud services tend to view available infrastructure resources as essentially infinite. It’s usually not until someone from the finance department starts raising cost concerns that developers even become aware there might be an issue.

The challenge is that adopting FinOps best practices is not quite as easy as it might seem. In fact, more than half (54%) of survey respondents cited challenges in building the right process and human support systems for FinOps into workflows that have been in place for years.

Chris Ortbals, chief product officer for Tangoe, said the simplest path to FinOps is to rely on a software-as-a-service (SaaS) platform designed from the ground up to leverage AI to help IT teams manage cloud computing and telecommunications expenses both before and after applications are deployed. Each DevOps team will ultimately need to determine how much they will implement metrics to foster more efficient consumption of cloud computing resources. The more aware of those costs DevOps teams are, the more likely that better decisions about what types of workloads should be run where and, just as importantly in the age of the cloud, at what time, given all the pricing options provided.

Developers, of course, tend to jealously guard their prerogatives. Convincing them to give up their ability to provision cloud infrastructure on demand is going to be a challenge, at least until someone makes it plain how much all those cloud instances wind up costing the organization each and every month.

October 19, 2023
Netlify Unfurls Composable Jamstack Framework for Web Apps

At its Compose 2023 conference, Netlify today launched a composable framework that promises to make it simpler to build web applications spanning multiple backend services.

Netlify CEO Matt Biilmann said the Netlify Composable Web Platform provides a more holistic approach for loosely coupling the frontend of a web application with backend services that can be swapped out as needed.

The goal is to make it simpler to unify content, data sources, code and infrastructure at a higher level of abstraction while providing development teams with a maximum amount of flexibility, he added.

The Netlify Composable Web Platform consists of Netlify Core to manage workflows, Netlify Connect to access content and data sources and Netlify Create to build applications using JavaScript, application programming interfaces (APIs) and markup languages, otherwise known as Jamstack, framework.

Collectively, the tools will enable development teams to move away from monolithic web applications that are more difficult to update and maintain, noted Biilmann. Instead, development teams can compose applications faster using a framework that automates many of the underlying processes that conspire to slow down application development, he added.

In the face of challenging economic headwinds, interest in improving developer productivity is rising because organizations need their existing teams to consistently deliver applications faster. Most organizations can’t afford to continue expanding development teams when the current processes relied on to build software result in developers spending only a quarter of their time writing code.

There is also a greater need for collaboration using a framework that enables different members of a development team to build elements of an application in isolation from one another without impacting the entire application. Each organization will need to decide for itself whether it wants to make applications from multiple services, but at a time when the speed of adding new capabilities to an application is core to an organization’s ability to differentiate itself, the need for a different methodology for building applications has never been more apparent.

There is, of course, no shortage of frameworks for building web applications, but DevOps teams have a vested interest in making sure that whichever one is adopted is based on a modern architecture. The time and effort required to support monolithic applications in an era where new capabilities need to be added regularly has become cost-prohibitive. The challenge is finding a framework that enables developers to quickly build applications that can be deployed within the context of a DevOps workflow that typically needs to support applications being developed in multiple programming languages.

It’s not clear who inside IT organizations is driving adoption of composable frameworks for building the next generation of applications, but as more organizations become dependent on software, legacy approaches to building monolithic applications have become antiquated. The good news is that in terms of the developer talent pool, there are large numbers that are already familiar with multiple variants of the JavaScript programming language. The challenge, as always, is making sure that whatever code finds its way into a production environment is actually reliable enough to consistently drive a business process.

October 19, 2023
Preparing for the Shift to Platform Engineering

Platform engineering is one of the hottest trends in the software delivery world and is expected to represent the next evolution of DevOps. According to Gartner’s 2023 study of top strategic technology, 80% of software engineering organizations will establish platform teams as internal providers of reusable services, components and tools for application delivery by 2026. What used to happen in one day in the developer world now needs to happen in five minutes. With high pressure for organizations to scale operations quickly, especially in a marketplace clamoring for the benefits of artificial intelligence (AI), automation is increasingly important in platform engineering. It reduces manual errors, streamlines the development process and boosts growth and efficiency. As technology advances rapidly, it becomes more critical for organizations to proactively navigate newly emerging trends within platform engineering to remain competitive. The keys to successful platform engineering include cloud-native architecture, AI and machine learning (ML), and evolving tools that use intuitive interfaces and pre-built components to create functional applications.

Platform Engineering is the Natural Progression of DevOps

Platform engineering expands on the practices of DevOps by moving development out of silos and taking a collaborative, bird’s-eye view of an organization’s technology. Platform engineering is built on a foundational infrastructure that enables self-service capabilities so developers can deploy code faster, more reliably and more securely. While platform engineering represents the next wave of change in DevOps, it’s essential for leaders and developers to understand that it isn’t a singular technology or approach. The shift to platform engineering also represents a cultural shift for developers and leaders to address the needs, challenges and rapid pace of modern software development by optimizing processes, accelerating delivery and reducing the complexity of operations. For platform engineering to succeed, it’s critical to approach it from the perspective of a technology-driven culture that creates solutions with a developer’s perspective in mind. By seeking to improve the developer experience first, enterprises can achieve better results.

Trends Shaping the Landscape

Platform engineering itself is an emerging trend, but several essential ingredients shape how it will impact businesses today and in the future. For organizations to reap its benefits, understanding these essential trends will keep them ahead of the pack.

Cloud-native architecture is fundamental to modern platform engineering by designing and building applications for cloud environments. Infrastructure-as-code (IaC) is a key component, enabling the automated provisioning and management of resources. Cloud-native platforms like Kubernetes, Terraform or Docker facilitate container orchestration and scaling, while serverless computing isolates infrastructure management entirely. This trend ensures that platform engineering is agile, scalable and cost-effective, with minimal downtime and easier maintenance.

Automation is pivotal in platform engineering, reducing manual errors and streamlining deployment workflows. It allows developers to build, test and deploy software more efficiently, leading to faster and more reliable product releases. In fact, 43.2% of early adopters of platform engineering saw faster product delivery. Automation tools like Jenkins, Ansible and Terraform are commonly used to automate infrastructure provisioning, configuration management and deployment pipelines. Machine learning (ML) models can perform automated troubleshooting by analyzing historical data to identify patterns and predict potential issues, proactively addressing problems and minimizing downtime. This also helps with performance optimization. By using AI that can analyze application performance data, automation takes platform engineering to the next level by suggesting optimizations to improve resource utilization and user experience. AI-powered tools assist in generating code or script, which is vital to reducing development time and human error. ML algorithms empower the security side of platform engineering, a top priority for any developer. It’s possible to leverage automated security tests during development and maintenance, even detecting and responding to security threats in real-time. When AI handles routine tasks, developers are less likely to feel overwhelmed or experience burnout. This allows developers to focus on more creative tasks, complex problem-solving and innovation, resulting in more efficient operations to help them achieve higher levels of productivity, security and consistency.

Evolving development tools that create functional applications using intuitive interfaces and pre-built components increasingly simplify application development and deployment. Low-code and no-code platforms allow developers to create applications with minimal coding effort, utilizing visual interfaces and pre-built modules. Serverless frameworks like AWS Lambda and Azure Functions enable developers to focus solely on writing code. An increasing reliance on and advancement of application programming interface (API) gateways provides a simplified way to manage and expose APIs, enhancing application integration and connectivity. Containerization platforms are becoming similarly powerful tools. Platforms like Docker and Kubernetes simplify application deployment across diverse environments by offering standardized containers and orchestration. Adopting a strategic suite of these evolving tools requires a clear understanding of product goals, organizational goals and developmental capabilities.

How Leaders can Support Platform Engineering

It’s projected that 30% of organizations are currently evaluating the adoption of platform engineering practices, and its prevalence will only grow. While platform engineering’s trends are empowered by digital transformation and new technologies, it’s vital to remember that platform engineering represents a shift in DevOps culture. To effectively support the transition, leaders must commit to a culture of platform engineering. Simply adopting technology isn’t enough. It needs to be backed by a thorough strategy that allows developers to truly benefit from the tools and structures of platform engineering. What does this look like? Success requires leaders and developers to encourage collaboration and break down silos between operations and development teams. It’s possible to build a bridge between developers and operations by committing to cloud migration, creating a centralized platform and investing in collaborative tools and the strategy to back it up.

To engage in platform engineering requires dedication to a collaborative culture instigated from the top, empowered by overall strategic decisions and operations. This includes continued learning for developers to stay on top of new languages, trends, challenges and priorities, internally and externally. Teams are more successful when they utilize performance metrics to track workflows that help them conduct effective maintenance and improve on a consistent and ongoing basis.

Embracing Change is Embracing Progress

In the fast-paced and continuously improving world of software development, it’s easy to think that each wave of technology will replace the last. Of course, experts know this isn’t true. Iteration is vital to advancing technologies, and it’s important to embrace the next iteration of progress to stay competitive. Automation and platform engineering are tools that enable developers rather than replace them. It’s important to note that platform engineering doesn’t mark the end of DevOps. Rather, it takes the core foundations that DevOps has built to the next level to empower developers with self-service capabilities, foster collaboration, enhance scalability, accelerate deployment and instigate sustained business growth.

October 19, 2023
Multi-Tenant Database Strategies for Mixed Workloads and Real-Time SLAs

Multi-tenancy—where a single instance or multiple instances of one or more applications operate in a single shared infrastructure environment—is not a new concept. First used in mainframe hardware, timesharing enabled a single massive piece of computing iron to be sliced up and used by multiple applications on a single hardware instance. This simplified both the hardware landscape as well as the skills and resources needed to manage environments. With the rise of software-as-a-service (SaaS) in today’s modern private and public cloud environments, multi-tenancy is now a common and critical function. It allows cloud companies to scale and customers to save money by essentially pooling resources. As more applications demand more and more data, multi-tenancy is now a core function at the database layer.

But when you have a lot of different applications running on the same infrastructure and accessing the same database, how do you prevent one application from impacting the data performance of another? And how do you determine from the underlying database infrastructure which tenants get what infrastructure and at what performance level?

Here’s a look at the driving forces behind today’s real-time data needs and how they impact multi-tenant data architecture management.

SLAs Are at the Heart of Multi-Tenant Data Management

Everyone wants faster performance and more resilient systems. We’re impatient, so reliable, real-time performance delivers large, competitive business benefits. However, every tenant does not require the same level of performance. Actual performance needs may be plenty fast for a user or machine without requiring the fastest execution path possible with the available resources.

Service-level agreements (SLAs) ultimately govern the resources delivered to a tenant. Real-time may need to be measured in sub-milliseconds in one instance (i.e., for a stock trade) but in sub-seconds for others (i.e., in a Zelle funds transfer). This is where developers, architects and operators partner with a business to understand its actual user or system needs and to build a system that can deliver the SLA using the minimum amount of resources possible.

Mixed Workloads and the ‘Noisy Neighbor’ Problem

As soon as workloads are shared, there’s the potential for one application to quickly grab and use resources in a way that impacts other applications. Often called a “noisy neighbor,” the impacting applications can dramatically interfere with real-time response time SLAs when multi-tenant apps share an infrastructure base. For example, one node in a database cluster becomes heavily loaded, and the performance of other nodes begins to be impacted, causing SLAs to be missed.

Typically, multiple virtual machines (VMs) are hosted on a single physical machine or node but share system resources (i.e., processor, network, storage, etc.) with the other VMs. It’s possible for applications to overuse those shared resources, impacting other applications and sometimes even the database itself. Such overusage of the infrastructure layer (in this case, sharing compute, storage, network queues, etc.) leads to poor database and end-to-end system performance. It’s like plugging a 100-amp appliance into a 50-amp circuit. All the circuit breakers will trip, and the entire system comes to a screeching halt!

Strategies to Maximize Performance for Database Multi-Tenancy

Configuring and managing the database itself (not just infrastructure) with multi-tenancy in mind can maximize performance and avoid problems like noisy neighbors. Isolating databases in your multi-tenant architecture can help minimize non-database problems that can be the root cause of poor performance and missed SLAs.

Start by picking databases that allow applications to isolate their data access based on sets (or tables in relational parlance) that can have storage limits attached to them. When combined with rate limits for transactions on a per-user basis, applications can be guaranteed their quota of runtime resources. Using the per-app quota configurations for items like storage and TPS (transactions per second), the database can evaluate the total capacity available in the system and ensure that enough resources are reserved and available to ensure that the SLAs of all applications are met all the time.

Note that unexpected events that cause rebalancing of data within the distributed database, such as node and network failures, require additional resources. The database system will subtract that resource capacity from what is available for apps so that the system is not overcommitted during failure situations.

Enterprises also need very good automation and observability to fix immediate problems quickly and have the requisite visibility into long-term trends that impact performance and SLA compliance. Otherwise, by the time a human notices a problem, the SLA may already have been missed and the business impacted.

Good observability can help you understand how data is used and create those automations. As more applications access and use the same data set in different ways, you need modern observability tools integrated into (or interoperable with) the underlying database.

With a good understanding of how your data-rich applications interoperate with other tenants in mixed-load environments, you can begin to deploy these strategies to make sure you conform to all required SLAs and keep all that data working hard for the business.

October 19, 2023
How to Build a Data Platform for Self-Service, Ad-Hoc Analytics

In a digital world, data is often the differentiator between success and failure. Whether it’s defending against cybersecurity threats, improving application performance, or resolving full-blown outages, data is critical to the modern DevOps team.

In fact, there are plenty of situations where DevOps engineers, SRE teams and other observability-focused departments need to explore reams of data rapidly and flexibly. An SRE trying to figure out the root cause of latency may need to dissect the movement of data across endpoints to find the malfunctioning one. Another engineer may need to compare historical and current performance to find anomalous behavior. Lastly, a DevOps engineer may have to dissect user performance metrics to understand the scope of a slowdown—whether it is global or regional.

Every use case requires dashboards that can provide detailed data through a wide variety of visualizations, including choropleth maps, stack areas, pie charts and bar graphs. Ideally, a dashboard will enable teams to isolate dimensions, apply filters, and dive into data for deeper insights.

Unfortunately, not all dashboards can support this type of fast, flexible analysis. Many still utilize outdated technologies, which lack the flexibility, agility and scalability required to seamlessly explore data. Many were also designed without the urgency that today’s data requires—after all, in previous years, most uses for dashboards (such as internal reporting) were not time-sensitive.

Other dashboards are constrained by templates, which offer a finite array of widgets, tools and drill-down capabilities. These dashboards may feel unwieldy and sluggish, unsuited for a rapidly evolving situation like an application outage.

Requirements

To truly fulfill the promise of self-service, ad-hoc analytics across huge datasets, teams need to work in real-time—and so they need a database capable of timely responses.

When an application goes down, an SRE may not know what to look for and, hence, needs to quickly comb through lots of data. While some dashboards rely on workarounds for faster queries, such as pre-aggregations, precomputing, or rollups, this is not possible in this instance because the SRE simply won’t know what to look for. After all, they can’t necessarily predict what will go wrong, and even if they work on assumptions from previous failures, this current issue could be much different.

Therefore, dashboards must provide plenty of functionalities and visualizations. Users should be able to filter data by time, isolate variables like location, and zoom into specific time intervals with a few clicks. Dashboards should also accommodate diverse data types, including intricate parent-child relationships and nested columns.

Further, dashboards must offer a depth of insight. An online streaming media platform may need to assess user metrics (such as latency or load times) across different devices, operating systems, and regions to fine-tune performance. A cloud provider has to monitor their physical hardware for high temperatures, slowed network switches or devices, and other anomalies.

Because data is now so important to success, many more people across a company, including data scientists, product managers and external users, themselves require data-driven insights. In these situations, dashboards must handle the increased user traffic and query activity, maintaining fast responses even under load. This is especially important considering that a single-user operation (such as a zoom) will require multiple queries on the backend to execute.

A database also must scale seamlessly. If an organization’s environment generates millions of events an hour, that equates to billions of events per day or week—challenging for any database to ingest, store, analyze and query. In fact, many databases cannot successfully provide fast response times while managing large datasets and high query volume. As an example, transactional databases (OLTP) can often query rapidly but cannot execute analytics at scale, while analytical databases (OLAP) can analyze massive volumes of data but not at speed.

Apache Druid for Independent, Ad-hoc Data Exploration

This is where open source Apache Druid comes in. Combining the scale and advanced analytics of an OLAP database with the speed of an OLTP database, Druid offers swift, real-time data exploration.

Upon ingesting data, Druid makes it immediately available for querying and analysis, removing the need to first batch or aggregate the data in some way. In addition, Druid natively integrates with streaming technologies like Apache Kafka and Amazon Kinesis, removing the need for workarounds or connectors.

Druid powers interactive visualizations, providing millisecond response times, enabling more versatile exploration, expanding the range of available dimensions and filters, and even maintaining subsecond speeds in the face of surging user and query volumes.

Druid’s unique architecture also enables easy scaling. By separating key duties among separate node types—data nodes for storage, master nodes for data ingestion and availability, and query nodes for executing queries and returning results via the scatter/gather method—Druid ensures that nodes can be independently scaled based on need. Afterward, Druid also automatically rebalances traffic to ensure consistent performance.

Salesforce: A Druid Success Story

Salesforce pioneered the customer relationship management (CRM) space, serves 150,000 customers worldwide and earns billions in annual revenue.

The Edge Intelligence Team is the division of Salesforce that tackles the massive task of ingesting, processing, filtering, aggregating, and querying the entirety of their log data — anywhere from billions to trillions of lines daily. Each minute, Salesforce ingests 200 million metrics, while each day, Salesforce processes five billion daily events globally. In total, Salesforce accumulates dozens of petabytes of data in their transactional store, five petabytes of logs in their data centers, and almost 200 petabytes in their Hadoop storage.

Salesforce teams use Druid to unlock real-time insights into product performance and user experiences, diving into large datasets instantly. Anyone from engineers to account executives can query a wide variety of dimensions, filters and aggregations to better understand trends, troubleshoot any issues that arise and set strategies for the future.

By using Druid’s compaction abilities, Salesforce also decreased the number of Druid rows by 82%, leading to an overall reduction of their storage footprint by 47% and accelerating their query response times by 30%.

October 19, 2023
Post service upgrade pods are taking too long to serve traffic – Answers
After a service upgrade the pods are taking too long to serve traffic. What would you need to find out to debug?
1. The validating webhook for pods has failed
2. The cluster doesn’t have enough CPU/memory to schedule the pod to a node
3. The new code added an init container and the init container is taking a long time to finish
4. The new deployment changed the readiness probe
5. There is a bug in the code

100 people answered the question. And their answers are reflected in the chart below.

Each of the provided choices is indeed a correct answer – meaning 16 people got the right answer. Let’s breakdown each of the choices in the context of troubleshooting slow pod response times after a service upgrade:

The validating webhook for pods has failed: Validating webhooks in Kubernetes are responsible for ensuring that pods conform to certain policies before being created. If the validating webhook fails, it can prevent pod creation altogether, which would certainly impact the ability of new pods to serve traffic. Therefore, this is a correct choice to consider when debugging service issues.

The cluster doesn’t have enough CPU/memory to schedule the pod to a node: When a service is upgraded, its resource requirements may change. If the cluster lacks the necessary CPU or memory resources to schedule the new pods, it can lead to delayed pod scheduling and slow response times. This is a common issue that SREs and DevOps need to investigate when traffic-serving problems arise after an upgrade.

The new code added an init container, and the init container is taking a long time to finish: Init containers are executed before the main application containers in a pod. If a new init container is introduced during an upgrade and it takes an extended time to complete its tasks, it can indeed cause delays in the pod becoming ready to serve traffic. This is a relevant consideration when diagnosing slow response times post-upgrade.

The new deployment changed the readiness probe: The readiness probe determines when a pod is ready to accept traffic. If the readiness probe is modified during an upgrade and is now misconfigured, it can delay the point at which the pod is considered ready to serve traffic.

Therefore, this is an accurate choice for troubleshooting slow response times.

One of our earlier quiz questions dealt with a similar issue which we discussed in the following blog: https://webb.ai/blog/quiz-1-aug-1-23-healthcheck-port-mismatch/

There is a bug in the code: While the broadest option, it’s also valid. Code bugs can absolutely result in slow response times. However, it’s crucial to note that this choice doesn’t specify what type of bug or where in the code it might be. It highlights the importance of code review and debugging as part of the troubleshooting process.

In summary, all the provided choices are correct because they represent various potential causes of slow pod response times after a service upgrade. Troubleshooting such issues often requires considering a combination of factors, from resource constraints to configuration changes and code-related problems. SREs and DevOps professionals must approach the problem with a holistic perspective to identify and address the root causes effectively.

There are additional reasons, not included in the options provided, for slow pod response times after a service upgrade:

Network Latency/Issues: Connectivity problems, network misconfigurations, or high network latency between pods and services can cause delays in serving traffic.

Storage Latency: Slow storage I/O or inadequate storage resources can impact data retrieval and processing, affecting pod performance.

Resource Contentions: Other resource-intensive workloads on the same nodes can lead to resource contentions, resulting in slow pod response times due to competition for CPU, memory, or disk resources.

Analyzing each potential root cause for each failure is not possible in today’s highly dynamic environments. Leveraging software and the recent advancements in AI to automate root cause analysis is one way to reduce toil and increase productivity allowing SRE and DevOps to become proactive and focus on addressing more complex failure scenarios.

–

In a Kubernetes environment, a recent pod scheduling failure occurred due to a specific configuration. Which Kubernetes resource type, often associated with node constraints, might have caused this failure, especially if it wasn’t defined correctly?
1. A) NodeSelector B) ResourceQuota C) PriorityClass D) Taint E) PodDisruptionBudget
From an SRE perspective, let’s explain why “Taint” is the right answer in the context of a pod scheduling failure due to a specific configuration, and why the other choices are not as suitable:

Taint (Correct Answer):

Taints in Kubernetes are node-level attributes that can be applied to nodes to affect pod scheduling. When a node is tainted, it essentially broadcasts a constraint to pods that they should not be scheduled on that node unless they have a corresponding “toleration.” Here’s why taints are the correct answer:

Node Constraints: Taints are directly related to node constraints. They allow you to specify criteria that restrict which pods can run on specific nodes based on attributes like hardware, software, or other node characteristics. This makes them a crucial resource for controlling where certain workloads are placed within the cluster.

Pod Scheduling: When taints are applied to nodes and pods do not have matching tolerations, they will not be scheduled on those tainted nodes. If a pod is failing to schedule due to a node constraint issue, it’s likely because of taints.

NodeSelector (Not the Best Choice):

NodeSelector is a Kubernetes feature that allows you to set node affinity for your pods based on labels assigned to nodes. While it does influence pod scheduling, it is not primarily associated with node constraints set at the node level like taints.

Node Affinity: NodeSelector is more about node affinity (i.e., preferring nodes with certain labels) rather than constraints. It doesn’t directly prevent pod scheduling but rather guides the scheduler’s preference.

ResourceQuota (Not Related to Node Constraints):

ResourceQuotas are Kubernetes objects that limit resource consumption (CPU, memory, etc.) within namespaces. They do not directly influence pod scheduling based on node constraints, making them less relevant to the given scenario.

Resource Limitation: ResourceQuotas control resource usage within namespaces, but they do not define node-specific constraints or affect where pods can be scheduled within the cluster.

PriorityClass (Not Related to Node Constraints):

PriorityClasses are used to prioritize pods in scheduling order, but they do not define node constraints like taints. They affect the order in which pods are scheduled but do not directly relate to why a pod may fail to schedule due to node-specific constraints.

Scheduling Priority: PriorityClasses are about setting scheduling priorities, not specifying where pods can or cannot run based on node characteristics.

PodDisruptionBudget (Not Related to Node Constraints):

PodDisruptionBudgets are used to control the disruption of pods during voluntary disruptions (e.g., draining a node). They are not associated with node constraints or pod scheduling based on node attributes.

Disruption Control: PodDisruptionBudgets are for controlling disruptions during node maintenance or other planned events, but they do not deal with node constraints affecting pod scheduling.

In summary, when debugging a pod scheduling failure due to a specific configuration, especially one related to node constraints, “Taint” is the most appropriate answer because taints directly influence pod scheduling based on node attributes, whereas the other options are not primarily associated with this aspect of Kubernetes resource management.
October 18, 2023