devopsy.kinsta.cloud

Category: DataOps

How to Optimize Data Platforms for Ultimate Efficiency

Data is the lifeblood of modern business. Nearly everything we interact with in our digital lives involves data management in some form. And large companies often deal with many unoptimized data-related operations, such as inefficient queries, un-optimized data workloads or large ungoverned data lakes.

I recently chatted with Capital One Software’s VP & Head of Slingshot Engineering, Salim Syed, on optimizing data operations to maximize efficiency. Below, we’ll consider how data management changes when migrating to the cloud and review some actionable ways technical leaders can direct more efficient DataOps.

Transitioning to Cloud-First Thinking

Arguably, the first and highest hurdle to overcome when optimizing data operations is transitioning from an older, on-premises way of thinking about data management to a newer, cloud-first model, Syed explained, because the two paradigms couldn’t be any more different.

Simply replicating on-premises behaviors could be bad for managing databases in the cloud because it enables unlimited power compared to on-premises computing. “When you go to the cloud, the shackles break,” said Syed. Cloud technologies allow for infinite scalability, but such power comes with a significantly higher degree of responsibility.

Another potential holdover is the organizational structure. Having a centralized team oversee a lot of controls could end up stifling innovation, he said. Yet, federating too many platform teams could have a siloing effect. Therefore, he recommended defining policies and regulations centrally and making them easy to enact with self-service tooling.

Ways to Optimize DataOps For Efficiency

Capital One was an early mover to the cloud. Syed explained that Capital One started its tech transformation journey in 2012 and exited its last data centers in 2020. It’s no surprise that, as a cloud-based digital financial company, they deal with extremely large amounts of data. For example, thousands of analysts rely on financial data and other sources to generate their insights.

Therefore, the company must constantly balance empowerment with operational efficiency for interacting with large quantities of data, Syed said. With this background, he shared some tips they’ve been using to enhance efficiencies within their data-based operations.

Avoid Inefficient Queries

Database queries used to have a minimal impact on resource consumption when you owned your infrastructure. But this all changes in the cloud era. For instance, poorly written, inefficient queries programmed to scan billions of rows could result in unnecessary computing effort and high cloud spending. Therefore, designing queries with smarter ranges can help avoid waste.

Syed shared that, at Capital One, his team has developed a query advisor that can look at prior patterns and give recommendations, such as fixing the range, joining queries or reducing the number of rows. By examining past query patterns and looking at the related metadata, the system can advise better-constructed queries that improve efficiency.

Optimize the Datasets

In addition to optimizing queries, database engineers should keep an eye on optimizing the datasets themselves. Because how you structure your tables and the data model itself is even more critical in the cloud world, said Syed.

Furthermore, the access pattern should be given forethought. Although some scenarios require real-time processing, it can be costly and isn’t always necessary. Therefore, Syed recommended understanding your access pattern for loading data and its associated cost implications.

Have a Data Retention Strategy

Next, it’s easy to let data accumulate over time. But, having a ‘set it and forget it’ policy for data is not a wise strategy for the cloud. Instead of letting petabytes of information accrue and produce excessive storage fees, it’s best to prune your data lakes where possible so they aren’t retained indefinitely, said Syed, and set policies for storage retention and deletion timelines upfront.

Monitor For Inefficiencies

A certain degree of inefficiencies with data operations will be inevitable, acknowledged Syed. For instance, this may be an instance accidentally left running or provisioning large clusters to do basic functions. Nevertheless, there are certain actions engineers can take to remediate inefficiencies as they arise. Part of this is identifying inefficiencies in real-time and correlating them to actionable changes. He encourages leaders to create learning opportunities out of these events and to spread internal knowledge.

Match the Technology to the Use Case At Hand

Lastly, and perhaps most importantly, are you using the right technology for the use case at hand? There are many types of database styles to choose from these days, from relational databases like MySQL or PostgreSQL to non-relational databases like NoSQL or MongoDB. Matching the right data technology to the use case at hand will be an important step toward improving efficiency.

Furthermore, Syed recommended right-sizing the compute for the workload as well. For instance, for his team at Capital One, Syed explained that the policy is not to have large-size computing in lower-level environments, as you shouldn’t need to process a lot of data there.

Smart DataOps Aids FinOps Objectives

The above strategies could be adopted to avoid wasted resources. Moreover, these tactics to optimize data processes could also aid FinOps, which seeks to optimize operations to reduce IT expenses. But reviewing data efficiencies should be done often—you have to be vigilant as new workloads are introduced, Syed cautioned.

Image source: joshua-sortino-LqKhnDzSF-8-unsplash.jpg

November 3, 2023
Measuring the Progress of the OpenTelemetry Project

The biggest compliment you can pay a technology standard is to say that it is “boring.”

Transport layer standards like TCP and UDP are boring. They are the underlying protocols powering pretty much everything we do on the internet. REST is boring. It brokers trillions of API transactions across the world’s distributed microservices.

Infrastructure standards get boring when they become common infrastructure plumbing that “just works.” They give open source creators and maintainers common foundations to innovate on top of. They give users confidence that their technology choices aren’t leading to dead ends. Look at any major evolution in computing, and it’s always propped up by boring standards.

When OpenTelemetry debuted at KubeCon in 2019, telemetry data for applications in distributed systems was far from boring. Two competing projects—OpenTracing and OpenCensus—were trying to encapsulate best practices around the still nascent distributed tracing discipline. They were “very similar in that they aim[ed] to unify app instrumentation and make observability a built-in feature in every modern application” (Merging OpenTracing and OpenCensus: Goals and Non-Goals). But having two fragmented communities around these two projects made it hard for distributed tracing to move forward—so it was a big deal when the two projects merged and OpenTelemetry was born.

Let’s take a look at how the OpenTelemetry standard is making this all possible.

Standard Semantic Conventions and a Single ‘Mental Model’

Semantic conventions are important to standardize how we describe common operations so that we have consistent telemetry for similar actions. Developers instrumenting applications need to be able to relate to clear semantic definitions to understand how to instrument everything from database drivers to SQL statements and how to consume that telemetry data for analysis.

For example, OpenTelemetry semantic conventions define where in the metadata the process name should be stored, but not what the process name should be. Similarly, if I’m deploying my application on Kubernetes on a cloud provider somewhere, I’d expect my telemetry data to have the Kubernetes pod and namespace it originates from, as well as the cluster and region.

OpenTelemetry also standardizes metadata for labels that span languages. So if I have an HTTP service and request, it doesn’t matter if it is in Java, Python or any other language — with OpenTelemetry, I can just start querying the data without having to think about the language behind it.

It brings a consistent mental model to how the different telemetry data types (traces, metrics, logs) are related: OpenTelemetry gives them the same base, so understanding how a span is modeled provides knowledge that can be reused when understanding metrics and logs. It can be used by any microservice and any HTTP- or gRPC-connected service or device. For any network device, OpenTelemetry gives developers semantic conventions that bring consistency to reporting telemetry data on any type of event. So, any engineer working with OpenTelemetry can compare latencies and queries between services—even if they are written in different languages—because of the semantic conventions.

Protecting Users From Vendor Lock-In

A decade ago, monitoring and logging vendors provided agents to their customers. They didn’t call it observability back then. But they told the market that if you want the best telemetry data, you need to use our agent. Every vendor had its own agent.

Adding these agents was risky and took a lot of time. Once a vendor got its agents into a customer’s environment, they were hard to change and you had to either pay the vendor or your team of engineers to remove that instrumentation in favor of another.

OpenTelemetry really commoditized the telemetry definition in such a way that we don’t need to use agents from vendors anymore. This means users can decide which vendor to use for observability tooling and have the flexibility to switch to different vendors down the line. With OpenTelemetry, they no longer have to instrument applications, restart applications and get into risky operations cycles to change the destination of their telemetry data.

Economies of Community – Scale for Drivers and Integrations

OpenTelemetry has achieved a very large and diverse set of contributors in a relatively short period of time. It’s become the second most popular CNCF project, behind only Kubernetes.

The project is drawing contributions both on the instrumentation side of things, as well as the collection side of things. Specifications are another big area where the project is evolving, including the API/SDK spec, the data model and the OTel protocol (OTLP). Accelerating all of this is the fact that OpenTelemetry has broad participation from the logging, monitoring and observability vendor ecosystem, which comprises many of its largest contributors.

We still have a ways to go. But in the last year, it’s become obvious that OpenTelemetry is so commonly regarded as the standard for tracing data that the language, library and database creators and maintainers are instrumenting for OpenTelemetry. This is the ultimate sign of maturity for any project or standard—when its popularity has created pull from maintainers designing around the spec and using it natively.

What’s Next For OpenTelemetry?

Looking to the future, I think we’re going to see an unintended consequence where OpenTelemetry moves the industry from operational data to more data intelligence. OpenTelemetry makes it much easier to instrument how users are interacting with systems and enables developers to add business metrics and compare those against actual user behavior.

OpenTelemetry also has a great potential marriage with eBPF and related projects like Cilium. These abstractions that make the Linux kernel “programmable” add a pool of much finer-grained telemetry data that can be extracted into logs from applications during run-time.

This all feeds into the concept of “Profiles” that OpenTelemetry is implementing. While distributed tracing emphasizes the horizontal cut of how a specific transaction is running or all the services that are tied through a single request, a profile is a different vertical cut that emphasizes the state of resources like memory, CPU and more at a point in time. Those profiles are collections that are stored continuously and another great slice of telemetry data that will be encapsulated by the OpenTelemetry standard.

October 31, 2023
How to Alleviate Your Observability Challenges

Observability is becoming a keystone of contemporary DevOps practices. Even departments that weren’t traditionally a part of DevOps are seeing the benefits of being brought under the auspices of observability teams. In 2023, however, organizations are finding that the road to adoption is bumpier than expected. Here are seven of the biggest challenges DevOps teams face with observability and some suggestions for mitigating them.

Increasing MTTR

MTTR, or mean time to recovery, is the time it takes to get a system back up and running after an outage or bug. A longer MTTR means more downtime and poorer service for clients. Worryingly, the DevOps Pulse Report indicates that the average MTTR is increasing. This year, 73% of its respondents reported an MTTR of multiple hours; last year’s figure was only 64%.

MTTR often results from an inability to diagnose incidents due to data siloes impairing observability. This can be improved by an observability platform that allows engineers to see the big picture.

Costs of Telemetry Data

Along with increasing MTTR, many organizations have to deal with costs incurred by high telemetry data volumes. It’s a big problem. An IDC survey of 200 companies found that 53% of respondents highlighted the costs of storing log data.

Much of the problem is due to an outmoded tiered pricing model. Many vendors charge per GB of data, so if your data volumes fluctuate, so will your data costs. They also have obscure pricing models, meaning many organizations find it difficult to know what they’re paying for. At Coralogix, we’ve created a new business model that is one-third of the cost of standard log storage solutions.

Tool Sprawl

To understand why this is a challenge, we need to answer the question: What is observability? Effective observability requires integrating data from every aspect of your application. Because many organizations implement monitoring with multiple tools, they suffer from tool sprawl. This has the effect of siloing telemetry data, making it harder to correlate data and extract insights into system performance.

There are a range of options for mitigating tool sprawl, such as thoroughly evaluating a tool’s costs and benefits before including it in your DevOps strategy. The most effective solution is a “single pane of glass” tool that combines insights synoptically on a single dashboard.

Kubernetes Complexity

Elastic reports that organizations are increasingly turning to cloud-based solutions such as Kubernetes for their DevOps. Kubernetes can supercharge organizations with its ability to dynamically scale infrastructure as needed, eliminating the cost overheads of dedicated servers.

However, Kubernetes is complex and comes with its own set of challenges. Kubernetes’ scalable architecture comes from containerization, a paradigm in which applications are hosted in objects called containers. This means that developing in Kubernetes requires the ability to work with a lot of spinning plates.

A good way to combat this is better training in organizations. Additionally, breaking down silos allows different teams to transfer knowledge.

Security Challenges

Kubernetes’ popularity brings security challenges. These can include privilege escalation, where a user manages to gain privileges such as write access, and security misconfigurations, where developers forget to change the non-secure default configurations.

There are several strategies for mitigating Kubernetes security risks. These include scoping roles to particular namespaces, using service meshes, and enhancing security with Coralogix’s Kubernetes Operator.

Beyond Kubernetes, there’s the larger issue of integrating security into an observability strategy, which is becoming a challenge for an increasing number of IT businesses. To counter this, more and more businesses are starting to incorporate observability and security monitoring under a single umbrella. Solutions such as infrastructure and application metrics can enhance security and monitoring.

Scaling Platforms

To meet the challenges posed by rising data costs and increasing cloud complexity, businesses are turning to open-source solutions. These come with their own challenges, however. According to the DevOps Pulse Report, around 30% of enterprises surveyed had problems with infrastructure management, scaling and upgrading relevant components. Because many open-source platforms require specialized knowledge to maintain, businesses have trouble sourcing the skills and expertise for them.

Tools such as OpenTelemetry can make scaling easier by integrating with platforms such as Coralogix.

Troubleshooting Data Pipeline Performance

Implementing observability requires having a reliable and high-performance pipeline for telemetry data. However, organizations using open-source platforms often have trouble monitoring and troubleshooting the performance of their data pipeline. This can impair observability as telemetry data is of lower quality.

Data engineer Abraham Alcantara suggests ten key steps to troubleshoot data pipelines successfully. These include identifying data pipeline software and infrastructure, reproducing and isolating issues and automating issue scenarios. Another strategy is to apply machine learning, such as Coralogix uses.

October 31, 2023
Maximizing Data Insights: How OLAP Empowers Modern Data Stacks

In the era of big data and cloud computing, online analytical processing (OLAP) has fallen out of favor. As modern data warehouses evolved to accommodate larger volumes of data and claims faster processing capabilities, OLAP fell out of fashion as a notion emerged that it would struggle to manage the combinatorial explosion. The focus shifted toward runtime querying engines. Instead of using predefined or pre-aggregated OLAP cubes, many business intelligence (BI) and analytics tools used their own data modeling applications that worked directly with raw data.

However, in recent times, there has been a realization that OLAP, when implemented intelligently, can be highly beneficial even in the context of a massive scale of data and cloud computing. Modern OLAP leverages advanced technologies and optimizations that overcome any previous limitations. Besides being cost-effective, especially for cloud implementations, it delivers real performance advantages.

Modern OLAP systems are designed to handle large datasets efficiently. One of the key strengths of OLAP lies in its ability to provide real-time or near-real-time insights, enabling businesses to make data-driven decisions with greater agility. The multidimensional data model of OLAP cubes facilitates instant query response, empowering users to derive valuable insights from complex data sets without compromising on query performance.

Consider These Advantages of Using OLAP in Modern Data Stacks

Performance: As data volumes continue to grow, OLAP remains a powerful technology to extract valuable insights and unleash the true potential of modern data stacks and cloud environments. OLAP’s multidimensional data model with pre-aggregation cubes enables efficient data processing and analysis, delivering faster insights and sub-second query responses. Modern algorithms further enhance performance, allowing organizations to handle large-scale data workloads without worrying about data explosion problems.

Unique OLAP Capabilities: Some domains, like finance, have an inherent need to preserve crucial OLAP capabilities even with the growing data storage and compute requirements for which they make a move to modern data stacks and cloud environments. Moving to the cloud enables data scalability, addressing the challenges of handling vast data volumes. However, they cannot afford to compromise on essential OLAP functionalities such as advanced hierarchies, like parent-child relationships, and alternate hierarchies with weighted rollups.

These capabilities are indispensable, especially in the world of financial analytics, regardless of how modern and advanced the new data stack may be. By leveraging OLAP in the cloud, organizations can continue to analyze and explore data with complex hierarchies efficiently, ensuring accurate and in-depth financial insights without sacrificing the flexibility and analytical power that OLAP provides.

Cloud Architecture and Cost Effectiveness: OLAP in cloud architecture presents a powerful and forward-looking solution that combines the benefits of OLAP technology with the scalability and cost-effectiveness of cloud computing. By harnessing the cloud’s resources, OLAP can efficiently process and analyze large volumes of data, thus meeting the growing demands of modern businesses.

Once OLAP cubes are constructed and populated, they provide a highly efficient and aggregated representation of the data. These pre-aggregated structures accelerate query performance, enabling teams to access insights in real-time or near-real-time. BI teams can seamlessly integrate existing tools with the OLAP model, fostering a cohesive and familiar analytics environment. It empowers decision-makers to explore data interactively, drill down into details, and gain valuable insights.

One of the primary advantages of OLAP in the cloud is its cost-effectiveness. OLAP technology, being primarily based on pre-aggregation, drastically reduces runtime querying compute cost, as answers to business queries are already pre-computed and stored in an aggregated structure. At querying time, OLAP tools have to look for right answers from the aggregated structure—not the cloud data storage—and return the query response, which results in much less compute cost as compared to runtime aggregation compute cost. It truly enables build once, query many times culture. Apart from this, businesses can leverage the cloud’s elasticity to scale computing and storage resources up or down as their data requirements fluctuate. This adaptability ensures companies can handle peak workloads without overprovisioning and reducing unnecessary expenses.

Universal Semantic Layer: A USL approach ensures consistent data definitions and analytics across the organization, reducing data discrepancies and improving data governance. A common understanding of the data enables better communication among teams and results in a smooth implementation of a data-driven decision-making process.

At the same time, using a USL means that multiple business intelligence tools and reporting platforms can connect to and leverage the same semantic model. Hence, various parts of the organization can continue to work with their familiar BI interfaces while accessing common underlying data.

OLAP lends itself naturally to delivering on a powerful universal semantic layer. The concept of dimensions, measures, hierarchies and calculations forms the foundation of an OLAP semantic model. OLAP semantic models also accommodate complex calculations, custom expressions or formulas required for complex business-specific logic. These components provide a powerful way to structure and organize data, making it more accessible and intuitive for business users who need to derive insights from the data.

If we consider typical sales data, product categories, geographic regions and time periods could be dimensioned. Measures, on the other hand, would be sales revenue, profit margins, quantities sold or any other KPIs. Hierarchies allow data to be analyzed at different levels of granularity. Users may roll up or drill down to view data at different time intervals such as year, quarter, month, day, etc.

USL and OLAP models are designed to shield business users from the underlying complex data modeling and schemas. They can work with filters, dimensions and measures that are business-focused representations of the data using familiar business terms while abstracting technical intricacies. They can view and explore data through intuitive hierarchies, filters and measures aligned with their business processes and KPIs.

Conclusion

OLAP in cloud architecture represents a forward-looking, fast and cost-effective solution. It capitalizes on the cloud’s scalability and pay-as-you-go model to process vast amounts of data efficiently, while its pre-aggregated structures and real-time capabilities empower business intelligence teams to gain valuable insights seamlessly. With its agility, OLAP in the cloud is well-positioned to meet the challenges of the future, providing organizations with a competitive edge with data-driven decision-making.

October 23, 2023
Data Reliability Needs to Shift Left

Data pipelines feed and transform data for consumption and are increasingly growing in complexity. As enterprises continue to add data sources, they run the risk of breaking their data pipelines due to data errors, flawed logic or insufficient resources for processing. The challenge for every modern data team is to establish data reliability as early as possible in the data journey and to create optimized data pipelines capable of performing and scaling to meet the business and technical needs of the enterprise. In the context of data observability, “shift left” refers to a proactive approach that incorporates observability practices during the early stages of the data lifecycle. This concept is borrowed from software development methodologies and emphasizes addressing potential issues and ensuring quality from the outset.

Shifting left involves integrating observability practices and tools into the data pipeline and infrastructure from the beginning. This approach avoids treating observability as an afterthought or applying it only in later stages to identify and resolve data reliability, integrity, and performance issues as early in the development process as possible, minimizing the likelihood of problems cascading downstream.

The Economic Impact of Poor Data

To effectively manage data and optimize data pipelines, it is crucial to detect and address data incidents as early as possible within the supply chain.

The “1 x 10 x 100 Rule” is applicable in a variety of processes, including software development. It states that the cost of fixing a problem increases exponentially, depending on the stage at which it is detected. The rule can be extended to data pipelines and supply chains, where it suggests that detecting and rectifying a problem in the data landing zone (i.e., where source data is fed) incurs a cost of $1, whereas identifying and addressing the issue in the transformation zone (i.e., where data is transformed into its final format) increases the cost to $10. If the problem is only detected in the consumption zone (where data is in its final format and is accessed by users), the cost rises significantly to $100.

Shift Left Data Reliability

The process of identifying data pipeline issues has become more challenging for data teams as data supply chains continue to evolve and grow in complexity, mainly resulting from the following factors:

● Expanding sources: The number of data sources (such as Databricks, Snowflake, Redshift, Teradata and many others) being incorporated into the supply chain has significantly increased. Organizations now integrate data from a wider range of internal and external sources, contributing to the complexity of data management and processing.
● Advanced data transformation logic: The logic and algorithms used to transform the data within the supply chain have become more sophisticated. Complex transformations and calculations are applied to raw data to derive meaningful insights, necessitating robust systems and processes to handle the intricacies involved.
● Resource-intensive processing: The processing requirements for data within the supply chain have significantly escalated. With larger volumes of data and more complex operations, organizations must allocate substantial computing resources, such as servers, storage and processing power, to handle the data processing workload effectively.

By proactively addressing data incidents at an early stage, organizations minimize the potential impact and cost associated with data issues. This not only ensures the reliability and accuracy of data consumed by users but also safeguards the integrity of downstream processes and decision-making. Ultimately, the shift left approach to data reliability promotes efficiency, reduces costs, and enhances overall data quality and trustworthiness.

How Can Data Teams Shift Left?

To effectively shift left in your data reliability solution, certain capabilities are crucial; these include:

● Early Data Reliability Checks: Conduct data reliability tests before data enters the data warehouse and data lakehouse. This ensures that bad data is identified and filtered out early in the data pipelines, preventing its propagation to the transformation and consumption zones.
● Support for Data-in-Motion Platforms: Enable support for data platforms like Kafka and monitor data pipelines within Spark jobs or Airflow orchestrations, which enables real-time monitoring and metering of data pipelines.
● File Support: Perform checks on different file types and capture file events to determine when incremental checks should be performed.
● Circuit-Breakers: Integrate APIs that incorporate data reliability test results into your data pipelines, which can halt data flow when bad data is detected. By preventing the spread of bad data, circuit breakers protect downstream processes from being affected.
● Data Isolation: Identify and isolate bad data rows, preventing their continued processing.
● Data Reconciliation: Implement data reconciliation capabilities to ensure consistency and synchronization of data across multiple locations.

Shifting left your data reliability practices empowers data teams to focus on innovation, reduces costs, enhances data trust and enables agile business processes. Embracing this approach makes data more usable and resilient and delivers a more robust and reliable data ecosystem that supports informed decision-making and drives organizational success.

October 23, 2023
Multi-Tenant Database Strategies for Mixed Workloads and Real-Time SLAs

Multi-tenancy—where a single instance or multiple instances of one or more applications operate in a single shared infrastructure environment—is not a new concept. First used in mainframe hardware, timesharing enabled a single massive piece of computing iron to be sliced up and used by multiple applications on a single hardware instance. This simplified both the hardware landscape as well as the skills and resources needed to manage environments. With the rise of software-as-a-service (SaaS) in today’s modern private and public cloud environments, multi-tenancy is now a common and critical function. It allows cloud companies to scale and customers to save money by essentially pooling resources. As more applications demand more and more data, multi-tenancy is now a core function at the database layer.

But when you have a lot of different applications running on the same infrastructure and accessing the same database, how do you prevent one application from impacting the data performance of another? And how do you determine from the underlying database infrastructure which tenants get what infrastructure and at what performance level?

Here’s a look at the driving forces behind today’s real-time data needs and how they impact multi-tenant data architecture management.

SLAs Are at the Heart of Multi-Tenant Data Management

Everyone wants faster performance and more resilient systems. We’re impatient, so reliable, real-time performance delivers large, competitive business benefits. However, every tenant does not require the same level of performance. Actual performance needs may be plenty fast for a user or machine without requiring the fastest execution path possible with the available resources.

Service-level agreements (SLAs) ultimately govern the resources delivered to a tenant. Real-time may need to be measured in sub-milliseconds in one instance (i.e., for a stock trade) but in sub-seconds for others (i.e., in a Zelle funds transfer). This is where developers, architects and operators partner with a business to understand its actual user or system needs and to build a system that can deliver the SLA using the minimum amount of resources possible.

Mixed Workloads and the ‘Noisy Neighbor’ Problem

As soon as workloads are shared, there’s the potential for one application to quickly grab and use resources in a way that impacts other applications. Often called a “noisy neighbor,” the impacting applications can dramatically interfere with real-time response time SLAs when multi-tenant apps share an infrastructure base. For example, one node in a database cluster becomes heavily loaded, and the performance of other nodes begins to be impacted, causing SLAs to be missed.

Typically, multiple virtual machines (VMs) are hosted on a single physical machine or node but share system resources (i.e., processor, network, storage, etc.) with the other VMs. It’s possible for applications to overuse those shared resources, impacting other applications and sometimes even the database itself. Such overusage of the infrastructure layer (in this case, sharing compute, storage, network queues, etc.) leads to poor database and end-to-end system performance. It’s like plugging a 100-amp appliance into a 50-amp circuit. All the circuit breakers will trip, and the entire system comes to a screeching halt!

Strategies to Maximize Performance for Database Multi-Tenancy

Configuring and managing the database itself (not just infrastructure) with multi-tenancy in mind can maximize performance and avoid problems like noisy neighbors. Isolating databases in your multi-tenant architecture can help minimize non-database problems that can be the root cause of poor performance and missed SLAs.

Start by picking databases that allow applications to isolate their data access based on sets (or tables in relational parlance) that can have storage limits attached to them. When combined with rate limits for transactions on a per-user basis, applications can be guaranteed their quota of runtime resources. Using the per-app quota configurations for items like storage and TPS (transactions per second), the database can evaluate the total capacity available in the system and ensure that enough resources are reserved and available to ensure that the SLAs of all applications are met all the time.

Note that unexpected events that cause rebalancing of data within the distributed database, such as node and network failures, require additional resources. The database system will subtract that resource capacity from what is available for apps so that the system is not overcommitted during failure situations.

Enterprises also need very good automation and observability to fix immediate problems quickly and have the requisite visibility into long-term trends that impact performance and SLA compliance. Otherwise, by the time a human notices a problem, the SLA may already have been missed and the business impacted.

Good observability can help you understand how data is used and create those automations. As more applications access and use the same data set in different ways, you need modern observability tools integrated into (or interoperable with) the underlying database.

With a good understanding of how your data-rich applications interoperate with other tenants in mixed-load environments, you can begin to deploy these strategies to make sure you conform to all required SLAs and keep all that data working hard for the business.

October 19, 2023
How to Build a Data Platform for Self-Service, Ad-Hoc Analytics

In a digital world, data is often the differentiator between success and failure. Whether it’s defending against cybersecurity threats, improving application performance, or resolving full-blown outages, data is critical to the modern DevOps team.

In fact, there are plenty of situations where DevOps engineers, SRE teams and other observability-focused departments need to explore reams of data rapidly and flexibly. An SRE trying to figure out the root cause of latency may need to dissect the movement of data across endpoints to find the malfunctioning one. Another engineer may need to compare historical and current performance to find anomalous behavior. Lastly, a DevOps engineer may have to dissect user performance metrics to understand the scope of a slowdown—whether it is global or regional.

Every use case requires dashboards that can provide detailed data through a wide variety of visualizations, including choropleth maps, stack areas, pie charts and bar graphs. Ideally, a dashboard will enable teams to isolate dimensions, apply filters, and dive into data for deeper insights.

Unfortunately, not all dashboards can support this type of fast, flexible analysis. Many still utilize outdated technologies, which lack the flexibility, agility and scalability required to seamlessly explore data. Many were also designed without the urgency that today’s data requires—after all, in previous years, most uses for dashboards (such as internal reporting) were not time-sensitive.

Other dashboards are constrained by templates, which offer a finite array of widgets, tools and drill-down capabilities. These dashboards may feel unwieldy and sluggish, unsuited for a rapidly evolving situation like an application outage.

Requirements

To truly fulfill the promise of self-service, ad-hoc analytics across huge datasets, teams need to work in real-time—and so they need a database capable of timely responses.

When an application goes down, an SRE may not know what to look for and, hence, needs to quickly comb through lots of data. While some dashboards rely on workarounds for faster queries, such as pre-aggregations, precomputing, or rollups, this is not possible in this instance because the SRE simply won’t know what to look for. After all, they can’t necessarily predict what will go wrong, and even if they work on assumptions from previous failures, this current issue could be much different.

Therefore, dashboards must provide plenty of functionalities and visualizations. Users should be able to filter data by time, isolate variables like location, and zoom into specific time intervals with a few clicks. Dashboards should also accommodate diverse data types, including intricate parent-child relationships and nested columns.

Further, dashboards must offer a depth of insight. An online streaming media platform may need to assess user metrics (such as latency or load times) across different devices, operating systems, and regions to fine-tune performance. A cloud provider has to monitor their physical hardware for high temperatures, slowed network switches or devices, and other anomalies.

Because data is now so important to success, many more people across a company, including data scientists, product managers and external users, themselves require data-driven insights. In these situations, dashboards must handle the increased user traffic and query activity, maintaining fast responses even under load. This is especially important considering that a single-user operation (such as a zoom) will require multiple queries on the backend to execute.

A database also must scale seamlessly. If an organization’s environment generates millions of events an hour, that equates to billions of events per day or week—challenging for any database to ingest, store, analyze and query. In fact, many databases cannot successfully provide fast response times while managing large datasets and high query volume. As an example, transactional databases (OLTP) can often query rapidly but cannot execute analytics at scale, while analytical databases (OLAP) can analyze massive volumes of data but not at speed.

Apache Druid for Independent, Ad-hoc Data Exploration

This is where open source Apache Druid comes in. Combining the scale and advanced analytics of an OLAP database with the speed of an OLTP database, Druid offers swift, real-time data exploration.

Upon ingesting data, Druid makes it immediately available for querying and analysis, removing the need to first batch or aggregate the data in some way. In addition, Druid natively integrates with streaming technologies like Apache Kafka and Amazon Kinesis, removing the need for workarounds or connectors.

Druid powers interactive visualizations, providing millisecond response times, enabling more versatile exploration, expanding the range of available dimensions and filters, and even maintaining subsecond speeds in the face of surging user and query volumes.

Druid’s unique architecture also enables easy scaling. By separating key duties among separate node types—data nodes for storage, master nodes for data ingestion and availability, and query nodes for executing queries and returning results via the scatter/gather method—Druid ensures that nodes can be independently scaled based on need. Afterward, Druid also automatically rebalances traffic to ensure consistent performance.

Salesforce: A Druid Success Story

Salesforce pioneered the customer relationship management (CRM) space, serves 150,000 customers worldwide and earns billions in annual revenue.

The Edge Intelligence Team is the division of Salesforce that tackles the massive task of ingesting, processing, filtering, aggregating, and querying the entirety of their log data — anywhere from billions to trillions of lines daily. Each minute, Salesforce ingests 200 million metrics, while each day, Salesforce processes five billion daily events globally. In total, Salesforce accumulates dozens of petabytes of data in their transactional store, five petabytes of logs in their data centers, and almost 200 petabytes in their Hadoop storage.

Salesforce teams use Druid to unlock real-time insights into product performance and user experiences, diving into large datasets instantly. Anyone from engineers to account executives can query a wide variety of dimensions, filters and aggregations to better understand trends, troubleshoot any issues that arise and set strategies for the future.

By using Druid’s compaction abilities, Salesforce also decreased the number of Druid rows by 82%, leading to an overall reduction of their storage footprint by 47% and accelerating their query response times by 30%.

October 19, 2023
Microsoft kills Python 3.7 ¦ … and VBScript ¦ Exascaling ARM on Jupiter

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: VS Code drops support for Python 3.7, Windows drops VBScript, and Europe plans the fastest ARM supercomputer. (more…)

October 11, 2023
Building Scalable Cloud Applications With MySQL

Relational database systems such as MySQL have long been the go-to choice for handling large and complex data handling tasks. But, with the introduction of new technologies like NoSQL, many developers started to question whether traditional relational databases still have the upper hand. However, when building high-scalability cloud applications, relational databases like MySQL still have the edge over NoSQL. Here are five reasons MySQL remains superior to NoSQL in supporting developers of high-scale cloud applications.

1. Enablement of Complex Queries

Structured query language (SQL) is the most widely adopted language for relational databases, and MySQL is no different. SQL gives developers a powerful platform for creating complex queries and ad hoc reports, efficiently managing large datasets and creating tables and indexes for data optimization. By contrast, NoSQL usually offers only rudimentary query and data manipulation capabilities, often proving inefficient when handling complex data sets. Most data analysts are also familiar with SQL, so it’s a drop-in tool they can use to help gain insights.

2. Scalability and Reliability

MySQL is noted for its scalability when handling massive data loads and is particularly well suited to cloud-based applications that require consistently high performance. MySQL ensures that the consistency of data is maintained. MySQL is the go-to option for scalability and reliability for larger cloud applications. Companies such as Facebook, GitHub and WordPress have built billion-dollar businesses on MySQL, and it’s been tested and validated in some of the harshest and most demanding workloads of any database.

3. High-Performance Replication at Scale

MySQL replication offers a reliable solution for cloud applications that need to scale. Unlike other databases, MySQL’s replication capabilities provide effective data safety, making it a robust choice for applications that require high availability and durability. Additionally, MySQL enables seamless version upgrades without disrupting operations, ensuring the system always has the latest features and security patches. The read scaling feature of MySQL improves performance by distributing the read load across multiple instances, making it ideal for large-scale cloud applications. With a proven track record at scale, MySQL replication is the optimal choice for applications that want to grow and scale efficiently.

4. High Availability and Durability

MySQL provides enhanced high availability and durability, ensuring that critical data is always available even during maintenance and disaster recovery. This is because MySQL is fully ACID-compliant (atomicity, consistency, isolation, durability) and provides full support for transactions. Even when the system fails, the transaction can be rolled back and the data is not lost. With NoSQL databases, data is primarily written in memory and doesn’t provide the same degree of durability or consistency.

5. Ease of Implementation and Debugging

MySQL is easy to install, implement and debug compared to NoSQL. MySQL is supported by a large community of developers and database administrators who use it for a diverse range of applications. MySQL is also open-source and free to use, making it a great choice for businesses of all sizes. NoSQL systems require the installation of additional tools, libraries, and other components for implementation, making it more complex and time-consuming.

MySQL provides many benefits for developers of high-scale cloud applications, ranging from its powerful SQL language for complex querying to its high availability and flexibility for managing multi-table joins. NoSQL remains a viable option in some cases, but developers who are building high-scale cloud applications are mainly sticking with MySQL due to its scalable and reliable nature, easy implementation and flexible query-handling capabilities. Choosing the right database engine is critical for any organization that relies heavily on large and data-heavy applications. MySQL continues to stand the test of time when it comes to ensuring performance and getting consistent results.

October 6, 2023
Cisco Acquires Splunk to Create Observability Powerhouse

Cisco today revealed it has signed a definitive agreement to acquire Splunk for $28 billion in cash. The deal will add Splunk, one of the most widely-used platforms for managing IT operations, to Cisco’s portfolio, joining the AppDynamics observability platform in addition to a wide range of IT infrastructure equipment.

Chuck Robbins, CEO of Cisco, told industry analysts the combined entity will be one of the largest software companies in the world. The Splunk and AppDynamics platform creates a comprehensive observability platform spanning applications, networks and hybrid cloud computing environments, Robbins added.

By combining AppDynamics with the data collected by the Splunk platform, the combined company will be able to provide greater insights into IT operations than any other company, added Robbins.

Splunk CEO Gary Steele said the acquisition represents the next natural, logical step for Splunk and furthers its long-term partnership with Cisco to keep IT environments secure and resilient. The two companies will also be able to apply artificial intelligence (AI) more effectively to IT operations, he added.

In its most recent quarter, Splunk reported revenues of $911 million, with cloud revenue accounting for $445 million, or 29%, of the total. Total Splunk revenues are forecasted to be between $3.925 billion and $3.95 billion for fiscal 2024.

Splunk previously acquired SignalFX, a provider of an observability platform, in 2019. It’s not yet clear how the platform might be integrated into the AppDynamics platform, but Robbins said there is little overlap between the observability platforms the two companies currently provide.

The Splunk acquisition comes at a time when it’s becoming more apparent that the need for observability extends well beyond DevOps workflows as organizations become more dependent on software.

Each IT organization will need to determine how long it will wait for the Splunk acquisition to close before investing further in observability. Historically, acquisitions of this scale have been accompanied by price hikes to help fund the merger once the deal is concluded.

In the meantime, IT and DevOps leaders will likely revisit the way IT is managed in the months ahead. Splunk came to prominence at a time when most workloads were running in on-premises IT environments. While the bulk of workloads continue to run in those environments, many workloads deployed in the cloud are being managed using platforms designed specifically for those environments. Splunk has been making a case for a hybrid cloud computing approach that assumes the management of on-premises and cloud computing environments will become more unified.

In theory, economic headwinds are driving more organizations in that direction to reduce the total cost of managing IT. But it’s just as feasible to extend cloud tools to on-premises IT environments as it is to extend legacy tools to the cloud. Despite the inevitable outcome, Splunk rivals will spend the better part of the next year coming up with offers to switch platforms both before and after the deal eventually closes.

September 21, 2023