devopsy.kinsta.cloud

Category: DataOps

New-Age Tech and Culture Driving Data Democratization

Data democratization has become critical for organizations to leverage data’s true value and fully realize its benefits. Making data available across the organization helps companies to serve customers better and make data-driven decisions that align with their goals and objectives. It essentially means enabling access to a company’s data resources to employees, bound by reasonable limitations on legal confidentiality and security.

As a means to remove data silos in organizations, the indispensability of data democratization is clear. It allows more widespread access to and use of data within an organization. This can lead to more informed decision-making, increased collaboration and innovation and more efficient use of resources.

Breaking down data silos can help reduce the risk of data breaches and improve data security. While the jury may be out on whether data democratization is beneficial in preventing security breaches, silos between networks or security systems make it difficult to identify large-scale attacks as they tend to represent disjointed security protocols. There is less likelihood of robust security protocols between non-communicating teams.

New-Age Tech as an Enabler of Data Democracy

New-age technology such as cloud computing, blockchain, and advanced analytics are aiding data democratization. For example, 5G enables more significant volumes of data to be moved faster and more securely in the form of big data. Blockchain provides higher-quality data and improves the operation of cyber and physical systems. In addition, IoT allows for exchanging data between devices, where processes can be established between devices without intermediaries.

A hybrid cloud adoption survey conducted in March 2022 of IT professionals primarily based in North America and Europe revealed 93% of the IT industry would adopt a hybrid of cloud and on-premises solutions or migrate fully to the cloud within five years, using it to store and manage data. Gartner estimated that 85% of enterprises would embrace a cloud-first principle by 2025 and that it will be essential to execute digital strategies in business.

This suggests that new-age technologies play a significant role in breaking down data silos and enabling organization-wide access to data. By leveraging these technologies, organizations can easily store, process and share data, leading to better decision-making, increased collaboration and improved business outcomes.

Data creation, development, and utilization directly relate to Industry 4.0 applications. In practice, data democratization is seamlessly transferable to Industry 4.0 applications. The question arises: Do industrial companies have the capabilities today to apply the concept of data democratization? And to what extent?

As new-age tech drives the emergence of new business models, companies that will lead a cultural shift as well as a technology evolution will be successful. This is possible via a business-centric data strategy, along with the democratization of analytical tools and platforms to enable stakeholders to help build better insights into their business processes.

Culture as a Prerequisite for Data Democracy

A Google Cloud and Harvard Business Review survey of industry leaders revealed 97% believe organization-wide access to data and analytics is critical to the success of their business. However, only 60% of respondents believed that their organizations effectively distribute that access today.

What are the prerequisites for data democracy? First, companies should inculcate a culture change to create a pull from inside the organization rather than first developing the technology. Second, decision-making in companies is based on experience and intuition than on data; they should compulsorily begin to use data as a baseline for business decisions. Third, leaders should encourage a data-driven culture to grow from within.

Formalizing and sharing knowledge can remove roadblocks to the democratization of data. Therefore, one form of action is to set specific incentives to foster a mindset of knowledge sharing. For instance, existing practices to incentivize participation in the suggestion or lessons-learned programs can be used as a reference.

Many companies have procedures to involve employees in the continuous improvement process. But while employees can use the existing IT systems to perform their day-to-day work, they need more capabilities to cover their individual information needs. Thus, as a way forward, capability building within is critical. Another strategy is decentralizing decision-making; if employees can gain insights independently, they should be able to act on these insights independently.

Organizational culture is vital to data democratization, as it determines the attitudes, behaviors and values that guide how data is shared and used. On the other hand, a culture that values secrecy or hierarchical control over data can lead to resistance to data democratization efforts and perpetuate data silos. A culture that lacks trust and transparency can also lead to reluctance to share data, which can impede collaboration and hinder data-driven decision-making.

Data Democratization – The Way Ahead

Organizations need to assess and work on their tech stack and internal culture to promote a data-friendly culture supporting data democratization. To foster a data-friendly culture, organizations should encourage open communication and collaboration among employees and provide training and resources for employees to access, understand, and analyze data.

Open communication, collaboration, innovation and trust are essential attributes for breaking data silos. A corporate culture that values data-driven decision-making will lead to more widespread access to and use of data and support the implementation of new-age technologies. Finally, there is a need to establish clear data governance policies and procedures that support data democratization and build a continuous improvement process by encouraging experimentation and learning from failures.

Image Source: Joshua Sortino, Unsplash

March 3, 2023
How to Achieve Data Mesh With Four Key Principles

Today’s world is becoming increasingly data-driven, which means businesses need to refine their practices and decision-making processes with the use of trustworthy data. However, this has proven challenging as organizations often don’t know what to do with the abundance of data available to them. According to a recent survey from Snow Software, 89% of IT leaders receive actionable data, yet 60% feel overwhelmed by the volume at their fingertips. These companies are both thriving in data collection and are bogged down by the sheer quantity available. While traditional data strategies have not yet adapted to meet modern business needs as the amount of data continues to grow, there’s a strong opportunity for companies to transform their data warehouses and data lakes and optimize the flow of data to consumers through an emergent approach called data mesh. Data mesh forges a path for companies to improve their business strategy and make better, faster decisions by leveraging the insights buried within their data.

Data Mesh and its Benefits

At its core, data mesh is an approach to data management that enables organizations to evolve from the pitfalls of legacy, centralized data architecture toward a decentralized, domain-driven design at scale. Breaking that down, data mesh is designed to help companies manage complex data networks by focusing on decentralized data ownership, treating data as a product, and providing a uniform infrastructure capable of sharing data across the company landscape. It improves upon legacy data management, which is often time-consuming, error-prone, unsustainable and unscalable. Unlike these traditional methods, data mesh gives domain-driven ownership to experts who strategize how it can best serve the company, as opposed to just dumping all data in one place and waiting for a centralized function to process it and make it available to the business.

A data mesh approach also enables organizations to democratize their data by allowing users to acquire trusted data when they need it, all while automating the complexity and being shielded by security and governance guardrails. It removes bottlenecks and helps businesses glean the data-driven insights they need at the scale and speed of their organization. This way, organizations can maximize the value of their data by ensuring it is clean, vetted and readily available to support business initiatives.

The Four Principles of Implementation

So how can organizations embrace this approach? The framework of data mesh is centered on four guiding principles to help organizations get the most out of their data. Leveraging these principles will help your organization unlock the benefits of a data mesh.

● Domain-driven ownership: As opposed to separating data from your organization’s subject matter experts, the first pillar of data mesh implementation involves putting domain experts in charge of their data ecosystem. This way, data will be cleaned and enriched at the source before it’s made available throughout the organization. Domain owners will maintain the quality of the data and keep all necessary facts and documentation available for audits, effectively combining data with business talent.
● Data as a product: The job of the data team is to provide the data that the company needs for whatever purpose, be it making decisions, building personalized products, or detecting fraud. By treating data as a product, the company is giving its data a vision, strategy and product roadmap, ensuring its value is executable.
● Self-service data infrastructure: A business is made up of logically autonomous domains; each domain not only supports a business function, product or process but also produces data products and therefore requires a data infrastructure. Self-service infrastructure ensures business domains don’t have to deal with the underlying complexity of networking, security, compute and storage requirements.
● Federated computational governance: Standards for data governance are defined centrally, but local domain teams are empowered and equipped to apply to these standards however is most suitable for their specific environment within predefined guardrails. The key to achieving this for policy, security, quality and definition at scale is automation and integration throughout the landscape of the data infrastructure.

By enacting these principles within your organization, your data mesh architecture is sure to provide the most value for your data.

Data mesh is an optimal solution for any business currently struggling to find value in its data. Traditional methods are time-consuming, unsustainable, unscalable and error-prone. Data mesh is an approach designed to transcend these barriers, decentralizing much of the IT team’s heavy lifting and putting the responsibility of managing data on individual business domains. With clear ownership, domain experts can develop deep data knowledge and expertise and ensure that it’s trustworthy and valuable to your organization before making it available for consumption. This way, enterprises can prioritize self-service and empower data consumers with access to accurate, high-quality data when needed.

March 3, 2023
Why You Need to Break the Observability Data Silo

In my role as a developer advocate for Coralogix, I speak with organizations across almost every conceivable industry, and there is a pattern across almost all of them. In the world of observability, the challenge is battling scale. At scale, observability becomes a complex problem to solve. Surprisingly, the most common pattern for handling observability data is also holding people back from unlocking the true power of their data.

What is the Data Silo in the World of Observability?

To understand the data silo, we first need to understand the typical architecture of an observability solution, whether it’s homegrown or SaaS. Observability data can be divided into three types: Logs, metrics and traces. These three types come together to give engineers a complete picture of the health and performance of their system.

These three data types are typically held in different storage mechanisms. A popular solution for logs is open search, while Prometheus holds the mindshare for metrics, and Jaeger is a very popular tracing solution. This means three solutions, three interfaces and, subsequently, three problem-solving workflows.

What’s wrong with separating data like this? Even though there are three types of data, there is still only one goal: To understand the health and performance of the system. This is the only use case. The data silo is the dominant trend for observability solutions to provide a fragmented view of the data. It requires engineers to piece together stories as they go, rather than correlating across all of this data at once, to provide a complete picture of system performance.

This slows down troubleshooting. If there is an issue, an engineer will first review one of the data types. Then, they will run different queries against other data types to confirm. So they may detect a 500 error in the logs, then move to the traces to see the impact on traffic, and finally, review the application metrics. This means three minimum queries when attempting to assess only one problem.

Breaking the Data Silo

Breaking the data silo can be a daunting challenge. There are some key points to focus on when a company starts out and some finishing touches that make for an excellent developer experience.

Unify data storage as much as possible. For example, traces can be stored in Prometheus rather than in a separate solution. This will enable engineers to query, for example, trace latencies alongside CPU metrics. It will also minimize the UI work necessary to join together data. If an organization has the expertise, it’s possible to store everything inside an OpenSearch cluster. This means a single interface, with a single query language, to query all available observability data. Anyone assessing this option should simply decide what their company can maintain and what skills they have in-house.

Present observability data as one. If storage unification isn’t an option, then focus on the presentation layer. Tools like Grafana can consume most observability solutions as external data sources and render them all on the same dashboard. While an engineer will still have to query them using the specific language of the tool, it means that they can render things alongside one another, making for an insightful dashboard.

Correlate data in one user interface. This means metrics are rendered in a single dashboard alongside logging and tracing data. This will give users a static view of the data at once, even if there is a limit to how queryable the data is.

Create universal values. If it’s impossible to make any of the previous recommendations, then this is an easy, quick win that will at least make querying observability data more consistent. An easy way to do this without making your code messy is to use a library like MDC (or your tech stack’s equivalent). Add some tags or values that will exist in all logs, metrics, and traces. For example, the name of the application and the feature it belongs to, i.e. application is credit-card-service and feature is payments. This means that if something goes wrong, there is already a set of key-value pairs that translate across every data type.

At the very least, this will mean an engineer can issue consistent queries across the three solutions, albeit in different syntaxes. This represents the least value, but also the least rework and effort, and will lower the cognitive load of engineers who are already trying to debug a potentially serious outage.

The Data Silo Can be Beaten

A siloed observability strategy drives waste and inefficiency in engineering teams, precisely when they need efficiency the most. By breaking the data silo, observability data can be queried, analyzed, and correlated together to form a single, cohesive and complete picture of your system health.

March 1, 2023
Vercel Adds Observability Tools to Web Application Framework

Vercel today announced it has added a suite of observability tools to its platform for building web applications using Next.js, an instance of the open source React framework based on a JavaScript library.

Vercel CTO Malte Ubl said the Vercel Monitoring tools make it simpler for developers to troubleshoot applications without relying on a DevOps team to surface data. Instead, developers can take advantage of Vercel Monitoring tools to access metrics that track user interface and application programming interface (API) issues, he added.

That capability is not intended to replace the need for observability tools employed by DevOps teams. Instead, the goal is to provide metrics that are specific to issues that developers encounter as an application is being built or after it has been deployed so they can be addressed.

Developers can select from pre-made queries provided by Vercel or create their own by defining what metrics they want the Vercel Monitoring tools to track, including bandwidth and web traffic spikes and requests by bots that are indicative of a cyberattack. In total, there are more than 500 potential errors that developers can track. Logs then make it possible to drill down into errors and issues once they are identified to pinpoint the root cause of an issue.

As more responsibility for applications shifts left toward developers, there’s a clear need for tools that provide more visibility into the applications they develop. Many of the issues that DevOps teams encounter today could have been resolved if developers had been aware of them earlier.

The Vercel platform employs caching, routing and a React framework to optimize application performance in a way that reduces the dependency on backend infrastructure to optimize application performance. That approach puts less strain on backend infrastructure as more requests are automatically processed using a framework that spans clients and servers. In effect, developers can take advantage of frameworks such as React to treat the underlying IT infrastructure as if it were serverless whenever possible with tools that have a familiar JavaScript construct. The overall goal is to increase developer productivity at a time when many organizations are much more focused on the issue.

As a result, DevOps teams will need to make some adjustments as more applications capable of providing richer experiences are built and deployed without relying as much on backend infrastructure. That shift has implications for everything from the amount of network bandwidth consumed to the performance of web applications. Frameworks such as Next.js also tend to increase the overall rate at which web applications can be built and deployed.

One way or another, it’s clear that the way web applications are being built is fundamentally changing. The only issue now is determining to whether DevOps teams are prepared to absorb that level of change across an ever-increasing portfolio of applications that still need to be deployed, secured, updated and optimized.

February 28, 2023
How to Modernize a Data Warehouse in Months, Not Years

Modernizing a data warehouse ranks among the worst imaginable tasks for IT. The end result is immensely desirable. The benefits of moving from an on-premises to a cloud data warehouse are well-documented. But database migrations require huge efforts and the stakes are high. What makes it so difficult? All applications, including ETL, BI, analytics and reporting, are designed to work with a specific legacy system. Hence, moving these applications to a different system requires extensive rewrites.

Database migrations may well be the world’s most underestimated problem. In particular, data warehouse migrations are notorious for their horrendous cost overruns. Even when organizations plan for these projects to take several years, they tend to miss even the most generous deadlines. Worse, the vast majority fail to meet their objectives.

In this article, we report on the exceptional success one major enterprise had migrating its systems. They managed to modernize their system in record time with little to no downtime at all. We’ll take a look at their journey and what sets this organization apart from their peers.

The Big Challenge

They don’t get much bigger. The system in question was long regarded as one of the largest, most complex data warehouses in EMEA. It is the backbone of a market-leading integrated container logistics and shipping company.

The company’s 165TB of highly compressed data is used to track goods and shipping vessels around the world. Uninterrupted availability 24×7 is critical for this kind of operation, of course. As the central data hub, this system connects thousands of users that execute over one million queries every day.

Migrating the workloads off an aging Teradata system to Microsoft Azure Synapse was key to modernizing the data estate. IT leadership found itself faced with the question of whether it would be possible to move without disrupting their business users.

Database Virtualization is the Game Changer

Database virtualization (DBV) is the extension of server virtualization to the database stack. Over the past two decades, we’ve seen a similar development in other spaces too. For example, networking and storage have been completely transformed by virtualization.

The principle of DBV is as simple as it is powerful. At its core, it is software that sits between the applications and the database and translating between the two in real-time. Applications connect to the DBV component instead of the database and DBV provides the same language surface and APIs as the legacy system. DBV converts and optimizes all queries and data formats for the new cloud database. Applications can’t tell the difference between the original system and the virtualized one.

Viewed in the context of migrations, DBV eliminates the most labor-intensive tasks. Instead of rewriting applications over the course of years, point them to DBV.

Planning the Implementation—There is no ‘Easy’ Button

If DBV eliminates the need for a rewrite, doesn’t that make database migrations easy? Indeed, it makes it simple. But easy it is not. Remember, DBV is only the runtime. As with other kinds of virtualization, the implementation requires extra effort.

DBV, in this case, had over 99.75% coverage. In theory, there is no limit to what DBV can emulate. In practice, developers of DBV systems have to weigh each feature request. In our case, of the 0.25%, the customer decided to redesign the application that was outside of the current coverage.

Another decision criterion is the choice of the destination system. For DBV to be successful, the new destination must be powerful enough to stomach the incoming workloads. This seems obvious, yet the failure of the Hadoop craze of the past decade showed it isn’t. In our case, the customer chose Azure Synapse, which was perfectly capable of processing these workloads.

The biggest task during an implementation, though, is the data transfer itself. As a runtime, DBV relies on the data transfer to the new system being complete.

People and Processes Are Your Biggest Obstacle

Conventional migrations are the open heart surgery of the IT world. In contrast, DBV is a non-invasive procedure, but heart surgery, nonetheless. Implementing DBV is a team sport.

To be successful, everybody must pull in the same direction. Executives must set the course and, once underway, relentlessly cheer on the team. Developers must refrain from wanting to rewrite code “just because.” Everybody else must execute in their clearly defined roles.

Divide and conquer is the key principle. By structuring the implementation in tranches, the scope remains manageable at all times. Moving a limited number of applications at one time reduces distraction and keeps the team focused.

Then, there are the naysayers. These are members of the organization who want to preserve the status quo. And finally, there is the legacy vendor. To them, DBV is a clear threat and they will do anything in their power to stop your project.

Upfront alignment across the organization keeps the team on course even when there’s pushback.

Move Fast to be Successful

Because a data warehouse is constantly under development, it remains a moving target throughout the project. The longer the project takes, the more complex things tend to get.

For each workload tranche, the team needs to execute two distinct steps. Check for functional correctness, then tune the performance of the workload. The latter concerns physical design choices such as indexes or table geometries.

Rarely do queries need to be adjusted.

At the end of the project, the team needs to catch up the data transfer so the new system is fully equivalent. Catching up on data transfer between tranches reduces the load in the home stretch.

The time needed to implement the full virtualization of a system varies. For a mid-range system, this can be a calendar quarter; for a large system, it could take six to 12 months. In our case, the full implementation took 10 months.

A New Way of Modernizing Data Estates Emerges

What is most remarkable was the final cutover from the old to the new system. Our customer made the transition without telling their thousands of business users. Instead, they flipped the switch on a weekend, repointing all business users to the new stack.

Business users coming back on Monday noticed their applications mysteriously running faster. But they had no idea of the magnitude of the change. That is unheard of. Even simple database upgrades cause more downtime.

Looking back, how does this new way of modernizing a data estate compare to the conventional approach?

Instead of 10 months, a conventional approach would have taken years. Instead of a small agile team of about 20, it would have required hundreds or even thousands of people to make changes. Coordination alone would have made the cutover a project for several calendar quarters.

In the end, the customer saved an estimated nine figures on their five-year TCO over a conventional approach, assuming that a conventional approach would have succeeded at all. Given the high failure rate of conventional migrations, this is a very optimistic assumption.

This project can serve as a blueprint for data warehouse migrations. Whenever time, cost and risk are of the essence, DBV is a strong contender.

February 17, 2023
Data Mobility: Why it’s a Must and How to Achieve It

The workplace has undergone a drastic change, and so has the way companies need to move their data. Wondering how those two points are connected? Let’s dig in.

It’s clear that remote work is here to stay: Just over a quarter (26%) of U.S. employees are currently working remotely, and a staggering 36.2 million Americans will work remotely by 2025. Naturally, this huge influx of remote (and hybrid) workers has necessitated changes in IT infrastructure. It has changed how people work, engage with their corporate databases and applications and what they expect from their corporate technology infrastructure.

In the past, companies migrated their data every four or five years when they refreshed their storage infrastructure. CIOs and storage experts now think differently about how data centers are built and optimized. Important questions all companies should consider are: Does it make sense to build out storage capacity ahead of the need? Do we really plan to have our databases and applications hosted on-premises over the next five years? One CIO recently told me, “If you aren’t talking about your future in the cloud, you aren’t paying attention.”

The Need to Migrate Data More Frequently

The shift toward a more flexible storage architecture, even in a hybrid cloud environment, has created the need to migrate data much more frequently, fueling the demand for data mobility. Robust data mobility is essential in today’s world: Companies need to securely and efficiently move their data to accommodate a strategy that accommodates both on-premises and cloud; even multiples of these. Data mobility constraints can’t dictate the success of an organization’s digital transformation. Moving data to the best location based on an organization’s goals—whether they’re related to cost, manageability, performance or some other factor—is critical.

Another critical factor to consider regarding data mobility: The data must always be accessible to the end user, regardless of when or where it’s moving. In the age of remote work, people are working based on their own unique schedules, so data must be accessible at all times.

In fact, data from Microsoft recently revealed that 30% of its employees choose to work at night. Gone are the days when systems could be offline for data migration during “non-working hours.” Remote work demands continuous data access so employees can stay productive and work at their convenience.

With this in mind, here are three key steps companies can take to achieve data mobility.

1. Strategy
When determining where to store data, consider who will need access to the data, what performance level will be required and what the cost will be for each data storage location. Is everything best contained in a single public cloud? Is a multi-cloud strategy or a hybrid cloud storage environment a better approach? We often talk about data mobility and data migration interchangeably, but not all data migration solutions can continuously move data between storage environments.

Organizations must also determine how they will move their database and applications to the new environment. If this step isn’t addressed during the strategy stage, it can become an afterthought and create problems down the road.

2. Deployment
Once an organization has determined which storage environment best fits its needs, it’s time to move the data from its source to its destination. A single, one-time migration involves moving the data from its current location to a new location and stepping away until the next migration. Data mobility solutions are ideal when data needs to move back and forth, requiring data mobility solutions (e.g., moving from on-premises to a multi-cloud environment).

During this step, organizations must ask the following questions and plan accordingly to minimize any negative impact: How long will this migration take? How secure is the data in transit from source to destination? How disruptive will the transition be to end users? What will the impact on the organization be? How much manual effort is required?

Sometimes moving data can make applications sluggish, affecting productivity and resulting in workers overloading the help desk with inquiries about what’s happening. Data migration tools can help organizations tackle deployment efficiently, securely and without disruption to end users (again, continuous data access is vital).

3. Assessment and cutover
After the data has arrived at its new destination, it’s time for the cutover process (i.e., running the application in its new location and removing the old storage). Cutover is when problems can arise if everything isn’t mapped correctly. Organizations need protections in place to ensure there is no risk to their operations should something go awry during the process. Additionally, minimizing the impact of this transition for end users is vital, so organizations should also be doing whatever they can to reduce downtime.

Companies can consider leveraging a data mobility tool that can synchronize in the background without impacting users. So, while users are still working on the old site, new infrastructure can be updated simultaneously, requiring just seconds to minutes of cutover. This time can be planned for off hours so that it’s minimally disruptive and doesn’t require notifying employees.

The technology landscape is constantly changing, and so are storage vendors’ offerings. Companies need solid data mobility to pivot at a moment’s notice to take advantage of newer, better and more cost-effective capabilities. Data mobility gives companies the flexibility to stay competitive, and data migration tools that allow continuous data access keep workers productive.

February 10, 2023
Where Does Observability Stand Today, and Where is it Going Next?

As we move into the second half of 2023, it’s only natural to engage in the annual ‘state of the industry’ reviews and take a general inventory of the past year while looking out into the immediate future. Observability has gained traction among DevOps and platform engineering teams over the past year and is poised to continue that velocity. But my observation is that the current observability model is broken.

Too Much Data

Long story short: Many organizations (and engineers specifically) agree that they are sending too much data to their monitoring tools, resulting in an inability to garner valuable insights because the systems are massively overloaded. Sending this high volume of data also costs too much and requires an inordinate amount of time for analysis, which many organizations don’t have. Put simply–more data doesn’t mean more clarity; in fact, we see the opposite.

Massive adoption of cloud-native technologies such as Kubernetes that heighten the challenge of efficiently monitoring environments is also accelerating this problem. Beyond increasing costs and complexity, many teams seem to feel that their current approach doesn’t create the desired level of observability for Kubernetes, full stop.

Another major issue we see on the observability landscape relates to the overall maturity of monitoring processes and teams. In many instances, even when organizations have effective tools, they struggle to interpret the data produced into targeted conclusions that their teams can translate into effective actions. And this is further amplified by the outstanding engineering talent shortage because good people are harder to find and retain than ever before.

We also see a lot of pervasive challenges related to organizations’ desire to use popular open source monitoring tools-especially if they are attempting to do so in an integrated manner or to support rapidly growing deployments at scale.

Overwhelmed by Observability Data?

In summary, if you’re overwhelmed by observability data, if complexity and costs grow out of control and you’re also struggling to make sense of the available information and improve your cloud applications, we think that means that the current observability model really is broken.

To that end, gaining a deeper perspective into end user opinions on these topics, among others, is the precise goal of the annual DevOps Pulse Report, currently in its sixth year and now in the survey stage. We predict that this year’s research project will underline the aforementioned challenges and help frame how teams might seek to address them.

What types of findings do we hope to unearth? Last year, respondents revealed that despite a growing perception of DevOps maturity, it was actually taking them more time to respond to emerging problems. Roughly 64% of respondents reported that MTTR during production incidents was over an hour, compared to only 47% the previous year.

Last year’s report also highlighted the reality that monitoring complex environments remains difficult with over 52% of respondents citing Kubernetes, microservices and serverless as their primary challenges in gaining improved observability.

Both of these previous findings also seemed to underline the growing issues of data overload and spiraling complexity. We’ve also seen previous evidence of a lack of maturity around distributed tracing—a critical observability practice aimed directly at increasing visibility into complex architectures—despite the fact that 75% of respondents listed it as a priority in last year’s research.

Is your team struggling to make sense of its available monitoring data, or do so in a way that aligns with your current processes or budgets? Hopefully, the 2023 DevOps Pulse Report, set to publish early this year, will put some numbers behind these assertions and help us all understand how to move forward.

If you want to participate, you can do so here.

February 6, 2023
OpenAI Hires 1,000 Low Wage Coders to Retrain Copilot ¦ Netflix Blocks Password Sharing

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: ChatGPT darling OpenAI wants people to write code in English, and the unintended consequences of blocking shared accounts. (more…)

February 2, 2023
Stream Big, Think Bigger: Analyze Streaming Data at Scale

Data streams are all the rage. Once a niche element of data engineering, streaming data is the new normal—more than 80% of Fortune 100 companies have adopted Apache Kafka, the most common streaming platform, and every major cloud provider (AWS, Google Cloud Platform and Microsoft Azure) has launched its own streaming service.

This shift is fueled by the growing demand to deliver events quickly, reliably and at scale to support a wide range of internal and external applications. From always-on, customer-facing applications to mission-critical operations, a growing number of use cases depend on subsecond updates—further upping the ante for real-time data processing and dissemination.

There’s no doubt 2023 will be a big year for streaming technology.

Meanwhile, as streaming moves toward ubiquity, we’re seeing another shift in the way businesses use the data: Analyzing events as they’re created for real-time insights. With the right tools in place, people can instantly compare what’s happening now to what happened in the past, triage problems as they occur and make time-sensitive decisions in the moment.

With the proliferation of data streams, a new set of use cases and requirements for real-time analytics has emerged. To unlock the power of streaming in 2023, data teams will need to look beyond the traditional batch-oriented stack and adopt a streaming-native approach to the analytics architecture.

The Streaming Paradigm Shift

Data has always been at the center of how businesses operate. Until recently, we lived in a very batch-dominant world of business and data. The goal of data infrastructure was to identify data at a fixed point in time and store it for eventual use. The evolution from daily batch operations on mainframes to the internet-driven always-on world replaces fixed, static “data at rest” with fast-moving data in motion, as information flows freely between applications and data systems within and between organizations.

Data at rest still exists, and batch systems continue to serve a variety of reporting purposes. But reality isn’t fixed or static, and to meet the demands for seamless, authentic data experiences, the systems we build must be designed for data in motion.

Hence, the rise of streaming technology—and with it, a new mindset for how we think about data. As streaming increases, streaming data platforms increasingly act as an organization’s central nervous system, connecting the different functions and driving mission-critical operations. Evolving technologies are purpose-built to support data in motion systems: Stream processors and event databases.

Apache Druid fits into the purpose-built category; as a real-time analytics database, it is designed to enable users to query every event as it joins the data stream and do it at a tremendous scale while enabling subsecond queries on a mix of stream and batch data.

Thousands of businesses are already using streaming processors like Kafka and Amazon Kinesis with Druid to build cutting-edge systems that make terabytes to petabytes of streaming data accessible to applications and humans in milliseconds. Some of those businesses—including Reddit, Citrix and Expedia—were highlighted at Current 2022, the annual streaming event organized by Confluent.

The next evolution of data intelligence is here, and it’s based on the ability to react to events as they are happening. But on the growth chart of streaming adoption, we are only at the beginning of an upward curve—a critical inflection point where streaming (and the technology that is built for it) starts to become the bedrock of everyone’s data architecture.

So, when it comes to enabling scalable, subsecond analytics on streaming data, many developers and data innovators are wondering, ‘How?’

We’re Streaming! Now What?

At Current, we spoke with hundreds of Kafka users who were turning their attention to building out the next phase of their data streaming pipelines and embedding them more deeply into their organizations.

While streaming adoption is becoming more widespread, most companies still have just one or two use cases they’re solving with a streaming platform. At Current, many people spoke about how Kafka effectively set their data in motion. But when it came time to analyze or put those streams to work in a user-facing application, their “data in motion” turned into “data in waiting” because their analytics system was designed for batch data, not streaming.

To solve this problem, a new type of database is needed—and that’s why Druid was created. Turning billions or trillions of events into streams that can then be immediately queried by thousands of users simultaneously is where stream processors like Kafka, in combination with Druid, have unlocked a new set of capabilities that are enabling new developer-built analytics applications.

For example, Reddit generates tens of gigabytes of event data per hour from advertisements on its platform. To let advertisers understand their impact and decide how to target their spending, Reddit needs to enable interactive queries across the last six months of data—hundreds of billions of raw events to slice and dice. But they also needed to empower advertisers to see user groups and sizes in real-time, adjusting based on interests (sports, art, movies, technology…) and locations to find how many Reddit users fit their target. To achieve this, they built a Druid-powered application that ingests data from Kafka and enables Reddit’s advertising partners to make real-time decisions and get the best possible ROI on their campaigns.

Stream-to-Batch Vs. Stream-to-Stream

Reddit chose Druid as the database layer of their application in large part because of its close integration with Kafka, as Druid was designed to ingest and analyze streaming data. This sets Druid apart from virtually every other analytics database, the rest of which were built for batch ingestion.

Batch ingestion works exactly the way it sounds—it takes chunks of stream data, dumps them into a file, processes the file, then loads it into the database. Using a batch-based system to analyze streams introduces a bottleneck to your streaming pipeline. By contrast, Druid provides a connector-free integration with the top streaming platforms and handles the latency, consistency and scale requirements for high-performance stream analytics cost-effectively.

In addition to out-of-the-box integrations, Druid has built-in indexing services that provide event-by-event ingestion. This means streaming data is ingested into memory and made immediately available for use. No waiting for events to be processed through batch processes before they can be queried. This capability, coupled with exactly-once semantics, guarantees data is always fresh and consistent. If there’s a failure during stream ingestion, Druid will automatically continue to ingest every event exactly once (including events that enter the stream during the outage) to prevent any duplicates or data loss.

However, perhaps the most powerful reason to use Druid for streaming data analytics is its near-infinite scalability. For even the largest, most complex ingestion jobs, Druid can easily scale into the tens of millions of events per second. With variable ingestion patterns, Druid avoids resource lag (as well as overprovisioning) by enabling dynamic scaling. It combines the query and ingestion performance of shared-nothing architecture with a cloud data warehouse’s flexibility and non-stop reliability. This means you can add computing power and scale out without downtime or rebalancing, which is handled automatically.

In essence, while using a cloud data warehouse to serve both batch-oriented and real-time use cases might sound efficient, doing so defeats the purpose of a streaming pipeline.

“The weakest link in the chain sort of dominates everything,” Databricks CEO Ali Ghodsi told the audience at Current. “If you have one step of your pipeline that is batch-processed and really slow, then it doesn’t matter how fast you are in the rest of the pipeline.”

That’s why when Confluent’s data engineering team needed to build a real-time observability platform, they chose to use Kafka and Druid together. Today, the platform ingests over five million events per second from Kafka and handles hundreds of concurrent queries from a large team of analysts.

“We do a lot of things today that we didn’t imagine doing three or four years ago,” Matt Armstrong, who leads the observability and data platform team at Confluent, said in a Druid Summit 2022 interview. “From a [product] perspective, you could think of Druid as being able to power a great product [Confluent Cloud] with a great support experience—but perhaps more importantly—an unconstrained future.”

It’s Time to Think Big

We’re already moving to this entirely new way of thinking about moving, analyzing, and sharing data. Streaming technology has opened the door for a plethora of new use cases and products to come to fruition. We believe organizations that view streaming data as a force multiplier and implement a streaming-native approach to analytics will gain a competitive advantage in 2023 and beyond.

Druid is purpose-built for these new streaming use cases, but it’s also built for a new type of application—one that draws characteristics from both the analytics and transactional database worlds. Industry-leading teams have built mission-critical applications that enable operational visibility, customer-facing analytics, drill-down exploration and real-time decision-making using streaming data and Druid. This trend is now starting to take hold and the next wave of analytics applications is already being built.

January 27, 2023
What’s Ahead for the Future of Data Streaming?

Data is everywhere. It’s being produced by almost everyone and everything around us. For businesses to be competitive, they need to build resilient, scalable systems that can efficiently tap into and react to this ever-increasing volume of data. Companies are meeting this need by moving to real-time data streaming to develop new solutions and improve their businesses and customer experience.

This category of data streaming has been growing. And, driven by the rise of real-time digital experiences like ride-sharing, fraud detection, inventory management and more, the growth has accelerated.

So what’s next for data streaming?

Data as a Product Grows in Popularity

The concept of data as a product will become more mainstream. Across many industries, data streaming is becoming central to how businesses operate and disseminate information within their companies. However, there is still a need for broader education about key data principles and best practices, like those outlined through data mesh.

Data as a product is foundational to rethinking how data produced in one part of a business can be reliably used in another. Data products with formalized ownership and responsibilities, combined with a platform for discoverability and accessibility, result in significantly fewer barriers and a much easier data access experience for all. Schemas, the Kafka API and maturing specification standards like ASyncAPI can be a part of this evolution.

As more folks embrace this mindset, I expect to see companies shift from using ad-hoc data pipelines to ones based on data products, driving a centralized system that can power both better analytical insights and operational execution.

Growth of Self-Service Solutions

With the rise of cloud-based offerings, I expect to see more companies offering more accessible, pay-as-you-go, self-service solutions. We’ve already seen this shift over the past few years; cloud-based offerings are making it easier for companies to try out a data product based on their usage (or lack thereof). Given the current market conditions, these offerings will be more appealing to organizations that aren’t ready to sign a multi-year contract for a product offering or focus solely on a single technology.

Increase in Cloud Migrations and the Continued Rise of SaaS

Cloud is here to stay. We’ve proven that it works, that it’s safe and secure, and for many folks, it just makes sense. This should come as no surprise to anyone, but expect the trend of cloud migrations to continue.

In the context of Apache Kafka, building out a Kafka-based data streaming solution used to mean hosting and managing your own Kafka cluster. If you’ve ever been in that position, you know that usually means more DevOps work and less focus on building applications to get the most out of your data. Not only do cloud-based Kafka offerings eliminate much of the DevOps headaches that come with a Kafka cluster, but they also make it much easier and faster to hit the ground running and begin streaming your data.

I’m personally excited by what the cloud and SaaS can do to make Kafka and data streaming more accessible. In my visits to developers and companies around the globe, I see this enthusiasm shared by so many. Entire geographic regions and industries are taking the opportunity to move to the cloud and dive headfirst into data streaming.

KRaft Leads to Rise in Kafka Adoption

As of Apache Kafka 3.3, the KRaft consensus protocol has replaced Zookeeper and is production-ready. Cluster metadata can now be maintained and managed from within the cluster, meaning there’s one less thing to worry about when deploying Kafka instances.

While there are many options for cloud-based Kafka-as-a-Service, improvements such as KRaft in open source Kafka make self-hosting even easier. This, along with the work done to make it simpler to run managed Kafka services on local machines and test environments, means that Kafka no longer has to be synonymous with complexity. With the difficulty of managing and maintaining a Kafka instance no longer preventing folks from making use of data streaming, I predict that the adoption of Kafka will increase even more.

Data streaming will continue to mature as a transformational technology for companies of all sizes, serving as a critical infrastructure that most organizations will need to run their business.

If you haven’t yet explored what data streaming has to offer you, maybe this will be your year to check it out!

January 27, 2023