devopsy.kinsta.cloud

Tag: data pipelines

Why Data Contracts Need Apache Kafka and Apache Flink
Imagine it’s 3 a.m. and your pager goes off. A downstream service is failing, and after an hour of debugging you trace the issue to a tiny, undocumented schema change made by an upstream team. The fix is simple, but it comes with a high cost in lost sleep and operational downtime.

This is the nature of many modern data pipelines. We’ve mastered the art of building distributed systems, but we’ve neglected a critical part of the system: the agreement on the data itself. This is where data contracts come in, and how they fail without the right tools to enforce them.

The Importance of Data Contracts

Data pipelines are a popular tool for sharing data from different producers (databases, applications, logs, microservices, etc.) to consumers to drive event-driven applications or enable further processing and analytics. These pipelines have often been developed in an ad hoc manner, without a formal specification for the data being produced and without direct input from the consumer on what data they expect. As a result, it’s not uncommon for upstream producers to introduce ad hoc changes consumers don’t expect and can’t process. The result? Operational downtime and expensive, time-consuming debugging to find the root cause.

Data contracts were developed to prevent this.

Data contract design requires data producers and consumers to collaborate early in the software design lifecycle to define and refine requirements. Explicitly defining and documenting requirements early on simplifies pipeline design and reduces or removes errors in consumers caused by data changes not defined in the contract.

Data contracts are an agreement between data producers and consumers that define schemas, data types, and data quality constraints for data shared between them. Data pipelines leverage distributed software to map the flow of data and its transformation from producers to consumers. Data contracts are foundational to properly designed and well behaved data pipelines.

Why We Need Data Contracts

Why should data contracts matter to developers and the business?

First, data contracts reduce operational costs by eliminating unexpected upstream data changes that cause operational downtime.

Second, they reduce developer time spent on debugging and break-fixing errors. These errors are caused downstream from changes the developer introduced without understanding (via the missing data contract) their effects on consumers.

Third, formal data contracts aid the development of well-defined, reusable data products that multiple consumers can leverage for analytics and applications.

The consumer and producer can leverage the data contract to define schema and other changes before the producer implements them. The data contract should specify a cutover process, so consumers can migrate to the new schema and its associated contract without disruption.

Three Important Data Contract Requirements

Data contracts have garnered much interest recently, as enterprises realize the benefits of shifting their focus upstream to where data is produced when building operational products that are data-driven. This process is often called shift left.

In a shift-left data pipeline design, downstream consumers can share their data product requirements with upstream data producers. These requirements can then be distilled and codified into the data contract.

Data contract adoption requires three key capabilities:
- Specification — define the data contract
- Implementation — implement the data contract in the data pipeline
- Enforcement — enforce the data contract in real-time
There are a variety of technologies that can support these capabilities, however, Apache Kafka and Apache Flink are among the best technologies for this purpose.

Apache Kafka and Apache Flink for Data Contracts

Apache Kafka and Apache Flink are popular technologies for building data pipelines and data contracts due to their scalability, wide availability, and low latency. They provide shared storage infrastructure between producers and consumers. In addition, Kafka allows producers to communicate the schemas, data types, and (implicitly) the serialization format to consumers. This shared information also allows Flink to transform data as it travels between the producer and consumer.

Apache Kafka is a distributed event streaming platform that provides high-throughput, fault-tolerance, and scalability for shared data pipelines. It functions as a distributed log enabling producers to publish data to topics that consumers can asynchronously subscribe to. In Kafka, topics have schemas, defined data types, and data quality rules. It can store and process streams of records (events) in a reliable and distributed manner. Kafka is widely used for building data pipelines, streaming analytics, and event-driven architectures.

Apache Flink is a distributed stream processing framework designed for high-performance, scalable, and fault-tolerant processing of real-time and batch data. It excels at handling large-scale data streams with low latency and high throughput, making it a popular choice for real-time analytics, event-driven applications, and data processing pipelines.

Flink often integrates with Apache Kafka, using Kafka as a source or sink for streaming data. Kafka handles the ingestion and storage of event streams, while Flink processes those streams for analytics or transformations. For example, a Flink job might read events from a Kafka topic, perform aggregations, and write results back to another Kafka topic or a database.

Contract Versioning and Evolution

Kafka supports schema versioning and can support multiple versions of the same data contract as it evolves. Kafka can keep the old version running with the new version, so new clients can leverage the new schema while existing clients are using the old schema. Mechanisms like Flink support for materialized views help accomplish this.

How Kafka and Flink Help Implement Data Contracts

Kafka and Flink are a great way to build data contracts that meet the three requirements outlined earlier — specification, implementation, and enforcement. As open source technologies, they play well with other data pipeline components that are often built using open source software or standards. This creates a common language and infrastructure around which data contracts can be specified, implemented, and enforced.

Flink can help enforce data contracts and evolve them as needed by producers and consumers, in some cases without modifying producer code. Kafka provides a common interface that supports specification while making implementation practical.

Kafka and Flink encourage the reuse of the carefully crafted data products specified by data contracts. Kafka is a data storage and sharing technology that makes it easy to enable additional consumers and their pipelines to use the same data product. This is a powerful form of software reuse. Kafka and Flink can transform and shape data from one contract into a form that meets the requirements of another contract, all within the same shared infrastructure.

Implementation and Enforcement

You can deploy and manage Kafka yourself, or leverage a Kafka cloud service and let others manage it for you. Any data producer or consumer can be supported by Kafka, unlike strictly commercial products that have limits on the supported producers and consumers.

You could get enforcement via a single database if all the data managed by your contracts sits in that database. But applications today are often built using data from many sources. For example, data streaming applications often have multiple data producers streaming data to multiple consumers. Data contracts must be enforced across these different databases, APIs, and applications.

You can specify a data contract at the producer end, collaborating with the producer to get the data in the form you need. But enforcement at the producer end is intrusive and complex. Each data producer has its own authentication and security mechanisms. The data contract architecture would need to be adapted to each producer. Every new producer added to the architecture would have to be accommodated. In addition, small changes to schema, metadata, and security happen continuously. With Kafka, these changes can be managed in one place.

In contrast, Kafka sits between producers and consumers. With Kafka Schema Registry, producers and consumers have a way of communicating what is expected by their data contract. Because topics are re-usable, the data contract may be re-usable directly or it could be incrementally modified and then re-used.

Kafka also provides shared, standardized security and data infrastructure for all data producers. Schemas can be designed, managed, and enforced at Kafka’s edge, in cooperation with the data producer. Disruptive changes to the data contract can be detected and enforced there

Data contract implementation needs to be simple and built into existing tools, including Continuous Integration and Continuous Delivery (or Deployment) CI/CD. Kafka’s ubiquity, open source nature, scalability and data re-usability make it the de facto standard for providing re-usable data products with data contracts.

Best Practices for Developers Building Data Contracts

As a data engineer or developer, data contracts can help you deliver better software and user experiences at a lower cost.

Here are a few guidelines for best practices as you start leveraging data contracts for your pipelines and data products.
1. Standardize Schema Formats: Use Avro or Protobuf for Kafka due to their strong typing and compatibility features. JSON Schema is an alternative but less efficient.
2. Automate Validation: Use CI/CD pipelines to validate schema changes against compatibility rules before deployment. Make sure your code for configuring, initializing, and changing your Kafka topic’s schemas is part of your CI/CD workflows and check-ins.
3. Version Incrementally: Use semantic versioning (e.g., v1.0.0, v1.1.0) for schemas and document changes clearly. This should be part of your CI/CD workflows and run-time checks for compatibility.
4. Monitor and Alert: Set up alerts for schema and type violations or data quality issues in Kafka topics or Flink jobs.
5. Collaborate Across Teams: Ensure producers and consumers (e.g., different teams’ Flink jobs) agree on the contract upfront to avoid mismatches. Leverage collaboration tools (preferably graphical) that allow developers, business analysts, and data engineers to jointly define, refine, and evolve the contract specifications.
6. Test Schema Evolution: Simulate schema changes in a staging environment to verify compatibility with Kafka topics and Flink jobs.
You can find out more on how to develop data contracts with Kafka here.

Conclusion

Kafka and Flink provide a common language to define schemas, data types and data quality rules. This common language is shared and understood by developers. It can be independent of the particular data producer or consumer.

Kafka and Flink have critical capabilities to make data contracts practical and widespread in your organization with:
- Broad support for potential data producers and consumers
- Widespread adoption, usage, and understanding, partly due to their open source origins
- Many implementations available, including on-prem, cloud-native, and BYOC (Bring Your Own Cloud)
- The ability to operate at both small and large scales
- Provides mechanisms to modify data contracts and their schemas as they evolve
- Sophisticated mechanisms for evolving schemas and reusing data contracts when joining multiple streams, each with its own data contract.
Data contracts require a new culture and mindset that encourages data producers to collaborate with data consumers. Consumers need to design and describe their schema and other data pipeline requirements in collaboration with producers, and guided by developers and data architects.

Kafka and Flink make it much easier to specify, implement and enforce the data contracts your collaborative producers and consumers develop, getting your data pipelines up and running faster, operating more efficiently, without downtime, while delivering more value to the business.
February 3, 2026
Data Reliability Needs to Shift Left

Data pipelines feed and transform data for consumption and are increasingly growing in complexity. As enterprises continue to add data sources, they run the risk of breaking their data pipelines due to data errors, flawed logic or insufficient resources for processing. The challenge for every modern data team is to establish data reliability as early as possible in the data journey and to create optimized data pipelines capable of performing and scaling to meet the business and technical needs of the enterprise. In the context of data observability, “shift left” refers to a proactive approach that incorporates observability practices during the early stages of the data lifecycle. This concept is borrowed from software development methodologies and emphasizes addressing potential issues and ensuring quality from the outset.

Shifting left involves integrating observability practices and tools into the data pipeline and infrastructure from the beginning. This approach avoids treating observability as an afterthought or applying it only in later stages to identify and resolve data reliability, integrity, and performance issues as early in the development process as possible, minimizing the likelihood of problems cascading downstream.

The Economic Impact of Poor Data

To effectively manage data and optimize data pipelines, it is crucial to detect and address data incidents as early as possible within the supply chain.

The “1 x 10 x 100 Rule” is applicable in a variety of processes, including software development. It states that the cost of fixing a problem increases exponentially, depending on the stage at which it is detected. The rule can be extended to data pipelines and supply chains, where it suggests that detecting and rectifying a problem in the data landing zone (i.e., where source data is fed) incurs a cost of $1, whereas identifying and addressing the issue in the transformation zone (i.e., where data is transformed into its final format) increases the cost to $10. If the problem is only detected in the consumption zone (where data is in its final format and is accessed by users), the cost rises significantly to $100.

Shift Left Data Reliability

The process of identifying data pipeline issues has become more challenging for data teams as data supply chains continue to evolve and grow in complexity, mainly resulting from the following factors:

● Expanding sources: The number of data sources (such as Databricks, Snowflake, Redshift, Teradata and many others) being incorporated into the supply chain has significantly increased. Organizations now integrate data from a wider range of internal and external sources, contributing to the complexity of data management and processing.
● Advanced data transformation logic: The logic and algorithms used to transform the data within the supply chain have become more sophisticated. Complex transformations and calculations are applied to raw data to derive meaningful insights, necessitating robust systems and processes to handle the intricacies involved.
● Resource-intensive processing: The processing requirements for data within the supply chain have significantly escalated. With larger volumes of data and more complex operations, organizations must allocate substantial computing resources, such as servers, storage and processing power, to handle the data processing workload effectively.

By proactively addressing data incidents at an early stage, organizations minimize the potential impact and cost associated with data issues. This not only ensures the reliability and accuracy of data consumed by users but also safeguards the integrity of downstream processes and decision-making. Ultimately, the shift left approach to data reliability promotes efficiency, reduces costs, and enhances overall data quality and trustworthiness.

How Can Data Teams Shift Left?

To effectively shift left in your data reliability solution, certain capabilities are crucial; these include:

● Early Data Reliability Checks: Conduct data reliability tests before data enters the data warehouse and data lakehouse. This ensures that bad data is identified and filtered out early in the data pipelines, preventing its propagation to the transformation and consumption zones.
● Support for Data-in-Motion Platforms: Enable support for data platforms like Kafka and monitor data pipelines within Spark jobs or Airflow orchestrations, which enables real-time monitoring and metering of data pipelines.
● File Support: Perform checks on different file types and capture file events to determine when incremental checks should be performed.
● Circuit-Breakers: Integrate APIs that incorporate data reliability test results into your data pipelines, which can halt data flow when bad data is detected. By preventing the spread of bad data, circuit breakers protect downstream processes from being affected.
● Data Isolation: Identify and isolate bad data rows, preventing their continued processing.
● Data Reconciliation: Implement data reconciliation capabilities to ensure consistency and synchronization of data across multiple locations.

Shifting left your data reliability practices empowers data teams to focus on innovation, reduces costs, enhances data trust and enables agile business processes. Embracing this approach makes data more usable and resilient and delivers a more robust and reliable data ecosystem that supports informed decision-making and drives organizational success.

October 23, 2023
Why We Need to Rethink Data Pipelines

Data is the fuel that powers modern business. But as demand for data surges, so does the pressure on data leaders and practitioners to deliver it. Businesses need resilient data pipelines that deliver critical insight for real-time decision-making to users on demand. However, against the backdrop of today’s chaotic modern data ecosystems, this is much easier said than done.

DevOps teams are struggling with a complex combination of legacy and diverse technology systems across multiple environments. The resulting siloed systems and an ever-shifting data supply chain cause significant data integration friction. This friction keeps businesses from creating seamless and resilient pipelines to drive digital transformation and other business outcomes. Businesses need a better way to take the pressure off data leaders and DataOps practitioners and enable line-of-business teams to do more of the “last mile” data collection and analysis themselves. To achieve this, they need a unified, end-to-end platform designed to build resilient pipelines.

Data Demand Sparks Friction

The acceleration of digital transformation has created a demand for data that far exceeds supply. This has become especially true as competitive advantage gets harder to carve out and worrying macroeconomic headwinds gather. In today’s climate, all areas of the business are demanding data, but it’s hard to meet every request all the time.

Research that polled data leaders and practitioners shows nearly half (48%) of admin and operations and customer service departments request data at least weekly, followed closely by accounting and finance (44%), IT and digital (43%) and sales and marketing teams (40%). And the strain is showing. Over half (59%) of data leaders said changing priorities have created significant data supply chain challenges.

The problem is one not just of capacity but complexity. Building pipelines from source to destination requires rules to integrate, transform and process data. But when data is siloed across cloud, legacy and mainframe systems and stored in inconsistent formats, creating bespoke data pipelines to fulfill departmental requests is a huge challenge.

Legacy systems also bring another challenge. Over half of data leaders say it’s so difficult to unlock data from legacy systems like mainframes that they simply don’t bother. That will ultimately undermine the value of the cloud analytics tools that many business leaders see as a single source of truth for decision-making–especially as legacy systems often hold decades of vital business insights.

This patchwork approach creates extra toil for DevOps, DataOps and technical teams and means that over two-thirds (68%) of data leaders said they are being prevented from delivering data at the speed the business needs.

The Scourge of Broken Pipelines

For many organizations, building pipelines is also a labor-intensive job requiring a high degree of manual effort to produce hand-coded, one-off solutions. The resulting pipelines are brittle and vulnerable to being disrupted whenever there is a shift in the environment–such as adding new data sources. In fact, two-fifths (39%) of data leaders admit their pipelines crack at the first sign of trouble, and 87% have experienced a break at least once a year. More than one in 10 say it happens at least once a day.

The most significant cause of breakages is due to bugs and errors being introduced during a change (44%) while infrastructure changes such as moving to a new cloud (33%) and credentials altering or expiring (31%) are also creating disruption. These broken pipelines immediately impact the corporate bottom line. They also force technical teams into a vicious cycle of firefighting. Data engineers spend, on average, almost a third (31%) of their time troubleshooting and recoding broken pipelines–time that could be better spent on value-adding tasks.

But the negative impacts aren’t only felt operationally. The breakage of any pipeline can also lead to bad decision-making. For example, a supply chain director working with old data may over- or under-order goods or a financial trader making stock picks on out-of-date intel may lose money for a client.

Unleash the Power of Data

For organizations looking to overcome these challenges, the first goal is to reduce the workload on DevOps, DataOps and technical data teams by empowering business units and end users to do more. Research shows that 70% of technical data leaders are responsible for the last mile of data collection and analysis while 86% of them would prefer lines of business teams, such as marketing or finance, to be empowered to do this independently. However, this can only become a reality if organizations invest in the right data integration platform. It must be able to build, run, monitor and manage smart data pipelines at scale from a single console–across all cloud and on-premises environments. And it must be able to deliver resilient pipelines built to withstand continuous change.

Only by eliminating data integration friction and enabling self-service analytics for lines of business can organizations accelerate digital transformation. Doing so will unleash the power of data across the enterprise and reduce the burden of overworked data leaders and practitioners.

May 9, 2023
How to Modernize Data Integration

In times of uncertainty, accurate real-time data becomes essential to informed decision-making. When the pandemic ends and businesses begin to reopen, this will be truer than ever. While there is no precedent for COVID-19’s impact on the economy, what is clear is that the enterprise is clamoring for more analytics than ever. Data engineering is at the heart of operationalizing the delivery of that data, wherever it originates and wherever it’s needed.

To operationalize data flows, engineers must be able to collaborate with different personas, reuse peer assets, support data pipelines in production and, as platforms and business requirements change, evolve quickly and with confidence. The focus of data engineering can no longer be limited to ease-of-use and developer productivity. In today’s fast-changing world, we need to enable modern engineers to embrace DataOps. In doing so, they will be able to guarantee continuous data for fast, confident decisions by and for the business.

Current Options Are Too Simple or Too Complex

Current approaches follow either a “grab-and-go” mentality that focuses on simplifying ad-hoc access for first-time analysis, or lurches towards specialized coding for anything complex. Both of these approaches lead to ongoing and recurring headaches. Significant effort is required to debug data pipelines, rerun them and rework them every time a change happens. Today’s data engineers end up spending 80% of their time just keeping the lights on, leaving very little time for new value-added work.

Using Intelligent and Responsive Data Pipelines

Data engineers face mounting pressures and challenges, from an ever-increasing project backlog to an accelerating pace of upstream changes to the emergence of cloud data platforms driving re-platforming projects. To tackle these issues, engineers need a solution that allows them to operationalize their work by abstracting the “what” of the data—the business meaning and the logic—away from the “how” of the data—the technical implementation details that the business doesn’t care about. To achieve this, businesses need to think about data pipelines as dynamic, intelligent sources that can (and should) be capable of real-time adjustments.

There are three ways that data pipelines can be considered smart or dynamic: they are easy to start and easy to extend; they are resilient to changes that occur over time and they are portable, supporting hybrid and multi-cloud environments.

Companies that have focused on removing the “how” by developing intelligent and responsive systems free up their engineers to stay focused on the “what” – what are the outputs? Are the analytics providing the right ROI or information? They also make it easy to get started building pipelines through intent-driven design, removing tedious tasks for engineers and empowering ETL developers to self-service. This includes the repetitive task of managing data drift, whether it’s a schema change, a database version upgrade or a file format change. Smart pipelines handle such changes with minimal to no intervention required.

Demand Continuous Data for the Business, Now

COVID-19 has segmented businesses into have-lots, haves and have-nots. The have-lots have experienced unimaginable growth and, consequently, are generating incredible amounts of data. They have had to deliver better engagement with their constituents, optimize their product offerings and stop bad actors, fraudsters and cybercriminals from exploiting the unexpected surge in demand for their business. If they have risen to the challenge, you can bet that they have modernized their data practice and can deliver continuous data to all consumers.

The haves and have-nots have generally been in survival mode, operating under short-term mindsets to reduce cost, consolidate as much as possible and wring out any waste. Depending on each individual situation, that may have meant staying on legacy platforms and postponing modernization efforts. Some have enjoyed cost savings by reducing consumption and negotiating with their vendors. Others have made a tactical move to the cloud, but simply as a path to lower costs of their data platforms. Only a select few have done the right thing and made the complementary investments in data engineering to reap the full benefits of their shift to cloud.

As this pandemic comes to an end, the prospects for the haves and the have-nots will start to look more promising. If you have been focused solely on keeping the lights on and managing costs, you may have been mortgaging your future. This needs to end. Don’t wait. By modernizing your ETL to create more dynamic, real-time data pipelines now, your engineers can operationalize their work, deliver a new level of innovation and self-service to the business and demonstrate their true worth as data heroes.

April 28, 2021
Falling Back in Love with Data Pipelines

It’s Spring 2020, and even in challenging and unprecedented times, love continues to bloom. That said, data pipelines are never far from my thoughts and the confluence of those two things got me thinking about data engineers and how their initial love for creating data pipelines often turns to loathing.

Data pipelines are the backbone of modern data systems, yet there’s still much to be desired when it comes to connecting and orchestrating the movement of data between data systems, including business intelligence, data processing, data warehousing or other applications. As an industry, we haven’t paid as much attention to the happiness of the people who build and maintain data pipelines as we have to the volume or velocity they support. As a consequence, today’s data engineers are building increasing resentment toward the technologies and tools that previously offered so much promise.

What is happening here, and can it be fixed?

Love at First Sight

When we first met, everything was magical. We traveled the data world, far and wide, no longer bound by the constraints of relational databases and ETL tools. Vertical scaling was a thing of the past, and with horizontal scaling, we felt we could reach the end of the earth together.

The introduction of Hadoop, Spark, Kafka and more was a truly magical evolutionary stage for data pipelines. We could store and process more data than ever before, derive insights never thought possible, and do it all at a fraction of the cost.

What was there not to love?

Definitely, Maybe

As our courting phase came to a close, we were happier than ever. We were spending more and more time together, eagerly building a future. We were both getting accolades at work, accompanied by exciting promotions and responsibilities. We were also growing increasingly comfortable with each other, and while at times it felt you were being a bit clingy, it was nice to be needed.

With pipeline technology continuing to advance, it felt scale and speed had few, if any, limits. We built more and more pipelines, linking outputs and inputs, tuning performance and performing increasingly complex transformations, all as part of an increasingly interconnected data ecosystem.

Our data teams grew, and with more data engineers we built even more pipelines. We also started to notice that these pipelines are quite a bit higher maintenance than originally thought. They require frequent tuning and optimizations, yet still break far more often than they used to, usually at the least desirable times–like at 3 a.m.

This is definitely turning into a love/hate kind of thing.

10 Things I Hate About You

I miss the old days. What happened? I’m just trying to hang out with my friends and you won’t stop calling–it’s suffocating me. When we do spend time together, the romance is gone. Horizontal scalability has lost its magic.

Most data engineering teams we encounter are firmly planted in this stage of relationship with data pipelines. They’re honestly feeling a bit trapped. They’ve built a ton of remarkable pipelines that do incredible things. These pipelines also require a tremendous amount of maintenance, and each new one they build only compounds the problem. As a result, we see most data engineering teams actively distancing themselves from data pipeline technology, building increasingly large layers of abstraction that sit between them and the raw technology.

Surely, all is not lost?

Something’s Gotta Give

Right when I thought we were at our tipping point, we had a breakthrough. We weren’t communicating effectively, our languages of love were entirely different. I wanted to focus on the destination, the end result, and you wanted to focus on the journey, the path to get there. There clearly was a gap, and we needed help bridging it.

Fortunately, for the data engineer and their muse, the data pipeline, there are advances in pipeline technology, bridging the gap and offering a path toward reconciliation. One such path combines declarative programming with an underlying control system that understands the declarative model. Such control planes translate declarative models into imperative instructions for the underlying infrastructure, making pipeline development far less plumbing, and far more architecting. Declarative programming allows data engineers to focus on the big picture, drawing on their experience and creativity to design what the optimal data pipeline should look like, rather than spending time repairing failures, tuning parameters or manually tracing code.

The underlying control system, or control plane as it’s called, imbues the pipeline with autonomy, data awareness and intelligent persistence. Autonomous data pipelines are less needy and can handle data changes on their own (rather than making the data engineer responsible for everything). Data awareness allows pipelines to intuitively know what changed and what subsequently needs to change. Intelligent persistence allows pipelines to be considerate of cost and resources and makes sure everything is fast, efficient and never duplicated.

Love Actually

I think it’s safe to say those among us who chose to enter data engineering didn’t do so imagining ourselves, infrastructure plumbers, spending hours upon hours maintaining data pipelines that fuel someone else’s data-driven creativity. No, data engineers want to create digital services and contribute business value, too–they’re just trying to keep everybody else out of the data muck in the meantime.

This is the dawn of a new era in data engineering, and many of the early adopters of this latest approach to pipeline development have found it enables creation with far less time, code and maintenance. By embracing this new method of development, data engineers can again see a happy future where more of their time and skills are devoted to the things for which they–and the business–have real passion.

May 18, 2020