devopsy.kinsta.cloud

Category: DataOps

  • DataOps Vs. DevOps: What’s the Difference?

    DataOps Vs. DevOps: What’s the Difference?

    There is a mindboggling amount of data today; to even measure it requires using a byte measurement called a zettabyte, which is one sextillion bytes (that’s 21 zeros). Currently, because such a ridiculous amount of data exists, there is a growing urgency to end wasteful data processes. From this environment, DataOps was born. Similar to the way that enterprises adopted DevOps to formalize and streamline wasteful development practices in the past, today, many large organizations turn to DataOps to formalize modern data management practices.

    What, Exactly, is DataOps?

    Primarily, many companies are adopting these principles because they are either trying to avoid or rectify data debt, which is the amount of money required to fix data problems due to the mismanagement of data processes. Data debt is a strong motivator for revamping outdated processes and policies, particularly when decision-makers and stakeholders require metrics before implementing change. Unpaid data debt can be detrimental to a business; the longer it remains unpaid, the more it costs to maintain a data landscape.

    By implementing DataOps principles and data governance, an organization can effectively reduce its data debt and prevent it from growing any larger. Moreover, DataOps practices and software engineering can be used to detect inefficiencies, minimize knowledge loss and capitalize on missed opportunities related to data usage.

    Similarities Between DataOps and DevOps

    Much of the processes that enable DataOps were borrowed initially from the same foundations that built DevOps. Likewise, just as companies need DevOps to provide a high-quality, consistent framework for software and feature development, data enterprises also rely on these same features to realize rapid data engineering and analytics development. For organizations that already have a DevOps framework in place, leveraging DataOps is relativity straightforward. Several important DevOps concepts adopted by DataOps include:

    • Agile development
    • Focus on delivering business value
    • Continuous integration and continuous delivery (CI/CD)
    • Automated testing and code promotion
    • Reuse and automation

    The Differences

    Despite the similarities between the underpinnings of DevOps and DataOps, there are several major differences.

    The human factor: The people using DataOps and DevOps have divergent personalities and skillsets. DataOps participants may be tech-savvy, but often their knowledge is theoretical. DataOps professionals can include data engineers, data scientists and analysts who focus on creating models and visual aids. DevOps, however, was made for software developers and engineers–coding is in their DNA.

    The process: The life cycles of DataOps and DevOps do share similar interactive properties. But, the former deviates in that it consists of a data pipeline and an analytics development process, both active and intersecting. While conceptually, the pipelines of DataOps resemble the development processes of DevOps, typically, experts note that the DataOps process is more challenging.

    Orchestration: In the DevOps process, application code does not require complex orchestration. However, for DataOps, both the data pipeline and analytics development orchestration is an essential component. Although orchestration in the DataOps pipelines occurs frequently and drives data flows, there is usually no such coordination of pipelines in application development and DevOps processes.

    Testing: Again, the two pipelines of DataOps create a significant difference from DevOps; testing in DataOps occurs during both the data pipeline and the analytic development process. These tests attempt to catch anomalies, flag abnormal data values and–unlike DevOps–validate new analytics before deployment. Likewise, these tests get embedded into a data quality framework for continual monitoring.

    Test data management: In most DevOps environments, test data management hardly takes priority; with DataOps, it’s vital to accelerate analytics development so that innovation keeps pace with agile iterations.

    Tools: DevOps is the ‘father’ of DataOps, and as such, the tools needed to support the latter are still in their infancy. While testing in DevOps is primarily automated, DataOps doesn’t have the same luxury – most users modify testing automation tools or build their own from scratch.

    Exploratory environment management: Generally, data teams use more tools than software development teams. Moreover, exploratory environments in data analytics are more challenging from a tools and data perspective; data teams also naturally depart from data islands across the enterprise.

    While foundationally, the concepts of DevOps serve as a starting point for DataOps, the latter involves additional considerations to maximize efficiency when operating data and analytical products. Nevertheless, both serve their intended audiences, reducing data debt and evolving data products or shortening systems development life cycles or providing continuous delivery. For businesses looking to make internal data-related processes more efficient, they should start by examining best practices associated with DevOps.

  • Open Source Vs. Proprietary DataOps

    Open Source Vs. Proprietary DataOps

    Core DataOps concepts are making their way into data engineering teams and, from there, into the broader enterprise. Data engineers are retooling how they create data products, and much of this work revolves around creating data pipelines.

    DataOps pipelines offer the kind of observability that traditional data integration and ETL processes don’t or can’t. They allow you to continuously integrate and test new data sources and deliver data in streaming or batch contexts with higher levels of quality and reliability than traditional, siloed approaches. They can also support machine learning efforts by preparing data for training, testing and deployment.

    DataOps pipelines shorten the cycle time between a data consumer’s request and how quickly and reliably the data producer can fulfill it, all while observing service quality throughout its life cycle. The challenge teams face when building pipelines is how to construct them. The two approaches, as I see it, are do-it-yourself using open source, or through commercial product acquisition.

    Open source certainly has its vocal adherents and some visible successes, but there are trade-offs to taking the open source route.

    Building Or Buying Your Pipeline

    One advantage of open source is in its flexibility and availability. Open source licenses, excluding the SSPL, gives users incredible freedom over what they can do with the software. If you have the skill, you can compose a DataOps pipeline that can take any data, enrich it and route it to the right place. That flexibility, though, is also a downside. While you can do anything you want, you also have to do it. Open source projects like Kafka, Pulsar, Spark, Airflow and Flink don’t know anything about the data they’re handling. That’s up to the developer.

    This may not sound like a problem, but today’s data engineers are handling dozens of data types in hundreds – or even thousands – of different formats. If you add in operational data, you’re also looking at data flooding in from firewalls, containers, SNMP traps and HTTP sources. And that’s just what’s coming at you. You also need to fetch data from object stores, multiple activity hubs and other messaging sources. No open source project natively supports the variety and volumes of data required in a modern DataOps pipeline. Every new data source you add means starting from primordial components rather than higher-level abstractions, adding time and undifferentiated effort to your work.

    You’ll also need essential features like per-source backpressure, support for a range of protocols and data stores, role and permission management and so on. That’s a lot for overworked data engineering teams to build and maintain over the life of a pipeline.

    There’s no ‘easy button’ for proprietary products, either. Many commercial products only support a handful of sources and destinations and may have excessive licensing costs related to data ingest rates or per-connector licensing fees. Also, they can be challenging to scale, from an operations perspective. Seeing a cool drag-and-drop interface may look great during the demo, but it quickly becomes a cluttered mess when you’re looking at thousands of similar data processing pipelines.

    The advantages to commercial products center around the features enterprises need, like governance, management and security. They may also integrate with other components more readily than open source. Finally, having someone to call when you need support is also essential when running in production, although open source core companies also offer support.

    Take a Higher-Level View

    As you’re evaluating DataOps pipeline options, keep two things in mind. First, regardless of which path you choose, ensure your chosen technology offers observability and monitoring features. DataOps is about reacting to change faster, and you can’t react if you don’t know what’s going on. Building observable DataOps pipelines ensures you are delivering a valuable data product to the range of data consumers in your enterprise.

    Next, get past the technology. DataOps is more a people discipline than a technology practice. A key component of a successful DataOps implementation is aligning metrics and incentives across teams. If the staff running the DataOps pipeline are disconnected from the business outcomes their data products inform and influence, they’re not incentivized to run the pipeline as a mission-critical infrastructure component. It’s key to align all of your teams with shared metrics and incentives.

    DataOps Determination

    Before you decide on your DataOps pipeline implementation, there are some things you should do.

    • Determine who your customers are and what their expectations are around data delivery, enrichment, user experience and observability requirements. Are you only serving data scientists, or are you also serving infrastructure and operations, business intelligence and marketing? Knowing your customer allows you to make better decisions.
    • Test the software in a real-world environment. Everyone dislikes being told to perform a proof of concept before you buy/build, because POCs are time consuming. Remember – they’re less time-consuming than picking the wrong tool and looking foolish later.

    Engineering teams sometimes choose technologies that are in fashion rather than selecting the right tool for the job. Taking a pragmatic view of performance, manageability and engineering effort, as well as involving a wide array of stakeholders, will give you a higher chance of success with DataOps.

  • DevOps’ Data Storage Problem

    DevOps’ Data Storage Problem

    The technology sector has always been about problem-solving. When the value of big data was finally embraced, thanks to new analysis capabilities developed in the late nineties and early aughts, the industry adapted its mindset toward storage by investing in on-premises data centers to help store the data that would drive better business decisions. When this rise of data accelerated software development and deployment, creating organizational silos between developers and operations, the industry responded by creating DevOps to improve collaboration.

    Now DevOps has revolutionized the way companies do work by eliminating silos, increasing agility and creating greater visibility, resulting in faster deployments and better service overall. These new work styles have only increased the amount of data produced and the necessity to access it anytime, anywhere has made migrating to the cloud the only real viable solution to any successful DevOps team. Unfortunately, this has spawned a new challenge for the technology industry to solve: the cloud.

    Data and the Cloud Predicament

    Early DevOps adopters quickly leveraged the benefits of the cloud in increasing collaboration brought on by improved data accessibility. However, these early adopters typically entered agreements with the big three cloud vendors that offered increased storage capacity and a seemingly more flexible and accessible format. In most cases, these agreements provide instant gratification, but as DevOps expands over time new pain points are discovered as service and egress fees pile up and teams are forced to either increase budgets to accommodate storage needs or risk meeting limits and data loss.

    This situation created a cloud storage predicament: Companies either pay for more cloud storage than they’ll ever need or have to choose what data was kept and what was deleted. These unsustainable vendor agreements introduce financial pain points for scaling up or scaling down storage space, outweighing the benefits of keeping all of the valuable data being produced and collected day-to-day. As a result, most are stuck thinking about how much and what to store instead of using all of its data to drive the business forward.

    This form of vendor lock-in is a detriment to DevOps teams that rely on seemingly endless amounts of data to experiment, conduct maintenance and develop and deploy new applications.

    Accelerating DevOps With a Bottomless Mindset

    An IDC report recently indicated that the amount of data stored is now expected to hit 59 zettabytes this year alone, with the next three years of creation and consumption almost eclipsing that of the previous 30 years combined. And while DevOps has certainly contributed to this spike, the reality of our digital worlds is that data is king, and a company’s ability to seamlessly store, access and leverage it is what will set it apart from the competition.

    The problem with this growth is finding ways to sustainably store this data in the cloud, particularly when teams are handcuffed to existing storage agreements that are draining budgets. This starts with DevOps teams rethinking their approach. Instead of thinking of cloud storage and accessibility as a recurring cycle of bills and service limit notifications, cloud strategies need to focus on the ability to only pay for the space needed at any given time without the worry of additional fees for accessing archived data. Need to scale up storage capacity? Great! Add as much as you’d like. Need to scale down for a particular reason? Don’t stress over unused real estate just sitting in your vendor’s data center. Need to access archived data at a moment’s notice? Don’t hesitate out of fear of a knee-buckling egress fee.

    With this shift, DevOps teams can move away from conserving, archiving and destroying data toward gathering and utilizing all of it to drive the data-driven insights required in today’s digital economy. This change in mindset toward a “bottomless” cloud removes constraints and enables the ability to innovate at a faster and more sustainable rate.

    Unlocking DevOps Innovation

    Too often people think of DevOps and data management as being separate entities, but as software and application development evolves to include capabilities such as predictive analytics, it’s clear that a marriage between the two entities is needed. By rethinking its data strategy and embracing a bottomless approach, DevOps teams can unlock even greater efficiencies, leading to the next revolution in technology innovation.

  • Consider DataOps for a Competitive Edge

    Consider DataOps for a Competitive Edge

    DataOps seeks to eliminate existing barriers between people, technology, tools and data

    It’s no secret that COVID-19 has put the economy under enormous strain and future economic prospects are uncertain. A smart way to give your organization a leg up against competitors is DataOps. Adopting a DataOps culture makes the best use of your data—it helps to optimize the data powering your business because it successfully monetizes and systematizes your approach to working with data. It will help to get your new products to market faster and also accelerates business agility.

    For example, a DataOps approach can help a data scientist who has spent weeks developing a new machine learning model that is complete but not ready to go into production. She will need to set up data stream monitoring and other basic requirements before she can put it in place.

    This type of challenge is routinely faced by DevOps teams and the best practices they implement help avoid potential setbacks to quickly put their hard work and innovation to use. Applying the same development practices to the deployment of data-intensive services and applications significantly speeds up project completion times.

    Why DataOps?

    DataOps puts the principles and powerful DevOps practices such as automation, continuous delivery and quick feedback cycles to work in a data environment. Its strategic focus is to bring people closer to data, by delivering proven tools that deliver data to people with big-picture business knowledge. DataOps also removes complicated engineering and IT processes that stand between data and those with business knowledge to give projects an important and meaningful business context that greatly improves outcomes.

    The goals of a DataOps approach are to eliminate the friction that’s holding business stakeholders, developers and data scientists back and to increase project productivity and business outcome success. Automating and streamlining processes help people to quickly and easily move from ideas to development to production without lag time.

    DataOps also helps take concepts into production by fitting data projects into familiar, standard CI/CD development workflows and plugging them into any existing framework to deploy applications.

    How to Put DataOps to Work With GitOps

    GitOps is becoming a core component of DataOps practices and puts them into action. It allows data engineers to define data processing applications and pipelines as configuration and manage them through Git and mature CI/CD pipelines. This brings with it higher levels of standardization, security, governance and, as a result, an accelerated rate of delivery. Adopting GitOps in the data management world can pose problems for some organizations working with powerful and fast-changing technologies such as Apache Kafka and Kubernetes, but the automated processes it brings turbo-charge delivery time.

    Cloud migration is another way in which a GitOps approach makes a difference for organizations that want to take advantage of managed services in Microsoft Azure, AWS or Managed Streaming for Apache Kafka (MSK). GitOps helps simplify migration of Kafka clusters used for streaming to the cloud service of an organization’s choice—simply define the desired state, step back and GitOps takes over.

    GitOps’ pull-based approach to deployment, in which an operator watches Git and applies the desired state, creates several benefits. One benefit is not needing to poke holes in production environments to provide push-based actions from software such as Jenkins, which has numerous common vulnerabilities and exposures (CVEs). With a GitOps pull-based approach, attack surfaces are reduced, enabling solution migration across technology and clouds.

    A Better Process Results in Better and Faster Outcome

    Real-time business is here to stay and there is huge pressure on businesses to change fast. As the saying goes, “You snooze, you lose.” With the right tools and strategy backed by a good communication plan, DataOps will work to your advantage, bringing the benefits of a DevOps culture.

    Faster time to market via increased productivity is one advantage of a DataOps approach. Putting people closer to data, together with the right tools, helps you empower business users and developers so they can add new features and capabilities to any deliverable, and deploy and monitor it quickly.

    When working at a faster clip, don’t overlook the importance of governance and security. Build governance into the DataOps process to ensure deployment of the correct applications in the right way. And, maintain business continuity without disruption to existing infrastructure and processes.

    DataOps seeks to eliminate existing barriers between people, technology, tools and data. By bringing business and technology imperatives closer together, you gain invaluable context and a better ability to focus on driving strategic business outcomes that matter.

  • The Taxonomy of DataOps

    The Taxonomy of DataOps

    Data Operations, or DataOps for short, is one of those IT buzzwords that lots of people use, yet few can define precisely. Like the cloud or DevOps, DataOps doesn’t make sense until you sit down and really think through what it entails.

    To that end, let’s take a look in this article at the taxonomy of DataOps—or, in other words, which IT disciplines, areas of expertise and types of work make DataOps happen. I won’t explain everything about how DataOps works (that would require more than one blog post), but I will offer a primer on how DataOps workflows break down.

    What Is DataOps?

    Before delving into the taxonomy of DataOps, let me first define at a high level what I’m referring to when I use the term “DataOps.”

    I use DataOps as a shorthand for data operations of all types—from data collection and storage to data management, monitoring, analytics and performance optimization.

    I mention this because when some people hear the term DataOps, they think only of the practices and ideas described in the “DataOps Manifesto” and similar writings, which focus on extending the core principles of the DevOps software delivery philosophy to data operations. I’m a big believer in the “DataOps Manifesto,” but because it is targeted at data scientists rather than the IT community at large, I think it presents a narrow image of what data operations entails.

    To me, data operations encompasses the work performed by anyone who works with data in any way. While data scientists and data engineers may perform the bulk of the heavy lifting related to data transformation and data analytics, developers and IT engineers also have important roles to play in data operations. When I use the term DataOps, I’m thinking of them, too.

    The Taxonomy of DataOps

    With that clarification out of the way, let’s explore what goes into DataOps.

    Data Collection

    You can’t have data operations without data—and so, data collection is a core part of DataOps.

    Data collection refers, of course, to all of the processes by which you accumulate data and store it somewhere, for some length of time.

    It’s important to note that data collection can be both passive and active. Much of the data that organizations store and analyze these days is not data that someone goes out and collects by hand. It’s the information produced through the everyday operations of these organizations—their data exhaust, so to speak.

    Data Storage

    Data storage may seem simple and straightforward. After all, how much do you really need to think about the storage of data? Don’t you just pick a storage location and put your data there?

    Well, no. Data storage is much more complex than that, which is why data storage is a distinct discipline within DataOps.

    Effective data storage requires finding the right type of storage infrastructure—the one that will deliver the best balance between data performance, availability and cost. It also involves defining data retention policies to determine how long data will be stored. It encompasses data backups, too, which are important for ensuring that the data you intend to keep in storage actually will be available to you if the unexpected happens.

    Data Management

    Data management is a broad discipline that involves all of the tasks required to keep data available and accessible once it has been collected and stored. It entails work such as transforming data from one format to another to make it compatible with a given set of tools. It also involves tasks such as moving data between storage locations and identifying and resolving bottlenecks within the overall DataOps pipeline that are slowing down performance.

    Data Monitoring

    In the IT world, the word “monitoring” usually refers to the monitoring of applications or infrastructure for problems. But you have to monitor your data also if you want to do DataOps effectively.

    Data monitoring includes tasks such as identifying data quality problems (such as missing data or data formatting issues) that will make your data less reliable or more difficult to work with. It also means checking for signs of failures, such as a failed database or infrastructure crash, that could cause a rupture in your overall DataOps pipeline.

    Data Analysis

    Everyone (or everyone’s boss, at least) wants to “turn data into dollars,” as Gartner put it. Data analysis, or the process of seeking insights within sets of data, is key to making that happen.

    Sophisticated data analytics is the only discipline on this list that is usually performed by data scientists alone, not the rest of the IT team—although that is changing somewhat as more and more simplified data analysis and visualization tools enter the market that makes it easy to make sense of large data sets without having a Ph.D. in statistics or knowing how to program in R.

    Data Security

    Data security is a big topic, especially in the present age of high-profile data security breaches. Keeping data secure requires a range of processes and tools, from setting the right access control policies for data to monitoring for signs of unauthorized access and scanning data to root out malware or other threats that could exist within data sets.

    DataOps is Everyone’s Ops

    You might have noticed that many of the tasks described in the DataOps taxonomy above typically are performed by employees of the IT department (or, in some cases, programmers), not data scientists or engineers who specialize in data operations.

    This is an important point to drive home because when many people talk about DataOps, they assume that every organization has a team of data experts on staff who handle all work related to data, from collecting and storing it to analyzing and securing it.

    In reality, many companies don’t have dedicated data operations teams. And even if they do, the general IT department still plays a huge role in tasks such as configuring tools and platforms for data collection, provisioning data infrastructure and keeping databases running healthfully. Security engineers, too, are key for things such as enforcing data access control and monitoring for signs of data security problems.

    Thus, whether your company has a dedicated data operations team or not, DataOps is the responsibility (at least in part) of everyone who does anything related to IT. It’s everyone’s Ops.

    This sponsored article was written on behalf of Unravel.

    — Chris Tozzi

  • DataOps: DevOps Plus Big Data

    DataOps: DevOps Plus Big Data

    In traditional DevOps, there are the complimentary forms of development operations (that I call DEVops), and development operations (that I like to call devOPS). Between them they automate the toolchain and bring the people working on getting an application out to users onto the same team—not necessarily the same organizational team, but using their strengths in tandem to meet the needs of the business.

    DataOps is an interesting extension of DevOps. You still need all of the coding side to get data into the system and queries consistently maintained. You still need all of the operations side to get your database/noSQL/whatever up and running. In fact, while the development side in a DataOps environment might be (but isn’t necessarily) lighter, the operations side is almost always more complex. No matter the big data engine in use, it is a complex system in addition to other supported systems in a normal environment. My first installation of a big data environment (Cloudera, as it happens) was a weeks-long learning voyage. Only after I’d completed it did I use an automation tool (which is no longer available) to make it easy. My second round was hours … But it assumed the knowledge I had gained in the first attempt.

    The other bit of DevOps—ongoing monitoring and management—is also more complex in a big data environment, but we’ll come back to that in a moment.

    Once the normal DevOps systems are in place, ETL/data import tools will need to be supported also. The volume that these tools crunch through in a day makes their inclusion in DevOps in a data-heavy environment critical. If the data uptake is slow or the data itself inaccurate, there is an impact on the organization. This step also requires inclusion of data scientists, people who traditionally have not been pulled into the DevOps model. But they are the ones that can gauge the accuracy of data, and normally are responsible for data acquisition anyway.

    Which brings us to the monitoring and management stage of DataOps. Normally when we talk monitoring and management in DevOps, we are talking about tools that ultimately help with things like availability, responsiveness, auto-scaling and recovery. DataOps needs all of these things, like any other application does. And we’ve already mentioned that it has a complex environment embedded in your complex environment. But it also (some would argue more importantly) needs data monitoring. A knowledge of the state of the data is imperative. Whether from a piece of code gone wrong or data poisoning, monitoring the health of big data archives is essential to keep results from getting out of whack. This monitoring will again involve the people who know the data—data scientists by whatever title. This iterative process will need to run relatively constantly, checking on whatever makes the data valuable, normally using cross-field checks to verify that data hasn’t somehow been corrupted, or comparing trending averages to current average to monitor for sudden swings in results.

    Of course, for a system (and in research there are many) that spins up, finds results and spins down, the monitoring part is less essential, but for anything that keeps accessing and adding to the data pool, it is the key to a healthy application or application portfolio based upon the dataset.

    If all that sounds a bit intimidating, you’re not alone. The phrase “DataOps” has been around for a few years, and while data scientists use it, it hasn’t yet gained serious traction in the enterprise. Here’s how I would approach it as CIO: Given that DevOps is running on other projects, that is the low-hanging fruit. Given the rate of change for servers (as queries—particularly long-running ones—increase and decrease, systems needs change is one example), traditional DevOps is a greater benefit to big data than many other corporate systems. So roll out DevOps first without serious concern for DataOps beyond how DevOps impacts ETL—not ETL data quality, but ETL data availability/performance. Only when the system and the team are in place and the software/hardware/code part of the overall system is fully in a DevOps environment, start looking at ways to move data into it. Data monitoring will be the big one, but ETL under the initial round of DevOps implementation will have shown places it could be improved and ETL data integrity can be added, so improving those areas also. A measured, step-by-step approach that builds off of organizational experience is key here. Data monitoring and management will require different tools than normal DevOps monitoring and management, so having the data scientists look into available toolsets and how they mesh with existing tools early on will help with eventual implementation.

    Make no mistake, it is a big job, moving large data driven environments to DevOps. But the benefits are increased awareness of data quality and data quality issues, along with more responsive systems and teams that DevOps offers every business application. The amount of data available to any give organization is currently on a massive increase as IoT, open datasets, partnerships and ML drive greater pools. It is worth getting the automation and cooperation of DataOps into the mix as early as possible.

    — Don Macvittie