devopsy.kinsta.cloud

Tag: data management

  • Dell Brings DevOps to Edge Computing Environments

    Dell Brings DevOps to Edge Computing Environments

    Dell Technologies today updated its edge computing platform this week to make it simpler to programmatically provision infrastructure using DevOps best practices.

    Phil Burt, senior director of project management for edge computing at Dell Technologies, said version 2.0 of Dell NativeEdge adds additional declarative blueprints based on YAML files and the Topology and Orchestration Specification for Cloud Applications (TOSCA) framework.

    Blueprints for Telit, Litmus and PTC for manufacturing use cases, Deep North for retail, Dell Streaming Data Platform for analytics and Rancher Labs K3s for Kubernetes environments are provided.

    Those blueprints make it simpler to either deploy applications on a lightweight operating system that Dell developed optimized for edge computing platforms or any other software stack an IT organization may prefer, he added.

    Dell is also adding support for the virtual Trusted Platform Module (vTPM) and UEFI Secure Boot capabilities to apply zero-trust IT principles to edge computing platforms using hardware-based cryptography and secure storage for encryption keys, certificates and passwords.

    Finally, Dell is also making available a three-year subscription plan for NativeEdge to lower the total cost of deploying edge computing platforms.

    Dell relies on virtual machines it developed for edge computing platforms to deliver software. IT teams can then opt to deploy monolithic applications or container-based applications running on an instance of Kubernetes deployed as an extension of the virtual machine provided by Dell. Those platforms can then also automatically connect to a network to provide the connectivity required.

    Regardless of approach, Dell is providing a method to programmatically orchestrate the management of highly distributed edge computing environments, noted Burt.

    In the longer term, Dell is also researching how to apply generative artificial intelligence (AI) to the management of edge computing environments using the data its monitoring tools provide, he added.

    While edge computing has been around in one form or another for decades, as more organizations start to process and analyze data at the point where it is created and consumed, the number of applications deployed at the network edge has increased substantially. As such, the number of applications that need to be built and deployed at the edge using DevOps best practices is growing exponentially. In fact, managing those workloads is only going to become that much more challenging as more AI models are deployed at the network edge.

    Of course, not every workload at the edge is managed by an IT team. For years, many organizations have relied on operational technology (OT) teams that have historically deployed applications at the network using manual processes. Organizations that embrace DevOps practices to automate the deployment of workloads at the edge will naturally have to navigate some cultural issues alongside the technical challenges encountered when programmatically deploying software.

    It’s not clear how much software will be running at the network edge in the years ahead, but as the capabilities of the IT infrastructure made available continue to increase, so will the complexity of the applications that will be deployed. The challenge, as always, is going to be finding the best way to manage all that distributed software at what will soon be an unprecedented level of scale.

  • 6 Transformative Mainframe Predictions for 2024

    6 Transformative Mainframe Predictions for 2024

    The success of organizations in today’s digital economy demands speed — the ability to quickly respond to market trends with new applications and services, instant access to critical data, and resolution of issues even before they affect service. As we move into the new year, the adoption and adaptation of emerging technologies on the mainframe will enable enterprises to embrace changing market forces, turning them into a competitive advantage.

    Below are six predictions of how mainframe organizations will leverage new technologies to advance the platform in 2024.

    Prediction: AI/ML Adoption Grows

    The roles of artificial intelligence (AI) and machine learning (ML) and their adoption across the mainframe, from application development to monitoring to data management, will continue to grow.

    The generative AI revolution is real and here to stay. In the coming year, organizations will continue to explore best-use cases for its integration into mainframe management. Development teams will leverage key AI-recommended insights into automated workflows and code generation, improving developer productivity. Database administrators (DBAs) will look to automate data management processes to efficiently handle the demand for the diverse datasets required by generative AI tools, and mainframe operations teams will increase their reliance on AI-driven predictive analytics and root cause analysis.

    As adoption increases and experience is gained, organizations will work past their AI trust issues and begin to harness the power of automation to become a fully autonomous digital enterprise. At the same time, there will be a growing need to govern AIOps and MLOps to ensure the validity and accuracy of ongoing model development.

    Prediction: DevEx Comes to the Forefront of Companies’ Innovation Strategies

    With the long-awaited shift of mainframe workforces from those with decades of experience to a new generation of stewards now in full effect, organizations will find a new imperative to attract and retain top talent by providing developers with familiar tools and development environments.

    Professionals who have recently graduated from college or are moving from other platforms bring with them an expectation of integrated and automated DevOps practices. The move towards enterprise-wide DevOps and continuous integration and continuous deployment (CI/CD) pipelines will continue, with many companies also looking to employ cross-platform tooling. By giving developers the freedom to use their integrated development environment (IDE) of choice, organizations can decrease onboarding time and get new mainframe developers up to speed faster, enabling them to understand code and its function quickly and easily, and creating a seamless, frictionless way to make changes, debug, and deploy.

    Prediction: Database DevOps Bridges the Gap Between DBAs and Developers

    Changing customer demand and the need for agility will accelerate the drive for developers to seamlessly access and manage data sources. The demand for an agile development environment that extends from applications to databases will increase the inclusion of database DevOps in CI/CD pipelines, including version-controlled database schemas and data, automated database provisioning, and database testing as part of the pipeline.

    By embracing agile practices, such as incremental database changes and rolling deployments, organizations will bridge the gap between DBAs and developers and ensure that mainframe databases evolve alongside application code. This cohesive approach will enhance collaboration, reduce conflicts, and expedite application development, positioning mainframes as agile and adaptable components within the rapidly changing IT landscape.

    Prediction: Democratizing Mainframe Data

    The move ― and the need ― to democratize mainframe data, making it available to AI/ML analytics engines, will gain momentum.

    Organizations have matured their data lakes and gained valuable insights from data housed on distributed and cloud platforms, but the efforts needed to extract and transform mainframe data stored on tape and virtual tape libraries (VTL) has left volumes of business-critical data behind. By operationalizing mainframe integration with the cloud, organizations can make it easier and faster to access mainframe data and transform it into open formats, making it accessible to analytics engines and AI/ML applications.

    The ability to realize the hidden potential of data they already store will make cloud-based storage of mainframe data a prerequisite for enterprises looking to digitally transform their businesses and gain competitive advantage.

    Prediction: Object Storage Becomes Critical for Security

    Mainframe-dependent organizations will seek to integrate hybrid cloud object storage to fortify mainframe data protection against escalating cyberthreats and ransomware attacks.

    In 2024, cyberthreats will almost certainly increase in number and sophistication. The European Union’s Digital Operational Resilience Act (DORA) seeks to ensure that organizations remain vigilant and protect their systems and their customers’ data against increasing threats.

    With DORA set to come into effect in January 2025, and further regulations and guidelines on the horizon, managing mainframe data across diverse locations while prioritizing immutable data copies will be imperative. Organizations will pivot towards proactive strategies, aligning their mainframe infrastructure with hybrid cloud solutions to meet evolving compliance standards and shield against the growing sophistication of cyberthreats.

    Prediction: Innovation Takes the Lead

    Organizations will adopt ruthless prioritization, shifting focus from maintaining services and applications to building new ones for the future.

    Having seen firsthand the success of ongoing mainframe modernization efforts, organizations will place a new focus on building skills and specialties that align with future goals and product roadmaps, putting them in a position to capitalize on coming trends and giving them the ability to quickly respond to changing market forces with innovative new products and services.

    As companies mature their approaches to AI on the mainframe, many will turn to building “applied AI” teams tasked with developing proactive solutions to better analyze data and forecast trends. In short, organizations will think logically about how to embrace AI rather than being embraced by it.

    In 2024, mainframe organizations will accelerate the adoption of emerging technologies like AI and the cloud to increase the adaptability, resilience, and efficiency of the platform while situating it for even greater success in the future. The mainframe will continue to be a platform of innovation, integrating with new technologies and practices borrowed from other platforms to further the evolution of a hybrid IT environment that meets and exceeds the demands of the modern digital economy.

    The role of AI, the impact of data governance regulations, the mainframe skills gap, affordability, and interoperability requirements are all brewing a perfect storm around the mainframe that will mark 2024 as a year of significant transformation that will define the leaders from the laggards.

  • KubeCon 2023: Telemetry and Data Management

    KubeCon 2023: Telemetry and Data Management

    Speaker 1: This is Techstrong TV.

    Alan Shimel: Hey everyone. We’re back here at KubeCon and it’s bustling, man. It’s a crazy big floor with a lot of people walking around, so we hope you like this. This is a little different than we usually do. Usually people can’t see the background because the cameras face this way, but we usually have sort of the typical background that you see at some of these other booths. But we wanted to give people kind of a flavor that we’re alive here.

    Tucker Callaway: We’re in the middle of it.

    Alan Shimel: We’re in the middle of it. So kind of like if you ever watch Sunday night football, they always have that. So that’s what we were shooting for. But anyway, let me introduce you to Tucker Callaway. Tucker is the CEO of a company called Mezmo. Now, some of you may have heard of, I think it was Log DNA.

    Tucker Callaway: Before. Yep.

    Alan Shimel: And they pivoted. Was it about four years ago?

    Tucker Callaway: About two years ago now.

    Alan Shimel: Two years ago.

    Tucker Callaway: I’ve been here for four, but big change was two years ago, renamed.

    Alan Shimel: Renamed to be Mezmo and a slightly different mission, let’s call it. And then I had a chance to catch up with Tucker at RSA last, I guess it was last April.

    Tucker Callaway: April. Yeah.

    Alan Shimel: This year it’s May. So I think it was around then last year. And for those of you who want to catch that interview, it’s actually available on Techstrong TV. If you look up RSA 2023, it’s there. But so Tucker, before we get into that, let’s back up. Mezmo. Not everyone out here. Look, not everyone out here knew Log DNA either, so don’t worry about it.

    Tucker Callaway: Fair enough. Yeah.

    Alan Shimel: Let’s start there. Let’s talk about the mission.

    Tucker Callaway: Yeah, so the mission of Mezmo is essentially to help people with their telemetry data. So we’ve found that there’s really kind of three main problems in the telemetry data space. There’s too much of telemetry data. It’s not in the right format. It’s not in the right location. So that was kind of one of our founding kind of principles of solving.

    Alan Shimel: Yeah, I think that.

    Tucker Callaway: Solving the problem.

    Alan Shimel: Those were three rock solid.

    Tucker Callaway: Hard to debate.

    Alan Shimel: Hard to debate. That’s the good way of putting that. And also, quite frankly, all three problems are like they compound each other.

    Tucker Callaway: They do.

    Alan Shimel: So it’s like you have too much data. That’s a big problem, but we have too much data that we don’t know how to categorize, so that exponentially makes that problem worse. So these are-

    Tucker Callaway: Compounding problems. No question about it.

    Alan Shimel: No doubt about it. So when we were at RSA, you felt like you were kind of on the cusp of a realization, right? You were going to help customers. Why don’t you? Well, you tell the story better than me.

    Tucker Callaway: So as we were talking about earlier, we had just launched this new product capability, the telemetry pipeline at RSA. We’ve got a lot of great customers and great experience since then. When we think about the core value of what it provides, it helps people with cost, it helps people get better insights and helps people enforce compliance. What we were surprised by though in the last couple months, is how little people actually understand of their data.

    In fact, we’ve put a lot of time and effort as an industry into getting control of our applications and getting control of our infrastructure, but we haven’t really gotten control of our telemetry data yet, and that’s like the big opportunity that we see. But with that, in order to get that value I described, people didn’t need to understand their data better. And so that was a big revelation for us.

    And we just launched our data profiling capability that helps people categorize and understand baseline, to do the schema management for their data, which is a really strong foundation for data ops principles applied to telemetry data. Then the next thing we found after we solved that problem was now that people understood their data and they were able to optimize it because they understood it. They also realized that the context and telemetry data changes rapidly. So for example, you don’t want to store all this data because you don’t need it until you need it, and then you really need it.

    Alan Shimel: I’ll take a little, they do want to store the data because people are data hoarders.

    Tucker Callaway: The data hoarder. Yeah.

    Alan Shimel: It’s when they get hit with the bill for storing all that data, they say, “Wait a second, I don’t need all this data. Not at that price.”

    Tucker Callaway: Something has to give somewhere.

    Alan Shimel: Exactly.

    Tucker Callaway: And so what we realized though was if you take the bookend of this value, we think of as understand, optimize and respond, we talked about understand and optimize the response stage was a really important one for us. So when new context comes in, the pipeline can change its behavior and change its optimizations to account for the current flow of data and the current needs. So if you’re sampling down to 20% and you see an event, you see performance, you can rehydrate, reprocess, dynamically change the responsiveness of the pipeline and get all the data to the right location for the right people, which helps address the cost problem, but also gives you the full visibility that you need.

    Alan Shimel: If one was going to do a business school case study on this, someone who’s not from this industry would say, “Hey, just the fact that you discovered that most of these organizations don’t even know how much data or what data they have would be enough to build a company around.”

    Tucker Callaway: Yeah, I’d like to think so.

    Alan Shimel: No, it’s logical. It makes sense, but it’s not, it’s the threshold. Because I think you’d hit the nail on the head here, Tucker, which is once they recognize that, now there’s a whole new vista of realization for them of, “Holy mackerel, I got all this data, now I want to start understanding it. Wow. I’m seeing stuff I never thought I’d see. Maybe I intuitively thought I had, but I got the goods now.”

    Tucker Callaway: We like to think of that as being mesmerizing, right?

    Alan Shimel: Okay, I love it. Mesmerized by Mezmo.

    Tucker Callaway: I couldn’t not take the shot, but that was just-

    Alan Shimel: Yeah, that was your shot.

    Tucker Callaway: That’s the derivative of the name.

    Alan Shimel: I feel used. I feel used.

    Tucker Callaway: I got back into it a little bit.

    Alan Shimel: So now you’re mesmerized and now like, “Oh my God, what can I do with this?” And that really becomes the business. And I guess that’s now where you are.

    Tucker Callaway: The next step for us is clearly that. Yeah, and I think there’s two plays on that, too. There’s a, “What do I do with it? Help me, tell me what to do. Give me the best practices that from all of your customers, what should I do?” And so we’ll be coming out probably later in the spring with something that actually allows you to simulate how that data might look when optimization’s applied to it so you can start to make those decisions more effectively.

    But then the second phase of that, too, is that also won’t be sufficient for certain organizations that want to look at their data as a strategic advantage. And they’re going to be looking for ways to take actions and drive more insights out of that data. And so the way we see that, we see that through recipes. We’ll give you the standard offerings, but then you have the ability to go in and edit as code and go do the things you need to do to go drive more value out of that data. If you’re ready for advanced yoga moves in the data space.

    Alan Shimel: Got it, got it, got it. My background is security.

    Tucker Callaway: Yep.

    Alan Shimel: Started a security company in 2001, and it was right when we were moving from IDS to IPS. So intrusion detection to intrusion prevention. Same thing with vulnerability management. We were moving from just finding vulnerabilities to patching, remediating, not always patching, remediating. Seemed like who wouldn’t want that? No brainer, right? Just don’t tell me I’m getting attacked. Block the thing. By the time you tell me, the attack went through. Don’t just tell me you found a vulnerability, make sure I’m not getting exploited. But a funny thing that I learned in that was a lot of people say, “Go slow here. I can’t afford, because sometimes the vulnerability or the attack or the data that you want me to act on can affect something that I consider more valuable.”

    Tucker Callaway: Yeah, fair enough.

    Alan Shimel: And I like to go slow when it comes to letting a program or an application or a product, just do things. Just give me my menu and I’ll decide what I want to do. Now, of course, that was before the world of AI and ML and a lot of the automations. And quite frankly, the speed of business was probably a little slower then. Do you see a way station where people are going to want that? Just give me best practices and I’ll decide when, how, and where I want to implement it. Or do we move right to, “Hey man, make it happen for me.”

    Tucker Callaway: I think there’s a way station, when we think about the data management, telemetry, data management in general, the word that’s the most important to us is trust, right? So it is one thing to optimize the data, but you have to trust that data is constantly being optimized and handled in the right way for all those varying needs you described.

    So I do see that the first step is a suggestion with a human in the loop. You want to hit that button still to say “go”, and then as you build trust in that data and you know that the algorithms are working, then you’ll let that go. Probably with some backstops on the side, I’m going to let it go, but I’m [inaudible 00:09:52] just in case. And then you’ll start to optimize it more and more. And I think naturally, almost like a realtime data platform, operating system will start to evolve.

    Alan Shimel: So really the key there is trust.

    Tucker Callaway: Key is trust.

    Alan Shimel: To build trust in the next. So as the CEO, how do you build that trust?

    Tucker Callaway: Well, trust is always a tough.

    Alan Shimel: I’m interested, you tell me.

    Tucker Callaway: Yeah. I think like anything else in life, you deliver trust through consistent delivery. And so that’s where I think that human in the loop step is important. It has to be sustaining. It can’t be a moment in time, it can’t be the very first implementation. It’s something that you earn over time where people can count on you or could count on the data or count on the ways that you treat data to get you to the outcome that you want to go through. So it’s really only through the delivery of outcomes to the customer that they will trust the systems and all those things.

    Alan Shimel: Makes sense?

    Tucker Callaway: Yeah.

    Alan Shimel: All right. Got to do a little housekeeping.

    Tucker Callaway: Okay, let’s do it.

    Alan Shimel: People who want to know more about Mezmo.

    Tucker Callaway: Come to the website. We’ve got an offer out there to profile anyone’s data for free right now.

    Alan Shimel: Do you really?

    Tucker Callaway: Yeah. Come to mezmo.com or stop by our booth.

    Alan Shimel: M-E-Z-M-O.com.

    Tucker Callaway: M-E-Z-M-O.com

    Alan Shimel: Out there in TV land or if you hear a coupon.

    Tucker Callaway: Come by the booth. I forget the number, but it’s over there.

    Alan Shimel: You know what, if you look on the, I don’t have it. If you look on the back of your ID thing, you could actually look stuff up like that.

    Tucker Callaway: Yeah, it’s back there. If you connected to the wifi here at KubeCon, it’s Mezmo Data.

    Alan Shimel: I’ve noticed that.

    Tucker Callaway: Is the wifi.

    Alan Shimel: I was like, “Wow, what a cool thing.” Yeah.

    Tucker Callaway: You have see us. Come see us. Profile your data will give you insight and understanding into that and talk to you about the next steps.

    Alan Shimel: All right, check it out at Mezmo. Tucker, thanks.

    Tucker Callaway: Thanks for having me.

    Alan Shimel: Actually, we’ll see you. Are you going to be at AWS?

    Tucker Callaway: Yeah, we’ll be there. Yeah, I’ll be there. Yeah.

    Alan Shimel: We’ll be doing videos, not on the show floor. It’ll be a little quieter in a private suite. And then of course you’ll be at RSA.

    Tucker Callaway: We’ll be at RSA.

    Alan Shimel: We’ll be there as well.

    Tucker Callaway: We might not be on the show floor at AWS just because of-

    Alan Shimel: It’s a little crazy there isn’t it? A little pricey.

    Tucker Callaway: Yeah.

    Alan Shimel: Yeah, I hear you. That’s why we’ll be at the wind in the studio.

    Tucker Callaway: I’ll see you there.

    Alan Shimel: Come on up. I’ll have coffee.

    Tucker Callaway: Sounds good.

    Alan Shimel: But RSA will be at broadcast alley and we’ll be doing our DevSecOps thing on Monday. And the usual kind of RSA craziness.

    Tucker Callaway: Looking forward to it.

    Alan Shimel: Absolutely.

    Tucker Callaway: Okay.

    Alan Shimel: Check out Mezmo.com.

    Tucker Callaway: Thanks everybody.

    Alan Shimel: Tucker Callaway here on Tech Drunk TV. We’re going to take a break. I guess it’s almost lunchtime here, but it’s still busy.

    Tucker Callaway: Near enough.

    Alan Shimel: Stay tuned. We’re here all day. We’ll be back. Bye-Bye.

  • Data Observability and its Importance: Everything you Need to Know

    Data Observability and its Importance: Everything you Need to Know

    In today’s tech-driven world, companies rely on almost incomprehensible amounts of data to dictate their business operations and decisions. Enterprises are focused on building multiple data repositories and data pipelines to process, store, manage and utilize data from various sources. Given the ever-growing size and complexity of the enterprise data environment, providing the accuracy and completeness of data is becoming increasingly challenging. So how can you understand how the data is performing across the entire IT infrastructure, all at once? The answer is data observability.

    What is Data Observability?

    Data observability is a process that aims to alert you to your data’s reliability and health while also delivering the insights and analysis required to identify and resolve issues before they affect your entire organization.

    Monitoring operations in the database serves as a proactive defense mechanism against potential security threats.
    This ensures accurate, complete, secure and valuable data and eliminates data downtime. Let’s dive deeper into the observability infrastructure and explore some of the best data observability platforms that can help you optimize and secure your data operations.

    Data Observability Framework

    We know that observability has three pillars: Logs, traces and metrics.

    However, data observability has five pillars, which, together, provide key insights into the reliability and quality of your data.

    Incorporating each pillar can help you craft an effective observability strategy.

    1. Recency: Also referred to as “freshness,” recency involves confirming if the data is up to date. It also analyzes if there are unusual temporal gaps in the data tables. This helps prevent timeliness issues in data pipelines.

    2. Volume: Volume involves checking if the amount of data intake in the database meets the expected thresholds. This ensures that data sets are complete.

    3. Distribution: This measures the quality of data at the field level and confirms if the data values are within expected ranges. Unexpected fluctuations in the distribution patterns are indicative of a data issue.

    4. Schema: This involves monitoring and auditing changes to data tables and data organization to look for signs of broken data. Schemas are extremely important as changes in the source data’s structure are often the cause of data downtime.

    5. Lineage: Lineage collects metadata and provides a complete picture of your organization’s data landscape, including upstream sources, downstream, and which teams can access the data at which stages. This process helps data teams troubleshoot where breaks occur in the data.

    Data Observability Benefits

    Data observability provides better visibility into your IT system’s internal state, including its behavior, performance and interactions with other systems. This can be beneficial in the following ways:
    ● Facilitates Root Cause Analysis: With end-to-end data visibility and monitoring across multi-layered IT infrastructure, data observability enables data teams to quickly spot issues in data sets with less effort. This also increases the chances of identifying new issues regardless of where they originate.

    ● Faster Mean Time to Detection and Resolution: As data observability monitors a wide range of output, data teams can actively triage and debug their systems. By providing valuable insights into how the data interacts and moves within the IT architecture, data observability helps teams spot problems they didn’t know about, resulting in quicker mean time to detection (MTTD) and mean time to resolution (MTTR).

    ● Automates Security Management: Data observability not only provides real-time visibility into the security posture but also makes it easy to automate parts of the triage process. This helps detect data health issues or data downtime instantly.

    Data Observability Challenges

    Depending on their existing IT architecture, organizations may face the following challenges when it comes to data observability:
    ● Data Silos: Due to the presence of several agents, silos monitoring tools and disparate data sources, it becomes difficult to comprehend interdependencies among applications, digital channels and various clouds.

    ● Integration with Entire Data Ecosystem: For data observability to work, the tool requires insights into the complete data pipeline and the servers, databases, software and applications involved. However, some organizations may find it challenging to connect all of the systems to a data observability platform.

    ● Manual Instrumentation and Configuration: Data observability tools aim to standardize telemetry data and logging guidelines to efficiently correlate information. However, as large organizations maintain several data sources (hundreds or even thousands), data from these sources might have varied standards. This requires manual effort to standardize data.

    Data Observability vs. Data Governance

    In today’s digital-first landscape, where data security is a global concern, there’s a growing focus on data governance. Data governance assists in the formation of policies and procedures required to keep a check on how an organization collects, analyzes, stores, shares and uses its data.

    A strong data governance program ensures data availability, integrity, usability and security. It eliminates data integration issues, data silos and poor data quality, thereby resolving the challenges of data observability.
    With data sets now scaling with more tables, more data sources and more complexity, data engineers and developers are under pressure to keep up with the data consistency, availability and security requirements. Any amount of downtime can lead to wasted resources and time while deteriorating confidence in decision-making.

    Data observability, along with data governance, helps organizations manage data security and quality problems in a streamlined manner.

    Data Observability vs. Data Quality

    Data quality measures the completeness and accuracy of data sets to determine if they can be used in analytics and operational applications. Data observability, on the other hand, allows organizations to detect and fix issues in the data pipeline efficiently and quickly.

    For effective data management, an enterprise must take both into account. Earlier manual checking and cleansing of data sets helped determine data quality. But today, many of these tasks have been automated, thanks to modern data stacks.

    As a result, there’s been a shift in focus from data quality to data observability. With data observability, businesses can effectively monitor and troubleshoot their data pipeline. Without it, they are at the risk of relying on incomplete or inaccurate data to make decisions. This can lead to expensive mistakes.

    Finding the Right Data Observability Tools

    Undoubtedly, data observability is a critical feature of any enterprise that uses data. However, not all data observability tools are equally beneficial to your business. Look out for the following traits when choosing a data observability platform:
    ● Compatible: The tool must be compatible with your data lakes, databases and cloud storage solutions.

    ● Autonomous: Autonomous technology responds to stimuli without any human intervention. This is essential in a data observability tool as it helps in the early detection of anomalies and an instant response to alerts.

    ● Timely: A data observability tool must help you identify errors as early as possible. The best platform constantly monitors data health from the moment the data is added to the ecosystem to the end of its lifecycle. Early detection of errors stops issues before they become critical.

    ● Sophisticated: The best platforms leverage machine learning (ML) and artificial intelligence (AI) to identify hard-to-find issues.

    ● Other features: The tool must be able to collect, sample, review and process telemetry data across several data sources. It should serve as a centralized data repository, offer comprehensive data monitoring services and provide data visualization.

    Eventually, the right data observability platform depends on your organization’s observability engineering needs and unique IT architecture.

    Top Data Observability Tools

    Data observability platforms are still a budding product category. The good news is that over a half-dozen data observability companies now offer commercial tools with excellent features. These include the following:

    Observability Vendors, Overview,  Pros and Cons

    Monte Carlo Data is headquartered in San Francisco. Monte Carlo is the creator of the industry’s first end-to-end data observability tool.

    ● Provides comprehensive data observability capabilities.
    ● Offers a high level of features, including automated alerting, data catalogs, etc.
    ● Supports fully automated setup.

    ● High data volumes can lead to user interface issues.
    ● Huge amounts of variables restricted by different constraints can lead to computational inefficiency.

    Bigeye is an industry-leading data observability platform that allows teams to improve, measure, and communicate quality data clearly and quickly at any scale.

    ● An easy-to-use interface that facilitates data configuration while ensuring consistency and accuracy.
    ● Provides powerful API integration capabilities.
    ● Features a versatile dashboard with real-time tracking and monitoring of data quality metrics by multiple people.

    ● It can be expensive for smaller organizations.
    ● The tool might crash at times, requiring performance improvements.

    Acceldata offers tools for Hadoop, cloud services, and companies which include end-to-end data reliability, data pipeline monitoring, and multi-layer data observability.

    ● Provides fully automated data reliability checks.
    ● Offers a drag-and-drop interface to analyze data pipelines across several layers and platforms.

    ● Initial setup can be complex and difficult.
    ● Node addition and node removal require human intervention.

    Databand is an IBM company that offers proactive capabilities to identify and resolve data issues in the early stages of the development cycle.

    ● Provides cross-stack visibility. This means you can get an overview of all the data tasks from start to end.
    ● Offers standardized DataOps and end-to-end data lineage, ensuring data reliability and accuracy.

    ● The program requires a significant amount of space, which makes it difficult to install it on the user’s system.
    ● It requires constant software updates making it much heavier.

    Datafold has a unique ability to proactively detect, investigate, and prioritize data quality errors before they affect production.

    ● Allows you to turn SQL queries into smart alerts, keeping you updated on any issues that may arise.
    ● Automates regression testing by integrating the CI process through GitLab and GitHub.

    ● Limited integration options.
    ● Does not provide any support for data analysis and data science.

    Closing Thoughts

    Data observability is the backbone of data engineers’ ability to be agile with their products. If you want to modernize your data management practices and improve your data quality, data observability is the way forward. Without it, your data team cannot rely on its tools and infrastructure, as issues can’t be detected efficiently and quickly.

  • Redgate Automates Pulling of Test Data From Production Databases

    Redgate Automates Pulling of Test Data From Production Databases

    Redgate today added a Redgate Test Data Management (TDM) tool that automates the pulling of masked test data from databases running in production environments.

    David Gummer, chief product officer for Redgate, said this capability eliminates manual tasks that previously inhibited testing or resulted in personally identifiable information (PPI) being inadvertently used to test applications.

    Redgate TDM automatically classifies and masks data residing in SQL Server, PostgreSQL, MySQL or Oracle database before it is used to test an application. It then produces a copy or clone of that data that is a fraction the size of the original, said Gummer.

    That approach creates a repeatable process for DevOps teams that can withstand a regulatory audit of application development processes, he added. In fact, with the increased focus on software supply chain issues, more auditors are specifically reviewing how data is secured during the application development process.

    Designed to be invoked via a command line interface (CLI), within an integrated development environment or via a graphical interface, Redgate TDM reduces the friction that currently occurs when DevOps teams request test data from database administrators (DBAs), said Gummer. Many DBAs are either too busy to fulfill those requests or reluctant to provide access to sensitive data that would result in the organization running afoul of a compliance mandate.

    Developers then often resort to using anonymous data that doesn’t actually reflect the environment in which their application will ultimately be deployed.

    The amount of data that developers need to build applications is only going to increase. In the absence of any way to effectively automate the process of accessing that data, it ultimately takes longer to build and deploy stable applications. In many cases, applications are simply not tested as thoroughly as they should be before being deployed in a production environment, resulting in rollbacks that might otherwise be avoidable.

    Advances in artificial intelligence (AI) are making it easier to create test scripts, but much of the testing setup process is still based on manual processes that increase the overall level of toil.

    At a time when more organizations than ever are trying to increase developer productivity, automating relatively mundane tasks such as pulling test data can have a big impact. Developers want to spend as much time as possible focused on writing code. Everything else is a task that gets in the way of achieving that goal. While there is a general desire to empower developers, that doesn’t necessarily mean they want the cognitive load associated with building applications to increase. In fact, one of the reasons platform engineering has emerged as a methodology for managing DevOps workflows at scale is to reduce the current level of cognitive load developers experience today.

    Of course, DevOps teams are, by their nature, ruthlessly committed to automating as many processes as possible. The challenge is bottlenecks in those processes conspire to slow down the overall rate at which applications can be built and deployed despite the best intentions of all involved.