devopsy.kinsta.cloud

Category: DataOps

  • KubeCon 2023: Telemetry and Data Management

    KubeCon 2023: Telemetry and Data Management

    Speaker 1: This is Techstrong TV.

    Alan Shimel: Hey everyone. We’re back here at KubeCon and it’s bustling, man. It’s a crazy big floor with a lot of people walking around, so we hope you like this. This is a little different than we usually do. Usually people can’t see the background because the cameras face this way, but we usually have sort of the typical background that you see at some of these other booths. But we wanted to give people kind of a flavor that we’re alive here.

    Tucker Callaway: We’re in the middle of it.

    Alan Shimel: We’re in the middle of it. So kind of like if you ever watch Sunday night football, they always have that. So that’s what we were shooting for. But anyway, let me introduce you to Tucker Callaway. Tucker is the CEO of a company called Mezmo. Now, some of you may have heard of, I think it was Log DNA.

    Tucker Callaway: Before. Yep.

    Alan Shimel: And they pivoted. Was it about four years ago?

    Tucker Callaway: About two years ago now.

    Alan Shimel: Two years ago.

    Tucker Callaway: I’ve been here for four, but big change was two years ago, renamed.

    Alan Shimel: Renamed to be Mezmo and a slightly different mission, let’s call it. And then I had a chance to catch up with Tucker at RSA last, I guess it was last April.

    Tucker Callaway: April. Yeah.

    Alan Shimel: This year it’s May. So I think it was around then last year. And for those of you who want to catch that interview, it’s actually available on Techstrong TV. If you look up RSA 2023, it’s there. But so Tucker, before we get into that, let’s back up. Mezmo. Not everyone out here. Look, not everyone out here knew Log DNA either, so don’t worry about it.

    Tucker Callaway: Fair enough. Yeah.

    Alan Shimel: Let’s start there. Let’s talk about the mission.

    Tucker Callaway: Yeah, so the mission of Mezmo is essentially to help people with their telemetry data. So we’ve found that there’s really kind of three main problems in the telemetry data space. There’s too much of telemetry data. It’s not in the right format. It’s not in the right location. So that was kind of one of our founding kind of principles of solving.

    Alan Shimel: Yeah, I think that.

    Tucker Callaway: Solving the problem.

    Alan Shimel: Those were three rock solid.

    Tucker Callaway: Hard to debate.

    Alan Shimel: Hard to debate. That’s the good way of putting that. And also, quite frankly, all three problems are like they compound each other.

    Tucker Callaway: They do.

    Alan Shimel: So it’s like you have too much data. That’s a big problem, but we have too much data that we don’t know how to categorize, so that exponentially makes that problem worse. So these are-

    Tucker Callaway: Compounding problems. No question about it.

    Alan Shimel: No doubt about it. So when we were at RSA, you felt like you were kind of on the cusp of a realization, right? You were going to help customers. Why don’t you? Well, you tell the story better than me.

    Tucker Callaway: So as we were talking about earlier, we had just launched this new product capability, the telemetry pipeline at RSA. We’ve got a lot of great customers and great experience since then. When we think about the core value of what it provides, it helps people with cost, it helps people get better insights and helps people enforce compliance. What we were surprised by though in the last couple months, is how little people actually understand of their data.

    In fact, we’ve put a lot of time and effort as an industry into getting control of our applications and getting control of our infrastructure, but we haven’t really gotten control of our telemetry data yet, and that’s like the big opportunity that we see. But with that, in order to get that value I described, people didn’t need to understand their data better. And so that was a big revelation for us.

    And we just launched our data profiling capability that helps people categorize and understand baseline, to do the schema management for their data, which is a really strong foundation for data ops principles applied to telemetry data. Then the next thing we found after we solved that problem was now that people understood their data and they were able to optimize it because they understood it. They also realized that the context and telemetry data changes rapidly. So for example, you don’t want to store all this data because you don’t need it until you need it, and then you really need it.

    Alan Shimel: I’ll take a little, they do want to store the data because people are data hoarders.

    Tucker Callaway: The data hoarder. Yeah.

    Alan Shimel: It’s when they get hit with the bill for storing all that data, they say, “Wait a second, I don’t need all this data. Not at that price.”

    Tucker Callaway: Something has to give somewhere.

    Alan Shimel: Exactly.

    Tucker Callaway: And so what we realized though was if you take the bookend of this value, we think of as understand, optimize and respond, we talked about understand and optimize the response stage was a really important one for us. So when new context comes in, the pipeline can change its behavior and change its optimizations to account for the current flow of data and the current needs. So if you’re sampling down to 20% and you see an event, you see performance, you can rehydrate, reprocess, dynamically change the responsiveness of the pipeline and get all the data to the right location for the right people, which helps address the cost problem, but also gives you the full visibility that you need.

    Alan Shimel: If one was going to do a business school case study on this, someone who’s not from this industry would say, “Hey, just the fact that you discovered that most of these organizations don’t even know how much data or what data they have would be enough to build a company around.”

    Tucker Callaway: Yeah, I’d like to think so.

    Alan Shimel: No, it’s logical. It makes sense, but it’s not, it’s the threshold. Because I think you’d hit the nail on the head here, Tucker, which is once they recognize that, now there’s a whole new vista of realization for them of, “Holy mackerel, I got all this data, now I want to start understanding it. Wow. I’m seeing stuff I never thought I’d see. Maybe I intuitively thought I had, but I got the goods now.”

    Tucker Callaway: We like to think of that as being mesmerizing, right?

    Alan Shimel: Okay, I love it. Mesmerized by Mezmo.

    Tucker Callaway: I couldn’t not take the shot, but that was just-

    Alan Shimel: Yeah, that was your shot.

    Tucker Callaway: That’s the derivative of the name.

    Alan Shimel: I feel used. I feel used.

    Tucker Callaway: I got back into it a little bit.

    Alan Shimel: So now you’re mesmerized and now like, “Oh my God, what can I do with this?” And that really becomes the business. And I guess that’s now where you are.

    Tucker Callaway: The next step for us is clearly that. Yeah, and I think there’s two plays on that, too. There’s a, “What do I do with it? Help me, tell me what to do. Give me the best practices that from all of your customers, what should I do?” And so we’ll be coming out probably later in the spring with something that actually allows you to simulate how that data might look when optimization’s applied to it so you can start to make those decisions more effectively.

    But then the second phase of that, too, is that also won’t be sufficient for certain organizations that want to look at their data as a strategic advantage. And they’re going to be looking for ways to take actions and drive more insights out of that data. And so the way we see that, we see that through recipes. We’ll give you the standard offerings, but then you have the ability to go in and edit as code and go do the things you need to do to go drive more value out of that data. If you’re ready for advanced yoga moves in the data space.

    Alan Shimel: Got it, got it, got it. My background is security.

    Tucker Callaway: Yep.

    Alan Shimel: Started a security company in 2001, and it was right when we were moving from IDS to IPS. So intrusion detection to intrusion prevention. Same thing with vulnerability management. We were moving from just finding vulnerabilities to patching, remediating, not always patching, remediating. Seemed like who wouldn’t want that? No brainer, right? Just don’t tell me I’m getting attacked. Block the thing. By the time you tell me, the attack went through. Don’t just tell me you found a vulnerability, make sure I’m not getting exploited. But a funny thing that I learned in that was a lot of people say, “Go slow here. I can’t afford, because sometimes the vulnerability or the attack or the data that you want me to act on can affect something that I consider more valuable.”

    Tucker Callaway: Yeah, fair enough.

    Alan Shimel: And I like to go slow when it comes to letting a program or an application or a product, just do things. Just give me my menu and I’ll decide what I want to do. Now, of course, that was before the world of AI and ML and a lot of the automations. And quite frankly, the speed of business was probably a little slower then. Do you see a way station where people are going to want that? Just give me best practices and I’ll decide when, how, and where I want to implement it. Or do we move right to, “Hey man, make it happen for me.”

    Tucker Callaway: I think there’s a way station, when we think about the data management, telemetry, data management in general, the word that’s the most important to us is trust, right? So it is one thing to optimize the data, but you have to trust that data is constantly being optimized and handled in the right way for all those varying needs you described.

    So I do see that the first step is a suggestion with a human in the loop. You want to hit that button still to say “go”, and then as you build trust in that data and you know that the algorithms are working, then you’ll let that go. Probably with some backstops on the side, I’m going to let it go, but I’m [inaudible 00:09:52] just in case. And then you’ll start to optimize it more and more. And I think naturally, almost like a realtime data platform, operating system will start to evolve.

    Alan Shimel: So really the key there is trust.

    Tucker Callaway: Key is trust.

    Alan Shimel: To build trust in the next. So as the CEO, how do you build that trust?

    Tucker Callaway: Well, trust is always a tough.

    Alan Shimel: I’m interested, you tell me.

    Tucker Callaway: Yeah. I think like anything else in life, you deliver trust through consistent delivery. And so that’s where I think that human in the loop step is important. It has to be sustaining. It can’t be a moment in time, it can’t be the very first implementation. It’s something that you earn over time where people can count on you or could count on the data or count on the ways that you treat data to get you to the outcome that you want to go through. So it’s really only through the delivery of outcomes to the customer that they will trust the systems and all those things.

    Alan Shimel: Makes sense?

    Tucker Callaway: Yeah.

    Alan Shimel: All right. Got to do a little housekeeping.

    Tucker Callaway: Okay, let’s do it.

    Alan Shimel: People who want to know more about Mezmo.

    Tucker Callaway: Come to the website. We’ve got an offer out there to profile anyone’s data for free right now.

    Alan Shimel: Do you really?

    Tucker Callaway: Yeah. Come to mezmo.com or stop by our booth.

    Alan Shimel: M-E-Z-M-O.com.

    Tucker Callaway: M-E-Z-M-O.com

    Alan Shimel: Out there in TV land or if you hear a coupon.

    Tucker Callaway: Come by the booth. I forget the number, but it’s over there.

    Alan Shimel: You know what, if you look on the, I don’t have it. If you look on the back of your ID thing, you could actually look stuff up like that.

    Tucker Callaway: Yeah, it’s back there. If you connected to the wifi here at KubeCon, it’s Mezmo Data.

    Alan Shimel: I’ve noticed that.

    Tucker Callaway: Is the wifi.

    Alan Shimel: I was like, “Wow, what a cool thing.” Yeah.

    Tucker Callaway: You have see us. Come see us. Profile your data will give you insight and understanding into that and talk to you about the next steps.

    Alan Shimel: All right, check it out at Mezmo. Tucker, thanks.

    Tucker Callaway: Thanks for having me.

    Alan Shimel: Actually, we’ll see you. Are you going to be at AWS?

    Tucker Callaway: Yeah, we’ll be there. Yeah, I’ll be there. Yeah.

    Alan Shimel: We’ll be doing videos, not on the show floor. It’ll be a little quieter in a private suite. And then of course you’ll be at RSA.

    Tucker Callaway: We’ll be at RSA.

    Alan Shimel: We’ll be there as well.

    Tucker Callaway: We might not be on the show floor at AWS just because of-

    Alan Shimel: It’s a little crazy there isn’t it? A little pricey.

    Tucker Callaway: Yeah.

    Alan Shimel: Yeah, I hear you. That’s why we’ll be at the wind in the studio.

    Tucker Callaway: I’ll see you there.

    Alan Shimel: Come on up. I’ll have coffee.

    Tucker Callaway: Sounds good.

    Alan Shimel: But RSA will be at broadcast alley and we’ll be doing our DevSecOps thing on Monday. And the usual kind of RSA craziness.

    Tucker Callaway: Looking forward to it.

    Alan Shimel: Absolutely.

    Tucker Callaway: Okay.

    Alan Shimel: Check out Mezmo.com.

    Tucker Callaway: Thanks everybody.

    Alan Shimel: Tucker Callaway here on Tech Drunk TV. We’re going to take a break. I guess it’s almost lunchtime here, but it’s still busy.

    Tucker Callaway: Near enough.

    Alan Shimel: Stay tuned. We’re here all day. We’ll be back. Bye-Bye.

  • Data Observability and its Importance: Everything you Need to Know

    Data Observability and its Importance: Everything you Need to Know

    In today’s tech-driven world, companies rely on almost incomprehensible amounts of data to dictate their business operations and decisions. Enterprises are focused on building multiple data repositories and data pipelines to process, store, manage and utilize data from various sources. Given the ever-growing size and complexity of the enterprise data environment, providing the accuracy and completeness of data is becoming increasingly challenging. So how can you understand how the data is performing across the entire IT infrastructure, all at once? The answer is data observability.

    What is Data Observability?

    Data observability is a process that aims to alert you to your data’s reliability and health while also delivering the insights and analysis required to identify and resolve issues before they affect your entire organization.

    Monitoring operations in the database serves as a proactive defense mechanism against potential security threats.
    This ensures accurate, complete, secure and valuable data and eliminates data downtime. Let’s dive deeper into the observability infrastructure and explore some of the best data observability platforms that can help you optimize and secure your data operations.

    Data Observability Framework

    We know that observability has three pillars: Logs, traces and metrics.

    However, data observability has five pillars, which, together, provide key insights into the reliability and quality of your data.

    Incorporating each pillar can help you craft an effective observability strategy.

    1. Recency: Also referred to as “freshness,” recency involves confirming if the data is up to date. It also analyzes if there are unusual temporal gaps in the data tables. This helps prevent timeliness issues in data pipelines.

    2. Volume: Volume involves checking if the amount of data intake in the database meets the expected thresholds. This ensures that data sets are complete.

    3. Distribution: This measures the quality of data at the field level and confirms if the data values are within expected ranges. Unexpected fluctuations in the distribution patterns are indicative of a data issue.

    4. Schema: This involves monitoring and auditing changes to data tables and data organization to look for signs of broken data. Schemas are extremely important as changes in the source data’s structure are often the cause of data downtime.

    5. Lineage: Lineage collects metadata and provides a complete picture of your organization’s data landscape, including upstream sources, downstream, and which teams can access the data at which stages. This process helps data teams troubleshoot where breaks occur in the data.

    Data Observability Benefits

    Data observability provides better visibility into your IT system’s internal state, including its behavior, performance and interactions with other systems. This can be beneficial in the following ways:
    ● Facilitates Root Cause Analysis: With end-to-end data visibility and monitoring across multi-layered IT infrastructure, data observability enables data teams to quickly spot issues in data sets with less effort. This also increases the chances of identifying new issues regardless of where they originate.

    ● Faster Mean Time to Detection and Resolution: As data observability monitors a wide range of output, data teams can actively triage and debug their systems. By providing valuable insights into how the data interacts and moves within the IT architecture, data observability helps teams spot problems they didn’t know about, resulting in quicker mean time to detection (MTTD) and mean time to resolution (MTTR).

    ● Automates Security Management: Data observability not only provides real-time visibility into the security posture but also makes it easy to automate parts of the triage process. This helps detect data health issues or data downtime instantly.

    Data Observability Challenges

    Depending on their existing IT architecture, organizations may face the following challenges when it comes to data observability:
    ● Data Silos: Due to the presence of several agents, silos monitoring tools and disparate data sources, it becomes difficult to comprehend interdependencies among applications, digital channels and various clouds.

    ● Integration with Entire Data Ecosystem: For data observability to work, the tool requires insights into the complete data pipeline and the servers, databases, software and applications involved. However, some organizations may find it challenging to connect all of the systems to a data observability platform.

    ● Manual Instrumentation and Configuration: Data observability tools aim to standardize telemetry data and logging guidelines to efficiently correlate information. However, as large organizations maintain several data sources (hundreds or even thousands), data from these sources might have varied standards. This requires manual effort to standardize data.

    Data Observability vs. Data Governance

    In today’s digital-first landscape, where data security is a global concern, there’s a growing focus on data governance. Data governance assists in the formation of policies and procedures required to keep a check on how an organization collects, analyzes, stores, shares and uses its data.

    A strong data governance program ensures data availability, integrity, usability and security. It eliminates data integration issues, data silos and poor data quality, thereby resolving the challenges of data observability.
    With data sets now scaling with more tables, more data sources and more complexity, data engineers and developers are under pressure to keep up with the data consistency, availability and security requirements. Any amount of downtime can lead to wasted resources and time while deteriorating confidence in decision-making.

    Data observability, along with data governance, helps organizations manage data security and quality problems in a streamlined manner.

    Data Observability vs. Data Quality

    Data quality measures the completeness and accuracy of data sets to determine if they can be used in analytics and operational applications. Data observability, on the other hand, allows organizations to detect and fix issues in the data pipeline efficiently and quickly.

    For effective data management, an enterprise must take both into account. Earlier manual checking and cleansing of data sets helped determine data quality. But today, many of these tasks have been automated, thanks to modern data stacks.

    As a result, there’s been a shift in focus from data quality to data observability. With data observability, businesses can effectively monitor and troubleshoot their data pipeline. Without it, they are at the risk of relying on incomplete or inaccurate data to make decisions. This can lead to expensive mistakes.

    Finding the Right Data Observability Tools

    Undoubtedly, data observability is a critical feature of any enterprise that uses data. However, not all data observability tools are equally beneficial to your business. Look out for the following traits when choosing a data observability platform:
    ● Compatible: The tool must be compatible with your data lakes, databases and cloud storage solutions.

    ● Autonomous: Autonomous technology responds to stimuli without any human intervention. This is essential in a data observability tool as it helps in the early detection of anomalies and an instant response to alerts.

    ● Timely: A data observability tool must help you identify errors as early as possible. The best platform constantly monitors data health from the moment the data is added to the ecosystem to the end of its lifecycle. Early detection of errors stops issues before they become critical.

    ● Sophisticated: The best platforms leverage machine learning (ML) and artificial intelligence (AI) to identify hard-to-find issues.

    ● Other features: The tool must be able to collect, sample, review and process telemetry data across several data sources. It should serve as a centralized data repository, offer comprehensive data monitoring services and provide data visualization.

    Eventually, the right data observability platform depends on your organization’s observability engineering needs and unique IT architecture.

    Top Data Observability Tools

    Data observability platforms are still a budding product category. The good news is that over a half-dozen data observability companies now offer commercial tools with excellent features. These include the following:

    Observability Vendors, Overview,  Pros and Cons

    Monte Carlo Data is headquartered in San Francisco. Monte Carlo is the creator of the industry’s first end-to-end data observability tool.

    ● Provides comprehensive data observability capabilities.
    ● Offers a high level of features, including automated alerting, data catalogs, etc.
    ● Supports fully automated setup.

    ● High data volumes can lead to user interface issues.
    ● Huge amounts of variables restricted by different constraints can lead to computational inefficiency.

    Bigeye is an industry-leading data observability platform that allows teams to improve, measure, and communicate quality data clearly and quickly at any scale.

    ● An easy-to-use interface that facilitates data configuration while ensuring consistency and accuracy.
    ● Provides powerful API integration capabilities.
    ● Features a versatile dashboard with real-time tracking and monitoring of data quality metrics by multiple people.

    ● It can be expensive for smaller organizations.
    ● The tool might crash at times, requiring performance improvements.

    Acceldata offers tools for Hadoop, cloud services, and companies which include end-to-end data reliability, data pipeline monitoring, and multi-layer data observability.

    ● Provides fully automated data reliability checks.
    ● Offers a drag-and-drop interface to analyze data pipelines across several layers and platforms.

    ● Initial setup can be complex and difficult.
    ● Node addition and node removal require human intervention.

    Databand is an IBM company that offers proactive capabilities to identify and resolve data issues in the early stages of the development cycle.

    ● Provides cross-stack visibility. This means you can get an overview of all the data tasks from start to end.
    ● Offers standardized DataOps and end-to-end data lineage, ensuring data reliability and accuracy.

    ● The program requires a significant amount of space, which makes it difficult to install it on the user’s system.
    ● It requires constant software updates making it much heavier.

    Datafold has a unique ability to proactively detect, investigate, and prioritize data quality errors before they affect production.

    ● Allows you to turn SQL queries into smart alerts, keeping you updated on any issues that may arise.
    ● Automates regression testing by integrating the CI process through GitLab and GitHub.

    ● Limited integration options.
    ● Does not provide any support for data analysis and data science.

    Closing Thoughts

    Data observability is the backbone of data engineers’ ability to be agile with their products. If you want to modernize your data management practices and improve your data quality, data observability is the way forward. Without it, your data team cannot rely on its tools and infrastructure, as issues can’t be detected efficiently and quickly.

  • Observability: The Central Watchtower Your All-Seeing IT Needs

    Observability: The Central Watchtower Your All-Seeing IT Needs

    If there’s something you need to keep secure, it’s natural to want to keep it sealed away under lock and key. However, even keeping something in a vault, far from prying eyes and ill intent, may not be enough to offer peace of mind. After all, how can you have the peace of mind of knowing it’s safe if you don’t regularly check to ensure it hasn’t bolted, been burgled or broken down?

    Now, multiply the volume and value of things being monitored tenfold — and you get a sense of the daily inefficiencies and anxieties IT professionals battle.

    Instead of jewels, these teams safeguard, manage and maintain something far more valuable: The data that underpins everything from business transactions to system performance to threat detection to service delivery and beyond. To make matters more complicated, today’s digital ecosystems are more complicated, distributed and disparate than ever before.

    Databases are critical to managing a successful IT department — and it’s time to discard the widespread industry’s perception of them as an unknowable black box. To be sure, databases represent the most difficult component of the IT ecosystems to observe, tune, manage and scale. But the database isn’t a Magic 8 ball in terms of its internal processes; We no longer have to shake the mystery toy and accept one of the six answers it’s designed to spit out.

    The fact of the matter is that database specialists and IT teams need a clear view of database performance telemetry if they want any hope of maintaining the health, stability and scalability of their services.

    Observability Tools

    This is where observability tools can play a transformative role. Taking its cues from the all-seeing panopticon watchtower, observability eliminates the dark corners of the database dungeon by providing a comprehensive view across the full cloud-native, on-premises and hybrid technology stacks.

    The word panopticon derives from the Greek word panoptes, or “all-seeing.” The panopticon model features a central, lighted guard tower at the center of a circular room — allowing 360-degree lines of sight that allow the guards to observe every surrounding cell from a single, centralized vantage point.

    English philosopher Jeremy Bentham had no way of foreseeing the challenges of today’s database professionals when he first conceived of the panopticon in the 1700s. The concept would lay the blueprint for a prison system designed around one core concept: Enabling the minimum number of guards to effectively monitor the maximum number of inmates. Bentham’s system also works as a strategy for ensuring the health and performance of your organization’s critical databases.

    The panopticon was designed so that every cell could be monitored from one central point — ensuring not only better and more straightforward security but preserving resources in saved time, manpower and effort expended. Think of your ITOps, database, DataOps and DevOps teams as the wardens of your IT performance. Traditional monitoring can be likened to patrolling a traditional prison: Rows of cells facing out into a hallway that is patrolled on a schedule or used to spot and address misbehavior as it’s noticed.

    How can they be sure they’re focusing their efforts in the right place? Or trust that Cell Block A won’t stage a breakout while they’re breaking up an argument in Cell Block B?

    In short: They can’t. Odds are, the team is either running from outage to outage in an everlasting firefighting mode or relying on random spot checks for something as vital as the health of our IT systems.

    Observability, as opposed to monitoring, puts teams into that central panopticon watchtower, allowing them to not just see everything, but to see it within the context of the bigger picture. This comprehensive visibility and always-on observation enable IT teams to identify critical issues as they occur, even those caused by complex dependencies between the database, operating system, storage subsystem and network.

    Database Performance Issues

    Furthermore, database performance issues are a starting gun for costly bottlenecks or serious outages that can handcuff your company’s ability to compete or grow. Without complete and precise database monitoring and observability, IT operations struggle to accurately determine an application’s root cause of performance issues. This increases the risk of downtime, data loss and poor customer experiences.

    Discovering the root cause of performance issues is quite literally a system-critical undertaking, whether you’re simply running your enterprise, scaling your operations or deploying new code. Observability also offers teams the rare opportunity to get ahead of performance issues since you can see performance anomalies in advance of a service disruption rather than relying on real-time remediation.

    With traditional methods, DevOps and IT teams must manually analyze the data presented to them, correlate it to the problem, and locate the error before they can finally begin addressing it. In contrast, observability collects data to provide information on what’s not performing as expected and why. This allows teams to take a proactive approach to solving downtime and bottlenecks and proactively preventing them.

    In the software world, that translates directly to more time spent where it matters: Developing new business value within your applications and infrastructures, fueling innovation and exceeding customer expectations.

    Given how complicated the modern database has — and continues to — become, arming your enterprise with the right observability solutions can provide a boost to efficiency and performance and pardon your teams from a sentence to time-consuming, mundane and error-prone work.

  • Couchbase Adds Columnar Database to DBaaS Environment

    Couchbase Adds Columnar Database to DBaaS Environment

    At the AWS re:Invent 2023 conference today, Couchbase, Inc. announced it added a columnar database based on the JSON file format to its database-as-a-service (DBaaS) portfolio to enable organizations to build real-time analytics applications.

    Scott Anderson, senior vice president for product management and business operations at Couchbase, said the Capella columnar database is designed to be integrated with the company’s JSON-based document database via a database change protocol (DCP) that Couchbase has embedded within its DBaaS environment.

    Deployed on cloud services from Amazon Web Services (AWS), the overall goal is to reduce the level of friction that IT teams would otherwise encounter when integrating a document database with a columnar database optimized to process analytics using columns rather than the rows typically associated with a relational database.

    That approach gives IT teams the ability to use a schemaless database architecture that provides an alternative to employing disparate databases that would need to be deployed and then integrated using extract, transform and load (ETL) processes to move data from one database to the other, noted Anderson.

    Capella columnar also uses the same SQL++ language as the company’s document database to streamline queries in addition to supporting Capella iQ, a copilot based on generative artificial intelligence (AI) that enables queries to be crafted in natural language.

    In addition, Capella columnar provides connectors to move data from Amazon DynamoDB, Amazon DocumentDB, Amazon Relational Database Service (Amazon RDS) and other data sources deployed on the AWS cloud.

    In the longer term, Couchbase is also planning to add vector capabilities to its DBaaS environment to make it simpler to securely extend generative AI models, said Anderson.

    In general, document databases have been widely adopted because they provide developers with an alternative to legacy database platforms that they can download and deploy themselves. The Capella DBaaS platform further simplifies that process via a managed service through which Couchbase assumes operational responsibility for managing databases. The columnar database extends that capability using a columnar database that can drive analytics capabilities to deliver a more personalized application experience, noted Anderson.

    It’s not clear to what degree organizations will ultimately wind up relying on as-a-service platforms, but since the start of the COVID-19 pandemic, the shift to this approach to consuming IT resources has increased significantly. As a result, the way internal IT teams are organized is shifting as lower-level tasks are either automated or handled by an external IT team that works for a vendor.

    Regardless of how databases are managed, however, there remains a need to integrate the data operations (DataOps) processes used to manage them with the DevOps workflows used to build modern applications. Each organization will need to determine how much economic sense it makes for them to manage that data versus relying on an external services provider. The challenge when relying on an external service provider is, as always, integrating that service into an existing IT environment.

  • Redgate Automates Pulling of Test Data From Production Databases

    Redgate Automates Pulling of Test Data From Production Databases

    Redgate today added a Redgate Test Data Management (TDM) tool that automates the pulling of masked test data from databases running in production environments.

    David Gummer, chief product officer for Redgate, said this capability eliminates manual tasks that previously inhibited testing or resulted in personally identifiable information (PPI) being inadvertently used to test applications.

    Redgate TDM automatically classifies and masks data residing in SQL Server, PostgreSQL, MySQL or Oracle database before it is used to test an application. It then produces a copy or clone of that data that is a fraction the size of the original, said Gummer.

    That approach creates a repeatable process for DevOps teams that can withstand a regulatory audit of application development processes, he added. In fact, with the increased focus on software supply chain issues, more auditors are specifically reviewing how data is secured during the application development process.

    Designed to be invoked via a command line interface (CLI), within an integrated development environment or via a graphical interface, Redgate TDM reduces the friction that currently occurs when DevOps teams request test data from database administrators (DBAs), said Gummer. Many DBAs are either too busy to fulfill those requests or reluctant to provide access to sensitive data that would result in the organization running afoul of a compliance mandate.

    Developers then often resort to using anonymous data that doesn’t actually reflect the environment in which their application will ultimately be deployed.

    The amount of data that developers need to build applications is only going to increase. In the absence of any way to effectively automate the process of accessing that data, it ultimately takes longer to build and deploy stable applications. In many cases, applications are simply not tested as thoroughly as they should be before being deployed in a production environment, resulting in rollbacks that might otherwise be avoidable.

    Advances in artificial intelligence (AI) are making it easier to create test scripts, but much of the testing setup process is still based on manual processes that increase the overall level of toil.

    At a time when more organizations than ever are trying to increase developer productivity, automating relatively mundane tasks such as pulling test data can have a big impact. Developers want to spend as much time as possible focused on writing code. Everything else is a task that gets in the way of achieving that goal. While there is a general desire to empower developers, that doesn’t necessarily mean they want the cognitive load associated with building applications to increase. In fact, one of the reasons platform engineering has emerged as a methodology for managing DevOps workflows at scale is to reduce the current level of cognitive load developers experience today.

    Of course, DevOps teams are, by their nature, ruthlessly committed to automating as many processes as possible. The challenge is bottlenecks in those processes conspire to slow down the overall rate at which applications can be built and deployed despite the best intentions of all involved.