Tag: nosql

  • Building Scalable Cloud Applications With MySQL

    Building Scalable Cloud Applications With MySQL

    Relational database systems such as MySQL have long been the go-to choice for handling large and complex data handling tasks. But, with the introduction of new technologies like NoSQL, many developers started to question whether traditional relational databases still have the upper hand. However, when building high-scalability cloud applications, relational databases like MySQL still have the edge over NoSQL. Here are five reasons MySQL remains superior to NoSQL in supporting developers of high-scale cloud applications.

    1. Enablement of Complex Queries

    Structured query language (SQL) is the most widely adopted language for relational databases, and MySQL is no different. SQL gives developers a powerful platform for creating complex queries and ad hoc reports, efficiently managing large datasets and creating tables and indexes for data optimization. By contrast, NoSQL usually offers only rudimentary query and data manipulation capabilities, often proving inefficient when handling complex data sets. Most data analysts are also familiar with SQL, so it’s a drop-in tool they can use to help gain insights.

    2. Scalability and Reliability

    MySQL is noted for its scalability when handling massive data loads and is particularly well suited to cloud-based applications that require consistently high performance. MySQL ensures that the consistency of data is maintained. MySQL is the go-to option for scalability and reliability for larger cloud applications. Companies such as Facebook, GitHub and WordPress have built billion-dollar businesses on MySQL, and it’s been tested and validated in some of the harshest and most demanding workloads of any database.

    3. High-Performance Replication at Scale

    MySQL replication offers a reliable solution for cloud applications that need to scale. Unlike other databases, MySQL’s replication capabilities provide effective data safety, making it a robust choice for applications that require high availability and durability. Additionally, MySQL enables seamless version upgrades without disrupting operations, ensuring the system always has the latest features and security patches. The read scaling feature of MySQL improves performance by distributing the read load across multiple instances, making it ideal for large-scale cloud applications. With a proven track record at scale, MySQL replication is the optimal choice for applications that want to grow and scale efficiently.

    4. High Availability and Durability

    MySQL provides enhanced high availability and durability, ensuring that critical data is always available even during maintenance and disaster recovery. This is because MySQL is fully ACID-compliant (atomicity, consistency, isolation, durability) and provides full support for transactions. Even when the system fails, the transaction can be rolled back and the data is not lost. With NoSQL databases, data is primarily written in memory and doesn’t provide the same degree of durability or consistency.

    5. Ease of Implementation and Debugging

    MySQL is easy to install, implement and debug compared to NoSQL. MySQL is supported by a large community of developers and database administrators who use it for a diverse range of applications. MySQL is also open-source and free to use, making it a great choice for businesses of all sizes. NoSQL systems require the installation of additional tools, libraries, and other components for implementation, making it more complex and time-consuming.

    MySQL provides many benefits for developers of high-scale cloud applications, ranging from its powerful SQL language for complex querying to its high availability and flexibility for managing multi-table joins. NoSQL remains a viable option in some cases, but developers who are building high-scale cloud applications are mainly sticking with MySQL due to its scalable and reliable nature, easy implementation and flexible query-handling capabilities. Choosing the right database engine is critical for any organization that relies heavily on large and data-heavy applications. MySQL continues to stand the test of time when it comes to ensuring performance and getting consistent results.

  • How Database DevOps Fuels Digital Transformation

    How Database DevOps Fuels Digital Transformation

    In today’s real-time, hyperconnected reality, data is a core driver behind most digital strategies. User data is constantly gathered and shared across various technology backends and platforms—as is system performance data to inform engineers. Data is pivotal for everything, from generating machine learning models, refining user experiences and even supporting new data-centric services. But although data is a clear backbone to many enterprises, there is no clear winner regarding which database to use.

    Large organizations often find themselves supporting different database styles, from relational databases to NoSQL and distributed SQL. And as organizations continue to adopt multiple databases, safely managing the processing and storage of data becomes more challenging to enact at scale.

    I recently met with Jakub Lamik, CEO, Redgate Software, to understand the role DevOps, specifically database DevOps, is playing in advancing data-driven digital transformations. According to Lamik, applying DevOps practices for database management can improve performance, increase productivity and help organizations stay compliant. Below, we’ll consider the state of today’s data-driven digital transformations and highlight ways to improve how DevOps is applied to database management.

    State of Data-Driven Digital Transformations

    The 2022 StackOverflow Developer Survey found the most popular databases to be MySQL, PostgreSQL, SQLite, MongoDB and Microsoft SQL Server. But most often than not, businesses deploy more than one of these in production. Redgate’s 2021 State of Database DevOps report found that 70% of businesses had more than one DBMS in use, and 48% had teams working on three or more different databases.

    Engineers operate in increasingly hybrid and distributed technology environments. This isn’t for no reason—in fact, 90% of organizations said multi-cloud is helping them achieve their business goals. Lamik also notes a diverse array of software technologies in practice today, from multiple clouds to decoupled microservices architectures and database types. But the desire to work with diverse toolsets causes fragmentation, which could hinder software delivery efforts. For example, organizations want to deploy the same framework and similar methodologies across different databases, said Lamik, and have it be database- and cloud-agnostic.

    Tips to Improve Database DevOps Operations

    To increase the pace of software delivery, infrastructure automation is often needed. And like other roles, database professionals are increasingly trying to automate operations and free themselves from mundane tasks. So, what are some ways to streamline database DevOps? First off, Lamik encourages adopting a common approach across multi-database environments, using standard toolsets that are flexible and portable. “The organizations that standardize deployments across different database environments can increase the frequency and reliability of deployments,” he said.

    Catching and resolving errors quickly is also becoming increasingly essential to meet objectives. And although DevOps brings many advantages, the CI/CD pipeline has gaps. “Observability supported by intelligent automation is key to closing those security gaps,” said Lamik.

    Thus, he encouraged additional oversight and tracking of changes to ensure stability throughout the database deployment pipeline. Also helpful is encouraging collaboration with security professionals (also known as DevSecOps) and integrating security principles into the DevOps workflow to ensure the process is secure by design.

    Another element of database security involves the proper observability and classification of sensitive data. As Lamik describes, organizations must be careful to distinguish between data used for production purposes and data used for development purposes. Ensuring sensitive data is cataloged correctly is imperative to avoid production data from being breached or accidentally publicly exposed.

    Final Thoughts

    In summary, applying DevOps ideals to database management could be a boon for development efforts. For those who haven’t yet considered it, a DevOps transformation will pay dividends, says Lamik. Furthermore, a common management layer could help wrangle multiple database types and help adopt new emerging database types, such as Snowflake and Postgres. Yet, when acquiring new database toolsets, Lamik admitted it may not be a quick transition. He recommends a land-and-expand strategy rather than a top-down approach (which typically isn’t as effective in empowering individual developers).

    Commenting on the industry’s recent move toward platform engineering, Lamik sees it as the next evolution of DevOps. Whereas DevOps brought helpful toolsets to streamline deployment and digitization, platform engineering is taking these practices and productizing them to be reusable at scale across teams and departments, he explained.

    Also, in the new cloud-native stack, containers are designed to be ephemeral, causing engineers to craft workaround solutions to enable stateful storage on platforms like Kubernetes. Regardless of deployment, data must have proper backup and recovery controls as well. In this new cloud-native paradigm, the role of database professionals is changing, explained Lamik, and “APIs are becoming the focal point of communication.”

  • DevOps Chat: Scaling Data Without Complexity with Cockroach Labs

    DevOps Chat: Scaling Data Without Complexity with Cockroach Labs

    Cockroach Labs the folks behind CockroachDB recently announced another round of funding as they continue their mission to allow you to scale your data without complexity.

    In this DevOps Chat, CEO of Cockroach Labs Spencer Kimball talks about about their plans and how Cockroach shall inherit the DB world.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel from devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. On DevOps Chat today, I’m really happy to be joined by Spencer Kimball. Spencer is CEO/co-founder of Cockroach Labs—hot company in the DB and DevOps space. Spencer, welcome.

    Spencer Kimball: Thank you, Alan. It’s a pleasure to be here.

    Shimel: Absolutely. So, Spencer, let’s get it out of the way. First, I’m assuming most of our listeners have already heard of Cockroach Labs, but it’s possible they haven’t. Why don’t you give us a little background?

    Kimball: Well, sure. I mean, first, we should probably get the obvious question out of the way—why would we name something Cockroach? I think that’ll become clear when I give you the explanation, but Cockroach Labs was founded to build a database, and that database replicates and it’s hard to kill, so that’s where the name came from. Honestly, the name was invented before the product and long before it was a company. So, it was—you know, now I’m in a position of explaining that same thing to Fortune 500 CIOs on a weekly basis. [Laughter] There’s some regret there, but the name makes sense.

    So, I can give you a little bit of the background. All three co-founders of Cockroach Labs worked at Google for about 10 years in the office and Google went through Bigtable which started the NoSQL movement, and it quickly moved to Megastore, which introduced some transactionality. Then they created Spanner, and Spanner is something that Google’s been working on for more than a decade now.

    When we left Google, we realized Spanner is a solution to many companies’ problems, not just Google’s. And that’s become more and more true in the years since. We did a startup, and even as a startup, we wanted capabilities that Google had built into Spanner, and those weren’t available in open source. So, that’s sort of where the original idea of Cockroach came from. That startup got acquired by Square, the payments processing company, and many other things now as well. And Square needed something like Spanner as well.

    So, the idea kind of had gained currency. We started as a GitHub project. That took off, VCs got interested. The company was started, now, four and a half years later, we’re busy selling to the Global 2000, Fortune 500 big companies out there as well as competing on one side with Oracle and on the other side with Amazon.

    Shimel: Only on the Internet, man, you know? [Laughter] It’s a crazy story, but true. So, first of all, I absolutely get the Cockroach thing, because they are gonna inherit the earth, you know, or so it seems, especially with the path we’re heading, but let’s not get into politics.

    But it is kind of an amazing story, because when—for people who are in the IT business and are somewhat familiar with the history of the dashboard and how Larry Ellison came to be the second richest man in the world or whatever it is he is these days, you know, it’s fascinating that all of everything we do online, almost, is predicated on the ability to store the data and to access that data in ways that allow us to do all of, you know, manipulate the data, analyze the data. I mean, without databases, none of that happens.

    And so, when you look at the history of the database and then the, you know, to relational databases and then you mentioned a few other things, SQL database and then NoSQL databases—

    Kimball: And back to SQL. [Laughter]

    Shimel: Right, and then back to SQL, then you’ve got time stamped databases are a thing now. You know, all revolving around, though, how do we store this data in a way that’s easy to put the data in there and easy for us to access that data, manipulate it, analyze it, you know, and make value out of it, right?

    And so, much like a cockroach, data doesn’t die, either, right? [Laughter] It’s eating the world, you know? Where was I recently, I was shopping at Costco or something, and they had eight terabyte, I guess, backup or drives, you know, external drives that were maybe a little bigger than credit cards, you know? Size of a good iPhone, let’s say, a little thicker. And I think it was $169.00 for eight terabytes. [Laughter] And I’m thinking, “My God, you know, when I had those 3.5 inch floppy disks, man, and the bigger ones”—

    Kimball: And the first 20 megabyte hard drive, it was—

    Shimel: Yeah! I didn’t think there were eight terabytes in the whole world.

    Kimball: And there weren’t for a while, right?

    Shimel: Yeah, it’s crazy. And, you know, as our data storage and data needs have grown, so has our databases. So, you mentioned the spinoff from Google, you mentioned certain things that the latest, greatest from Google didn’t do and hence, why Cockroach.

    But you guys recently also announced a new funding round, if I’m not mistaken, right?

    Kimball: Correct, right. We’re going to—that will happen on August 6th.

    Shimel: Well, by the time people are listening to this, it’ll be August 6th already, so.

    Kimball: Got ya, okay.

    Shimel: If you’re listening to this at home, it’s after August 6, they’ve announced their funding round.

    Kimball: Yeah, so, we did close that, and it’s a big round. And basically, the additional money equals all the money we’ve raised in the past which, you know, isn’t so unusual, but it’s a lot of money. In aggregate, we’re talking over $100 million.

    Shimel: Wow.

    Kimball: You know, I think, to put that into perspective, you just have to look at who we’re competing with, right? I mean, certainly, it’s some of the biggest companies in the world, the biggest companies in the world, and we’re also serving the biggest companies in the world, so you do need that money both to create a sales force that’s capable of educating and reaching and selling to those companies, which all—I’ve been humbled and amazed by the process necessary to actually sell to a Fortune 500 company. In some cases, it’s quite involved.

    But also, to continue to iterate on the product, and we have to, you just need to hire the best people and create an environment where they can collaborate to move the product forward so that you can compete.

    Shimel: Absolutely. But, you know, look—in the Internet and the IT world today, compete means different things to different people, right? We compete by offering great service. You know, I just got done, I was over at DevOps Enterprise Summit in London a month or two ago, and there was this notion there of what folks were calling BVSSH—Better Value Sooner, Safer, Happier, right? And that’s what we try to deliver in DevOps and stuff like that.

    So, does competing on that and—but then, in the database world, too, it’s just really, sometimes it’s very, very, very feature driven.

    Kimball: Yeah.

    Shimel: Right?

    Kimball: It’s incremental features, and then there are sort of generational features. And we’re, we’ve been pushing on truly changing the nature of the database, and that is an opportunity that exists because of all the other things happening in the ecosystem—in particular, the global public cloud, which is, I think, one way to describe the combination of AWS and GCP and Alibaba Cloud and Azure. There’s many others, of course, that are far more regional. But in aggregate, all of those can be utilized by new database architectures to allow a company to store data in new ways that ____ their business versus their competitors.

    So, just to give you an example, things like GDPR and recent regulations around privacy and Russia and China and Vietnam and South Korea and South America—the list goes on—there’s a lot of attention being placed on how data is localized, where it’s stored in relation to the customer, where it’s processed.

    And also—and this is, I think, really exciting—there is an ability, because of this global public cloud, for new services to come online that are able to deliver the end to end user experience that happens instantaneously to the user. So, that means in under 100 milliseconds, which is sort of the user threshold for noticing time intervals. If something feels instantaneous, but it happens in less than 100 milliseconds, you literally, because of physics and the speed of light, you can’t do that for a customer in Australia if all your data is in one data center in Virginia. But that’s the model that’s prevalent right now.

    Shimel: Sure is.

    Kimball: [Cross talk] on public cloud and with a solution like Cockroach, you’re actually able to have a globally spanning database that’s able to domicile data near the customer and actually deliver both the assurance of some data privacy, that their data is in process and stored close to them in their legal jurisdiction, but also to have an end to end user experience where you click a button on your mobile app, it goes all the way through the application logic, accesses the database, comes all the way back and updates. So, that ability to make that instantaneous, I think, is gonna open a new era of applications.

    Shimel: It sure is. You know, so, I remember, Spencer, back—so, one of the companies during the dot com days, one of the companies I helped build was a company called, it came to be known as Interliant, and we were an early ASP application service provider.

    But before we changed our name, we were Sage Networks, and we were a large hosting provider, a rollup. We had bought a lot of different hosting companies. And one of the key things for us, like, you know, what really brought us scale versus all these smaller hosting companies we had bought, is that all of these hosting companies’ storage and the data was sort of part and parcel with the same infrastructure, if you will, that was hosting the website and the web server and all of that.

    And what we did is, you know, we actually—I don’t wanna say abstracted, because it wasn’t really abstract, but we separated the data and the database from the web server and the applications they were running, right? And now, that freed us up to store that data. This is before there was a cloud, but we had multiple data centers. So, we were at the point, almost, where we could store data, let’s say, in our Atlanta data center and access it or use it on websites where the web servers were stored, let’s say, in Tysons Corner in Virginia at the time. And what was holding us back there was, we didn’t have the connectivity you have now.

    Kimball: Yeah.

    Shimel: Right? If you had a T3 line, that was hot stuff, you know? That was carrier grade. And so, it was a little different. But, you know, this is the age we live in now where you do have distributed applications. So, of course, why wouldn’t you have distributed data storage and the databases that organize them?

    Kimball: Yeah. I mentally have distributed audiences, right? I mean, every time a fledgling startup puts their first mobile app on the app store, the total global population that can have access to that, it’s pretty staggering. I mean, the opportunity there is just tremendous. But, you know, everyone’s—90 some-odd percent of efforts out there are focused on deploying to a single availability zone, right? So, this is, there’s a legacy of how things were built, which is rapidly changing, and you have to do it through the whole stack. At Cockroach Labs and CockroachDB, we’re attacking a fundamental part of that stack, very important, probably one of the harder parts, which is the data storage, the state behind those applications.

    Shimel: Yep. No doubt about it. I wanted to talk a little bit about CockroachDB sort of as a project, if you will. Cloud native, right? When you guys were starting out, obviously, you came out of Google, you have a big open source kind of heritage at Google—what were the discussions then about, “Is this something we open source? Is it something we make free?” How do we walk that line, right? Go ahead.

    Kimball: So, when I was at Google, from 2002 to 2012 with my co-founders, Google was only starting to do open source towards the end of that tenure. Previously, they were pretty adamant about keeping things closed source. It was just sort of the model that Google evolved with. Now, they’re much more open, and some of their more recent efforts, including TensorFlow and Kubernetes, are examples of where open source in the hands of Google actually is a tremendous advantage both to the company and to the larger ecosystem. But things like Spanner were closed source and many of the things that Spanner is built on are also closed source.

    But what Google did do, which is great, is they always publish about these things.

    Shimel: Right.

    Kimball: When we started Cockroach, we had the benefit of not only working at Google and seeing how these things were built—you know, Peter, one of my co-founders, and myself worked on something called Colossus, which is the exoskeletal distributed file storage system at Google, which runs underneath Spanner.

    So, we had a really great opportunity to see why and how Google built the things that they built. But, you know, once we were kinda kicked out of the Google nest—or we left it, I guess, is the better way to say it—

    Shimel: Flew the nest. [Laughter]

    Kimball: Yeah. We flew the coop, right? We had the benefit of all of the published work that Google had created around things like Spanner, but you know, out in the wider world, you realize, for a database—and when we started this, I guess it was 2014. In 2014, and it’s certainly true today, people aren’t going to pay attention to your database unless it’s open source. And it’s just, it’s sort of a fundamental requirement. And there have been databases that have tried as closed source software, but they haven’t gotten very far, to my knowledge.

    So, I just think that—and this is true for me, too, as a potential user of the database, as a potential developer on the database, something that’s not open source feels a little bit icky. [Laughter] Right? Like, there’s not enough surface area that I can latch onto where I feel like I can be the master of my own fate, right?

    Shimel: Yeah.

    Kimball: Like, if something—that’s what I hated about closed source products. If something goes wrong, it’s just this big black box and, you know, it’s sort of a black hole if you wanna send a support ticket in, what’s gonna happen? Whereas it would be much nicer for everyone involved, in my opinion, if I were able to dig down in there, find out what was wrong, probably my fault, but let’s just say it’s a bug in the product, I could send that into the company and they could take that as a patch.

    That’s the beauty of open source and that, as a personal preference and sort of even as an idealistic sort of mindset. I would not have been able to start the database as anything except for open source.

    Shimel: Yeah. I mean, and look, quite frankly, there is a bit of a history in the database space of successful open source databases that not only were successful as open source projects, but were equally or more successful as commercial entities or the companies that manage these open source projects—you know, very successful open commercial entities that had great exits and great value for the investors and founders and employees.

    So, I mean, in that regard, it’s not like you’re pioneering, right? It’s not like someone says, “Oh, they’re crazy to do this in open source,” right? Well, 15 years ago, they might have said you were crazy, right? [Laughter] You know—

    Kimball: I think MySQL kind of, you know—

    Shimel: Yep, ____.

    Kimball: —blazed the trail, as they say. And Mongo is a very successful—

    Shimel: Absolutely.

    Kimball: [Cross talk] that uses open source as well. And there’s a number of other ones. But you know what, even when we started this company in 2015, I think the open core model and open source in general seemed like a catalyst, right? Just something that could create business value far more quickly than closed source software and the sort of procurement department and budgets and all this stuff that used to be involved in the cycle of adopting a new technology in order to build your products or services that the engineers or developers at a company needed to—

    Shimel: Oh, absolutely. I mean, look, I’ve been in IT 30 years, right? I’ve seen the rise of open source. I was involved in open source security stuff in 2001, ’02, ’03, ’04. The deal is, today, for a lot of developers, if it’s not open, they don’t wanna work on it, number one.

    Kimball: Yeah. But—

    Shimel: Go ahead.

    Kimball: What’s really fascinating is that 2015 might have been sort of the beginning of the end for open source being the true, the most important catalyst of business value. I would argue that, in 2019, the true catalyst to business value, faster business value—business value in sort of relation to your competitors, I think it’s services, right? Cloud-based services. That is—you know, it’s always about, it’s not about the idealism of open source. That’s not why it was successful.

    Shimel: No, I think we’ve passed the idealistic stage of open source.

    Kimball: It’s basically like, what are the tools out there that a software engineer at company X can use in order to get their project done faster so that it’s more stable, it’s more performant, it costs less, right? All those things—

    Shimel: But it’s their preference, and these developers and the Ops and the DevOps folks, they’re the top predator on the food chain at a lot of these companies. They’re the hardest piece of the puzzle. And so, companies are hesitant to tell them, “No, I don’t want you to use that tool” or, “I don’t want you to use this service.” They’re the ones making these choices, where I think, in years past, it was like, “Oh, no, you can’t bring that in here” or, “You can’t use that one” or something like that.

    Kimball: Yeah. It kinda [Cross talk].

    Shimel: But there’s more, Spencer. Let me tell you this. You know, I’ve been going to a bunch of the Cloud Native cons and Kube cons and those kinds of shows, and I spend my life at IT conferences. I’m telling you that the open source community—and not because of the idealistic ____ and the bizarre and free as in freedom and free as in beer kind of crap.

    We’re past that, you know, Dr. Stallman stage, if you will. It is the, that community is the energy driving almost everything we’re seeing in IT today, right? There’s such an energy in there that it’s driving what you’re doing at Cockroach, for sure. It’s driving what we do in DevOps. It’s driving what we see in Cloud Native and Kubernetes and all of these things. It’s where—it’s almost like this well of, like, a nuclear power plant or something, you know what I mean?

    Kimball: Yeah, absolutely. And, you know, I think the best model right now that I’m aware of is, you know, it’s called cloud-based services with a very fast time to value, incremental pricing, right? So, you kinda pay for what you use as you go. If you can deliver that with an open source product that’s backing it that people feel that they can get into and they can use if they had to move off that sort of freedom and flexibility, that sort of offers the best of all worlds.

    And I think that—and that’s why Amazon’s been so successful in adopting open source technologies, because they’re very good at providing things as a service and they have so many things altogether into one, you know, that can be packaged into one bill, one sort of monitoring platform, one identity access, you know, management platform. It’s very impressive, but, like that, in aggregate, creates so much value for business.

    Shimel: It’s crazy. We’re running low on time, so let us bring this back to Cockroach Labs. We seem to have taken off there on a tangent. You guys are offering a managed version.

    Kimball: That’s right. So, we have—some of our biggest customers, they wanna run the database themselves, it’s the way they’ve been running it, but over time, that will shift, of course. So, we have a self-service, self-hosted version where you buy subscription licenses if you wanna pay us for it. There’s also, of course, open core, which you can use for whatever purpose.

    Then there’s the managed service offering, which we’re calling Cockroach Cloud. And that, we run it for you, and we can run that on AWS or GCP, and the pricing model is much more like something like DynamoDB or Amazon Aurora or MongoDB Atlas, those are sort of similar kinds of offerings.

    Shimel: Got it. Excellent, man. Just, again, Spencer, for people who may not be familiar, the website is CockroachLabs.com.

    Kimball: That’s right.

    Shimel: So, that’s where people can go get more information, there. Any other news that we didn’t hit on, here?

    Kimball: Well, you know, I think maybe we didn’t quite cover the funding well enough. I kinda jumped ahead, but we’re raising more than 15 million and the lead investor is Altimeter Capital.

    Shimel: Great.

    Kimball: So, there’s sort of growth equity added, along with Tiger Global, and then all of our existing investors, Google Ventures—GV is what they’re called now—Benchmark, FirstMark, Redpoint, Index Ventures. So, it’s a great—

    Shimel: Impressive list.

    Kimball: Yeah, a great group of investors. We’re looking forward to sort of this being an inflection point and really moving forward with our penetration into the Global 2000 and also fundamentally putting a lot of focus on managed services, so, the cloud delivered Cockroach experience, in addition to enabling more and more of this global architecture capability that I was talking about earlier.

    Shimel: Absolutely. I mean, it takes resources today to make all of that happen and bring it all to market, obviously, and hiring good people is not—even with the money, it’s not an easy thing to do, as I’m sure you’re finding out. Finding good people is—

    Kimball: A lot of these guys have—I liked what you said, at sort of the top of the food chain.

    Shimel: Yep, the alpha predator in the thing. It’s crazy.

    Kimball: They vote with their feet, right? So, it’s—

    Shimel: Yeah, they do. I mean, they don’t even ask for raises, they just go get another job. It’s not—I mean, it’s a crazy, crazy dynamic at play there, but it is what it is.

    Well, listen, this was—obviously, we blew past our 15 minute time, but it happens, it’s all good. I want to wish you success. I mean, just a quick digression from the point—you know, there’s a lot of people out here who are gonna listen to this, Spencer, who will say, “You know, I have a great idea for a company. We should go raise a few million bucks in capital, and we can do similar to what Cockroach is doing.” Not necessarily in the DB perhaps, but you know, in startup world, it’s not easy—$100,000,000.00 is a lot of frickin’ money for people to write checks to you for. And it doesn’t come without tremendous responsibility, and having raised a lot of money in my lifetime for venture backed companies, I know what goes into it.

    So, congratulations on a successful run with this, and looking forward to seeing what you do with it.

    Kimball: I appreciate that, Alan.

    Shimel: No problem. Alright. Our guest was Spencer Kimball, CEO, co-founder of Cockroach Labs, makers of the CockroachDB, and this is Alan Shimel for devopsy.kinsta.cloud. You’ve just listened to another DevOps Chat.

    — Alan Shimel

  • Rethinking Data Management in a DevOps Environment

    Rethinking Data Management in a DevOps Environment

    You often hear data is the new oil. This valuable, ever-changing commodity has begun to play a starring role in many cloud-native applications. Yet, according to a number of DevOps teams, data issues continue to plague their efforts to continuously integrate, test and deploy frequent software releases. More specifically, issues with persistent data (and its underlying database engine) often appear to be the culprit.

    Your organization might be pursuing emerging IoT applications that incorporate and analyze sensor data from multiple sources. Or, your DevOps team may be trying to develop applications that extract further actionable insight about customers. Whatever the use case, there’s no doubt back-end database architectures have gained growing importance to the success of such projects. Yet, in many cases, such database systems appear to be having difficulty just keeping up with the pace of today’s DevOps pipelines.

    According to one report on the state of database deployments in 2019, 46% of DevOps teams on an accelerated release schedule (with weekly, or even daily, releases) found it “extremely or very difficult” to speed their database release cycles accordingly. In a related Redgate report on the 2019 State of Database DevOps, 20% of respondents cited slow development and release cycles as one of the biggest drawbacks to “traditional siloed database development practices.” Another 23% saw a higher risk of deployment failure and extra downtime when database changes were introduced in a traditional database environment.

    Getting to the Heart of the Data Problem

    Are such database issues the result of the wrong choice of underlying database technology–such as the use of an RDBMS, NoSQL or even a NewSQL system? Possibly.

    Maybe database issues are also caused by poor (or non-existent) cross-communication between database experts and their developer counterparts? Undoubtedly, this is true as well.

    Would you be surprised to learn both situations (wrong technology and poor communication) are often to blame? Also, to blame is the urge to solve new data problems in the same old way organizations approached their earlier, legacy use cases.

    Ultimately, as with many things DevOps-related, the answer to the data problem is likely to require change along three, separate fronts: People, process and technology. It will also require a fundamental shift in how such issues are approached at the start.

    Starting with People and Process

    Instead of worrying about database slowdowns, what if you began again with how DevOps teams first approach the many facets of the underlying data layer? This means:

    1. From the start of a project, include DBAs as part of the cross-functional DevOps team. This will help promote healthy cross-communication. They will also positively influence the development of appropriate underlying data layers and infrastructure that support your emerging use cases. In larger organizations that manage many data sets, these may be specialized database administrators (DBAs). In smaller companies, these may be more full-stack engineers who also have high-quality expertise in database operations.
    2. Identify early the different data types, domains, boundaries and optimal cloud-native patterns associated with your data. Once these are established, the team can gain a better understanding about the different ways to develop applications and data architectures for each new use case.

    In effect, organizations need to make a concerted effort to shift left with their overall data architecture discussions. This shift allows more of the right questions about data to be asked, answered and incorporated early on in the design of the overall application.

    Asking the Right Questions About Data

    Part of this early collaborative work may involve asking key questions about data. These include the best ways for an application and its users to interact with the underlying data, such as:

    • How will data be ingested and from where?
    • How will we access the data?
    • How and where will we store it?
    • How do we update and/or maintain legacy databases as we build for new use cases?
    • What types of data will we have? How will the types change or be increased over time?
    • How will we query from it? And what are the most common queries users will want to run?
    • How will we manage and scale the data in use?
    • How will we protect the data from data loss, disaster or corruption?
    • How will we secure the data?

    If these questions sound a lot like features in a data lifecycle, you’re right. But, such basic questions about data are often overlooked in the rush to deliver, integrate, test and deploy application code. So often, what happens instead is an application team creates a design, then hands it off saying, “Okay, now create a data architecture to support the application.”

    This fundamental miss with data and collaboration at the start often leads to poor database architecture decisions later. It can also cause DevOps teams to resort to workarounds to accommodate such wrong architecture choices made at the start–which may cause some development teams to choose to circumvent DBA involvement altogether. In this case, they might even find themselves opting to use public cloud services instead (or some other types of data management) to support their application.

    Instead, application teams should choose the path of early collaboration between developers and all functions of IT (including DBAs). This approach can offer better DevOps outcomes. It also helps move organizations toward a future where data architects provide an effective bridge between all parties critical to the application’s success: From the needs of developers to the platforms provided by DBAs and the underlying infrastructures needed to enable them.

    Changing Skillsets, Changing Mindsets

    As with most things DevOps-related, success at the data layer has as much to do with changing the education, mindset and culture toward database architectures and database deployment methods as it does with the specific steps in the DevOps pipeline.

    For example, if most DBAs in your organization are narrowly focused in one area (such as an Oracle DBA or SQL Server admins), your organization may benefit from investing in education to close skill gaps. Such investment can help expand the knowledge and exposure of that workforce to DevOps practices or emerging database approaches to cloud-native application patterns.

    It can also help some organizations to first identify those individuals in the organization who want to transform, innovate and learn about new tools, methods and practices.

    Hiring those with that expertise already in place is another option. When you hire new skillsets, however, the answer may not be just to hire several NoSQL administrators. This tactic will not necessarily solve your data problem, either. It may be just as beneficial (or more so) to hire someone with little specific NoSQL skills but who is good at thinking about data in a different way.

    Business First, Technology Second

    In our practice, we are often asked to weigh in on technology choices to support DevOps efforts in emerging areas such as IoT, Big Data and advanced analytics. Organizations ask us about using traditional relational databases (SQL Server, Oracle, etc.) versus NoSQL databases (MongoDB, Cassandra, etc.). They ask about the merits of one NoSQL iteration over another. They ask about how to manage persistent data in container environments. They even ask about workarounds when the data choices they’ve made cause other unexpected problems.

    We do our best to answer these questions. But, whenever possible, we also tell organizations to back up a few steps and start, instead, from their business objectives:

    • What are you trying to accomplish?
    • What are your business drivers?
    • How do you hope data will be used in these contexts?

    Answers to these questions can help you make better choices about the best data architecture to support your growing applications.

    Ultimately, new tools and technologies can enable a lot of impressive data architectures. But, just throwing tools at the data problem is not enough. Early effort and investment in people, process and cross-departmental communication is just as important to a successful data outcome for any project.

    Many of the discussions we have now are trying to take everyone back to the steps they skipped. This harks back to the fundamentals of software engineering: Addressing things early so you don’t have problems later.

    Unfortunately, data is the one component in the technology stack that you can’t easily undo once you’ve chosen a specific path with your application’s data architecture. When it comes to planning for data, it really is a pay now or pay later equation.

    Isn’t it best to take the time and choose your data path wisely rather than paying later for slowdowns or other issues with deployment and performance?

    — Jeff Bozic

  • The Linux Foundation Breathes New Life into Osquery

    The Linux Foundation Breathes New Life into Osquery

    Facebook’s useful but neglected DevOps tool Osquery has gotten a new lease on life, thanks to The Linux Foundation.  

    Anyone who has been tasked with monitoring the security of server instances in a data center or cloud knows how laborious and time-consuming it can be. Osquery, a project started by Facebook, aims to lessen this burden by reframing how developers engage with their infrastructures. DevOps professionals can use Osquery to expose an operating system as a high-performance relational database, making it possible to use SQL commands to access data about a system, just as they would for a database. 

    Osquery works on Mac, Linux and Windows systems and is provided as an open source download via GitHub. Although Osquery was developed by Facebook to monitor and safeguard the security of its own platform, the social media giant quickly realized the utility of the platform would extend to other enterprises that depend upon insight into the low-level behavior of operating systems. 

    Facebook publicly released the project’s code and documentation in 2014, in response to feedback from a small number of other companies that tested Osquery. The public availability of Osquery resulted in a number of major companies including Airbnb, Dropbox, Netflix, Etsy and Uber to bring Osquery into their software development environments. Nevertheless, users have voiced frustrations with Facebook’s handling of Osquery, and have accused the internet giant of neglecting the project.

    To address those complaints, The Linux Foundation has taken over the mantel of Osquery and recently announced the formation of a new foundation to support the Osquery community. 

    The new Osquery Foundation brings together engineers and developers from Dactiv, Facebook, Kolide, Trail of Bits, Uptycs and many others committed to supporting the project with technical contributions and long-term stewardship. A Technical Advisory Board will be created to handle the shift in management from Facebook to the Osquery Foundation and will oversee the priorities initiated by the foundation’s members.

    This restructuring of management promises to extend the usability of the project as a number of developers contribute to the codebase.

    “Osquery has drastically simplified the process of operating system monitoring, which has unlocked new methods for securing infrastructure, detecting anomalies and more. We look forward to working with the Osquery community to develop the foundation in a manner that will continue to foster the growth and adoption of Osquery while supporting the community’s diverse needs,” said Jim Zemlin, executive director at the Linux Foundation.

    Developers working to maintain the security of their systems believe it would be wise to incorporate Osquery into their workflow. “Although Osquery will never do everything that a security team needs, it gets about 80% of what you need for endpoint insight in one package,” said Doug Wilson, director of security for Uptycs.

    Although the platform requires some configuration, developers familiar with SQL will feel accustomed to using Osquery to retrieve information about their systems. Furthermore, Osquery’s cross-platform support and customizability make it especially attractive to businesses with newer types of infrastructures and ones that work at scale. 

    The move away from Facebook toward a more community-friendly foundation is sure to breathe new life into the project, and as development on Osquery expands, it is likely to become a more valuable resource for security teams moving forward.

    This article was co-authored by Tyler Ohlhorst, research analyst and freelance copy editor and copywriter at Magnum Consulting.

    — Frank Ohlhorst

  • ScyllaDB Announces Major Release of Its NoSQL Database and Support for Concurrent OLTP and OLAP

    Scylla Open Source 3.0 Brings Production-Ready Materialized Views, Secondary Indexes, Cassandra 3.0 Compatibility and More
     
    PALO ALTO, Calif., Nov. 6, 2018 — ScyllaDB, the real-time big data database company, kicked off its two-day Scylla Summit by announcing a major release of its database, Scylla Open Source 3.0. The company also previewed Scylla support for concurrent OLTP and OLAP, an industry first that enables simultaneous transactional and analytical processing. This release marks a significant milestone, as the company surpasses feature parity with Apache Cassandra, the popular open source NoSQL database. As a drop-in replacement for Apache Cassandra with as much as 10 times better performance, Scylla Open Source has added production-ready capabilities much anticipated by the NoSQL community, while introducing ground-breaking new features that improve utilization and minimize operational overhead. 
     
    Scylla Open Source 3.0 features a close-to-the-hardware design that makes optimal use of modern servers. Written from the ground-up in C++ to provide significant improvements to throughput, latency and administration, Scylla delivers scale-up performance of more than 1,000,000 IOPS per node, scales out to hundreds of nodes, and consistently achieves a 99% tail latency of less than 1 millisecond.
     
    New Scylla Capabilities Announced Today
    • Concurrent OLTP and OLAP: Scylla will enable its users to safely balance real-time operational workloads with big-data analytical workloads within a single database cluster. Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) have very different data access characteristics. OLTP involves many small and varied transactions, including mixed writes, updates, and reads, with a high sensitivity to latency. In contrast, OLAP emphasizes the throughput of broad scans across datasets. By introducing capabilities that isolate workloads, Scylla will uniquely support simultaneous OLTP and OLAP workloads without sacrificing latency or throughput.
    • Materialized Views: Production-ready release of a long-awaited experimental feature designed to enable automated server-side table denormalization. Notably, the Apache Cassandra community reverted this feature from production-ready to experimental mode in 2017.  
    • Secondary Indexes: Production-ready with its 3.0 release, Scylla’s global secondary indexes can scale to any cluster size, unlike the counter local-indexing approach adopted by Apache Cassandra. Secondary indexes allow querying data through non-primary key columns. 
    • Cassandra 3.x File Format Compatibility: Scylla Open Source 3.0 includes Apache Cassandra 3.x compatible storage format (SSTable), improving performance and reducing storage volume by as much as 3x.
    Speaking at Scylla Summit 2018, CEO and co-founder of ScyllaDB, Dor Laor said, 
    “Scylla 3 represents a huge leap forward in functionality and scalability. We finally reached parity with Apache Cassandra, and developed a ground-breaking OLTP+OLAP service level agreement guarantee that puts us on a path towards pure multi-tenancy and positions us favorably against Amazon DynamoDB, Microsoft’s Cosmos DB among others.”
    AVAILABILITY
    Scylla Open Source 3.0 will be available in November 2018, with concurrent OLTP and OLAP support available shortly after. Existing users are encouraged to upgrade to the new version in order to take full advantage of performance gains and new capabilities. 
     
    To see the advantages of Scylla:
     
    ABOUT SCYLLADB 
    ScyllaDB is the real-time big data database company. Fully compatible with Apache Cassandra, Scylla embraces a shared-nothing approach that increases throughput and storage capacity to 10X that of Cassandra. AppNexus, Samsung, Ola Cabs, CERN, Grab, Investing.com, Snapfish, IBM’s Compose and many more leading companies have adopted Scylla to realize order-of-magnitude performance improvements and reduce hardware costs. ScyllaDB was founded by the team responsible for the KVM hypervisor and is backed by Bessemer Venture Partners, Innovation Endeavors, Wing Venture Capital, Qualcomm Ventures, Magma Venture Partners, Western Digital Capital, Samsung Ventures, and TLV Partners. 
     
    For more information, please visit: scylladb.com
  • Why NoSQL is the New Database Darling of DevOps

    Why NoSQL is the New Database Darling of DevOps

    If developers are from Mars and testers are from Venus, your database administrator is stuck in Saturn. By nature, the database—and specifically the data it carries—has to remain stable even as the application code on top of it is changing faster than Donald Trump’s staff.

    When it comes to the speed of your release cycle, issues with your database are like potholes. You don’t see them up ahead, and you only feel the pain after the damage is already done. The least flexible layer in an application is its database, rendering it the weakest link in any agile development process.

    Get Your Database Up to Speed

    The right NoSQL database offers the following features that can speed up your release cycle without falling into a pothole.

    It’s Schema-less:  When you make a new build, or issue a new release, your code will change. Obviously, it will go through QA and check out. However, just because it’s working doesn’t mean it’s working correctly. You may have changed the types of data your application is accepting. You may have changed the way your app works with data. Your schema is the blueprint for how you accept data. For a relational database, you need one. The more complicated the schema, the harder it is to change.

    A NoSQL document database is schema-less, offering you the ultimate flexibility and speed in how to rearrange the way you handle your application’s data.

    Rapid Scalability: The moment your application is a success, expect greater load. Greater load requires more hardware to handle that load while maintaining peak performance. A relational database demands you scale up, investing large amounts in a new server. NoSQL solutions let you scale out, creating data clusters on commodity hardware.

    NoSQL lets you add new nodes to your database topography in minutes, expanding your ability to process requests from users at optimum performance.

    Immediate Adaptability: New code doesn’t only change the type of data you are working with; it also can change how you work with the data itself. New queries mean whenever a user clicks a button, your database will have to relearn how to sort your data to answer new requests for specific items or aggregated totals.

    A relational database will join tables and scour the data row by row to get your users what they want. If the whole thing changes, the process starts from square one. A good NoSQL solution uses dynamic indexing. It means that it won’t perform a query until after it finds the right index. It will either use a pre-existing index, improve on one or create one from scratch that will be made available for future queries.

    Such a system reduces the performance hiccup by releasing new code in that the more you use it, the better indexes are created to maximize the performance of the database. Your Agile DevOps team can release new builds every day and the database will be able to keep up.

    Platform testing: Agile without testing is like a lunchbox without the apple. It may not be the most appealing part of the meal, but you gotta have it. The right database lets you make queries in test mode without touching the disk itself. The data in real-time is separated from what you are testing.

    Due to its inherent flexibility, a NoSQL database lets you re-create whatever test scenarios you need and measure function and performance to see how every layer is working under the new changes. You can also do this fast, keeping your development process moving right until the end.

    Technical support that doesn’t hold you back: One of the biggest pitfalls in Agile is when you hand the fate of your next release over to people who have no idea how urgent it is that you get the issue resolved now. Bad tech support can take an issue that shouldn’t slow you down and freeze you up for hours, even days.

    The right NoSQL solution offers tech support with quick response time, and engineers who have experience developing the database itself. It is important that they know how the database works, and can do better by you than reciting a series of standard test scripts.

    Databases can also notify you of underperformance or any indications that there are escalating problems that need immediate attention before they become full-fledged issues. Small features such as these save you time in having to resort to tech support in the first place.

    Conclusion

    There are always risks involved with making changes to your application. The risk databases pose is that some changes can shut down your database and paralyze your application. A NoSQL solution can reduce this risk, and even speed up areas of your DevOps process.

    — David Ben Horin

  • NoSQL Databases: Lessons Learned Growing a Globally Focused Startup

    NoSQL Databases: Lessons Learned Growing a Globally Focused Startup

    Startups deploying NoSQL databases and global citizens learning new languages actually share something in common: Both want to remove the limitations on how they access information and express themselves. In launching and growing a platform that enables anyone around the world to connect and speak with native English-speaking tutors on-demand, we’ve had the fortune of helping countless people learn new language skills—and we’ve also learned a great deal ourselves about the best ways to handle the requisite user and application data.

    NoSQL, No Problem

    Here are four NoSQL database strategy takeaways we learned from recent experience:

    Keep Your Data Options Open with Flexible Technology

    When a startup is just out of the gate, it isn’t immediately clear what shape the final product may take, or what data it will require.

    In our case, the relatively broad scope of our project also defined the need for a database able to function globally with 24/7 availability. The concept for Cambly was born out of its founders’ real experiences traveling to other countries and learning foreign languages through conversations with native speakers. Immersive experiences such as these are critical for gaining and retaining conversational language abilities, and provide learning opportunities that no language course or software can really match. However, these language skills can recede rapidly if a person no longer has access to native speakers to practice with. Seeing this as an issue that technology could address, we began building an application to provide global on-demand access to immersive conversations with native speakers, initially concentrating on English as the largest language market.

    As a startup beginning to explore how exactly we’d accomplish the task ahead of us, it was critical that we could store whatever types of data we found to be of value as we quickly iterated the application. This meant utilizing a NoSQL database solution to free ourselves from any schema constraints. We selected MongoDB as a flexible, NoSQL solution with which our technical team had some existing familiarity.

    Get Help …

    Startup success is dependent on managing finite internal resources in a way that maximizes the attention that can be devoted to product development and other growth-critical initiatives. This means having the wisdom to rely on managed services where they’re beneficial in saving time and simplifying operations, versus building up internal headcount. For example, we enlisted Heroku to use as our application platform. And, even though our team was experienced in running databases and managing machines on AWS, we still saw the value in keeping production as hands-off as possible and opted for a database-as-a-service (DBaaS) strategy as well.

    … And Make Sure It’s the Right Help

    It can be tricky, but ensuring that any and all technology service providers a startup works with are a strong fit for its needs is vital to moving quickly. We actually learned this the hard way when it came to our original MongoDB DBaaS. For us, reliable 24/7 availability and insights into database performance were essential requirements. Unfortunately, our initial provider presented us with two challenges to meeting these requirements in a short time frame: Its tools became unable to provide us with visibility into our database, and it announced a required platform migration that could not be completed without downtime. Naturally, these changes led us to seek a better-fitting option. Soon after, we partnered with mLab in large part due to rolling node replacement capabilities for migrations and 24/7 uptime, as well as support and tooling that could help our team quickly optimize MongoDB instances.

    Get the Database Tools You Need to Succeed

    From reliable data backups to performance insights and beyond, the right toolset can make or break a burgeoning company’s success. Just as language skills must be constantly attended to, your database requires intelligent upkeep and backups to ensure it delivers a satisfactory user experience and remains available in case of disaster. This is one lesson which—very fortunately—we were able to absorb without hardship. We had an incident in which our team accidentally deleted a trove of valuable data. But, with data backups in place, our DBaaS was able to help us restore that data quickly—and without us stressing over it as we would have otherwise. At the same time, the DBaaS tools allow us to recognize and address database issues while improving indexing and performance.

    — Gar Higgins

  • Quest Software Aims to Mend DevOps Fences for DBAs

    Quest Software Aims to Mend DevOps Fences for DBAs

    It’s no secret there’s been a lot of contention between application developers, who are under pressure to build applications faster, and database administrators (DBAs), who are charged with providing those developers with access to data. Historically, developers have been dependent on DBAs to create schemas that would allow them to access data. But it takes time to build schemas, which is why many developers as of late have opted to employ a NoSQL database that don’t require a schema to access data. While that provides developers with some much-needed flexibility, most of the data stored in an enterprise today is still inside a relational database.

    To make it possible for DBAs to respond faster to developer demands, Quest Software has unveiled the Toad for DevOps toolkit, which makes it easier to include relational databases such as Oracle within a larger continuous integration/continuous development (CI/CD) process managed by frameworks such as Jenkins, Bamboo and Maven.

    Greg Davoll, executive director for product management and marketing for Quest database management solutions, notes the DBA traditionally has been the last holdout in embracing DevOps—DBAs didn’t really have access to tools that would enable them to participate in an agile development environment. The Toad for DevOps Toolkit address that issue by enabling them to test PL and SQL code during the application build process and ensure schema, data and database integrity with the context of a DevOps pipeline process, Davoll says, adding DBAs participating in a DevOps process can now also perform static code reviews and promote artifacts into a target environment to see how they perform in a production environment.

    The arrival of Toad for DevOps Toolkit is scheduled to coincide with the release of Toad for Oracle 2017 R2, which will add pass/fail status notifications within the Code Analysis code review process, a redesigned multi-schema compare and sync function, and an enhanced REST API as part of the Benchmark Factory testing tool to help automate performance testing within a CI/CD workflow. Over time, Quest Software will add similar capabilities across the full range of databases it supports, Davoll says. Quest Software just announced a Toad for MySQL offering to complement its existing support for Oracle and Postgres databases.

    Many of the decisions over the years concerning what database to employ have come down to which platform incurs the least amount of friction for developers to deploy. That hasn’t always resulted in the right database decision being made. In fact, it’s usually not too long before a developer wants to offload the management of the database back to the IT organization. Invariably, that results in DBAs having to manage multiple database platforms. While there may be a perfectly rational reason to employ something other than a relational database, that decision should be based on the attributes of the database rather than internal IT politics and purchasing processes. The challenge facing DBAs now is how to re-engage developers who have developed a bias against specific classes of databases, no matter how much critical data they may hold.

    — Mike Vizard

  • Backup and Recovery for Distributed Databases

    Backup and Recovery for Distributed Databases

    Over the past five years, applications have undergone a paradigm shift. They now require more agility, scalability and availability. The key trends in this era are driven by the use of social platforms, mobile devices and software as a service (SaaS).

    Being able to ingest large amounts of data from various sources and process that data in real-time to deliver contextual information or business insights is a major competitive advantage. To meet these new application requirements, organizations are forced to look beyond traditional relational databases. This has resulted in a slew of new database systems that are distributed and scale-out in nature, can be deployed on commodity hardware and offer tunable consistency-versus-performance trade-offs. It’s important to note that to cater to the agility requirements, database-as-a-service model in cloud is being adopted as well.

    To show just how big the shift to distributed databases is, below is a Keyword Search Trend graph showing the sharp rise in NoSQL search.

    screen-shot-2016-09-27-at-2-03-59-pmThe implication of this is that requirements for data protection have changed forever, driven by this data-centric world (web-scale applications and distributed databases). Organizations are collecting vast amounts of data and seek to drive more value from that data to make better and faster business decisions. Most distributed and cloud databases already provide replication capability and thus satisfy the availability requirements of data protection. However, data protection requirements for scalable point-in-time backup and recovery need to be addressed. Without point-in-time backups, organizations are at substantial risk of losing data due to human error, logical corruption and other operational failures. Traditional backup solutions were built to address the requirements of relational databases that used shared storage and had the ACID transaction model. Unfortunately, they fall well short in addressing the point-in-time backup requirements of distributed databases (local storage, eventual consistency and the elastic nature of infrastructure).

    As database architecture has fundamentally changed to meet new application requirements, data protection needs to be redefined and rearchitected as well. The following are the new requirements for data protection.

    • Take a durable point-in-time backup copy of an eventually consistent database; we call it “versioning” the new paradigm of data protection for this new distributed era: backups to snapshots to replication to copy data management to versioning
    • Minimize time to recover from failures (low RTO)
    • Scale with the growing needs of application
    • Enable easy refresh of test/dev environments for continuous development
    • Provide operational resiliency in event of failures
    • Provide deployment flexibility in public cloud or on-premises data center

    Most organizations are investing in enterprise-grade point-in-time backup and recovery products so they may deploy and scale their next-generation application on distributed databases with confidence. In the next five years, it will be exciting to watch companies redefine the data protection technologies to meet the requirements of next-generation applications.

    — Jeannie Liou