Tag: databases

  • Hasura SDK Integrates GraphQL Platform With More Data Sources

    Hasura SDK Integrates GraphQL Platform With More Data Sources

    At its HasuraCon ’22 conference, Hasura today announced the early release of a software development kit (SDK) that extends the reach of its GraphQL platform to additional data sources.

    In addition, Hasura is making generally available integrations with the GitHub repository, support for OpenTelemetry Traces within the Hasura Cloud Platform and Microsoft SQL event triggers.

    Finally, Hasura Cloud can now be deployed on the Google Cloud Platform while the Hasura Cloud Enterprise edition is now available on the Amazon Web Services (AWS) marketplace.

    Hasura CEO Tanmai Gopal said that while the company already provides support for widely used data sources such as the Postgres database, there are many data sources that organizations also want to be able to query via GraphQL application programming interfaces (APIs). Organizations want to be able to join data using GraphQL regardless of what repository it resides in, he added.

    The open source Hasura engine at the core of the company’s offerings automatically generate a GraphQL schema from a data source. That can then be used to accelerate the development of APIs to make it easier for a wider range of developers to build applications that aggregate data across multiple sources.

    That approach eliminates the backend complexity that IT teams encounter when adding GraphQL APIs to an application environment that already has a range of APIs for internal IT teams to support, noted Gopal.

    GraphQL was originally created by Facebook, but it’s still not clear to what degree GraphQL APIs will supplant REST APIs. Developers tend to prefer GraphQL APIs because they provide more granular control over what data is accessed. The challenge is the number of backend services that expose GraphQL APIs is still comparatively limited.

    It’s not likely IT teams will replace REST APIs with GraphQL-based APIs overnight, but the percentage of new applications that rely on GraphQL APIs will steadily increase. Overall, the number of APIs employed within IT environments is rapidly expanding as more microservices-based applications are deployed. Each microservice generates its own API. Over time, IT teams will find themselves managing a wider variety of types of APIs.

    The challenge is that it’s not always clear who within IT organizations will manage those APIs. They are often created by developers and are left to IT operations teams to maintain. It’s not uncommon for IT teams to find themselves managing hundreds, perhaps thousands of APIs.

    In addition, many of those organizations are now consuming data via APIs exposed by other organizations. Over time, the level of interdependency between APIs and applications will serve to make application environments much more complex to manage.

    In the meantime, DevOps teams will need to extend existing DevOps workflows to both support and secure GraphQL-based APIs. The amount of sensitive data being accessed via those APIs is likely to accelerate the rate at which organizations are adopting DevSecOps best practices. Let’s hope this happens before they wake up to discover GraphQL APIs have already proliferated across the enterprise.

  • Data Monoliths Go Cloud-Native with Kubernetes

    Data Monoliths Go Cloud-Native with Kubernetes

    Breaking monolithic applications into independently scalable services is the core of the cloud-native application model. By making cloud-native apps easy to deploy and manage, Kubernetes has contributed to a big change in IT practices.

    Initial cloud-native conquests started with stateless services and quickly grew to encompass applications that manage data. Database management systems have long relied on centralized servers, but cloud-native architectures are now reaching even the largest monoliths. Among the most interesting cloud-native converts are analytic databases, which hold data arranged in storage formats optimized for extremely fast analytic queries over billions ,or even trillions, of rows.

    The transformation of databases to cloud-native models has profound implications for data-driven applications. Intersecting trends in analytic database architecture and Kubernetes promise substantial benefits to the users of such applications as well as companies that operate them.

    The Rising Importance of Analytics

    Over the last decade the quantity of data that businesses generate has decoupled from the size of the business itself. Ad networks, website analytics and service observability applications often ingest 1 million events per second (or far more) and may have tables running into trillions of records. Businesses need to analyze this information rapidly–for example, to diagnose faults in cloud applications or to retrain models for displaying online ads. Flexible, real-time analytics have gone from nice-to-have to business critical.

    Meanwhile, new analytic engines have emerged that eliminate dependencies on specialized hardware and complex networking required by legacy data warehouses. Druid, ClickHouse or Elasticsearch run on commodity hardware and depend on basic compute, network and storage. Everything else, such as high availability (HA), is built into the application. Newer analytic databases offer built-in horizontal scaling using fleets of server processes. The server processes themselves typically run well in containers.

    With these changes, analytic databases are increasingly cloud-friendly if not cloud-native. It is a short hop to deployment on Kubernetes.

    Kubernetes Operators for Databases

    At the same time, as analytic databases have shifted their operational model, Kubernetes itself has become more welcoming for databases.

    One particularly important innovation for the database is custom resource definitions, or CRDs, which extend the Kubernetes API to define new types of manageable resources. Using CRDs makes it possible to have a single definition that shows the desired state of a complex application such as a data warehouse.

    Another addition is Kubernetes operators, which manage custom resource definitions. Operators track custom resource definitions and take actions when they change. They can also track other information that might be relevant to managing the custom resource, such as changes to pods, storage or even related cloud-native applications. It’s like having a human operator who is constantly watching the application, except the rules are programmed into software that runs inside Kubernetes itself.

    The operator pattern is a perfect match for analytic databases, which are complex distributed applications and may require definition of dozens or even hundreds of individual Kubernetes resources in a running system. Operators encapsulate these details as well as important procedures such as upgrade, scale-out and other operations that require careful manipulation of resources to prevent outages.

    It’s not surprising operators are popular with database developers. The OperatorHub.io registry for database operators alone has 19 entries covering databases of every kind, including analytic databases such as ClickHouse and Elastic. Many more exist outside the registry.

    A Cloud-Native Way to Consume Analytic Databases

    New data warehouse architectures and Kubernetes operators means users can now use analytic databases very differently from legacy data warehouses. Our work on ClickHouse, including development of the ClickHouse Kubernetes Operator, prompts two observations.

    First, operators make spinning up analytic databases simple and fast for all users. Amazon Redshift kicked open the door to data warehouse-as-a-service in the public cloud. Kubernetes operators enable similar capabilities not just in public clouds but any Kubernetes cluster. Individual developers can cycle through dozens of configurations in a single day while testing new analytic applications. Thanks to quick pod spin-up new analytic databases can be running in minutes or even less.

    More profoundly, users can use Kubernetes to move to an on-demand model for analytics. Rather than forcing all applications to compete for resources on a single shared cluster, each application can have an analytic database tuned to its specific requirements. Analytic databases can integrate tightly with applications in the same locations where those applications already run. Businesses can scale databases flexibly to deliver quality analytics at exactly the time they are needed and at reasonable cost. This is the heart of the cloud-native programming model.

    Modern database architectures combined with Kubernetes features, such as the operator pattern, are creating a new path to manage enormous volumes of data. For readers who are hesitant about managing data on Kubernetes, it’s time for another look.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Robert Hodges

  • DevOps Chat: Scaling Data Without Complexity with Cockroach Labs

    DevOps Chat: Scaling Data Without Complexity with Cockroach Labs

    Cockroach Labs the folks behind CockroachDB recently announced another round of funding as they continue their mission to allow you to scale your data without complexity.

    In this DevOps Chat, CEO of Cockroach Labs Spencer Kimball talks about about their plans and how Cockroach shall inherit the DB world.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel from devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. On DevOps Chat today, I’m really happy to be joined by Spencer Kimball. Spencer is CEO/co-founder of Cockroach Labs—hot company in the DB and DevOps space. Spencer, welcome.

    Spencer Kimball: Thank you, Alan. It’s a pleasure to be here.

    Shimel: Absolutely. So, Spencer, let’s get it out of the way. First, I’m assuming most of our listeners have already heard of Cockroach Labs, but it’s possible they haven’t. Why don’t you give us a little background?

    Kimball: Well, sure. I mean, first, we should probably get the obvious question out of the way—why would we name something Cockroach? I think that’ll become clear when I give you the explanation, but Cockroach Labs was founded to build a database, and that database replicates and it’s hard to kill, so that’s where the name came from. Honestly, the name was invented before the product and long before it was a company. So, it was—you know, now I’m in a position of explaining that same thing to Fortune 500 CIOs on a weekly basis. [Laughter] There’s some regret there, but the name makes sense.

    So, I can give you a little bit of the background. All three co-founders of Cockroach Labs worked at Google for about 10 years in the office and Google went through Bigtable which started the NoSQL movement, and it quickly moved to Megastore, which introduced some transactionality. Then they created Spanner, and Spanner is something that Google’s been working on for more than a decade now.

    When we left Google, we realized Spanner is a solution to many companies’ problems, not just Google’s. And that’s become more and more true in the years since. We did a startup, and even as a startup, we wanted capabilities that Google had built into Spanner, and those weren’t available in open source. So, that’s sort of where the original idea of Cockroach came from. That startup got acquired by Square, the payments processing company, and many other things now as well. And Square needed something like Spanner as well.

    So, the idea kind of had gained currency. We started as a GitHub project. That took off, VCs got interested. The company was started, now, four and a half years later, we’re busy selling to the Global 2000, Fortune 500 big companies out there as well as competing on one side with Oracle and on the other side with Amazon.

    Shimel: Only on the Internet, man, you know? [Laughter] It’s a crazy story, but true. So, first of all, I absolutely get the Cockroach thing, because they are gonna inherit the earth, you know, or so it seems, especially with the path we’re heading, but let’s not get into politics.

    But it is kind of an amazing story, because when—for people who are in the IT business and are somewhat familiar with the history of the dashboard and how Larry Ellison came to be the second richest man in the world or whatever it is he is these days, you know, it’s fascinating that all of everything we do online, almost, is predicated on the ability to store the data and to access that data in ways that allow us to do all of, you know, manipulate the data, analyze the data. I mean, without databases, none of that happens.

    And so, when you look at the history of the database and then the, you know, to relational databases and then you mentioned a few other things, SQL database and then NoSQL databases—

    Kimball: And back to SQL. [Laughter]

    Shimel: Right, and then back to SQL, then you’ve got time stamped databases are a thing now. You know, all revolving around, though, how do we store this data in a way that’s easy to put the data in there and easy for us to access that data, manipulate it, analyze it, you know, and make value out of it, right?

    And so, much like a cockroach, data doesn’t die, either, right? [Laughter] It’s eating the world, you know? Where was I recently, I was shopping at Costco or something, and they had eight terabyte, I guess, backup or drives, you know, external drives that were maybe a little bigger than credit cards, you know? Size of a good iPhone, let’s say, a little thicker. And I think it was $169.00 for eight terabytes. [Laughter] And I’m thinking, “My God, you know, when I had those 3.5 inch floppy disks, man, and the bigger ones”—

    Kimball: And the first 20 megabyte hard drive, it was—

    Shimel: Yeah! I didn’t think there were eight terabytes in the whole world.

    Kimball: And there weren’t for a while, right?

    Shimel: Yeah, it’s crazy. And, you know, as our data storage and data needs have grown, so has our databases. So, you mentioned the spinoff from Google, you mentioned certain things that the latest, greatest from Google didn’t do and hence, why Cockroach.

    But you guys recently also announced a new funding round, if I’m not mistaken, right?

    Kimball: Correct, right. We’re going to—that will happen on August 6th.

    Shimel: Well, by the time people are listening to this, it’ll be August 6th already, so.

    Kimball: Got ya, okay.

    Shimel: If you’re listening to this at home, it’s after August 6, they’ve announced their funding round.

    Kimball: Yeah, so, we did close that, and it’s a big round. And basically, the additional money equals all the money we’ve raised in the past which, you know, isn’t so unusual, but it’s a lot of money. In aggregate, we’re talking over $100 million.

    Shimel: Wow.

    Kimball: You know, I think, to put that into perspective, you just have to look at who we’re competing with, right? I mean, certainly, it’s some of the biggest companies in the world, the biggest companies in the world, and we’re also serving the biggest companies in the world, so you do need that money both to create a sales force that’s capable of educating and reaching and selling to those companies, which all—I’ve been humbled and amazed by the process necessary to actually sell to a Fortune 500 company. In some cases, it’s quite involved.

    But also, to continue to iterate on the product, and we have to, you just need to hire the best people and create an environment where they can collaborate to move the product forward so that you can compete.

    Shimel: Absolutely. But, you know, look—in the Internet and the IT world today, compete means different things to different people, right? We compete by offering great service. You know, I just got done, I was over at DevOps Enterprise Summit in London a month or two ago, and there was this notion there of what folks were calling BVSSH—Better Value Sooner, Safer, Happier, right? And that’s what we try to deliver in DevOps and stuff like that.

    So, does competing on that and—but then, in the database world, too, it’s just really, sometimes it’s very, very, very feature driven.

    Kimball: Yeah.

    Shimel: Right?

    Kimball: It’s incremental features, and then there are sort of generational features. And we’re, we’ve been pushing on truly changing the nature of the database, and that is an opportunity that exists because of all the other things happening in the ecosystem—in particular, the global public cloud, which is, I think, one way to describe the combination of AWS and GCP and Alibaba Cloud and Azure. There’s many others, of course, that are far more regional. But in aggregate, all of those can be utilized by new database architectures to allow a company to store data in new ways that ____ their business versus their competitors.

    So, just to give you an example, things like GDPR and recent regulations around privacy and Russia and China and Vietnam and South Korea and South America—the list goes on—there’s a lot of attention being placed on how data is localized, where it’s stored in relation to the customer, where it’s processed.

    And also—and this is, I think, really exciting—there is an ability, because of this global public cloud, for new services to come online that are able to deliver the end to end user experience that happens instantaneously to the user. So, that means in under 100 milliseconds, which is sort of the user threshold for noticing time intervals. If something feels instantaneous, but it happens in less than 100 milliseconds, you literally, because of physics and the speed of light, you can’t do that for a customer in Australia if all your data is in one data center in Virginia. But that’s the model that’s prevalent right now.

    Shimel: Sure is.

    Kimball: [Cross talk] on public cloud and with a solution like Cockroach, you’re actually able to have a globally spanning database that’s able to domicile data near the customer and actually deliver both the assurance of some data privacy, that their data is in process and stored close to them in their legal jurisdiction, but also to have an end to end user experience where you click a button on your mobile app, it goes all the way through the application logic, accesses the database, comes all the way back and updates. So, that ability to make that instantaneous, I think, is gonna open a new era of applications.

    Shimel: It sure is. You know, so, I remember, Spencer, back—so, one of the companies during the dot com days, one of the companies I helped build was a company called, it came to be known as Interliant, and we were an early ASP application service provider.

    But before we changed our name, we were Sage Networks, and we were a large hosting provider, a rollup. We had bought a lot of different hosting companies. And one of the key things for us, like, you know, what really brought us scale versus all these smaller hosting companies we had bought, is that all of these hosting companies’ storage and the data was sort of part and parcel with the same infrastructure, if you will, that was hosting the website and the web server and all of that.

    And what we did is, you know, we actually—I don’t wanna say abstracted, because it wasn’t really abstract, but we separated the data and the database from the web server and the applications they were running, right? And now, that freed us up to store that data. This is before there was a cloud, but we had multiple data centers. So, we were at the point, almost, where we could store data, let’s say, in our Atlanta data center and access it or use it on websites where the web servers were stored, let’s say, in Tysons Corner in Virginia at the time. And what was holding us back there was, we didn’t have the connectivity you have now.

    Kimball: Yeah.

    Shimel: Right? If you had a T3 line, that was hot stuff, you know? That was carrier grade. And so, it was a little different. But, you know, this is the age we live in now where you do have distributed applications. So, of course, why wouldn’t you have distributed data storage and the databases that organize them?

    Kimball: Yeah. I mentally have distributed audiences, right? I mean, every time a fledgling startup puts their first mobile app on the app store, the total global population that can have access to that, it’s pretty staggering. I mean, the opportunity there is just tremendous. But, you know, everyone’s—90 some-odd percent of efforts out there are focused on deploying to a single availability zone, right? So, this is, there’s a legacy of how things were built, which is rapidly changing, and you have to do it through the whole stack. At Cockroach Labs and CockroachDB, we’re attacking a fundamental part of that stack, very important, probably one of the harder parts, which is the data storage, the state behind those applications.

    Shimel: Yep. No doubt about it. I wanted to talk a little bit about CockroachDB sort of as a project, if you will. Cloud native, right? When you guys were starting out, obviously, you came out of Google, you have a big open source kind of heritage at Google—what were the discussions then about, “Is this something we open source? Is it something we make free?” How do we walk that line, right? Go ahead.

    Kimball: So, when I was at Google, from 2002 to 2012 with my co-founders, Google was only starting to do open source towards the end of that tenure. Previously, they were pretty adamant about keeping things closed source. It was just sort of the model that Google evolved with. Now, they’re much more open, and some of their more recent efforts, including TensorFlow and Kubernetes, are examples of where open source in the hands of Google actually is a tremendous advantage both to the company and to the larger ecosystem. But things like Spanner were closed source and many of the things that Spanner is built on are also closed source.

    But what Google did do, which is great, is they always publish about these things.

    Shimel: Right.

    Kimball: When we started Cockroach, we had the benefit of not only working at Google and seeing how these things were built—you know, Peter, one of my co-founders, and myself worked on something called Colossus, which is the exoskeletal distributed file storage system at Google, which runs underneath Spanner.

    So, we had a really great opportunity to see why and how Google built the things that they built. But, you know, once we were kinda kicked out of the Google nest—or we left it, I guess, is the better way to say it—

    Shimel: Flew the nest. [Laughter]

    Kimball: Yeah. We flew the coop, right? We had the benefit of all of the published work that Google had created around things like Spanner, but you know, out in the wider world, you realize, for a database—and when we started this, I guess it was 2014. In 2014, and it’s certainly true today, people aren’t going to pay attention to your database unless it’s open source. And it’s just, it’s sort of a fundamental requirement. And there have been databases that have tried as closed source software, but they haven’t gotten very far, to my knowledge.

    So, I just think that—and this is true for me, too, as a potential user of the database, as a potential developer on the database, something that’s not open source feels a little bit icky. [Laughter] Right? Like, there’s not enough surface area that I can latch onto where I feel like I can be the master of my own fate, right?

    Shimel: Yeah.

    Kimball: Like, if something—that’s what I hated about closed source products. If something goes wrong, it’s just this big black box and, you know, it’s sort of a black hole if you wanna send a support ticket in, what’s gonna happen? Whereas it would be much nicer for everyone involved, in my opinion, if I were able to dig down in there, find out what was wrong, probably my fault, but let’s just say it’s a bug in the product, I could send that into the company and they could take that as a patch.

    That’s the beauty of open source and that, as a personal preference and sort of even as an idealistic sort of mindset. I would not have been able to start the database as anything except for open source.

    Shimel: Yeah. I mean, and look, quite frankly, there is a bit of a history in the database space of successful open source databases that not only were successful as open source projects, but were equally or more successful as commercial entities or the companies that manage these open source projects—you know, very successful open commercial entities that had great exits and great value for the investors and founders and employees.

    So, I mean, in that regard, it’s not like you’re pioneering, right? It’s not like someone says, “Oh, they’re crazy to do this in open source,” right? Well, 15 years ago, they might have said you were crazy, right? [Laughter] You know—

    Kimball: I think MySQL kind of, you know—

    Shimel: Yep, ____.

    Kimball: —blazed the trail, as they say. And Mongo is a very successful—

    Shimel: Absolutely.

    Kimball: [Cross talk] that uses open source as well. And there’s a number of other ones. But you know what, even when we started this company in 2015, I think the open core model and open source in general seemed like a catalyst, right? Just something that could create business value far more quickly than closed source software and the sort of procurement department and budgets and all this stuff that used to be involved in the cycle of adopting a new technology in order to build your products or services that the engineers or developers at a company needed to—

    Shimel: Oh, absolutely. I mean, look, I’ve been in IT 30 years, right? I’ve seen the rise of open source. I was involved in open source security stuff in 2001, ’02, ’03, ’04. The deal is, today, for a lot of developers, if it’s not open, they don’t wanna work on it, number one.

    Kimball: Yeah. But—

    Shimel: Go ahead.

    Kimball: What’s really fascinating is that 2015 might have been sort of the beginning of the end for open source being the true, the most important catalyst of business value. I would argue that, in 2019, the true catalyst to business value, faster business value—business value in sort of relation to your competitors, I think it’s services, right? Cloud-based services. That is—you know, it’s always about, it’s not about the idealism of open source. That’s not why it was successful.

    Shimel: No, I think we’ve passed the idealistic stage of open source.

    Kimball: It’s basically like, what are the tools out there that a software engineer at company X can use in order to get their project done faster so that it’s more stable, it’s more performant, it costs less, right? All those things—

    Shimel: But it’s their preference, and these developers and the Ops and the DevOps folks, they’re the top predator on the food chain at a lot of these companies. They’re the hardest piece of the puzzle. And so, companies are hesitant to tell them, “No, I don’t want you to use that tool” or, “I don’t want you to use this service.” They’re the ones making these choices, where I think, in years past, it was like, “Oh, no, you can’t bring that in here” or, “You can’t use that one” or something like that.

    Kimball: Yeah. It kinda [Cross talk].

    Shimel: But there’s more, Spencer. Let me tell you this. You know, I’ve been going to a bunch of the Cloud Native cons and Kube cons and those kinds of shows, and I spend my life at IT conferences. I’m telling you that the open source community—and not because of the idealistic ____ and the bizarre and free as in freedom and free as in beer kind of crap.

    We’re past that, you know, Dr. Stallman stage, if you will. It is the, that community is the energy driving almost everything we’re seeing in IT today, right? There’s such an energy in there that it’s driving what you’re doing at Cockroach, for sure. It’s driving what we do in DevOps. It’s driving what we see in Cloud Native and Kubernetes and all of these things. It’s where—it’s almost like this well of, like, a nuclear power plant or something, you know what I mean?

    Kimball: Yeah, absolutely. And, you know, I think the best model right now that I’m aware of is, you know, it’s called cloud-based services with a very fast time to value, incremental pricing, right? So, you kinda pay for what you use as you go. If you can deliver that with an open source product that’s backing it that people feel that they can get into and they can use if they had to move off that sort of freedom and flexibility, that sort of offers the best of all worlds.

    And I think that—and that’s why Amazon’s been so successful in adopting open source technologies, because they’re very good at providing things as a service and they have so many things altogether into one, you know, that can be packaged into one bill, one sort of monitoring platform, one identity access, you know, management platform. It’s very impressive, but, like that, in aggregate, creates so much value for business.

    Shimel: It’s crazy. We’re running low on time, so let us bring this back to Cockroach Labs. We seem to have taken off there on a tangent. You guys are offering a managed version.

    Kimball: That’s right. So, we have—some of our biggest customers, they wanna run the database themselves, it’s the way they’ve been running it, but over time, that will shift, of course. So, we have a self-service, self-hosted version where you buy subscription licenses if you wanna pay us for it. There’s also, of course, open core, which you can use for whatever purpose.

    Then there’s the managed service offering, which we’re calling Cockroach Cloud. And that, we run it for you, and we can run that on AWS or GCP, and the pricing model is much more like something like DynamoDB or Amazon Aurora or MongoDB Atlas, those are sort of similar kinds of offerings.

    Shimel: Got it. Excellent, man. Just, again, Spencer, for people who may not be familiar, the website is CockroachLabs.com.

    Kimball: That’s right.

    Shimel: So, that’s where people can go get more information, there. Any other news that we didn’t hit on, here?

    Kimball: Well, you know, I think maybe we didn’t quite cover the funding well enough. I kinda jumped ahead, but we’re raising more than 15 million and the lead investor is Altimeter Capital.

    Shimel: Great.

    Kimball: So, there’s sort of growth equity added, along with Tiger Global, and then all of our existing investors, Google Ventures—GV is what they’re called now—Benchmark, FirstMark, Redpoint, Index Ventures. So, it’s a great—

    Shimel: Impressive list.

    Kimball: Yeah, a great group of investors. We’re looking forward to sort of this being an inflection point and really moving forward with our penetration into the Global 2000 and also fundamentally putting a lot of focus on managed services, so, the cloud delivered Cockroach experience, in addition to enabling more and more of this global architecture capability that I was talking about earlier.

    Shimel: Absolutely. I mean, it takes resources today to make all of that happen and bring it all to market, obviously, and hiring good people is not—even with the money, it’s not an easy thing to do, as I’m sure you’re finding out. Finding good people is—

    Kimball: A lot of these guys have—I liked what you said, at sort of the top of the food chain.

    Shimel: Yep, the alpha predator in the thing. It’s crazy.

    Kimball: They vote with their feet, right? So, it’s—

    Shimel: Yeah, they do. I mean, they don’t even ask for raises, they just go get another job. It’s not—I mean, it’s a crazy, crazy dynamic at play there, but it is what it is.

    Well, listen, this was—obviously, we blew past our 15 minute time, but it happens, it’s all good. I want to wish you success. I mean, just a quick digression from the point—you know, there’s a lot of people out here who are gonna listen to this, Spencer, who will say, “You know, I have a great idea for a company. We should go raise a few million bucks in capital, and we can do similar to what Cockroach is doing.” Not necessarily in the DB perhaps, but you know, in startup world, it’s not easy—$100,000,000.00 is a lot of frickin’ money for people to write checks to you for. And it doesn’t come without tremendous responsibility, and having raised a lot of money in my lifetime for venture backed companies, I know what goes into it.

    So, congratulations on a successful run with this, and looking forward to seeing what you do with it.

    Kimball: I appreciate that, Alan.

    Shimel: No problem. Alright. Our guest was Spencer Kimball, CEO, co-founder of Cockroach Labs, makers of the CockroachDB, and this is Alan Shimel for devopsy.kinsta.cloud. You’ve just listened to another DevOps Chat.

    — Alan Shimel

  • The Power of Autoscaling and the Mistake You’ll Never Make Again

    The Power of Autoscaling and the Mistake You’ll Never Make Again

    Many years ago when I was interning as a software engineer, I made a mistake that could have ended my career just as it was getting started …

    It was my birthday, and I was given the task of loading queries to a database. The task was straightforward—even then, when I was only an intern learning my craft. I was using a program to upload the queries automatically. It looked like the program was running well, so I decided to go home and leave the program running overnight.

    The next day at the office I met any intern’s worst nightmare: Everything had stopped. And it was all my fault. The queries were loading with a paralyzing level of inefficiency. In fact, it took up all of the processing power that we had in that company server, making it crash. All operations were brought to screeching halt … just for my measly query project.

    It took several hours for the operations team to have everything running again. Luckily, I didn’t lose my job. In fact, the company ended up handling the situation graciously, and it turned out to be a good learning opportunity for me.

    These days, something like this is entirely avoidable, and it’s all thanks to the magic of autoscaling. As recently as 10 years ago, obtaining more server space was a slow and rigid process owned by a few gatekeepers. Now anyone can acquire more server space with just a few clicks.

    The democratization of processing servers has completely transformed the internet. It has given any organization virtually unlimited processing power at a moment’s notice. When done strategically, autoscaling allows for a more flexible and agile processing infrastructure. If implemented in a way that’s ad hoc or rushed, your organization will end up spending exorbitant sums of money without seeing the full potential of autoscaling.

    Here are three questions you need to answer to develop a winning autoscaling deployment strategy.

    What Should You Autoscale?

    Before the advent of autoscaling, projecting server needs took a lot of guesswork. Imagine, for example, that a timely and effective advertisement suddenly brings a 1000x spike in traffic to a company that sells sunglasses. If the company is not ready for this spike in traffic, everything comes to a screeching halt. Orders are not filled—and some orders can’t even be made because the server space is not ready to handle the unexpected surge in volume.

    With autoscaling, processing power can be offloaded to a new server automatically, based on the conditions set by the system’s administrators. This is valuable not only in the case of spikes in traffic, but also for lag periods. This is a way to avoid paying for processing power that you’re not using.

    In any organization’s computing architecture, there are a number of components that can be autoscaled, whether it’s central processing units (CPU), random-access memory (RAM) or bandwidth. The first step is to collect as much data as possible to understand the current operational loads of each of these items. Any organization using Amazon Web Services (AWS) has access to some data monitoring services, but they tend to be very basic. It’s better than nothing, but there are other options out there.

    One option is black box testing, which allows you to measure various components within your  infrastructure through monitoring. Among many things, it can help you determine where bottlenecks are occurring. Elastisearch is another great resource for word searching and metadata collection within a document or a digital framework.

    Another effective tool to assess autoscaling needs is something called an ‘agent.’ It works by sending pulses to measure processing data throughout an entire system. Collecting data like this will give you crucial insight on the state of your processing infrastructure.

    Once you collect all of this data, a thorough analysis is in order. At present, analyzing this data must be done manually, but before long machine learning will take care of this part of the process. It’s important that you consider all layers of your applications (front end, back end, database) to autoscale effectively. It can be tempting to scale at will, given how easily it can be done. Be vigilant against abusing this capability. Otherwise, it won’t be long before you spend outrageous sums of money on something you don’t need.

    How Should You Autoscale?

    An important concept to understand before you decide on your autoscaling approach is containers. Containers are small pieces of code that are meant to perform a specific function. In this way, a single machine can contain several apps used to perform a variety of functions: CPU, RAM, bandwidth, disk space, etc.

    Containers are a sophisticated way to get creative with the server space you’re working with, and should be considered even before you start building on a vertical or horizontal autoscaling strategy. Better algorithms in existing hardware can help you to avoid making new hardware purchases. In any case, once you do decide to autoscale, containers should most definitely be considered as you strategize and build.

    However, while containers can be great for scaling, it should be taken into account that they have the potential to add a great deal of complexity to your server. For example, if your server can run 20 containers, but suddenly you need three more, buying a new server with more containers is an expensive solution, since the server would be mostly empty.

    In principle, there are two ways to autoscale: vertically or horizontally. Scaling vertically means adding more physical machines with more gigabytes for more processing power. The biggest AWS machine server that’s currently available (at least as far as I’m aware) is 128 CPUs. That’s the most processing growth you can make from just adding one server. The cost of a new machine like this will vary: anywhere from 80 cents to $7 an hour is what you can expect to pay.

    The other approach is to scale horizontally, which would involve a cloud-based solution. For organizations not ready to make the commitment that comes with adding a new physical server, this is the approach worth considering. Plus, with a horizontal approach to autoscaling, it’s possible to adjust processing needs on the fly.

    It’s important to understand that not all parts of an application can scale the same way. Databases, for example, can scale up with ease. But scaling in (horizontally) would take a lot of time. If your data is stored in 30 front-end containers and you want to scale in to 15 containers, you can’t do it in one step. You’d first need to scale from 30 containers to 28, wait for the data to replicate, then scale in from 28 containers to 26, again wait for the data to replicate, then from 26 containers to 24, etc. Otherwise you risk losing all of that data.

    When Should You Autoscale?

    Consider the above example of the sunglasses vendor. While it’s impossible to predict exactly when your business or customer base will reach a tipping point, something such as publishing out an advertisement with a broad reach is certainly an instance where your organization ought to be ready for a sudden spike in traffic. Your business decisions should be made mindfully, so that the potential effects of actions like these—and the strain they create on your organizational architecture—are understood ahead of time.

    Anticipating peaks and troughs in demand is something businesses have been doing for centuries. In Colombia, for example, utility companies have long understood the power of “arepa hour”: This is when Colombians are at home eating breakfast (specifically arepas, a traditional breakfast food in Colombia). Knowing that the majority of the country is likely to be in their homes sometime between 7:30 and 9:30 a.m. consuming power, that is a crucial data point that can allow for utility companies to be ready for this disproportionately high level of demand.

    The dilemma to determine “the arepa hour” of your business’s autoscaling needs often comes down to CPU and memory. Should you be ready to autoscale based on when your site crosses a predetermined CPU load? Or should it be based on the memory running on your servers?

    The answer is really neither. There is no “golden metric” that will point the way for your autoscaling strategy, and you should never decide to scale based on a single data point in isolation. For example, it may make sense for processing data to be offloaded to a new server once the processing capacity is at 90 percent, but it’s also entirely plausible that your application continues to run completely normally with zero issues at this processing load.

    You should be aiming to strike the right balance between efficiency and not compromising the user experience for those who visit your site or use your application. And the only way to do this is to consider all of your data points holistically and how they interact with one another—not in isolation.

    Conclusion

    We are past the time in which a query loading faux pas like the one I made as an intern can completely shut down a company’s processing activities. Given how accessible processing servers have become, some may find it tempting to “brute force” their way through every problem: Make a few clicks and pay for more server space at will. Don’t be goaded into taking the easy way out. Think about autoscaling strategically by answering these deceptively basic questions, and you will in fact make it easier (and more cost efficient) for your organization in the long run.

    — Alejandro Calderon

  • DevOps Chat: DevOps for Databases, a DBmaestro Update

    DevOps Chat: DevOps for Databases, a DBmaestro Update

    DevOps is coming to databases. While there are several companies who have been dedicated to this over the last few year, by all accounts DevOps for databases has been catching stride as more DBAs and organizations realize the benefits of DevOps for databases.

    In this DevOps Chat we sit down with Yaniv Yehuda, co-founder, CTO of DBmaestro, to discuss what is driving the increased awareness and what is coming to the market. Yaniv is always a great interview with lots of good insight. Enjoy.

    As usual, the audio of our discussion is immediately below, followed by the transcript of our conversation.

    Audio


    Transcript

    Alan Shimel: Hey, everyone. It’s Alan Shimel, DevOps dot com, and you’re listening to another DevOps Chat. Welcome to DevOps Chat today to our – a repeat guest. We’ve – he’s been a guest on DevOps Chat several times. Yaniv Yehuda, co-founder, CTO at DBmaestro. Yaniv, welcome to DevOps Chat.

    Yaniv Yehuda: Hey, Alan. A pleasure being here again – pleasure being here today.

    Shimel: Yes, thank you. Always a pleasure to have you, my friend. Yaniv, what we’re going to spend a little time talking about today is, you know, as DevOps continues to mature and we see it moving beyond, you know, the orthodox Dev and Ops only to encompass, you know, more pieces of the IT stack, it’s becoming, you know, increasingly obvious to people that having the database and the DBA as part of the DevOps team, as part of the – you know, under the DevOps umbrella – is a must have.

    And I guess this is something that is kind of elemental, if you will, to the DBmaestro folks. It’s what you’ve been preaching now for, how long?

    Yehuda: For several years. Several years already, yes. Absolutely. Absolutely. No, so I’m in perfect agreement here, you know. It – if you’re trying to start and practice the new or modern process as – may that be DevOps, variations of agility, trying to practice CI/CD – at the end of the day, you need to take care of everything. You can’t just do two thirds or half of the equation. You – it’s like having a car that, you know, you have three wheels spinning at one speed and one spinning slower. So it doesn’t make sense. You really need to cater for everything in order to be efficient, in order to be repeatable.

    And recently, you know, talking about DevOps, it’s really expanding. People now really want to cater for security at the same – at the same rate. Because you’re trying to – to shorten feedback loops, you don’t want to deal with stuff too late, so it’s – it’s mainly DevSecOps and this is what we’re seeing from our customers, the need to take care of security, they need to take care of the database, they need to take care of – of agility and productivity together. So with security.

    Shimel: Yes. Yep. Now, but Yaniv, when we start talking about, you know, DevOps for the databases, DBAs, and so forth, one of the things – right, the database market is a fairly mature market with more than several players in it. Right? You have your  Oracle, DB2, Microsoft SQL, then you have sort of the open source crowd, like the MySQL and the MariaDB, you know, and so forth, and then of course you have the NoSQL group or the so-called NoSQL group which have come on, let’s say, in the last six, seven years.

    I would – is there – from DBmaestro’s point of view, let’s say, is there a difference which database you’re using or they – they’re relatively the same across the board?

    Yehuda: So discussing, like, from a customer challenge point of view, it’s all of the same. It’s all variations of the same challenges. For Oracle you might be pushing, I don’t know, structural changes and – and index creation or even DMLs just to tune the data and for NoSQL you might be deploying an update or a patch that needs to be implemented in each one of the databases but you have a thread of them. It’s – it’s like shards and – and, you know, replications that – and you actually need to take care of a lot of instances.

    So in all cases, you need to have a process to implement changes whether, you know, it’s one type of a database or another. And, you know, it’s just the same problem from different angles.

    Shimel: Yep. Is there anything – let’s say NoSQL. Right? Is there anything specific to NoSQL from a DBmaestro point of view?

    Yehuda: So first of all, we’re about to ship a MongoDB edition for our software.

    Shimel: Okay.

    Yehuda: And I think, talking about specific databases, you tend to see different aspects of – as I said, different aspects of the same thing. I’ll give you an example, a quick one. So many times, when you talk to DBAs and database developers and DevOps people, they want to move fast. And, you know, people say, move fast, break things, and be effective, be productive, but at the same time you need to be careful what you break because some things are just harder to fix.

    So, you know, if you drop a table from production then all break – all hell breaks loose but if you do things that would just take too much time to fix, even if it’s reversible, it’s not a good thing. So you need to come up with a way to move fast, break whatever is breakable, but not break Humpty Dumpty because it’s going to be hard to fix.

    And this is where we introduce a lot of safety nets into the process. So these safety nets might be more database flavored and – and I’ll give you an example. So let’s say you have a team of developers who wants to be agile and they want to run quick, they have their datalines, they have their tasks that they need to provide, and they need to go to the DBA and ask him to do stuff. If he actually has to go to the database and log in and run strips and do stuff, it’s – it takes time.

    You know, I’ve talked to DBAs who had – you know, a few days of tickets to go through and execute what people want to do. And it’s nothing that really benefits anyone because it’s all about just having the right credentials to the correct place and knowing who asked for what and should he be able to run that.

    So we’re automating all of these things in order to make sure that you can run fast, but about the breaking stuff, you break what should be broken. And I’ll give now the example about, like, Mongo versus Oracle.

    So for instance, we have what we call organizational policies and this is really the – helps balance the amount of work with the need to control what goes forward and not destroy things. So for instance, you have policies for Oracle or SQL or DB2 that says you cannot drop a table. Why? Because, you know, you might lose the data. Or you cannot rebuild an index on production in the middle of the day. Obviously why.

    But when you go to something like Mongo, then it’s all about fine tuning these policies. So if you’re trying to, like, create an index with background faults, that means that when you’re going to try and rebuild it, it will be a nightmare. So it’s all about stopping it at the first time when they – when the person puts in the command and saying, “Listen, you have an attribute missing from what you’re trying to do,” in order to live with that safely later on.

    So it’s all about catering for the, like, problem areas of each database and providing the benefit so people can run and if there’s something that they’re doing wrong, they will get very quick feedback, so that’s, you know, all about trolling the feedback loop just like DevOps and Agile are trying to achieve, and this is very much database-specific – so to your question.

    Shimel: Excellent, excellent. Yaniv, let me – let me turn a little – well, before we do, I wanted to just get maybe a – give our audience a quick update on DBmaestro. You mentioned a new MongoDB version coming out, but what else do we have?

    Yehuda: So actually we’ve – we’ve released a DB2 edition – I’m saying edition, it’s actually everything within the same product. It’s DB2 Sport, so DB2 for the mainframe z/OS and for the open systems – Linux, UNIX, Windows – we’re about to ship Postgre support and MariaDB so this is on top of Oracle and – and Microsoft SQLs that are already out there in the market for quite some time. And, you know, Mongo, as we’ve mentioned before.

    So what we’re doing is we’re not, like, sitting in a lab and dreaming, you know, we should do this at – we’re actually, in daily conversations with our customers, getting a lot of feedback, you know, working with the people who need to crack their own tests and trying to make sure that DBmaestro actually helps them in the most efficient way.

    So, you know, I’m proud to say that – you know, we have quite a big vision of where we’re going but a lot of it is fine-tuned with what’s happening in the market so it’s not just lab dreaming and wishful thinking.

    Shimel: Absolutely. So I – I don’t know if your being here told us everything though. I think I heard somewhere about an award or something.

    Yehuda: Oh yes. That is – this is quite, quite new. It’s actually today. We’ve just been announced that we won the Stevie Award for – we won a Bronze award for the best new product. So of course we’re very proud of this and probably will be boasting about this for quite some time now.

    But actually, with – you know, it’s – it’s very encouraging because we’re getting a lot of great feedback. The feedback that I like the most actually, more than awards and, you know, the stuff that you put on the website, is just customer feedback. You know, when you – when you see a person telling you, you know, “I used to wait three days for the tickets to go forward and now the developer is pushing changes five times a day, 10 times a day.”

    This is all I need. This is, like, what we dreamt when we wanted to – to do what we’re doing, to make sure that the database can play the modern processes, that people can practice, you know, the database stuff safely because, you know, if you’ve been talking to, like, DBAs two, three years ago. I think talking to them about automation would be a big, scary thing because, you know – you need to know what you’re doing, you need to make sure that you don’t introduce any risks. And at the end of the day, this is something that you just can’t do manually. You need to be repeatable and the only way to be truly repeatable is to automate stuff and to keep improving it.

    If you do something manually, at the end of the day you’ll make mistakes and, you know, as – as a certain big bird once says, you know, everybody makes mistakes. Oh yes they do. And this is it. You need to make sure that DevOps plays to the values of what you’re doing and – and this is what we’re putting for ourselves as targets. You know? Help people be more agile, be more productive, but not at the expense of safety. So more safe at the same time.

    And that, you know, if it can give you the ability to be also compliance-worthy at the same breath, yeah. Sure, why not?

    Shimel: I agree – you’re preaching to the choir over here, Yaniv. So let me – I wanted to bring up another topic, Yaniv, and this may be a little close to home for you, a little too sensitive, but, you know what, obviously I follow the DevOps market very closely now for a number of years and, you know, I think back to three or four years ago when, you know, the biggest question we got on DevOps dot com was always, “What’s the definition of DevOps? What exactly is DevOps? Is it just a marketing word?”

    And then when we – you know, when it was just Dev and Ops and the idea, when you would talk to someone about, you know, database for DevOps or DevOps for database, excuse me, DevOps for databases, there were a lot of – I don’t know what you want to call them, cynics or just – they were a little bit – a little stiff, who said, “Oh, that has nothing to do with DevOps.” Where do – within the DevOps community itself, how much – how has the attitude changed where DBmaestro and the whole, hey, DevOps is for databases too, has become more accepted within the DevOps community?

    Yehuda: That’s actually – you’re just describing my history here. So for a long time I have been trying to educate and tell the story as I think it should be told and, at the beginning, you know, when we were just bootstrapping and didn’t have the product that we’re so proud of today, cynics were even more extreme and it was, like, DBAs on one side, developers on the other side, a big wall in between, people tossing things over that wall.

    And actually, if you go back two years, two years back I think, this is no longer the case. People are implementing DevOps not just to be modern. They don’t have a choice. Gene Kim once said, you know, it’s DevOps or die. You don’t have to do it but you don’t also have to survive.

    So today – in today’s business, everybody’s trying to run fast. If you’re not running fast, that means that you’re running slower than your competition and nobody wants to afford that.

    You know, you had a webinar, I think, with ABN AMRO a couple of months ago. I was just delighted to sit there and listen to how they described what they built. It’s just – it’s just amazing. So – so – you know, it’s a carefully thought out process that takes out all of the junk time out of the process and makes everything efficient, whatever can be automated is automated, and, you know, it saves a lot of time. It makes the whole company more efficient.

    And then you reached a point where, you know, you’re looking at what you’re doing and that is working – I’m not talking about specifically them, but in general – everything is working but you’re not as efficient as – as you could be because every time that you go to the database side, you go back – you fall back to the manual processes – the single steps, God forbid, the DBA that goes – and side commands in production to make the change go forward.

    You know, the funny thing is when I’m recruiting people to the company, I’m telling them what we’re doing and they said, “This is amazing. This is what we’ve been doing all our lives wrong and this has always been a headache and you’re just solving it.” So that’s – that is certainly something that has drastically changed and I think the last two years has been an accelerated – I would say, accelerated approach to the whole thing that – it cannot be left aside. You cannot just do application, you cannot just do database, by the way, as well.

    You have to introduce a testing automation. You have to build with your infrastructure automation. So may that be containers or virtualization depending on what – where you’re coming from. It all has to be combined in order to reach that Holy Grail of efficient automation.

    Shimel: Got it. Excellent. You know, it – I think it’s a good thing for the DevOps community as a whole but it’s interesting, Yaniv, you almost have to be an evangelist to the DevOps community at the same time you’re evangelizing to the database community because I don’t know if they raise their hand and say, I want to be part of that DevOps thing too. Right? They – they need to be shown the way and evangelized and converted, if you will.

    Yehuda: So on this we do see a changing approach. It used to be, like, trying to convince why they should care. Today it’s not – I’m happy to say, it’s no longer the case. Some customers we have, you know, DevOps champions that push us forward. In other organizations, it’s the DBA or the DBA manager because they see what’s happening. They see the projects around them in the company.

    They don’t want to be left out and there’s – you know, there’s really no gain in – in doing a mundane job. Let’s do the fun stuff. Let’s – let’s improve logic, let’s fine-tune the database. Let’s do the stuff where, you know, computers can’t – can’t still, at the moment, solve the problems for us and leave that, you know, the mundane – mundane back-work or whatever to automated process.

    Shimel: Yep.

    Yehuda: And it’s not just DBA. You know, today you talk to – to, you know, all of our European or the ones who – who need to cater for GDPR. They have a set of new problems they need to cater for. It’s not just, you know, automation. It’s not just DevOps, they need to now make sure that they manage their roles and responsibilities and auditing and obfuscation of data, if you’re taking, like, data from production, now there’s no private data inside and how you do that process. So you want to have holes.

    You have automation for many reasons now. It’s not just efficiency. It’s creating the good process and sticking to it because if you have a great process but it’s just a manual process, each time you do it, you’re doing it slightly different. And this is exactly where I think the whole ARA market is spot on. This is why we’ve – I think we’re seeing the changes we’re seeing in the market in the  last couple of years.

    Shimel: Agreed. Excellent. Yaniv, you know what, I apologize, we’re way over our time here. So we’ll wrap it up. Before we wrap up, any last thoughts – anything you want to share with the audience?

    Yehuda: No, I think – I think, as you said, we did – took a bit more time so I think we can wrap it up.

    Shimel: Okay. Well, look, I wanted to mention that DBmaestro will be at Gene Kim’s DevOps Enterprise Summit in London, I think June 22nd to the 24th, maybe it’s the 23rd to the 25th. Something like that, towards the end of June. So if you were planning on attending DevOps Enterprise Summit, London, please stop by and say hello to the DBmaestro folks. And then Yaniv, you’ll also be at Cloud and DevOps World in London in June as well. Correct?

    Yehuda: Correct, correct. Yep.

    Shimel: Excellent.

    Yehuda: So, as I said, going – meeting our peers, meeting our partners, meeting our customers – just talking.

    Shimel: Excellent. All right, Yaniv Yehuda, CTO, co-founder, DBmaestro, frequent DevOps Chat guest, thanks for being here today with us and continued success.

    Yehuda: A pleasure being here again. Thank you. Bye-bye.

    Shimel: All right. This is Alan Shimel for DevOps dot com and you’ve just listened to another DevOps Chat. Have a great day, everyone.

    — Alan Shimel

  • NoSQL Databases: Lessons Learned Growing a Globally Focused Startup

    NoSQL Databases: Lessons Learned Growing a Globally Focused Startup

    Startups deploying NoSQL databases and global citizens learning new languages actually share something in common: Both want to remove the limitations on how they access information and express themselves. In launching and growing a platform that enables anyone around the world to connect and speak with native English-speaking tutors on-demand, we’ve had the fortune of helping countless people learn new language skills—and we’ve also learned a great deal ourselves about the best ways to handle the requisite user and application data.

    NoSQL, No Problem

    Here are four NoSQL database strategy takeaways we learned from recent experience:

    Keep Your Data Options Open with Flexible Technology

    When a startup is just out of the gate, it isn’t immediately clear what shape the final product may take, or what data it will require.

    In our case, the relatively broad scope of our project also defined the need for a database able to function globally with 24/7 availability. The concept for Cambly was born out of its founders’ real experiences traveling to other countries and learning foreign languages through conversations with native speakers. Immersive experiences such as these are critical for gaining and retaining conversational language abilities, and provide learning opportunities that no language course or software can really match. However, these language skills can recede rapidly if a person no longer has access to native speakers to practice with. Seeing this as an issue that technology could address, we began building an application to provide global on-demand access to immersive conversations with native speakers, initially concentrating on English as the largest language market.

    As a startup beginning to explore how exactly we’d accomplish the task ahead of us, it was critical that we could store whatever types of data we found to be of value as we quickly iterated the application. This meant utilizing a NoSQL database solution to free ourselves from any schema constraints. We selected MongoDB as a flexible, NoSQL solution with which our technical team had some existing familiarity.

    Get Help …

    Startup success is dependent on managing finite internal resources in a way that maximizes the attention that can be devoted to product development and other growth-critical initiatives. This means having the wisdom to rely on managed services where they’re beneficial in saving time and simplifying operations, versus building up internal headcount. For example, we enlisted Heroku to use as our application platform. And, even though our team was experienced in running databases and managing machines on AWS, we still saw the value in keeping production as hands-off as possible and opted for a database-as-a-service (DBaaS) strategy as well.

    … And Make Sure It’s the Right Help

    It can be tricky, but ensuring that any and all technology service providers a startup works with are a strong fit for its needs is vital to moving quickly. We actually learned this the hard way when it came to our original MongoDB DBaaS. For us, reliable 24/7 availability and insights into database performance were essential requirements. Unfortunately, our initial provider presented us with two challenges to meeting these requirements in a short time frame: Its tools became unable to provide us with visibility into our database, and it announced a required platform migration that could not be completed without downtime. Naturally, these changes led us to seek a better-fitting option. Soon after, we partnered with mLab in large part due to rolling node replacement capabilities for migrations and 24/7 uptime, as well as support and tooling that could help our team quickly optimize MongoDB instances.

    Get the Database Tools You Need to Succeed

    From reliable data backups to performance insights and beyond, the right toolset can make or break a burgeoning company’s success. Just as language skills must be constantly attended to, your database requires intelligent upkeep and backups to ensure it delivers a satisfactory user experience and remains available in case of disaster. This is one lesson which—very fortunately—we were able to absorb without hardship. We had an incident in which our team accidentally deleted a trove of valuable data. But, with data backups in place, our DBaaS was able to help us restore that data quickly—and without us stressing over it as we would have otherwise. At the same time, the DBaaS tools allow us to recognize and address database issues while improving indexing and performance.

    — Gar Higgins

  • 5 Ways to Keep DBAs Happy, Alive and Efficient

    5 Ways to Keep DBAs Happy, Alive and Efficient

    As DevOps adoption continues to grow, organizations are competing to find the best and brightest application developers available. Meanwhile, database administrators (DBAs) are hard at work in the background to keep up with the burgeoning number of databases platforms developers deploy.

    DBAs have it rough. If developers are the James Bonds of the IT world, then DBAs are the Qs who quietly toil away to create the next big toy that Bond will destroy.

    The average DBA in a large enterprise can be tasked to manage 40 or more databases spread across various development stages. These disparate and sometimes rogue databases compete for the same compute and storage resources. Nearly all come with their own reporting and management tools. And when they break or fail, any number or combination of factors can be blamed.

    Database growth is especially difficult to control when companies such Amazon Web Services make it easy for any developer to create one. The AWS site, for example, provides just five easy steps to create a MongoDB database. Compare that to the time and work that goes into hiring, training and retaining more DBAs.

    Overloaded DBAs can overlook issues that lead to data loss or missed insights that affect revenue and productivity. If you can’t establish policies that control database growth, then think about ways to help your DBAs keep up and stay ahead of the game. Here are five possible tips:

    Focus and eliminate distractions

    DBAs can sometimes be lumped into general, overarching IT roles including help desk, backup and recovery or system upgrades. You probably wouldn’t want your developer to multitask, so why ask the DBA on the back end to do that? Give DBAs the opportunity to move beyond the firefighting role to focus more on proactively improving application performance. In the long run, this approach will eliminate fires before they occur.

    Give DBAs time to train on new databases

    By eliminating outside distractions, DBAs have more time to spread their knowledge across traditional databases including MySQL, Oracle or Microsoft SQL, and develop their expertise on more modern open-source databases such as Couchbase, Cassandra or PostgreSQL.

    Centralize monitoring tools

    The management tools that come with most databases provide a limited view of their broader environment. Centralized tools can provide visibility across the entire application stack. They show how databases interact with shared resources and unveil interdependent factors that lead to failures and performance issues. A centralized tool allows DBAs that “Monday morning coffee” view of all databases on one screen to determine which one needs the greatest or more immediate attention.

    Deputize your DBAs to enforce database policies

    Give your DBAs the power to evaluate and regulate new databases. This role is difficult to maintain if you confine DBAs to a traditional operations team. If databases can cross boundaries, so should your DBA. They should be empowered to seek out and evaluate databases that might be redundant, resource-intensive or unused. This means giving them the ability to say no to new databases unless they have a clear and compelling argument to be created. DBAs also deserve the authority to create a common set of goals, metrics and SLAs that must be met by all databases.

    Hire and treat your DBAs like rock stars

    It is becoming increasing more challenging to find and hire talented DBAs—especially the ones who can multitask across multiple database platforms. According to a survey in 2015, “The Vanishing Database Administrator,” more than 41 percent of database professionals surveyed said they intended to leave the data management field within the next 10 years. Possibly because of the impending shortage, the DBA profession is now seen as one of the hottest IT careers over the next several years. CNN Money ranks it as No. 9 in its 2017 Best Jobs in America. Employers should give particular consideration to these candidates and do everything they can to retain them.

    If you want your data to continue playing a central role in your business, give your DBAs as much recognition and credit as your developers. Behind every  developer, there’s a DBA keeping applications and data running at top speed. Remember Bond could never be successful with his Q.

    About the Author / Cameron Jones

    Cameron Jones is the Principal Product Manager at Blue Medora. Based in Grand Rapids, MI, he has been with Blue Medora for over 5 years working with partners and customers to ensure the highest value of Blue Medora products. This has involved initial investigation, feature planning and ensuring customer satisfaction. Connect with him on LinkedIn and Twitter.

  • Enterprise Cassandra Deployments

    Enterprise Cassandra Deployments

    Successful Cassandra deployment in the cloud takes a certain amount of skill and knowledge. In this blog I will share best practices for a successful Cassandra deployment in the cloud such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). I will also discuss how you can protect and recover the data in your Cassandra cluster when deployed in the public cloud.

    Cassandra Deployments in the Public Cloud

    There are two main ways to have successful enterprise Cassandra deployments in AWS:

    • Use a ‘managed’ Cassandra deployment. This is a reasonable approach; the only downside is the inability to customize data placement beyond what is typically provided by the vendor.
    • Manage the cluster yourself, if you want to have full control over your Cassandra deployment and customize it to scale and perform according your business needs.

    Please refer documentation from Datastax or Apache Cassandra for detailed best practices for Cassandra deployments. The following best practices are reiterated here to simplify deployment of a scalable Cassandra cluster in AWS.

    Storage

    Both AWS and GCP offer two types of local storage options for their compute instances. For AWS, the options are Elastic Block Storage (EBS) and Ephemeral Storage. In GCP, the options are Persistent Disks and Local SSD. Both options have their own trade-offs. Pick the one that best suits your deployment.

    Persistent Storage

    EBS volumes have performance characteristics similar to network storage such as a NAS or SAN. Because storage from EBS volumes is multitenant in nature, it is not suitable for distributed, log structured databases such as Cassandra. If you want to ensure reliable performance through EBS volumes, use “EBS optimized instances” with “provisioned IOPS.” Since the IO traffic to EBS volumes goes over the instance network, EBS optimized will provide reliable performance, but does not guarantee any increased performance. To get the most performance out of EBS volumes, use instances with multiple disks and separate the volumes used for “Data” and “Commitlog.”

    If you have to use EBS Volumes, go with “EBS Optimized Instances,” with “Provisioned IOPS” and split data and Commit Log on different volumes.

    Ephemeral Storage

    Ephemeral storage is a great option to use for downstream Cassandra environments such as test/dev and staging. Since ephemeral volumes are typically SSDs, there is no need to separate the data and commitlog into separate drives. To optimize performance, choose EC2 instances with multiple ephemeral storage drives and stripe them together into one logical volume. Send data and commitlog to the logical volume. Remember that when you stop the EC2 instance, the data in Ephemeral storage will be deleted.

    Stripe all Ephemeral drives into a single volume to get maximum performance.

    Data Distribution

    Cassandra uses Snitch to determine the topology of the cluster. The Snitch is also used to determine the data center and rack for each node in the cluster. Replication strategy is used to determine the placement of data across the different nodes in the cluster.

    AWS

    1. EC2Snitch works well for single region deployments.
    1. Ec2MultiRegionSnitch works well for cluster distributed across multiple AWS regions. Please note, Multi-DC, Active-Active configuration makes your cluster resilient to region failures and for multi-dc cluster, use “local quorum” to improve performance.

    Google Cloud Platform (GCP)

    1. Use GoogleCloudSnitch for Cassandra deployments in GCP.
    1. GoogleCloudSnitch treats regions as Cassandra data centers and availability zones as racks.

    Other Features

    Both AWS and GCP offer a series of operational capabilities that seem to make it easier to maintain high uptime on Apache Cassandra cluster. These capabilities have to be chosen carefully depending your deployment needs.

    1. Load Balancing: Network-level load balancing service is offered by both AWS (Elastic Load Balancer) and GCP. It is not recommended to put a load balancer in front of Cassandra. It is better to use a client connector that can handle this, consider open source utilities such Presto and Astyanax
    1. Autoscaling: Both AWS and GCP offer Autoscaling facility for compute instances. However, although this functionality looks very enticing, putting all C* nodes in an autoscaling group will not produce the desired result. Instead, create a separate autoscaling group of ‘1 node’ for every node in your cluster. This will ensure that anytime a node goes down, it is automatically brought back up. This will ensure the cluster data placement is not disturbed when nodes go down and nodes coming up do not cause excessive data movement.

    Scalable Backups and Reliable Point-in-Time Recovery

    Backing up your data is a must-have for enterprise organizations. The impact of data loss is so high that no enterprise wants to risk relying alone on replication. Protecting data in a public cloud environment may seem daunting, but there are several options available that can make this easier compared to an on-premise deployment. A simple option may look like one of the native snapshots:

    Storage Snapshots

    I have seen few customers use snapshots of EBS volumes as a way to protect the data in their cluster against logical and/or operational errors. This solution will work but with some severe limitations. On top of the overhead of maintaining the scripts to take snapshots, there is a large overhead in storage costs. Taking a snapshot of volumes does not allow for incremental backups of your data. This causes an exponential bloat in the amount of data stored in snapshots, leading to high cost of the backup solution.

    Amazon S3 or Google Cloud Storage

    Using an S3 bucket as the secondary storage enables enterprises to use low-cost, highthroughput storage, while reducing overall operating costs for storing backup data. Here are a few recommendations when using S3:

    1. S3 Region: Amazon S3 buckets are region-specific. A bucket created in one region will provide better performance characteristics to other infrastructure in the same region. Subsequently, it is important to ensure the S3 bucket used as the secondary storage is in the same region as the production database being protected and the Datos IO cluster.
    1. Storage Class and Lifecycle Policy: Amazon S3 allows users to create lifecycle policies on buckets. These policies dictate the performance and response time characteristics of objects in the bucket. STANDARD_1A is more suitable for longer lived and less frequently access data, and is not suitable as secondary storage. Additionally, do not turn on any lifecycle policies on the buckets used for the secondary storage. All buckets should be in the standard storage class.

    Conclusion

    As enterprises leverage the agility provided by public cloud environments, it’s important to understand the various facets of Cassandra deployments to ensure the efficient use of your money and time.