Tag: SQL

  • DevOps Chat: Scaling Data Without Complexity with Cockroach Labs

    DevOps Chat: Scaling Data Without Complexity with Cockroach Labs

    Cockroach Labs the folks behind CockroachDB recently announced another round of funding as they continue their mission to allow you to scale your data without complexity.

    In this DevOps Chat, CEO of Cockroach Labs Spencer Kimball talks about about their plans and how Cockroach shall inherit the DB world.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Alan Shimel: Hey, everyone, this is Alan Shimel from devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. On DevOps Chat today, I’m really happy to be joined by Spencer Kimball. Spencer is CEO/co-founder of Cockroach Labs—hot company in the DB and DevOps space. Spencer, welcome.

    Spencer Kimball: Thank you, Alan. It’s a pleasure to be here.

    Shimel: Absolutely. So, Spencer, let’s get it out of the way. First, I’m assuming most of our listeners have already heard of Cockroach Labs, but it’s possible they haven’t. Why don’t you give us a little background?

    Kimball: Well, sure. I mean, first, we should probably get the obvious question out of the way—why would we name something Cockroach? I think that’ll become clear when I give you the explanation, but Cockroach Labs was founded to build a database, and that database replicates and it’s hard to kill, so that’s where the name came from. Honestly, the name was invented before the product and long before it was a company. So, it was—you know, now I’m in a position of explaining that same thing to Fortune 500 CIOs on a weekly basis. [Laughter] There’s some regret there, but the name makes sense.

    So, I can give you a little bit of the background. All three co-founders of Cockroach Labs worked at Google for about 10 years in the office and Google went through Bigtable which started the NoSQL movement, and it quickly moved to Megastore, which introduced some transactionality. Then they created Spanner, and Spanner is something that Google’s been working on for more than a decade now.

    When we left Google, we realized Spanner is a solution to many companies’ problems, not just Google’s. And that’s become more and more true in the years since. We did a startup, and even as a startup, we wanted capabilities that Google had built into Spanner, and those weren’t available in open source. So, that’s sort of where the original idea of Cockroach came from. That startup got acquired by Square, the payments processing company, and many other things now as well. And Square needed something like Spanner as well.

    So, the idea kind of had gained currency. We started as a GitHub project. That took off, VCs got interested. The company was started, now, four and a half years later, we’re busy selling to the Global 2000, Fortune 500 big companies out there as well as competing on one side with Oracle and on the other side with Amazon.

    Shimel: Only on the Internet, man, you know? [Laughter] It’s a crazy story, but true. So, first of all, I absolutely get the Cockroach thing, because they are gonna inherit the earth, you know, or so it seems, especially with the path we’re heading, but let’s not get into politics.

    But it is kind of an amazing story, because when—for people who are in the IT business and are somewhat familiar with the history of the dashboard and how Larry Ellison came to be the second richest man in the world or whatever it is he is these days, you know, it’s fascinating that all of everything we do online, almost, is predicated on the ability to store the data and to access that data in ways that allow us to do all of, you know, manipulate the data, analyze the data. I mean, without databases, none of that happens.

    And so, when you look at the history of the database and then the, you know, to relational databases and then you mentioned a few other things, SQL database and then NoSQL databases—

    Kimball: And back to SQL. [Laughter]

    Shimel: Right, and then back to SQL, then you’ve got time stamped databases are a thing now. You know, all revolving around, though, how do we store this data in a way that’s easy to put the data in there and easy for us to access that data, manipulate it, analyze it, you know, and make value out of it, right?

    And so, much like a cockroach, data doesn’t die, either, right? [Laughter] It’s eating the world, you know? Where was I recently, I was shopping at Costco or something, and they had eight terabyte, I guess, backup or drives, you know, external drives that were maybe a little bigger than credit cards, you know? Size of a good iPhone, let’s say, a little thicker. And I think it was $169.00 for eight terabytes. [Laughter] And I’m thinking, “My God, you know, when I had those 3.5 inch floppy disks, man, and the bigger ones”—

    Kimball: And the first 20 megabyte hard drive, it was—

    Shimel: Yeah! I didn’t think there were eight terabytes in the whole world.

    Kimball: And there weren’t for a while, right?

    Shimel: Yeah, it’s crazy. And, you know, as our data storage and data needs have grown, so has our databases. So, you mentioned the spinoff from Google, you mentioned certain things that the latest, greatest from Google didn’t do and hence, why Cockroach.

    But you guys recently also announced a new funding round, if I’m not mistaken, right?

    Kimball: Correct, right. We’re going to—that will happen on August 6th.

    Shimel: Well, by the time people are listening to this, it’ll be August 6th already, so.

    Kimball: Got ya, okay.

    Shimel: If you’re listening to this at home, it’s after August 6, they’ve announced their funding round.

    Kimball: Yeah, so, we did close that, and it’s a big round. And basically, the additional money equals all the money we’ve raised in the past which, you know, isn’t so unusual, but it’s a lot of money. In aggregate, we’re talking over $100 million.

    Shimel: Wow.

    Kimball: You know, I think, to put that into perspective, you just have to look at who we’re competing with, right? I mean, certainly, it’s some of the biggest companies in the world, the biggest companies in the world, and we’re also serving the biggest companies in the world, so you do need that money both to create a sales force that’s capable of educating and reaching and selling to those companies, which all—I’ve been humbled and amazed by the process necessary to actually sell to a Fortune 500 company. In some cases, it’s quite involved.

    But also, to continue to iterate on the product, and we have to, you just need to hire the best people and create an environment where they can collaborate to move the product forward so that you can compete.

    Shimel: Absolutely. But, you know, look—in the Internet and the IT world today, compete means different things to different people, right? We compete by offering great service. You know, I just got done, I was over at DevOps Enterprise Summit in London a month or two ago, and there was this notion there of what folks were calling BVSSH—Better Value Sooner, Safer, Happier, right? And that’s what we try to deliver in DevOps and stuff like that.

    So, does competing on that and—but then, in the database world, too, it’s just really, sometimes it’s very, very, very feature driven.

    Kimball: Yeah.

    Shimel: Right?

    Kimball: It’s incremental features, and then there are sort of generational features. And we’re, we’ve been pushing on truly changing the nature of the database, and that is an opportunity that exists because of all the other things happening in the ecosystem—in particular, the global public cloud, which is, I think, one way to describe the combination of AWS and GCP and Alibaba Cloud and Azure. There’s many others, of course, that are far more regional. But in aggregate, all of those can be utilized by new database architectures to allow a company to store data in new ways that ____ their business versus their competitors.

    So, just to give you an example, things like GDPR and recent regulations around privacy and Russia and China and Vietnam and South Korea and South America—the list goes on—there’s a lot of attention being placed on how data is localized, where it’s stored in relation to the customer, where it’s processed.

    And also—and this is, I think, really exciting—there is an ability, because of this global public cloud, for new services to come online that are able to deliver the end to end user experience that happens instantaneously to the user. So, that means in under 100 milliseconds, which is sort of the user threshold for noticing time intervals. If something feels instantaneous, but it happens in less than 100 milliseconds, you literally, because of physics and the speed of light, you can’t do that for a customer in Australia if all your data is in one data center in Virginia. But that’s the model that’s prevalent right now.

    Shimel: Sure is.

    Kimball: [Cross talk] on public cloud and with a solution like Cockroach, you’re actually able to have a globally spanning database that’s able to domicile data near the customer and actually deliver both the assurance of some data privacy, that their data is in process and stored close to them in their legal jurisdiction, but also to have an end to end user experience where you click a button on your mobile app, it goes all the way through the application logic, accesses the database, comes all the way back and updates. So, that ability to make that instantaneous, I think, is gonna open a new era of applications.

    Shimel: It sure is. You know, so, I remember, Spencer, back—so, one of the companies during the dot com days, one of the companies I helped build was a company called, it came to be known as Interliant, and we were an early ASP application service provider.

    But before we changed our name, we were Sage Networks, and we were a large hosting provider, a rollup. We had bought a lot of different hosting companies. And one of the key things for us, like, you know, what really brought us scale versus all these smaller hosting companies we had bought, is that all of these hosting companies’ storage and the data was sort of part and parcel with the same infrastructure, if you will, that was hosting the website and the web server and all of that.

    And what we did is, you know, we actually—I don’t wanna say abstracted, because it wasn’t really abstract, but we separated the data and the database from the web server and the applications they were running, right? And now, that freed us up to store that data. This is before there was a cloud, but we had multiple data centers. So, we were at the point, almost, where we could store data, let’s say, in our Atlanta data center and access it or use it on websites where the web servers were stored, let’s say, in Tysons Corner in Virginia at the time. And what was holding us back there was, we didn’t have the connectivity you have now.

    Kimball: Yeah.

    Shimel: Right? If you had a T3 line, that was hot stuff, you know? That was carrier grade. And so, it was a little different. But, you know, this is the age we live in now where you do have distributed applications. So, of course, why wouldn’t you have distributed data storage and the databases that organize them?

    Kimball: Yeah. I mentally have distributed audiences, right? I mean, every time a fledgling startup puts their first mobile app on the app store, the total global population that can have access to that, it’s pretty staggering. I mean, the opportunity there is just tremendous. But, you know, everyone’s—90 some-odd percent of efforts out there are focused on deploying to a single availability zone, right? So, this is, there’s a legacy of how things were built, which is rapidly changing, and you have to do it through the whole stack. At Cockroach Labs and CockroachDB, we’re attacking a fundamental part of that stack, very important, probably one of the harder parts, which is the data storage, the state behind those applications.

    Shimel: Yep. No doubt about it. I wanted to talk a little bit about CockroachDB sort of as a project, if you will. Cloud native, right? When you guys were starting out, obviously, you came out of Google, you have a big open source kind of heritage at Google—what were the discussions then about, “Is this something we open source? Is it something we make free?” How do we walk that line, right? Go ahead.

    Kimball: So, when I was at Google, from 2002 to 2012 with my co-founders, Google was only starting to do open source towards the end of that tenure. Previously, they were pretty adamant about keeping things closed source. It was just sort of the model that Google evolved with. Now, they’re much more open, and some of their more recent efforts, including TensorFlow and Kubernetes, are examples of where open source in the hands of Google actually is a tremendous advantage both to the company and to the larger ecosystem. But things like Spanner were closed source and many of the things that Spanner is built on are also closed source.

    But what Google did do, which is great, is they always publish about these things.

    Shimel: Right.

    Kimball: When we started Cockroach, we had the benefit of not only working at Google and seeing how these things were built—you know, Peter, one of my co-founders, and myself worked on something called Colossus, which is the exoskeletal distributed file storage system at Google, which runs underneath Spanner.

    So, we had a really great opportunity to see why and how Google built the things that they built. But, you know, once we were kinda kicked out of the Google nest—or we left it, I guess, is the better way to say it—

    Shimel: Flew the nest. [Laughter]

    Kimball: Yeah. We flew the coop, right? We had the benefit of all of the published work that Google had created around things like Spanner, but you know, out in the wider world, you realize, for a database—and when we started this, I guess it was 2014. In 2014, and it’s certainly true today, people aren’t going to pay attention to your database unless it’s open source. And it’s just, it’s sort of a fundamental requirement. And there have been databases that have tried as closed source software, but they haven’t gotten very far, to my knowledge.

    So, I just think that—and this is true for me, too, as a potential user of the database, as a potential developer on the database, something that’s not open source feels a little bit icky. [Laughter] Right? Like, there’s not enough surface area that I can latch onto where I feel like I can be the master of my own fate, right?

    Shimel: Yeah.

    Kimball: Like, if something—that’s what I hated about closed source products. If something goes wrong, it’s just this big black box and, you know, it’s sort of a black hole if you wanna send a support ticket in, what’s gonna happen? Whereas it would be much nicer for everyone involved, in my opinion, if I were able to dig down in there, find out what was wrong, probably my fault, but let’s just say it’s a bug in the product, I could send that into the company and they could take that as a patch.

    That’s the beauty of open source and that, as a personal preference and sort of even as an idealistic sort of mindset. I would not have been able to start the database as anything except for open source.

    Shimel: Yeah. I mean, and look, quite frankly, there is a bit of a history in the database space of successful open source databases that not only were successful as open source projects, but were equally or more successful as commercial entities or the companies that manage these open source projects—you know, very successful open commercial entities that had great exits and great value for the investors and founders and employees.

    So, I mean, in that regard, it’s not like you’re pioneering, right? It’s not like someone says, “Oh, they’re crazy to do this in open source,” right? Well, 15 years ago, they might have said you were crazy, right? [Laughter] You know—

    Kimball: I think MySQL kind of, you know—

    Shimel: Yep, ____.

    Kimball: —blazed the trail, as they say. And Mongo is a very successful—

    Shimel: Absolutely.

    Kimball: [Cross talk] that uses open source as well. And there’s a number of other ones. But you know what, even when we started this company in 2015, I think the open core model and open source in general seemed like a catalyst, right? Just something that could create business value far more quickly than closed source software and the sort of procurement department and budgets and all this stuff that used to be involved in the cycle of adopting a new technology in order to build your products or services that the engineers or developers at a company needed to—

    Shimel: Oh, absolutely. I mean, look, I’ve been in IT 30 years, right? I’ve seen the rise of open source. I was involved in open source security stuff in 2001, ’02, ’03, ’04. The deal is, today, for a lot of developers, if it’s not open, they don’t wanna work on it, number one.

    Kimball: Yeah. But—

    Shimel: Go ahead.

    Kimball: What’s really fascinating is that 2015 might have been sort of the beginning of the end for open source being the true, the most important catalyst of business value. I would argue that, in 2019, the true catalyst to business value, faster business value—business value in sort of relation to your competitors, I think it’s services, right? Cloud-based services. That is—you know, it’s always about, it’s not about the idealism of open source. That’s not why it was successful.

    Shimel: No, I think we’ve passed the idealistic stage of open source.

    Kimball: It’s basically like, what are the tools out there that a software engineer at company X can use in order to get their project done faster so that it’s more stable, it’s more performant, it costs less, right? All those things—

    Shimel: But it’s their preference, and these developers and the Ops and the DevOps folks, they’re the top predator on the food chain at a lot of these companies. They’re the hardest piece of the puzzle. And so, companies are hesitant to tell them, “No, I don’t want you to use that tool” or, “I don’t want you to use this service.” They’re the ones making these choices, where I think, in years past, it was like, “Oh, no, you can’t bring that in here” or, “You can’t use that one” or something like that.

    Kimball: Yeah. It kinda [Cross talk].

    Shimel: But there’s more, Spencer. Let me tell you this. You know, I’ve been going to a bunch of the Cloud Native cons and Kube cons and those kinds of shows, and I spend my life at IT conferences. I’m telling you that the open source community—and not because of the idealistic ____ and the bizarre and free as in freedom and free as in beer kind of crap.

    We’re past that, you know, Dr. Stallman stage, if you will. It is the, that community is the energy driving almost everything we’re seeing in IT today, right? There’s such an energy in there that it’s driving what you’re doing at Cockroach, for sure. It’s driving what we do in DevOps. It’s driving what we see in Cloud Native and Kubernetes and all of these things. It’s where—it’s almost like this well of, like, a nuclear power plant or something, you know what I mean?

    Kimball: Yeah, absolutely. And, you know, I think the best model right now that I’m aware of is, you know, it’s called cloud-based services with a very fast time to value, incremental pricing, right? So, you kinda pay for what you use as you go. If you can deliver that with an open source product that’s backing it that people feel that they can get into and they can use if they had to move off that sort of freedom and flexibility, that sort of offers the best of all worlds.

    And I think that—and that’s why Amazon’s been so successful in adopting open source technologies, because they’re very good at providing things as a service and they have so many things altogether into one, you know, that can be packaged into one bill, one sort of monitoring platform, one identity access, you know, management platform. It’s very impressive, but, like that, in aggregate, creates so much value for business.

    Shimel: It’s crazy. We’re running low on time, so let us bring this back to Cockroach Labs. We seem to have taken off there on a tangent. You guys are offering a managed version.

    Kimball: That’s right. So, we have—some of our biggest customers, they wanna run the database themselves, it’s the way they’ve been running it, but over time, that will shift, of course. So, we have a self-service, self-hosted version where you buy subscription licenses if you wanna pay us for it. There’s also, of course, open core, which you can use for whatever purpose.

    Then there’s the managed service offering, which we’re calling Cockroach Cloud. And that, we run it for you, and we can run that on AWS or GCP, and the pricing model is much more like something like DynamoDB or Amazon Aurora or MongoDB Atlas, those are sort of similar kinds of offerings.

    Shimel: Got it. Excellent, man. Just, again, Spencer, for people who may not be familiar, the website is CockroachLabs.com.

    Kimball: That’s right.

    Shimel: So, that’s where people can go get more information, there. Any other news that we didn’t hit on, here?

    Kimball: Well, you know, I think maybe we didn’t quite cover the funding well enough. I kinda jumped ahead, but we’re raising more than 15 million and the lead investor is Altimeter Capital.

    Shimel: Great.

    Kimball: So, there’s sort of growth equity added, along with Tiger Global, and then all of our existing investors, Google Ventures—GV is what they’re called now—Benchmark, FirstMark, Redpoint, Index Ventures. So, it’s a great—

    Shimel: Impressive list.

    Kimball: Yeah, a great group of investors. We’re looking forward to sort of this being an inflection point and really moving forward with our penetration into the Global 2000 and also fundamentally putting a lot of focus on managed services, so, the cloud delivered Cockroach experience, in addition to enabling more and more of this global architecture capability that I was talking about earlier.

    Shimel: Absolutely. I mean, it takes resources today to make all of that happen and bring it all to market, obviously, and hiring good people is not—even with the money, it’s not an easy thing to do, as I’m sure you’re finding out. Finding good people is—

    Kimball: A lot of these guys have—I liked what you said, at sort of the top of the food chain.

    Shimel: Yep, the alpha predator in the thing. It’s crazy.

    Kimball: They vote with their feet, right? So, it’s—

    Shimel: Yeah, they do. I mean, they don’t even ask for raises, they just go get another job. It’s not—I mean, it’s a crazy, crazy dynamic at play there, but it is what it is.

    Well, listen, this was—obviously, we blew past our 15 minute time, but it happens, it’s all good. I want to wish you success. I mean, just a quick digression from the point—you know, there’s a lot of people out here who are gonna listen to this, Spencer, who will say, “You know, I have a great idea for a company. We should go raise a few million bucks in capital, and we can do similar to what Cockroach is doing.” Not necessarily in the DB perhaps, but you know, in startup world, it’s not easy—$100,000,000.00 is a lot of frickin’ money for people to write checks to you for. And it doesn’t come without tremendous responsibility, and having raised a lot of money in my lifetime for venture backed companies, I know what goes into it.

    So, congratulations on a successful run with this, and looking forward to seeing what you do with it.

    Kimball: I appreciate that, Alan.

    Shimel: No problem. Alright. Our guest was Spencer Kimball, CEO, co-founder of Cockroach Labs, makers of the CockroachDB, and this is Alan Shimel for devopsy.kinsta.cloud. You’ve just listened to another DevOps Chat.

    — Alan Shimel

  • The Linux Foundation Breathes New Life into Osquery

    The Linux Foundation Breathes New Life into Osquery

    Facebook’s useful but neglected DevOps tool Osquery has gotten a new lease on life, thanks to The Linux Foundation.  

    Anyone who has been tasked with monitoring the security of server instances in a data center or cloud knows how laborious and time-consuming it can be. Osquery, a project started by Facebook, aims to lessen this burden by reframing how developers engage with their infrastructures. DevOps professionals can use Osquery to expose an operating system as a high-performance relational database, making it possible to use SQL commands to access data about a system, just as they would for a database. 

    Osquery works on Mac, Linux and Windows systems and is provided as an open source download via GitHub. Although Osquery was developed by Facebook to monitor and safeguard the security of its own platform, the social media giant quickly realized the utility of the platform would extend to other enterprises that depend upon insight into the low-level behavior of operating systems. 

    Facebook publicly released the project’s code and documentation in 2014, in response to feedback from a small number of other companies that tested Osquery. The public availability of Osquery resulted in a number of major companies including Airbnb, Dropbox, Netflix, Etsy and Uber to bring Osquery into their software development environments. Nevertheless, users have voiced frustrations with Facebook’s handling of Osquery, and have accused the internet giant of neglecting the project.

    To address those complaints, The Linux Foundation has taken over the mantel of Osquery and recently announced the formation of a new foundation to support the Osquery community. 

    The new Osquery Foundation brings together engineers and developers from Dactiv, Facebook, Kolide, Trail of Bits, Uptycs and many others committed to supporting the project with technical contributions and long-term stewardship. A Technical Advisory Board will be created to handle the shift in management from Facebook to the Osquery Foundation and will oversee the priorities initiated by the foundation’s members.

    This restructuring of management promises to extend the usability of the project as a number of developers contribute to the codebase.

    “Osquery has drastically simplified the process of operating system monitoring, which has unlocked new methods for securing infrastructure, detecting anomalies and more. We look forward to working with the Osquery community to develop the foundation in a manner that will continue to foster the growth and adoption of Osquery while supporting the community’s diverse needs,” said Jim Zemlin, executive director at the Linux Foundation.

    Developers working to maintain the security of their systems believe it would be wise to incorporate Osquery into their workflow. “Although Osquery will never do everything that a security team needs, it gets about 80% of what you need for endpoint insight in one package,” said Doug Wilson, director of security for Uptycs.

    Although the platform requires some configuration, developers familiar with SQL will feel accustomed to using Osquery to retrieve information about their systems. Furthermore, Osquery’s cross-platform support and customizability make it especially attractive to businesses with newer types of infrastructures and ones that work at scale. 

    The move away from Facebook toward a more community-friendly foundation is sure to breathe new life into the project, and as development on Osquery expands, it is likely to become a more valuable resource for security teams moving forward.

    This article was co-authored by Tyler Ohlhorst, research analyst and freelance copy editor and copywriter at Magnum Consulting.

    — Frank Ohlhorst

  • Redgate enhances the appeal of compliant database DevOps with new provisioning dashboard for SQL Clone

    Cambridge UK, Wednesday 24 October 2018 – The leading Microsoft SQL Server tools vendor, Redgate, has introduced a new at-a-glance dashboard to its database provisioning solution, giving users an easy way to manage, organize and make available masked copies of databases.

    Copies of production databases are commonly used in development to test changes against and ensure they are not breaking changes. Shifting left the testing in such a manner is one of the ways a DevOps approach can be applied to the database in order to speed up development and minimize deployment errors.

    Redgate’s provisioning solution uses the tried and tested virtualization technology built into 64-bit Windows to create database copies, or clones, which are a fraction of the size of the original. While they work just like normal SQL Server databases and can be connected to and edited with any program, they typically take up only megabytes of space, even for databases of 64TB in size.

    As importantly in an age when data breaches are prompting the introduction of stricter data privacy legislation, the provisioning solution also enables sensitive data in the clones to be masked at the time they are created.

    Rather than simply randomizing the data, it uses pseudonymization and anonymization techniques to ensure the database is truly representative of the original and retains its referential integrity and distribution characteristics. The sanitized copies can then be used to speed up development, accurately test code, and fix issues faster. Their small size also encourages a dedicated development model where developers are free to have their own personal copy of the database to work with.

    The increasing demand for and use of masked copies, however, can make managing multiple copies across disparate teams a complicated task. To address this, Redgate has developed a dashboard to make the searching, grouping, renaming and management of copies fast and easy.

    The copies in use can now be grouped by location, the database version they were created from, and the instance they are on, and searched by name, creation date or size from the dashboard. Importantly, if a developer tests a change which breaks their copy of the database, it can be reset to its original state in seconds rather than having to be deleted and recreated.

    The activity view additionally gives Database Administrators an instant picture of what has happened over the last seven days, who did it, and any errors that occurred. For errors that occurred previously, the log files can be retrieved in a couple of keystrokes.

    To increase security and further demonstrate compliance with data protection legislation, Active Directory role-based access can also be used so that users without appropriate permission only have access to masked copies of the database.

    As Rebecca Edwards, Redgate Product Marketing Manager, concludes: “At Redgate, we’re constantly talking to companies to discover what challenges they face in their desire to develop applications and databases faster while also keeping data safe. This latest addition to our database provisioning solution will help them comply with data protection legislation – and demonstrate compliance – while making their jobs easier.”

  • DBmaestro DevOps Platform 7.5 released with PostgreSQL Support – DB2, SQL, Oracle, and PostgreSQL

    Manages multi-database environments for DB2, SQL, Oracle, and PostgreSQL

    BOSTON, MA, October 11th, 2018 – DBmaestro, the pioneer and leading DevOps for database solution provider announced today the launch of DBmaestro DevOps Platform 7.5 which enables CI/CD and release automation for PostgreSQL. PostgreSQL is becoming the preferred database for more and more enterprises and smaller organizations. It is currently ranked #4 in popularity amongst hundreds of databases worldwide (See more info at DB-Engines Ranking).

    DBmaestro’s DevOps Platform 7.5 includes database release automation solutions to streamline database changes with confidence and control while enabling companies to reduce the time to market for their applications. The fast adoption of continuous delivery and deployment methodologies to accelerate application development has created a strong demand for database release automation tools to prevent errors and downtime.  According to the 2018 DevOps for Database Report, 83% of companies have already embraced DevOps, while continuous delivery for databases lags behind, with only a 37% adoption rate.

    DBmaestro’s DevOps Platform 7.5 zero-friction solution provides the ability for DevOps teams and database professionals to manage multiple databases more easily, without having to change existing processes.  Providing an overarching solution for CI/CD adoption across platforms and operating systems including Oracle, DB2, Microsoft SQL Server and now PostgreSQL, DBmaestro meets the evolving database release automation needs of enterprises across industries and borders.

    “Our DevOps platform unleashes the power of PostgreSQL databases by enabling DevOps and database teams to implement DevOps best practices and accelerate time to market with a high level of confidence, minimal risk and maximum efficiency,” said Yaniv Yehuda, CTO at DBmaestro.

    DBmaestro will be demonstrating DBmaestro DevOps Platform 7.5 solution at DOES18  from October 22– 24th in Las Vegas, Booth #811.

    About DBmaestro

    DBmaestro brings DevOps best practices to the database, delivering a new level of efficiency, speed, security and process integration for databases. DBmaestro’s platform enables organizations to run database deployments securely and methodically, increase development team productivity and significantly decrease time-to-market. The solution enables organizations to implement CI/CD practices for database activities, with repeatable pipeline release automation and automatic drift prevention mechanisms. For more information, visit www.dbmaestro.com.

  • Jobs-as-Code: The Business Processing DevOps Forgot

    Jobs-as-Code: The Business Processing DevOps Forgot

    Developing web and mobile apps is cool. But in the words of a longtime ballad: “Don’t forget who’s taking you home and in whose arms you’re gonna be …” It’s the back-end services that underpin a lot of those cool apps and make them possible. So yes, servers and relational databases and enterprise applications and sometimes even mainframes are definitely a thing. And in those environments, scripting, batch processing, application integration and operational visibility are a very big thing that, more often than not, is managed by workflow and job scheduling tools.

    I argue that the “something—application, code, other” elements used to deliver all those capabilities are logically part of the application and should be treated as such. The fact that “process X” needs to start only after certain data has arrived, been cleansed and inserted into some database is just as much a part of the application as calculating an invoice amount or determining that inventory needs to be replenished.

    And if you are still reading, then I submit that you need jobs-as-code.

    Simply stated, jobs-as-code means that the jobs—automation rules for business applications—you need to manage and execute systems of record, back-end application components and data pipelines in production are built and managed in the same way as Java, Python, C++ or any other business logic code. In other words, everything that is part of the application should be included in the software development life cycle (SDLC).

    Of course, that statement is pretty simple to say but not that easy to accomplish, because traditional workflow, job and batch processing solutions have lived (and most continue to) in the operations world. The practitioners of enterprise scheduling tend to prefer graphical interfaces with a strong emphasis on ease of use and rigid control over access. Traditionally, capabilities such as developer self-service, automated build and test and deploy have not been high on most enterprise job scheduling new feature requirements wish lists. Most commercial solutions, if they do anything, add an API or two and consider the job done. Open-source authors build tools that cater to developers with little to no regard for the multitude of other user types that rely on batch workload.

    The resulting state in the industry is that developers spend lots of time building “snowflake” operational plumbing which may be difficult to test, difficult to embed in an automated delivery pipeline and either has to be reworked to make it ready for production or becomes an albatross around the necks of operations teams trying to keep it running. It’s time for the DevOps community and the industry to recognize there is a better way and we all instinctively know what it is.

    Jobs-as-Code

    To understand jobs-as-code, let’s examine the traditional dev-to-ops handoff from a developer and DevOps perspective.

    I write some code and now I want to test. The execution is simple; I want to pull some data from a relational back end, run my code to manipulate the data and push the result set to some other location to serve as a data source for a different application. I need a SQL query, run my Java code and ftp or sftp the result.

    I’ve used some interactive tool to extract my test data, debugged my code in Eclipse and just browsed the output with Notepad++. Now I want to construct a flow that either I or any of my teammates can run whenever they want.

    Today, I probably brush off my scripting skills, bash or perl or whatever, and write a quick script to run a SQL query, run my java code and then run the file transfer. Easy, right?

    Maybe not.

    Connecting to the database interactively was so trivial I didn’t even think about it, but doing it in the script is a whole different exercise. I’ve got to get my quotes and escape characters right; I have to make sure I connect without compromising the credentials needed to log in to the database; I even have to figure out what return codes I’m getting and what to do with them. Of course, I need to add some notification so that I’m informed if there was a problem.

    OK, got all that done. Now running my Java code is a piece of cake right? Well, it is when running on my machine with my credentials but maybe my teammate has a different environment. Or he’s a .net guy and doesn’t have Java configured the same way as me. Or worse, maybe doesn’t even have it installed. Better add the code to check for all these problems and notification for that, too. And when I get that failure notification, what do I look at to figure out what went wrong? I’d better pipe everything to a log and make sure others can view the log, too, so I make sure I put it on a shared filesystem that others can access.

    Let’s not event start on the file transfer! Or, when you have to add additional steps or when you have to run this a 3 a.m. as part of some testing that another group is doing or making sure that database you rely on is up and running, etc., etc., etc.

    Eventually, after way more time than I ever thought it would take, that massive script is done. I run it through some test cases and I’m done.

    On to the rest of the delivery phases with more testing and probably some more extensions that make the script truly epic.

    In those last few stages of getting to production, someone may raise the issue that this new application interacts with some other application that’s managed by some production automation tool. OK, let the rework—or, really, brand new unplanned work—begin. The developers aren’t familiar with the tool and the tool administrators aren’t familiar with the application so you can imagine how well those conversations go. However, they are all professionals and eventually manage to get the job done and the application proceeds to production.

    If this long, drawn-out story didn’t already make you uneasy, you may have caught the huge fail that is now just waiting to blow up in our proverbial faces. Most, if not all, the testing was already completed. The new work done at the eleventh hour gets minimal, if any testing, is deployed and—surprise!—has a high failure rate in production.

    To add insult to injury, somewhere down the road after all the heroics of getting the application deployed are long forgotten, a failure occurs, as is only natural and to be expected. The poor on-call folks must negotiate through that production tool they aren’t familiar with and isn’t part of their world. And then they encounter that massive, epic script. What’s going on inside of that thing and where did the failure occur? Where are the logs and do we even have all the ones we need? And saving the best for last, after the problem has been identified, how do we restart this process in the middle when half of the processing was already completed?

    So What Would Nirvana Look Like?

    What if you could create a database query by just coding the SQL statement(s), picking the database you wanted from a list of logical names and requesting notification in case of error and you could express those requirements in a few simple statements? If you want to run it at 3 a.m., you just add that requirement too and then say “run.” The availability of the database is checked, the credentials are secure, there is no scripting, log and output are captured automatically (not just from this run but as many runs as you want, and you can compare one run with another, etc.) and you get whatever notifications you requested. The logic would be solid because it’s been in use for years by organizations like yours, rather than being written under pressure while trying to meet a deadline.

    You could write your automation rules (jobs) in some simple notation you were already familiar with such as JSON, store it together with your Java code in git or whatever SCM you use. The jobs could be packaged with the rest of your application and your build, config management and testing tools and could push your entire application including your jobs through the entire delivery pipeline so there are no surprises or additional work required before deploying to production. The facilities to deliver operational visibility, support problem analysis and log and output management would be second to none because they have been evolving with input from community and IT professionals.

    Imagine if this magical-sounding solution that implemented Jobs-as-Code was available from github and other public sources? You could download, install and have access to Jobs-as-Code in minutes. You wouldn’t have to submit access requests to IT folks, no purchase requisitions, no messy and complex installations.

    You don’t have to imagine. Jobs-as-code solutions already exist. If the tool(s) you’re using doesn’t give you these capabilities, switch! At least start talking to your peers and let them know the better way has arrived. With sufficient demand from the community, the vendors and authors implementing the myriad job management tools may just accommodate you.

    — Joe Goldberg

  • The Coming Earthquake  in IIS and SQL Configuration Management

    The Coming Earthquake in IIS and SQL Configuration Management

    A marketing-free, engineering-led webinar: What was “state of the art” in IIS configuration management has changed a lot in just a few years. It is about to change a lot more. Hear DevOps and IT automation practitioners and visionaries outline recent IIS configuration management evolution and contrast that to the coming revolution. Q&A afterwards.

    RECORDING

    SLIDES

    [slideshare id=75142033&doc=orcawebinar-thecomingearthquakeiniisandsqlserverconfigurationmanagement-170418175248]

     

    — Jules Louis

  • Does MariaDB replace the “M” in LAMP

    Does MariaDB replace the “M” in LAMP

    If you were starting a new application tomorrow, which backend would you use? Chances are you said MySQL. However post Oracle acquisition MySQL has changed. After Oracle purchased the code they broke the hearts of many. And there is hope that MySQL has re-invented itself in the form of MariaDB

    (more…)

  • SQL lovers fear not – NoSQL just got friendlier

    SQL lovers fear not – NoSQL just got friendlier

    Today Couchbase, the creator of an Enterprise NoSQL database server, announces N1QL ( pronounced Nickle) at their user conference. N1QL is designed to ease the pain for SQL developers when considering NoSQL, by creating a new query language to sit on top. It is essentially SQL for JSON documents.

    (more…)

  • SQL vs. NoSQL – Unsettled Debate & DevOps Pitfalls

    SQL vs. NoSQL – Unsettled Debate & DevOps Pitfalls

    Big Data, a relatively new phenomenon driving the industry hype-cycle is a mainstay in the database business. IT companies are convinced over the endless value proposition of the operational insight that comes with leveraging Big Data ( or Big Small Data ), so much so that it’s easy to overlook its impact on traditional database technologies and practices.

    (more…)