devopsy.kinsta.cloud

Tag: mongodb

  • MongoDB’s Atlas Turns Five

    MongoDB’s Atlas Turns Five

    In June 2021, Andrew Davidson, VP of cloud products at MongoDB, talked with Alan Shimel about MongoDB and the fifth anniversary of launching Atlas, the company’s database-as-a-service offering. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Alan Shimel: Hi, everyone. Welcome to another segment here on TechStrong TV. Our guest for this segment is Andrew Davidson. Andrew, welcome to TechStrong TV.

    Andrew Davidson: Thanks, Alan. Good to be here.

    Shimel: It’s great to have you on. Andrew, I purposely didn’t say a lot about – I just said Andrew Davidson. I didn’t give your company, your title, or anything. I’m gonna let you do that, if you don’t mind.

    Davidson: Well, thank you very much. Yeah, so I’m Andrew Davidson. I’m VP of cloud products here at MongoDB. I’ve been here nearly nine years. I’m a Silicon Valley native originally, but I moved to New York City and found this amazing opportunity here. MongoDB, for those who don’t know, is actually headquartered in New York City. And it’s been an amazing journey since that time, just seeing this incredible explosive growth out there over this time.

    Shimel: Absolutely. Y’know, if you couldn’t tell from my funny French accent, I’m from New York originally. So I got into tech around 1995, 1996. And at the time, I think the mayor of New York was Mayor David Dinkins, and he had started a project that they were calling Silicon Alley. Right?

    Davidson: There you go.

    Shimel: It was downtown. And I actually had an office – first company I sold – on John Street, 55 John Street. But it was shortly after that – eh, shortly; it was probably seven or eight years after that at least, I bet, that Mongo started in that Silicon Alley, New York kind of cauldron of tech stuff.

    Davidson: That’s exactly right.

    Shimel: Yeah. Oh, I remember. I was in New York then. And it was a point of pride, actually, right? I mean, it wasn’t Oracle. We didn’t wanna be Oracle, right? It was the new guys, the new sequel guys, and they made a name for themselves and did well, right? And so I’m glad to hear you made that move from Silicon Valley –

    Davidson: Totally.

    Shimel: – to New York, and it’s a pretty cool thing. Of course that, as I said, was 15, 17 years ago, right? We’re here to celebrate a birthday/anniversary today, too, and that’s the – If I’m wrong, Andrew, correct me. It’s the fifth anniversary of Atlas from MongoDB, right?

    Davidson: That’s exactly right. We launched MongoDB Atlas, which is our global cloud database service, five years ago this month, and we’re just so excited. We’ve come so far. We’ve reached the psychological milestone this same month of 51 percent of our company revenue growing more than 70 percent year over year. We all know that this is the future of the platform. This is the foundation on which we can deliver really the world’s preeminent application data platform. And the value –

    Compared to where we were, y’know, when you mentioned when MongoDB was coming out of Silicon Alley so many years ago, it’s just a completely different universe today. At that time when MondoDB was first created, I think it was largely about timing. There was this combination of factors, giving developers this document model that was totally native for them, and also democratizing a distributed system right at the time in which cloud was democratizing commodity hardware.

    That combination led MongoDB to explode. But this idea of self-managing databases that we realized six, seven years ago, that was gonna go away. People are gonna move up a whole level of abstraction. It’s gonna be all declarative. An end-point in the cloud you can read and write from, and build wonderful applications – that’s what MongoDB Atlas was all about. And five years in, over 25,000 customers, over 100,000 developers signing up every month, it’s just unbelievable.

    Shimel: It really is. Y’know, I mean, look. Again, having been there, the original attraction with MongoDB was “no SQL,” right? It was a no sequel database. Everybody was using SQL scripts, and whether you were using MySQL or MS SQL, it was – And then the idea of no SQL was really huge. But why was it huge? I think you hit it, Andrew. Number one, it was easier for developers to do what they wanted to do, how they wanted to do it. Number two, it was just lighter and easier and more fungible, if you will, especially as we were moving to cloud in 2005 and 2006, right?

    Cloud burst on the scene. And it lent itself. It lent itself to multi-nodes and all of these things much better than relational SQL database.

    Davidson: Totally.

    Shimel: But of course Atlas comes in as a service.

    Davidson: Absolutely.

    Shimel: And I think that – it’s more than just a service. A database is a service, or data is a service. But fundamentally I think that is the next leap forward, right? We went from like 2001: An Odyssey, right? Someone _____ to throw a rock, and that was a big leap forward. But it was the next leap forward with fire, that defined it.

    Davidson: That’s right. The obelisk, I suppose you could say.

    Shimel: Right, the obelisk, no, totally.

    Davidson: ‘Cause I mean, like you were saying a second ago, why is it so important for developers to have a data model that’s so native for them? Well, it’s because every company today has that incredible sense of urgency that they must be come software companies. In every vertical, every part of the world, everyone’s either fighting to keep what they’ve got or being challenged by upstarts. And you can’t buy this off the shelf any longer.

    You have to bring that DNA in house and really reflect your company’s unique value propositions and users and internal logistics and all of the above for software, and data is the heart and soul of software. It’s always been the hardest part of, frankly, building software. We change how we build software. We layer in microservice as next-gen frameworks, et cetera. But if the data model hasn’t kept up, and if the data model continues to be this sort of tabular model that goes back to the ’70s, a time in which storage was the key bottleneck in computing because it was expensive to have storage then –

    We flash forward to today. The bottleneck is actually our technical people’s ability to understand and build that software. And so MongoDB flips this whole thing on its head and allows those people to be kind of these artisans of the business today, unlocking their ability to really make those businesses competitive. And that’s why I think the strategic value of this is rising. And then when you deliver it in a service, you get so far away from this kind of plumbing of dealing with infrastructure management, operating systems –

    Who wants to be talking about that stuff? It’s non-differentiating, right? You have to move to your custom software, and you have to be able to utilize the best-in-class capabilities. Best in class database as a service globally on the Big Three public cloud, like MongoDB Atlas. Use that with best-in-class other capabilities, and really weave together your unique business, your unique company. That’s what’s happening out there.

    Shimel: Agreed. You said something earlier, too, that wasn’t lost on me, and I wanna make sure it wasn’t lost on the audience. Here on the first anniversary of MongoDB Atlas, it now accounts for more than 50 percent of revenue for the company, which means it’s the dominant revenue-maker for the company. And how long are you working on the Atlas project now, Andrew?

    Davidson: Since the beginning. I mean, really five, six years on Atlas, but at MongoDB almost nine.

    Shimel: Yep. So I guess the question is, and be honest –

    Davidson: Yeah.

    Shimel: – don’t fudge it – did you imagine five years ago that this would be the dominant revenue source for Mongo? Or did you always think it was?

    Davidson: That’s a great question. I think part of me thought, “Y’know, it’s gonna be so hard to get to the point where you can be trusted to power the back end of these mission-critical applications all over the world. This is gonna be such a massive mountain to climb!” I think I had doubts at times, but on the flip side I knew: this is the future of databases. The only way we’re really gonna get to that next level, the only way we truly can disrupt this massive database market, is if we nail it with this service.

    So we knew, and I think we had pretty solid buy-in at every level of the company, that we had to make Atlas successful. In the early days, y’know, as is always difficult, we were kind of a start-up in the company, a whole new go-to-market model, largely self-service led, credit card customers coming in, long tail. We weren’t ready yet to go out to the enterprises on year one. And kind of earning that right, getting battle tested, getting all the security controls, the regulatory capabilities, the sort of the reference wins.

    Y’know, those early customers like 7-11, and Liberty Mutual, and NBCUniversal. Doing great big things on the platform, really showing that you could do it at scale, you could do it safely and securely. And just validating the value prop that we always knew was gonna be there – it just got easier and easier. The flywheel starts moving, and you started feeling like, “Y’know, we really have a shot, here.” And we started doing more and more deals – SMB, corporate, kind of mid-market segment.

    And then probably year three, we started really going big in the enterprise, and you just kinda knew, “It’s gonna happen. We’re gonna do this.” The growth rate was so high. And now, boom. Yeah, you get to that point. The 50 percent chasm is crossed, growing twice as fast as the traditional software, and it’s like, “What a feeling!” I talked to so many customers, even, who are transitioning from kind of being open-source software companies to SAS companies.

    It is a very hard DNA shift to go through. You go from really just writing software that other people run on their own, to being responsible for the business of your customer. And it’s a whole different trust dynamic that’s really important.

    Shimel: Absolutely. Look, congratulations to you and to the whole team with that. It’s an accomplishment, for sure, right? ‘Cause it’s easy to go from zero to one; it’s easy to go from one to two. Two to four is harder. Four to eight is harder. Eight to 16 is yet harder. When you’re talking about 50 percent of the revenue of a company like Mongo, it’s not a zero to one, right? Five years ago, Mongo was already – I guess it was public. Was it public five – ? I don’t know. How long – ?

    Davidson: I think we went public more – no, I think it was more like three and a half years ago, if I’m not mistaken.

    Shimel: It was still a sizeable amount of revenue, right? It wasn’t just _____, y’know –

    [Crosstalk]

    Davidson: Oh, yeah, absolutely. Yeah, yeah, totally. Yeah, yeah.

    Shimel: So that is a mountain to climb. All right. I’m gonna ask you – We’re running low on time. I got a few other things. I’m gonna ask you to put on your visionary goggles now.

    Davidson: Sure.

    Shimel: Here we are, Andrew, talking to Alan five years from now. I hope not. I hope I’m somewhere in Maui or something. But five years from now we’re talking, celebrating the tenth year of Atlas. It’s now 80 percent of Mongo’s revenue, unless something new came up to disrupt it – who knows?

    Davidson: Totally.

    Shimel: But what does Atlas look like five years from now?

    Davidson: Yeah, look. I mean, the whole philosophy of our platform in many ways is it’s fundamentally about user experience. It’s about making it possible for these application developers and operators and enterprises and start-ups, everyone in between – make it easier for them to build modern applications on a modern application data platform. And what we think makes us really unique is, it all starts with this document data model. The document data model is very much the superset data model.

    You can express key value inside a document. That’s just, y’know, a document with a single index. You can express relational inside a document; it’s essentially a flat document. Y’know, spatial, graph – it’s all expressible in this unified interface. So you can do all these data models and all these use cases, which makes it very widely applicable, very multimodal. But then the important step beyond that is to drive multiple workloads behind the scenes.

    The workloads are kind of how you store the data behind the scenes, the algorithms you use to analyze it. Our philosophy is, behind that same unified, elegant interface, you should be able to express all of the wide array of workloads that power modern applications – operational, transactional, the most mission-critical applications; search, which is about human-powered experiences; mobile experiences on iOS and Android devices, synchronizing data back up and down from the cloud to the mobile device –

    real-time analytics; data-like use cases, with object storage economics. All of that should be accessible at your fingertips through this superset data model. And when we think about kind of wrapping all of that up, continuing to deliver it at scale, secure, highly available, highly scalable, multi-cloud service on the Big Three public cloud, we’re just gonna keep taking this vision to the next level. Be laser focused on the user and on delivering what they need to build the next-generation applications of tomorrow.

    Shimel: I love it, man. Good stuff. Andrew, we’re about out of time. For people who wanna get more information on MongoDB Atlas, where do they go?

    Davidson: Well, you can absolutely go to mongodb.com, but you should also check out our upcoming developer conference, mongodb.live, on July 13. Look forward to seeing you there.

    Shimel: Is that virtual only?

    Davidson: Virtual. Absolutely. Virtual conference this year. Next year I’m hopeful, very hopeful, we’ll be back with the real deal, what we used to call MongoDB World, in New York City.

    Shimel: Absolutely. I’ve been there, been there more than a few times.

    Davidson: There you go.

    Shimel: I’m looking forward to it.

    Davidson: Absolutely.

    Shimel: Y’know, I’m gonna be in New York that week of July 13th. I wish you guys were doing it in person. But –

    Davidson: Well –

    Shimel: Next year. We’ll wait.

    Davidson: Yeah, yeah. Look forward to seeing you _____.

    [Crosstalk]

    Shimel: Andrew, congrat – Absolutely. Andrew, congratulations on your fifth anniversary with Atlas, and great work. It’s something to be proud of, man.

    Davidson: Thank you, Alan.

    Shimel: It’s good stuff. Big stuff.

    Davidson: My final comment is, this community’s growing like crazy. Like I said, 100,000 people sign up for Atlas every month. You should sign up today if you haven’t experienced a free forever sandbox cluster, build your Hello World application. We’ve got courses for you to learn from, documentation up the wazoo, because this is big tent community. Thank you, Alan.

    Shimel: Absolutely. Talk to you soon, Andrew. A pleasure. Andrew Davidson, MongoDB. Fifth year anniversary of Atlas – check it out. We’re gonna take a break here on TechStrong TV, and we’re gonna be right back with another guest.

    [End of Audio]

  • MongoDB Announces General Availability of Device to Cloud Data Sync to Accelerate Mission-Critical Mobile Development

    Realm Sync empowers mobile developers to easily build cross-platform and offline-first applications with bi-directional syncing of data between the Realm Mobile Database and MongoDB Atlas

    NEW YORK — February 2, 2021 — MongoDB, Inc. (NASDAQ: MDB), the leading, modern general purpose database platform, today announced the general availability of MongoDB Realm Sync, a fully managed service which syncs data between the popular Realm mobile database and  MongoDB Atlas. This new solution addresses the unique technical challenges of mobile and offline-first development, allowing organizations to rapidly build responsive applications for their customers and remote workforces that drive user adoption, improve productivity, and deliver ROI.

    Realm Sync will enable teams to take advantage of robust bidirectional data sync between devices and the industry-leading multi-cloud database, MongoDB Atlas, without having to write complex conflict resolution and networking code. The Realm mobile database allows end users to read and write data to their devices, enabling zero-latency data retrieval and offline application functionality. With more than two billion downloads, applications built on Realm are used by millions each day. Bidirectional sync to MongoDB Atlas enables data captured on the device to be processed, analyzed, and combined with other data sets; this can then be delivered back to the user’s device to deliver a better and more robust experience. Over 20,000 customers use MongoDB Atlas, now available in 79 regions across Amazon Web Services, Google Cloud, and Microsoft Azure.

    “We’re witnessing a mobile-led digital transformation that is changing how companies interact with their customers and how they support their workforces. The proliferation of 5G will only enable richer application experiences that rival what can be delivered on desktops today,” said Sahir Azam, Chief Product Officer, MongoDB. “For developers who aspire to build these best-in-class experiences on mobile, Realm Sync can be transformative in reducing the amount of time to get these apps in the hands of users
. Building on the strong foundations of the Realm mobile database and MongoDB Atlas, Realm Sync simplifies the development of mission-critical mobile applications with global reach.”

    “To support our edge computing strategy we’ve begun expanding the use of automation technologies within our distribution centers. We’re evaluating using Realm and Realm Sync to assist with inbound and outbound parcel shipping use cases,” said James Fairweather, Chief Innovation Officer, Pitney Bowes. “As an example, we are exploring building an app on Realm for our front-line workers to scan a package that would automatically sync the data back to MongoDB Atlas providing consistent reporting and up-to-date logistics throughout the shipping journey.”

    “Business apps tend to be rated less highly than consumer apps, suffer a high churn rate and have lower retention rates. They are designed to improve productivity, but fail to do so if they are not being used,” said William Fellows of 451 Research. “Apps should work offline—regardless of the strength of signal—be reactive to changes, show data that is synced in real-time across multiple users and provide a seamless experience when users move between devices. This is where Realm Sync comes in, with support for real-time data sync between the Realm Mobile Database and MongoDB Atlas.”

    Further Resources:

    About MongoDB

    MongoDB is the leading modern, general purpose database platform, designed to unleash the power of software and data for developers and the applications they build. Headquartered in New York, MongoDB has more than 22,600 customers in over 100 countries. The MongoDB database platform has been downloaded over 130 million times and there have been more than one million MongoDB University registrations.

  • MongoDB, Storj Labs Work to Reduce Data Security Costs

    MongoDB, Storj Labs Work to Reduce Data Security Costs

    MongoDB Inc. and Storj Labs announced today they have jointly created a backup service for the open source MongoDB database that leverages a global network for excess storage capacity.

    Storj Labs has created a Tardigrade Decentralized Cloud Storage Service based on an open source object storage system that is compatible with the S3 application programming interface (API) defined by Amazon Web Services (AWS). That object storage system distributes 80 slices of a file across a subset of thousands of nodes that each contain only a slice of the data required to reconstruct that file when needed. Only 29 of those slices are needed to reconstruct a file, which enables IT teams to secure data without having to manage encryption keys.

    Via a connector from Storj Labs, the Tardigrade Decentralized Cloud Storage Service now can be invoked via MongoDB DevOps Manager, which is made available in the commercially supported enterprise edition of the open source databases provided by MongoDB Inc.

    Storj Labs CEO Ben Golub said the Tardigrade Decentralized Cloud Storage Service provides a less expensive alternative to backup and recovery platforms based on the cloud, as excess capacity on devices ranging from servers to individual desktops is readily available, The challenge has been finding a way to marshal all that excess capacity in a way that makes it simple for IT organizations to consume, he said.

    IT organizations, Golub said, should expect Storj Labs to build additional connectors to databases and other platforms that require a less expensive approach to data protection. In the wake of the economic downturn brought on by the COVID-19 pandemic, he noted, more organizations than ever are open to innovative approaches to reducing storage costs.

    Of course, most IT teams are well-used to employing a range of data protection platforms to back up and recover data. The Tardigrade Decentralized Cloud Storage Service is likely to be viewed as one more option that might find traction at a time when the number and types of databases being deployed across the enterprise continue to expand rapidly. Developers are especially fond of adopting open source document databases that don’t require a database administrator (DBA) to get approval for a license from a central purchasing organization. It’s usually not too long, however, before developers look for help from central IT to manage all those databases, especially after an application is deployed in a production environment.

    Naturally, not all those applications are of equal value, so given the current economic climate, there might be a greater willingness to at the very least experiment with alternative approaches to data protection.

    Regardless of platform, however, the one thing that is clear is data protection alongside the rest of IT infrastructure is being driven more by APIs. As such, responsibility for data protection is also being pushed further left toward DevOps teams, who have the expertise required to continuously automate back up and recover using APIs. That might not always sit well with traditional IT administrators who have been using tools based on graphical user interfaces (GUIs) for decades. Nevertheless, IT times are clearly changing.

  • MongoDB Addresses Host of DevOps Issues

    MongoDB Addresses Host of DevOps Issues

    At its MongoDB Europe conference this week, MongoDB updated its open-source document database with the addition of MongoDB Ops Manager to simplify management of distributed instances of version 3.6 of its database.

    In addition, the company has added a capability that enables all updates to the database to be automatically reflected in the application in real time without having to take the database offline.

    Additional new capabilities include a MongoDB Compass tool to analyze schemas and a governance capability that makes it easier to apply and tune data governance polices before and after an application is deployed.

    The company has added an aggregation pipeline that allows users to perform real-time analytics directly in the database to improve performance. Developers and analysts now can write queries combining data from multiple collections for complex analytics. Its Connector for BI, which can now be deployed and managed by its Ops Manager, has been updated to support this new query capability.

    Finally, in the event of a system failure or network outage, MongoDB says it now automatically will replay any writes to the database when those services come back online.

    Brooks Circhlow, vice president of product for MongoDB, says developers have embraced document databases as an alternative to other database platforms because of their flexibility. Unlike, for example, relational databases, a document database doesn’t impose a schema that must be crafted by a database administrator. But as more instances of document databases proliferate across the enterprise, DevOps challenges start to manifest.

    In fact, Circhlow notes that IT organizations are also making extensive use of cross-replication capabilities across multiple database clusters that they manage or are managed on their behalf via the MongoDB Atlas managed service. In many of those instances, DevOps teams are responsible for both the applications and the document databases they run on.

    MongoDB is also moving to address a DevSecOps issue stemming from a wave of attacks aimed at misconfigured instances of its database. Circhlow says the most widely used installer for MongoDB limits network access by binding to localhost by default. With version 3.6, binding to the localhost is now extended to all MongoDB packages across all platforms. In addition, with new IP whitelisting support, administrators can configure MongoDB to only accept external connections on approved IP addresses.

    Document databases such as MongoDB arguably have been a primary catalyst for DevOps. Once developers began to bypass database administrators by downloading instances of document databases, it became clear that traditional approaches to IT operations were starting to erode. Most organizations would prefer their developers to write code than manage databases, so some new means for managing databases that doesn’t compromise the dexterity that document databases provide developers was required. Once those processes get established, the next question then becomes how to make traditional database administration involving other classes of databases part of that process. Most IT organizations are still working through those processes. But it’s clear database administration will never be the same.

    — Mike Vizard

  • Backup: Not So Easy in Certain Environments

    Backup: Not So Easy in Certain Environments

    We discussed how backup needs to be reinvented for a new era cloud-native applications, and what you should look for in a modern database backup and recovery solution. But even if you scrap the traditional approach to back up and write scripts to handle the task, backup and recovery of next-generation applications and databases (Apache Cassandra, MongoDB, Amazon DynamoDB, Microsoft DocumentDB, Apache HBase, et al) can be frustratingly difficult. But why is that?

    The short version is this: In an eventually consistent write-anywhere non-relational database architecture, it is almost impossible to capture a consistent state backup. Successful data recovery is even less likely!

    There isn’t just one reason why this is the case. It comes down to the basic nature of a distributed architecture that’s designed to both scale and withstand node failures without downtime. As we’ve highlighted below, there are several challenges from the start that make backing up a distributed architecture difficult:

    • Data is written to an available node. The first landing point for data is not predictable, so there’s no way to catch data as it lands on the node.
    • Then data is replicated to at least one other node before verification. That ensures a valid write, but also immediately creates a copy of the data.
    • … then replicated to one or two more nodes for availability. This happens afterwards, and the same data can get updated in quick succession.
    • … meaning there are three or four copies of data at any time.
    • … and nodes are never immediately consistent.
    • … and backups are hugely inefficient since every replica is copied.
    • And when a node fails, the topology changes mid-flight.

    Theoretically, a good DevOps team might be able to write scripts that will back up the database successfully 80 percent to 90 percent of the time (cases such as multi-node failures, changing topologies, compaction of databases and many more are complex to model, so scripts are difficult to get right).

    Unfortunately, backup is the “easy” part of the equation. Restoring a backup is where it really gets challenging. A successful restore is far more complicated than most people assume. It involves the following steps:

    • Rebuilding the correct topology. Because each node was backed up individually, the database has to be restored to the same topology (six nodes to six nodes, 12 nodes to 12 nodes), etc. This relates to enabling recovery across clouds, test/dev and CI/CD use cases.
    • Waiting for the database to repair and restore itself. The non-relational database architecture can withstand node failures and keep running, but a full restore with data reconciliation will bring back memories of the worst RAID rebuilds on slow drives.
    • Deduplicating multiple inconsistent copies. The backup copy has three or more copies of all data that may be in an inconsistent state. It needs to be deduplicated or reconciled first.
    • But data history isn’t always available. In most distributed architectures, the oplog is a circular buffer that essentially eats itself as it goes! It may not be possible to even recover the necessary log data to reconcile the database.
    • And there’s no reliable way to go back to a point-in-time. Even with the oplog data, it would be very difficult to reconstruct point-in-time data. At best, you’d get a recent, mostly accurate copy of the database.

    In the real world, restores can take days or weeks—if the data can be recovered at all.  But as the recent Gitlab outage shows, it is difficult for even technically savvy organizations to get it right. Human error can be as fatal to a database as a natural disaster if there isn’t a reliable backup and recovery process in place.

    — Shalabh Goyal

  • Why Reinvent Deduplication? Isn’t Cloud Storage Cheap?

    Why Reinvent Deduplication? Isn’t Cloud Storage Cheap?

    Most people assume cloud storage is cheaper than on-premises storage. After all, why wouldn’t they? You can rent object storage for $276 per terabyte per year or less, depending on your performance and access requirements. Enterprise storage costs between $2,500 to $4,000 per terabyte per year, according to analysts at Gartner and ESG.

    This comparison makes sense for primary data, but what happens when you make backups or copies of data for other reasons in the cloud? Imagine that an enterprise needs to retain three years of monthly backups of a 100TB data set. In the cloud, this can be easily equate to 3.6PB of raw backup data, or a monthly bill of more than $83,000. That’s about $1 million a year, even before factoring in and data access or retrieval charges.

    That is precisely why efficient deduplication is hugely important for both on-premise and cloud storage, especially when enterprises want to retain their secondary data (backup, archival, long-term retention) for weeks, months and years. Cloud storage costs can add up quickly, surprising even astute IT professionals, especially as data sizes get bigger with web-scale architectures—data gets replicated and they discover it can’t be deduplicated in the cloud.

    The Promise of Cloud Storage: Cheap, Scalable, Forever Available

    Cloud storage is viewed as cheap, reliable and infinitely scalable—which is generally true. Object storage such as AWS S3 is available at just $23/TB per month for the standard tier, or $12.50/TB for the Infrequent Access tier. Many modern applications can take advantage of object storage. Cloud providers offer their own file or block options, such as AWS EBS (Elastic Block Storage) that starts at $100/TB per month, prorated hourly. Third-party solutions also exist that connect traditional file or block storage to object storage as a back end.

    Even AWS EBS, at $1,200/TB per year, compares favorably to on-premises solutions that cost 2 to 3 times as much and require high upfront capital expenditures. To recap, enterprises are gravitating to the cloud because the OPEX costs are significantly lower, there’s minimal upfront cost and you pay as you go (vs. traditional storage, which you have to buy far ahead of actual need).

    Copies, Copies Everywhere: How Cloud Storage Costs Skyrocket

    The direct cost comparison between cloud storage and traditional on-premises storage can distract from managing storage costs in the cloud, particularly as more and more data and applications move there. There are three components to cloud storage costs to consider:

    • Cost for storing the primary data, either on object or block storage
    • Cost for any copies, snapshots, backups or archive copies of data
    • Transfer charges for data.

    We’ve covered the first one. Let’s look at the other two.

    Copies of data. It’s not how much data you put into the cloud; uploading data is free, and storing a single copy is cheap. It’s when you start making multiple copies of data—for backups, archives or any other reason—that costs spiral if you’re not careful. Even if you don’t make actual copies of the data, applications or databases often have built-in data redundancy and replicate data (or, in database parlance, a Replication Factor).

    In the cloud, each copy you make of an object incurs the same cost as the original. Cloud providers may do some deduplication or compression behind the scenes, but this isn’t generally credited back to the customer. For example, in a consumer cloud storage service such as DropBox, if you make one copy or 10 copies of a file, each copy counts against your storage quota.

    For enterprises, this means data snapshots, backups and archived data all incur additional costs. As an example, AWS EBS charges $0.05/GB per month for storing snapshots. While the snapshots are compressed and only store incremental data, they’re not deduplicated. Storing a snapshot of that 100TB dataset could cost $60,000 per year, and that’s assuming it doesn’t grow at all.

    Data access. Public cloud providers generally charge for data transfer either between cloud regions or out of the cloud. For example, moving or copying a TB of AWS S3 data between Amazon regions costs $20, and transferring a TB of data out to the internet costs $90. Combined with GET, PUT, POST, LIST and DELETE request charges, data access costs can really add up.

    Why Deduplication in the Cloud Matters

    Cloud applications are distributed by design and are deployed on non-relational massively scalable databases as a standard. In non-relational databases, most data is redundant before you even make a copy. There are common blocks, objects and databases such as MongoDB or Cassandra that have replication factor (RF) of 3 to ensure data integrity in a distributed cluster, so you start out with three copies.

    Backups or secondary copies are usually created and maintained via snapshots (for example, using EBS snapshots as noted earlier). The database architecture means that when you take a snapshot, you’re really making three copies of the data. Without any deduplication, this gets really expensive. And existing solutions, designed for on-premises legacy storage, can’t help.

    Not Just Deduplication — Semantic Deduplication

    Most deduplication technology works at the storage layer, deduplicating blocks of data. This is highly efficient on centralized SAN or NAS storage, but breaks down if the data layer is abstracted from the storage—as it is in a distributed database such as MongoDB. Deduplication in this world needs to address two fundamental issues:

    • It has to work at the data layer, not the storage layer. To deduplicate data from a distributed cluster, the software has to understand and interpret the underlying data structure.
    • It has to eliminate redundant data before it gets written to the database. Once data is written, it gets replicated within the cluster, so it needs to be deduplicated in-flight.

    The good news is there is semantic deduplication technology that works efficiently with distributed cloud applications that can help cut storage costs by up to 80 percent for databases such as MongoDB and Cassandra.

    — Shalabh Goyal

  • 7 Best Practices New MongoDB Users Should Know

    7 Best Practices New MongoDB Users Should Know

    Being a developer in today’s technology landscape is tricky. While the proliferation of different cloud services makes it faster and simpler than ever to turn ideas into reality, this rapid development comes at a price. Developer responsibilities have grown to include not only product/feature development, but also other application-critical tasks such as configuring and managing a database deployment, in this case, MongoDB.

    While this guide looks at MongoDB databases specifically, the broader (and proverbial) fork in the road for developers is this: Manage a production database deployment yourself or go the database-as-a-service (DBaaS) route. A DBaaS manages the database on a developer’s behalf, so the developer can focus more on feature and product development operations instead of database ops. However, sometimes this isn’t an option. If you’re embarking on a MongoDB journey and don’t use a DBaaS, consider this checklist to ensure you have the best practices down before getting started.

    Host Your Application in the Same Data Center as Your Database

    Your application infrastructure and your database should be located in the same local network (i.e., data center/cloud region), as it will be the most secure method of deployment and will minimize latency between your application and database. For example, if you are hosting your application in Azure West US, you should also host your database in Azure West US.

    When you connect to your database from within the same data center/region, you communicate through your cloud hosting provider’s internal network. All major cloud hosting providers provide a good deal of network security infrastructure to isolate tenants. The hypervisors they use do not allow VMs to read network traffic addressed to other VMs; thus, no other tenant can “sniff” your traffic.

    Use a Replica Set for High Availability and Review Deployment Best Practices

    MongoDB Replica Sets synchronize data over multiple redundant servers and protect your application from downtime in the face of node failure. Should your primary node fail or become unreachable, your application will automatically fail over to a secondary node, ensuring the availability of your database.

    AWS users should take special care to deploy their replica set nodes across separate Availability Zones to protect against zone failures.

    Not all workloads have the same requirements, so there are multiple strategies for deploying a replica set. You can find a list of considerations in the MongoDB documentation here.

    Host Different Environments on Separate Hardware/VMs

    In addition to the production environment, you may be responsible for setting up environments for development and staging applications. Create a separate database deployment for each of your application environments, with each deployment running on its own VM or hardware to protect against resource contention (CPU, RAM, etc.) across your multiple environments.

    Enable Database Access Control

    MongoDB does not require authentication by default. Protect your database against unauthorized access by enabling database access control. Enabling access control also allows you to limit actions that particular users can perform on the database.

    Use an Application Driver Maintained by the MongoDB team

    MongoDB is a very popular database technology with a large community behind it. As a result, many third-party drivers have been created. However, third-party drivers may become abandoned or outdated over time. It’s critical to find a driver that is well-maintained and stays up to date with the latest MongoDB features.

    If possible, use an official driver by the MongoDB team. You can find a list of drivers that are maintained by the MongoDB team here.

    Index Early and Often

    Without question, the most common database performance issue is due to improper indexing (or a complete lack thereof). Proper indexing is critical because even one unindexed query is enough to prompt significant performance degradation.

    If you’re unsure about indexing best practices, we’ve created a list of indexing tips for improving your MongoDB performance.

    Schedule Regular Backups

    While replica sets can offer redundancy to protect against system failure, they will not protect you from accidents caused by human error such as dropping a collection or even your database. Backups are crucial for ensuring you do not lose valuable data.

    Two common backup strategies are to take filesystem snapshots or to use the mongodump utility. Schedule frequent, recurring backups to keep your data safe!

    About the Author / Chris Chang

    chris-changChris Chang is a developer advocate at mLab. At mLab, Chris has provided database support to thousands of developers. Previously, Chris has held IT roles at VMware and NetApp. He is interested in developer tools, fitness, and house music. Chris holds a B.S. in Computer Science from the University of California, San Diego. Connect with him on LinkedIn and Twitter.