devopsy.kinsta.cloud

Author: Shalabh Goyal

  • When Bad Things Happen to Good Clouds: Why You Need a Recovery Plan

    When Bad Things Happen to Good Clouds: Why You Need a Recovery Plan

    Amazon S3 storage service recently experienced a widespread five-hour outage in its East Coast availability zone. Many next-generation consumer and business applications rely on a cloud storage, so the S3 outage quickly cascaded and temporarily crippled organizations from Netflix to Slack.

    Cloud outages like this are normal, and notable because they affect so many businesses. The last significant AWS outage was in August 2016. While Amazon S3 is architected for data durability, that doesn’t equal fast recoverability during an outage. As good as cloud platforms are, they still leave a few gaps:

    • Availability zones don’t equal recoverability. S3 is designed to withstand a site outage within a zone, but as the most recent outage shows, networking issues can lead to a widespread outage across an entire region.
    • Data still needs to be backed up. As we noted in a previous blog post, even Amazon recommends backing up data.
    • Recovery can be slow and tedious. It’s one thing to back up data. It’s another thing entirely to recover it. It can take hours or days to recover data after a failure—especially for hyperscale applications and databases.
    • Data is often in one “basket.” If backup data is stored in the same cloud service as the primary data, in the same availability zone, there’s no way to recover data during a widespread outage.
    • Data can get compromised or enter an inconsistent state. The cloud itself doesn’t protect data from application- or database-level corruption, or human error.

    A data backup, recovery and continuity strategy needs to be designed with the cloud in mind. To make sure you can recover quickly, even from a cloud outage:

    • Keep backup data in another service or region. Failures like this one often affect an entire region. A backup and recovery strategy needs to include the ability to recover in another region, cloud service or even a private cloud.
    • Have a fast recovery process. Traditional backup solutions and scripting-based approaches can’t recover data quickly, particularly if the application needs to be recovered to a different topology.
    • Have point-in-time recovery. Since data can get compromised at the early stages of an outage, being able to restore applications quickly to a point in time is important as well.

    The cloud as-a-service architecture is more resilient than traditional infrastructure, and gives you agility. But when failures do happen, they are also entirely out of your control. Don’t ignore recoverability and resiliency of the data because it’s in the cloud, and don’t expect the same recovery processes and tools to work on next-generation, hyperscale applications.

    — Shalabh Goyal

  • Backup: Not So Easy in Certain Environments

    Backup: Not So Easy in Certain Environments

    We discussed how backup needs to be reinvented for a new era cloud-native applications, and what you should look for in a modern database backup and recovery solution. But even if you scrap the traditional approach to back up and write scripts to handle the task, backup and recovery of next-generation applications and databases (Apache Cassandra, MongoDB, Amazon DynamoDB, Microsoft DocumentDB, Apache HBase, et al) can be frustratingly difficult. But why is that?

    The short version is this: In an eventually consistent write-anywhere non-relational database architecture, it is almost impossible to capture a consistent state backup. Successful data recovery is even less likely!

    There isn’t just one reason why this is the case. It comes down to the basic nature of a distributed architecture that’s designed to both scale and withstand node failures without downtime. As we’ve highlighted below, there are several challenges from the start that make backing up a distributed architecture difficult:

    • Data is written to an available node. The first landing point for data is not predictable, so there’s no way to catch data as it lands on the node.
    • Then data is replicated to at least one other node before verification. That ensures a valid write, but also immediately creates a copy of the data.
    • … then replicated to one or two more nodes for availability. This happens afterwards, and the same data can get updated in quick succession.
    • … meaning there are three or four copies of data at any time.
    • … and nodes are never immediately consistent.
    • … and backups are hugely inefficient since every replica is copied.
    • And when a node fails, the topology changes mid-flight.

    Theoretically, a good DevOps team might be able to write scripts that will back up the database successfully 80 percent to 90 percent of the time (cases such as multi-node failures, changing topologies, compaction of databases and many more are complex to model, so scripts are difficult to get right).

    Unfortunately, backup is the “easy” part of the equation. Restoring a backup is where it really gets challenging. A successful restore is far more complicated than most people assume. It involves the following steps:

    • Rebuilding the correct topology. Because each node was backed up individually, the database has to be restored to the same topology (six nodes to six nodes, 12 nodes to 12 nodes), etc. This relates to enabling recovery across clouds, test/dev and CI/CD use cases.
    • Waiting for the database to repair and restore itself. The non-relational database architecture can withstand node failures and keep running, but a full restore with data reconciliation will bring back memories of the worst RAID rebuilds on slow drives.
    • Deduplicating multiple inconsistent copies. The backup copy has three or more copies of all data that may be in an inconsistent state. It needs to be deduplicated or reconciled first.
    • But data history isn’t always available. In most distributed architectures, the oplog is a circular buffer that essentially eats itself as it goes! It may not be possible to even recover the necessary log data to reconcile the database.
    • And there’s no reliable way to go back to a point-in-time. Even with the oplog data, it would be very difficult to reconstruct point-in-time data. At best, you’d get a recent, mostly accurate copy of the database.

    In the real world, restores can take days or weeks—if the data can be recovered at all.  But as the recent Gitlab outage shows, it is difficult for even technically savvy organizations to get it right. Human error can be as fatal to a database as a natural disaster if there isn’t a reliable backup and recovery process in place.

    — Shalabh Goyal

  • Why Reinvent Deduplication? Isn’t Cloud Storage Cheap?

    Why Reinvent Deduplication? Isn’t Cloud Storage Cheap?

    Most people assume cloud storage is cheaper than on-premises storage. After all, why wouldn’t they? You can rent object storage for $276 per terabyte per year or less, depending on your performance and access requirements. Enterprise storage costs between $2,500 to $4,000 per terabyte per year, according to analysts at Gartner and ESG.

    This comparison makes sense for primary data, but what happens when you make backups or copies of data for other reasons in the cloud? Imagine that an enterprise needs to retain three years of monthly backups of a 100TB data set. In the cloud, this can be easily equate to 3.6PB of raw backup data, or a monthly bill of more than $83,000. That’s about $1 million a year, even before factoring in and data access or retrieval charges.

    That is precisely why efficient deduplication is hugely important for both on-premise and cloud storage, especially when enterprises want to retain their secondary data (backup, archival, long-term retention) for weeks, months and years. Cloud storage costs can add up quickly, surprising even astute IT professionals, especially as data sizes get bigger with web-scale architectures—data gets replicated and they discover it can’t be deduplicated in the cloud.

    The Promise of Cloud Storage: Cheap, Scalable, Forever Available

    Cloud storage is viewed as cheap, reliable and infinitely scalable—which is generally true. Object storage such as AWS S3 is available at just $23/TB per month for the standard tier, or $12.50/TB for the Infrequent Access tier. Many modern applications can take advantage of object storage. Cloud providers offer their own file or block options, such as AWS EBS (Elastic Block Storage) that starts at $100/TB per month, prorated hourly. Third-party solutions also exist that connect traditional file or block storage to object storage as a back end.

    Even AWS EBS, at $1,200/TB per year, compares favorably to on-premises solutions that cost 2 to 3 times as much and require high upfront capital expenditures. To recap, enterprises are gravitating to the cloud because the OPEX costs are significantly lower, there’s minimal upfront cost and you pay as you go (vs. traditional storage, which you have to buy far ahead of actual need).

    Copies, Copies Everywhere: How Cloud Storage Costs Skyrocket

    The direct cost comparison between cloud storage and traditional on-premises storage can distract from managing storage costs in the cloud, particularly as more and more data and applications move there. There are three components to cloud storage costs to consider:

    • Cost for storing the primary data, either on object or block storage
    • Cost for any copies, snapshots, backups or archive copies of data
    • Transfer charges for data.

    We’ve covered the first one. Let’s look at the other two.

    Copies of data. It’s not how much data you put into the cloud; uploading data is free, and storing a single copy is cheap. It’s when you start making multiple copies of data—for backups, archives or any other reason—that costs spiral if you’re not careful. Even if you don’t make actual copies of the data, applications or databases often have built-in data redundancy and replicate data (or, in database parlance, a Replication Factor).

    In the cloud, each copy you make of an object incurs the same cost as the original. Cloud providers may do some deduplication or compression behind the scenes, but this isn’t generally credited back to the customer. For example, in a consumer cloud storage service such as DropBox, if you make one copy or 10 copies of a file, each copy counts against your storage quota.

    For enterprises, this means data snapshots, backups and archived data all incur additional costs. As an example, AWS EBS charges $0.05/GB per month for storing snapshots. While the snapshots are compressed and only store incremental data, they’re not deduplicated. Storing a snapshot of that 100TB dataset could cost $60,000 per year, and that’s assuming it doesn’t grow at all.

    Data access. Public cloud providers generally charge for data transfer either between cloud regions or out of the cloud. For example, moving or copying a TB of AWS S3 data between Amazon regions costs $20, and transferring a TB of data out to the internet costs $90. Combined with GET, PUT, POST, LIST and DELETE request charges, data access costs can really add up.

    Why Deduplication in the Cloud Matters

    Cloud applications are distributed by design and are deployed on non-relational massively scalable databases as a standard. In non-relational databases, most data is redundant before you even make a copy. There are common blocks, objects and databases such as MongoDB or Cassandra that have replication factor (RF) of 3 to ensure data integrity in a distributed cluster, so you start out with three copies.

    Backups or secondary copies are usually created and maintained via snapshots (for example, using EBS snapshots as noted earlier). The database architecture means that when you take a snapshot, you’re really making three copies of the data. Without any deduplication, this gets really expensive. And existing solutions, designed for on-premises legacy storage, can’t help.

    Not Just Deduplication — Semantic Deduplication

    Most deduplication technology works at the storage layer, deduplicating blocks of data. This is highly efficient on centralized SAN or NAS storage, but breaks down if the data layer is abstracted from the storage—as it is in a distributed database such as MongoDB. Deduplication in this world needs to address two fundamental issues:

    • It has to work at the data layer, not the storage layer. To deduplicate data from a distributed cluster, the software has to understand and interpret the underlying data structure.
    • It has to eliminate redundant data before it gets written to the database. Once data is written, it gets replicated within the cluster, so it needs to be deduplicated in-flight.

    The good news is there is semantic deduplication technology that works efficiently with distributed cloud applications that can help cut storage costs by up to 80 percent for databases such as MongoDB and Cassandra.

    — Shalabh Goyal

  • Protecting Data in Public Cloud and Hybrid Cloud

    Protecting Data in Public Cloud and Hybrid Cloud

    Backup, business continuity (BC) and disaster recovery (DR) have all been a critical part of IT for over 30 years—ever since we began relying on technology to run our businesses. Traditional solutions were designed for the world of on-premises infrastructure and structured applications and relational databases. But the world is changing. In the last blog, I talked about the era of digital transformation and the impetus it has on rethinking and reinventing the fundamental backup and recovery architecture for workloads moving to the cloud and applications being born in the cloud.

    What Changed? Reinventing Backup and Recovery

    Application and data platforms are undergoing the biggest transformation since the dawn of computing. There are several forces at work:

    • New applications. The third-generation applications are geo-distributed, scale across multiple systems, always on and typically are deployed in a cloud-first model.
    • Existing applications are moving to the cloud. They aren’t going away, but companies are moving some or all of them to the cloud. They still need backup and recovery.
    • RPO and RTO windows are shrinking: Enterprises want “always-on” and gone are the days when you could do nightly backups.
    • Smaller companies will go all-in on public cloud. SMBs don’t want to be in the business of IT. They’ve been driving the rapid growth of cloud applications and platforms.
    • Enterprises will build hybrid clouds. Enterprises will deploy applications and data across on-premises and public cloud environments. Scale, compliance and other factors mean they’ll need to keep some systems on-premises.
    • Everyone will use multiple clouds. No one will bet their business on one cloud or one provider. Even now, enterprises are splitting workloads across clouds or cloud and on-premises. Development and test may use one cloud, while the same application might be deployed in a private cloud or a different public cloud.

    Cloud’s Impact on Backup, Recovery and Continuity

    Cloud gives organizations much more agility, operational savings, and a pay-as-you go model. Public cloud providers can also build much more resilient infrastructure. Amazon guarantees 99.95 percent availability for EC2 and 99.99 percent for S3; S3 is designed for 11 nines of data durability, with multiple availability zones. Because the cloud is so reliable—and cheap—it’s quickly becoming a backup target for on-premises data. But that shouldn’t trick us into believing backup and recovery are “built-in” when we run applications in the cloud. Even Amazon itself recommends backup services for all AWS-native applications and cloud databases.

    While service availability and data resiliency addresses infrastructure business continuity and disaster recovery, it doesn’t provide point-in-time recovery or application level intelligence for backup and recovery. As good as the cloud platforms are, they don’t protect against logical errors. And research shows 8 out of 10 errors are logical errors—data corruption, user errors.

    Existing Backup Products and the Cloud

    As we noted above, traditional backup and recovery products don’t meet the needs of cloud applications—even for existing applications that have moved to the cloud, and not just because they were built in a different era. Cloud and distributed architectures present several other challenges:

    • Cloud breaks the media server based architecture of traditional solutions. Cloud applications and data don’t reside on a specific array or disk, so you can’t easily back up what you can’t see. Backups also don’t capture configuration data in the cloud, such as AWS Cloud Formation templates.
    • Cloud doesn’t speak the same language. Legacy solutions talk to tape, disk or virtual disks. Backup and recovery in the cloud means integrating with the right protocols, such as the S3 API or Google Cloud Storage.
    • Backup appliances can’t be moved to the cloud. Existing backup appliances such as EMC Data Domain or NetBackup that work extremely well on-premises can’t be picked up and moved to the cloud.
    • Traditional backup agents won’t scale. If you could get a backup agent running in the cloud, it wouldn’t scale well across dozens or possibly hundreds of nodes.
    • VM is not the right layer of abstraction: This is precisely why the core principle of the Datos IO CODR architecture is a scalable application-centric view of data management and data protection that distinguishes it from conventional approaches. This is the exact reason why the CODR architecture introspects application data and uses global semantic de-duplication to achieve storage efficiencies, instead of relying on traditional de-duplication techniques which treat data as an opaque object (such as a VM or a LUN). The benefit of this approach is fine-grained and highly space-efficient data protection that can span clouds over network links.
    • Cloud Gateway or Migration Services: Unidirectional only.

    Data Protection Must Be Reinvented

    The problem of backup and recovery of cloud applications is novel as there are three critical things a cloud backup and recovery architecture should have:

    • Elastic compute only. The architecture should scale efficiently on elastic compute instances. There shouldn’t be any CapEx costs for servers or appliances.
    • No media servers. Backing up large, scale-out databases requires direct a parallel streaming architecture for data movement between the database and secondary storage. Legacy backup architectures rely on media servers that quickly would become choke points. Direct parallel streaming also allows the data to remain available in native formats.
    • Semantic deduplication. Scale-out application databases typically have a 3x replication factor. If you back up individual nodes or even manage to snapshot the whole database, two-thirds of the backup data is redundant. Over time, backups will be 75 percent to 80 percent inefficient without semantic deduplication that works across a distributed architecture.

    — Shalabh Goyal

  • Evolving Data Requirements in Multi-Cloud Environments

    Evolving Data Requirements in Multi-Cloud Environments

    By design or by accident, multi-cloud is a reality that most enterprise IT teams have to cope with today. Multi-cloud (or, cross-cloud, as Dell EMC calls it) refers to an IT model where an organization uses services of one or more public cloud service providers, sometimes in addition to using its own data centers. In a multi-cloud model, different clouds may be used in different stages of the application life cycle such as test/dev and production, and for different types of applications such as traditional structured applications and next-gen cloud applications. Nonetheless, IT leaders face challenges in identifying data management tools they need to embrace the multi-cloud environment to improve application agility, reduce operational hassles and enhance application resilience. In this blog, I will discuss how enterprises are using a multi-cloud model and the requirements it places on next-generation data protection products.

    One common reason for multi-cloud is to support geo-replicated applications to enable global reach of e-commerce organizations. In such a scenario, a production database may reside on a private cloud behind an enterprise firewall and replicated to multiple regions in a public cloud to service reads (i.e. consumers are shopping on these e-commerce sites) from different regions at low latency. Such a configuration also protects against disasters at the data center level because the database and application are replicated to public cloud (with multiple availability zones).

    Another use case is to service the requirements of production environment versus test and development environments using different clouds. While security, control and performance optimization may mandate a production environment to stay within a private data center, agility, access to new capabilities such as Google BigQuery and developer preferences may drive application development teams to use a public cloud service providers.

    Finally, using a multi-cloud model could also be driven by the IT strategy of an enterprise to prevent overdependence (or, vendor lock-in) on any one public cloud service provider. To hedge the risks and get leverage, an enterprise may balance its application workloads across multiple cloud providers—for example, deploy Tier-1 applications on one cloud and Tier-2 applications on another. It is a real phenomenon we are seeing with enterprise customers! However, a multi-cloud model presents the following unique challenges to enterprises.

    • Data protection in multi-cloud environment: Legacy data protection (backup and recovery, archiving, replication) products were architected for on-premises environments and optimized to work with legacy applications. Using legacy backup and recovery tools for next-generation applications in a public cloud environment is equivalent to fitting a round peg in a square hole. For one, legacy products don’t leverage the elastic and scalable compute and storage that is available in public cloud. True infrastructure-independent (software-only and elastic compute-based) data protection products that may be deployed in heterogenous environments are required as enterprises look to migrate applications to the cloud.
    • DevOps agility: Moving data across clouds for disaster recovery or to refresh test & development environments with production data is another key requirement for enterprises. Doing so in a multi-cloud environment is not an easy task given the challenges in moving data over the WAN. There are also differences in configuration between test/dev and production environments. Further, the test/dev environments may not be as large as production environment so sub-sampling is required. Finally, personally identifiable data needs to be masked before it leaves the production firewall.
    • Ease of management and governance: Finally, federated management is required for applications and workloads running across multiple clouds. It is extremely inefficient to get familiar with different management tools and their intricacies.

    This is by no means a complete list. There are several other requirements, such as support for cloud storage with consistent performance, handling of identity access and management processes and so on. The point is that data protection needs to be re-thought in the context of multi-cloud environments.

    — Shalabh Goyal

  • Multi-Cloud: The Danger of Data Lock-In

    Multi-Cloud: The Danger of Data Lock-In

    Enterprises rarely think about cloud portability issues before transferring large amounts of data onto a cloud service provider such as Amazon AWS, Microsoft Azure or Google Cloud Platform. They were likely to ask questions on usage by other players in the industry (“the herd effect”) as well as the reliability of the cloud service and durability of data, among other things. Enterprises are starting to realize that as they store increasing amounts of data onto the cloud, it is easy to import data but difficult to get data out. While it is true that sophisticated users may be able to solve the problem, most run the risk of locking in their databases—the “crown jewels”—with the cloud provider.

    A common solution to the cloud portability issue is to develop an abstraction layer. Having an abstraction layer in place between an application and the cloud data service helps by minimizing the amount of code restructuring that would be required should an enterprise wish to migrate to a new cloud service. One example is the Amazon S3 API: The Google Cloud Framework has developed S3-compliant API to make it easy for enterprises to migrate their applications. The open-source community is also working to make it seamless; for example, the Python Boto library supports both the Amazon S3 service as well as the Google Cloud Storage service under the same abstraction layer.

    However, while an abstraction layer solves the application migration issue, what about the “data” itself? How easy would it be to migrate swathes of data from one cloud service provider to another? Let us do some simple mathematics on this subject. For example, let us assume you would like to migrate 10TB of bucket data from Amazon S3 to Google Cloud Storage. How long do you think it will take? The answer would surprise you—almost a month, by reasonable estimates! The sheer operational cost of managing the data transfer over a month would dwarf the storage and network bandwidth charges accrued in both cloud providers. It might be cheaper to transfer the contents of an Amazon S3 bucket onto a local hard drive, ship the drive to Google and ask the company to upload the drive contents onto a Google Cloud Storage bucket.

    How would someone solve the multi-cloud data lock-in problem? Would you:

    • Run copies of applications on multiple clouds and keep data in sync?
    • Maintain one cloud as the primary and other clouds as passive secondaries?
    • Trust one cloud for Tier 0 applications and other cloud for Tier 2 or Tier 3 applications?

    One of the  key issues is that of the data transfer time. We use intelligent change capture techniques to minimize the amount of data sent over the WAN links that connect multiple cloud providers. For example, take a version of an Apache Cassandra database running in AWS S3, store the version in Google Cloud Storage and restore the version to an Apache Cassandra database running in Microsoft Azure. This is the future of data protection: Not only does it allow you to protect against user errors and logical corruptions, but it also gives you protection across clouds to provide insurance against lock-in onto a single cloud service provider.

    Application Recovery Management: The Rise of Polyglot Persistence

    Enterprises are using a variety of different data stores to capture the needs of a varying set of applications and their access patterns. Relational data stores might be the right choice for multi-faceted normalized data, but data stores today have to cater to a varying degree of access patterns—for example, a relational database cannot keep up with the high volume of a social media stream, and, likewise, a NoSQL data store cannot perform joins efficiently on a normalized data set. This term is referred to as “polyglot” persistence, wherein composite applications are sprayed over multiple data stores for varying types of data and methods of data manipulation. In the earlier days, data architects chose a relational data store such as Oracle and then mapped the application to this data store. Today, the landscape has changed significantly: Data architects first classify the data types and expected manipulation methods and then choose the appropriate data store to fit the needs.

    Let us use the example in which an enterprise needs to identify all the customers in the past year, their purchase characteristics, the customer acquisition method, the social media comments on the post-purchase experience as well as any support interactions. Remember, that in today’s information age, the processing has to be in real time: it would be very imprudent for a business to react to a negative social media reference months after it happened. As one can guess, multiple sources of data with varying degrees of structure need to be collected and analyzed to meet these basic requirements of the enterprise. This type of problem cannot be solved easily or cost-effectively with one type of database technology. Even though some of the basic information is transactional and probably in a relational data store, the other information is non-relational and will require a few different types of persistence engines: document stores for static images and text collateral, spatial for geo-locating mobile customer interactions and graph for deriving relations between the multi-faceted data described above.

    But has data protection kept up? Remember that the data is highly correlated with complex interdependencies between the data stores. A logical error in one data store is likely to propagate to others, so recovery has to span the entire spectrum of polyglot persistence. Today’s data protection is highly siloed with recovery limited to a single type of data store. This is clearly not enough!! What one has to do is to synchronize versioning across multiple data stores according to the needs of the application—this is an application consistent polyglot version. The key advantage of this version is to allow us to recover an entire application rather than sub parts of it.

    Conclusion

    It is time to rethink and reinvent data protection. The world of enterprise IT is experiencing a massive change as cloud is becoming the de-facto infrastructure of choice and multi-faceted big data is becoming the hallmark of the next generation of applications. Data protection must adapt to these changing environments and deliver the next generation of services to the enterprise.

    — Shalabh Goyal

  • MongoDB Atlas: One Giant Leap, But More Steps Needed

    MongoDB Atlas: One Giant Leap, But More Steps Needed

    Atlas, holding up your databases in the cloud. I’m not sure if that’s what MongoDB was going for, but I personally like the analogy.

    At MongoDB World’16, MongoDB announced Atlas, a database-as-a-service offering (in the cloud) of the company’s popular open-source NoSQL database. Rather than dealing with the complexities of setting up, provisioning and configuring a MongoDB cluster yourself, Atlas allows companies to leave all those details for the masters of MongoDB to manage, freeing up application developers to focus on more important things (such as application development). Atlas currently is available on Amazon AWS, but there are plans to expand it to other public cloud platforms such as Microsoft Azure and Google Cloud Platform.

    Depending on the needs of your application, Atlas allows you to scale your cluster up or down, both vertically (with more powerful servers) and horizontally (with more shards in your cluster). You can also provision more disk space as your data grows. Simply tell Atlas and it will automatically update your cluster in the cloud to match your specifications. The documentation is unclear whether Atlas leverages AWS’ autoscaling features, but it appears that scaling must be done manually.

    Bringing Up a MongoDB Cluster

    When you first log in to Atlas, you’re met with a pretty simple user interface (UI). The very first thing you can do is to start up a new cluster in the cloud.

    2Before deploying a cluster, you need to know the AWS region to place the cluster, AWS instance size, amount of disk space needed, whether to use encrypted storage, whether to enable backups, the speed of the underlying storage (i.e. number of IOPS) and the number of shards and replica set members.

    You first specify the type of AWS instance you want to run your cluster on (ranging from M10 to M100) and how much disk space you want to provision, anywhere from 10GB to 16TB. All instances can be configured with encrypted storage with no additional cost. I first started by selecting the default settings from the M10 instance, which also happens to be the cheapest.

    3Instance sizes, specs, and configurable options. Details including cost are consolidated on the right.

    After specifying your instance, you can then configure the number of nodes in your cluster. For replica sets, you can have either three, five or seven members. Additionally, you can specify whether the cluster should be sharded and the number of shards in the cluster. Note that sharded clusters requires an AWS instance of M50 or higher, which is one of the more expensive instances, so the jump from a basic unsharded cluster to sharded cluster will be fairly expensive!

    4

    5The cheapest sharded cluster is 56 times more expensive than the cheapest unsharded cluster. You probably don’t want to be using Atlas for any sharded testing clusters if you can help it.

    After you’re satisfied with your cluster configuration, you can just click “Confirm & Deploy.” My cluster was spun up within a couple minutes, but your mileage may vary. After adding the IP addresses from which you’ll access the Atlas instances to a security whitelist, you can begin using your new mongo cluster. Something small that I really appreciated about Atlas is that it gives you the exact command and hostnames you should use to connect to your cluster (both through the mongo shell as well as through MongoDB’s many drivers). It makes connecting to your cluster incredibly simple. Just copy and paste.

    6Atlas gives you high-level stats of your cluster, including IOPS, read and write performance, disk usage, etc. You can also get more detailed metrics for each node.

    7Connecting to your cluster made easy! Just download an updated mongos binary and then copy and paste.

    After creating your cluster, it is fairly simple to scale your clusters horizontally by adding more shards. For me, the whole process of upgrading my 1×3 to a 2×3 cluster took around 10 minutes (again, your mileage may vary). Pretty slick! In this process, Atlas will automatically provision more servers and configure your cluster with new config servers, mongos and shards. Notably, once you decide to shard a cluster, you are unable to go back down to an unsharded instance, so make sure that you’re absolutely sure when you decide to scale up to a sharded cluster—you won’t be able to go back!

    Overall, outside of a few unsupported cases, the process of setting up a cluster and manually scaling it was quite smooth. While what has been achieved thus far in reducing cluster configuration to a few tunable options is definitely a step in the right direction in simplifying cluster setup for the user, Atlas can do much more. Compare with other existing cloud database services, such as Amazon’s DynamoDB or Microsoft’s DocumentDB, where all you need to specify is the IOPS requirement of your application (application IOPS, not storage IOPS as it is in Atlas). In these competing cloud databases, issues such as replication, number of shards, and high availability are hidden away from the user. Granted, DocumentDB has been out for a year already and DynamoDB many years before that, so it may just be a matter of time before Atlas catches up. Regardless, there is a lot more Atlas can do to be truly serverless and even simpler to use—a solid first step, but lot more needs to be developed to compete with cloud databases and to win enterprise customers.

    Under the Covers: Backup and Recovery

    Now since I work on something similar here at Datos IO, I was quite interested in Atlas’ backup and recovery options. At a first glance, it looks similar to the backup and recovery services provided by MongoDB’s earlier products such as CloudManager and OpsManager. After playing around with the feature, I’m more convinced that the backup technology Atlas uses is essentially the same as previous offerings, only that it’s applied to Atlas instances.

    Backups are free for the first 1GB and $2.50 per GB per month for every GB after that. Backups can be scheduled at set intervals but with a minimum interval of six hours. While I was initially put off by the minimum snapshot interval (after all, a LOT can happen in six hours!), after some investigation, Atlas also allows you to schedule “checkpoints” in between snapshots. Checkpoints can be used as restore points, but they take longer to restore because it requires applying all deltas since the last snapshot. It’s important to note that this feature is only available for sharded clusters. If you want to protect data on an unsharded cluster, you’ll be stuck with a minimum snapshot interval of six hours.

    Backup can be set when by toggling a flag during cluster initialization. During restore time, you can select a specific snapshot to restore. If you need a finer granularity of restore, Atlas also provides point-in-time recovery, which is pretty neat. You can specify a time between two snapshots to restore at a minute-level granularity. If you need an even finer granularity, you can specify the specific oplog timestamp and interval you want to restore.

    8Existing cluster snapshots and the types of restores you can do.

    As I performed a couple of backups and restores, I had a couple reservations about the feature. Most of my qualms arose from the fact that Atlas treats the entire cluster as its ‘unit’ of backup. What I mean by this is that backups created by Atlas are snapshots of the entire cluster. So there is no granularity at all, and this is problematic in a few ways.

    First, since Atlas takes snapshots of the entire cluster, you can’t control what data you want to back up. Say, for example, only a small portion of your database contains critical sensitive information that needs to be backed up. With Atlas, to back up your critical data, you’ll have to back up all your noncritical data as well! Considering that the cost of backup increases with the size of your data, you’ll end up unnecessarily paying a lot for backups of data you don’t really care about.

    Moreover, your options when recovering your backups becomes constrained by the fact that backups are at the cluster-level. By nature of being cluster-level snapshots, backups must restore back the entire cluster; you can’t restore back a specific database or collection. This can be problematic when, say, only one of your collections in your production cluster becomes corrupted. In that case, Atlas takes a very heavy-handed approach of bringing down the entire cluster and resetting every collection back to the point of the snapshot, as opposed to only restoring the problematic collection.

    Lastly, Atlas only allows you to restore snapshots to clusters with the same number of shards as the snapshot. For example, if you have a cluster with three shards, you can only restore to other clusters with three shards. One potential reason you may want to restore data to a cluster with a different topology is for testing purposes: Perhaps you want to measure application performance as you increase the number of shards in your cluster, or perhaps you just need a small, cheap cluster for basic tests. If you want to restore production data to a test cluster in Atlas, you’ll either have to spin up a cluster with the same topology as the production cluster (which is good for MongoDB, but sad for your wallet), or you’ll have to manually download from Atlas and then manually restore that data back to Atlas into a cluster of different topology. Both cases work, but are far from being ideal.

    Concluding Thoughts

    Atlas is a pretty good move by MongoDB to make it even easier to build up applications on top of its database by abstracting out a lot of the more mundane details of cluster configuration to the people who know MongoDB best. For me, Atlas was a simple and fast way to create and scale up MongoDB clusters. While a good step in the right direction, Atlas can do more to make it a truly serverless approach like other cloud databases in the market right now.

  • Backup and Recovery of MongoDB, Part 2

    Backup and Recovery of MongoDB, Part 2

    This blog is the second in a two-part series on backup and recovery for MongoDB. In one of my previous blogs, I covered why companies require both replication and backup for enterprise grade data protection. And in the first part of this blog series, I discussed the existing solutions for backup and recovery of MongoDB and their drawbacks. Now in this blog, I will discuss the key requirements for protecting data that resides on MongoDB (deployed either on-premises, on private cloud with an as-a-service model or in public cloud with Amazon AWS or Google Cloud Platform).

    Requirement 1: Online Cluster-Consistent Backups

    Next-generation applications deployed on MongoDB must be always-on. This means pausing the database to make backups is not feasible and the backup operation should not impact the performance of the application. As the application scales, the underlying MongoDB also needs to scale-out to multiple shards. In this case, a backup solution must provide a consistent backup copy across shards without disrupting database and application performance during the backup.

    Advanced Data Management ServicesRequirement 2: Flexible Backup Options

    Depending on the application, data may have different change rate and patterns. For example, in a product catalog, certain items may be refreshed every day (fast-selling goods, for example), while the others may have longer shelf life, such as premium items. Based on the application requirements, some collections may need to be backed up every hour versus others that may be backed up daily. Providing this flexibility to schedule backups at any interval and at collection level granularity is another requirement we have heard from customers using MongoDB. More importantly, these backups always should be stored on the secondary storage in native formats to avoid vendor lock-in.

    Requirement 3: Scalable Recovery

    During its life cycle, data resides in multiple stages such as development, test, preproduction and production, and also may reside in multiple clouds (private cloud and public cloud). The topology of MongoDB clusters at each stage is different. For production, the application could be deployed on a sharded MongoDB cluster on-premises, but the test team might have access only to unsharded MongoDB clusters in the Amazon AWS (public cloud). Hence, the backup solution should allow multiple restore operations such as sharded to sharded (such as from 5×3 cluster to 2×3 sharded cluster) or sharded to unsharded (such as 5×3 cluster to 1×3 unsharded) across such cloud configuration.

    chart

    Requirement 4: Handling Failure

    Failures are a norm in the distributed database world. However, the backup solution should be resilient to database process failures, node failures, network failure and even logical corruption of data during backup and recovery operations. Finally, the backup solution should be able to handle failures of MongoDB config servers that store metadata for sharded clusters.

    Customers are deploying MongoDB in physical servers, private clouds and microservices such as frameworks, and in public cloud.  Backup and recovery should be seamless across these deployment example and the ease of backup and recovery deployment is a big one for MongoDB customers and I will cover this in the next blog. Stay tuned!

  • Using MongoDB for a Product Catalog Application

    Using MongoDB for a Product Catalog Application

    In today’s era of data-driven business, enterprises are continuously adopting new technologies to gain a competitive advantage. Distributed and cloud databases is one such technology. These databases are fundamentally different from relational databases, offering scalability, availability and flexible data models that are beyond the capabilities of traditional relational databases.

    MongoDB is one of the leading next-generation scale-out databases. It is a document-oriented database that provides dynamic schemas, making the integration of data in certain types of applications easier and faster. MongoDB also provides easy scaling through auto-sharding, tunable consistency for reads and built-in replication.

    In this blog, I will describe an application, product catalog management, that distinctly benefits from features MongoDB offers.

    Most e-commerce enterprises need a product catalog with the ability to store varied types of objects with different sets of attributes. These kinds of data collections are quite compatible with MongoDB, which can offer dynamic queries on a constantly changing dataset.  However, it is important to understand the major requirements of industry verticals such as retail, media and entertainment, insurance, health care and e-commerce that are addressed by MongoDB before designing an optimal data model and for your own product catalog.

    For example, Ebay offers several use cases in which MongoDB has been implemented successfully and has delivered much value for its product catalog:

    • Product Availability – Being a JSON document store, not only can MongoDB help in complex querying of a product catalog in real time but it also can keep track of placed orders to accurately calculate the available quantity for popular products. The data is important from customer service point of view, as it keeps them aware of available choices at the time of order placement and prevents order cancellations later due to non fulfillment.
    • Dynamic Pricing – Product prices change quite frequently in the online world. E-commerce enterprises need to keep pace with these fluctuations and dynamically adjust product prices to retain their existing customer base. MongoDB helps tremendously through its ability to update data on live sites without shutting down/restarting any systems. On top of that, ability to run geospatial queries using geospatial index allows the sites to show converted pricing and taxes based on the location of their customers. The example below shows how to find all theatres within 10 miles of a location using a geospatial query.

    db.theatres.find({ location:
    { $geoWithin:
    { $centerSphere: [ [ <longitude> , <latitude> ], 10 / 3963.2 ] } } })

    • Shipping Estimation – Today’s customers enjoy the convenience of shopping online at home but also demand the flexibility to control the delivery of their products. Some customers will pay extra money for expedited delivery while others are content with waiting longer to pay no shipping costs. The ability to provide correct shipping costs and time estimates is a critical functionality for e-commerce applications. Again, MongoDB comes to rescue with its geospatial queries to calculate correct shipping estimates based on the locations of the user and the inventory.
    • Personalized experience – E-commerce enterprises want their customers to enjoy the online shopping experience by understanding their needs as well as their social profile. This allows them to provide every single user with a personalized version of the site, which can be based on their past order history and their interaction with the company via stores, customer service and social media. Not only that, e-commerce enterprises run applications that recommend new products based on customers’ indicated preferences. MongoDB is being used by online retailers such as Otto to consume data from multiple sources and build customized catalog experiences for individual e-commerce customers.
    • Sales/Discounts – To engage their customers at the next level, e-commerce enterprises have built capabilities to develop customized discount offerings. This can be done via “flash sales” that allow customers to buy their coveted products at lower prices within the constraints of quantity, time and minimum spend. MongoDB has been the with technology of choice here as well (for example GILT Groupe) because it can keep track of time-bound and volume-bound parameters in real time to make such deals successful.

    ecommerceIt can be argued that e-commerce industry and dynamic product catalogs have been in existence for a long time. So, how is MongoDB changing the landscape? The answer lies in the rapidly evolving landscape of customer buying patterns. Today, a wait of an extra minute often means losing customers and revenue to the next best online competitor. Always-on applications are a norm and scalable infrastructure is a must-have requirement. Hence, MongoDB is helping enterprises to meet their changing business needs.

    However, product catalog applications generally require consolidating data from disparate sources and present a uniform view to customers. This also makes the underlying database, MongoDB, prone to logical and human errors. Hence, a highly available data protection infrastructure that can guarantee minimal data loss (low recovery point objective and recovery time objective) is required to keep online retail engines running. Further, certain catalog portions could be updated more frequently such as perishable goods; hence, those databases need to be backed up every hour versus the slow-moving goods.

    The data protection system should provide enough flexibility to set backup policies per the business requirements (granularity, etc.).

  • Backup and Recovery of Next Gen Apps in MongoDB

    Backup and Recovery of Next Gen Apps in MongoDB

    This blog is the first in a two-part series about MongoDB backup and recovery. In this first part, I will discuss the motivations for protecting data that resides on MongoDB and the existing backup and recovery solutions. In the second part, I will discuss how to solve this problem.

    In the era of big data, enterprise applications create a large volume of data that may be structured, semi-structured or unstructured in nature. In addition, application development cycles are much shorter and application availability is a critical requirement. Given these application requirements, enterprises are forced to look beyond traditional relational databases to onboard the next-generation, cloud-native Platform 3 applications—infrastructure-as-a-service (IaaS) or platform-as-a-service (PaaS) applications. NoSQL databases such as MongoDB are now being adopted and evaluated by enterprises for these next-generation applications, which include ecommerce and content management. The benefits of MongoDB are many, including dynamic schema, easy scaling through auto-sharding, tunable consistency for reads and built-in replication.

    MongoDB database also has native replication capabilities that satisfy availability requirements. However, data protection requirements for scalable point-in-time backup and recovery must be addressed. For robust data protection, enterprises need both backup and replication; Without point-in-time backups, organizations are at substantial risk of losing data due to human error, logical corruption and other operational failures. Traditional backup solutions were built to address the requirements of structured Platform 2 applications on relational databases that used shared storage and had the ACID (atomicity, consistency, isolation and durability) transaction model. Unfortunately, they fall short of addressing point-in-time backup requirements of Platform 3 applications and distributed databases (local storage, eventual consistency and the elastic nature of infrastructure). There are a few alternate script-based solutions such as Strata that enterprises are using to fill the data protection gap, but these solutions are sub-optimal at best.

    Manual Scripted Solutions

    Manual scripted solutions (MSS) leverage native MongoDB snapshot utility and scripts to transfer data to secondary storage. The scripts (via mongodump) are customized for each MongoDB cluster and require significant operational effort to scale or adapt to any topology changes (such as addition or removal of nodes to your MongoDB database). Further, these scripts are not resilient to failure scenarios such as failure of a node (primary or secondary) or intermittent network issues. Finally, recovery (the paramount value of “backup”) is a manual process, hence, time-consuming (resulting in very high application downtime) and contains data loss risk due to any bugs in the scripts. Overall, these solutions work when the MongoDB environment is small and some data loss may be permitted in the application. Some of the key issues that these solutions face are:

    • Lack of enterprise backup solution for sharded configurations
    • Database needs to be offline when the snapshots are taken
    • Both backup and recovery fail under node failure and other infrastructure failures
    • Recovery process is manual and requires verifications, which increases the recovery time
    • Recovery at collection-level requires manual recovery that is time consuming
    • Recovery to unlike topologies (sharded → unsharded) for test/dev refresh is not available

    Most enterprises use these scripted methods as a temporary quick-fix solution. It is like driving your car with flat tires: You can keep going; however, you can’t go at the speed you want to go, nor are you risk-free from disasters.

    MongoDB Paid Backup and Recovery

    MongoDB (the company) provides a couple of ways to back up MongoDB databases. Enterprises may choose from either a managed backup offering (MMS) that runs in public cloud or, if they are paid MongoDB customers, they may deploy the backup service on-premise. In addition to being exorbitantly costly, the managed backup service stores customers’ data in public cloud. Backup data transfer over WAN may not work for customers who deploy MongoDB on-premise and for the customers who need to keep their sensitive data in-house. Further, there are significant data limitations per shard to use this service.

    Using the MongoDB on-premise backup service is possible but is overly complex to deploy and operationalize (this deployment diagram speaks for itself!). Enterprises need to deploy eight servers, additional databases (with additional licensing) and about 6 to 9 times storage capacity (of the database that is backed up) for enabling on-premise backups.

    Overall, on-premise backup service is a theoretical solution that brings with it significant CAPEX and OPEX investments:

    1. Complexity of deploying multiple databases
    2. Cost of additional infrastructure (servers and storage)
    3. Cost of licensing additional MongoDB nodes
    4. Risk of failed backups when nodes fail (secondary from which backup is taken)
    5. Siloed backup infrastructure for only MongoDB database

    Look for Part two of this discussion to come shortly.