devopsy.kinsta.cloud

Tag: data protection

  • Protecting Data in Public Cloud and Hybrid Cloud

    Protecting Data in Public Cloud and Hybrid Cloud

    Backup, business continuity (BC) and disaster recovery (DR) have all been a critical part of IT for more than 30 years—ever since we began relying on technology to run our businesses. Traditional solutions were designed for the world of on-premises infrastructure and structured applications and relational databases. But the world is changing. In the last blog, I talked about the era of digital transformation and the impetus it has on rethinking and reinventing the fundamental backup and recovery architecture for workloads moving to the cloud and applications being born in the cloud.

    The Motivations Leading to Reinvent Backup and Recovery

    Application and data platforms are undergoing the biggest transformation since the dawn of computing. There are several forces at work:

    • New applications. The third-generation applications are geo-distributed, scale across multiple systems, are always on and typically deployed in a cloud-first model.
    • Existing applications are moving to the cloud. They aren’t going away, but companies are moving some or all of them to the cloud. They still need backup and recovery.
    • RPO and RTO windows are shrinking: Enterprises want “always-on” and gone are the days when you could do nightly backups.
    • Smaller companies will go all-in on public cloud. SMBs don’t want to be in the business of IT. They’ve been driving the rapid growth of cloud applications and platforms.
    • Enterprises will build hybrid clouds. Enterprises will deploy applications and data across on-premises and public cloud environments. Scale, compliance and other factors mean they’ll need to keep some systems on-premises.
    • Everyone will use multiple clouds. No one will bet their business on one cloud or one provider. Even now, enterprises are splitting workloads across clouds or cloud and on-premises. Development and test may use one cloud, while the same application might be deployed in a private cloud or a different public cloud.

    Cloud’s Impact on Backup, Recovery and Continuity

    Cloud gives organizations much more agility, operational savings and a pay-as-you go model. Public cloud providers can also build much more resilient infrastructure. Amazon guarantees 99.95 percent availability for EC2 and 99.99 percent for S3; S3 is designed for 11 nines of data durability, with multiple availability zones. Because the cloud is so reliable—and cheap—it’s quickly becoming a backup target for on-premises data. But that shouldn’t trick us into believing backup and recovery are “built-in” when we run applications in the cloud. Even Amazon itself recommends backup services for all AWS-native applications and cloud databases.

    While service availability and data resiliency addresses infrastructure business continuity and disaster recovery, it doesn’t provide point-in-time recovery or application-level intelligence for backup and recovery. As good as the cloud platforms are, they don’t protect against logical errors. And research shows 8 out of 10 errors are logical errors—data corruption, user errors.

    But Won’t Existing Backup Products Work in the Cloud?

    As we noted above, traditional backup and recovery products don’t meet the needs of cloud applications—even for existing applications that have moved to the cloud, and not just because they were built in a different era. Cloud and distributed architectures present several other challenges:

    • Cloud breaks the media server based architecture of traditional solutions. Cloud applications and data don’t reside on a specific array or disk, so you can’t easily backup what you can’t see. Backups also don’t capture configuration data in the cloud, such as AWS Cloud Formation templates.
    • Cloud doesn’t speak the same language. Legacy solutions talk to tape, disk or virtual disks. Backup and recovery in the cloud means integrating with the right protocols, such as the S3 API or Google Cloud Storage.
    • Backup appliances can’t be moved to the cloud. Existing backup appliances that work extremely well on-premises can’t be picked up and moved to the cloud.
    • Traditional backup agents won’t scale. If you could get a backup agent running in the cloud, it wouldn’t scale well across dozens or possibly hundreds of nodes.
    • VM is not the right layer of abstraction: Look for technologies that provide a scalable application-centric view of data management and data protection, introspect application data and use global semantic de-duplication to achieve storage efficiencies, instead of relying on traditional de-duplication techniques. The benefit of this approach is fine-grained and highly space-efficient data protection that can span clouds over network links.
    • Cloud Gateway or Migration Services: Unidirectional only.

    The problem of backup and recovery of cloud applications is novel there are three critical things a cloud backup and recovery architecture should have:

    • Elastic compute only. The architecture should scale efficiently on elastic compute instances. There shouldn’t be any CapEx costs for servers or appliances.
    • No media servers. Backing up large scale out databases requires direct a parallel streaming architecture for data movement between the database and secondary storage. Legacy backup architectures rely on media servers that would quickly become choke points. Direct parallel streaming also allows the data to remain available in native formats.
    • Semantic deduplication. Scale-out application databases typically have a 3x replication factor. If you backup individual nodes or even manage to snapshot the whole database, two-thirds of the backup data is redundant. Over time, backups will be 75 percent to 80 percent inefficient without semantic deduplication that works across a distributed architecture.

    — Brian Yin

  • Don’t Be Samsung: Here’s How You Can Protect Yourself

    Don’t Be Samsung: Here’s How You Can Protect Yourself

    Imagine a factory building 100,000 units a day, and you can see how a small error can have huge consequences. Now imagine that factory is a database.

    The world’s top smartphone maker this month pulled the plug on its $882 Galaxy Note 7 after phones overheated and caught fire, concluding in a disastrous recall for Samsung that could cost the company as much as $17 billion and shutter a global revenue stream. In an unprecedented move, Samsung has told mobile carriers to halt sales or exchanges of the device, requested that users to shut off their phones and issued a mass recall while the South Korean manufacturer investigates reports of fires in original and replacement Note 7s.

    This a scant two months after the Galaxy Note 7’s launch, a device that one TechCrunch consumer electronics expert raved as “undoubtedly one of the most iconic handsets around.” While it is not clear what caused these fires, there is speculation that it was due to a faulty component that interacts with the battery, rather than the battery itself.

    It goes without saying that there are ample opportunities for error in the assembly of a smartphone. The complicated, highly coordinated process is fraught with thousands of input touchpoints, from hard- and software components to tightly controlled manufacturing processes to manual labor. And though there may be countless checks and balances in place, mistakes do happen. What may seem to be an innocuous error, such as not sufficiently tightening a screw or the shorting of a circuit, in this case has led to epic consequences.

    The truth is, these instances happen more often than we think, and to technologies even more vital to the population than a personal smartphone.

    Consider today’s era of big data and cloud applications, where enterprise applications ingest and process vast amounts of information from various sources, often in real time, to deliver critical analytics, contextual information or business insights.

    This data has weight. It can influence whether fleets of airplanes are grounded at gates all across the country. It can restrict customers from accessing funds from online banking. It can dictate whether millions of online retail dollars go unspent. The reliance on accurate data, and its grip on lives and business, is growing at an exponential rate—like ever-widening ripples in a lake.

    To support all of this data, new applications (such as real-time analytics, internet of things, ecommerce and others) are developed and deployed on scale-out, non-relational databases or ported from traditional relational databases.

    But, like the manufacturing gaffs experienced by Samsung, corruptions and operational errors by developers occur in enterprise IT environments. Enterprises know this, and billions have been invested in traditional backup and recovery and data protection products to protect against such “err” moments. However, the irony is that data protection products like those that exist for traditional relational databases do not exist for today’s modern, next-generation scale-out databases supplying the new types of data that increasingly control our lives. We’re essentially trusting that the Ma Bell rotary phone repairman will be able to recover personal data—such as contacts, photos and messages—after your Galaxy Note 7 starts smoldering.

    So how did we get from the world’s most explosive phablet to distributed databases and big data? Because, like the Galaxy Note 7, the tiniest data error can rapidly expand into magnificent, catastrophic consequences.

    Enterprises are adopting cloud applications that gather large amounts of data at a high-ingestion rate and process that data in real-time to deliver actionable insights. These applications are deployed on non-relational databases. As migration to cloud and multi-cloud application environments increases, enterprises also must be able to manage and recover at scale without data loss or corruption and with minimal application downtime.

    However, enterprises today face a significant gap in addressing their data protection requirements. Their most valuable data, often needed 24/7 and without fail, runs absent any point-in-time backup options for that ‘“Houston, we have a problem”’ moment. Most studies suggest that more than half of data downtime and loss is triggered by human error, corruption or virus attacks. And yes, while native replication helps enterprises meet business continuity and disaster recovery requirements by providing protection from hardware failures or even natural disasters, it is insufficient (meaning it cannot scale) to meet next-gen enterprise data protection demands. Case in point: If a minor schema corruption impacts the primary data copy, it causes a multitudinous ripple effect that impacts all replicas down the line. Yet, even without protection, industries continue to adopt and deploy scale-out database systems at an accelerated pace.

    So just as Samsung witnessed an error replicate itself into such a massive meltdown of global proportions that it threatens to kill an entire product line, organizations also must understand how replication-based protection mechanism can actually act like a virus—spreading a corruption uncontrollably across non-relational, cloud databases.

    Samsung’s troubles won’t end with the simple recall of a single product, and their predicament should resonate. There will be punitive measures, and perhaps even a total enterprise shift from the smartphone market. It is not an enviable position, but is does offer a clear example of how a peripheral manufacturing error can swell to engulf an entire brand.

    For enterprises rushing to deploy scale-out databases, let Samsung serve as a warning: that deploying any infrastructure that could potentially lead to exponential, irreversible corruption of all data copies is a recipe for harm (if not disaster). As enterprises make the switch to next-gen applications running on scale-out, non-relational databases, an assembly line e-stop needs to be built in, in the form of a point-in-time, version-friendly data protection solution.

    Data corruption is not a matter of “if,” but “when.” So have a kill switch ready.

    errors-replicating-image

    — Brian Yin

  • The Cost of Rework when Developing for OpenStack Clouds

    The Cost of Rework when Developing for OpenStack Clouds

    According to the 2016 State of DevOps Report by Puppet and DORA, the mean time to recover when a service incident occurs ranges from less than one hour for high IT performers to less than one day for medium and low IT performers. Put that in a real-world scenario: your OpenStack application and the data associated with it that spans multiple virtual machines failed (for one reason or another). Now you need to rebuild something you’ve already spent time developing. Are you—and your boss—OK with that rework taking hours to days? Or do you think you’d be wishing you fell in the range of high IT performers, and you could quickly recreate what you lost in less than one hour?

    Now, take it a step further. What could you be doing with that time you now need to dedicate to rebuilding something? What will that rework take you away from? The next phase of your development? Lunch? Happy hour? Sleep? What price would you put on the personal cost to you?

    The 2016 State of DevOps Report authors wanted to calculate the cost of excess rework per year for organizations. They created a formula to calculate the number:

    Cost of excess rework = Technical staff size x average salary x benefits multiplier x percentage of technical staff time spent on excess rework

    Here’s what they found:

    • High performers have 11 percent excess rework
    • Medium performers have 22 percent excess rework
    • Low performers have 17 percent excess rework

    At first glance, those numbers seem odd. Why would the low performers have less excess rework than the medium performers? The report clarified that by explaining “low performers spend less time on rework … because they report spending a greater proportion of time on new work. We believe this greater amount of new work could be occurring at the expense of rework, thus racking up technical debt.”

    It’s clear all organizations have some amount of rework as a result of a failure of some kind. The question is, What is the cost of that rework? Let’s take a look at the results of the study.

    Cost of rework - 2016 State of DevOps Report by Puppet and DORA; OpenStack DevOps

    For the most part, the simple answer is that rework will likely cost your organization millions (unless it’s costing you billions). It’s just a matter of how many millions.

    We often get asked why it’s necessary for OpenStack environments to have a data protection strategy and a backup solution. The conventional (but wrong) thinking is that virtualization can take the place of data recovery. Frankly, the answer to why is contained within those numbers above. If you don’t have a backup and recovery solution for your OpenStack development efforts, are you willing to spend money and resources on reworking the lost data? And if you are, how many millions are you willing to spend on it? Let me remind you that you are trying to recover to a point in -time … persistent data included. If you say, “If you built your cloud appropriately, this shouldn’t be an issue ….” Madame or Sir, if your virtualized cloud has that kind of resiliency, then I would like to work for you.

  • Things to Know When Planning for Cassandra Backup

    Things to Know When Planning for Cassandra Backup

    NoSQL databases such as Apache Cassandra, MongoDB, Amazon DynamoDB, Azure DocumentDB and others are being adopted by enterprises to meet the data requirements of next-generation applications. Specifically, Apache Cassandra database provides linear scalability, a flexible consistency model and shared-nothing architecture that enables enterprises to scale their database footprint according to performance needs.

    As mission-critical applications are being deployed on Cassandra databases, there are enterprise data management requirements also, such as backup and recovery. Today, there are various options available for backup and recovery of Cassandra. These options have pros and cons, which I will elaborate on in this blog post. It is important for a Cassandra database administrator (DBA) to understand the pros and cons of these options before choosing any option and validating products and solutions available to achieve enterprise goals for backup and recovery.

    Let me first point out the important features that DBAs should look for while planning the backup:

    1. Secondary Storage footprint: Efficient storage utilization is key for any backup and recovery design and plan. The backup data set may be much larger than primary setup depending on the retention period of the backup data. Hence, if storage management is not efficient, customer storage capital expenditure could increase multifold, defeating the purpose of data protection.
    2. RPO and RTO: Recovery point objective (RPO) is the maximum targeted period in which data might be lost from an IT service due to a major incident. Recovery time objective (RTO) is the targeted duration of time and a service level within which a business process must be restored after a disaster. Typically, enterprise requirements for these next-generation applications dictate very small RPOs and RTOs.
    3. Backup performance: This is the time spent in backing up the desired data set. Typically, the backup performance should be scalable enough to match the data change rate at the primary cluster. In addition, the time spent in taking the initial full backup of the primary cluster also matters, so periodical backup is not affected.

    Cassandra Backup Options

    Now let’s take a look at four options available for performing Cassandra backup:

    Snapshot-based Backup

    Cassandra provides native node-level snapshot option via the nodetool command. Snapshot creates a hard link to existing Cassandra data files and expects users to delete it after backing up the files. This way, despite changes in Cassandra data, the backup is performed from snapshot files.

    Pros:

    • Simpler to manage: It requires periodical snapshot commands on each of the nodes, triggering backup on snapshot files and removal of snapshot files after backup.

    Cons:

    • Not a true backup: Node-level snapshot doesn’t mean cluster-consistent backup and such a simplistic strategy will result in heavy repairs upon recovery.
    • Potential large RPO: The snapshot operation is not trivial, as it requires flushing of all in-memory data on disk and, hence, frequent invocation of snapshots will impact the production cluster’s performance dearly.
    • Large storage footprint due to data compaction at primary cluster: As part of data compaction, Cassandra merges multiple data files on a node into a new file and, hence, reduces multiple versions of a record to a single and latest version. Compaction not only helps in reduction of data at the primary cluster, it also improves the read performance as it reduces the number of files to look up for a record. As a side effect of compaction, it’s possible that same records will appear in multiple versions, as they are merged into a new compacted file. It isn’t a trivial task to filter out already captured records from a compacted file. Depending on various aspects such as workload type, compaction strategy or versioning interval, compaction may cause multifold more data to be backed up, causing a serious impact on CapEx planning.
    • Snapshot storage management at primary cluster: Cassandra expects users or admins to clean up snapshot files after they are no longer required. It requires a reliable job to clean up the snapshot folders; otherwise, they may affect the primary cluster’s availability due to lack of storage.
    • Siloed solution: Enterprises require multiplatform backup and recovery product and native tools are built specifically for a particular database and don’t scale for multiple databases.

    Incremental Backup

    Cassandra supports incremental backup functionality. If enabled, every time Cassandra flushes memory content as a new file, a hard link for the new file is created in the incremental backup folder. Once the backup of these files is done, the administrator is supposed to delete these hard links. The best part of incremental backup is that the hard link is created only for newly created file due to the flushing of in-memory data contents. Files created due to compaction are not hard-linked.

    Pros:

    • Better storage utilization: There are no duplicate records in the backup, as the compacted files are not backed up.
    • Point-in-time backup: Companies can achieve better RPO, as backing up from the incremental backup folder is a continuous process and, hence, versioning can be done more granularly.

    Cons:

    • Space management at primary cluster: With this option, companies need a reliable space management process on the primary cluster. The incremental backup folder must be emptied after being backed up. Failure to do so may cause serious space issue on the primary cluster.
    • Creates lots of small size file in backup: Since incremental backup facilitates a backup of new files created due to the flush of in-memory data from Cassandra, typically the file size is small. This leads to many small files, making file management and recovery not a trivial task (and fairly expensive compared to the RTO guarantees customers want and expect).

    Incremental Backup in Combination with the Snapshot

    Incremental backup in combination with snapshots is another rudimentary way to take Cassandra point-in-time backups. Periodically, data is backed up from the snapshot, while incremental backup-based files are used for point-in-time needs. This means periodically the incremental backup-based files, which are already covered via snapshot-based backup, must be deleted.

    Pros:

    • Fairly large-size backup files: Only the data between periodical snapshots are from the incremental backup.
    • Point in time: It provides a point-in-time backup and restores.

    Cons:

    • Space management in backup: After every snapshot, the incremental backed-up data needs to be cleaned up. Also, for backed-up files from the snapshot, a large backup storage footprint due to compaction at primary always is a concern.
    • Operationally very heavy: This method requires DBA admins to script solutions; this is not scalable for enterprise scale.

    Commit-Log Backup in Combination with the Snapshot

    This method is similar to incremental backup. Rather than backing up the newly added SSTables, DBA admins archive the commit-log. Periodical snapshot-based backup provides the bulk of backup data, while the archived commit-log is used for the point-in-time backup.

    Pros:

    • Point in time: Snapshot-based backups in the combination of commit-log archival provide a point-in-time backup option.

    Cons:

    • Space management: Again, the commit-log archival space management is admin’s responsibility; this method results in operational heavy solution (OpEx).
    • Restore complexity: The restore is a lot more complex, as part of restore will happen from the commit-log replay.
    • Storage overhead: Snapshot-based backup will have a storage overuse issue because of duplication of data due to compaction. This results in a very large use of secondary storage and, thus, a very high CapEx expenditure.

    Summary

    Overall, Cassandra provides multiple options for backup. Typically, snapshot-based backup in the combination of commit-log archiving/incremental backup is used if the point-in-time restore is a requirement. If point-in-time restore is not a requirement, simple snapshot-based backup is much easier to implement, though it is not a true backup and will result in heavy repairs.

    Most of these options require significant manual effort and are error prone. They work for staging or preproduction and test/dev environments that are smaller in scale and can sustain data loss and high RTO. However, for large enterprise customers and production deployments, there are key requirements such as data reduction via novel compression/de-duplication techniques, various failure handling and faster recovery with desired consistency.

    For mission-critical, always-on applications where minimizing data loss and operational resiliency are important factors,  administrators should consider leveraging next-generation data protection products from companies that specialize in this area.

  • Backup and Recovery of MongoDB, Part 2

    Backup and Recovery of MongoDB, Part 2

    This blog is the second in a two-part series on backup and recovery for MongoDB. In one of my previous blogs, I covered why companies require both replication and backup for enterprise grade data protection. And in the first part of this blog series, I discussed the existing solutions for backup and recovery of MongoDB and their drawbacks. Now in this blog, I will discuss the key requirements for protecting data that resides on MongoDB (deployed either on-premises, on private cloud with an as-a-service model or in public cloud with Amazon AWS or Google Cloud Platform).

    Requirement 1: Online Cluster-Consistent Backups

    Next-generation applications deployed on MongoDB must be always-on. This means pausing the database to make backups is not feasible and the backup operation should not impact the performance of the application. As the application scales, the underlying MongoDB also needs to scale-out to multiple shards. In this case, a backup solution must provide a consistent backup copy across shards without disrupting database and application performance during the backup.

    Advanced Data Management ServicesRequirement 2: Flexible Backup Options

    Depending on the application, data may have different change rate and patterns. For example, in a product catalog, certain items may be refreshed every day (fast-selling goods, for example), while the others may have longer shelf life, such as premium items. Based on the application requirements, some collections may need to be backed up every hour versus others that may be backed up daily. Providing this flexibility to schedule backups at any interval and at collection level granularity is another requirement we have heard from customers using MongoDB. More importantly, these backups always should be stored on the secondary storage in native formats to avoid vendor lock-in.

    Requirement 3: Scalable Recovery

    During its life cycle, data resides in multiple stages such as development, test, preproduction and production, and also may reside in multiple clouds (private cloud and public cloud). The topology of MongoDB clusters at each stage is different. For production, the application could be deployed on a sharded MongoDB cluster on-premises, but the test team might have access only to unsharded MongoDB clusters in the Amazon AWS (public cloud). Hence, the backup solution should allow multiple restore operations such as sharded to sharded (such as from 5×3 cluster to 2×3 sharded cluster) or sharded to unsharded (such as 5×3 cluster to 1×3 unsharded) across such cloud configuration.

    chart

    Requirement 4: Handling Failure

    Failures are a norm in the distributed database world. However, the backup solution should be resilient to database process failures, node failures, network failure and even logical corruption of data during backup and recovery operations. Finally, the backup solution should be able to handle failures of MongoDB config servers that store metadata for sharded clusters.

    Customers are deploying MongoDB in physical servers, private clouds and microservices such as frameworks, and in public cloud.  Backup and recovery should be seamless across these deployment example and the ease of backup and recovery deployment is a big one for MongoDB customers and I will cover this in the next blog. Stay tuned!

  • Why Legacy Backup Solutions Aren’t Ideal for OpenStack

    Why Legacy Backup Solutions Aren’t Ideal for OpenStack

    Our team has recently returned from the OpenStack Summit in Austin. While there were many hot topics, we discovered people fell within two camps when it comes to backup and recovery: those who understand why a native OpenStack solution is required, and those who question why legacy backup and recovery won’t fit the bill.

    I sat down with Trilio Data’s co-founder, Murali Balcha, to detail why legacy backup and recovery solutions are not suited to protect OpenStack applications without considerable retooling. Even then, these solutions do not naturally fit into OpenStack operating model and will increase your total cost of ownership (TCO) considerably.

    First, let’s look at the OpenStack operating model. It includes:

    Horizontal Scale

    OpenStack does not come with the maximum capacity that it can grow into. The architecture is inherently scalable without any theoretical boundaries; it can be deployed in a single box or it can scale to thousands of nodes.

    Contrast this with existing backup solutions that always have some predefined capacity, whether it is number of applications it can back up or the amount of backup storage capacity that it can manage. So essentially you have a forever scalable architecture, tied to a backup solution with a limit on capacity.

    Multi-Tenancy

    Multi-tenancy is an integral part of OpenStack cloud. The tenancy is different than supporting multiple active directory users in a backup application. Active directory users are typically OS users, whereas cloud tenancy usually represents individual organizational units with in a bigger organization. So there is a mismatch on how existing backup solutions are created versus how OpenStack multi-tenancy works.

    One way to make existing solutions work with OpenStack is to deploy a backup application for each tenant and make that tenant responsible for managing its own instance of the backup application. This means each tenant has to manage the application and any storage that is provisioned to save backup images.

    Self-Service Portal

    Some backup applications provide an aspect of a self-service portal for individual users to manage their own backup policies. Backup applications use host agents on either virtual machines, or on servers to back up files and provide a portal to restore a backup copy in place through its agent infrastructure. This kind of self-service portal is, again, in contrast to what an OpenStack self-service portal is—it provides the ability to provision virtual resources.

    File-Level Backups

    Existing backup solutions are only good at doing file-level backups. Yes, they have evolved to do virtual machine (VM) backups for VMware, but vCenter is not a cloud platform, and the mechanism to do a VM backup on VMware platforms is totally different from doing VM backups in OpenStack. Anyone trying to retrofit existing backup solutions for OpenStack will only have their file-level backup feature to rely on.

    File-level backups only back up bits and pieces of your application. File-level backups are the legacy of backing up scale-up systems, where huge databases are confined to a beefed-up single host. But cloud applications are lot more distributed. Instead of provisioning one beefed-up server to host a large database, cloud applications are deployed as multiple VMs, with each VM performing discrete functionality. The data in these applications are also are distributed to multiple VMs. File-level backups will not ensure consistent backups. They are notoriously insufficient to restore an entire application environment and may seriously affect your recovery time.

    Infrastructure-Aware Backup Solutions

    File-level backups are not well-suited to address cloud application needs. Instead, a backup application that is aware of cloud infrastructure provides a perfect solution for your cloud applications. Since cloud applications are distributed, your backup should include not only the data but also the application context to successfully recover the application.

    An application context includes the VMs, flavors of the VMs, network settings and all the volumes attached to those VMs. Let’s not forget that all of the aforementioned will be tweaked and will evolve over time to suit that moment in time. So only backup solutions that can provide a single snapshot that includes all the VMs, storage volumes mapped to each VM, network settings and the meta data associated with each VM and its resources at that point in time are suited for OpenStack backup needs. Since such backup solutions capture the entire application environment, recovery with this backup solution is very deterministic and reliable, irrespective of the size and complexity of application.

    Solutions that are aware of the infrastructure also can back up applications into one availability zone/region and restore into a different zone/region. Beyond speed of recovery, this now empowers data protection (or backup) with resource management capabilities.

    DevOps

    DevOps is an integral part of OpenStack cloud deployment and management. Your choice of backup solution should play with your cloud DevOps scripts. For example, if you standardize on Ansible playbooks to manage your cloud deployments, the backup application should play into your DevOps strategy so you can reduce the TCO of your backup application.

    Other Use Cases

    In traditional IT, the data management functions such as backup and recovery, disaster recovery and application migration require different applications. However, your backup solution for OpenStack should be more than just a backup and restore of the files. A backup solution that is well-designed for OpenStack is a re-orchestration from a point-in-time and should be able to provide additional uses cases such as:

    1. File(s) restore
    2. VM/volume restore
    3. Intra/inter-cloud application migration
    4. Disaster recovery
    5. Instant restore of point in time for application validity checks, network scanner and other forensics
    6. Archival
    7. Environmental restore during cloud upgrades

    OpenStack offers an operating model that warrants a completely new thinking around backup solutions. Retrofitting existing solutions not only is inadequate in protecting your applications, it also increases your TCO. So do yourself a favor and look for solutions that are purpose-built for OpenStack. That will provide the best data protection possible, decrease your TCO and simplify recovery.

  • Enterprise Cassandra Deployments

    Enterprise Cassandra Deployments

    Successful Cassandra deployment in the cloud takes a certain amount of skill and knowledge. In this blog I will share best practices for a successful Cassandra deployment in the cloud such as Amazon Web Services (AWS) and Google Cloud Platform (GCP). I will also discuss how you can protect and recover the data in your Cassandra cluster when deployed in the public cloud.

    Cassandra Deployments in the Public Cloud

    There are two main ways to have successful enterprise Cassandra deployments in AWS:

    • Use a ‘managed’ Cassandra deployment. This is a reasonable approach; the only downside is the inability to customize data placement beyond what is typically provided by the vendor.
    • Manage the cluster yourself, if you want to have full control over your Cassandra deployment and customize it to scale and perform according your business needs.

    Please refer documentation from Datastax or Apache Cassandra for detailed best practices for Cassandra deployments. The following best practices are reiterated here to simplify deployment of a scalable Cassandra cluster in AWS.

    Storage

    Both AWS and GCP offer two types of local storage options for their compute instances. For AWS, the options are Elastic Block Storage (EBS) and Ephemeral Storage. In GCP, the options are Persistent Disks and Local SSD. Both options have their own trade-offs. Pick the one that best suits your deployment.

    Persistent Storage

    EBS volumes have performance characteristics similar to network storage such as a NAS or SAN. Because storage from EBS volumes is multitenant in nature, it is not suitable for distributed, log structured databases such as Cassandra. If you want to ensure reliable performance through EBS volumes, use “EBS optimized instances” with “provisioned IOPS.” Since the IO traffic to EBS volumes goes over the instance network, EBS optimized will provide reliable performance, but does not guarantee any increased performance. To get the most performance out of EBS volumes, use instances with multiple disks and separate the volumes used for “Data” and “Commitlog.”

    If you have to use EBS Volumes, go with “EBS Optimized Instances,” with “Provisioned IOPS” and split data and Commit Log on different volumes.

    Ephemeral Storage

    Ephemeral storage is a great option to use for downstream Cassandra environments such as test/dev and staging. Since ephemeral volumes are typically SSDs, there is no need to separate the data and commitlog into separate drives. To optimize performance, choose EC2 instances with multiple ephemeral storage drives and stripe them together into one logical volume. Send data and commitlog to the logical volume. Remember that when you stop the EC2 instance, the data in Ephemeral storage will be deleted.

    Stripe all Ephemeral drives into a single volume to get maximum performance.

    Data Distribution

    Cassandra uses Snitch to determine the topology of the cluster. The Snitch is also used to determine the data center and rack for each node in the cluster. Replication strategy is used to determine the placement of data across the different nodes in the cluster.

    AWS

    1. EC2Snitch works well for single region deployments.
    1. Ec2MultiRegionSnitch works well for cluster distributed across multiple AWS regions. Please note, Multi-DC, Active-Active configuration makes your cluster resilient to region failures and for multi-dc cluster, use “local quorum” to improve performance.

    Google Cloud Platform (GCP)

    1. Use GoogleCloudSnitch for Cassandra deployments in GCP.
    1. GoogleCloudSnitch treats regions as Cassandra data centers and availability zones as racks.

    Other Features

    Both AWS and GCP offer a series of operational capabilities that seem to make it easier to maintain high uptime on Apache Cassandra cluster. These capabilities have to be chosen carefully depending your deployment needs.

    1. Load Balancing: Network-level load balancing service is offered by both AWS (Elastic Load Balancer) and GCP. It is not recommended to put a load balancer in front of Cassandra. It is better to use a client connector that can handle this, consider open source utilities such Presto and Astyanax
    1. Autoscaling: Both AWS and GCP offer Autoscaling facility for compute instances. However, although this functionality looks very enticing, putting all C* nodes in an autoscaling group will not produce the desired result. Instead, create a separate autoscaling group of ‘1 node’ for every node in your cluster. This will ensure that anytime a node goes down, it is automatically brought back up. This will ensure the cluster data placement is not disturbed when nodes go down and nodes coming up do not cause excessive data movement.

    Scalable Backups and Reliable Point-in-Time Recovery

    Backing up your data is a must-have for enterprise organizations. The impact of data loss is so high that no enterprise wants to risk relying alone on replication. Protecting data in a public cloud environment may seem daunting, but there are several options available that can make this easier compared to an on-premise deployment. A simple option may look like one of the native snapshots:

    Storage Snapshots

    I have seen few customers use snapshots of EBS volumes as a way to protect the data in their cluster against logical and/or operational errors. This solution will work but with some severe limitations. On top of the overhead of maintaining the scripts to take snapshots, there is a large overhead in storage costs. Taking a snapshot of volumes does not allow for incremental backups of your data. This causes an exponential bloat in the amount of data stored in snapshots, leading to high cost of the backup solution.

    Amazon S3 or Google Cloud Storage

    Using an S3 bucket as the secondary storage enables enterprises to use low-cost, highthroughput storage, while reducing overall operating costs for storing backup data. Here are a few recommendations when using S3:

    1. S3 Region: Amazon S3 buckets are region-specific. A bucket created in one region will provide better performance characteristics to other infrastructure in the same region. Subsequently, it is important to ensure the S3 bucket used as the secondary storage is in the same region as the production database being protected and the Datos IO cluster.
    1. Storage Class and Lifecycle Policy: Amazon S3 allows users to create lifecycle policies on buckets. These policies dictate the performance and response time characteristics of objects in the bucket. STANDARD_1A is more suitable for longer lived and less frequently access data, and is not suitable as secondary storage. Additionally, do not turn on any lifecycle policies on the buckets used for the secondary storage. All buckets should be in the standard storage class.

    Conclusion

    As enterprises leverage the agility provided by public cloud environments, it’s important to understand the various facets of Cassandra deployments to ensure the efficient use of your money and time.

  • Top 5 Reasons to Back Up Your OpenStack Environment

    Top 5 Reasons to Back Up Your OpenStack Environment

    I am a big fan of lists. If you recall the movie “High Fidelity,” John Cusack’s character Rob and his co-workers—armed with an encyclopedic knowledge of all things musical—compile “top five” lists for every conceivable occasion, openly mock the tastes of people, and, every so often, sell a few records. In the spirit of a lists, here are the top five reasons to back up your OpenStack environment. We thought that it would be helpful to many.

    If you’re currently working with OpenStack clouds, or are in the process of embracing it, you need to include backup and recovery for OpenStack on your list of to-dos from the start. You may ask, “Why? I have orchestrated/automated everything, I just need to spin resources up and down.” Cloud environments are fundamentally different from the virtualization we have become used to. Virtualization relies on infrastructure for resiliency. In the cloud, distributed applications are responsible for their own resiliency, and are independent of the underlying infrastructure. Therefore, cloud apps need backup and recovery options, lest you face a complete machine, data or file meltdown.

    We’ve compiled the top five reasons to backup your OpenStack environment. Take a read and it will become clear why you must include backup in your OpenStack plans.

    1. Compliance — Compliance may seem like an issue for only certain industries—finance and health care come to mind—but nearly every organization has to deal with compliance issues. For some, it’s compliance to external regulations, while for others it’s simply complying with internal policies and best practices. Backup for OpenStack is an essential component for compliance; it ensures the availability of the data at a specific point in time within your OpenStack applications. I was recently involved in a dialogue/string on LinkedIn watching experienced OpenStackers go back and forth debating “pet vs. cattle.” “I am getting tired of it,” one person said. “This is cloud. It’s about scaling up and out … With automation and orchestration there is no need for cattle.” This conversation went on for days. Theory is great. Practice is reality. What people failed to discuss is that compliance or the business units will drive many requirements (such as backup) either today or tomorrow. End of story.
    2. DevOps — Backup is often the forgotten tool for DevOps. Containers are great to use, but the ability to test with real data is an issue. The reason to use backup is quite simple: test and development. With a copy of a production environment, you now can test upgrades and new features in your agile development process. Furthermore, if you lose a machine or your data, or you end up with corrupted files while in the middle of testing or development, your life will be much easier if you have an appropriate backup solution in place. Restoring machines, data and files is a better option than re-creating them entirely. Cookbooks and recipes are great as well, but you still need to couple it with the appropriate “ingredients.” It is like playing with a fake kitchen as a child … do you want to make breakfast with plastic pancakes and toast? Nothing beats the real thing. Yes, more resources are required for one over the other, but when something goes wrong—it is unfortunate—people point fingers.
    3. Disaster recovery plans — While the terms “backup” and “disaster recovery” often are used interchangeably, they are actually not the same thing. You might have what appears to be a great recovery environment (ideally one that mirrors your production environment for high availability), but without a comprehensive backup solution, your recovery might not be the painless experience you expect it to be. Perhaps the solution for intelligent disaster recovery is to marry backup with a Swift environment so you can recover from any point in time, anywhere in your organization.
    4. Save time — Often recovery can be a frantic process, driven by a business need to have the machine, data or files recovered as soon as possible. Or, more likely, sooner than possible. If you are trying to re-create something under that type of environment, there’s a likelihood that mistakes could be made. This becomes increasingly true the more complex and mature your cloud becomes (think about the growing number of workloads and applications). A much easier approach—one that provides you with peace of mind—is to have a comprehensive backup solution ready and available for when you need it. There’s also another reason backup can save you time: When you have to re-create the lost information, you are taking your time away from something else—and that means you are taking a big hit to your productivity. Save yourself and your organization a big headache with backup.
    5. Security and forensics — You never know when you are going to be asked to provide data or files related to a security issue or a forensics investigation. The right backup solution not only makes it easier to assist with the activity, but think how good you’ll look by being able to deliver the information the forensics team needs quickly.

     Are you currently looking into backup solutions for OpenStack? Now that you know why you need backup for OpenStack, learn more about how to backup OpenStack.

  • Continuous Security Key to Defending Your Data Well

    Continuous Security Key to Defending Your Data Well

    Defending data isn’t easy. If it were, we wouldn’t see new data breach headlines on a weekly—or sometimes even daily—basis. That said, defending data also isn’t rocket science. A new report from Nuix highlights some of the major concerns with protecting data, and also illustrates some of the ways in which consistent vigilance is half the battle.

    The Nuix report, compiled by principal researcher Ari Kaplan, covers a broad spectrum of concerns related to data security. Some of the key findings from the report include:

    • Increased granularity of security budgets: Respondents report there is a growing trend to specify how the security budget should be broken down and allocated.
    • Regulatory impact on spending has doubled: Government and industry compliance mandates continue to take a sizable bite out of security budgets.
    • Greater focus on insider threats: More than 70 percent of respondents have some sort of insider threat policy or program.
    • BYOD continues to play a role in security: 82 percent of respondents have some sort of BYOD (bring your own device) policy in place. A growing number of companies, however, are prohibiting remote access by employees.
    • Cloud adoption slowing: According to the study, about 70 percent of companies are migrating data to the cloud—a number consistent with the previous year—but fewer companies are moving systems or servers to the cloud. More than four out of five respondents agreed that the cloud creates unique cybersecurity concerns, and only 43 percent indicated plans to move systems or servers to the cloud.

    What sorts of unique security concerns does the cloud bring? According to the survey respondents, there are a variety of potential issues, such as:

    • losing visibility into the management of data
    • being at the mercy of the cloud provider’s security skills
    • reduced control over access to data; and
    • confusion about the privacy/legal issues when a government or law enforcement agency wants to access data.

    In a nutshell, none of this is really new or different. Compliance, insider threats, BYOD and concerns over security and privacy in the cloud have been prevalent factors in network and computer security for years—and likely will continue to be for years to come. What has changed, though, is the focus on ways to implement better security and the tools to do it.

    One of the primary issues with traditional computer and network security is that it’s reactionary. Scans provide moment-in-time confirmation that there are no known threats, but do little or nothing to protect against unknown threats or prevent exploits and compromises between scans. DevOps, and continuous security, change the game by enabling organizations to monitor for and detect security issues on a consistent, proactive basis.

    “Security practitioners cannot continue to rely on manual, human controlled processes to defend against attacks,” agreed Andrew Storms, VP of Security Services for New Context. “Automation is a requirement for the next evolutionary phase of security protection.”

    Nick Galbreath, founder and CTO of Signal Sciences, also believes that automation and continuous DevOps principles are key components of effective security. “I have argued that continuous deployment (how rapid the ‘continuous’ is up to the organization and risk-dependent on the nature of the code) is actually the most important security tool since without it, I don’t really know how you make things better rapidly.”

    There are a lot of names tossed around for various aspects of DevOps automation. The actual name is not as important as the actual practice of proactively automating security processes for more effective security. Take a look at the report from Nuix to learn more about the general state of data security, but make sure you embrace DevOps and continuous security if you really want to do something about it.