Author: Jeannie Liou

  • When Did Safety Nets Become Optional?

    When Did Safety Nets Become Optional?

    There are various reasons why some organizations would not be interested in protecting their data. These reasons could range from the lack of financial resources to the lack of non-financial resources, such as engineers to implement a solution. Regardless of whether there is a good reason to not protect your data, the net result is the same: If organizations don’t do it and something bad happens, they will be in a world of hurt.

    Data is the lifeblood of any business. It’s true that not all data is created equal, so some organizations may be comfortable losing some data. And it’s also true that there are some systems or applications that are built to not only tolerate data loss, but also seamlessly absorb the loss of data. The reality is that this is not 100 percent of the systems in your business, nor is it 100 percent of the data. If you were to completely lose the users table in your application or have it suffer from data corruption, how would you explain this situation to your customers? Can you realistically tell them that the money wasn’t there, or you didn’t have the resources to protect their data and yours? What type of trust would that instill in them for you as a provider of services? What type of trust would that instill in the business for your technology team?

    For most businesses, data protection should be thought of as a component of the overall security strategy. If you wouldn’t set up a network infrastructure without a firewall for protection, why would you create an application without protecting the data that powers it?

    Unfortunately, some form of system failure is inevitable. There will come a time when something bad happens. The best thing you can do is to be prepared as possible. Providing yourself with a safety net when this happens is doable, but shouldn’t be thought of as optional. There are some basics tenets that should be involved in providing a proper safety net. The most important of them is to plan for and failure and test your recovery procedures. This team education is your primary line of defense against disaster. It will help mitigate many of the potential failure scenarios through awareness. Data protection is for what happens when things get past this point. And protecting data should not be optional.

    — Jeannie Liou

  • Ransomware will Double in 2017. Are you Prepared?

    Ransomware will Double in 2017. Are you Prepared?

    A recent report found that ransomware attacks quadrupled in 2016, and are likely to double in 2017. Not surprisingly, a research survey showed that 48 percent of organizations had been hit by ransomware in the past year. And as we highlighted in a recent blog post, next generation cloud applications and databases are quickly becoming an attractive target for criminals.

    Attackers are targeting critical data or operations now, not just files. A few examples from the past two weeks highlight this trend:

    Attackers are getting much more clever and picking valuable targets within a business. It’s only a matter of time before ransomware hits cloud-based applications that run on these platforms. What can you do? Be prepared.

    A layered “onion strategy” means using a set of security tools to spot and isolate malware and ransomware, but that alone isn’t enough. The Department of Homeland Security issued an alert last year highlighting the key role backup plays in defending against ransomware:

    Employ a data backup and recovery plan for all critical information. Perform and test regular backups to limit the impact of data or system loss and to expedite the recovery process. Note that network-connected backups can also be affected by ransomware; critical backups should be isolated from the network for optimum protection.

    As the Texas police department example shows, recovering from a backup won’t work if the backup itself gets infected. A versioned backup of business data and files is essential to recovery. That’s where most backup solutions fall short, especially for next-generation applications and databases that cannot be quiesced to make a consistent backup copy across the database cluster due to their distributed nature. More importantly, given the large amount of data that is processed by these applications, it is critical that there is no bottleneck to data movement in and out of the cluster for an application consistent backup.

    — Jeannie Liou

  • A Modern Database Backup and Recovery Checklist

    A Modern Database Backup and Recovery Checklist

    Do you have single points of failure? How do you handle deduplication? These questions and more need consideration when you’re dealing with database backup and recovery.

    Enterprises are quickly onboarding next-generation, mission-critical applications such as eCommerce, artificial intelligence, machine learning and Internet of Things (IoT) on non-relational databases and distributed local storage. However, as organizations move these applications to scalable databases because they are easier to deploy, always available and cost-effective, they also need a robust and scalable backup and recovery solution to protect these modern databases.

    The challenge is that although enterprise adoption of next-generation applications has been rising, the risk of data loss is prevalent. That’s because non-relational databases lack any viable data protection and recovery solution.

    Here is what I mean:

    • Currently, there is a lack of a solution that allows for corrupted data to be removed, replayed and propagated with minimal downtime to customer-facing applications.
    • Reputable studies have concluded that as much as 75 percent of data disasters ultimately leading to downtime are the result of some sort of human error. Because human errors have a big impact on business operations and occur randomly, they cannot be solved by solutions that are offered today, such as native database replication or node-level snapshots.
    • Database replication does not provide point-in-time backup and any-point-in-time recovery, so enterprises cannot go back and fix these errors. In fact, if errors are introduced, the databases’ redundant-node replication can lead to almost immediate corruption across all nodes of the database.

    So, as you begin to explore backup and recovery solution for your next-generation applications, below is a list of items to keep in mind.

    Storage Is No Longer Unified or Shared Storage

    A key architectural transition is occurring where monolithic pools of shared storage are no longer used for next-gen applications and databases. Next-generation databases are scale-out in nature and use commodity storage at a node level, creating a highly distributed storage pool. This distributed storage pool provides both low cost and high performance, but is not conducive to using traditional database backup and recovery tools, which rely on database durability logs (from a shared storage LUN) for a point-in-time backup.

    Eventual Consistency Has a Big Impact on Backup and Recovery

    Next-generation databases are not built around the traditional ACID (Atomicity, Consistency, Isolation, Durability) transaction model. Transactions are a mechanism for guaranteeing these properties; they are a way of grouping related actions together such that as a whole, a group of operations can be atomic, produce consistent results, be isolated from other operations and be durably recorded. Eventual consistency, however, brings forth new challenges to generate a consistent point-in-time view of the database for operational recovery or test/dev needs. Therefore, novel and scale-out data protection technology to address eventual consistency is needed.

    Deduplication Matters at Scale

    With traditional databases, unified or shared database storage, block-level snapshots made image-based recovery and rollback relatively efficient. Distributed storage combined with a scale-out database that commonly involves replication of data rather than joins across tables led to a rapid proliferation of redundant data and increasing storage requirements (three to four copies of every write operation). Effective data reduction at scale is critical to success and economics for effective data protection in this environment.

    Single Points of Failure Do Fail

    Legacy data protection approaches have commonly involved a media server, which presents a choke point for secondary data path and single point of failure. Given the very high ingest rates coupled with the real-time nature of applications deployed on modern scale-out databases, next-generation data protection infrastructure has to be distributed to handle performance (RPO and RTO windows) and resiliency for always-on environments.

    Checklist for Modern Database Data Protection

    • How do you protect your next-generation scale-out databases today? What scenarios of data loss are you worried about?
    • What downtime can your applications or business handle in the event of data corruption?
    • Have you validated backup and recovery for your next-generation databases?
    • Can you prove compliance with regulatory or line-of-business requirements for backup of your critical next-generation infrastructure?
    • Can you recover to a desired or any point in time quickly? What manual effort is required to do this?
    • How much storage is required to store your desired pool of recovery versions?
    • Does your data protection approach scale out with your application and data growth?
    • Can you efficiently refresh your test/dev environment to a smaller or larger cluster size as needed?
    • What happens to your backup solution when a database node fails? Can you still perform backups? How about intermittent network failures?
    • Native formats free you from vendor lock-in and allow for data management services, such as search and analytics on secondary data.
    • Can you recover your data in native format?

    — Jeannie Liou

  • Backup and Recovery for Distributed Databases

    Backup and Recovery for Distributed Databases

    Over the past five years, applications have undergone a paradigm shift. They now require more agility, scalability and availability. The key trends in this era are driven by the use of social platforms, mobile devices and software as a service (SaaS).

    Being able to ingest large amounts of data from various sources and process that data in real-time to deliver contextual information or business insights is a major competitive advantage. To meet these new application requirements, organizations are forced to look beyond traditional relational databases. This has resulted in a slew of new database systems that are distributed and scale-out in nature, can be deployed on commodity hardware and offer tunable consistency-versus-performance trade-offs. It’s important to note that to cater to the agility requirements, database-as-a-service model in cloud is being adopted as well.

    To show just how big the shift to distributed databases is, below is a Keyword Search Trend graph showing the sharp rise in NoSQL search.

    screen-shot-2016-09-27-at-2-03-59-pmThe implication of this is that requirements for data protection have changed forever, driven by this data-centric world (web-scale applications and distributed databases). Organizations are collecting vast amounts of data and seek to drive more value from that data to make better and faster business decisions. Most distributed and cloud databases already provide replication capability and thus satisfy the availability requirements of data protection. However, data protection requirements for scalable point-in-time backup and recovery need to be addressed. Without point-in-time backups, organizations are at substantial risk of losing data due to human error, logical corruption and other operational failures. Traditional backup solutions were built to address the requirements of relational databases that used shared storage and had the ACID transaction model. Unfortunately, they fall well short in addressing the point-in-time backup requirements of distributed databases (local storage, eventual consistency and the elastic nature of infrastructure).

    As database architecture has fundamentally changed to meet new application requirements, data protection needs to be redefined and rearchitected as well. The following are the new requirements for data protection.

    • Take a durable point-in-time backup copy of an eventually consistent database; we call it “versioning” the new paradigm of data protection for this new distributed era: backups to snapshots to replication to copy data management to versioning
    • Minimize time to recover from failures (low RTO)
    • Scale with the growing needs of application
    • Enable easy refresh of test/dev environments for continuous development
    • Provide operational resiliency in event of failures
    • Provide deployment flexibility in public cloud or on-premises data center

    Most organizations are investing in enterprise-grade point-in-time backup and recovery products so they may deploy and scale their next-generation application on distributed databases with confidence. In the next five years, it will be exciting to watch companies redefine the data protection technologies to meet the requirements of next-generation applications.

    — Jeannie Liou

  • Will Deduplication Solutions Work for Cassandra?

    Will Deduplication Solutions Work for Cassandra?

    Cassandra is a popular next-generation database system (NoSQL) that powers the back end of high-performance web-scale applications in enterprises. It is database software for cloud applications that accelerates the organizations’ ability to power the growing number of cloud applications that require data distribution across data centers and clouds.

    While Cassandra offers high availability, it also provides significant opportunities for meeting data protection requirements. Creating cluster-consistent and space-efficient backup of a distributed database can be a challenging task. In this blog post, I will highlight existing deduplication solutions for Cassandra, why deduplication matters, and how to achieve deduplication in Cassandra.

    What is Cassandra?

    Here is a brief introduction. You might be curious, “Aren’t existing deduplication solutions enough to save space for Cassandra snapshot files?”  There are multiple deduplication solutions that can eliminate redundant data from the different levels of the storage layer. So why do we need a new one for Cassandra cluster backups?

    Let’s begin with some background information.

    Cassandra is a distributed database that is becoming increasingly popular with the emergence of big data applications such as software as a service (SaaS), internet of things (IoT) and real-time analytics. These applications require high availability and scalability over consistency. Cassandra supports eventual consistency rather than strict consistency, which is provided by traditional database systems such as Oracle, MySQL and IBM DB2. “Eventually consistent” means consistency will be achieved eventually rather than immediately. As well-known in CAP theorem, we cannot have following three properties in a single system: consistency, availability and partition tolerance. In short, Cassandra is an eventually consistent database that provides high performance, high availability and high scalability—but not strong consistency.

    Replication and its Role in Deduplication

    One of the most important mechanisms of distributed scale-out database systems such as Cassandra is data replication. By replicating the same data on different nodes across failure boundaries, distributed database systems can continue to service application requests even with a certain number of node failures. The downside is performance overhead to maintain multiple data copies; both write and read operations will be slower to create multiple copies and to check the consistencies among multiple copies. Although asynchronous data replication technique can be used to minimize the performance overhead of writes, it also would lower the level of guaranteed consistency level, which called is eventual consistency.

    As I just explained, replication plays a very important role in a distributed database system, and therefore we should not remove the redundancy from a live Cassandra cluster to save storage space.

    The situation becomes different when we think about backup files (or, secondary data) from a Cassandra cluster. Like any other database system in an enterprise organization, backups are needed for Cassandra; it is not because Cassandra is not reliable or not available enough, but primarily because people make mistakes (“fat fingers”) and enterprise applications sometimes have to keep the history of their databases. As they say, to err is human!

    Cassandra has a nice node-level snapshot feature, which can persist a complete state of an individual node to snapshot files. One very important point is that a Cassandra snapshot is a “per-node” operation, which does not guarantee anything about the cluster status as shown in the figure below.

    1

    Backing up a Cassandra Cluster

    To create a backup of a Cassandra cluster, we have to trigger a snapshot operation on each node, collect created snapshot files, then claim the set of collected snapshot files as a backup. In this backup, the replicated data exist as is, and the size of the backup will be N times bigger than the size of user data, where “N” is the replication factor. Replication has an important role in a “live” Cassandra cluster to provide high availability and scalability, but what’s the use of replication for backups? If we upload the backup files to an object store such as S3 or Swift, the “already replicated” data will be replicated again by the object store to provide reliability and availability for their own sake. In short, there are redundant data copies in Cassandra backup files (secondary data). If we can eliminate the redundant data copies we will save massive storage space for Cassandra backups without sacrificing retention periods. Saving on storage space directly translates to saving big bucks for the operational cost of maintaining and operationalizing a big data system across an organization.

    But, let’s return to the question of whether existing deduplication solutions would work for Cassandra backup files. You can test this by collecting and placing Cassandra database files to a deduplication system. Will the solution can save storage consumption? No! Existing deduplication solutions wouldn’t work for Cassandra data files for the following two reasons:

    • Cassandra has a masterless peer-peer architecture. Each node receives a different set of rows and, therefore, there are no identical nodes in a cluster, which means data files will look different, as shown in the following figure. Since different combination of rows are stored in Cassandra data files, each data file will be hardly identical even at the chunk level.2
    • Cassandra data files are compressed with 64KB-sized chunks regardless of row boundaries. If you know enough about deduplication algorithms, you can easily understand why Cassandra data files cannot be easily deduplicated. Fixed-length, chunk-based deduplication will not work because of the chunk alignment, and variable-length chunk-based deduplication will not work because of compression.

    Cassandra’s compaction and small record size are other reasons why the existing block- or file-level deduplication solutions will not work for Cassandra backup files. Compaction is an independent operation that merges multiple database files into a new data file; small-sized records can be hardly deduplicated with bigger-sized chunk.

    Summary

    The existing file- and block-level deduplication solutions will not work for Cassandra backup files because:

    1. Each Cassandra data file contains different set of records.
    2. Cassandra data files are compressed regardless of the row boundary.
    3. Cassandra runs compactions independently.
    4. Cassandra can have smaller record size than deduplication chunk size.