devopsy.kinsta.cloud

Tag: big data

  • DevOps and Data Portability

    DevOps and Data Portability

    You have a classic car. You’re in the middle of restoring it, and it looks mighty sweet. Paint is polished, wheels are stock, hubcaps (or spokes) are shiny, engine is completely restored to original spec.

    But it has no gas tank.

    A bit of a stark comparison, but that’s where we’re at with application release automation (ARA)—at least in terms of portability. And where we’re likely to stay.

    Hold on, I’m a Fan

    Before the DevOps aficionados get out the torches and pitchforks, let me be completely clear that I’m a huge fan of ARA, and explain again why.

    Tools such as Jenkins allow DevOps teams to automate the entire build/test stack—or, as much of it as their organization requires. That is a huge step forward, for all the reasons we’ve covered in the past, including the one I think is often overlooked or underemphasized. With frequent build/test feedback, the code is still fresh in the developers’ head when they learn of problems. Most of the time, they are fixing before moving on to the next task, which is a huge plus compared to waterfall, where many problems show up long after the developer in question has forgotten what the code is doing because they’re off on other things.

    Likewise, tools such as Puppet, Ansible and Chef (to name a few) allow DevOps and security to set standards via script, and use those scripts to deploy over and over. This has all the benefits we’ve discussed, too—speed of delivery, reduction of human error, repeatability that becomes stability, etc. This gives operations the ability to answer, “Is server X set up to do this new thing?” by looking in deploy scripts and quickly updating things if the answer is no, but X will be required going forward. That’s one of my favorite bits of Application Provisioning.

    Riding the Gap

    For those unfamiliar, ARA simply bridges the gap between DEVops and devOPS. Tying code being released to platforms and configurations it will be released under. It has the ability to rapidly re-target a release to a new environment, and it can do predictable and repeatable releases, offering the benefits of application provisioning for the actual release segment. Most tools do this by tying together your favorite build tools and provisioning tools, and giving Ops a control panel that can manage the process end-to-end.

    The benefits are huge. Particularly for Operations and Security (who have another place to standardize and insert essential monitoring), but also for development because it offers one further step in testing and by doing so reduces the surprises six months down the road, when you’ve forgotten what

    Bubblegum += 10

    Is in your code for.

    So don’t take my intro as offensive. I am a fan. But I’m a realist. There is no panacea.

    It’s About the Data (as Always)

    The two weakest parts of most ARA systems are server provisioning and data issues. Server provisioning is setting up OS/hardware, and many ARA systems tend to assume you magically did elsewhere, but this problem is only an issue in highly physical environments and there are tools to handle the problem; they’re just not integrated well with most ARA systems. Data issues are the bigger problem.

    It’s cool that we can drag and drop a couple boxes, and magically switch our deployment from inside the DC to a cloud provider in the time it takes the ARA to build/deploy. But we all know that’s not how the world works. Now you have an application, complete with database that is correctly configured and security that does AAA solidly … and no data.

    “Just move the data, then!” the choir sings. No, that’s really not how it works in an enterprise. Data itself is an object of security and compliance, and some source databases are so massive that moving them is a nearly impossible chore in and of itself. Amazon tips its hat to this fact with its import/export service.

    This was the reason why I gut reacted to the “No more physical data centers!” crowd back in the midst of the hype with an eye roll. The amount of data stored in data centers is staggering, and often not portable at all. Add to this the fact that such data isn’t going to reduce, and we have a quandary.

    What About Moving Forward?

    Sometimes, it is necessary to accept that the current state is not changeable, but take steps to improve the future state. I don’t see this as a reasonable approach in this case, because the rules of data aren’t changing in a way to change future state in a positive way. We’re generating more data every day, and that data will end up stored somewhere. Be careful where you store it, think long-term strategic. Having ARA doesn’t magically make that data portable, so make sure that where you store it today, while it’s small, is where you’ll want to store it in five years, when it is a massive, largely immobile asset.

    What to do … What to do

    Use ARA to multitarget all of those apps that are portable without terabytes of data, and make sure data generation/collection/analysis apps are on the architecture you will want them on moving forward. ARA offers a ton of other benefits, so data portability isn’t a deal breaker—or even a significant negative, since you probably already knew some datasets weren’t going anywhere—but don’t expect miracles out of it. It can’t make your large datasets portable, so don’t expect it will. And use it to target the platform your organization feels is most appropriate for apps that will generate or collect a lot of data.

    — Don Macvittie

  • Study: CIOs Want a Chief Data Officer

    Study: CIOs Want a Chief Data Officer

    Quick, what does CDO stand for? Before you say chief digital officer, think again. A new study points to the chief data officer as the more important of the two titles, finding 82 percent of CIOs believe there is a compelling case to hire a CDO in most organizations. The survey, commissioned by Experian Data Quality, was completed by 200 CIOs and 50 chief data officers in June and released in August.

    But while the report stresses the importance of the CDO to companies that want to become data-driven, it also reveals severe challenges for some CDOs, including very high expectations, insufficient resources and, despite the C-level title, a lack of authority to get the job done.

    The survey states that the core responsibility of the CDO is “to define a data-management strategy and translate it into tactical, [achievable] steps for the business.” To make data actionable, it needs to be accessible, accurate and standardized. Here’s a simplistic example of why that is important: Let’s say a data analyst needs to create a report that isolates customers based in New York City. If some of them are listed as being headquartered in NY, NY and others are listed as based New York, NY and still others are in the database as New York City, NY, that’s not standardized. The data analyst would need to realize the data is splintered and needs to be stitched together to get accurate results.

    Making your business data-driven is part of the potential promise of digital transformation. CDOs and their ilk face a daunting task. Transforming data into business intelligence is the alchemy of this age.

    Chart courtesy of Experian Data Quality

    The study identifies three main drivers for hiring a CDO: capitalizing on big data opportunities, creating a competitive advantage and reducing risk on data-driven projects. These roles go far beyond the data management and data governance toward enabling business innovation. Over the next few years, the CDOs surveyed predict that their top business challenges will concern data security, keeping pace with technology advances and managing increasing user expectations.

    Perhaps one of the biggest challenges some CDOs face is the inconsistency across companies of the nature of their role and their stature within the organization. Bottom line: They may not have the authority they need to succeed. The survey reveals, for example, that the average tenure of a CDO is only two years. Many CDOs report that despite their C-level title, they are not considered to be a part of the C-suite. Although 37 percent report directly to the CEO, a larger percentage reports to the CIO, CTO or some other part of IT. That can create some friction because it’s important that CDOs are able to bridge the gap between IT and lines of business—and empower the business.

    Tips for Structuring the CDO’s role

    In creating a new CDO position, the C-suite should be advised of these common challenges faced by CDOs identified by the survey:

    • Staying within budget
    • Keeping up with evolving technologies
    • Knowing how to interpret information
    • Finding/hiring people with data expertise
    • Managing ever-increasing user expectations
    • Balancing the needs of business users and IT
    • Having a lot of responsibility with little room for error
    • Getting pulled into the weeds
    • Maintaining data security

    You should consider extending these responsibilities to your CDO:

    • Having significant control of the management and use of company data
    • Playing an active role in digital transformation
    • The responsibility for enterprise-wide governance and utilization of information as an asset
    • Manager of a team of experienced data employees
    • The purview to implement different approaches to managing data as an asset
    • The purview to generate business intelligence that unlocks and supports business innovation
    • The authority to be recognized by the entire company is the person overseeing data strategy and process
    • Recommend: At least a dotted-line report to the CEO.

    — Scot Finnie

  • 4 Predictive Analytics Challenges Facing SRE Teams

    4 Predictive Analytics Challenges Facing SRE Teams

    “Predictive analytics” refers to the many techniques used to analyze data to make predictions about the future, such as data mining, statistics, modeling, machine learning and artificial intelligence. Being able to predict technical and business outcomes is one of the main promises of big data. Indeed, many businesses have invested heavily in infrastructure to anticipate key performance indicators such as demand, pricing or maintenance. The investment required is substantial but, when it works, it yields a very positive ROI.

    In this post, we’ll examine why implementing predictive analytics in a site reliability engineering (SRE) context requires a significant investment in both setting up the right data sets and applying the right logic to that data.

    SRE should in theory be a prime target area for predictive analytics. Why? Because companies have already widely adopted monitoring tools over the past decade. In doing so, they’ve amassed extensive system health and behavior data that is ripe for analysis. SRE teams have also increased their ability to make sense of this data as data science tools have become more accessible. Basic data science aptitude is increasingly become a desired skill when hiring engineers.

    Additionally, the SRE team is responsible for a mission-critical KPI: system availability. This single metric is directly tied to customer SLAs in digitally dependent businesses. SRE teams have a strong incentive to implement predictive analytics. Why? Having the ability to prevent downtime by addressing potential issues before they hit the production environment is hugely valuable. More system availability translates into more revenue, higher customer satisfaction and lower costs of problem mitigation.

    However, the reality is that an SREs job is still mostly focused on the diagnostic, mitigation and “fixing” parts, as opposed to more proactive tasks. This is in big part because predictive analytics applied to SRE/DevOps has proven difficult to implement and perform in a disciplined manner. Therefore, it is still an aspirational goal at most companies.

    SRE Challenges in Predictive Analytics

    Up next, let’s explore the four challenges that need to be overcome to implement high-quality predictive analytics practice in an SRE context, including:

    • Data collection
    • Data quality
    • Data volume
    • Model usability

    Challenge 1: Data Collection

    First, the working data sets need to fit what you are trying to predict. In this case, it is challenging to get the right data set because the SRE team’s mission is very broad; specifically, ensuring the reliability of the entire system. The difficulty lies in the fact that there is no such thing as “full system” data readily available. All the relevant data is distributed across siloed, or overlapping repositories and monitoring tools. Centralizing the data requires either a full system agent (which takes a lot of work to set up) or through APIs, which requires mastering a multitude of API integrations. An additional obstacle in getting the right data lies in the inconsistent nature of events, time series data and logs. These disparate data types can’t be analyzed together unless they can be transformed into a uniform data set.

    Challenge 2: Data Quality

    SRE environments emit lots of data and some won’t make sense for the purpose of the analysis. Whichever data collection method is used, the information tracked often reflects some level of bias on the part of whoever set up the tracking mechanism in the first place. This leads to capturing both irrelevant data or inadvertently leaving out important information. In other words, the data is often both noisy and lacking at the same time.

    The key to successful modeling lies in the selection of prediction variables with data that is obtained before the event predicted happens (in this case, a failure). Defining the variables that lead to the determination of a subsystem failure is critical and requires specific domain knowledge. It is difficult to define and extract a pure data set sanitized of noise. The more data you leverage, the higher the chances that you are actually “feeding the beast” with irrelevant information. Too much irrelevant data leads to inaccurate models and false positives.

    Challenge 3: Data Volume

    Here you can have two different types of challenges: too little data and too much.

    Not enough data

    You typically need at least 100 instances of what you are trying to predict and probably at least 100 where it didn’t work to train a model. This assumes the data set is clean and directly relevant to a very narrow, specifically defined problem. In the case of SRE, the scope of issues that can affect a system is such that the data set required is very substantial. Likely, it’s several months worth of a wide variety of system behavior measurements. For example, if you collect data only through passive collection, think about the lag time between t-zero (when you start collecting data) and the time when you would have results. During that time, you have a bootstrapping problem—your model can’t be trained to predict accurately what you want.

    Too much data

    There are two practical points to make around the notion of “too much data.”

    • Lack of standardization: Processing large volumes of data for analysis remains a difficult engineering challenge. There is no standardization for data pipeline; every method has a limit in terms of how many inputs it can ingest on a per-second basis, and ingest rates often end up becoming bottlenecks in large enterprise environments.
    • Volume does not equal accuracy: There is a point beyond which you get diminishing returns: statistical theorems show that after a certain point, feeding more data into a predictive analytics model will not provide more accurate results.

    Overall, getting the quality and quantity of the training data right is the best investment you can make in building your predictive capabilities.

    Challenge 4: Model Usability

    From a pure “mechanics” standpoint, the analytical concepts and methods are there to reflect the complex thinking required to predict the behavior of an enterprise’s combination of technology, information and infrastructure. Predictive software relies heavily on advanced algorithms and methodologies such as logic regressions, time series analysis and decision trees. The problem is that the workings of the kinds of systems that SREs deal with are very hard to understand in the first place, let alone predict. It is not uncommon even for SREs to not entirely understand how a system works if they were not the ones who initially set it up. In addition, every company’s system is different.

    When data scientists design predictive models without years of experience in an SRE capacity in multiple environments—in other words, without leveraging subject matter expertise—it rarely works perfectly. Often the models they create lack usability, even after the validation process to confirm its quality. A modeler will split the available data into a training set where the model is run to generate predictions and compare performance against the actual outcomes. That will require multiple iterations to generate a model that has high overall predictive accuracy, and 9 out of 10 times the result is overfitting, or perfect for the training data. A subject matter expert in the domain of technical operations can cut the cycles of iteration and get to a predictive model that works with an acceptable level of risk.

    In short, implementing predictive analytics in an SRE context takes a significant investment both in setting up the right data set and applying the right logic. Stay tuned for a future blog post that will provide some pointers on how to achieve the best prediction results.

    About the Author / JP Emelie Marcos

    JP Emelie Marcos is the co-founder and CEO of SignifAI. Previously, JP served as the Chief Operating Officer of Tango, a leading mobile consumer application that reached 400 million regular users in five years. JP’s career spans 20+ years leading teams in technology startups and large companies as an investor, GM and CEO.

  • Gartner: Digital Business Driving Spending

    Gartner: Digital Business Driving Spending

    With half the year gone, Gartner just revised its 2017 worldwide IT spending projection up 1 percent to 2.4 percent. That may not sound like a lot, but last year’s IT spending growth was effectively flat at 0.3 percent. It isn’t just the uptick in the overall spending level that’s interesting, it’s how Gartner predicts the money is being spent. It sees the move to digital business and the related adoption of new technologies such as internet of things (IoT), blockchain and smart machines as a growth engine that’s creating new potential reasons for spending.

    “Digital business is having a profound effect on the way business is done,” said John-David Lovelock, vice president and analyst at Gartner. “The impact of digital business is giving rise to new categories; for example, the convergence of ‘software plus services plus intellectual property.’ These next-generation offerings are fueled by business and technology platforms that will be the driver for new categories of spending. Industry-specific disruptive technologies include the internet of things in manufacturing, blockchain in financial services (and other industries), and smart machines in retail. The focus is on how technology is disrupting and enabling business.”

    Worldwide, Gartner is forecasting that the enterprise software market will be the highest growth segment of IT spending—7.6 percent growth in 2017 from 5.3 percent in 2016. (Device spending is also up markedly in the report.) Enterprise software will be one of the keys to driving and controlling revenues derived from digital business advances.

    Takeaways for senior IT leaders: Digital disruption is the probable stick slowly driving the adoption of digital business. The carrot is innovation. This is putting it too simply, but it’s true: Disrupt or be disrupted. And just because many of the high-profile digital disruptions have been business-to-consumer, don’t think you’re safe in the business-to-business sector, either. As this plays out, probably over the next five to 10 years, a lot more companies are going to be burned and a lot more companies are going to be made. You can no longer laugh this off as a buzzword.

    It starts with analytics, preferably sifting through meaningful big data. Digital business is rewriting business process and the way data is collected, interpreted, moved and consumed in profound ways. That could make your analytics easier to implement and analyze. But if you don’t have a major business analytics upgrade just rolling out, you’re flying blind.

    Ride the wave, but as you adopt new cloud-based services and enterprises apps, there’s a potential for management complexity to trip you up.

    “With the increased adoption of SaaS-based enterprise applications, there also comes an increase in acceptance of IT operations management tools that are also delivered from the cloud,” said Lovelock. “These cloud-based tools allow infrastructure and operations organizations to more rapidly add functionality and adopt newer technologies to help them manage faster application release cycles. If the I&O (infrastructure and operations) team does not monitor and track the rapidly changing environment, it risks infrastructure and application service degradation, which ultimately impacts the end-user experience and can have financial as well as brand repercussions.”

    — Scot Finnie

  • Is DevOps Making Life Better for Data Scientists?

    Is DevOps Making Life Better for Data Scientists?

    The path to success and happiness of the data science team working with big data project is not always clear from the beginning. It depends on the maturity of the underlying platform, their cross skills and DevOps processes around their day-to-day operations.

    The past few years have been like a dream come true for those who work in analytics and big data. There is a new career path for platform engineers to learn Hadoop, Scala and Spark, while Java and Python programmers have a chance to move to the big data world. There they find higher salaries and new challenges and get to scale up to distributed systems.

    But recently, I have started hearing some complaints and dashed hopes from engineers who have spent time working there.

    Their expectations are not always met. Big data engineers spend most of their time in extract, transform and load (ETL), preparing data sets for analysts and coding models developed by data scientists into scripts. They have become cogs between data scientists and big data.

    I see the same feedback from data scientists. Real big data projects differ from Kaggle competitions. Even minor tasks require Python and Scala skills as well as multiple interactions and design sessions with big data engineers and operation teams.

    DevOps engineers get to go no further than just using the basic tools and running through tutorials. They grow weary of being given nothing more to do than deploying the next Hadoop cluster through the Cloudera wizard and semi-automated Ansible scripts.

    In this environment, the big data engineer will never get to work on scalability and storage optimization. Data scientists will never invent a new algorithm. And the DevOps engineer will never put the entire platform on autopilot. The risk is the team’s ideas might not make it beyond proof of concept.

    How do we change this situation? There are three key points:

    1. Tools evolution—the Apache Spark/Hadoop ecosystem is great. But it is not stable and user-friendly enough to just run and forget. Engineers and data scientists should contribute to existing open-source projects and create new tools to fill the gaps in day-to-day operations.
    2. Education and cross skills—when data scientists write code they need to think not just about abstractions but also consider the practical issues of what is possible and what is reasonable. For example, they need to think how long their query will run and whether the data they extract will fit into the storage mechanism they are using.
    3. Improve the process—DevOps might be a solution. Here DevOps does not just mean writing Ansible scripts and installing Jenkins—we need DevOps working in optimal fashion to reduce hand-off and invent new tools to give everyone self-service to make them as productive as possible.

    The Continuous Analytics Environment

    With continuous analytics and self-service, data scientists own the data project from the original idea all the way to production. The more autonomous they are, the more time they can dedicate to producing actual insights.

    Data scientists start from the original business idea and work through data exploration and data preparation. Then they move to model development. Next comes deployment and validating the environment. Finally, there is the push to production. Given the proper tools, they could run this complete iteration multiple times a day without relying on the big data engineer.

    Big Data engineers, in turn, work on scalability and storage optimization, developing and contributing to tools such as Spark, enabling streaming architectures and so on. They provide an API and a DSL for the data scientist.

    Product engineers should be given analytical models developed by data scientists as neatly bundled services. Then they can build smart applications for business users to allow them to use those in self-service.

    Working like this, no one is stuck waiting on the next person. Each person is working with their own abstractions.

    This might sound like a utopia. But I believe this could motivate big data teams to improve their existing big data platforms and processes, with the added benefit of making everyone happy.

    About the Author / Stepan Pushkarev

    Stepan Pushkarev is the Head of DevOps Practice at Squadex.com and CTO of Hydrosphere.io. Co-founded and managed engineering teams for eCommerce, IoT and Ad-tech companies. He has been responsible for the full products stack: math models, infrastructure & operations, enterprise applications as well as hiring, establishing engineering culture and delivery process. Stepan combines strong technology, management skills and entrepreneurial spirit. Connect with him on LinkedIn.

  • Backup: Not So Easy in Certain Environments

    Backup: Not So Easy in Certain Environments

    We discussed how backup needs to be reinvented for a new era cloud-native applications, and what you should look for in a modern database backup and recovery solution. But even if you scrap the traditional approach to back up and write scripts to handle the task, backup and recovery of next-generation applications and databases (Apache Cassandra, MongoDB, Amazon DynamoDB, Microsoft DocumentDB, Apache HBase, et al) can be frustratingly difficult. But why is that?

    The short version is this: In an eventually consistent write-anywhere non-relational database architecture, it is almost impossible to capture a consistent state backup. Successful data recovery is even less likely!

    There isn’t just one reason why this is the case. It comes down to the basic nature of a distributed architecture that’s designed to both scale and withstand node failures without downtime. As we’ve highlighted below, there are several challenges from the start that make backing up a distributed architecture difficult:

    • Data is written to an available node. The first landing point for data is not predictable, so there’s no way to catch data as it lands on the node.
    • Then data is replicated to at least one other node before verification. That ensures a valid write, but also immediately creates a copy of the data.
    • … then replicated to one or two more nodes for availability. This happens afterwards, and the same data can get updated in quick succession.
    • … meaning there are three or four copies of data at any time.
    • … and nodes are never immediately consistent.
    • … and backups are hugely inefficient since every replica is copied.
    • And when a node fails, the topology changes mid-flight.

    Theoretically, a good DevOps team might be able to write scripts that will back up the database successfully 80 percent to 90 percent of the time (cases such as multi-node failures, changing topologies, compaction of databases and many more are complex to model, so scripts are difficult to get right).

    Unfortunately, backup is the “easy” part of the equation. Restoring a backup is where it really gets challenging. A successful restore is far more complicated than most people assume. It involves the following steps:

    • Rebuilding the correct topology. Because each node was backed up individually, the database has to be restored to the same topology (six nodes to six nodes, 12 nodes to 12 nodes), etc. This relates to enabling recovery across clouds, test/dev and CI/CD use cases.
    • Waiting for the database to repair and restore itself. The non-relational database architecture can withstand node failures and keep running, but a full restore with data reconciliation will bring back memories of the worst RAID rebuilds on slow drives.
    • Deduplicating multiple inconsistent copies. The backup copy has three or more copies of all data that may be in an inconsistent state. It needs to be deduplicated or reconciled first.
    • But data history isn’t always available. In most distributed architectures, the oplog is a circular buffer that essentially eats itself as it goes! It may not be possible to even recover the necessary log data to reconcile the database.
    • And there’s no reliable way to go back to a point-in-time. Even with the oplog data, it would be very difficult to reconstruct point-in-time data. At best, you’d get a recent, mostly accurate copy of the database.

    In the real world, restores can take days or weeks—if the data can be recovered at all.  But as the recent Gitlab outage shows, it is difficult for even technically savvy organizations to get it right. Human error can be as fatal to a database as a natural disaster if there isn’t a reliable backup and recovery process in place.

    — Shalabh Goyal

  • Evolving Data Requirements in Multi-Cloud Environments

    Evolving Data Requirements in Multi-Cloud Environments

    By design or by accident, multi-cloud is a reality that most enterprise IT teams have to cope with today. Multi-cloud (or, cross-cloud, as Dell EMC calls it) refers to an IT model where an organization uses services of one or more public cloud service providers, sometimes in addition to using its own data centers. In a multi-cloud model, different clouds may be used in different stages of the application life cycle such as test/dev and production, and for different types of applications such as traditional structured applications and next-gen cloud applications. Nonetheless, IT leaders face challenges in identifying data management tools they need to embrace the multi-cloud environment to improve application agility, reduce operational hassles and enhance application resilience. In this blog, I will discuss how enterprises are using a multi-cloud model and the requirements it places on next-generation data protection products.

    One common reason for multi-cloud is to support geo-replicated applications to enable global reach of e-commerce organizations. In such a scenario, a production database may reside on a private cloud behind an enterprise firewall and replicated to multiple regions in a public cloud to service reads (i.e. consumers are shopping on these e-commerce sites) from different regions at low latency. Such a configuration also protects against disasters at the data center level because the database and application are replicated to public cloud (with multiple availability zones).

    Another use case is to service the requirements of production environment versus test and development environments using different clouds. While security, control and performance optimization may mandate a production environment to stay within a private data center, agility, access to new capabilities such as Google BigQuery and developer preferences may drive application development teams to use a public cloud service providers.

    Finally, using a multi-cloud model could also be driven by the IT strategy of an enterprise to prevent overdependence (or, vendor lock-in) on any one public cloud service provider. To hedge the risks and get leverage, an enterprise may balance its application workloads across multiple cloud providers—for example, deploy Tier-1 applications on one cloud and Tier-2 applications on another. It is a real phenomenon we are seeing with enterprise customers! However, a multi-cloud model presents the following unique challenges to enterprises.

    • Data protection in multi-cloud environment: Legacy data protection (backup and recovery, archiving, replication) products were architected for on-premises environments and optimized to work with legacy applications. Using legacy backup and recovery tools for next-generation applications in a public cloud environment is equivalent to fitting a round peg in a square hole. For one, legacy products don’t leverage the elastic and scalable compute and storage that is available in public cloud. True infrastructure-independent (software-only and elastic compute-based) data protection products that may be deployed in heterogenous environments are required as enterprises look to migrate applications to the cloud.
    • DevOps agility: Moving data across clouds for disaster recovery or to refresh test & development environments with production data is another key requirement for enterprises. Doing so in a multi-cloud environment is not an easy task given the challenges in moving data over the WAN. There are also differences in configuration between test/dev and production environments. Further, the test/dev environments may not be as large as production environment so sub-sampling is required. Finally, personally identifiable data needs to be masked before it leaves the production firewall.
    • Ease of management and governance: Finally, federated management is required for applications and workloads running across multiple clouds. It is extremely inefficient to get familiar with different management tools and their intricacies.

    This is by no means a complete list. There are several other requirements, such as support for cloud storage with consistent performance, handling of identity access and management processes and so on. The point is that data protection needs to be re-thought in the context of multi-cloud environments.

    — Shalabh Goyal

  • Don’t Be Samsung: Here’s How You Can Protect Yourself

    Don’t Be Samsung: Here’s How You Can Protect Yourself

    Imagine a factory building 100,000 units a day, and you can see how a small error can have huge consequences. Now imagine that factory is a database.

    The world’s top smartphone maker this month pulled the plug on its $882 Galaxy Note 7 after phones overheated and caught fire, concluding in a disastrous recall for Samsung that could cost the company as much as $17 billion and shutter a global revenue stream. In an unprecedented move, Samsung has told mobile carriers to halt sales or exchanges of the device, requested that users to shut off their phones and issued a mass recall while the South Korean manufacturer investigates reports of fires in original and replacement Note 7s.

    This a scant two months after the Galaxy Note 7’s launch, a device that one TechCrunch consumer electronics expert raved as “undoubtedly one of the most iconic handsets around.” While it is not clear what caused these fires, there is speculation that it was due to a faulty component that interacts with the battery, rather than the battery itself.

    It goes without saying that there are ample opportunities for error in the assembly of a smartphone. The complicated, highly coordinated process is fraught with thousands of input touchpoints, from hard- and software components to tightly controlled manufacturing processes to manual labor. And though there may be countless checks and balances in place, mistakes do happen. What may seem to be an innocuous error, such as not sufficiently tightening a screw or the shorting of a circuit, in this case has led to epic consequences.

    The truth is, these instances happen more often than we think, and to technologies even more vital to the population than a personal smartphone.

    Consider today’s era of big data and cloud applications, where enterprise applications ingest and process vast amounts of information from various sources, often in real time, to deliver critical analytics, contextual information or business insights.

    This data has weight. It can influence whether fleets of airplanes are grounded at gates all across the country. It can restrict customers from accessing funds from online banking. It can dictate whether millions of online retail dollars go unspent. The reliance on accurate data, and its grip on lives and business, is growing at an exponential rate—like ever-widening ripples in a lake.

    To support all of this data, new applications (such as real-time analytics, internet of things, ecommerce and others) are developed and deployed on scale-out, non-relational databases or ported from traditional relational databases.

    But, like the manufacturing gaffs experienced by Samsung, corruptions and operational errors by developers occur in enterprise IT environments. Enterprises know this, and billions have been invested in traditional backup and recovery and data protection products to protect against such “err” moments. However, the irony is that data protection products like those that exist for traditional relational databases do not exist for today’s modern, next-generation scale-out databases supplying the new types of data that increasingly control our lives. We’re essentially trusting that the Ma Bell rotary phone repairman will be able to recover personal data—such as contacts, photos and messages—after your Galaxy Note 7 starts smoldering.

    So how did we get from the world’s most explosive phablet to distributed databases and big data? Because, like the Galaxy Note 7, the tiniest data error can rapidly expand into magnificent, catastrophic consequences.

    Enterprises are adopting cloud applications that gather large amounts of data at a high-ingestion rate and process that data in real-time to deliver actionable insights. These applications are deployed on non-relational databases. As migration to cloud and multi-cloud application environments increases, enterprises also must be able to manage and recover at scale without data loss or corruption and with minimal application downtime.

    However, enterprises today face a significant gap in addressing their data protection requirements. Their most valuable data, often needed 24/7 and without fail, runs absent any point-in-time backup options for that ‘“Houston, we have a problem”’ moment. Most studies suggest that more than half of data downtime and loss is triggered by human error, corruption or virus attacks. And yes, while native replication helps enterprises meet business continuity and disaster recovery requirements by providing protection from hardware failures or even natural disasters, it is insufficient (meaning it cannot scale) to meet next-gen enterprise data protection demands. Case in point: If a minor schema corruption impacts the primary data copy, it causes a multitudinous ripple effect that impacts all replicas down the line. Yet, even without protection, industries continue to adopt and deploy scale-out database systems at an accelerated pace.

    So just as Samsung witnessed an error replicate itself into such a massive meltdown of global proportions that it threatens to kill an entire product line, organizations also must understand how replication-based protection mechanism can actually act like a virus—spreading a corruption uncontrollably across non-relational, cloud databases.

    Samsung’s troubles won’t end with the simple recall of a single product, and their predicament should resonate. There will be punitive measures, and perhaps even a total enterprise shift from the smartphone market. It is not an enviable position, but is does offer a clear example of how a peripheral manufacturing error can swell to engulf an entire brand.

    For enterprises rushing to deploy scale-out databases, let Samsung serve as a warning: that deploying any infrastructure that could potentially lead to exponential, irreversible corruption of all data copies is a recipe for harm (if not disaster). As enterprises make the switch to next-gen applications running on scale-out, non-relational databases, an assembly line e-stop needs to be built in, in the form of a point-in-time, version-friendly data protection solution.

    Data corruption is not a matter of “if,” but “when.” So have a kill switch ready.

    errors-replicating-image

    — Brian Yin

  • DataOps: Time to Bring DevOps Ideas to Big Data

    DataOps: Time to Bring DevOps Ideas to Big Data

    What can data scientists learn from DevOps? A lot, actually. By applying DevOps principles to the world of big data, you get DataOps, which streamlines data analytics.

    If you’re reading this site, you’re probably already familiar with DevOps. The DevOps philosophy emphasizes continuous collaboration between software developers (the “dev” people) and system admins (the operations, or “ops,” folks)—as well as everyone in between, such as software testers.

    It’s possible to apply similar ideas to the world of big data and data analytics. There, as in the world of software development and delivery, different teams traditionally have worked in isolation from one another. Those teams include storage admins, who keep data storage systems running; data analysts, who write and execute programs for interpreting large volumes of data; data quality specialists, who ensure data consistency; and data security teams, which protect data from attackers.

    As DevOps professionals know, keeping different teams isolated from one another is not efficient. It limits collaboration and maximizes confusion when problems occur.

    The data world, therefore, has much to gain by ending this isolation between teams and adopting a DataOps workflow. This will keep data operations flowing smoothly and provide continuous insight into the data that organizations collect.

    DataOps is not a brand new concept. People like Andy Palmer have been writing about it for a little while now. But the term and the idea have yet to gain nearly as wide a following as DevOps. (There is not yet even a Wikipedia entry for DataOps—and you know an idea is still in its infancy when no one has taken the time to write about it on Wikipedia.)

    Making DevOps Bigger

    Meanwhile, from the DevOps perspective, the growth of interest in DataOps is significant because it exemplifies the broad applicability of DevOps principles. So far, the DevOps conversation has focused on the relatively narrow context of software development and delivery. But it can extend much further, too. DataOps is a start.

    So, if you’re a data analyst, storage admin, data security professional or someone who works with data in any other capacity, it will be worth your while to borrow some ideas from your DevOps brethren. And if you work in software development, testing or delivery, it may be worth your time to educate your data friends on DevOps principles. It will make their workflows more efficient, while introducing DevOps to a wider audience.

    — Chris Tozzi

  • Backup and Recovery for Distributed Databases

    Backup and Recovery for Distributed Databases

    Over the past five years, applications have undergone a paradigm shift. They now require more agility, scalability and availability. The key trends in this era are driven by the use of social platforms, mobile devices and software as a service (SaaS).

    Being able to ingest large amounts of data from various sources and process that data in real-time to deliver contextual information or business insights is a major competitive advantage. To meet these new application requirements, organizations are forced to look beyond traditional relational databases. This has resulted in a slew of new database systems that are distributed and scale-out in nature, can be deployed on commodity hardware and offer tunable consistency-versus-performance trade-offs. It’s important to note that to cater to the agility requirements, database-as-a-service model in cloud is being adopted as well.

    To show just how big the shift to distributed databases is, below is a Keyword Search Trend graph showing the sharp rise in NoSQL search.

    screen-shot-2016-09-27-at-2-03-59-pmThe implication of this is that requirements for data protection have changed forever, driven by this data-centric world (web-scale applications and distributed databases). Organizations are collecting vast amounts of data and seek to drive more value from that data to make better and faster business decisions. Most distributed and cloud databases already provide replication capability and thus satisfy the availability requirements of data protection. However, data protection requirements for scalable point-in-time backup and recovery need to be addressed. Without point-in-time backups, organizations are at substantial risk of losing data due to human error, logical corruption and other operational failures. Traditional backup solutions were built to address the requirements of relational databases that used shared storage and had the ACID transaction model. Unfortunately, they fall well short in addressing the point-in-time backup requirements of distributed databases (local storage, eventual consistency and the elastic nature of infrastructure).

    As database architecture has fundamentally changed to meet new application requirements, data protection needs to be redefined and rearchitected as well. The following are the new requirements for data protection.

    • Take a durable point-in-time backup copy of an eventually consistent database; we call it “versioning” the new paradigm of data protection for this new distributed era: backups to snapshots to replication to copy data management to versioning
    • Minimize time to recover from failures (low RTO)
    • Scale with the growing needs of application
    • Enable easy refresh of test/dev environments for continuous development
    • Provide operational resiliency in event of failures
    • Provide deployment flexibility in public cloud or on-premises data center

    Most organizations are investing in enterprise-grade point-in-time backup and recovery products so they may deploy and scale their next-generation application on distributed databases with confidence. In the next five years, it will be exciting to watch companies redefine the data protection technologies to meet the requirements of next-generation applications.

    — Jeannie Liou