devopsy.kinsta.cloud

Author: Haoyuan Li

  • Predictions 2020: Ushering in 2020 Data Predictions

    Predictions 2020: Ushering in 2020 Data Predictions

    2019 brought us more data organizations running more advanced analytics, AI and ML workloads than ever before. 2020 is the year where we’ll see a spike in both the number of technologies and data teams that support these types of workloads internally. We’ll see AI and analytics teams merge into one as the new foundation of the data organization, focused on areas such as moving to the cloud while maintaining on-prem Hadoop, “Kubernetifying” the analytics stack and Hadoop compute. These are the trends I believe we’ll see come about in 2020.

    Rise of the Hybrid Cloud

    We’ve been hearing people talk about the hybrid cloud for the past three years now. For the most part, that’s all it’s been talk—but in 2020 it gets real. We are seeing large enterprises refusing to add capacity on-prem to their Hadoop deployments and instead invest in the public cloud. But they are still not willing to move their core enterprise data to the cloud. Data will stay on-prem and compute will be burst to the cloud, particularly for peak demands and unpredictable workloads. Technologies that provide optimal approaches to achieve this will drive the rise of the hybrid cloud. 

    One Machine Learning Framework to Rule Them All

    Machine learning with models has reached a turning point, with companies of all sizes and at all stages moving towards operationalizing their model training efforts. While there are several popular frameworks for model training, a leading technology hasn’t yet emerged. Just like Apache Spark is considered a leader for data transformation jobs and Presto is emerging as the leading tech for interactive querying, 2020 will be the year we’ll see a front-runner dominate the broader model training space with pyTorch or Tensorflow as leading contenders. 

    Kubernetifying the Analytics Stack

    While containers and Kubernetes works exceptionally well for stateless applications such as web servers and self-contained databases, we haven’t seen a ton of container usage when it comes to advanced analytics and AI. In 2020, we’ll see a shift to AI and analytic workloads becoming more mainstream in Kubernetes land. Kubernetifying the analytics stack will mean solving for data sharing and elasticity by moving data from remote data silos into K8s clusters for tighter data locality. 

    Hadoop Storage (HDFS) is Dead but Hadoop Compute (Spark) Lives Strong

    There is a lot of talk about Hadoop being dead, but the Hadoop ecosystem has rising stars. Compute frameworks such as Spark and Presto extract more value from data and have been adopted into the broader compute ecosystem. HDFS is dead because of its complexity and cost and because compute fundamentally cannot scale elastically if it stays tied to HDFS. For real-time insights, users need immediate and elastic compute capacity that’s available in the cloud. Data in HDFS will move to the most optimal and cost efficient system, be it cloud storage or on-prem object storage. HDFS will die but Hadoop compute will live on and live strong.

    AI and Analytics Teams Will Merge Into One as the New Foundation of the Data Organization

    Yesterday’s Hadoop platform teams are today’s AI/analytics teams. Over time, a multitude of ways to get insights on data have emerged. AI is the next step to structured data analytics. What used to be statistical models has converged with computer science to become AI and ML. So data, analytics and AI teams need to collaborate to derive value from the same data they all use. This will be done by building the right data stack—storage silos and computes, deployed on-prem, in the cloud or in both, will be the norm. In 2020 we’ll see more organizations building dedicated teams around this data stack.

    Talent Gap Will Inhibit Data Technology Adoption

    Building the stacks that enable data technology into practice is hard, and this will only become more obvious in 2020. As companies discuss the importance of data in their organizations, they’ll need to hire the data, AI and cloud engineers to architect it. But there aren’t enough engineers who have expertise in these technologies to do that. This super-power skill is the ability to understand data, structured and unstructured, and pick the right approach to analyze it. Until the knowledge gap closes, we’ll continue to see a shortage of these types of engineers—many companies will come up short on their promises of “data-everywhere.”

    China Is Moving to the Cloud on a Scale Much Larger than the US and Will Leap Frog From On-Prem to Massive Cloud Deployments for Advanced Workloads

    Over the past five years, while enterprises in the U.S. have been moving in leaps and bounds to public clouds, enterprises in China have been investing mostly in on-prem infrastructure primarily for data-driven platform infrastructure. 2020 will be the inflection point where this changes. China will leapfrog into the cloud at a scale much larger than the U.S. by adopting the public cloud for new use cases, bursting in the cloud for peak loads and over time move existing workloads. Public cloud leaders in China will see dramatic growth that might outpace the growth of the current cloud giants.

    2020 is the year where the rubber meets the road when it comes to advanced analytics and AI. Companies that back the types of technologies that enable and support this kind of data and workloads will emerge as leaders in the space. On the other side, companies that structure their data teams to meet the requirements of the new data stack will emerge as leaders as well. I’m excited to see how advanced analytics and AI opens up new and innovative applications and use cases.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Haoyuan Li

  • Building Hybrid and Multi-Cloud Architectures for Analytics and AI

    Building Hybrid and Multi-Cloud Architectures for Analytics and AI

    We are in the early stage of the data revolution. Today’s conventional wisdom states that hybrid latency prevents you from running analytic and machine learning workloads in the cloud with the data on-prem. As a result, most companies either do not burst their workloads into the cloud or have to copy their data into a cloud environment and maintain that duplicate data—and even this can take days or weeks.

    Counter to conventional wisdom, you can create a high performance hybrid/multi-cloud data platform for analytics and machine learning by using data orchestration technology. Container orchestration technology (e.g. Kubernetes) enables containers to run in any environment agnostic to the hardware that is running the application, similarly data orchestration technology enables active working sets and data to be orchestrated on demand. This enables enterprises to create a compute agnostic, storage agnostic and cloud agnostic stack to serve all their data driven workloads, including analytics (e.g. Apache Spark, Presto, Apache Hive) and AI (e.g. Tensorflow, PyTorch).

    An enterprise hybrid cloud analytics strategy should burst compute that spans on-prem and cloud data stores. By bringing the hot data to the analytics and machine learning applications, the performance is the same as having the data co-located in the cloud. Also, the on-prem data stores will have offloaded the computation and minimized the additional I/O overhead.

    Specifically, a data orchestration system should:

    • Abstract data access across storage systems. Today, data is siloed in various storage systems and deployments. For example, lots of on-premise data is stored in systems such as Apache HDFS, Ceph, EMC Isilon and IBM Cleversafe, and data in the cloud is stored in systems such as AWS S3, GCP Store, Alibaba OSS, etc. Historically, there have been many attempts to unify all data into a single storage deployment with limited success. As a result, we’ve seen more of a trend around embracing these data silos by abstracting the data in various storage systems/deployments. To do this, the data orchestration system needs to abstract and virtualize across various storage deployments/deployments by server side API translation, which enables users to run existing applications without modifying any code. The system also needs to provide a single global namespace to enable apps and users to access all the data in their environments without the need to understand which storage the data is stored in.
    • Bring data locality in a separated compute and storage environment. While cloud vendors and leading companies in data-heavy industries have adopted a decoupled compute and storage architecture, it can still be very challenging to implement this type of architecture, especially when it comes to bursting on-premises data into the public cloud. A data orchestration system resolves this challenge by providing caching functionality that enables the system to automatically orchestrate the hot data to be closer to compute while at the same time keeping cold data in a cheap storage. Because each workload may have a different usage pattern, the system must have a policy that enables customized policies best suited for those specific workloads. Moving forward, it is also very critical to have policies across various clouds to enable a multi-cloud environment.
    • Provide security for data in motion and at rest with secure access methods. A data orchestration system plays a critical role in the success of an organization’s data strategy. While open source software is important and can help in avoiding potential vendor lock-in, it is also critical to provide necessary security features in this environment given the system touches a lot of data in the organization.

    Besides empowering data driven application developers, a data orchestration system also brings value to data and infrastructure engineers. It de-risks vendor lock-in by providing organizations with flexibility at the infrastructure level. Transitioning to different storage systems (including cloud storage), adopting another application framework, or even having a hybrid or multi-cloud environment are all possible without the need for many different technologies or incurring large development costs.

    — Haoyuan Li