devopsy.kinsta.cloud

Tag: data streaming

  • What’s Ahead for the Future of Data Streaming?

    What’s Ahead for the Future of Data Streaming?

    Data is everywhere. It’s being produced by almost everyone and everything around us. For businesses to be competitive, they need to build resilient, scalable systems that can efficiently tap into and react to this ever-increasing volume of data. Companies are meeting this need by moving to real-time data streaming to develop new solutions and improve their businesses and customer experience.

    This category of data streaming has been growing. And, driven by the rise of real-time digital experiences like ride-sharing, fraud detection, inventory management and more, the growth has accelerated.

    So what’s next for data streaming?

    Data as a Product Grows in Popularity

    The concept of data as a product will become more mainstream. Across many industries, data streaming is becoming central to how businesses operate and disseminate information within their companies. However, there is still a need for broader education about key data principles and best practices, like those outlined through data mesh.

    Data as a product is foundational to rethinking how data produced in one part of a business can be reliably used in another. Data products with formalized ownership and responsibilities, combined with a platform for discoverability and accessibility, result in significantly fewer barriers and a much easier data access experience for all. Schemas, the Kafka API and maturing specification standards like ASyncAPI can be a part of this evolution.

    As more folks embrace this mindset, I expect to see companies shift from using ad-hoc data pipelines to ones based on data products, driving a centralized system that can power both better analytical insights and operational execution.

    Growth of Self-Service Solutions

    With the rise of cloud-based offerings, I expect to see more companies offering more accessible, pay-as-you-go, self-service solutions. We’ve already seen this shift over the past few years; cloud-based offerings are making it easier for companies to try out a data product based on their usage (or lack thereof). Given the current market conditions, these offerings will be more appealing to organizations that aren’t ready to sign a multi-year contract for a product offering or focus solely on a single technology.

    Increase in Cloud Migrations and the Continued Rise of SaaS

    Cloud is here to stay. We’ve proven that it works, that it’s safe and secure, and for many folks, it just makes sense. This should come as no surprise to anyone, but expect the trend of cloud migrations to continue.

    In the context of Apache Kafka, building out a Kafka-based data streaming solution used to mean hosting and managing your own Kafka cluster. If you’ve ever been in that position, you know that usually means more DevOps work and less focus on building applications to get the most out of your data. Not only do cloud-based Kafka offerings eliminate much of the DevOps headaches that come with a Kafka cluster, but they also make it much easier and faster to hit the ground running and begin streaming your data.

    I’m personally excited by what the cloud and SaaS can do to make Kafka and data streaming more accessible. In my visits to developers and companies around the globe, I see this enthusiasm shared by so many. Entire geographic regions and industries are taking the opportunity to move to the cloud and dive headfirst into data streaming.

    KRaft Leads to Rise in Kafka Adoption

    As of Apache Kafka 3.3, the KRaft consensus protocol has replaced Zookeeper and is production-ready. Cluster metadata can now be maintained and managed from within the cluster, meaning there’s one less thing to worry about when deploying Kafka instances.

    While there are many options for cloud-based Kafka-as-a-Service, improvements such as KRaft in open source Kafka make self-hosting even easier. This, along with the work done to make it simpler to run managed Kafka services on local machines and test environments, means that Kafka no longer has to be synonymous with complexity. With the difficulty of managing and maintaining a Kafka instance no longer preventing folks from making use of data streaming, I predict that the adoption of Kafka will increase even more.

    Data streaming will continue to mature as a transformational technology for companies of all sizes, serving as a critical infrastructure that most organizations will need to run their business.

    If you haven’t yet explored what data streaming has to offer you, maybe this will be your year to check it out!

  • Dell Pushes Computing to the Edge

    Dell Technologies today set its sights on becoming a major edge computing player with the launch of a Dell EMC PowerEdge XE2420 server optimized for space-constrained environments and a smaller version of a pre-integrated Dell EMC Modular Data Center Micro 415 (MDC Micro) ruggedized platform.

    In addition, Dell launched the Dell EMC Streaming Data Platform, a streaming analytics platform based on open source Kafka software. Dell is using Kafka also to provide an embedded management capability known as the Dell Remote Access Controller (iDRAC 9 Datacenter) for its edge computing platforms.

    Matt Baker, senior vice president for Dell EMC strategy and planning, said edge computing applications require a different approach to IT infrastructure because data is being processed and analyzed in real-time. Dell expects that eventually 75% of all data will be created and processed outside of a data center, he said.

    A primary driver of that transition will be the rise of wireless 5G networks, which will make it feasible to build and deploy real-time applications on edge computing platforms.

    Dell, of course, is making a case for distributed IT environment made up of its platform, contending will be easier for IT organizations to deploy and manage.

    The iDRAC 9 Datacenter platform, for example, provides a remote deployment capability that Dell claims can reduce administrator-attended time by up to 99.1% per server after initial setup. The streaming analytics capability, meanwhile, makes it possible to discover trends, fine-tune operations and create predictive analytics to help ensure peak performance and reduce downtime. Dell has also made available an automated certificate enablement tool to enable zero-touch security at the edge and in central locations.

    All told, Dell claims to be collecting nearly 2.9 million data points per server that can be surfaced in as many as 20 different types of reports.

    Baker said Dell has no plans to extend its data streaming analytics capabilities to other platforms, so IT teams will have to decide to what degree they want to standardize on Dell infrastructure versus setting up a streaming analytics platform of their own that is capable of supporting edge computing platforms from multiple vendors.

    Whatever the approach, the one thing that is certain is the way applications historically have been built and deployed is fundamentally changing. Rather than relying primarily on batch-oriented applications that often require large amounts of data to be transferred across networks to either a data center or public cloud, a lot more data is going to be processed at the edge because of application latency requirements. Edge computing applications, for example, will need to process analytics in near real-time to optimize a process or end user experience in milliseconds.

    Naturally, that shift will have profound implications for the management of IT going forward as IT environments become even more distributed than they are today. In fact, the real issue isn’t so much how computing platforms will be deployed at the edge so much as it is how they will be managed, at a time when most IT organizations are already short-staffed.

    — Mike Vizard