devopsy.kinsta.cloud

Tag: hybrid cloud

  • Best of 2019: Implementing Message Queue in Kubernetes

    Best of 2019: Implementing Message Queue in Kubernetes

    As we close out 2019, we at devopsy.kinsta.cloud wanted to highlight the five most popular articles of the year. Following is the fourth in our weeklong series of the Best of 2019.

    The adoption of Kubernetes as enterprises’ tool of choice is increasing very rapidly. Consequently, the architectural consideration and operational requirements to support production environments have gained increasing attention. A major challenge to consider when planning a scalable microservices architecture (MSA) that is orchestrated by Kubernetes is how to manage the heavy traffic of messages in the system.

    After migrating from the monolith architecture to a microservices architecture or setting up a new MSA, each microservice holds its data model and is disengaged from the rest of the system. When scaling, it is observed that a project can grow to hundreds or more microservices, and the messaging traffic can become enormous in systems that generate millions of messages a day. Hence, there is a need for a new robust communication between the microservices to be established.

    Using Point-To-Point connectivity such as REST to solve the communication gap in Kubernetes can end up creating a complicated structure between the services. A consequence of this is the need for comprehensive maintenance by the development team each time service requirements are changed. This is where a messaging queue system comes in handy to take care of the messaging challenge in MSA and Kubernetes. Each service communicates with one focal point (message broker queue) in its own language and the message queue system is in charge to deliver the messages to the services which are waiting for it.

    To build a well-managed messaging solution for Kubernetes, the message queue system should be native to Kubernetes, support the relevant messaging patterns, robust and secured. Being native to Kubernetes is a critical parameter to achieving a healthy system in Kubernetes. It includes a simple and fast deployment to Kubernetes (and not outside of the Kubernetes cluster), low DevOps maintenance and well connected to Kubernetes ecosystem for logging, monitoring and tracking.

    Hybrid Cloud

    A hybrid cloud solution enables enterprises to make use of both on-premises and public cloud services. An advantage of a hybrid cloud is the flexibility it offers–allowing workloads to alternate between the two environments when capacity and costs change, for example. Sensitive workloads and data can be hosted in the private cloud or on-prem while less critical workloads are hosted in a public cloud. Regulatory requirements for data handling and storage can be provided in the private cloud. A commercial advantage of public cloud services is cost, both in terms of CapEx and OpEx, as the scalability it provides allows an organization to pay only for the resources it uses. Deploying message queue in Kubernetes provides the technology that allows the hybrid clouds to smoothly and transparently connect and interact.

    Use Cases

    There are many use cases for message queues. The message queue needs to support diversified messaging patterns, enabling flexibility in creating different use cases.

    Here are some of the common use cases a messaging queue system should support in Kubernetes.

    Multi-stage Pipeline

    The synchronous pattern is used when we need to process the messages in a coordinated way in a workflow. Pipeline use case allows processing messages in a sequence between services. Each service can be considered as a stage, and the message passed between all the stages in a sequence. In some cases where the message is lost or cannot be processed (for any reason), the dead-letter queue mechanism takes the message and process it in a pre-defined way.

    Job/Task Distributed Queue

    The A-synchronic pattern is used by some producers to many consumers when we need to distribute tasks/jobs between workers. In this use case, there is no need to coordinate between tasks. Each service is pulling messages from the queue to process it. The sequence of processing the messages is not important as each worker takes the task from the queue and process it in his own time.

    Figure 1: Tasks/Jobs Distributed Queue.

    Stream

    The A-synchronic pattern is used when we need to stream data from many external sources (big data, IoT) and process it in a dedicated service such as database, pipeline, machine learning, storage, etc. This case typically aggregates many producers to a small number of consumers. The order of messages is not important, but the delivery guarantee is important.

    Pub/Sub Real-Time

    The A-synchronic pattern is suitable for cases of a small number of producers sending real-time data to many consumers. A service acting as a publisher sends a message to a channel where a set of services subscribe to that channel and get the messages.

    Application Decoupling

    Connectivity between services such as edge, API, databases and storage act like a router, and message order is not important; timeout and sync between producers and consumers are important.

    Ease of Use

    One key feature of the microservices architecture is its ease of use. Deploying it saves your organization time and money by unifying development and operations workflows. The message queue system needs to be easy to use as well. Its ease of use and DevOps’ friendliness should minimize the need for dedicated experts and, at the same time, accelerate development and production cycles. While considering this, it’s important to ensure support in high volume messaging with low latency, efficient memory usage and fundamental messaging patterns (real-time pub/sub, request/reply and queue) are not compromised.

    Gradual Migration to Kubernetes

    When IT professionals evaluate the significant advantages of using the Kubernetes environment, they cannot ignore the operational requirements necessary to maintain the revenue-generating monolith infrastructure. Legacy systems also contain the processes and data related to the on-going business and are crucial for the service contingency.

    To support a gradual migration that keeps the business operation, the Kubernetes message queue must enable connectivity between the old and new environments. A bridge is required to connect the legacy side to IBM MQ, TIBCO, MSMQ or Kafka (just to mention a few popular enterprise solutions), to the Kubernetes message queue that allows a seamless bi-directional transposing messages between the services in the legacy monolith environment and the microservices-based deployment in the Kubernetes environment.

    The bridge installed in the legacy environment is “listening” to the MQ on behalf of the relevant services in the Kubernetes environment, ensuring the designated messages are transferred from the monolith to Kubernetes and vice versa. This bridging capability enables gradual migration by implementing a step-by-step replacement of components from the old environment or the creation of new services that can still connect with the legacy resources.

     

    Figure 2: An example of gradual migration from MSMQ on the on-premises environment to AKS.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

    — Lior Nabat

  • Predictions 2020: Ushering in 2020 Data Predictions

    Predictions 2020: Ushering in 2020 Data Predictions

    2019 brought us more data organizations running more advanced analytics, AI and ML workloads than ever before. 2020 is the year where we’ll see a spike in both the number of technologies and data teams that support these types of workloads internally. We’ll see AI and analytics teams merge into one as the new foundation of the data organization, focused on areas such as moving to the cloud while maintaining on-prem Hadoop, “Kubernetifying” the analytics stack and Hadoop compute. These are the trends I believe we’ll see come about in 2020.

    Rise of the Hybrid Cloud

    We’ve been hearing people talk about the hybrid cloud for the past three years now. For the most part, that’s all it’s been talk—but in 2020 it gets real. We are seeing large enterprises refusing to add capacity on-prem to their Hadoop deployments and instead invest in the public cloud. But they are still not willing to move their core enterprise data to the cloud. Data will stay on-prem and compute will be burst to the cloud, particularly for peak demands and unpredictable workloads. Technologies that provide optimal approaches to achieve this will drive the rise of the hybrid cloud. 

    One Machine Learning Framework to Rule Them All

    Machine learning with models has reached a turning point, with companies of all sizes and at all stages moving towards operationalizing their model training efforts. While there are several popular frameworks for model training, a leading technology hasn’t yet emerged. Just like Apache Spark is considered a leader for data transformation jobs and Presto is emerging as the leading tech for interactive querying, 2020 will be the year we’ll see a front-runner dominate the broader model training space with pyTorch or Tensorflow as leading contenders. 

    Kubernetifying the Analytics Stack

    While containers and Kubernetes works exceptionally well for stateless applications such as web servers and self-contained databases, we haven’t seen a ton of container usage when it comes to advanced analytics and AI. In 2020, we’ll see a shift to AI and analytic workloads becoming more mainstream in Kubernetes land. Kubernetifying the analytics stack will mean solving for data sharing and elasticity by moving data from remote data silos into K8s clusters for tighter data locality. 

    Hadoop Storage (HDFS) is Dead but Hadoop Compute (Spark) Lives Strong

    There is a lot of talk about Hadoop being dead, but the Hadoop ecosystem has rising stars. Compute frameworks such as Spark and Presto extract more value from data and have been adopted into the broader compute ecosystem. HDFS is dead because of its complexity and cost and because compute fundamentally cannot scale elastically if it stays tied to HDFS. For real-time insights, users need immediate and elastic compute capacity that’s available in the cloud. Data in HDFS will move to the most optimal and cost efficient system, be it cloud storage or on-prem object storage. HDFS will die but Hadoop compute will live on and live strong.

    AI and Analytics Teams Will Merge Into One as the New Foundation of the Data Organization

    Yesterday’s Hadoop platform teams are today’s AI/analytics teams. Over time, a multitude of ways to get insights on data have emerged. AI is the next step to structured data analytics. What used to be statistical models has converged with computer science to become AI and ML. So data, analytics and AI teams need to collaborate to derive value from the same data they all use. This will be done by building the right data stack—storage silos and computes, deployed on-prem, in the cloud or in both, will be the norm. In 2020 we’ll see more organizations building dedicated teams around this data stack.

    Talent Gap Will Inhibit Data Technology Adoption

    Building the stacks that enable data technology into practice is hard, and this will only become more obvious in 2020. As companies discuss the importance of data in their organizations, they’ll need to hire the data, AI and cloud engineers to architect it. But there aren’t enough engineers who have expertise in these technologies to do that. This super-power skill is the ability to understand data, structured and unstructured, and pick the right approach to analyze it. Until the knowledge gap closes, we’ll continue to see a shortage of these types of engineers—many companies will come up short on their promises of “data-everywhere.”

    China Is Moving to the Cloud on a Scale Much Larger than the US and Will Leap Frog From On-Prem to Massive Cloud Deployments for Advanced Workloads

    Over the past five years, while enterprises in the U.S. have been moving in leaps and bounds to public clouds, enterprises in China have been investing mostly in on-prem infrastructure primarily for data-driven platform infrastructure. 2020 will be the inflection point where this changes. China will leapfrog into the cloud at a scale much larger than the U.S. by adopting the public cloud for new use cases, bursting in the cloud for peak loads and over time move existing workloads. Public cloud leaders in China will see dramatic growth that might outpace the growth of the current cloud giants.

    2020 is the year where the rubber meets the road when it comes to advanced analytics and AI. Companies that back the types of technologies that enable and support this kind of data and workloads will emerge as leaders in the space. On the other side, companies that structure their data teams to meet the requirements of the new data stack will emerge as leaders as well. I’m excited to see how advanced analytics and AI opens up new and innovative applications and use cases.

    Want to learn more about what to expect in 2020? Join us Jan. 23 for our Predict 2020 Virtual Summit  featuring discussions from some of the industry’s best and brightest offering up their visions for the future. Sign up today for this free daylong virtual event.

    — Haoyuan Li

  • Why Integration Must Be Hybrid

    Why Integration Must Be Hybrid

    Integration is the central piece that ties your enterprise together. The need to integrate has only increased with the proliferation of SaaS applications, IoT sensors and devices, B2B e-commerce supply chains and mobile app development in addition to the vast number of applications that still reside in on-premises data centers which also host mainframe, legacy and custom applications. One of the biggest drivers today is digital transformation—organizations are rapidly trying to become digital enterprises in order to differentiate themselves from the competition and develop new business models to create new revenue channels.

    In the meantime, APIs are becoming the standard way of connecting applications, data and devices, and providing services to partners and end users. The API economy is the term used for creating new products as part of an organization’s transformation journey. APIs are also rapidly becoming a new channel for B2B partner interaction. Since APIs and integration are two sides of the same coin, this allows organizations to quickly realize the value of integration to unlock data from different silos, drive consumption of this data and leverage business benefits for the application of this data in various end user and consumer apps.

    It’s a Hybrid World

    The most common understanding of hybrid is that organizations have applications both on-premises as well as the cloud. Pure cloud integration is a fantasy since most organizations will have applications and systems that have to stay on-premises for a variety of reasons: the cost and risk of redeveloping core capabilities (eg. An ERP application that has been heavily customized for the business or a core banking application that is mission critical), the lack of critical services in the cloud, concerns about security and sunk cost of on-premises applications and data sources (mainframe and legacy applications that just work and will be extremely expensive to rip and replace with a SaaS application).

    However, the definition of hybrid is much broader and according to Gartner, such a strategy must address a set of challenges that go across personas, domains, endpoints and deployment models. Lets explain each of these in more detail:

    • Personas: Integration development today is not just the purview of IT specialists but extends into ad-hoc developers and business users (often called citizen integrators) who despite not being technical cannot wait for IT to develop integrations for them.
    • Domains: Integration has always been a continuum supporting a variety of use cases that go across applications, B2B partners, data (for the purpose of doing analytics) and process (to support automation and orchestration).
    • Endpoints: The sources and targets for integration started originally with on-premises, then expanded to the cloud along with the need for mobile connectivity and more recently has exploded with IoT with the proliferation of sensors and devices.
    • Deployment Models: This covers the more traditional definition of hybrid and includes on-premises, cloud and hybrid (cloud and on-premises) but is now expanding into a new style called embedded, using a headless deployment model which is required for SaaS applications that need underlying integrations to satisfy their end user requirements.

    The Hybrid Integration Platform

    A true hybrid integration architecture makes integration available across all environments, for all the personas, overlapping all the domains and connecting all the endpoints. This will span on-premises ERM, CRM and mainframe, cloud and SaaS environments, edge devices and sensors and B2B partners–we call this integrating edge to app.

    A key architectural principle in hybrid integration is to allow integrations to be done close to where the applications are in order to reduce latency and increase performance. From an application connectivity perspective, hybrid requires applications hosted on-premises, public cloud and private cloud to all be connected seamlessly so an integration developer does not worry about where they are and only needs to easily wire together the mapping to move data from one application to another.

    From a B2B perspective, hybrid requires data from B2B partners (customer and suppliers) to easily be identified, exchanged and processed within the integration layer. From an IoT perspective, hybrid requires that data from sensors and devices either directly or aggregated after analysis is integrated with back-end applications. Finally, all the integrations need to be exposed via an API layer for internal business units and external customers to consume the data in a secure and privileged way through mobile or custom apps.

    These requirements have created the need for a hybrid integration platform (HIP) that supports hybrid deployment across on-premises and cloud, democratizes integration development across IT and business users, is versatile enough to support all integration patterns and is extensive enough to integrate everything.

    Components of a HIP

    A lot of organizations originally solved the integration problem by adopting enterprise service bus (ESBs) and integration platforms. These are still widely in use and in many cases powering mission critical applications for organizations. These integration products solved the application to application (A2A) problem and in conjunction with message oriented middleware (MOM) provided capabilities to decouple publishers and subscribers and allow new applications to be easily added.

    However, ESBs are undergoing an evolution since they are considered a monolith and organizations are seeing the need to deploy light weight integrations as microservices. This has led to the emergence of a microservices runtime (MSR) that can be scaled very quickly to meet the needs of the business. MSRs are typically deployed using Docker containers and scale using a container orchestration standard such as Kubernetes and are subsequently monitored using open source tools, such as Prometheus.

    A big requirement for deploying microservices is the need for DevOps which provides the discipline to manage the dev/build/test/deploy process using automated techniques. Many of these organizations also adopted B2B gateways to extend the reach of integration to their trading partners for the purpose of exchanging e-commerce documents such as purchase orders and invoices via standards, such as EDI and XML.

    A lot of transactions still occur using files leading to the need for a managed file transfer (MFT) offering to securely manage file interactions. These in conjunction with adapters and connectors provide reach to a wide variety of applications and data sources. A newer component is API management, which allows existing integrations to be exposed as APIs for access to a variety of consumers while providing policy based enforcement and security of the underlying services. APIs needs end-to-end lifecycle management and governance to control the creation to retirement process and also drives the need for asset management to track dependencies and relationships and provide impact analysis.

    IoT integration is becoming a common use case. An IoT platform along with edge capabilities will allow data from sensors and devices to be easily integrated and analyzed for the purpose of moving to other systems or to take action. In-memory is a key embedded capability for ensuring high performance and scale.

    Finally, iPaaS is a pure cloud offering which enables integration in the cloud but in conjunction with an on-premises integration platform, or agent, it can ensure data is seamlessly exchanged between cloud and on-premises with end-to-end visibility and monitoring across the different environments. Here is a quick summary of the components:

    • ESB or integration platform (with broad connectivity capabilities)
    • Microservices and DevOps
    • API management and governance
    • B2B and MFT
    • Messaging
    • In-memory
    • IoT platform
    • iPaaS

    A Practical Approach to Hybrid Integration

    A HIP can be extremely powerful but involves a number of components which can also make it overwhelming to tackle all at once. A practical approach is to add components (using a building blocks approach) as the need arises to address various integration use cases.

    Most customers already have an ESB or integration platform, so that is usually the starting point for integrating on-premises applications. Once an organization starts adopting SaaS apps then there is a decision to be made whether to simply do cloud integration using the on-premises ESB or to start adopting an iPaaS. A critical factor in this decision process is to follow the general guideline and allow integrations to be done close to the applications. The decision matrix for such a scenario can be described as follows:

    • If the use case is mostly cloud to on-premises and most of the applications are still on-premises (80/20 rule) then this can be done within the ESB by adding cloud connectivity options.
    • If the use case involves a number of cloud-to-cloud integration needs then it may be worthwhile to invest in an iPaaS for efficiency reasons (there is no need to go from cloud to ground and then back to cloud).
    • If the application mix gets switched to mostly being SaaS apps then an iPaaS should be adopted since most integrations will be done in the cloud anyway.

    Once you bring together an ESB and iPaaS you are in a great position to start leveraging API management and add B2B capabilities to round out most of your integration needs and start deriving value from these integrations.

    Business Benefits

    Hybrid integration provides a way for customers to move ahead with critical projects and strategic transformation by exploring more flexible solutions. Not long ago, integration had been painted as a back-office capability with little direct impact on the business. However, integration needs have moved to the front-office and addressing these requirements are key for successful digital transformation initiatives.

    The business benefits are as follows:

    • Organizations are seeing more demand from customers for a better customer experience. Customers have many choices, the best experience wins the business.
    • Businesses need to react faster to disruptive changes by increasing the delivery cadence and deploy integrations faster.
    • The technology world is experiencing many disruptions. Businesses need flexible architectures to disrupt instead of being disrupted. They need to be able to implement new projects, scale rapidly to take advantage of short-term opportunities and try new fast-fail ideas to leapfrog the competition.
    • Gain benefits of the cloud. There is constant pressure to reduce TCO, use an opex model instead of capex and be able to access and leverage all data available from new IoT and SaaS sources.
    • Democratize integrations. IT departments are at breaking point and can no longer meet the needs of the business which means business analysts need to develop integrations on their own in a self-service way, which allows more people to contribute to the company’s goals.

    Real World Customer Case Studies

    Retail Pharmacy: Our first case study is a large global enterprise with more than half a million employees and hundreds of distribution centers. One of the biggest changes in the company’s history was the merger of two iconic brands. The company needed a global platform strategy to stay competitive and remain ahead of Amazon and other digital disruptors in the retail space, and coined their version of a HIP: Independent Federation Layer.

    Software AG provided a solution to standardize global operations—a “cook once, serve many” solution. Rather than buy more tools, the company would invest in a unified set of software capabilities using webMethods HIP to integrate all systems. They set up a 200-person integration center of excellence. All retail stores are being migrated onto webMethods for a mission-critical retail transformation program to simplify and standardize sales and inventory processing. The next phase will be standardized processes for warehouse and order management, replenishment management, financial management and more. With webMethods HIP, this retail pharmacy has the agility to react to market challenges and opportunities.

    Transport Company: Our second case study is a multinational public transport company that operates bus, train and tram services in Europe, Canada and the U.S. They needed to lift and shift services to the cloud in order to create a new business offering that could scale easily. They refactored their webMethods integration code base and moved key services to the webMethods integration cloud to enable customers to purchase tickets using mobile and web platforms via third-party websites. They provided secure access to on-premises databases by exposing APIs to the cloud using webMethods API gateway. In the process, they created a reusable ticketing platform that they’re offering to other transportation providers. They now have a new line of business.

    — Subhash Ramachandran

  • Going Cloud-Native with IBM Z as a Hybrid Cloud Platform

    Going Cloud-Native with IBM Z as a Hybrid Cloud Platform

    When people use the phrase “they don’t make ‘em like that anymore,” 100-year-old bridges, Jon Snow’s sword and American muscle cars often come to mind. If the topic of discussion happens to be computer technology, however, mainframes are without a doubt the Valyrian steel of the enterprise and lucky for us, are still in production today. While the cloud computing concept as we know it has been around for only about a decade, mainframes have been delivering on-demand access to compute and storage resources that are safe, scalable, highly available and resilient for half a century.

    Big Iron Advantage

    In today’s enterprise scenario, the mission to gain an edge over the competition in terms of pure horsepower can be elusive, to say the least. Although 20% of the easy enterprise workloads have moved to cloud platforms, it is estimated a large amount of data and applications continue to be run on-premise as enterprises determine their hybrid cloud strategy. Far from being irrelevant, mainframe architecture now finds itself at the center of this digital battle, causing disruption and giving its users a much-coveted and undeniable edge over the competition. In addition to running over 8,000 VMs on a single system, mainframes offer some of the most scalable storage in the industry.

    In case that isn’t enough, couple that with pervasive encryption, multi-factor authentication, a co-processor dedicated to cryptography and the most trusted and secure Linux environment in the industry today. What you get in the end is a major breakthrough with regard to cloud limitations that many early adopters of cloud technology found out about the hard way. These limitations are especially true with regard to cybersecurity, and though private clouds help alleviate these risks to an extent, IBM Z removes these risks by enabling all-encompassing encryption.

    Enabling the Hybrid Cloud

    While most people were skeptical of mainframes making a comeback until about a year ago, the majority seem to have changed their tune. There are two reasons for this: One is IBM’s new take on the mainframe, called IBM Z; and the other reason is its applications with regard to the hybrid cloud. Hybrid basically refers to organizations using a broad combination of public, private, IaaS and Paas in the same application solution. Mainframes are proving an important part of hybrid infrastructure thanks to the massive compute, storage and scalability features they can deliver.

    Another advantage mainframes have in a hybrid environment is simplicity. Every engineer knows the more moving parts, the higher the risk of a breakdown, and the same holds true for the hybrid cloud. In a complex, multi-cloud environment that often spans more than one public cloud and a variety of on-prem resources, the more discrete devices involved in a network, the higher the risk of something failing. With a mainframe like the IBM Z, you have one device that delivers broad functionality at high efficiency and low cost. And it does it for a few decades, without complaining.

    IBM Z and the Cloud

    IBM’s Z models are lean, mean, transacting machines. Recent innovations in hardware and software design have made it possible to run not just native mainframe apps, but also most Linux-based operating systems and applications on these systems. IBM Z also features machine learning and AI frameworks, as well as support for Apache SparkML, TensorFlow and languages such as Scala and Python. Capable of running more than 12 billion fully encrypted transactions per day on a single machine and 850 million on IBM’s skinny version, the z14 has a 35% capacity increase for workloads compared to the z13, and it’s being touted as the most powerful transaction machine in existence today.

    Combining mainframes and cloud platforms in today’s hybrid environments may sound like old wine in a new bottle, but it effectively gives you the best of both worlds–the scalability of the cloud, along with the power, security and dependability of the mainframe. Additionally, the mainframe’s ability to house both data and applications on the same device, while dedicating processors to specific operations independently, make it ideal for ensuring speed in such complex and distributed environments. IBM Z also features a multiple processing structure that makes it ideal for dealing with the varying bandwidth needs of modern applications. It can process Java workloads up to 50% faster than x86 servers.

    Cost Efficiency

    Forty-four of the top 50 banks in the world run on IBM Z servers right now. They understand that running their applications on a mainframe provides security and scalability for a compelling return on investment. Large enterprise systems have had a perception of being expensive, but when you factor in such things as space, power and cooling efficiency, as well as the number of personnel required to administer the system, the total cost of ownership is a value-add when leveraging Z in your hybrid cloud strategy.

    — Twain Taylor

  • The Linux Foundation Announces New Open Hardware Technologies and Collaboration

    OpenPOWER to become Linux Foundation project to advance development of data-driven architectures required for hybrid cloud environments and increasingly intensive workloads.

    San Diego, CA – August 21, 2019 – The Linux Foundation, the nonprofit organization enabling mass innovation through open source, today announced the OpenPOWER Foundation will become a project hosted at The Linux Foundation, operating within The Linux Foundation’s open governance principles. The project includes IBM’s open POWER Instruction Set Architecture (ISA) and contributed Source Design Implementations required to support data-driven hardware for intensive workloads like Artificial Intelligence (AI).

    OpenPOWER is the open steward for the Power Architecture and has the support of 350 members, including IBM, Google, Inspur Power Systems, Yadro, Hitachi, Wistron, Mellanox, NVIDIA and Red Hat.

    The governance model within the Linux Foundation gives software developers assurance of compatibility while developing AI and hybrid cloud native applications that take advantage of POWER’s rich feature set and open compute hardware and software ecosystems.

    As the demand rises for more and more compute-intensive workloads like AI and in-memory analytics, commodity systems vendors have struggled to improve performance with the looming predictions of the end of Moore’s Law. Central processing units (CPUs) may no longer handle the rising demands alone, and data-centric systems are built to maximize the flow of data between CPUs and attached devices for specialized workloads. By hosting OpenPOWER at The Linux Foundation, a cross-project, cross-community collaboration, it will accelerate development of hardware and software to support data-centric systems, by making it available to a growing global audience.

    “The OpenPOWER community has been doing critical work to support the increasing demands of enterprises that are using big data for AI and machine learning workloads. The move to bring these efforts together with the worldwide ecosystem of open source developers across projects at The Linux Foundation will unleash a new level of innovation by giving developers everywhere more access to the tools and technologies that will define the next generation of POWER architecture,” said Jim Zemlin, executive director at The Linux Foundation.

    “IBM believes that open collaboration through groups like Linux Foundation is key to delivering value for our clients,” said IBM’s OpenPOWER General Manager Ken King. “With our recent Red Hat acquisition and today’s announcement, IBM becomes the only commercially-available processor vendor and POWER becomes the only architecture that can boast of a completely open system stack, from the foundation of the hardware through the software stack.”

    “The OpenPOWER Foundation was created in 2013 to promote open standards and specifications of systems designed around the IBM POWER architecture,” said Hugh Blemings, executive director, OpenPOWER Foundation. “Since then, hundreds of companies from around the world have contributed to the framework and collaborated with each other to build technology for the benefit of their customers. Collaborating with The Linux Foundation marks the next step in opening up the community further.”

    To find more information on the OpenPOWER Foundation or get involved in the project, please visit The Linux Foundation blog, OpenPOWER blog or IBM blog.

    # # #

    About the Linux Foundation

    The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and industry adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

    — Veronica Haggar

  • IBM Closes $34B Red Hat Acquisition

    IBM Closes $34B Red Hat Acquisition

    Following the official closing of IBM’s $34 billion acquisition of Red Hat today, leaders from both companies reiterated Red Hat will continue to operate independently of IBM but will leverage IBM resources to advance the hybrid cloud computing ambitions of the newly combined entity.

    Arvind Krishna, senior vice president for cloud and cognitive software at IBM, said IBM and Red Hat together will be in a better position to address what he described as Chapter Two of a continuing cloud journey. With 80% of applications still running in on-premises IT environments some 10 years after the arrival of public clouds, it’s apparent hybrid cloud computing will become the dominant form of IT, he said.

    Red Hat will play a critical role in helping customers achieve that goal using a common operating model based on distributions of Linux and Kubernetes container orchestration software, Krishna said. However, he added, IBM sees no need to rationalize its current middleware software portfolio, which often addresses use cases that Red Hat doesn’t.

    Paul Cormier, executive vice president and president for products and technologies for Red Hat, said Red Hat will continue to invest in products and projects independently of IBM. As such, Red Hat does not expect its relationship with its partners that compete directly with IBM to change substantially, he said. At the same time, he stressed that working more closely with IBM will accelerate a 100% open source approach to hybrid cloud computing that otherwise would have taken Red Hat “many years” to execute on its own. That hybrid cloud computing strategy, in contrast to rivals, will span virtual machines, bare-metal servers and public clouds, Cormier said.

    Other than introducing Red Hat to a potentially larger customer base, it’s not quite clear how being a unit of IBM accelerates the Red Hat strategy. Red Hat and IBM sales teams, for example, will have no incentives to cross-sell offerings and Red Hat will continue to promote its brand independently of IBM, in much the same way VMware operates independently of the rest of Dell Technologies. Both Red Hat and IBM also stressed there is no plan to impose an IBM culture on Red Hat or lay off any personnel. There is a consensus that open source software will play a role in enabling hybrid cloud computing that should generate opportunities for IBM to generate more professional services revenue.

    Longer-term, there’s clearly an opportunity for both Red Hat and IBM to promote the adoption of a common set of best DevOps practices across a hybrid cloud computing environment. Many organizations, however, are already moving to extend those best practices across all the platforms that make up very diverse IT environments.

    For now at least, organizations that have already committed to Red Hat appear to be lending their continued support. Red Hat’s fiscal year 2019 revenue was $3.4 billion, up 15% year over year. Fiscal first-quarter 2020 revenue was $934 million, up 15% year over year. Services revenue for Red Hat also grew 17%. Precisely how IBM and Red Hat intend to unlock the value of this merger beyond the momentum Red Hat already enjoys, however, remains a bit of a mystery.

    — Mike Vizard

  • HPE Extends Hybrid Cloud Computing Strategy

    HPE Extends Hybrid Cloud Computing Strategy

    Hewlett-Packard Enterprise (HPE) today advanced its hybrid cloud computing strategy by announcing at its HPE Discover 2019 conference an alliance with Google to embrace Kubernetes in hybrid cloud computing environments and then extending the reach of its software-defined infrastructure for building private clouds to legacy ProLiant servers.

    Lauren Whitehouse, director of marketing for the software-defined and cloud group at HPE, said the HPE Composable Cloud now extends beyond new ProLiant servers to include everything from legacy hardware to cloud computing platforms running virtual machines or as bare metal servers optimized for Kubernetes.

    To achieve that latter goal, HPE will make it possible to deploy Google Anthos, a previously announced framework for delivering cloud-native computing services across public clouds and on-premises IT environments, on HPE-managed platforms.

    HPE is giving customers the option of deploying configuring local servers with virtual machines with either a VMware vSAN storage software or the HPE SimpliVity hyperconverged infrastructure platform. Additionally, customers can choose to deploy Kubernetes on those platforms or run Kubernetes on a bare-metal server. On top of those stacks, HPE also embeds the HPE OneView IT infrastructure management and HPE Composable Fabric networking solution to create what the company calls a composable rack environment.

    Finally, HPE announced it will extend the reach of the HPE InfoSight artificial intelligence engine, which it gained for storage environments when it acquired Nimble Storage, to the rest of its data center portfolio.

    Whitehouse said that, by extending the reach of its composable computing strategy to include legacy servers, HPE seeks to make it easier to modernize on-premises IT environments within the context of a larger hybrid cloud computing strategy.

    One of the primary reasons so many workloads are being deployed in the cloud is because of the inherent flexibility of the IT environment. HPE is investing in higher levels of infrastructure abstractions to enable an equivalent amount of agility to be injected into on-premises IT environments. Those efforts will then make it easier for organizations to extend best DevOps practices across a hybrid cloud computing environment, said Whitehouse. The company is now removing the objection by organizations to have to implement a forklift upgrade across their entire on-premises IT environment to achieve that goal, she added.

    It’s not immediately clear who will manage what in this brave new world of hybrid cloud computing. Application developers have been exercising a lot more influence over platforms, while advances in automation appear to be heralding a consolidation of management functions within the local data center. As site reliability engineers, or so-called “super administrators” capable of managing compute, storage and networking infrastructure continue to emerge, the silos around job functions in the enterprise were built are breaking down. In fact, the more organizations embrace DevOps best practices, the faster consolidation of job functions is occurring.

    In the meantime, whoever provides the infrastructure that remains in an on-premises IT environment had better be able to enable local IT teams to be just as flexible as any cloud service provider an application developer is likely to encounter.

    — Mike Vizard

  • With RHEL 8, Red Hat Adds More Automation to IT Ops

    With RHEL 8, Red Hat Adds More Automation to IT Ops

    At the Red Hat Summit 2019 conference, Red Hat today announced the general availability of Red Hat Enterprise Linux (RHEL) 8, which includes among other things a Smart Management add-on designed to make it easier to manage RHEL 8 across hybrid cloud computing environments.

    In addition, RHEL 8 includes Application Streams, a set of utilities for updating languages, frameworks and developer tools without impacting the core resources, as well as updating RHEL 7 to RHEL 8 in place.

    RHEL 8 also adds Red Hat Enterprise Linux System Roles, which provides modules to automate complex tasks using the Ansible automation framework along with support for the Red Hat container toolkit and the general availability of the Red Hat Universal Base Image, a userspace image derived from RHEL that makes it easier to build Linux containers certified to run on RHEL.

    Red Hat also revealed that Red Hat Insights, a set of predictive analytics tools, will be included with the RHEL 8 subscription and that RHEL 8 supports both the OpenSSL 1.1.1 and TLS 1.3 cryptographic standards.

    Gunnar Hellekson, director for product management for Red Hat, said Red Hat is focusing on automating as many tasks as possible as part of an effort to make the life of IT operations teams simpler. As part of that effort, Hellekson said Red Hat is moving to lower the barrier to becoming a RHEL administrator by abstracting away many of the complexities historically associated with trying to manage Linux environments using a command line interface (CLI). Much of that focus will manifest themselves via ongoing enhancements to the Compose and Cockpit tools Red Hat provides for deploying and managing instances of RHEL, said Hellekson.

    Red Hat is making it clear it wants to make RHEL much more accessible to a larger number of IT administrators, especially those that historically have favored a more graphical approach to managing operating system environments. At the same time, however, Red Hat is infusing more automation into RHEL 8 as part of an effort to make it easier to manage deployments at scale. That latter capability is crucial for DevOps teams that, for example, need to be able to update a development environment without waiting for an IT operations team to reconfigure and reboot an operating system. Via add-ons to the core RHEL platform, Red Hat is now starting to expand those automation capabilities across both on-premises and public cloud computing environments.

    It’s unclear going forward how many instances of RHEL any one IT administrator might be able to manage on their own. Trying to determine that number might not even be worth the effort because advances in automation are now coming so rapidly that trying to determine that right number of IT administrators per environment is not possible to determine consistently. Instead, IT organizations should just assume that, thanks to automation enabled by machine learning algorithms and predictive analytics, the operating system environment is now flexible enough to scale up and down as needed.

    — Mike Vizard

  • VMware Advances Hybrid Cloud Ambitions

    VMware Advances Hybrid Cloud Ambitions

    VMware today advanced its hybrid cloud strategy by making a series of updates to existing offerings, including the CloudHealth by VMware software-as-a-service (SaaS) application for managing and optimizing cloud applications, which VMware acquired last summer.

    In addition, VMware Cloud on AWS is now available in data centers operated by Amazon Web Services in Canada, Paris and Singapore, the company noted. VMware also announced that the latest version of VMware Cloud Foundation—which provides and automatically updates all the VMware compute, storage and networking software required to build a cloud—has been optimized to deploy VMware Horizon virtual desktop infrastructure (VDI) software, and will available on Dell EMC VxRail hyperconverged infrastructure in April.

    New capabilities being added to CloudHealth by VMware include enhanced multi-cloud reporting, multidimensional reporting and workspaces reporting.

    The CloudHealth platform also is now integrated with the Wavefront by VMware cloud analytics and monitoring platform, enabling users of CloudHealth to employ Wavefront metrics to rightsize and optimize applications running on multiple cloud applications as part of their DevOps practices.

    A future update to the CloudHealth platform will add enhanced amortization as well as support for convertible Reserved Instance (RI) exchanger automation, container cluster visibility and cross-family rightsizing for Amazon EC2, according to the company.

    Finally, VMware announced it is making available version 9.7 of VMware vCloud Director to cloud service providers. This offering provides a single pane of glass through which multiple VMware environments can be managed regardless of whether they are deployed on-premises or in the cloud. The latest version of VMware vCloud Director adds support for version 3.0 of VMware Cloud Availability, which unifies onboarding, migration and disaster recovery services between multi-tenant clouds. VMware vCloud Director is only available to cloud service providers, but Ajay Patel, senior vice president and general manager for the cloud provider software business unit for VMware, said VMware and its cloud service provider partners plan to make available a software-defined data center (SDDC)-as-a-service offering through which they will manage deployments of VMware from the edge to the cloud on behalf of IT organizations.

    VMware also revealed there are now more than 60 partners building or delivering managed service offerings for VMware Cloud on AWS, and more than 280 partners have achieved the VMware Cloud on AWS Solution Competency. VMware also now has 35 VMware Cloud Provider Program partners that have achieved Cloud Verified status, which is required to be eligible to deliver the forthcoming SDDC-as-a-service offering. Patel also revealed the number of organizations running instances of VMware on public clouds now exceeds 1,000, in the wake of the launch of VMware Cloud on AWS last year.

    Patel said the shift to hybrid clouds is more about transforming IT operational models than it is about deploying workloads at new physical locations. The challenge many IT organizations will need to come to terms with going forward is deciding what portions of that IT operational model they want to run themselves versus opting to rely on VMware and its partners.

    — Mike Vizard

  • DevOps and Security: The Path to DevSecOps

    DevOps and Security: The Path to DevSecOps

    A secure DevOps is highly beneficial for organizations. However, a secure DevOps environment shouldn’t be the end goal. When it comes to DevOps security, it’s important to integrate security right from scratch, secure the entire architecture, automate the security and use this technology to test the environment and the codes and respond to concerns immediately. This can help organizations safeguard their creations, protect the confidentiality of their customers and strengthen their applications, all before data breaches can occur.

    DevSecOps is a philosophy that focuses on introducing a culture of security in the DevOps environment. It is a collaboration of the DevOps teams with cybersecurity and system security personnel, where the focus is on finding faster and more efficient ways to safely deliver codes in an agile architecture. DevSecOps strives to bridge the gaps between IT and security while responding to bottlenecks in the existing environment.

    Some of the biggest benefits that organizations will get when they move from traditional DevOps philosophy to the DevSecOps philosophy is:

    • Improved operational effectiveness and efficiency.
    • Stronger and healthier collaboration between teams across the company.
    • Greater agility for security teams.
    • More conducive environment for automated builds and quality assurance testing.
    • Easier to identify system and application vulnerabilities.
    • Greater freedom for personnel to focus on high-value projects.
    • A greater degree of transparency into the environment.
    • Improved scalability in the cloud.
    • Increased ROI.

    Key Elements of the DevSecOps Environment

    Microservice-Based Infrastructure

    Single-function modules that contain well-defined interfaces and operations is essential for the success of focused DevSecOps. Constantly monitoring, upgrading and tweaking the microservice-based infrastructure will help organizations to be equipped for new developments.

    Hybrid cloud environments, software-defined networking and network micro-segmentation must be integrated into the infrastructure, and these must be used to better define parameters, assess and identify connections, verify access and monitor the organization’s online assets.

    Continuous Feedback Loop

    The next most important element of the DevSecOps environment is feedback. Setting up a continuous feedback loop will help developers and machines get a comprehensive insight into system/platform vulnerability to security threats. This type of real-time, continuous feedback can help organizations set in place the right policies and rule sets, which can keep the application security testing tools updated and relevant regarding the security status of the organization’s software/network/platform, in addition to keeping all parties updated about the potential threats to the DevOps environment.

    This type of continuous feedback loop acts as an enabler, as opposed to an inhibitor of business, by allowing organizations to stay well-equipped and constantly on guard.

    Focused Automation

    Finally, continuous and focused automation is essential to the success of the DevSecOps environment. Automation, when woven into the software development life cycle right from the start, can reduce the friction that may occur between development and security teams over software/platform security by quickly addressing existing and potential concerns at the lowest cost.

    There are certain open source tools on the market that can help organizations automate their security:

    • Continuum Security: This tool works on a BDD-Security framework, which is compatible with unit testing frameworks, issue trackers, SAST and DAST. It offers an open IriusRisk API for anything the tool doesn’t natively support.
    • WhiteSource: This particular tool specializes in addressing open source vulnerabilities, alerting users about the existence of threats throughout the pipeline. It is compatible with 200 programming languages and it actively analyzes the license, quality and security of all open source codes.
    • ThreatModeler: This platform uses the functional information that developers input to analyze the software and provides information about potential threats. It also offers actionable inputs and security test cases to enable easy implementation of security.
    • Evident.io: This solution is designed for the deployment stage and helps assess and manage cloud security risk, particularly on Azure and AWS. Evident Security Platform (ESP) offers continuous monitoring, early threat identification and threat mitigation.
    • Aqua Security: This tool is ideal to manage end-to-end DevSecOps pipeline security. It has very tight runtime security processes and controls in place. It offers full control over the containerized environment and prevents any intrusion or vulnerability across the pipeline.

    There are many other tools that organizations can use to automate their DevSecOps environment. These include: Dome9 (SaaS) Security, Contrast Security RASP & IAST tool, IMMUNIO RASP tool and Checkmarx SAST tool.

    Shifting to the Left

    In today’s agile world of development, following outdated waterfall philosophies is a death sentence for companies.

    For successful implementation of DevSecOps, organizations should shift left. In other words, they should integrate deployments and testing right from the start, and these must continue till the very end when the software/platform in question is vulnerability-free. Etsy and Amazon, for example, conduct more than 1,000 deployments per day, each aimed at recognizing potential threats to their systems. Such a highly dynamic DevSecOps environment ensures that the velocity of new development isn’t slowed, and the quality of new developments is maintained at a high level.

    This type of “shit left” philosophy not only accelerates new developments but also helps limit the security threats and addresses existing threats at the least cost with minimal damage to the software/platform.

    Incorporating AI and Machine Learning into DevSecOps

    Research shows that more than 50 percent of all companies will implement machine learning in their DevSecOps operations in 2019, due in part to the various benefits AI and ML offer to organizations that are actively maintaining a DevSecOps environment. These benefits include:

    • AI and ML will bring down DevSecOps security review time. They will do this by increasing the speed and quality of false positive identification, which will reduce the time spent on threat vector identification. This will help developers improve the speed at which they recognize the threats to their systems.
    • AI and ML will make up for the cybersecurity talent shortage, thereby ensuring that an organization’s DevSecOps environment isn’t left unmanned at any time. AI and ML will dip into human intelligence and use this to fuel their involvement in the DevSecOps environment. Additionally, AI and ML give organizations the ability to understand how other machines function, thereby helping cybersecurity personnel understand the psyche of cyberattackers.
    • AI and ML will actively find defects in the DevSecOps, help developers identify what they’re doing wrong and what they need to do to rectify these concerns. Constant monitoring is a core feature of DevSecOps, and AI and ML will serve to add more advanced technological backup to power this feature. AI and ML will also offer developers customized secure code patterns based on the organization’s unique DevSecOps landscape, which will assist in vulnerability detection.
    • AI and ML can help organizations move their DevSecOps from a small scale endeavor to large-scale. This will be a possibility because of the sheer technical prowess and the real-time information that AI and ML will furnish to developers and cybersecurity personnel.

    7 Rules to Successfully Implementing a DevSecOps Environment

    • Integrate security from the start and across the pipeline.
    • Automate your security.
    • Track and monitor each software stack meticulously to identify which needs patching.
    • Implement code dependency checks such as the OWASP Dependency Check, vulnerability assessment and discovery tests regularly.
    • Put in place robust policies to manage the DevSecOps environment.
    • Break your tasks into manageable chunks—this will improve the consistency of deployments.
    • Set in place one-click compliance reporting to increase traceability and transparency of the pipeline, right from code planning to code changes to release.

    Conclusion

    DevSecOps will bring massive economic and technical advantages to organizations, apart from equipping them with the capabilities to create, run and offer state-of-the-art software/applications. Organizations wanting to remain relevant and competitive in the industry must consider DevOps security as their primary goal in 2019.

    — Subramonian