devopsy.kinsta.cloud

Author: Mike Vizard

BMC Software Advances Multi-Cloud Agenda

Long before anyone ever coined the term DevOps there were terms such as IT service management (ITSM) and IT infrastructure library (ITIL) that at one time epitomized the state of the art in help desk. Now, as IT organizations enter 2018, much of the focus in the new year will be integrating DevOps processes with legacy approaches to delivering IT as a service that still permeate the enterprise.

To achieve that goal, BMC Software has integrated BMC Remedy service desk software with the Jira project tracking software developed by Atlassian. In addition, the latest release of BMC Remedy provides support for Amazon Web Services (AWS) and Microsoft Azure public clouds.

Nayaki Nayyar, president of digital service management at BMC Software, says the rise of both DevOps and cloud computing has clearly changed the way organizations need to manage IT. At the same time, however, traditional service desks still play a critical role in managing IT. Most enterprise IT organizations are looking to extend their investments in service desks by providing tighter integration with DevOps tools in addition to managing applications and IT infrastructure running on-premises and in a public cloud, she says.

Nayyar also notes that modern IT management tools need to be accessible to a much broader constituency. Not only do IT operations teams want to be able to manage distributed IT resources, but also developers, business leaders and others need the ability to drill down into IT processes that affect specific applications.

While there is widespread recognition of that requirement, Nayyar notes progress has been at best uneven. A recent survey of IT professionals published by BMC Software finds 80 percent of the respondents said new management approaches and tools are required for effective multi-cloud management. At the same time, however, 40 percent of the respondents readily admit their organization does not have a firm grasp on how much they are spending on cloud services.

Nayyar says it’s clear IT organizations will need to rely more on various types of cognitive and artificial intelligence technologies to manage IT at scale. To provide those capabilities, BMC Software previously outlined a strategy for infusing cognitive and AI technologies across its entire portfolio. The latest release of BMC Remedy adds support for virtual agents and chatbots, in addition the ability to automatically classify and route IT events and incidents.

Less clear is how long it will take for traditional IT organizations to modernize their environments. Large web-scale companies have already taken advantage of modern DevOps processes to transform the way IT is managed. More traditional enterprise IT organizations are still evolving, clearly borrowing many of the concepts pioneered by DevOps practitioners. But most of them don’t have the software engineering skills required to build, deploy and maintain a modern DevOps environment built from scratch. Many of them are looking to providers of packaged IT management tools to provide them with a modern DevOps experience capable of spanning both modern and legacy application environments.

Most IT organizations will employ a raft of tools to accomplish that goal. But over time, it’s also clear most of them will eventually look to consolidate many of those tools to reduce as much of the friction inside their IT environments as possible.

— Mike Vizard

January 9, 2018
HPE Rises to Multi-Cloud Management Challenge

Hewlett-Packard Enterprise (HPE) is making available a suite of tools to unify the management of public and private clouds via a software-as-service (SaaS) applications.

Bryan Jacquot, HPE worldwide user experience architect, says HPE OneSphere provides IT organizations with a means to manage a multi-cloud IT environment using a SaaS application they can consume in the same on-demand manner they employ to deploy cloud applications on-premises or on a public cloud service.

HPE OneSphere makes uses of application programming interfaces (APIs) to unify the management of diverse IT environments under a common software-defined framework.

Within that management application, HPE has also provided Project workspaces for different types of individuals and teams, including IT operations, business users, developers and data scientists.

Jacquot says HPE makes use of Docker and Kubernetes container technologies to provide the foundation for HPE OneSphere. The goal is to make it easier for IT organizations to manage any combination of hybrid cloud computing environment they choose to craft.

To one degree or another, most IT environments today are multi-cloud. They typically employ at least one public cloud alongside an on-premises IT environment that is becoming more cloud-like. At the same time, many IT organizations either already have or are planning to add an additional public cloud service, if for no other reason than to play Amazon Web Services (AWS) against Microsoft Azure. However, each stack of public and private cloud computing requires them to master a separate management plane, which drives up the total cost of ownership (TCO) of an IT environment—which creates a challenge at a time when many IT organizations are looking for both lower costs and increased agility. Through OneSphere, however, HPE is resolving that issue by providing access to one common management portal for both on-premises IT environment and public cloud services, says Jacquot.

There are other means to develop a common management plane spanning multiple clouds. But building, deploying and maintaining a management framework spanning multiple clouds can be a complicated endeavor. HPE OneSphere presents IT organizations with a portal through which all the underlying APIs are abstracted away from the IT operations team, making the process simpler, says Jacquot.

It’s unclear at what rate IT organizations will move to unify the management of multiple clouds in 2018. Different parts of an organization often fund adoption of different classes of public cloud services. At the some point internal IT teams are tasked with managing those services. But the constitutional crises that drive those events might not occur until 2019 or beyond within many organizations.

Regardless of the approach taken to managing multiple clouds, there’s no doubt that the distribution of application workloads across multiple internal and external platforms represents arguably the biggest DevOps challenge of 2018. Organizations that get in front of this issue today are likely to find managing IT to be much easier in the years ahead. Those that don’t are likely to experience more pain than they might have ever bargained.

— Mike Vizard

January 8, 2018
LinkedIn Open-Sources Fossor Log Mining Tool

Over the years there has been a steady flow of technologies developed by web-scale companies that have benefited the larger IT community by becoming open source projects. The latest two examples of this trend are Fossor, a tool for investigating broken hosts and services, and Ascii Etch, a Python library designed to complement Fossor that takes streams of numbers and turns them into visual graphs using ascii characters. Both were developed by LinkedIn.

Steven Callister, senior site reliability engineer for LinkedIn, said Fossor, which is Latin for grave digger, was developed to provide a mechanism to mine logs to more easily discover unknown issues such a memory fragmentation. Typically there are 10 commands that DevOps teams employ to investigate whether certain issues might be occurring. Fossor eliminates the repetitive tasks associated with issuing the commands manually by enabling them to be automated then executed in parallel. This approach allows DevOps teams to continuously execute tests to discover any potential unknown issues that might arise at any time, he said.

Callister noted LinkedIn decided to open-source Fossor to give back to the IT community in a way that is extensible. Fossor has two basic components: a core engine and a mechanism for creating plugins that DevOps teams can use to add tests specific to their site architecture. Linkedin is hoping that an entire library of plugins will emerge around a Fossor open source community.

Of course, no organization is obligated to contribute the plugin they create to the open source community. There may be information in that plugin an organization considers too proprietary to share. But Callister said there are plenty of use cases where a plugin developed for one type of site might apply to another.

One of the single biggest issues most IT organizations struggle with in managing IT at scale is a lack of engineers on staff. The average IT organization tends to rely on administrators that don’t have much in the way of coding skills. Organizations that manage IT environments that operate at web-scale typically have hired software engineers capable of developing their own tools. Open source projects such as Fossor provide a means through which software engineers working at web-scale companies can share their skills and expertise with the rest of the IT community.

It’s not clear to what degree open source tools will replace commercial software tools. It’s not unusual for an IT vendor to leverage an open source project to create a commercial offering around which they provide extensions and professional support. But many DevOps teams are becoming more self-reliant out of necessity, either technical or budgetary. The challenge now becomes sorting through all the open source code available to determine which of the many tools available today solves a problem well enough to not only rely on every day, but also to make ongoing contributions.

— Mike Vizard

January 4, 2018
HashiCorp Terraform Enterprise Empowers IT Administrators

Most of what passes for management of IT infrastructure often amounts to little more than a Word document describing how a system is configured. HashiCorp developed Terraform to enable IT organizations to programmatically manage IT infrastructure. Now the company is taking that concept a step further with the release of Terraform Enterprise, a collaboration application that can be employed to programmatically manage IT infrastructure without requiring IT administrators to know how to program.

HashiCorp co-CTO Armon Dadgar says Terraform provides an application that makes its Sentinel policy management framework more accessible. Previously, use of Terraform and Sentinel were limited largely to developers looking for a common way to manage infrastructure as code. But Dadgar says customer requests made it clear there was also demand for a collaboration application that did not require any programming skills to employ.

Terraform Enterprise includes Workspaces, a set of modules that that can be delegated to separate teams. Dadgar says Workspaces are roughly equivalent to Git repositories that have access controls for each IT team, using a secure access markup language (SAML) that comes with built-in encryption.

In addition, Terraform Enterprise exposes an application programming interface (API) that can be invoked for integration with existing tooling or application delivery pipelines. Dadgar says with the rise of multiple clouds in an organization, it’s now only a matter of time before existing approaches to managing IT infrastructure simple become obsolete. Pressure from developers to increase the amount of time it takes to provision and update IT infrastructure is only going to increase, says Dadgar.

HashiCorp is effectively trying to smooth over one of the more contentious aspects of DevOps. In theory, developers want more control over IT infrastructure largely because they often view internal IT teams as being too slow. To enable IT organizations to be more responsive, various APIs have been exposed that can be invoked using IT automation framework. Many of those frameworks, however, require programming skills that most IT administrators don’t possess. Many organizations also don’t want developers to be managing IT infrastructure when they could be devoting that time to writing and test applications. Dadgar says Terraform Enterprise provides a graphical user interface (GUI) through which all the benefits of managing infrastructure as code can be attained without having to teach administrators how to program, or allowing programmers to invoke IT infrastructure resources without any adult supervision.

HashiCorp is not the only provider of IT management frameworks trying to solve this issue. There are rival declarative frameworks for automating IT and many providers of the latest generation of software-defined infrastructure are embedding modern IT management frameworks within pre-integrated systems. But HashiCorp is one of the few vendors providing organizations with a dual approach using the same core underlying set of APIs. Rather than continuing to be at odds with one another, developers and IT administrators now could find it easier to collaborate using whatever approach to infrastructure as a code they best see fit.

— Mike Vizard

December 21, 2017
ZeroStack Advances Self-Service Provisioning on Private Cloud

Many IT organizations have aspired to build a private cloud only to find that they don’t have the granular control over IT infrastructure to deliver on that promise. To solve that problem, ZeroStack unveiled DevOps Workbench, a framework internal IT organizations can employ to present developers with a set of options that can self-select to automate the deployment of an application in an on-premises environment.

DevOps Workbench is designed to layer on top of the ZeroStack Intelligent Cloud Platform, which combines 40 developer tools and templates on top of a layer of abstraction based on a Kernel-based virtual machine (KVM) to unify the management of underlying IT infrastructure via a cloud management service. Steve Garrison, vice president of marketing for ZeroStack, says DevOps Workbench adds a self-service framework on top of that private cloud to present developers with an experience similar to the way they provision resources on a public cloud. The company also announced it has added DeployHub, continuous deployment software from OpenMake, to the portfolio of DevOps tools it makes available.

Garrison says developers have been doing an end run around internal IT organizations for years now because it takes too long for internal IT teams to make infrastructure resources available. ZeroStack aims to solve that problem by first applying an abstraction layer to create the private cloud, which is then managed via a suite of software-as-a-service (SaaS) applications that ZeroStack exposes in the cloud.

Rather than trying to force collaboration between developers and internal IT teams, Garrison says DevOps Workbench allows both groups to accomplish what they need without getting in each other’s way. Developers can access a user interface to provision IT infrastructure within a set of use cases predefined by the internal IT team.

Because of performance issues and compliance concerns, there are whole hosts of applications that will not be deployed in a public cloud. But the IT teams who manage those applications are still under pressure to provide developers with a cloud-like experience. In fact, Garrison notes that in 2017 research by Gartner found that 76 percent of respondents indicated that they are using DevOps in regulated situations, an increase from 47 percent in 2015.

ZeroStack rivals also are advocating transitions to software-defined data centers (SDDC). But most of those efforts involve expensive upgrades to both software running on virtual machines and the underlying infrastructure. Garrision says a private cloud approach that relies on cloud management services provides a faster, less costly, means to achieving the same goal, especially in midmarket IT organizations that don’t have access to much internal expertise to begin with. He adds that ZeroStack is committed to providing support for additional hypervisors soon.

Most IT organizations are still a long way from being able to build, deploy and manage a true private cloud at scale. Most of them have what might be more aptly described as large virtual machine farms. The operating costs associated with managing virtual machine farms will eventually push most IT organizations to some form of an SDDC platform. It’s just not clear yet what the preferred path for making that journey will be.

— Mike Vizard

December 18, 2017
VMware Adds vRealize Automation to Cloud Platform

At the core of the VMware approach to cloud computing is a VMware Cloud Foundation platform that combines storage and network virtualization software with the company’s core hypervisor platform. Now via the release of VMware Cloud Foundation 2.3, integration with the VMware vRealize portfolio of IT automation software is being added.

Matt Herreras, director of product marketing for VMware, says the inclusion of VMware vRealize Automation, vRealize Operations and vRealize Log Insight will make it easier to deploy application workloads on a platform that VMware manages on behalf of customers, whether it’s deployed on-premises or in a public cloud.

IT organizations will also be able to use the management console VMware provides with VMware Cloud Foundation to configure the vRealize automation framework.

At the same time, VMware is adding support for heterogeneous servers within a single rack, including be able to specify specific servers for each workload domain.

Additionally, VMware has revealed it is working with Intel to include the latest family of Intel Xeon Scalable processors in an existing Intel Select Solutions for VMware Cloud Foundation program. Those processors are expected to be certified in the first half of 2018.

VMware Cloud Foundation is rapidly becoming a centerpiece of the VMware approach to DevOps. VMware claims that a VMware-based cloud can be spun up six to eight times faster using VMware Cloud Foundation, while total cost of ownership of a private cloud will drop 30 percent to 40 percent and administrator productivity will increase by a factor of two.

IT organizations that have standardized on VMware are under significant pressure from developers to become more agile. In many cases, those developers end-run the internal IT organization by programmatically invoking cloud services, even if it would be more cost-effective for the application workloads to deploy locally. The challenge those administrators and VMware both face is convincing developers that VMware environments are flexible enough to meet their need to access IT infrastructure on demand, now that an IT automation framework has been tightly coupled with VMware Cloud Foundation.

In the meantime, VMware is also betting that as large numbers of existing applications are lifted and shifted into the cloud, most IT organizations would prefer to not have to refactor them. To allow IT organizations to avoid that issue, VMware is making VMware Cloud Foundation available jointly with Amazon Web Services (AWS), while partners including IBM provide their own managed instance of VMware Cloud Foundation. Herreras notes that the VMware vRealize automation suite also includes tools for managing AWS and Microsoft Azure environments as an extension of a local VMware environment.

In addition, VMware has invested in container technologies including Docker containers and Kubernetes clusters, which can be deployed on VMware. Herreras says that from an IT operations perspective, the company is betting that most IT organizations will prefer to deploy legacy monolithic applications and emerging microservices applications on top of a common base infrastructure to minimize costs.

It might be a little while longer before VMware can completely pull all the elements of its DevOps strategy together. But the one thing that is clear is that, at the very least now, the company understands just how much of its continued future relevance in enterprise IT is at stake.

— Mike Vizard

December 13, 2017
Morpheus Data Advances DevOps Across Hybrid Clouds

Thanks to the rise of multi-cloud computing, the challenges associated with managing DevOps are becoming exponentially greater. Most IT organizations are not likely to be 100 percent sure at any given time which application workload is running where.

To address that issue, Morpheus Data has added a Cross-Platform Discovery module to its Unified Ops Orchestration platform that employs machine learning algorithms to determine not only what applications, virtual machines and containers have been deployed on-premises or in a public cloud, but also how much capacity, memory usage and power consumption each is using and their and overall performance.

At the same time, Morpheus Data is adding integration with Github and Jenkins continuous integration/continuous delivery (CI/CD) environments as well as support for both Docker Swarm and Kubernetes clusters.

Finally, Morpheus Data has added connectors to IBM Cloud, Upcloud and HPE OneView services and been certified as a ServiceNow partner. The Unified Ops Orchestration platform already supports Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform.

Brad Parks, vice president of marketing and business development for Morpheus Data, notes that as IT environments become more distributed in the age of the cloud, new approaches to change management will be required. The Unified Ops Platform provides an opportunity to consolidate change management processes within the context of a single control plane.

While more IT organizations are making use of multiple cloud services, most are managing each IT environment in isolation. Over time, the lack of unification of the control planes for each of those environments conspires to increase the total cost of ownership for IT. Morpheus Data is making a case of for unifying the management of multiple platforms running on-premises or in public cloud instead of requiring IT staff to master multiple management tools.

That single management plane, Parks says, also makes it much easier for IT organizations to take dynamically employ multiple cloud environments. Price points for cloud services are liable to change at any moment, but it doesn’t necessarily follow that the same type of workload will continue to be deployed on the same cloud. At the same time, the characteristics of a workload may have evolved to the point where it doesn’t make financial sense to deploy it on a public cloud. A common management platform makes it possible to move that workload with the least amount of disruption possible.

Most IT organizations today are concerned about getting locked into a cloud service provider. Being able to demonstrate an ability to switch between cloud service providers gives IT organizations an opportunity to negotiate cloud service contracts from a position of strength. Most cloud service providers are counting on the fact that the cost of switching from one provider to another is simply too high. Integrated DevOps processes wrapped around a common management plane, however, will make it clear to all concerned that the platform provider serves at the pleasure of the internal IT department, not the other way around.

— Mike Vizard

December 6, 2017
LightStep Unveils APM for Microservices

Various forms of application performance management (APM) platforms have been around for some time. But with the rise of microservices many IT organizations are discovering that legacy APM platforms were not designed to handle the volume of software modules or the rate at which those modules are updated. But LightStep, a startup founded by former Google employees and flush with an additional $29 million in funding, is positioning itself as the first APM written from the ground up for microservices environments.

LightStep CEO Ben Sigelman says regardless of where software modules and components get deployed, LightStep [x]PM is capable of monitoring and troubleshooting performance issues across all web and mobile clients, monoliths and microservices.

Most of the demand for LightStep [x]PM, he says, is being driven primarily by microservices based on containers such as Docker. But the APM platform itself can support both microservices and legacy monolithic applications. That creates an opportunity to rationalize APM platforms as organizations make the shift to microservices while continuing to need to support existing legacy applications running in a production environment, Sigelman says.

Twilio, Lyft, Yext, GitHub and DigitalOcean are already using LightStep [x]PM, which traces its roots back to Dapper, an always-on distributed tracing system that Sigelman developed at Google in production environments. A decentralized architecture enables LightStep [x]PM to continuously analyze 100 percent of transactions across all services. A statistical engine is then employed to detect anomalies, which the APM can then record and replay as part of an end-to-end tracing capability based on OpenTracing, an open API standard co-created by Sigelman and others that has now become a project administered by the Cloud Native Computing Foundation (CNCF).

Other capabilities embedded within LightStep [x]PM include support for service mesh and load-balancing technologies such as Envoy, linkerd, nginx and haproxy.

With every new era of enterprise computing, a contest occurs between startup companies such as LightStep and incumbent providers of technology platforms. LightStep is clearly trying to fill a void before incumbent providers of APM platforms can add support for microservices. Incumbents occasionally rely on their own organic efforts to upgrade their platforms, but more often they buy a startup company and then spend several years trying to unify disparate code bases.

Of course, there are many organizations that are just beginning their DevOps journey. Many of the them may not have any kind of APM platform in place. Most tend to be cost-sensitive as well, so offerings based on open-source code tend to have greater appeal.

However the APM wars play out, it’s apparent they are about to enter a new phase. The winners and losers of those wars will be determined by DevOps teams rather than C-level IT executives trying to impose a framework from on high. After all, if IT organizations have learned anything in the last few years, it’s that DevOps tends to have a more lasting impact when it comes from the bottom up.

— Mike Vizard

November 30, 2017
Logz.io Extends Scope of Log Analytics Service

Log analytics is all the rage these days among IT and security professionals trying to discover the root cause of any performance issue or security breach. But there’s little consensus about how best to go about analyzing logs. Many IT organizations have made major investments in commercial platforms such as Splunk, while others prefer to rely on the less expensive, open-source tools Elasticsearch, Logstash and Kibana, collectively known as the ELK stack.

As a provider of a cloud service based on the ELK stack, Logz.io has been providing organizations that prefer to rely on open-source software an alternative to deploying and maintaining their own ELK stack. Now Logz.io is extending that service to include Logz.io Application Insights, which employs machine learning algorithms to create a model of normal IT operations that can be used to more easily identify anomalies, and Logz.io Data Optimizer, which helps identify which logs are worth keeping long-term.

Fresh off raising an additional $23 million in capital, Logz.io CEO Tomer Levy says his company is building up a pool of machine data to drive advanced analytics fueled by machine learning algorithms. Those insights then can be shared proactively with other Logz.io customers to head off production issues before they even occur. At present, Logz.io has more than 400 customers.

Levy says the Logz.io Data Optimizer should enable IT organizations to substantially reduce their costs. Most providers of log management tools charge based on the amount of data collected. By helping organizations identify extraneous log data, the cost of implementing analytics of machine data drops considerably, Levy says.

Logz.io Application Insights, meanwhile, provides a simple way to identify the last changes made to the environment that might have created a performance issue or introduced a security vulnerability.

Levy says Logz.io, via application programming interfaces (APIs), can be integrated with multiple continuous integration/continuous development (CI/CD) platforms, including Jenkins. In fact, log analytics is now a fundamental element of any mature set of DevOps processes, he notes.

Platforms for analyzing machine data is one of the fastest growing sectors of enterprise IT. New research from Zion Market Research forecasts that the IT operations analytics market is expected to grow on a 23.7 percent compound annual basis to reach $9.2 billion by 2020.

Most DevOps issues only take a few minutes to resolve. But identifying the root cause of an issue can take weeks, sometimes even months, to identify. In more advanced IT organizations, centers of excellence around log analytics are now being created to identify not only potential IT and security problems, but also issues impacting, for example, the amount of revenue being generated via a specific application.

It still may be a while before log analytics become pervasive across the enterprise. But every event in the IT environment generates a log, so the real challenge and opportunity is correlating all those log events into something that manifests itself as a truly actionable intelligence.

— Mike Vizard

November 29, 2017
Axway Extends API Management Reach

With application programming interfaces playing a larger role in the DevOps environment, Axway has added to its API toolkit for developers. The company expanded its AMPLIFY API management platform to include a visual tool for composing and integrating APIs and enhanced its API Manager tool so that it now can be invoked to scale resources up or down as needed. In addition, Axway unveiled API Central Service that makes it easier to integrate APIs across an ecosystem of organizations that make their services available via the AMPLIFY marketplace.

Other new capabilities include an open-source MQTT connector for building internet of things (IoT) applications, a connector to the Microsoft Azure Cloud and tighter integration between Axway’s MFT file transfer software and the Syncplicity file synchronization software it acquired earlier this year.

Axway also merged the Titantium SDK for building native mobile applications using JavaScript into the Core SDK for developers and is previewing the Query Builder Analytics tool, which enables users to do ad hoc query of mobile statistics. The Titanium code is based on technology developed by Appcelerator, which Axway acquired early last year.

The announcements were made at the recent Axway SPARK 2017 conference. Vince Padua, vice president of platform and products for Axway, says these and other enhancements to the Axway API portfolio are intended to make it easier for organizations to craft microservices spanning multiple platforms. In some instances, those microservices will be crafted by developers, but Axway believes more advanced users will be stitching together microservices on their own using visual tools, says Padua.

While most early use of APIs was driven by web-based applications facing external users, using APIs to interconnect services inside the enterprise now dwarfs the number of instances of published external APIs, he notes. The proliferation of those APIs has created a significant DevOps challenge for organizations and requires an API management platform to streamline the management of various back-end service. At the same time, Padua notes that DevOps teams working in collaboration with line-of-business units have taken over much of the API management, which previously was left up to individual developers. Now, IT organizations are trying to strike a balance between maintaining control and developer productivity, he says.

The shift in API management responsibilities has also driven a wave of mergers and acquisitions over the last two years as API management has become embedded in mainstream application development platforms. But there’s also a fierce debate over the degree to which legacy API management platforms can support microservices. As microservices based on emerging technologies such as containers or more mature Java programming tools continue to evolve, API management is becoming more critically important. Each microservice generates its own API that needs to be maintained as each microservice gets continually updated. That requires an API management platform that can support all levels of API call activity that once would have been assumed to be unimaginable.

— Mike Vizard

November 28, 2017