devopsy.kinsta.cloud

Author: Mike Vizard

BMC Pushes Automation to the DevOps Left

Much of the inherent conflict that exists between developers and IT operations teams often comes down to how IT automation processes are implemented. IT operations teams generally have their own preferred tools for automating operations while developers have another. To resolve that issue, BMC Software unveiled Control-M Workbench, a set of development tools that extends an application programming interface (API) to allow developers to embed instructions for automating infrastructure using BMC Control-M automation platform.

Gur Steif, president for digital automation at BMC Software, says Control-M Workbench moves the responsibility for automating the IT infrastructure resources that their application consumes, including the ability to code, debug and test workflows. The IT operations team can then more easily deploy that application in a production environment, he says.

In effect, BMC Software is enabling IT organizations to put the control of how jobs are executed in a production environment in the hands of developers by moving that function further to the left of the DevOps life cycle. Control-M Workbench represents the first time BMC is venturing beyond its traditional IT operations base to create a tool that will primarily be employed by developers.

Steif says the goal is to enable IT operations teams to manage much higher volumes of application rollouts. In return for spending a few minutes on installing Control-M Workbench, IT organizations can deliver application 20 percent faster, while running jobs using 50 percent fewer full-time employees. In addition, Stief says production incidents can be reduced by up to 25 percent and batch processing failure rate can drop by as much as 80 percent.

Control-M Workbench is clearly part of a trend to give developers more control over and responsibility for IT operations. A recent survey of 650 IT decision makers conducted by BMC finds 94 percent of IT decision makers expect automation to spread into all other areas of business by 2020.

In addition, the survey notes that 73 percent of IT decision makers believe businesses that do not embrace IT automation to achieve the larger digital business strategy within the next five years will cease to exist in 10 years. The top three areas of investment priority in the next 24 months to achieve those goals are containerization, workload automation/scheduling and DevOps. Almost 9 in 10 (89 percent) respondents concur that IT automation needs to achieve their desired digital business objectives.

However, there are considerable obstacles that need to be overcome, including lack of budget (67 percent), skills (44 percent) and time (51 perent). In fact, 42 percent of the respondents concede that when it comes to digital business initiatives there are conflicting objectives between business units and CIOs that can get in the way of crafting a comprehensive IT strategy.

DevOps, of course, is more about people and processes than it is technology. Then again, DevOps isn’t going to be willed into existence without providing the tools needed to change the processes. Given the chicken-and-egg relationship between processes and tools, it becomes incumbent on the IT leadership to serve as the catalyst for bringing it all together.

— Mike Vizard

June 20, 2017
Microsoft Lifts Investment in Cloud Foundry

At the Cloud Foundry Summit Silicon Valley 2017 conference this week Microsoft formally pledged its support for the open-source platform-as-a-service (PaaS) environment by becoming a gold member of the Cloud Foundry Foundation (CFF).

Beyond influencing the future direction of the Cloud Foundry PaaS, Microsoft also revealed this week it will tighten the integration between the Cloud Foundry PaaS running on Microsoft Azure and backend database services. There also now command line tools for Cloud Foundry available within Azure Cloud Shell, a browser-based tool for managing Azure resources.

Corey Sanders, partner director for Microsoft Azure, told conference attendees that Microsoft is fully committed to Cloud Foundry as any member of the foundation. In fact, there have been instances of Cloud Foundry running on Azure for years, including the version of Cloud Foundry that SAP makes available as part of a multi-cloud computing strategy it has crafted. Microsoft claims usage of Cloud Foundry on Azure has increased 300 percent over the last year.

CFF CTO Chip Childers says that endorsement of Microsoft is significant because, along with Google, it means two of the three largest public cloud providers are now investing in Cloud Foundry. There are instances of Cloud Foundry running on Amazon Web Services (AWS), but the largest public cloud service provider has not seen fit to join the CFF.

The mission of CFF is to promote the adoption of multi-cloud computing via technologies that can be deployed across both public clouds and an on-premises IT environment, Childers says. He expects Cloud Foundry and the Kubernetes container orchestration engine to play similar roles in that regard. The actual operating system deployed in those environments, he says, is now much less relevant. The choice regarding when to deploy one or another will come down to the nature of the application workload and the relative cloud-native maturity of the organization deploying those workloads.

Microsoft has already made it clear it it is committed to offering multiple classes of services with Azure. In addition to Cloud Foundry, Microsoft already offers a PaaS environment built on its own proprietary stack of Windows software. There’s also both a container-as-a-service offering based on Kubernetes and implementations of the container stack developed by Docker Inc., in the wake of a strategic alliance between those two companies.

Obviously, the range of deployment options now being made available on platform such as Microsoft Azure creates some significant DevOps challenges. Rather than trying to limit the number of services developers might invoke, IT operations teams will need to focus more on making sure the right type of application workload invokes the class of cloud service that is best optimized for that workload. Naturally, that approach will require IT organizations to address both technology and cultural issues within their organizations. But given the limited effectiveness most IT operations teams historically have had in terms limiting deployment options, there’s no better time than the present to get started.

— Mike Vizard

June 15, 2017
Atlassian Builds a DevOps Stack for the Enterprise

One of the major challenges associated with DevOps is that it’s not something an enterprise IT organization can buy. Because of that issue, it’s often challenging for IT organizations to forecast what types of DevOps tools they are going to need. To make it a lot easier to cope with that issue, Atlassian has announced Atlassian Stack, a suite of software that makes Atlassian DevOps tools available in a single package.

Sean Regan, head of product marketing for Atlassian, says this approach will make it easier for enterprise IT organizations to embark on a DevOps journey without having to worry about what type of tool they may need now versus in the future. For example, an IT organization may opt to get started using the Atlassian Jira software to track software development projects and then work its way up to deploying Atlassian Bamboo continuous integration/continuous deployment (CI/CD) software. From there, Regan says, a DevOps team may opt to embrace collaboration software to streamline communications across their various teams.

The move to launch Atlassian Stack comes after the company updated a slew of its DevOps offerings, including the release of Bitbucket Server and Data Center 5.0 and Bamboo 6.0. In addition to tightening the integration between both DevOps tools, the company added capabilities for verifying commits, smart mirroring and the ability to define a Bamboo configuration as code.

Rather than making DevOps teams acquire each tool individually, he says, many senior IT leaders now request a means through which the entire Atlassian portfolio can be made more readily available. That shift in purchasing behavior suggests that the level of DevOps maturity inside many organizations is advancing considerably, he notes.

Of course, there are many who would contend that level of DevOps maturity has been a long time in coming. Web-scale companies have been employing DevOps tools for years to accelerate the rate at which new applications are released and updated. In contrast, traditional enterprise IT organizations have been slow to implement DevOps processes. In fact, it’s taken the pressure arising from digital business transformation initiatives that have accelerated application deployment timetables to get enterprise IT organizations to fundamentally rethink IT operations. Business leaders are making it painfully obvious that their patience with extended rollouts for new applications is at an end.

Naturally, the rate at which those transformations are occurring within organizations will still vary considerably. However, the silos that exist within enterprise IT organizations need to be consolidated. The next issue is determining to what degree altering those processes needs to come in the form of a DevOps edict from above versus letting teams come together of their own accord to meet a specific application deadline. Whatever the approach, the way enterprise IT has been managed historically is no longer up to the digital business transformation tasks at hand.

— Mike Vizard

June 15, 2017
Node.js Becoming More DevOps-Friendly

Over the last several years use of Node.js has exploded across the enterprise, as developers moved to embrace a variant of Javascript that made it easier to build both client and server-side applications. However, every time it is upgraded, all the dependencies between it and application modules written in C or C++ breaks.

To solve that problem, the Node.js Foundation is making available on an experimental basis a Node.js API (N-API), developed collaboratively by Google, IBM, Intel, Microsoft, nearFrom, NodeSource and other contributors. The API will be in Node.js 8, and promises to eliminate the need to update all those dependencies. Pending additional testing, it is expected to be given Long Term Support (LTS) status later this year.

At the same time, The Linux Foundation is adding new features to experimental tracing and async tracking features. A diagnostics API will make it easier to monitor the operation of the Node.js event loop across the complete life cycle of a service request. Finally, an inspector core module, which allows developers to leverage the debug protocol used by the Google Chrome inspector to inspect currently running JavaScript code, is also being added.

Each of these new capabilities are nice to have on their own. But collectively, they signal the beginning of a maturation process in the Node.js community. Increased use of Javascript in general—and Node.js especially—have arguably caused many enterprise IT organizations re-evaluate their DevOps processes. Developers using Node.js have been able to increase the number of applications they could develop considerably. Since then, the number of languages being employed within the enterprise has exploded.

Mark Hinkle, executive director of the Node.js Foundation, says moving forward there will be more focus on reducing the friction surrounding upgrades of Node.js applications as part of an effort to further expand the ecosystem. Many organizations commonly end up supporting multiple versions running across a variety of applications. Hinkle says N-API is a step in helping to reduce the maintenance costs associated with supporting those applications. In an ideal world, the foundation would be gathering additional input from DevOps experts pursuant to achieving that goal. In most IT organizations, the care and feeding of Node.js applications is still left in the hands of IT operations teams after the apps are deployed in production.

In the meantime, Hinkle says its performance continues to improve. Naturally, there’s a corresponding relationship between better performance and the number of use cases for which it will be considered. Of course, that doesn’t mean Java and other legacy programming languages won’t be used to develop new applications. But it does means that in terms of the number of programming languages employed across the enterprise, it’s now a polyglot universe.

— Mike Vizard

June 13, 2017
Survey: DevOps Divide Grows a Little Wider

The gap between the number of organizations that have invested in DevOps is starting to narrow, and IT organizations that made the transition to DevOps earlier are starting to see accelerated returns on that investment, according to the latest survey by Puppet.

The annual “State of DevOps Report” finds that out of 3,200 IT professionals surveyed, only 11 percent were part of organizations that could be considered high DevOps performers. High performers are deemed to have higher deployment frequencies, lower lead times for changes, better mean time to recovery (MTTR) and better change failure rates. The higher performers in this year’s survey are deploying applications 46 times faster than the same designated group of higher performers were last year.

The survey also finds that higher-performing organizations have automated 72 percent of all configuration management processes. Those same high performers spend much less time (28 percent) employing manual configuration processes. Lower performers are spending almost half of their time (46 percent) on manual configuration.

Just over a quarter (27 percent) of the respondents self-identified themselves as being part of a DevOps team, according to the survey. In contrast, only 16 percent said the same last year. Puppet CTO Nigel Kersten says those results suggest that understanding of the power of DevOps is increasing. It’s also apparent the early implementers of DevOps processes are enjoying a level of IT agility that is much greater than the average IT organization.

In general, Kersten says that while much DevOps progress has been made, most IT organizations have automated only various pockets of silos. Developers, for example, may have automated some processes on their own, while IT operations specialists may have developed a custom script to automate the configuration of a handful of servers. Kersten notes IT leaders should be cognizant of the technical debt their organizations are incurring every time different elements of the IT organization employ different tools to automate the same basic processes time and again.

Very few have automated their operations within the context of a continuous integration/continuous deployment (CI/CD) pipeline, says Kersten. That becomes more challenging to accomplish as new computing models based on containers, for example, are added to the enterprise mix. On a more positive note, Kersten says more processes are being well-defined, as they are bounded by an application programming interface (APIs). Those APIs then become the core technology over which IT automation frameworks are then laid.

The issue many IT organizations have yet to confront is the degree to which they want to stumble toward DevOps versus implementing a concerted plan of action. Much of the DevOps progress to date has been driven from the bottom up inside many organizations. The time is now approaching for IT leaders to imbue DevOps as a part of the IT culture that permeates the entire organization.

To one degree or another it’s a safe bet every organization will implement some form of DevOps going forward. The challenge will be figuring out how to transform all those isolated instances into something more akin to an actual IT strategy.

— Mike Vizard

June 12, 2017
Salesforce Moves Compliance to the DevOps Left

Compliance is one of those tasks that trips up developers multiple times over. More times than anyone cares to admit developers finish an application only to discover there are any number of compliance controls that were overlooked. Sometimes, bringing that application into compliance is trivial. Other times, it can delay the rollout of the application by months.

To help mitigate this issue, Salesforce this week added a Heroku Shield module to Heroku Enterprise, its container-based platform-as-a-service (PaaS) environment available to IT organizations needing access to a dedicated platform running on Amazon Web Services (AWS). Dylan Steele, vice president of marketing for the Salesforce platform, says Heroku Shield enables developers to comply with a broad range of mandates ranging from payment card standards to any number of healthcare regulations. The goal is to automate meeting regulation compliance by making it possible for developers to automatically add the appropriate controls to their applications.

Heroku Shield requires mandatory encryption for data at rest in addition enforcing a stricter level of TLS encryption for applications deployed in a Shield Private Space. It also enforces trusted IP ranges and manages application-level authentication tokens.

The move is part of the effort to shift left more of the responsibility for IT security, by making it easier for developers to address security and compliance issues early in the application development process. That’s when the cost of fixing those issues is substantially less than when that application is deployed in a production environment.

In general, there’s not a lot of love lost between developers and auditors. Developers often feel that the controls needed to achieve compliance are not all that clear. Auditors feel that many developers are simply too lazy to implement the appropriate controls in the first place. In truth, however, there’s a lot of commonality between the control required to meet various compliance requirements. That creates a lot of opportunity for PaaS environments that abstract the underlying infrastructure away from developers to automate implementing a broad range of compliance and security controls. The biggest issue now may be simply making developers aware that’s possible.

In the meantime, DevOps teams that regularly struggle with compliance issues at the back end of an application release cycle might want to reconsider their approach. Whether it involves a PaaS environment or not, there’s a strong case to be made for moving testing of compliance controls as far left as possible in the application development process. Not only does that approach minimize the number of compliance issues that may arise before and after an application is deployed, but the documentation generated during the development process should substantially reduce the amount of time auditors spend crawling over and around an IT environment. That’s critical, since most auditors charge by the hour, and every minute they spend on an audit also increases the probability of finding something else that needs to be fixed.

— Mike Vizard

June 8, 2017
VMware Moves to Automate Management of Hybrid Clouds

Moving to expand the reach of its portfolio of IT management software well beyond virtual machines, VMware today announced that the vRealize suite of IT automation tools can be employed across containers and virtual machines running on-premises or in a public cloud.

In addition to providing support for VMware’s Admiral container management portal, version 7.3 of vRealize Automation adds much tighter integration with both Amazon Web Services (AWS) and Microsoft Azure public clouds. For Microsoft Azure, vRealize now can deploy and manage middleware and application services. For AWS, vRealize Network Insight now can microsegment networks in an AWS Virtual Private Cloud as well as troubleshoot network traffic issues in AWS.

From a DevOps perspective, however, the most significant new capability is being added via an update to the vRealize Operations module: the ability to automatically deploy workloads across multiple clusters and datastores based on business policies defined by the IT organization, says Mahesh Kumar, senior director of product marketing for VMware. This update also adds a predictive Distributed Resource Scheduler (pDRS) that leverages analytics provided via vRealize Operations to predict anomalies and identify potential I/O contention issues before they occur.

At the same time, the latest update to VRealize Operations will incorporate a tab for vRealize Business for Cloud 7.3, which has been extended to provide cost and capacity utilization metrics for AWS and Microsoft Azure alongside VMware-based private cloud costs.

Other additions to the VMware portfolio include tighter integration between VRealize Operations and VMware vSAN storage software, NSX network virtualization software, enhanced blueprints for sizing deployments and integration with configuration management tools from Puppet. Kumar says that latter capability will be extended to include support for Chef and Ansible configuration management tools as well.

With these additions to the IT automation portfolio, VMware is clearly signaling its ambitions lie beyond its own hypervisor. Within enterprise IT environments VMware is dominant. That position has enabled it to gain significant traction in terms of driving adoption of its management tools. Now, VMware is moving to extend the reach of those tools into public clouds and containers. At the recent Dell EMC World 2017 conference last month, VMware debuted instances of its portfolio of software running on AWS. Most containers today are deployed either on top of a virtual machine or in a platform-as-a-service environment. But VMware is also aware of the fact that over time more containers will be deployed on bare-metal servers that are not running any type of virtual machine software at all.

Of course, just about every other provider of an IT management framework knows this as well. In addition, there’s no shortage of management tools designed specifically for containers. But as DevOps continues to evolve, most IT organizations are naturally expected to favor approaches that enable them to manage the broadest number of platforms possible.

— Mike Vizard

June 6, 2017
IBM: DevOps, AI to Drive Digital Business Transformation
Conventional wisdom holds that one of the biggest challenges organizations face today is a chronic shortage of application developers. But as artificial intelligence (AI) advances, the process of writing code will become more automated. But rather than there being not enough developers to write all the code necessary, the shortage will be in software engineers who can develop innovative systems.

Bob Lord, chief digital officer for the Digital Business Group at IBM, says a glimpse of the future of application development is already being provided today. Instead of writing low-level code, most developers today simply combine modules to create new applications. Anytime new functionality is required, another module gets dropped into the application. The number of developers writing the low-level code needed to create those reusable modules is relatively few, says Lord.

Over time, however, AI models will learn how to write that code. People will add value by providing the engineering expertise needed to combine modules to create an innovative application, Lord says. Naturally, most of the processes employed to deploy that software also will be automated. But IT operations teams will be needed to continuously train the AI systems being used to deploy that software. This is in much the same way that we need human pilots to fly a plane whenever something new or different occurs, despite the existence of autopilot technology.

In general, Lord says, IT organizations would be well-advised to focus on three things:
- They need to decide what architecture strategy is going to be employed to accelerate DevOps integration.
- They need to develop a data strategy that drives digital business transformation.
- They need to layer a cognitive computing strategy that employs advanced algorithms to maximize the business value of the data they’ve accumulated.
While some organizations approach these issues in isolation, Lord says the reality is they’re all closely connected. Organizations are investing in DevOps not just to reduce costs—DevOps is about attaining a level of IT agility capable of driving digital business transformations. In fact, Lord believes the way IT organizations are measured will change: Rather than being rated on traditional IT metrics, they will be measured on how effective they were in attaining business goals—especially when to comes to velocity at which applications are rolled out and updated.

In the meantime, Lord says, developers are starting to exercise more influence over the technologies organizations are employing to achieve those goals. Because of that issue, IBM is making an effort to get closer to the developer community. But as application development advances, many of those developers will be software engineers rather than people who write code, while IT operations teams will become teachers who train intelligent systems how to respond to the needs of deployed code.

— Mike Vizard
May 30, 2017
New Relic Extends Monitoring Reach

New Relic this week at its Future Stack 2017 conference extended the reach of its monitoring services via the release of a software development kit (SDK) that makes it possible for IT teams to monitor custom applications.

In addition, New Relic unveiled Health Map, which makes it much easier for IT administrators to correlate data collected from New Relic application and infrastructure monitoring services. Todd Etchieson, vice president of product management for New Relic, says developing Health Map was a natural next step after the expansion of its services into the realm of infrastructure monitoring. That capability should significantly reduce the finger-pointing that often occurs between developers and IT operations teams anytime there’s an application performance issue, says Etchieson.

New Relic is also adding billing and cost management features that enable customers to monitor costs in real time as well as plan for future Amazon Web Services (AWS) resource requirements. Rather than having to invoke a separate tool, users can leverage New Relic’s extensive history of relying on AWS to host its tools.

Application performance management (APM) tools from New Relic first gained prominence among developers who preferred to monitor their applications using a service. Because New Relic is deployed on AWS it’s especially well-suited to monitoring applications deployed on AWS. The company this week also announced that its monitoring tools now come with 20 pre-built integrations to AWS services including Amazon Kinesis Firehose, Amazon Elasticsearch Service, Amazon Route 53, Amazon EC2 Container Service and Amazon EC2 Container Registry.

Now Etchieson says New Relic is expanding its reach in the enterprise as organizations embrace a more integrated approach to DevOps. In fact, he notes half of its customers are already consuming multiple services. Driving that adoption is a need to roll out more applications, many of which are based on microservices enabled by containers. This fact is forcing organizations to confront their DevOps issues because of the dynamic and ephemeral nature of containers. New Relic added support for container monitoring back in 2015.

Etchieson says New Relic doesn’t necessarily care where IT organizations choose to deploy their applications. Because New Relic made its APM software available as a service, use of APM expanded considerably. The company currently has 1,700 enterprise customers, he says.

Not all those customers are very far down the DevOps path, but more than a few organizations have begun their DevOps starting with an APM tool. In fact, in many organizations DevOps is a bottom-up transformation that begins with developers seeking more information about how their applications are performing.

As DevOps continues to evolve, the divide between application and infrastructure performance monitoring tools will continue to narrow. There may be different roles inside an IT organization that require different views to address specific issues. But at its core DevOps is all about providing cross-functional IT organizations with the context needed to resolve issues faster. In almost every case, that will require correlating data across both the application and infrastructure stack.

— Mike Vizard

May 26, 2017
Chef Applies IT Automation to Meet Compliance Mandates

Much of the productivity waste associated with compliance requirements stems from the fact that most developers and IT operations teams aren’t aware how an application might be non-compliant until it’s too late. Aiming to eliminate that fundamentally inefficient approach to meeting compliance mandates, Chef this week tightened the integration between its Automate continuous automation platform and InSpec, an open-source project lead by Chef for specifying compliance and security policies.

At the same time, Chef enhanced support for Docker within InSpec and released InSpec-AWS, InSpec-Azure and InSpec-vSphere as incubation projects that IT operations teams can employ to test and audit for compliance. Chef also pledged to extend InSpec beyond operating systems to include support for middleware to address compliance issues across an entire platform. The announcements were made at the ChefConf 2017 conference.

Chef CEO says Barry Crist says that in much the same way Chef automation software turns infrastructure into code, InSpec turns compliance into code. Tighter integration between the two platforms should enable IT operations teams to automate compliance testing before code gets deployed in a production environment. In effect, Crist says, compliance becomes just another continuous process automated by Chef. This approach not only leads to faster application deployments, it also reduces the amount of time auditors must spend checking on where production applications comply with a particular mandate, as all the documentation has already been generated.

Compliance mandates are, in many ways, the enemy of innovation. In fact, it often takes longer these days to navigate compliance issues than it does to write a piece of code. Rewriting code to comply with multiple compliance requirements over several different manual tests is a waste of time and money. InSpec makes it possible to make compliance testing a part of an integrated DevOps process in much the same way security testing is fueling DevSecOps.

As the number of platforms that IT operations teams now need to support has expanded, the compliance testing process has become even more complicated. Crist notes that for DevOps to be truly effective, it needs to encompass not just the deployment of code but also making it possible to make compliance and security testing an integrated component of the application deployment process. Of course, as an open-source project there’s no reason developers can’t incorporate InSpec into any continuous integration/continuous deployment (CI/CD) framework they choose. Chef is just moving to make InSpec a more natural extension of its IT automation framework.

There’s a lot of commonality in the controls implemented across various compliance specifications. That creates opportunity to automate controls to reduce the need to rework an application, either before it gets deployed or, worse, after it has been deployed into production. Longer-term, the level of commonality across compliance controls also creates an opportunity to leverage machine-learning algorithms to automate implementing those controls.

Of course, there will also be controls that will be unique to one compliance standard or another. But the days when IT organizations need to manually address those compliance controls one application at a time are coming to an end.

— Mike Vizard

May 25, 2017