devopsy.kinsta.cloud

Author: Mike Vizard

AWS Advances DevOps Agenda at Summit Event

Amazon Web Services (AWS) has added services to its cloud platform designed to automate the creation of virtual private clouds (VPCs) and made available additional tools to build applications and integrate external data.

Speaking at the AWS Summit event, CTO Werner Vogels said these latest services are part of an ongoing effort to automate DevOps processes to the point at which developers will need to write only the code for business logic that runs on top of the AWS public cloud.

One of the latest tools the company is making available in pursuit of that goal is AWS Cloud Development Kit (CDK), a software development framework for defining cloud infrastructure in code as constructs and provisioning it through AWS CloudFormation. AWS CDK currently supports TypeScript, JavaScript and Python with support for C#/.NET and Java available in preview. That capability will eliminate the need to write low-level code using a different tool just to spin up a VPC on AWS, Vogels said.

AWS CDK also allows organizations to design and compose their own custom components for managing infrastructure alongside their application code.

The cloud services provider also announced AWS EventBridge, a programmable event bus for integrating data on AWS with data residing in software-as-a-service (SaaS) applications on any platform based on a serverless computing framework, and the general availability of AWS Toolkit for Visual Studio Code, a plug-in for the Microsoft integrated development environment (IDE) that makes it easier to build, debug and deploy code on AWS.

Finally, the company announced that Amazon CloudWatch Container Insights is now available in preview. This offering provides access to automated dashboards summarizing the performance and health of their Amazon Elastic Container Service (ECS) and Fargate clusters by tasks. Those offerings come on the heels of AWS Fluent Bit, a tool for container logging being made generally available earlier this week.

Vogels stressed that the way applications are built is now fundamentally different in the age of the cloud. Developers are making greater use of containers and serverless computing frameworks. The challenge with containers is that in the absence of a cloud service provider to manage the environment, the complexity of the underlying middleware and infrastructure conspires to reduce the amount of time developers spend on writing actual application code, he said.

To address that issue, AWS is investing more in container and Kubernetes services as well as in on-demand serverless computing services such as AWS EventBridge that don’t require organizations to pay for idle processes, Vogels explained.

In general, Vogels said AWS continues to advocate for the elimination of separate IT operations and cybersecurity teams in favor of a more unified approach leveraging automated services provided by AWS that ensure, for example, the entire continuous integration/continuous deployment (CI/CD) platform is secure. However, while the number of organizations embracing best DevOps and DevSecOps processes continues to increase steadily, it’s much less clear how many of them will standardize those processes on a single cloud platform at a time when reliance on multiple clouds also continues to increase steadily.

— Mike Vizard

July 12, 2019
HashiCorp Tools Bridge DevOps, Legacy IT Environments

At its HashiConf EU 2019 conference this week, HashiCorp extended the reach of its Terraform Enterprise tools for automating the management of IT infrastructure beyond the realm of DevOps into legacy IT environments.

Via integration with the Service Catalog from ServiceNow, IT administrators working with a traditional ITIL-based framework now can invoke Terraform Enterprise to provision IT services. HashiCorp CTO Armon Dadgar said rather than requiring organizations to hire an infrastructure engineer to automate the provisioning of IT infrastructure, IT administrators now can do the job.

At the same time this week, HashiCorp announced that policy sets may now be configured using the source policies available in multiple version control systems (VCSs), including GitHub, GitLab and Bitbucket. The goal is to make it simpler to enforce policies that would limit, for example, the number of virtual machines a developer might be allowed to spin up within the context of a set of GitOps processes, said Dadgar.

Finally, HashiCorp also announced that HashiCorp Consul service mesh networking software has been enhanced to include Layer 7 application traffic management capabilities alongside integration with the HashiCorp Nomad application and cluster scheduler software.

In general, Dadgar said HashiCorp is moving toward enabling its portfolio of offerings to span multiple types and classes of IT environments. HashiCorp Consul, for example, can be run across both legacy IT environments as well as Kubernetes. Most organizations will need to support both legacy monolithic and microservices applications using a combination of DevOps and legacy ITIL processes, noted Dagar. Rather than having to master new tools for each environment, HashiCorp is making a case for a common set of tooling that can be applied across multiple IT classes of processes.

While some organizations have fully transitioned to DevOps, many more organizations are still struggling to make the transition. Reasons for not making that transition faster range from simple inertia to a lack of skills. Rather than having to take an all-or-nothing approach to modernizing IT processes, Dadgar said HashiCorp is enabling each IT department to embrace IT automation wherever they happen to be on that journey, including being able to embrace new platforms as they become available.

Dadgar noted that level of extensibility is critical because most tools are still being brought into organizations from the bottom up by individual members of the IT staff, rather than being championed from the top down. In the absence of an executive mandate, Dagar said the path of least resistance is to make IT automation tools as accessible as possible to members of the IT team regardless of skill level.

There will, of course, come a day when IT leaders will try to rationalize processes around a common standard set of practices. The degree to which any IT organization has been able to achieve that goal, however, has been mixed at best. In the meantime, IT organizations will always need to keep their options as open and flexible as possible.

— Mike Vizard

July 10, 2019
source{d} Delivers Enterprise Edition of SDLC Platform

source{d} has made available an enterprise edition of its software development lifecycle (SDLC) platform that includes visualization and analytics tools along with additional management capabilities. A free community edition of the core source{d} platform, which is stripped of most of the enterprise-class tools, also is now available in beta.

Company CEO Eiso Kant said the enterprise edition of source{d} provides the additional tools enterprise IT organizations require to employ the platform as the equivalent of a business intelligence (BI) application for managing DevOps processes.

The tools can launch queries against a big data repository based on the Apache Spark in-memory analytics engine running on a Kubernetes cluster, which can be populated with actual data or metadata pulled from multiple DevOps tools and platforms. That capability enables IT leaders to then query data that has been joined together in a way that surfaces actionable intelligence, said Kant.

source{d} Enterprise achieves that goal by including Apache Superset, an open source business intelligence web application. The platform also can be extended to support SQL queries launched from other BI tools and project management applications. The enterprise edition also provides access to advanced queries and user-defined functions to address tasks such as code duplication analysis or identity matching.

Finally, source{d} Enterprise provides authentication integration with LDAP and OpenID to limit who can edit or view queries, data sources, charts and dashboards.

Longer-term, source{d} plans to leverage predictive analytics and machine learning algorithms more aggressively as the amount of SDLC data that gets collected by enterprise IT organizations increases.

Kant said many enterprise IT organizations have tried and failed to build the equivalent of source{d} platform mainly because bringing together and then maintaining all the required technologies often becomes too complex an undertaking. Those efforts also have been confined largely to managing processes such as bug fixes versus including all the BI tools needed to manage software development in a way business and IT leaders can align technology initiatives to specific business goals, he said, noting the source{d} approach brings together enabling technologies such as Spark and Kubernetes to enable IT organizations to deploy an SLDC platform on-premises or in a public cloud.

As software development increasingly grows more complex, the days when it could be managed efficiently through a project management application alone have come to an end. At the same time, more organizations realize their success ultimately comes down to how consistently they can deliver a high-quality application experience. For all the hype surrounding digital business process transformation, the core tenet is that most organizations are indeed becoming software companies. As such, they need to borrow many of the SDLC processes that successful software companies employ routinely. The only real difference is they don’t need to develop those processes from the ground up when more turnkey SDLC platforms are emerging.

— Mike Vizard

July 9, 2019
IBM Closes $34B Red Hat Acquisition

Following the official closing of IBM’s $34 billion acquisition of Red Hat today, leaders from both companies reiterated Red Hat will continue to operate independently of IBM but will leverage IBM resources to advance the hybrid cloud computing ambitions of the newly combined entity.

Arvind Krishna, senior vice president for cloud and cognitive software at IBM, said IBM and Red Hat together will be in a better position to address what he described as Chapter Two of a continuing cloud journey. With 80% of applications still running in on-premises IT environments some 10 years after the arrival of public clouds, it’s apparent hybrid cloud computing will become the dominant form of IT, he said.

Red Hat will play a critical role in helping customers achieve that goal using a common operating model based on distributions of Linux and Kubernetes container orchestration software, Krishna said. However, he added, IBM sees no need to rationalize its current middleware software portfolio, which often addresses use cases that Red Hat doesn’t.

Paul Cormier, executive vice president and president for products and technologies for Red Hat, said Red Hat will continue to invest in products and projects independently of IBM. As such, Red Hat does not expect its relationship with its partners that compete directly with IBM to change substantially, he said. At the same time, he stressed that working more closely with IBM will accelerate a 100% open source approach to hybrid cloud computing that otherwise would have taken Red Hat “many years” to execute on its own. That hybrid cloud computing strategy, in contrast to rivals, will span virtual machines, bare-metal servers and public clouds, Cormier said.

Other than introducing Red Hat to a potentially larger customer base, it’s not quite clear how being a unit of IBM accelerates the Red Hat strategy. Red Hat and IBM sales teams, for example, will have no incentives to cross-sell offerings and Red Hat will continue to promote its brand independently of IBM, in much the same way VMware operates independently of the rest of Dell Technologies. Both Red Hat and IBM also stressed there is no plan to impose an IBM culture on Red Hat or lay off any personnel. There is a consensus that open source software will play a role in enabling hybrid cloud computing that should generate opportunities for IBM to generate more professional services revenue.

Longer-term, there’s clearly an opportunity for both Red Hat and IBM to promote the adoption of a common set of best DevOps practices across a hybrid cloud computing environment. Many organizations, however, are already moving to extend those best practices across all the platforms that make up very diverse IT environments.

For now at least, organizations that have already committed to Red Hat appear to be lending their continued support. Red Hat’s fiscal year 2019 revenue was $3.4 billion, up 15% year over year. Fiscal first-quarter 2020 revenue was $934 million, up 15% year over year. Services revenue for Red Hat also grew 17%. Precisely how IBM and Red Hat intend to unlock the value of this merger beyond the momentum Red Hat already enjoys, however, remains a bit of a mystery.

— Mike Vizard

July 9, 2019
DevSecOps Survey Finds Failure to Communicate

While it’s generally agreed that shifting more cybersecurity responsibility onto the shoulders of developers is a good idea, a failure to communicate across application development and cybersecurity teams has contributed to little progress in achieving DevSecOps.

A survey of 1,310 IT decision-makers conducted by the market research firm Vanson Bourne on behalf of Trend Micro finds that while nearly three-quarters of respondents (74%) said integrating developer, IT operations and cybersecurity processes has become more important over the past year, more than one-third (34%) said these silos are making it more difficult to create a DevOps culture in the organization. A full 89% said software development and IT security teams needed to be in closer contact, while 77% said the same for developers, security and operations.

Greg Young, vice president of cybersecurity for Trend Micro, said the two biggest challenges in achieving DevSecOps historically are the lack of trust between developers and cybersecurity professionals and a lack of tooling to foster collaboration and communication across teams.

The rift between developers and cybersecurity is well-known: Application developers have been known to give short shrift to cybersecurity guidelines to make a deadline, only to have that application breached in a production environment. However, developers can counter that cybersecurity professionals can slow things down too much, in the name of being overly cautious.

Young said organizations need to find tools that make it easier for those two teams to establish trust within the context of an integrated DevSecOps process. Rather than just relegating cybersecurity teams to defining policies, cybersecurity professionals need to find a way to provide input during all stages of the application development process without compromising the overall agility of the development team, he said.

In fact, survey respondents identified fostering greater integration between teams (61%), setting common goals (58%) and sharing learning experiences across teams (50%) as the best ways to achieve DevSecOps. The trouble is, 78% of respondents said their organization needs to improve in all these areas.

Unfortunately, nearly half of respondents (46%) said they have only partially developed their DevOps strategy, with only a third of respondents (33%) saying DevOps today is a shared responsibility between software development and IT operations.

Regardless of how organizations achieve DevSecOps, the one certain thing is most organizations have no alternative. The rate at which applications are being developed today generally exceeds the ability of short-handed cybersecurity teams to keep pace. The only way to make sure applications are more secure than they have been historically is to make implementing security controls part of the application quality assurance process. That said, cybersecurity teams most likely will take a “trust, but verify” attitude toward developers to ensure the right controls have been put in place.

Of course, none of this is going to happen of its own accord. Senior leaders need to make DevSecOps best practices a requirement. Otherwise, DevSecOps will always remain one of those many aspirational IT goals that go unfulfilled.

— Mike Vizard

July 9, 2019
Pivotal Embraces Spinnaker for Continous Delivery

Pivotal Software has added support for the Spinnaker continuous delivery platform to its platform-as-a-service (PaaS) environment for the open source Cloud Foundry PaaS.

Richard Seroter, vice president of product marketing at Pivotal, said version 2.6 of Pivotal Cloud Foundry (PCF) extends the reach of the Spinnaker CD platform developed by Netflix to its PaaS environment and Enterprise PKS, an instance of Kubernetes curated by Pivotal and its sister company, VMware.

At the same time, the latest version of PAS adds support for a beta instance of custom sidecar processes, which enable developers to embed additional functions running in an isolated container directly within their applications. That capability makes it possible to deploy, for example, a service mesh within the application, said Seroter.

Finally, Pivotal plans to add multicenter replication capabilities for MySQL databases and a permissions model to Concourse for PCF, a continuous integration tool for building application pipelines with a Pivotal CF environment. That latter capability provides a mechanism through which DevOps teams can automate application updates, including patches, end to end, Seroter noted. As more patches are released to address the growing number of critical flaws involving microprocessors that make use of hyperthreading, the ability to automate their deployment is now more critical than ever, he said.

Pivotal is seeing the deployment of both traditional 12-factor applications and more modern containerized applications on its PaaS environment. At the same time, the Cloud Foundry Foundation, which oversees the development of the Cloud Foundry PaaS, has, via Project Eirini, made it possible to deploy containers that rely on Kubernetes rather than the Diego orchestration platform developed for Cloud Foundry. However, Eirini is still an alpha project and Pivotal has not yet fully committed to a specific level of support for Kubernetes with its overall PaaS strategy.

In the meantime, Seroter said PCF continues to gain momentum in organizations looking to build and deploy software much like a technology company. As more organizations realize how dependent they are on software to drive revenue, they want to rely on a platform that makes it easier for them to manage software development and deployment platforms at scale, he noted.

As part of that effort, Pivotal Software is aggressively incorporating tools that will enable organizations using its PaaS to implement best DevOps process. Many have adopted various modern development platforms in isolation but continue to rely on the Cloud Foundry PaaS to automate the building and deployment of mission-critical applications. Now, two classes of development platforms are merging into a single entity that makes it easier to embrace agile development methodologies using a lightweight development platform.

Whatever the outcome, PaaS environments are not going away anytime soon. However, what organizations perceive as the role a PaaS environment plays in any enterprise soon might expand considerably as more container technologies are infused into the platform.

— Mike Vizard

July 8, 2019
Clubhouse Aims to Simplify Software Project Management

Clubhouse Software today announced it is adding a Clubhouse Iterations module to its project management application to make it easier to plan sprints involving product and software engineering teams.

Mitch Wainer, chief marketing officer for Clubhouse Software, said Iterations will be included as part of its project management application service, which aims to bridge the divide that often exists between software development teams and the lines of business for which they are building applications.

Wainer said the company’s namesake project management software makes it easier for non-technical professionals to participate in the same project management workflow as a development team, rather than requiring them to become conversant in a project management application designed for software engineers, such as Jira. Clubhouse’s approach eliminates silos that get created when two teams within the same organization rely on different project management applications, he said.

Clubhouse Iterations goes a step further by surfacing what teams need to focus on as priorities and timetables change within a specific project, added Wainer.

Clubhouse Iterations allows users to define sprints using start and end dates, and the tool then sends reminders so the team can reserve time to focus on specific projects, he added.

Organizations that have successfully embraced DevOps almost invariably seek to extend those processes beyond the software development team. In theory, at least, the entire business should aspire to become more agile. However, the project management software many DevOps teams employ today is too complex and intimidating for the average business executive. Rather than forcing business users to learn how to master these applications, a case can be made for a more intuitive set of project management tools for both software development teams and the product managers they ultimately serve. Wainer said Clubhouse is already being used within 2,000 organizations to address that issue.

Wainer also noted the Clubhouse Software also has no bias as far as continuous integration/continuous delivery (CI/CD) platforms are concerned. Clubhouse is designed to integrate with multiple CI/CD platforms via application programming interfaces (APIs) to enable the development of custom workflows that include, for example, integration with real-time conversations occurring via the Slack service.

Next up, Wainer said Clubhouse intends to improve the search capabilities it provides on the platform and offer more sophisticated reporting tools.

In the meantime, the rise of DevOps would not be possible without the capabilities enabled by project management applications. But the agile nature of DevOps processes is forcing providers of project management software to make their applications both more dynamic and accessible. Those advances, in turn, make it easier to bring new members on board to the DevOps team.

Project management software may not always be viewed as the linchpin around which most software development processes are constructed. Without some form of project management application, however, most DevOps processes would have devolved into utter chaos a long time ago.

— Mike Vizard

June 25, 2019
Linus Torvalds Sees Lots of Hardware Headaches Ahead

Linux founder Linus Torvalds, today at the KubeCon + CloudNative + Open Source Summit China conference, warned attendees that managing software is about to become a lot more challenging, largely because of two hardware issues that are beyond the control of DevOps teams.

The first, he said, is the steady stream of patches being generated as new cybersecurity issues related to the speculative execution model that Intel and other processor vendors rely on to accelerate performance. That model is the root cause of malware such as Spectre and Meltdown that have roiled the IT industry. Additional bugs in speculative execution with colorful names such as Fallout and ZombieLoad are showing up more frequently. Each of those bugs requires another patch to the Linux kernel that, depending on when they arrive, can require painful updates to the kernel, Torvalds told conference attendees.

Short of disabling hyperthreading altogether to eliminate reliance on speculative execution, each patch requires organizations to update both the Linux kernel and the BIOS to ensure security. Turning off hyperthreading eliminates the patch management issue, but also reduces application performance by about 15 percent.

The second major issue hardware issue looms a little further over the horizon, Torvalds said. Moore’s Law has guaranteed a doubling of hardware performance every 18 months for decades. But as processor vendors approach the limits of Moore’s Law, many developers will need to reoptimize their code to continue achieving increased performance. In many cases, that requirement will be a shock to many development teams that have counted on those performance improvements to make up for inefficient coding processes, he said.

In the meantime, Torvalds noted updates to the Linux kernel are still coming at a rate of every three months, and the Linux team is basically working on a six-month planning cycle—there is no master five-year plan the Linux team is working from. Roughly 1,500 developers work on contributions to the Linux kernel, with 100 maintainers overseeing the implementation of those contributions.

Naturally, cybersecurity patches at the kernel level have significant implications for all of DevOps. Changes to the kernel need to be absorbed by all the various distributions of Linux, which in turn impacts all the stacks of software that depend on Linux. Jim Zemlin, executive director for The Linux Foundation, said that in the wake of the rise of these hardware issues and previous cybersecurity issues involving open source software such as the Heartbleed vulnerability, cybersecurity is the top priority for The Linux Foundation. As part of that effort, The Linux Foundation is researching various DevSecOps approaches to better securing the global open source supply chain, he said.

In the meantime, organizations large and small alike will need to up their DevSecOps game significantly if they want to continue to push application performance limits.

— Mike Vizard

June 25, 2019
Agile Stacks Extends Reach of DevOps Automation Platform

Agile Stacks this week announced it is extending the reach of its namesake DevOps automation platform to Kubernetes and additional cloud services.

Company CEO John Mathon said that by extending the reach of the company’s platform to include Google Cloud Platform (GCP), Microsoft Azure and Red Hat’s OpenShift distribution of Kubernetes alongside existing support for Amazon Web Services (AWS), Agile Stacks will make it easier for organizations to implement best DevOps practices across hybrid cloud computing environments.

Agile Stacks made the announcement at the HPE Discover 2019 conference. The company counts Hewlett-Packard Enterprise (HPE) among its investors, and at the conference this week, HPE this week outlined how Kubernetes is core to the company’s evolving hybrid cloud computing strategy.

Mathon said the Agile Stacks platform provides a framework for automating the creation of DevOps pipelines across multiple tools and continuous integration/continuous deployment (CI/CD) platforms. In all, there are more than 80 DevOps tools and platforms available in the Agile Stacks Service Catalog, including Kubeflow, Jenkins, GitLab, Spinnaker, Harbor, Minio, Ceph and Portworx.

At the core of the Agile Stacks platform is a SuperHub automation hub, which accelerates software delivery based using a set of custom composable stacks that can be configured by the DevOps team. Each component of a stack is auto-configured to deploy, upgrade, rollback, promote, clone, backup, generate test scripts, perform merges, conduct performance testing, scan for vulnerabilities and execute functional testing.

In effect, Mathon said Agile Stacks is essentially automating the DevOps automation. Organizations that have embraced best DevOps practices tend to rely on multiple tools from different vendors, which inevitably leads to silos of automation that often work at cross purposes. Agile Stacks provides an extensible GitOps framework for integrating all those tools that enable organization to evolve more easily as their DevOps processes mature, added Mathon.

In addition, he said Agile Stacks expects to play a role in making DevOps more generally accessible by reducing the complexity organizations initially encounter when trying to embrace DevOps practices.

As hybrid cloud computing starts to become more commonplace, thanks in part to the rise of Kubernetes, Mathon said DevOps complexity is only likely to increase. Organizations have made it clear they want to leverage multiple clouds alongside their existing on-premises IT environment. Right now, however, the DevOps processes associated with each of those platforms is, at best, fragmented. In addition to making it easier to unify DevOps processes across multiple platforms, Mathon noted Agile Stacks can also enable organizations to keep better track of cloud spending by applying automated cloud tagging via Superhub.

Of course, most organizations are challenged automating the development and delivery of software on a single platform. However, regardless of how far down the path any organization is toward DevOps, the business is moving forward. In many cases, the rise of hybrid cloud computing will finally force many organizations to implement a set of best DevOps process enterprisewide. As part of that process, organizations will look to standardize on a specific set of DevOps tools whenever possible, or at least until developers discover the latest DevOps tool to supplant all others.

— Mike Vizard

June 21, 2019
GitLab Expands Scope of DevOps Ambitions

GitLab, with the release of its 12.0 update, is extending the reach of its namesake continuous integration/continuous delivery (CI/CD) platform to include both cybersecurity professionals and traditional IT operations staff.

John Jeremiah, senior product marketing manager at GitLab, said the goal is to provide organizations with a single CI/CD platform that unifies the management of everything from source code to best DevSecOps processes.

To address that latter requirement, GitLab 12.0 adds a dashboard designed specifically for security personnel along with the ability for cybersecurity teams to approve merge requests and automatically apply patches for vulnerabilities detected. Those capabilities complement existing support for scanning and secrets detection.

New capabilities embedded in the platform for IT Operations professionals include an operations dashboard and an embedded incident management capability that makes it easier to create a coordinated response to a specific event.

GitLab 12.0 also adds support for feature flags to better manage the application delivery process, along with a platform-as-a-service (PaaS) environment to provision and maintain the infrastructure needed for developing and launching an application.

Jeremiah said the new and existing capabilities in GitLab will make it possible for organizations to unify their DevOps tool chains in a way that also enables IT operations teams and cybersecurity professionals to be included in the process.
At the same time, GitLab is making DevOps more accessible by eliminating many of the silos between various tools that have emerged over the years, including, for example, built-in support for feature flagging versus requiring organizations to acquire and maintain a separate tool, he noted.

Because those capabilities are embedded in the platform, organizations will be able to make the cultural shift required to adopt DevOps more easily, he added. IT teams don’t always know what other tools they will need as their DevOps skills mature. With all the tools embedded in GitLab readily available, it becomes easier for organizations to expand their DevOps processes as the organization starts to implement more sophisticated processes, Jeremiah said.

That approach is starting to resonate with the 100,000-plus developers actively using the GitLab platform and more than 2,000 contributors actively contributing code to the underlying open source platform, he added. GitLab in March saw its millionth merge request.

It’s too early to tell how broadly any CI/CD platform might be able to consolidate the DevOps toolchain. Developers are notorious for adopting new tools as they see fit. Internal IT teams, of course, are not always excited to be asked to support yet another tool. The right level of balance between ultimate flexibility and the need for a standard set of processes that can be implemented across multiple application development teams must be determined by each organization.

In the meantime, DevOps processes are starting to expand beyond traditional development team to now include IT operations staff and cybersecurity personnel. The challenge now that they’ve been invited to the proverbial DevOps party is making them feel welcome and appreciated.

— Mike Vizard

June 20, 2019