devopsy.kinsta.cloud

Author: Mike Vizard

IBM: DevOps Begins and Ends with Culture

Making the transition to DevOps requires as much a shift in cultural mindset as it does new platforms for building, deploying and managing applications. Between the two, however, it’s often the extent of the cultural challenges needed to be overcome that gets most underestimated.

Michael Elder, a IBM Distinguished Engineer, said one of the most important DevOps philosophies that organizations must embrace is that No Code is Sacred (NCIS). NCIS fosters an experiment-driven culture that encourages self-editing and promotes a “don’t be afraid to fail” mentality. This approach allows teams to form questions, quickly implement means to gather data, try alternatives and incorporate real user behavior and feedback, he said.

Manage the Scrum (Meeting)

Other cultural issues that need to be addressed include leadership and how daily scrum meetings are managed.

An overzealous build champion can wind up adding technologies to the continuous integration/continuous deployment (CI/CD) platform that don’t get adopted simply because the team as a group did not agree to their inclusion, said Elder.

Daily scrum meetings don’t necessarily need to take a full hour. Non-critical, time-consuming status updates should be eliminated from the agenda. Daily scrum is more about problem-solving and looping the wider team in on critical decisions than talking about updates most team members can already read about for themselves, Elder noted. The goal of the meeting is not necessarily to inform team members of events. They should be expected to know what’s occurred. The real goal of the meeting is to decide what to do next.

Mindsets Matter

Elder said DevOps teams be cognizant of Conway’s Law, which says that organizations that design systems are constrained to produce designs that are copies of the communication structures of these organizations. If the goal is to develop a more agile application environment by employing microservices, Elder said, then the organization should create smaller, more independent teams to construct and manage them. However, that doesn’t mean every monolithic application needs to turned into a set of discrete microservices.

Elder also advised organizations to embrace Design Thinking principles that encourage DevOps teams to focus on what needs to be achieved instead of enabling processes. The focus ultimately needs to be on the experience of the users who are being asked to employ an application, he said.

The Tools Issue

The biggest cultural challenge, however, still may be to tooling. Organizations trying to embrace DevOps clearly need new tools. But acquiring new tools won’t drive meaningful change. Too many organizations buy tools in the hopes that processes will eventually be transformed. The decision to acquire new tools needs to be closely connected with the transformation of processes, said Elder. That said some tools, such as an enterprise instance of Github, are core to encouraging the sharing and reviewing of code within a DevOps process, he added.

Of course, Elder noted the most critical cultural change as far as tooling is concerned is keeping an open mind. Developers and IT operations teams tend to get overly attached to specific programming languages and platforms. A true DevOps culture, above all else, encourages experimentation.

— Mike Vizard

March 19, 2018
NATS Messaging Entering Cloud Native Purview

The Cloud Native Computing Foundation (CNCF) has expanded the scope of its purview by adding open source NATS messaging software to an expanding list of technologies it now oversees.

NATS messaging software has been commonly employed in environments such as platform-as-a-service (PaaS) environments. But with the rise of microservices, a campaign to make NATS a part of the portfolio of technologies that members of CNCF consortium will advance has begun in earnest, said Derek Collison, who lead the development of NATS and is now CEO of Synadia Communications, which is dedicated to enabling organizations to more easily deploy and manage messaging systems based on NATS.

The ultimate goal, Collison said, is to not only make messaging ubiquitous, but also effectively invisible.

Collison said he and the CNCF envision a world where messaging just comes baked into every application. PaaS environments may have driven mainstream adoption of NATS messaging to integrate various elements of an application, but microservices will make messaging a fundamental requirement for building applications that can efficiently scale, he said, adding too many organizations already run into these issues because they are depending on HTTP to integrate various microservices.

NATS is already widely employed by a number of vendors and organizations including Apcera, Apporeto, Baidu, Bridgevine, Capital One, Clarifai, Cloud Foundry, Comcast, Ericsson, Faber, Fission, General Electric, Greta, HTC, Logimethods, Netlify, Pex, Pivotal, Platform9, Rapidloop, Samsung, Sendify, Sensay, StorageOS, VMware Acadiant, Weaveworks and Workiva.

The core NATS server and NAT Streaming software are maintained by Synadia. There are clients for Python, Ruby, Node.js, Elixir, Java, NGINX, C and C#, as well as third-party contributions that add support for Arduino, Rust, Lua, PHP, Perl and others.

Longer term, Collison sees NATS evolving to include functionality to support real-time applications, many of which today are using Apache Kafka software. Rather than having to deploy NATS and Kafka for different use cases, Collison said it will make more sense to have a common control plane to address multiple classes of use cases for messaging.

Messaging systems in one form or another have been at the heart of integration platforms for decades now. But the assumption was always that integration would be addressed after the applications were deployed. In the future, the assumption is that all cloud-native applications need to be integrated, which is much easier to accomplish if they all come with some form of messaging capability baked in. Integrating and managing federated instances of messaging systems still will be required. But much of the lower-level functions now being centrally processed will be pushed out into the applications being integrated.

Collison is the first to admit that messaging is not some new, emerging technology. But as microservices architectures continue, messaging will prove to be core to a new generation of cloud-native applications, he said. The CNCF is clearly betting that a consortium of vendors working on a common code base for messaging will accomplish that goal a whole lot sooner.

— Mike Vizard

March 16, 2018
Rundeck Gains Funding to Support Open Source IT Automation Framework

Rundeck has emerged as a popular open source job scheduler and orchestration option for IT automation that enables IT teams to build runbooks that can be easily shared with others. The company behind it—Rundeck Inc.—has garnered $3 million in seed funding to provide enterprise-class support for instances in the form of the Rundeck Operations as a Service Platform.

Company CEO Stephanie Fohn says the Rundeck framework has gained popularity because it provides an agentless approach to creating runbooks capable of spanning multiple platforms, tools and scripting languages. That approach doesn’t require that software be on each endpoint to be automated or that the IT operations team standardize on one set of tools and languages to start integrating various DevOps processes, said Fohn.

In fact, the goal is to reduce resistance to DevOps in enterprise IT organizations by making it easier for IT professionals to employ their existing skills to create runbooks that can shared and executed by others, said Fohn.

Rundeck Inc. claims there already are more than 40,000 IT professionals participating in the open source project, which is already being used by more than 100 commercial organizations.

IT automation frameworks are not a new idea, but resistance to change traditionally has been high. Many IT professionals have been developing their own scripts for years, which they consider to be intellectual property they take with them from job to job. Frameworks that require them to learn new tools or languages are not readily adopted unless the organization they work for mandates them. But the number of IT organizations that have embraced IT automation frameworks as a percentage of the whole remains less than 30 percent.

Fohn said commercial support provided by Rundeck Inc. should make more organizations more comfortable adopting an IT automation framework that extends the shell scripts many IT professionals have already developed.

In the meantime, pressure on IT operations teams to manage IT infrastructure more adroitly at scale only continues to mount. Organizations are deploying and updating more applications at a faster rate than ever. Add to that a mix of emerging microservices technologies based on containers, and it’s apparent the complexity of the IT environment will only increase in the months and years ahead. Fohn said that means it’s almost a sure thing organizations will embrace IT automation and DevOps.

Rundeck offers an alternative agentless approach to managing IT at scale with the benefits of being free and having a growing population of IT professionals that know how to employ it. The degree to which Rundeck can move beyond a grass-roots initiative to become a bona fide movement remains to be seen. But given the total percentage of organizations that have yet to embrace an IT automation framework, clearly there’s plenty of opportunity.

— Mike Vizard

March 14, 2018
OpsRamp Looks to Fuel IT Operations Consolidation

One of the biggest challenges associated with DevOps is cutting through all the various platforms and applications alerts to make sense of what’s occurring in an IT environment, much less manage it. In fact, there’s been a pressing need of consolidate management tools for IT operations for quite some time now.

OpsRamp is now making just that case with the unfurling of Unified Service Intelligence, a set of integrated IT management applications that are all delivered using a software-as-a-service (SaaS) model that makes it possible to unify multiple dashboards.

Deepak Jannu, director of marketing for OpsRamp, said that although there are more application programming interfaces (APIs) being exposed than ever, it’s still too difficult for IT operations teams to correlate data in a way that creates meaningful actionable intelligence. For the most part, IT operations teams are still swiveling between different consoles in the hopes that some insight will magically make itself apparent, he said.

To make it easier to separate the signal from the noise, Unified Service Intelligence combines discovery and monitoring health dashboards with incident management capabilities in single application layered with hundreds of best-practice monitoring templates to simplify IT administration.

Jannu noted many IT organizations struggle when transitioning to DevOps because they are confronted with a classic chicken-and-egg challenge. Most of them rely on legacy tools that are not integrated. Trying to modernize IT operations processes is simply too difficult without the right set of tools in place. The only way to make modern tools affordable for most organizations is to deliver them as a set of SaaS applications, he said.

As IT organizations find themselves embracing multiple clouds as well as emerging technologies such as containers, Jannu noted it’s now only a matter of time before IT operations reaches a breaking point inside most organizations. It was difficult enough trying to navigate different dashboards for IT infrastructure and applications running on-premises. The cloud, in most cases, just adds another series of platforms to an already-inflexible mix of isolated processes, he said.

Just as significantly, he added, an integrated platform also is key to cutting down IT operations fatigue. Most IT operations teams don’t discover critical issues early enough because it’s simply too difficult to understand what an “alert storm” being generated by disparate management tools signifies. After a while, the IT operations team becomes inured to all the alerts being generated.

Longer term, Jannu said OpsRamp will be better positioned to apply machine and deep learning algorithms to automate IT management than the average IT organization would be by running tools locally on its own. OpsRamp already generates 3 billion records of events a day, all of which can be used to inform artificial intelligence (AI) tools that require access to massive amounts of data to effectively train models. Locally installed IT management tools will never generate the required volume of data required to train an AI model.

It remains to be seen how fast the IT operations management space will consolidate. There are a lot of constituencies inside IT organizations that have a vested interest in one tool or another. That said, the writing on the wall heralding that consolidation already has been inscribed.

— Mike Vizard

March 13, 2018
Datical Helps Pull DBAs into Modern DevOps Age

As organizations move to embrace DevOps, some have inadvertently left database administrators (DBAs) behind. As a result, developers increasingly find themselves managing databases, as DBAs can’t keep up with the rate of change introduced by DevOps processes. To help narrow that growing gap between developers and DBAs, Datical created a namesake platform that applies continuous deployment/continuous integration (CI/CD) processes the management of databases.

An update to the Datical platform now adds real-time views and dashboards that illustrate the status of database deployments. DevOps teams now can simulate the impact of changes before they are deployed in a production environment and enforce rules to meet audit and compliance requirements. A new management tool also makes it easy to track database changes across the pipeline as well.

Pete Pickerill, vice president of product strategy for Datical, said Datical 5 now makes it possible to automatically apply the updated changes to a database without many of the steps that slow down application development on top of databases.

Finally, with this release, Dacital also provides a mechanism for centrally managing all the credentials attached to database.

While agile development methodologies have lead to an increase in the rate at which code is being written, database management continues to act as a drag on the rate at which applications are being deployed and updated. Datical decided to address that problem by creating a platform that specifically applies DevOps processes to the management of databases, said Pickerill.

Organizations that still rely largely on manual database processes often find the level of tension between developers and database administrators to be high. Developers often will simply opt for a database they can manage themselves rather than engage a DBA. That can result in suboptimal choices being made for different classes of application workloads that become apparent once the application starts to scale. The challenge and opportunity is to put tools in the hands of DBAs that enable them to respond rapidly to changing application dynamics, said Pickerill.

Of course, in many instances developers have simply taken over the database administration task. But every minute spent on those tasks is one less minute a developer is writing code. The more efficient option is to find a way to integrate DBAs into the overall DevOps process. The effort requires both providing DBAs with access to new tools as well as changing an IT culture where many DBAs assumed the application development strategy revolved around their ability to find time to manage a database. Today, DBAs are now being asked to manage a multitude of types of databases that are being deployed in unprecedented numbers. Clearly, processes rooted in approaches to database management developed a decade ago or longer no longer suffice.

Not every DBA will make that transition. But at this juncture it’s clear that fundamental change to the way databases are managed is now a matter of when rather than if.

— Mike Vizard

March 12, 2018
Sauce Labs Adds Analytics to Continuous Testing Platform

Sauce Labs has added additional debugging and analytics capabilities to its testing platform to make it easier to manage continuous testing across multiple application development projects.

Lubos Parobek, vice president of product at Sauce Labs, said DevOps teams need to prioritize which bugs to address first across multiple development projects occurring simultaneously. In fact, according to the latest annual survey of 1,091 technology professionals conducted by Dimensional Research on behalf of Sauce Labs, DevOps teams on average are now taking longer to address bugs. These days there are many more issues that require debugging across multiple development projects, which is overwhelming DevOps teams, who must then prioritize some debugging requirements over others.

Specifically, the survey found only 21 percent of respondents report they fix bugs immediately, down from 23 percent in 2017.

Analytics capabilities included in Sauce Labs’ rechristened Continuous Testing Cloud include the ability to view test results by browser and operating system, visibility into pass/fail and error trends and the ability to see how a single test behaves over time so DevOps teams can more easily identify tests inconsistent with overall test run time and pass/fail rates.

Scheduled to be available in the second quarter, the Continuous Testing Cloud makes it possible to more easily reconcile advances in the rate at which applications are being developed and constrained testing and debugging tools, Parobek said.

The Sauce Labs survey results also show progress being made in the realm of mobile application testing. A full 77 percent of respondents said they now use a mix of both real devices and simulators or emulators for mobile testing, up from 34 percent in 2015. That approach results in higher fidelity to the actual experience of an end user on each device, said Parobek.

That shift, however, may also explain why there is more manual testing. More than half (56 percent) said they do as much or more automated as manual testing, a minor decrease from 59 percent (2017) and 60 percent (2016), the report found. Just under half (45 percent) said they do, however, expect to increase spending on test automation in 2018.

Parobek said organizations are moving into what he describes as the third era of testing. The first two eras were manual testing followed by a shift to automated testing. The third era is marked by continuous testing that leverages automation to make it possible to test code as it’s being developed rather than after, when the application is complete.

Naturally, each organization is not as far down the path toward continuous testing as every other. Parobek said for the most part, early adopters of DevOps practices employ continuous testing more widely than traditional enterprise IT organizations that are still in the early stages of embracing DevOps. The good news is the latter have an opportunity to leverage a much larger body of DevOps best practices than the organizations that originally pioneered the concepts.

— Mike Vizard

March 8, 2018
VMware Adds More DevOps Depth to Cloud Strategy

VMware today added a bevy of services to its portfolio that make managing instances of its software running in a public cloud easier.

Additions to the VMware cloud portfolio include VMware Hybrid Cloud Extension Service for Private Cloud, a software-as-a-service (SaaS) offering that enables IT organizations to employ a self-service portal to move workloads between public clouds in a way the minimizes downtime.

Other additions include the ability to apply both the Wavefront by VMware monitoring service and VMware Cost Insight Service against VMware Cloud on AWS, in addition to other public clouds. VMware has also added 45 integrations to expand the number of processes this SaaS application can monitor.

Finally, VMware has added a VMware Log Intelligence Service to provides access to real-time analytics of log data.

The arrival of these services coincides with the availability of VMware Cloud for AWS in the London region of AWS, which will be followed by the VMware stack becoming available in the Frankfurt, Germany, region of the AWS cloud.

Chris Wolf, vice president and CTO of global field and industry for VMware, said that as more organizations look to move out of data centers they operate, they are asking VMware to extend the mobility capabilities the company developed for its hypervisor into the public cloud. In fact, many of them are now moving to deploy VMware software across multiple clouds, he said.

Naturally, that expansion of VMware beyond the local data center is creating a raft of DevOps challenges, many of which the company is trying to address by making available a series of SaaS applications under the moniker VMware Cloud Services.

VMware is making a concerted effort to expand its DevOps portfolio at a time when many developers have shifted application workloads onto public clouds based on virtual machines other than VMware. Obviously, VMware has developed many partnerships with cloud service providers. But the number of VMware hypervisors running in public clouds remains a relatively small percentage of the total number of hypervisors. VMware is hoping that will change, however, once developers discover VMware hypervisors on public clouds, which in many cases can reduce the headaches associated with trying to manage hybrid clouds based on disparate hypervisors. As part of that overall strategy, VMware also has been aggressively embracing emerging cloud-native platforms such as Kubernetes that can run anywhere.

Most enterprise IT organizations are still learning to walk when it comes to hybrid cloud computing. The most common use case is still disaster recovery. Most IT organizations are a long way from dynamically moving workloads between clouds. Most of the use cases for moving workloads initially will be confined to simply lifting a workload from an on-premises environment. But as IT organizations also gain more experience, soon there may be as many workloads moving between clouds as there are workloads moving between servers in a local data center.

— Mike Vizard

March 7, 2018
CA Technologies Advances DevOps Automation Agenda

CA Technologies further advanced its staged goal of unifying its disparate portfolio of DevOps automation tools by announcing today a new release of CA Workload Automation AE (AutoSys) software now directly integrated with the CA Automic One Automation platform the company acquired early last year.

CA AutoSys is the older of two offerings and is positioned by CA Technologies as being able to support more complex workloads running on a larger number of platforms, including mainframes. CA Automic One Automation was acquired to bolster the automation capabilities CA can offer optimized for distributed computing environments.

Gwyn Clay, vice president of product management for CA Automic, said it’s no secret CA Technologies has multiple automation engines. The long-term goal is to enable organizations to employ any mix of automation engines and associated DevOps tools as they see fit to create what the company describes a software factory.

Other new capabilities being added to CA AutoSys include the ability to centrally manage all the various agents the company provides, support for connecting to Amazon Web Services (AWS) and Azure Database Services, improved load balancing and the ability to set up more custom email notifications.

Clay noted the rate at which IT organizations are embracing DevOps varies widely. Just about every organization has at least one DevOps project or pilot in place, but most of them are still relatively nascent compared to the level of sophistication being applied by web-scale companies, he said.

CA Technologies is betting that, rather than stitching together components themselves, enterprise IT organizations will prefer a more modular platform that requires less effort to set up without locking them into a specific tool.

DevOps is coming to the enterprise in one form or another. But unlike web-scale companies, where developers and engineers have tended to work closely together from the inception of the company, enterprise IT organizations will navigate myriad culture issues. There has been a historic predisposition in enterprise IT organizations for developers to “throw code over the wall” to IT operations, with little regard for how that code might run in the actual production environment. IT operations teams, meanwhile, have not always been as forthcoming about the state of that production environment as they might be. That lack of trust and visibility means both sides often make decisions that are not always in the best interest of the overall organization. For example, an application workload might wind up being deployed on a public cloud, even though that platform may be both more expensive and less secure.

At the same time, it’s still not clear to what degree warring cultures inside the enterprise will be able to put aside their historic animosities to create a coordinated approach to automation and DevOps. Mainframe and distributed computing teams to this day still battle for control over workloads inside the enterprise. Whatever the outcome, there’s no doubt various teams will eventually be forced to work more closely together to achieve DevOps goals regardless of whether they like it.

— Mike Vizard

March 6, 2018
SAP Cloud Price Changes Reflect DevOps Realities

To foster more application development in the DevOps era, SAP at the Mobile World Congress 2018 conference announced it is adding a consumption-based pricing option to its SAP Cloud Platform environment.

Based mainly on the Cloud Foundry platform-as-a-service (PaaS) overseen by the Cloud Foundry Foundation, the SAP Cloud Platform provides a way for SAP customers to build custom applications that can be more easily integrated with SAP applications delivered as a cloud service using a standard set of application programming interfaces (APIs). The basic idea is to enable organizations to innovate in a way that doesn’t impact their applications every time SAP, for example, updates the SAP S/4 HANA application portfolio, said Dan Lahl, vice president of product marketing at SAP.

SAP Cloud Platform currently is made available via a set of services an organization subscribes to for a defined amount of time. Under the consumption approach, an organization buys credits that can be applied within a specific amount of time. That makes it more economical for organizations to start multiple custom projects without having to know precisely upfront what services will be employed on the SAP Cloud Platform, Lahl said.

In effect, SAP is creating a licensing model more in keeping with how developers consume services in the age of DevOps, he noted.

SAP also announced an enhanced SAP Cloud Platform SDK for iOS, providing tighter integration with the Apple Xcode integrated development environment (IDE) and additional analytics controls that can be embedded within an iOS application. SAP has also made it easier to consume SAP Leonardo services for building internet of things (IoT) applications using APIs that can be accessed via the SAP API Business Hub.

Lahl said in the coming months SAP also will share with customers best DevOps practices it follows to develop and maintain cloud applications. SAP is now seeing traditional IT organizations now starting to more aggressively adopt DevOps processes, he said, primarily because DevOps processes enable IT organizations to show a proof-of-concept (PoC) to line of business executives much faster. That, in turn, also serves to condense the feedback loop between business units and the internal IT team.

Those projects now span multiple classes of runtimes, including SAP ABAP, Java and Docker containers—all three of which are being employed to create microservices. The challenge many organizations now face is determining when to build a microservice versus invoking a function via an API that already exists in an ERP application. Application developers don’t often have visibility into an existing function, which is why they run into resistance from business executives whenever they attempt to reinvent a wheel that already exists, Lahl said. The challengeis making it easier to discover the APIs that expose functions within an existing monolithic application, especially when it doesn’t make sense to reconstruct a general ledger function as a microservice.

Whatever the ultimate approach employed, the one thing that is clear is that costs associated with either building or consuming a service keep coming down in the age of DevOps.

— Mike Vizard

February 27, 2018
Chef Advances Compliance Automation

Chef today expanded the reach of its InSpec platform for automating compliance management with an update that adds support for Amazon Web Services (AWS) and Microsoft Azure public clouds, as well as integration with additional third-party tools. In total, InSpec 2.0 adds more than 30 additional capabilities, including support for Docker containers, Microsoft IIS and NGINX web server and PostgreSQL databases software.

In addition, InSpec results now can be exported as JUnit format for integration into continuous delivery tools such as Jenkins. Compliance profiles can now also be pulled from Chef Automate.

Plus, InSpec 2.0 runs 90 percent faster than InSpec 1.0 on Windows and 30 percent faster on Linux, according to Chef.

Julian Dunn, director of product marketing for Chef, said the primary goal of InSpec is to enable DevSecOps teams to address compliance issues before an application gets deployed in a production environment. It’s not feasible for any DevSecOps teams to remember all the compliance requirements that must be applied when any piece of software or hardware should be configured, Dunn said.

InSpec provides a mechanism to automate compliance management using a set of declarative tools that don’t require anyone on the IT team to possess programming skills. Dunn conceded it’s still early on in terms of incorporating compliance into any DevSecOps initiative, most of which are nascent efforts themselves. But it’s now only a matter of time before compliance shifts left along with security, Dunn noted.

Security becomes even more pronounced in the age of the cloud because developers are now routinely deploying workloads on top of infrastructure that is not often managed by an internal IT operations teams, Dunn said. In fact, there have been a recent raft of security breaches stemming from the fact that data was left exposed on a public cloud such as AWS.

There’s always been a fierce debate concerning the degree to which compliance encourages security. Advocates say that without some basic compliance requirement, the level of security inside most organizations would be more abject than it already is. Others contend that compliance requirements only encourage organizations to implement the bare minimum of security required; thereby creating a false sense of security that cybercriminals easily exploit. Regardless of viewpoint, however, compliance requirements are not going away anytime soon. In fact, major security breaches usually are not accompanied by hefty fines that are based on the number of compliance requirements that were ignored.

Developers don’t always appreciate all the nuances of compliance when under pressure to deploy and update applications faster than ever. Organizations today need to find ways to enforce compliance mandates without slowing down the rate at which applications are developed. That’s simply going to impossible to achieve without increased reliance on automation. Today, unfortunately, most of what passes for compliance processes don’t begin until long after an application is deployed in production, which makes fixing those oversights once discovered a much more expensive proposition than if they were discovered much earlier in the application development life cycle.

— Mike Vizard

February 20, 2018