devopsy.kinsta.cloud

Author: Mike Vizard

CA Technologies Acquires Runscope for API Monitoring

There’s a pressing need for a variety of monitoring tools more than ever. But given the proliferation of application programming interfaces (APIs) inside the enterprise, demand for API monitoring tools in particular is rapidly expanding. To address that requirement, CA Technologies has acquired Runscope, a provider of an API monitoring service delivered as a software-as-a-service (SaaS) application.

Ali Siddiqui, general manager for agile operations at CA Technologies, says Runscope will complement CA Blazemeter continuous testing tools the company acquired last year and the CA API Management platform based on the technology CA Technologies gained when it acquired Layer 7 Technologies in 2013.

While the API Economy has been a phenomenon for external-facing web and mobile applications for years now, Siddiqui notes that IT organizations have only begun to rely on APIs to ingrate internal applications in recent years. The number of private internal APIs that have been published already dwarf the number of external public APIs. Because those internal APIs have now reached a critical mass, Siddiqui says the time is now right for CA Technologies to add an API monitoring service to portfolio of DevOps tools.

He notes that the rise of microservices inside enterprise IT organizations also will exacerbate existing API monitoring and management challenges. Each microservice generates its own API, which needs to be maintained and monitored. Before too long IT organizations will find themselves trying to unravel the interdependencies of thousands of microservices that all expose an API that can be developed in any number of programming languages.

These days it’s not uncommon to see dedicated management teams focusing on specific APIs. Rather than treating the API as an afterthought, many IT organizations have come to realize it’s the quality and robustness of the API that ultimately determines the quality of the application experience. In fact, most service level agreements (SLAs) going forward will require access to data being generated by APIs to make sure the terms of any agreement are being met. As such, it’s starting to make sense in some organizations to organize DevOps teams around critical APIs, rather than specific elements of application logic that, in the age of the containers, will soon be more frequently updated.

There’s no shortage of API monitoring tools these days, and Siggiqui says CA Technologies is committed to making sure the Runscope service remains open in terms of integration with other API management and testing tools. However, eventually there will be incentives provided to organizations that choose to standardize on multiple DevOps tools from CA Technologies.

Despite the proliferation of monitoring tools, the percentage of application workloads that are instrumented remains relatively small. The reason for this is that the cost of monitoring every enterprise application has often proven prohibitive. The issue that enterprise IT organizations will have to confront as dependencies between microservices increase will be figuring out how to implement instrumentation across all those services in a way that doesn’t wind up breaking the budget.

— Mike Vizard

October 4, 2017
Oracle on the March Toward Automated CI/CD Services

At its OpenWorld 2017 conference today, Oracle charted a course forward that essentially creates a managed DevOps platform spanning a broad number of application types and styles.

The Oracle Application Development Platform is a subset of the Oracle Cloud Platform optimized for application development. Siddhartha Agarwal, vice president of product management and strategy, says the company is working to automate every aspect of DevOps with an eye toward making it possible for organizations to concentrate all their efforts on developing applications versus managing them.

The latest step in achieving that goal are updates to the Oracle Application Development Platform, which now incorporates the Oracle Container Native Application Development Platform based on the open-source Kubernetes container orchestration engine, and Oracle Container Engines, a container pipeline management service based on the Wercker software the company acquired earlier this year. The company also announced it is adding support for the Go and Java Enterprise Edition programming languages to existing support for Java Standard Edition, Node.js, PHP Ruby and Python runtimes.

Other options being added to the Oracle Application Development Platform include Fn.a, an open source serverless computing project based on software the company gained when it acquired Iron.io, as well as an implementation of the Open Service Broker that enables applications developed for the Cloud Foundry platform-as-a-service (PaaS) environment to be deployed on Oracle Cloud. The company is also making available an instance of a BOSH Cloud Provider Interface to automate the deployment of those Cloud Foundry workloads.

Meanwhile, the Oracle Data Hub Cloud Service now also supports the open-source Cassandra database alongside existing support for MongoDB, Postgres and other databases to be added in the future. However, the core Oracle database remains the primary data engine service. In fact, Oracle Database 18c, announced earlier in the week, employs machine learning algorithms to manage instances of the company’s relational database autonomously with no human intervention required.

Finally, the company is adding a Blockchain Cloud Service that allows developers to access an implementation of open-source Hyperledger blockchain software, as well as an intelligent chatbot platform that can incorporate in mobile computing applications.

Agarwal says his company is committed to making available both commercial and open-source platforms available via an Oracle Cloud Platform that provides access to x86, Sparc and graphical processing units (GPUs). Agarwal says the company expects IT organizations to mix and match back-end services as they see fit. But increasingly all those back-end services will be managed under the auspices of a fully managed continuous integration/continuous development (CI/CD) platform.
Long-term, Agarwal says there may even come a day when application workloads are able to self-describe themselves, which would then enable the company to deploy them automatically on the best-suited back end.

Naturally, it may be a while before the level of DevOps nirvana gets achieved. But as machine learning algorithms and other forms of artificial intelligence (AI) advance, it’s clear that the weight IT organizations are expected to carry is increasingly shifting more toward building applications than managing the underlying infrastructure on which they run.

— Mike Vizard

October 2, 2017
Using AI to Automate Incident Management Sparks Debate

When it comes to incident response with an integrated DevOps environment so much depends on the ability of an IT team to respond quickly and consistently to a specific set of events. Because of that requirement, many IT organizations have invested in incident management systems that everyone in IT can consult whenever an issue arises. Now PagerDuty, a provider of incident management software, is taking that concept a step further by applying artificial intelligence (AI) to automate as many incident response tasks as an organization deems appropriate.

As promising as that may sound, however, Resilient Systems, a unit of IBM that is an arch-rival to PagerDuty, contends most organizations today do not have anywhere near the level of incident management maturity to make investing in AI worth their time and effort yet.

Eric Sigler, head of DevOps for PagerDuty, says the the latest edition of the company’s namesake incident management platform is designed to augment the capabilities of the local IT administrator by providing:

Alert Grouping: Rules-based automation and machine learning algorithms automatically group related issues together to make it simpler to identify related issues.

Similar Incidents: IT administrators and support specialist can see previous related issues and surface information around incident severity, impact and remediation steps.

Response Automation: A Response Plays capability allows teams to design and execute an automated response pattern to recruiting responders and stakeholders by a single click.

Dynamic Notifications and Event Routing: IT teams can select notification and assignment behavior and automatically route events to different teams based on event payloads.

Sigler says PagerDuty has also redesigned the company’s user interface, creating an enhanced live incident details page that makes discovering events simpler.

Those capabilities, says Sigler, will collectively prove to be critical in an emergency because most of the time the specialist associated with a specific IT discipline is usually not immediately available when needed. First responders to any given event need as much information as possible as their fingertips to resolve issues as quickly as possible.

But Ted Julien, vice president of product management for Resilient Systems, says most organizations simply don’t have incident response processes in place that would benefit from that level of AI. Machine learning algorithms require access to massive amounts of incident response data to learn what to do, but most organizations don’t collect that data, much less learn from it, he says. That doesn’t mean AI doesn’t have potential. But he believes AI is not nearly as far along when it comes to incident response as anyone would like. In fact, he notes, machine learning algorithms and other forms of IT automation could just as easily wind up doing more harm than good.

There may come a day when incident response in an IT environment is fully automated. But the distance between where most IT organizations are today and the achievement of that goal is still nothing less than considerable.

— Mike Vizard

October 2, 2017
LogicMonitor Extends Reach to the Cloud

LogicMonitor has been making available performance monitoring tools as software-as-a-service (SaaS) applications for several years now. At the Microsoft Ignite conference this week, the company announced it is adding a module through which IT organizations can gain similar insights into public cloud computing services.

Steve Francis, founder and chief evangelist for LogicMonitor, says LM Cloud is designed to provide real-time visibility into the performance of Amazon Web Services (AWS) and Microsoft Azure clouds, including resource performance monitoring, detailed billing analysis and cloud provider availability.

Many IT organizations don’t realize just how different managing a public cloud is compared to an on-premises IT infrastructure environment, he says. Not only is too easy to let a workload waste resources, spiraling cloud service costs out of control, but also cloud service providers have policies that limit consumption of resources in a specific region.

Capabilities provided by LM Cloud include automated discovery of an organization’s entire cloud environment and dashboards that can be employed to troubleshoot issues.

Francis says that as an extension of the core LogicMonitor service, LM Cloud provides a means for IT organizations to employ one monitoring tool across what have become de facto hybrid cloud computing environments. In fact, using multiple clouds will require IT organizations to rely more on monitoring tools based in the cloud. Otherwise, Francis notes, IT organization would have to correlate data across multiple incompatible tools.

One of the things that holds IT organizations back from using public clouds is simply that they are more familiar with their on-premises environments. Francis says tools such as LM Cloud will make it much simpler for IT administrators to explore a public cloud environment using a set of capabilities that cloud service providers are not inclined to provide.

Thanks to the rise of public clouds, DevOps arguably has never been more complicated, with applications distributed across multiple public clouds. But it’s very difficult to manage what can’t be seen, and it’s nearly impossible to compare one public cloud with another—much less an on-premises IT environment—without access to a common baseline of metrics. All too often deciding where to deploy a workload is based on either instinct or corporate fiat rather than detailed analysis.

While many IT organizations expect to reduce costs by moving workloads to the public cloud, the truth is more nuanced. Long-running applications tend to be less expensive to deploy on-premises. Shorter-run applications lend themselves well to public clouds. Yet, many IT organizations are simply not fast enough when it comes to deploying applications on-premises. Internal IT organizations that have embraced DevOps processes, however, are now more agile. That, coupled with software-defined infrastructure, is narrowing the dexterity gulf that exists between on-premises and cloud computing environments.

All things may not be equal just yet. But the day when an on-premises IT environment is just another cloud that needs to be managed is that not that far off.

— Mike Vizard

September 29, 2017
OutSystems: Low-Code Apps Require Sound DevOps Processes

As more IT organizations embrace low-code application development platforms, many of them are discovering that accelerating the rate at which they develop applications can be too much of a good thing.

António Barros, product manager for OutSystems, says most of the organizations that have adopted the company’s namesake low-code platform are just now coming up to speed on automated testing. Barros estimates that only 20 percent of the organizations using the OutSystems platform are also employing automated testing tools. The majority of OutSystems customers continue to rely on manual testing even though it’s apparent low-code platforms will produce applications much faster than manual testing can keep pace with. OutSystems has identified five stage of DevOps maturity, he notes, but none of its customers has achieved that level of DevOps maturity.

To help them get a little further down the DevOps path, OutSystems, at its recent NextStep 2017 conference, announced an extension of its partnership with Indigo to provide tighter integration with the OutSystems platform. In addition, OutSystems plans to make available digital transformation playbooks, including one for DevOps, beginning in 2018. Outsystems has already integrated its platform with the Jenkins automated server to provide continuous integration and continuous development (CI/CD) capabilities.

Barros says it’s not only critical to engage in continuous testing, but the application needs to be designed to make it easier to isolate user interface, application programming interface (API) and unit testing. Otherwise, automating the testing process becomes more difficult than it should be, says Barros.

Developers for years have resisted low-code platforms, which automatically generate much of the code that needs to be written for an application. Some fear the platforms can’t build applications that scale. Others have simply decided they prefer to write code themselves. But these days, organizations are under more pressure than ever to deliver applications faster, which has led many organizations to rely more on low-code platforms such as OutSystems.

It’s not clear just how popular low-code platforms will become. OutSystems revealed at its conference it has signed up 122 new customers in the last quarter alone. On average, it now takes organizations eight weeks to build and deploy an application on the OutSystems platform, with new release cycle updates occurring every 48 hours, says OutSystems CEO Paulo Rosado.

Michael Facemire, an industry analyst with Forrester Research, told conference attendees that adoption of low-code platforms is now a matter of economics. The starting salary for a developer in the San Francisco Bay area is $145,000, says Facemire. As the salary of a developer with a few years of experience now reaches more than $200,000 in the Bay area, companies in that geography will have no other option but to employ low-code platforms, he says.

Obviously, developers might be less expensive in other regions. But Facemire notes that no matter what the region, developers are in short supply. That need for speed, he adds, is also why organizations are rapidly adopting containers and serverless computing framework. The challenge IT organizations will face is finding a way to put quality control processes in place to don’t compromise either the speed of the development process or the quality of the end user experience, he says.

By the same token, however, no one really appreciates code that arrives late, no matter how great it might be.

— Mike Vizard

September 26, 2017
New Relic Delivers on AI Promise to Automate IT Operations

New Relic at a FutureStack 2017 conference unveiled an instance of its monitoring tools that employs machine learning technologies and other forms of artificial intelligence (AI) to provide advice on how to optimize an IT environment.

Nadya Duke Boone, director of product management for New Relic, says New Relic Applied Intelligence (NRAI) will be made available as part of the company’s Digital Intelligence Platform. In addition, New Relic is making available three dedicated NRAI capabilities in the form of Radar, NRQL Baseline Alerting and New Relic APM Error Profiles services. These services are designed to help IT operations teams spot abnormal behavior across billions of pieces of data and predict where problems may arise. The services ultimately will surface prescriptive recommendations for problems before they occur.

The goal, says Duke Boone, is to leverage New Relic’s software-as-a-service (SaaS) approach to IT monitoring tools democratize access to AI technologies. That’s possible, she says, because of the massive amount of data New Relic collects. That data is then fed into machine learning algorithms to detect patterns indicative of a variety of potential issues.

Duke Boone says that intelligence is then delivered as a personalized Radar news feed that is similar to a social media news feed. That news alert might, for example, might inform an IT administrator that, based on characteristics of their IT environment, an issue is likely to arise. IT operations teams also have the option to integrate that Radar news feed with a collaboration platform such as Slack to alert the rest of the IT operations team to a potential issue.

NRQL Baseline Alerting allows IT operations team to set thresholds for specific applications that when exceeded will generate an alert. New Relic APM Error Profiles allows IT administrators to launch queries against NRAI to detect potential issues that might arise from outlier applications in the IT environment, adds Duke Boone.

New Relic doesn’t see AI replacing IT operations teams as much as augmenting them, Duke Boone notes. Most organizations can’t hire enough IT staff to manage their IT environment. As those IT environments become more complex, it’s becoming clear that IT professionals will need to leverage AI technologies to keep all the applications they deployed running optimally, she says.

New Relic has been investing extensively in AI ever since it previewed these capabilities in the form of a Project Seymour initiative it revealed last year. Now the company and its rivals are locked in an arms race to develop and implement machine learning algorithms that should enable IT operations teams to manage applications at unprecedented levels of scale.

In some quarters AI is viewed as an existential threat to the IT operations teams. In reality, AI will be used to automatically resolve low-level issues that collectively conspire to rob IT organizations of the time they need to focus on new innovative ideas that add value to the business. In fact, assuming many of those issues will be soon disappear it’s conceivable that all the drudgery associated with managing IT today made soon just fade away.

— Mike Vizard

September 14, 2017
New Relic Tracing Software Pinpoints Source of IT Issues

At a FutureStack 2017 conference this week, New Relic previewed an implementation of distributed tracing software that has the potential to close the war rooms where IT operations assign blame for application failure.

New Relic CEO Lew Cirne says a Distributed Tracing capability that New Relic previewed this week will enable developers or IT operations teams to pinpoint the precise source of any given performance issue. Rather than wasting hours on root cause analysis, Cirne says DevOps teams will now be armed with dashboard they can navigate to visually inspect what elements of a distributed computing environment are specifically affecting their applications.

Based on the open-source Open Tracing software being advanced by the Cloud Native Computing Foundation (CNCF), Cirne says New Relic opted to embrace an open initiative that should lead to more actionable intelligence being derived from the data New Relic collects.

At this juncture, Cirne contends that deploying an application in production that has not been instrumented borders on the criminal. He concedes, however, the preponderance of applications deployed in production environments today are not instrumented largely because the total cost of instrumenting legacy applications has been cost-prohibitive. But as IT monitoring software becomes less expensive to invoke using agents that communicate back to a software-as-a-service (SaaS) application, most new applications being deployed in the cloud are now being instrumented.

To help further that goal, New Relic this week also announced support for .Net Core 2.0 framework recently made available on Microsoft Azure and added support for six additional Amazon Web Services (AWS) services. New Relic now supports a total of 25 AWS services.

At the conference Cirne also showed how IT organizations can now invoke New Relic application programming interfaces (APIs) to spin up a dashboard to track specific events associated with, for example, the rollout of a new application.

Access to the data New Relic surfaces, notes Cirne, in turn helps fuel adoption of DevOps because organizations gain confidence in their ability to more rapidly build and deploy applications. That confidence will only increase as IT organizations avail themselves of artificial intelligence (AI) software in the form of machine learning and, soon, deep learning algorithms to make more reliable decisions based on patterns in their IT environments. New Relic, he adds, is able to fuel those algorithms because the service collects 1.5 billion metrics and events every minute. That said, humans will still be required to manage the overall IT environment.

Cirne maintains that the best way to collect all that data is to employ agents, noting New Relic is adopting those agents to make them lightweight enough to even deploy on microservices based on containers.

New Relic naturally is not the only provider of IT monitoring tools investing heavily in advanced algorithms. In the not-too-distant future, it’s probable that algorithms deployed by different vendors will share insights with one another. In the meantime, IT administrators can take some comfort in the fact that it should soon become easier to manage IT at scale.

— Mike Vizard

September 14, 2017
VMware Unveils IT Automation Tools Suite
VMware over the last two years has been making steady advances in automating the management of IT. At the VMworld Europe 2017 conference VMware unveiled a vRealize Suite 2017 offering that combines the latest releases of vRealize Operations, vRealize Automation, vRealize Business for Cloud and vRealize Log Insight software into a single offering designed to optimize lifecycle management across a software-defined data center (SDDC).

Mahesh Kumar, senior director of product marketing for cloud management at VMware, says the driving force behind the development of vRealize Suite 2017 is to enable the automation of IT operations in support of an IT organization’s larger DevOps strategy.

The elements of VMware vRealize have been upgraded as follows:
- vRealize Operations 6.6 adds tools to help organizations analyze workload placement decisions to automate the balancing of workloads across clusters and data stores. It includes a predictive distributed resource scheduler (pDRS) and native VMware vSAN management and monitoring capabilities.
- vRealize Automation 7.3 adds support for Admiral 1.1, the lightweight container management portal that VMware previously announced. Virtual Container Host instances generated by VMware vSphere Integrated Containers, Docker hosts, and for Docker volumes can employ vRealize Automation to create and attach volumes to containers. It also delivers increased support for VMware NSX network virtualization software.
- vRealize Business for Cloud 7.3 is now being exposed as a tab in vRealize Operations 6.6 to better drive cost efficiencies by making operational and cost metrics readily available. It also delivers improved insights into the costs of Amazon Web Services (AWS) and Microsoft Azure public clouds alongside VMware-based private clouds.
- vRealize Log Insight 4.5 now provides integration with vRealize Operations to provide root-cause analysis of logs and metrics.
Resistance to IT automation continues to decline. Many IT operations professionals naturally are concerned that IT automation will lead to unemployment. But managing IT operations at the present level of scale required simply can’t be accomplished using manual processes. In addition, developers are voting with their code to employ public clouds that have already been automated. IT teams who want to make a case of continuing to deploy application workloads in a local data center need to be able to provide a cloud-like experience, which can be accomplished only by employing automation. Kumar notes that VMware is trying to arm IT administrators with the analytics required to make a financial case for continuing to deploy applications on instances of VMware running on-premises or in a public cloud.

Most IT organizations today are focused on automating disparate deployments of IT infrastructure and software. While that represents an advance over manual processes, many of those same organizations now are wrestling with various islands of automation. VMware is trying to unify automation across all the compute, storage and networking technologies it delivers to fashion an SDDC that enables IT organizations to be more agile. Longer term, however, IT organizations that employ stacks of software other than VMware will need to find a way to automate DevOps across heterogeneous IT environments.

— Mike Vizard
September 14, 2017
CloudBees Applies Analytics to DevOps

At the heart of many DevOps initiatives is an automation server based on open-source Jenkins software developed by CloudBees. At the recent Jenkins World 2017 conference, CloudBees showed that Jenkins is now becoming the foundational element through which DevOps teams can now manage a continuous integration/continuous deployment (CI/CD) process end to end.

New additions to the portfolio include CloudBees DevOptics, an analytics application to aggregate data from software development pipelines as it moves through a Jenkins server. CloudBees CTO Kohsuke Kawaguchi says once organizations are armed with those insights they will be able to, for example, identify potential bottlenecks in their software development processes. CloudBees DevOptics creates a single point of reference on the status of any application delivery project, all the way down to individual commits, says Kawaguchi.

CloudBees also launched CloudBees Jenkins Advisor, a free plug-in for a Jenkins environment that Kawaguchi says provides another watchful eye capable of continuously analyzing a Jenkins environment using algorithms the company developed. Cloudbess Jenkins Advisor comprises a knowledge base and diagnostic tools that can be employed to detect and resolve issues ranging from simple configuration issues to security and other forms of best practices.

Finally, the company revamped the user interface provided with CloudBees Jenkins Enterprise, an implementation of Jenkins that comes pre-configured with built-in security and integrations with more than 1,400 DevOps tools. It also announced that a few select managed services providers are now building a practice around Cloudbees Jenkins Enterprise.

CloudBees reports there are now more than 150,000 active installations of Jenkins worldwide that span a user community of more than a million. Those numbers represent a 50 percent growth rate from January of 2015.

Most organizations employ DevOps processes to accelerate development of software. But along the way it’s not uncommon for many organizations to lose control over those processes as the amount of software being developed begins to overwhelm their management teams. CloudBees clearly sees providing tools to optimize DevOps processes based on Jenkins as its next big opportunity.

In general, DevOps is much more of a journey than an event. But rather than being a simple article of faith, analytics can be used to show all the participants in a process that progress is in fact being made. Otherwise, resistance to increasing the number of application development projects being undertaken at the same time starts to mount. Whatever the reason for gathering those metrics, however, that old “Things Measured Are Things Done” maxim continues to apply. Those metrics also help IT organizations justify the return on investment in DevOps to senior managers.

Of course, managing DevOps processes at scale is as much a cultural challenge as it is a technical issue. When many organizations first embrace DevOps there’s a lot of initial enthusiasm. But the further down the DevOps maturity curve an organization gets, the more they encounter various hurdles. Metrics collected and analyzed by the tools provided by CloudBees help organizations determine to what degree those hurdles are related to simple technical glitches versus organizational issues that are almost always a lot harder to resolve.

— Mike Vizard

September 11, 2017
Quest Software Extends DevOps Reach to MySQL Databases

Thanks mainly to the rise of DevOps, the number of open-source database deployments in the enterprise has risen sharply in the last two years. The driving force behind those deployments has been developers that have shown a marked preference for downloading an open-source database instead of requesting access to a commercial database from a database administrator (DBA) working in an IT department. As more of those applications get deployed, however, either the developer or a DBA still needs to find a way to manage those open-source databases.

To address that issue, Quest Software is now making available its Toad Edge tools for managing databases available for IT organizations that have deployed MySQL databases. Greg Davoll, executive director of product management and marketing, says Quest Software already supports open-source Postgres databases and will be adding support for other open-source databases.

In a lot of instances Davoll says the same DBA that is being tasked with managing IBM DB2 or Oracle databases is being tasked to manage MySQL databases. Most of them are already familiar with Toad database tools that are already widely used by internal IT organizations. Davoll says Quest Software is well aware there are a variety of open-source tools available for managing open-source databases. But Davoli says most of those offerings are not nearly as rich in terms of features as Toad database management tools that have been continually enhanced now for the better part of a decade.

For example, Toad Edge can be used to identify and manage schema differences and initiate changes in real time. Toad Edge can also be used to explore and edit JSON data as well as keeping all the scripts being employed to manage databases up to date.

Davoll notes that Toad Edge has also been integrated with Jenkins Continuous Integration server to allow databases to better facilitate continuous integration/continuous deployment (CI/CD) processes.

Most DBAs have come to terms with the fact they are being required to support multiple databases. They don’t always appreciate the extra work that involves. But as the number of applications deployed across the enterprise dramatically increases, the cost of employing commercial databases to support them all would be prohibitively expensive. Developers often download an open-source database on their own because they don’t want to spend time participating in an extended procurement process involving salespeople and purchasing managers.

Developers don’t always appreciate what’s required to keep the databases their applications up and running. Many of them initially believed they would be able to eliminate the need for a DBA altogether. But developers soon discover how much time is required to do routine database maintenance. DBAs may not always get the respect they deserve. But as most developers have come to appreciate, it’s difficult to have a truly integrated DevOps process that doesn’t include at the very least someone willing to manage those tasks.

— Mike Vizard

September 8, 2017