devopsy.kinsta.cloud

Author: Mike Vizard

Scalyr Alters Log Analytics Economics

Scalyr today announced that its log analytics cloud service is now capable of ingesting over 200TBs of data per customer per day in real-time.

Company CEO Christine Heckart said the goal is to make as much log data available at a price point that will continue to be less than $5 per GB at scale.

Heckart said Scalyr expects to be able to continue to expand the amount of data it makes available to customers because its namesake platform is built on top of a columnar data store rather than an index engine. The goal of a project dubbed Sonic Boom is to eventually make a petabyte of data available to customers of the cloud service, she said.

One of the paradoxes of log analytics is that while IT organizations are encouraged to use this data to proactively identify issues before they become a major problem, the cost of storing that data often results in IT organizations limiting the amount of log data they analyze. As a result, they often miss outlier events that could have major implications for the IT environment.

As a cloud service that is compatible with the S3 storage interface defined by Amazon Web Service (AWS), the columnar store engine employed by Scalyr makes it easier to pass the economics of cloud storage back to the end customer, she said. IT costs in the wake of the COVID-19 pandemic are, of course, going to be a much bigger issue going forward. Almost every aspect of IT, including log analytics, is likely to be subject to a cost review in the week and months ahead as organizations adjust to an economy that has changed for the worse overnight.

Heckart said the issue many organizations will face when it comes to logs is that data is the lifeblood of any digital process. Organizations may not need to store that data, forever, but analyzing log data to identify anomalies indicative of, for example, a cybersecurity breach is now almost a daily routine. As such, how much log data to make available where at what cost becomes a subject of debate.

Of course, it may be less costly to store log data locally. However, someone on the IT staff will need to manage those logs. Scalyr and other providers of SaaS-based platforms for log analytics contend it’s more cost-effective to rely on a cloud service to manage log data that also is more accessible via shared analytics tools.

Regardless of the amount of IT resources made available, the one thing that is certain is neither the amount of data nor the rate at which it needs to be analyzed will be slowing down anytime soon. In fact, many organizations are likely to accelerate their transition to digital processes that are ultimately both more flexible and cost-efficient than existing processes. It will be up to DevOps leaders to figure out how best to keep pace with that rate of change with whatever resources are at hand.

— Mike Vizard

April 14, 2020
Splunk Adds Remote Workforce Metrics Dashboard

Splunk has made available a Remote Work Insights dashboard designed to help IT administrators track and identify the most relevant IT metrics for a workforce that is working from home to help combat the spread of the COVID-19 pandemic.

Company CTO Tim Tully said the dashboard is initially configured to track virtual private network (VPN) traffic as well as access to cloud applications being made via an instance of an identity management platform from Okta.

In addition, the dashboard tracks the level of usage of Zoom video conferencing services to help determine to what degree employees are continuing to collaborate during the pandemic.

The level of productivity being achieved in most corporate environments is tied directly to the amount of network bandwidth being made available via a VPN, which can be deployed inconsistently on various client devices. Cloud application experiences, meanwhile, can vary widely based on the geographic region an end user is located.

The dashboard helps IT teams to determine where there are issues they can address and where there are issues that may need to be addressed by a third-party service provider.

Built on top of the Splunk Mobile applications, the dashboard provides a way for IT administrators to employ a tablet or smartphone to continuously monitor key remote connectivity metrics, Tully said, noting that capability means IT administrators don’t have to stay tethered to PC all day to monitor their environments.

Over time, Splunk plans to add support for additional metrics to make it simpler for IT teams to monitor remote workforces, he added.

Beyond simply monitoring metrics, the Remote Insights Dashboard also makes it possible to send rich notifications based on thresholds defined by the IT team. That approach allows Splunk Mobile to reduce the number of alerts being generated by the various IT platforms being monitored, said Tully. Those notifications then provide the relevant actionable intelligence IT administrators need to share with other IT professionals to remediate an issue.

It’s not clear to what degree IT teams will work from home after the COVID-19 pandemic subsides. Many organizations may decide it’s less expensive to have IT teams work remotely at a time when many of them will be looking to, for example, reduce the amount of office space they require to reduce costs. Many organizations are about to discover that IT staffs, much like customer support personnel, don’t necessarily need to be in an office to be effective.

Regardless of where they are located, many IT professionals are about to discover how productive they can be working remotely. That may create some boundary issues between the home and office. However, it should also go a long way toward eliminating the occasional need to go to the office at odd hours of the night, only to determine the source of a problem can only be fixed by a third-party service provider the IT professional could have contacted from home in the first place.

— Mike Vizard

April 13, 2020
BigPanda Provides Free 90-Day Access to IT Ops Platform

BigPanda, a provider of an IT ops platform infused with machine learning algorithms, this week announced it will make available for free for 90 days an instance of its software available to IT professionals working from home to help combat the COVID-19 pandemic.

Mohan Kompella, vice president of product marketing for BigPanda, said the IT Ops From Home program is intended to make it easier for IT teams to manage IT operations using a platform that was designed from the ground up to remotely manage highly distributed computing environments.

BigPanda is also providing free 90-day support along with free training via a virtual classroom. Organizations that opt to participate in the program are under no obligation to renew.

Kompella said the BigPanda platform is designed to correlate events in a way that automates IT incident management. It provides built-in integrations with software and services from Amazon Web Services (AWS), AppDynamics, Cloudwatch, Splunk, Azure, Jira and Slack.

It may take a little effort for IT organizations to implement a new platform for managing IT in the midst of a pandemic, he conceded, but also noted there are organizations that are finding their existing IT platforms are not designed to enable IT teams to support an entire workforce that is now accessing distributed applications from home. The BigPanda platform makes it possible to unify the management of all the silos that now make up an enterprise IT environment—in fact, Kompella said managing IT environments remotely may soon become the “new normal.” In much the same way many organizations now employ remote workers to provide customer service, Kompella noted it’s possible many IT support staff will now simply work from home. That approach would, among other things, reduce the cost of office space for IT personnel.

As organizations struggle to implement business continuity plans there’s no doubt many legacy IT operations platforms are being stretched to their limit. Many of the plans organizations had to slowly transition to a modern IT management platform now need to be accelerated, said Kompella.

There’s no doubt the COVID-10 pandemic will have a profound impact on the way IT is managed. Many processes will be forever altered, especially as organizations embrace DevOps processes to manage application workloads that are increasingly likely to be deployed in the cloud, as part of a concerted effort to make business processes more resilient in the face of global disruptions such as the COVID-19 pandemic. After all, no one is quite sure when the current crisis might subside or return. There may even additional pandemics to contend with in the future.

Whatever the motivation, an evolutionary approach to modernizing IT ops will be moving forward at a faster pace among organizations that have the wherewithal to stave off the current economic downturn caused by the pandemic. In fact, there may even be a high correlation between organizations that fail to modernize those processes and those organizations that don’t re-emerge stronger in 2021.

— Mike Vizard

April 10, 2020
LF Edge Moves Open Networking Push Forward

A vendor-neutral initiative dubbed the State of the Edge was acquired this week by LF Edge, an arm of The Linux Foundation focused on defining an open source stack of software for edge computing applications.

The State of the Edge project will be merged with the existing Open Glossary of Edge Computing initiative that LF Edge already had in place. That combined initiative will now be known as the State of the Edge project within the larger umbrella of LF Edge projects.

State of the Edge was launched by Vapor IO, Arm, Rafay Systems, Packet by Equinix and Edge Gravity by Ericsson to provide free research reports. Arpit Joshipura, general manager for networking, edge and Internet of Things (IoT) for The Linux Foundation, said that mission will continue as part of a larger effort to advance open software-defined networking software.

The goal is to define stacks of software and implementation blueprints that will make it simpler for both telecommunications carriers and enterprise IT organizations to implement open source networking software both at the edge and in the cloud, said Joshipura.

That’s critical because as IoT continues to evolve it’s apparent that IT and operations technology (OT) teams will converge, he added, noting the challenge is enabling organizations to achieve that goal around using a suite of programmable open source platforms that are truly interoperable with one another.

Existing LF Edge projects include Akraino Edge Stack, an edge computing platform; Baetyl, a cloud-based management platform; EdgeX Foundry; a framework for sharing data between devices; Fledge, a set of application programming interfaces (APIs) for common administrative tasks; Home Edge, a platform for deploying consumer applications; and Project EVE, a virtualization engine.

Interest in building and deploying applications at the network edge is on the rise in anticipation of 5G wireless networking services becoming widely available. Most of the organizations building those applications will be relying on best DevOps practices to achieve that goal. Less clear for now is the degree to which traditional network operations will be rolled up into those DevOps practices. The more programmable the networking environment becomes the more control each developer theoretically can have over the network services being accessed by their applications. However, network bandwidth is often a finite resource; there most likely will need to be some governing mechanism that determines how much network bandwidth any one application would be allowed to access at any given time.

Of course, the open source community is not the only entity eyeing the rise of edge computing. Providers of commercial networking platforms ranging from Cisco Systems to VMware are all focused on enabling edge computing applications. Historically, open source platforms such as OpenStack have been embraced by carriers. However, within enterprise IT environments open source networking platforms have not gained much traction. The open source community will need to galvanize around a set of platforms and blueprints for edge computing platforms quickly. Otherwise, many organizations will most likely turn once again to commercial networking platforms to provide services to edge computing applications.

— Mike Vizard

April 10, 2020
CD Foundation Serves Up Tekton Pipelines Beta

The team overseeing the development of the open source Tekton Pipelines under the auspices of the Continuous Delivery (CD) Foundation announced today the project is now in beta.

Christie Wilson, Tekton Project Lead and a software engineer at Google, said Tekton Pipelines are not necessarily a tool most DevOps teams will interact with directly. Rather they provide a foundation on which DevOps platforms can be built that will make it easier for DevOps teams to construct workflows spanning multiple continuous integration/continuous delivery (CI/CD) platforms.

As such, Tekton Pipelines should play a critical role in not just fostering interoperability but also alleviating concerns about become locked into a specific CI/CD platform.

The beta release of Tekton Pipelines is significant because it signals that the project is now stable enough to be incorporated in DevOps platforms and from here on will follow the same deprecation policies as Kubernetes in terms of supporting previous releases. However, Wilson noted that Tekton Triggers, Tekton Dashboard, Tekton Pipelines CLI and other components are still alpha and as such may evolve from release to release in a way that is not necessarily backward-compatible just yet.

In the meantime, the Tekton Pipeline team is encouraging all Tekton projects and users to migrate their integrations to the latest version of Custom Resource Definition (CRD), which is the application programming interface (API) supplied. The Tekton Pipeline team is also making available a migration guide.

The Tekton Pipelines project is one of several initiatives being advanced under the guidance of the CD Foundation, which is an arm of The Linux Foundation. Other projects include Jenkins and Jenkins X, a pair of open source CI/CD projects developed originally by CloudBees and Spinnaker, a CD platform originally created by Netflix. Tekton Pipelines are a framework that runs natively on Kubernetes that traces its lineage back to Google.

Premier members for the CD Foundation include CloudBees, Google, Circle CI, JFrog, IBM, CapitalOne, Salesforce and Netflix. Other members of the CD Foundation include GitLab, Red Hat, Harness and Rancher Labs.

Wilson said Jenkins X already supports Tekton Pipelines, while work involving Spinnaker is ongoing. It’s not expected that Jenkins, currently the most widely used CI/CD platform, will be able to employ Tekton Pipelines, said Wilson.

It’s not clear when Tekton Pipelines might be pervasively employed across multiple CI/CD platforms. In theory, as more applications are built on top of Kubernetes, the use of CI/CD platforms that run natively on Kubernetes should increase. However, most organizations that have already adopted a CI/CD platform such as Jenkins are continuing to use the same platform to build and deploy both monolithic and microservices-based applications. At some point, however, it’s expected those organizations will eventually migrate to a CI/CD platform that runs natively on Kubernetes. In many cases, different teams within the same organization may opt to standardize on different CI/CD platforms while still be able to share workflows, said Wilson.

Naturally, providers of DevOps platforms are already jockeying for position now that the race to support Tekton Pipelines as specified by the CD Foundation has formally started.

— Mike Vizard

April 9, 2020
Google Names CloudBees App Dev Partner of the Year

Google this week named CloudBees its 2019 Cloud Technology Partner of the Year for Application Development at a time when the rate at which application workloads are heading into the cloud is expected to accelerate.

Beyond the Jenkins continuous integration/continuous delivery (CI/CD) platform on the Google Cloud Platform (GCP), CloudBees also makes available a software-as-a-service (SaaS) edition of Jenkins X, a CI/CD platform optimized for Kubernetes, along with a CloudBees Software Delivery Management (SDM) service for managing DevOps processes.

François Dechery, chief strategy officer for CloudBees, said CloudBees chose to partner with Google at that level versus other cloud service providers because Google made it clear it would partner at both the technology and business development level. In addition, given Google’s long history with Kubernetes, it also made sense to leverage existing Kubernetes expertise, he noted.

In the wake of the COVID-19 pandemic, more application development work will be moving onto cloud platforms that can be remotely accessed more easily by development teams, he added.

In some cases, organizations will opt to self-manage those platforms or rely on a traditional managed service provided by a third-party IT services firm. However, Dechery said there also has been a significant increase in the number of organizations willing to consume a DevOps platform such as Jenkins X as a SaaS application, as long as the IT environment is proven secure. SaaS platforms are now a mainstream element of any enterprise IT environment, said Dechery.

Many of those decisions will be predicated on what degree of abstraction any organization is comfortable with as IT platforms become both more complicated and diverse, added Dechery, noting one thing that attracts organizations to Jenkins X is that it abstracts away the complexity associated with deploying Kubernetes. The Kubernetes deployment process is automated by Jenkins X, which essentially provides a level of abstraction on top of another abstraction.

Dechery conceded there is some level of inherent competition with cloud service providers that provide their own DevOps tools. However, most of those DevOps platforms are not up to the task of building and deploying applications at scale, he said. In addition, platforms such as Jenkins and Jenkins X make it feasible to manage application development projects at scale without having to worry about becoming locked into a single cloud service provider, he added.

Competition across the CI/CD platform space remains fierce. CloudBees has long benefited from a large open source Jenkins community that often leads to organizations implementing an enterprise edition of Jenkins supported by CloudBees. However, as organizations embrace microservices-based applications based on containers, many of them are revisiting their platform options. That doesn’t necessarily mean they will rip and replace existing investments in Jenkins platforms; however, given the rate and scale at which modern applications are being built and delivered today, many organizations may be moving toward more prescriptive approaches to bring some order to the chaos.

— Mike Vizard

April 9, 2020
Sysdig Extends Cloud Monitoring Alliance With IBM

Sysdig and IBM announced they have extended their alliance to make Sysdig monitoring tools more broadly available across IBM public cloud services.

IBM first adopted the Sysdig platform to enable DevOps teams to monitor workloads running on a subset of the IBM Cloud in June 2018. Now the company will extend those managed instances of the Sysdig monitoring platform running on its own public cloud to include additional cloud services such as IBM Watson, Event Streams, Cloud Databases, Cloud Object Storage and Cloud Foundry.

Janet Matsuda, chief marketing officer for Sysdig, said monitoring capabilities as especially critical as the number of microservices-based applications running on public clouds continues to increase. IBM also makes use of the open source Prometheus monitoring platform to monitor many of those applications. Sysdig last month announced compatibility with Prometheus as a data source.

In the wake of the COVID-19 pandemic, it’s probable that more monolithic applications will also be moving into the cloud as part of organizations’ business continuity strategies. Applications that run in the cloud are easier to manage remotely. Regardless of whether applications are monolithic or based on microservices, it’s clear IT teams will be “flying blind” unless they have access to monitoring tools, Matsuda noted.

Over time, many of the monolithic applications being moved into the cloud will be transformed into a more flexible set of microservices that are easier to update. As that transition occurs, the need to observe the myriad microservices running on a public cloud becomes an even bigger challenge for IT teams.

Of course, some IT teams will prefer to employ their own monitoring tools versus relying on a managed service, especially if they are trying to centralize the management of IT across multiple public clouds and on-premises IT environments. However, for now, at least, the bulk of IT teams are still employing various public clouds in isolation from one another.

Longer-term, the relationship between Prometheus and commercial monitoring platforms is still being defined. Many application developers are relying on Prometheus to observe applications during the development process. Many IT teams, however, are already making extensive use of commercial monitoring platforms to identify and correlate issues in production environments. Sysdig is betting most of those organizations will continue to rely on those tools because they are easier to set up and navigate. In addition, each instance of a Kubernetes cluster currently requires its own instance of Prometheus.

On the plus side, the amount of time and effort being applied to instrumentation and observability as part of any set of best DevOps practices continues to increase sharply. Furthermore, the need to combine security and operational IT metrics and analytics to advance DevSecOps adoption is also going to push IT organizations toward more robust monitoring platforms. Arguably, the real challenge is no longer finding a way to instrument applications as much as it will be turning all the data being collected into actionable intelligence that can be relied on to automate the management of IT at an unprecedented scale.

— Mike Vizard

April 9, 2020
HPE Expects Increased Reliance on Virtual Desktops

Hewlett-Packard Enterprise (HPE) is making it easier for organizations to migrate to virtual desktops that can be deployed on servers in an on-premises IT environment.

Peter Mansell, global practice director for digital workplace at HPE, said that in the wake of COVID-19 pandemic more organizations will be moving to centralize the management of desktops to make their IT environments more resilient. To facilitate that transition, HPE is making available servers that come pre-integrated with virtual desktop infrastructure (VDI) software from Citrix or VMware. Those servers can be acquired under standard acquisitions with 90-day payment deferrals if required or as short-term rentals, he said.

Alternatively, organizations can deploy VDI software on HPE Greenlake, a set of servers that HPE manages on behalf of customers.

Historically, the adoption of virtual desktops has been limited for reasons that range from total cost to simple inertia. However, one of the downstream impacts of the COVID-19 pandemic will be the greater centralization of desktop applications. In most cases, that will mean relying on either cloud services or VDI platforms that make it easier to access desktops remotely, Mansell said. The cloud, however, is not an option for many highly regulated industries that are required to maintain control over their IT infrastructure, he noted.

Of course, end users would still need to find a secure way to remotely access those servers assuming there is enough available bandwidth. However, from a management perspective, it would be much easier for IT administrators, who are now trying to push updates to a wide variety of devices as the bulk of the workforce continues to work from home indefinitely.

Up until now at least, resistance to centralizing desktops has been high among both end users and Microsoft. However, with the rise of the Azure cloud, Microsoft has been encouraging organizations to centralize management of desktops. Mansell said in addition to making VDI servers available, HPE is working with Microsoft to make virtual desktops running on the Azure cloud available to HPE customers as well. End users, meanwhile, increasingly rely on cloud applications, so the dependency they once had on applications that run locally has decreased considerably.

Each organization will need to determine whether it makes sense to centralize the management of desktops. Beyond ensuring business continuity, most organizations will be experiencing a significant amount of financial pressure in the months ahead. One way to reduce costs would be to reduce the number of IT administrators required to manage operating system and application updates across fleets of devices. In fact, some organizations may not even have to acquire fleets of devices if end users are employing their own machines to access virtual desktops.

Whatever the path forward, the COVID-19 pandemic has made more organizations aware of how easily their operations can be disrupted by events beyond their control. The issue now is coming up with a business continuity plan that minimizes the impact those disruptions can have on IT as much as possible.

— Mike Vizard

April 8, 2020
HashiCorp Strengthens Vault Secrets Management

HashiCorp has released an update to its HashiCorp Vault platform for managing secrets and encrypting data that now includes an internal storage option that eliminates the need to manage a separate storage backend.

Version 1.4 of Vault also adds an OpenLDAP Secrets Engine, which automates the management of static entities within OpenLDAP directories, and the ability to authenticate users and applications using the Kerberos authentication protocol.

HashiCorp has also updated the enterprise edition of Vault to add support for a Transform Secrets Engine that can be used to secure data that reside outside of Vault, such as credit card numbers, by applying tokenization techniques to mask data, as well as a Vault Helm Chart tool that makes it easier to deploy Vault on Kubernetes clusters to the enterprise edition of the tool.

Finally, the enterprise edition of Vault adds improved disaster recovery capabilities and support for NetApp enterprise key management to more easily manage encrypted data.

Chris Kent, director of product marketing for HashiCorp, said thanks to the increased focus on data privacy there is a greater appreciation for the need to maintain application secrets in a platform that resides outside of the core application. Regulations such as the General Data Protection Rule (GDPR) enacted by The European Union and the California Consumer Privacy Act (CCPA) penalize companies for losing control over credentials and data. As a result, more organizations are looking for a way to better manage application secrets, noted Kent.

At the same time, cybercriminals are becoming more adept at discovering the credentials of end users that developers sometimes inadvertently leave accessible within their applications, he noted. Vault provides a way to manage and maintain application secrets outside of the application in a way that serves to significantly enhance overall application security.

As developers and cybersecurity teams collaborate more closely as part of any effort to embrace best DevSecOps processes, it’s becoming apparent that a lot more attention must be paid to how application secrets are managed. In fact, arguably one of the biggest impediments to improving the overall security of applications is the way developers have historically managed application secrets. It’s not uncommon for cybercriminals to compromise an application in a way that gives them access to thousands of individual user names and their associated passwords, many of which unfortunately are reused both inside and outside of the extended enterprise. The challenge at this point is not so much making the tools available to manage application secrets as much as it is changing the processes and culture within organizations that are just now starting to embrace DevSecOps.

In general, as organizations become more proficient at managing application secrets the more secure application environments should become. Of course, secrets management is only one aspect of application security. However, the absence of an effective strategy for managing application secrets can prove catastrophic. By the time most IT organizations realize their application secrets have been compromised, it’s usually far too late to undo the damage.

— Mike Vizard

April 8, 2020
VMware Extends Network Virtualization Reach

VMware today announced the general availability of an update to its NSX-T network virtualization software that adds support for NSX Federation, which enables IT teams to apply global policies and isolate domains to contain problems to a single zone. Other capabilities being added via NSX-T 3.0 include support for Level 3 Ethernet virtual private networks (L3 EVPN), multicast routing and improved data plane performance.

In addition, VMware revealed that next month it will make available an update to VMware vRealize Network Insight that employs machine learning algorithms and statistics to discover application flows across a virtual network.

VMware is also making generally available intrusion detection and prevention (IDS/IPS) capabilities to a previously announced software-defined firewall, along with support for AWS Direct Connect, VMware SD-WAN application and business policy statistics; enhanced Kubernetes visibility, integrations with the VMware Tanzu cloud service for managing containers; the latest release of VMware Cloud Foundation; and VMware NSX-T 3.0.

Finally, VMware announced that VMware SD-WAN by VeloCloud software has now been integrated with the Microsoft Azure Zones portal to simplify provisioning on that public cloud.

Tom Gillis, senior vice president and general manager for the Networking and Security Business Unit at VMware, said as part of the overall effort to reduce costs in the wake of the sudden downturn in the economy as a result of the COVID-19 pandemic, VMware expects many organizations will accelerate the rate at which they are transitioning to network virtualization to replace many of the hardware appliances they currently rely on. For the amount many organizations are paying for software and support for those appliances, they could be transitioning to a much more flexible approach to networking that provides a cloud experience spanning on-premises and public cloud computing platforms, he said.

Gillis also noted it is now only a matter of time before VMware extends the control plane of its network virtualization software to the physical network underlay on which NSX-T is deployed. That capability will enable IT organizations to further reduce costs by eliminating the need to deploy multiple control planes, he said.

According to VMware, the number of customers that are consuming some element of its Virtual Cloud Network portfolio now exceeds 15,000, which represents on average a 50% growth rate each fiscal year since May 2018.

Gillis made it clear that in the months ahead a much larger swath of networking will become automated to drive down costs. Most organizations have adopted some level of network automation. However, Gillis said partial automation is an “oxymoron”—once most organizations start down the automation path the network becomes just as programmable as other IT infrastructure that is now managed as code.

It may take a while for networks to come fully automated at a time when many network engineers still prefer to use command-line interfaces (CLIs) to manage network infrastructure. However, as networking continues to become more complex in the age of the cloud, it’s not so much a question of whether organizations will automate networking as it is to what degree and using which platform.

— Mike Vizard

April 8, 2020