devopsy.kinsta.cloud

Author: Mike Vizard

InfluxData Delivers Revamped Database for Real-Time Processing

InfluxData has made good on a promise to deliver a next-generation time-series database that enables observability and monitoring tools used by DevOps teams to ingest high volumes of data more cost-efficiently.

In addition, for the first time, the latest iteration of the InfluxDB database supports the de facto standard SQL query language. Developers can run unlimited time-series workloads and contextualize data by any dimension without restrictions using SQL or existing tools such as the InfluxDB application programming interface (API), Flux scripting language or the InfluxQL query language

Brian Gilmore, director of internet-of-things (IoT) and emerging technologies for InfluxData, said InfluxDB is now based on a columnar engine, dubbed IOx, that leverages the open source Apache Arrow memory format and the Rust programming language accessed via a multi-tenant InfluxDB Cloud service that company manages on behalf of application developers. Use cases that require processing large volumes of data such as metrics, events, traces and other high cardinality data can now be queried in milliseconds, he noted.

The goal is to make this latest incarnation of InfluxDB available via a managed cloud service before providing IT teams with a version they can install themselves.

The arrival of a revamped InfluxDB architecture comes as the amount of data being generated across application environments continues to rapidly expand. Many DevOps teams are especially struggling with ways to process and store all the data being generated by cloud-native applications made up of microservices generating a steady stream of logs, metrics and traces.

InfluxData, of course, has ambitions for its database that go beyond application development and deployment environments. Edge computing platforms, for example, will increasingly process and analyze data in real-time at the point where it is generated and consumed. Similarly, applications infused with machine and deep learning algorithms will require databases that can process data at scale. Regardless of the use case, however, it’s apparent more IT teams will be embracing DataOps best practices alongside DevOps to manage data as applications are deployed and continuously updated.

It remains to be seen how many observability and monitoring tools might embrace this latest iteration of the InfluxDB database, but it’s clear DevOps teams should expect to see major advances as the underlying databases upon which these platforms depend continue to evolve. The goal is to ultimately eliminate the tradeoff that now often occurs between how much data is collected and analyzed versus the cost of processing and storing it.

In the meantime, DevOps teams will need to decide whether they want to manage databases themselves as more managed services become easier to invoke in the age of the cloud. Most database selections these days are made by development teams that aren’t especially eager to take on the tasks normally performed by a database administrator. A managed service essentially outsources responsibility for managing a database to a services provider instead of requiring IT teams to hire a DBA themselves.

One way or another, however, the speed at which data can now be processed in real-time should sharply reduce the need to rely on existing and cumbersome batch-oriented approaches to processing data.

November 2, 2022
Nobl9 Simplifies SLO Identification

Nobl9 today added a Replay capability to its service level objectives (SLO) management platform that makes it simpler to collect the historical data a DevOps team needs to get started.

Nobl9 COO Kit Merker said Nobl9 Replay makes service level indicator (SLI) data instantly available in minutes using 30 days of historical data collected from observability and application performance monitoring (APM) platforms. Previously, DevOps teams would have had to wait as long as a month before the Nobl9 platform could collect enough spreadsheet data to create a graph that identified the relationship between various services.

Nobl9 now enables historical data to be merged with data collected in real-time to allow DevOps teams to view SLO charts and generate reports shortly after creating an SLO within a Git-based workflow.

Nobl9 is trying to spur greater adoption of SLOs by making available an open source SLO specification that defines a common interface for constructing them that has garnered support from Dynatrace, Sumo Logic and others. SLOs, of course, are not a new idea; they have been used as a metric to track the performance of IT services for decades. However, as more microservices-based applications are built and deployed, it’s becoming more challenging to maintain SLOs across applications that have many more dependencies than legacy monolithic applications.

The Nobl9 platform is designed to enable DevOps teams to employ an SLO-as-code approach that makes it simpler to gather the metrics that confirm whether SLOs are being achieved.

It’s not clear whether reliance on SLOs has waned over the years or if application environments simply became too complex to track meaningful metrics. A recent survey conducted by Dimensional Research on behalf of Nobl9, however, found a full 94% of respondents that have or plan to implement SLOs intend to map them directly to business operations, with 91% reporting they expect that effort to improve decision-making. More than 80% also said their organizations are planning to increase the use of SLOs, with 87% indicating SLOs should improve overall microservices performance.

The issue is that most organizations have limited visibility into their IT environments. The survey found less than half of respondents (46%) had visibility into all their IT environments. Only 45% and 35% claim to have visibility into containers and microservices, respectively. More than three-quarters (78%) said hybrid clouds make observability more difficult, even though 45% said they already employed 11 or more observability and monitoring tools. Trying to track SLOs using tools such as Grafana dashboards that DevOps teams need to set up and maintain is simply too difficult, noted Merker.

As applications are increasingly constructed using services that drive digital business transformation initiatives, it’s now only a matter of time before SLOs are more widely tracked. That will enable DevOps teams to not only ascertain their current level of performance but also more easily work toward achieving their actual performance goals, he added.

The challenge is that defining an SLO in a highly dynamic application environment where services are added and replaced with much greater frequency is often a lot easier than maintaining it.

November 1, 2022
Update to Open Source ZAP Tool Improves DAST Performance

An update to the OWASP Zed Attack Proxy (ZAP) open source dynamic application security testing (DAST) tool made available today improves performance by employing a multi-threaded passive scanner engine.

Version 2.12.0 of ZAP also adds support for HTTP/2 and should make it simpler to update the vulnerability scanning tool by making the spider that discovers new resources an add-on that is now capable of discovering more URLs.

ZAP is an open source penetration testing tool maintained by the Open Web Application Security Project (OWASP) designed specifically for testing web applications.

Simon Bennetts, founder and chief maintainer of ZAP, said the latest update will make it simpler for developers to discover vulnerabilities in their code faster. One of the reasons developers often don’t frequently scan their code for vulnerabilities is because the scans are perceived to be slow, he said. Of course, the longer a developer waits before scanning their code the longer it will take to complete the scan.

DAST tools are a critical element of any DevSecOps workflow because they enable developers to view weaknesses in application security as an attacker would—by analyzing a web application through its front end. That approach doesn’t eliminate the need for static application security testing (SAST) tools to discover vulnerabilities in source code; rather, a DAST tool helps developers better understand how cyberattacks target specific types of vulnerabilities in their applications, noted Bennetts. As an open source tool, ZAP is intended to remove the barriers to adoption of DAST tools by making it simpler for developers to download and install themselves, he added.

Interest in DAST tools is, of course, rising in the wake of a series of high-profile attacks against software supply chains. The debate that has ensued is how far to shift application security responsibility left toward application developers. More organizations are starting to require developers to embed security tools within their application development tools. Others are also embedding various types of security tools within their DevOps workflows to ensure scans are run should developers fail to do so.

No developer sets out to build an insecure application, of course. The primary issue is that many of the DAST tools being provided are not designed to be employed by developers, said Bennetts. Instead, they are designed for cybersecurity teams that then share results with developers after an application has been constructed, he added. That approach creates a dysfunctional relationship between application development and cybersecurity teams because developers are being asked to remediate vulnerabilities long after code was initially written, said Bennetts. In most cases, those developers have moved on to other projects so any context for how code was constructed is often lost, he noted.

Implementing DevSecOps best practices is, naturally, as much about culture as it is the tools being provided. However, the one thing that is clear is that tools adopted willingly by developers are going to be much more widely used than tools chosen by others on their behalf. In many cases, the shortest path to improving the overall state of application security is going to be an open source tool that developers can readily access themselves.

November 1, 2022
Five Great DevOps Job Opportunities

DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers. Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

Indeed.com

Bobcat Company
Bismarck, N.D.
Solutions Architect AWS and SecDevOps
$130,000 to $180,000

LinkedIn

Lenovo
Morrisville, N.C.
Software DevOps
$100,000 to $140,000

Monster.com

Softweb Solutions
Plano, Tex.
AWS Engineer
$100,000 to $140,000

Dice.com

Estuate
Remote
DevOps Engineer
$100,000+

Indeed.com

Landmark Global
Remote
DevOps Engineer
$90,000 to $105,000

October 31, 2022
Mezmo Adds Observability Pipeline to Analyze DevOps Data

At the KubeCon + CloudNativeCon North America conference this week, Mezmo launched an Observability Pipeline platform that promises to make it simpler to manage, enrich and correlate machine data.

Previously known as LogDNA, Mezmo is expanding the scope of its reach to add the ability to augment and analyze data on top of its core log management platform. Over time, that capability will be extended to add support for metrics and traces using the open source OpenTelemetry agent software.

Mezmo CEO Tucker Callaway said the goal is to provide a framework for centralizing the flow of observability data that is now being generated by a wide range of sources across a DevOps workflow. It’s clear that DevOps teams require a means to enrich and correlate high volumes of data in motion that goes beyond simply pre-processing so it can be stored more cost-effectively, he added.

The Mezmo Observability Pipeline achieves that goal by ensuring the right data is flowing into the right systems in the right format for analysis, noted Callaway.

As the volume of data being collected by DevOps teams continues to dramatically increase the need to embrace a set of DataOps best practices is becoming more apparent. DataOps has emerged recently to describe a set of data engineering practices that applies many DevOps principles to the way data should be managed. Mezmo’s Observability Pipeline aims to make it simpler to apply DataOps best practices specifically to the data collected by DevOps teams.

Of course, each DevOps team will need to decide for themselves how much data they want or need to store long term. However, increasingly DevOps teams are holding onto data much longer as IT environments become more complex, noted Callaway. That creates a need to more efficiently analyze as much data as possible and store it more cost-effectively, he said. Those twin goals, however, often create a physics problem in the sense that they can be diametrically opposed.

In the longer term, it’s not quite clear how much DevOps teams are embracing observability. Most DevOps teams make use of monitoring tools that enable them to track a set of pre-defined metrics. Observability platforms promise to make it easier for DevOps teams to query a wide range of data types to discover potential issues before they disrupt an IT environment. The challenge is that many DevOps teams may not know what queries to form to discover those issues. It’s likely that, over time, machine learning algorithms will be increasingly applied to observability data to make it simpler to surface potential issues.

In the meantime, however, the need for DevOps teams to find ways to manage all the data being generated by multiple types of applications is becoming a higher priority. In fact, in the absence of any meaningful way to manage all that data, the benefits of an investment in an observability platform is likely to be limited, at best.

October 28, 2022
CNCF Survey Predicts Growing Wasm Momentum

A survey of 93 developers conducted by the Cloud Native Computing Foundation (CNCF) found more than a quarter (28%) are using WebAssembly (Wasm) in their cloud-native development projects, with another 36% planning to do so during the next six to 12 months.

The survey, published this week, is effectively a straw poll that found 42% of respondents are developing or plan to develop server-side applications using Wasm, while 48% are combining or planning to combine server-side workloads with client-side applications.

Originally developed five years ago by the World Wide Web Consortium (W3C) to create a common format for browsers executing JavaScript code, Wasm is now starting to be used to rapidly build lighter-weight applications that can be deployed on any server platform. The promise of being able to write an application once and deploy it anywhere is finally being realized some 25 years after the initial introduction of the Java programming language.

The survey found 70% of respondents viewed Wasm runtimes as a disruptive emerging technology. The primary programming languages used in association with Wasm are Rust (59%), Go (47%) and JavaScript (46%), the survey found.

Nearly two-thirds (63%) said they are both porting new applications using Wasm and migrating existing workloads, compared to 37% who said they were creating new applications only. The primary reason cited for migrating workloads to Wasm was application portability (78%), better security (77%) and improved runtime performance (70%).

Security ranks high because existing approaches to building applications rely on the aggregation of software components that tend to lack distinct boundaries between them. As a result, it becomes relatively simple for malware to infect all the components of an application. Wasm code runs in a sandboxed environment that isolates execution environments in a way that eliminates the ability of malware to laterally move across an application environment.

Target platforms for Wasm applications include serverless computing frameworks (63%), edge platforms (54%) and the web (52%), the survey found.

The biggest barriers to adoption were tools (46%) and lack of standards (45%). Just over half (56%) of respondents said Wasm developer tools are missing features and are difficult to use. A full 81% said they would like to see vendors provide full-featured and stable Wasm tools and projects.

Matt Butcher, CEO of Fermyon, a provider of a platform for building Wasm applications, said that while survey respondents most likely represented more advanced developers, it’s clear Wasm momentum is starting to build. Much like JavaScript before it, Wasm is about to jump from the client to server-side application environments, he noted.

Less clear is whether Wasm represents the next era of cloud-native computing or is simply another type of software artifact that will be employed to build applications alongside other artifacts such as containers.

Wasm isn’t necessarily going to replace existing approaches for building applications any time soon, but it does provide an alternative to serverless computing frameworks for developing lighter-weight applications that run faster. Existing serverless computing frameworks typically use containers that are much slower to start up than Wasm code running natively on a platform, which accounts for why developers are looking to embrace the format to improve overall application performance.

One way or another, more Wasm code will soon be finding its way into DevOps pipelines. The issue is determining how to incorporate it alongside all the other code moving through those pipelines.

October 28, 2022
Fermyon Adds Managed PaaS for Building Wasm Applications

Fermyon Technologies this week launched a hosted platform-as-a-service (PaaS) environment, dubbed Fermyon Cloud, that promises to make it simpler to build WebAssembly (Wasm) applications.

Announced during the Cloud Native Wasm Day event during the KubeCon + CloudNativeCon North America conference, Fermyon Cloud is now available in open beta.

Fresh from raising $20 million in funding, Fermyon CEO Matt Butcher said IT teams can quickly build and deploy Wasm applications using a Fermyon Spin development tool and a Fermyon Platform the company launched earlier this year.

Fermyon Cloud adds an opinionated PaaS environment managed by Fermyon that can start up Spin applications in less than a millisecond. It uses a serverless application programming interface (API) that makes it possible to cache and optimize applications as they are loaded, noted Butcher.

Originally developed more than five years ago by the World Wide Web Consortium (W3C) to create a common format for browsers executing JavaScript code, Wasm is now used to rapidly build lighter-weight applications that can be deployed on any server platform. In effect, the promise of being able to write an application once and deploy it anywhere is finally being realized some 25 years after the initial introduction of the Java programming language.

Wasm isn’t necessarily going to replace existing approaches for building applications, at least not any time soon, but it does provide an alternative to serverless computing frameworks for developing lighter-weight applications that run faster.

In addition, at a time when organizations are becoming more concerned about application security, the appeal of Wasm is clear. Existing approaches to building applications rely on the aggregation of software components that tend to lack distinct boundaries between them. As a result, it becomes relatively simple for malware to infect all the components of an application. In contrast, Wasm code runs in a sandboxed environment that isolates execution environments to eliminate the ability of malware to laterally move across an application environment.

Wasm is especially well-suited for building lightweight applications that previously might have been deployed on a serverless computing framework, noted Butcher. The issue developers are encountering when using a serverless computing framework is that those frameworks are based on containers that have comparatively slow startup times, he added.

It remains to be seen how much traction Wasm gains in the months ahead, but Butcher said he expects most backend services, such as databases, will continue to be built using containers. Fermyon is betting, however, that many future cloud-native applications built by the average developer will be constructed using Wasm. In the longer term, Wasm will also soon be used to build applications infused with machine learning algorithms, said Butcher.

In the meantime, DevOps teams should get ready for more Wasm code to move through continuous integration/continuous delivery (CI/CD) pipelines. The challenge, of course, is that code will need to be managed alongside all the other software artifacts moving through those pipelines, as well.

October 27, 2022
Sigstore Code Signing Service Becomes Generally Available

A free digital signing service for software created by the Sigstore open source community has become generally available this week via the cloud.

Announced at the SigstoreCon event that occurred during the KubeCon + CloudNativeCon North America conference, the cloud service makes it possible for developers to both cryptographically sign artifacts and verify that the components used to build applications are secure.

The Sigstore community operates under the auspices of the Open Source Security Foundation (OpenSSF), an arm of the Linux Foundation. Sigstore project sponsors, including Google, Red Hat, GitHub and Chainguard, have committed to operating a service with a 99.5% uptime and round-the-clock pager support. More than 70 organizations, including Shopify, Autodesk, Trail of Bits and Rancher Government Solutions are actively involved in maintaining and scaling Sigstore.

More the four million Sigstore signatures have been logged using Sigstore thanks mainly to Sigstore support that is already included by maintainers of open source Kubernetes and Python projects. The npm community also announced it is working to integrate Sigstore in all packages used to build JavaScript applications.

Bob Callaway, co-founder of Sigstore and technical lead and manager for the Open Source Security Team at Google, said this service will make it much simpler for developers to add digital signatures to code in a way that can be more easily verified within a software bill of materials (SBOM).

The goal now is to expand the number of open source projects that will commit to employing Sigstore to better secure software supply chains, he added.

The security of open source software became a major concern in the wake of the open source Log4j zero-day vulnerability. Sigstore was under development prior to that disclosure, but the Biden administration’s executive order requiring Federal agencies to verify the security of open source software has enabled the availability of more resources for the project.

In response to the executive order, the OpenSSF outlined a 10-point plan to secure open source software that would require more than $150 million in funding. One of those goals is to have 50 of the top 200 projects and 1,000 of the top 10,000 projects adopt an interoperable approach to software signing. That project was estimated to cost $13 million for the first year and $4 million per year beyond, with a one-time additional $10 million required after the first year.

It’s not clear how broadly developers will employ digital signatures, but given the current level of concern, it’s probable most organizations will soon require any code they use to be digitally signed so it can be verified as immutable. The next big challenge, of course, is finding a way to operationalize the collection of all the SBOMs that will include those signatures. After all, SBOMs are only the first step in a larger DevSecOps workflow that will enable organizations to accept or reject software components based on all the data being collected.

October 27, 2022
BMC Extends Scope of ServiceOps Initiative

At the BMC Exchange conference this week, BMC revealed it has updated its Helix IT service management (ITSM) platform to include mapping and discovery capabilities. Those updates make it simpler for cross-functional teams to correlate incidents, alerts and data and pinpoint root causes faster.

In addition, BMC is now applying pre-trained artificial intelligence (AI) models to observability data and the service models it tracks to assess service health, predict service impacts and proactively manage IT environments as the volume of metrics, events, and alerts continues to increase.

Finally, BMC also announced that the BMC Helix Control-M orchestration engine now supports open source Apache Airflow, a workflow management platform for defining data pipelines using a Python programming tool.

Margaret Lee, vice president and general manager for digital service and operations management at BMC, said these updates extend a unified ServiceOps initiative the company launched last year to unify ITSM, DevOps and customer service desk workflows. While each of these disciplines may have distinct objectives—such as speed and reliability—that are often in conflict with one another, Lee said it’s apparent IT organizations need to reduce the current level of friction that is often encountered when managing IT.

As a software-as-a-service (SaaS) platform, Lee said the BMC Helix platform achieves that goal via a consumer-like user interface that can be easily personalized. The goal is to make it simpler to create virtual agents, knowledge bases, live chat and tickets that make it easy for customers and employees to request IT help, she added.

Achieving that goal, however, will require IT organizations to rely more heavily on AI models to automate routine tasks as IT environments become more complex. There are simply not enough IT professionals available to manage IT environments using legacy processes, she noted.

In addition, end users and customers are expecting service requests to be automated in way that allow them to resolve issues via a mobile device, added Lee. In many cases, how cumbersome IT processes now has a direct impact on both customers and employee retention. The primary goal is to reduce the number of interactions required to complete a task such as changing a password or provisioning a laptop whenever a new employee needs to be onboarded.

While the pace at which IT management is evolving will vary greatly from one organization to the next, it’s apparent that many DevOps principles are being absorbed into traditional ITSM platforms. The challenge is that cultural divide between DevOps teams and centralized IT teams is still very wide. Most organizations are still trying to determine how far left they want to shift responsibilities toward application development teams versus continuing to rely on IT administrators using graphical tools to centrally manage IT workflows. It’s rarely going to be an all-or-nothing proposition. Rather, the issue is finding a way to strike a balance that enables organizations to be as agile as possible without introducing unacceptable levels of risk to resiliency.

October 26, 2022
JFrog Gives Pyrsia to CD Foundation to Secure Software Supply Chains

At the KubeCon + CloudNativeCon North America conference this week, JFrog announced it contributed the Pyrsia project, which uses blockchain technologies to secure software packages, to the Continuous Delivery (CD) Foundation.

Stephen Chin, vice president of developer relations at JFrog and governing board member for the CD Foundation, said the goal is to increase the number of contributors to the project. Current contributors to the Pyrsia project include Docker, Inc., DeployHub, Futurewei and Oracle.

As part of those efforts, the CD Foundation will bring a centralized governance model through which a roadmap for the project will be defined.

Pyrsia provides IT teams with a decentralized blockchain network to access an immutable software package repository that is designed to seamlessly integrate with the package management systems developers already employ. It creates a digitally signed, immutable chain of evidence for code that can be incorporated into a software bill of materials.

Securing software supply chains became a more urgent issue following the discovery last year of the zero-day Log4Shell vulnerability that impacted Java applications. Many developers routinely reuse open source software, but many of those projects are maintained by a small number of programmers that voluntarily contribute their time and effort to build components that others are free to use. Like any other developer, the amount of security expertise those individuals have is limited; the onus for making sure that software is secure falls on the organizations that decide to deploy it.

The trouble is, many organizations that rely on open source software assume it is more secure than it really is. Initiatives like Project Pyrsia are part of a larger effort to make it simpler for maintainers to secure open source software.

It’s not clear to what degree security concerns are prompting organizations to review the amount of open source software they consume. Most organizations are more dependent on open source software than they realize, as most packaged applications include a large amount of open source code that is downloaded from any number of repositories. Whenever a zero-day vulnerability is discovered, organizations can easily spend months looking for all the instances of an open source component that might be vulnerable.

In theory, blockchain platforms and DevSecOps best practices should make it easier to secure software packages by reducing the number of vulnerabilities in production environments. The challenge organizations face today is the amount of technical security debt that exists due to vulnerabilities in legacy applications. It might be close to a decade before applications based on more secure code replace existing solutions.

In the meantime, multiple approaches to securing software supply chains are advancing, including the use of some type of immutable platform. It may be a while before those platforms are widely incorporated into software supply chains, but it’s apparent that it’s now more a question of how long it will take rather than if those technologies will be applied to securing software development environments.

October 26, 2022