devopsy.kinsta.cloud

Author: Mike Vizard

DeepFactor Unifies Performance and Security Monitoring for Developers

DeepFactor today launched what it describes as the first continuous monitoring platform designed specifically for pre-production environments.

Company CEO Kiran Kamity said the DeepFactor Pre-Production Monitoring platform combines security, performance and behavior monitoring in one offering. Based on a “Deep Passive Monitoring” capability that requires developers to add a file to a container image to instrument an application, the platform feeds the telemetry data collected via that file to an Application Runtime Intelligence engine. That engine surfaces potential security and performance issues as well as risky and unexpected behavior changes between versions of applications, he said.

Scans of application environments are launched via a DeepFactor portal. Each scan is then compared to a set of configurable rules that are developed and maintained by the DeepFactor research team to identify indicators of application misbehavior. The engine also learns the application environment, which over time enables DeepFactor to generate alerts based on threshold triggers, known bad behaviors, anomaly detection and other attributes.

The DeepFactor Pre-Production Monitoring platform also comes with pre-packaged integrations for widely employed DevOps tools including Jira, Jenkins, Slack and GitHub.

In addition to the commercial platform, DevOps teams can also make use of a DeepFactor Standard Edition, which is free for non-commercial open source projects with no user limitations.

In an ideal world, developers would address issues, including security, long before an application is deployed to a production environment. The issue is that developers are asked to navigate too many tools to achieve that goal. The DeepFactor Pre-Production Monitoring platform will make it easier for developers to surface insights in a frictionless way, said Kamity.

Most organizations that embrace best DevOps practices embrace observability as a core tenet. The challenge they face is making it easy for developers to instrument applications as they are built and then making sense of all the telemetry data collected. That’s especially critical as developers attempt to strike a balance between remediating security issues and optimizing performance to better ensure the overall user experience. Existing developer tools are optimized for performance; asking developers to switch tools to address potential cybersecurity issues is a task that can be delayed to the point where it is essentially forgotten or simply ignored.

It’s not clear to what degree DevOps teams might have the budget dollars required to deploy their own monitoring tools versus relying on the existing monitoring tools provided deployed by an IT operations team. However, it does usually take a significant amount of time to deploy and update the agent software that legacy IT monitoring platforms require. DeepFactor is making the case for capturing telemetry data using a comparatively lightweight container that can be added to either a monolithic or microservices-based application.

Of course, if an application is already instrumented in a pre-production environment it might make sense to use the same monitoring platform to observe applications in a production environment. However, convincing IT operations teams to swap out their existing monitoring tools for something that makes the lives of developers easier might be a tall order.

August 31, 2020
Security Compass Unveils DevSecOps Platform

Security Compass today launched a DevSecOps platform that guides developers toward building secure applications faster using a set of security analysis tools and processes embedded within it.

Company CEO Rohit Sethi said SD Elements is the first instance of what the company is positioning as a balanced development automation (BDA) platform. By defining a set of best practices for building secure applications, organizations can strike a better balance between building applications fast and ensuring those applications are secure and compliant with any number of regulatory requirements.

Too much of the focus on application development is on pure speed, said Sethi. SD Elements is designed to offer a more structured approach that provides developers with access to threat modeling and risk assessment tools to reduce the number of vulnerabilities finding their way into an application.

SD Elements first gathers details such as the technology stack, deployment environment and compliance requirements. It then classifies risks based on security and compliance policies defined by the organization, which are then employed to surface recommendations to the developer. SD Elements also automatically tracks the status of security activities via integrations with security testing tools such as Veracode, Checkmarx or Fortify to generate reports that identify risks and current mitigation status.

Finally, SD Elements provides access to a library of thousands of security and compliance recommendations and controls, including code samples and test cases, that are mapped to regulatory and compliance standards from all over the world.

To some degree DevSecOps will slow down the rate at which applications are developed. Security Compass, via BDA, is making a case for minimizing the impact security gates will have on the rate at which applications are developed. It’s not clear BDA as a market category will catch on, but as providers of DevOps platforms continue to embed security tools in their platforms the entire category is evolving.

The challenge many organizations face today is they want developers to embrace DevSecOps as part of an effort to shift responsibility for application security to the left. However, organizations are still working to put the tools and processes in place that would enable developers to achieve that goal. Most developers want to build and deploy secure applications; this needs to occur within the context of a structured approach that has a foundation in best DevOps practices.

Each organization will need to determine to what degree they may need to slow down application development to build more secure applications. In theory, the smaller the workloads the easier it is to secure them while continuing to rapidly iterate. However, that level of DevSecOps maturity takes time and effort to achieve.

In the meantime, it’s clear DevSecOps has become a requirement as part of an overall effort to better secure the software supply chain on which so many organizations now depend. The challenge is finding a way to adopt DevSecOps in ways that developers will embrace rather than resist.

August 31, 2020
OverOps Taps Into Git Repos to Accelerate Code Reviews

OverOps this week announced it has added support for a git blame and automated source attach capability to its namesake platform for analyzing code that surfaces both root cause data for every error as well as the identity of the developer who owns that code.

Eric Mizell, vice president of solution engineering at OverOps, said once those developers are identified, the OverOps platform will automatically alert them about the errors in the code.

Every Git repository provides access to a git blame tracking command that identifies the authors of source code and the last time files were modified source file. The git blame and automated source attach capability extends that tracking command to accelerate the process of fixing errors that are discovered by the OverOps platform, which leverages machine learning algorithms and other forms of advanced analytics to perform static and dynamic analysis of code to detect errors in code before a DevOps team promotes code into a production environment.

Mizell said this latest capability accelerates the remediation process by automatically routing critical alerts to the developers responsible for the last commit.

Ultimately, DevOps teams should be able to spend less time in “war rooms” trying to resolve issues in a production environment because errors in code will have been addressed much earlier in the application development life cycle, he said.

Naturally, there’s a lot of sensitivity when it comes to analyzing code. It will be up to each IT organization to determine the level of visibility into the quality of the code being created by each developer. Most organizations will simply route alerts to individual developers to address without making each issue the topic of a team review. However, organizations may want to track what types of errors are being made consistently to better inform training requirements.

Regardless of approach, most experienced developers don’t want to spend a lot of time discovering the same issues multiple times when reviewing code someone else wrote. Tools that automate the discovery of those issues should enable development teams to focus their efforts on the more complex issues that inevitably arise when building modern applications.

In the wake of economic downturn brought on by the COVID-19 pandemic, there’s a lot more pressure on DevOps teams to become more efficient. In many cases, the velocity at which applications need to be built, tested and deployed in the wake of the economic downturn has increased as organizations invest more in digital business transformation.

Not every developer will enjoy have the code they wrote analyzed by a machine. However, most coders take pride in their work and would prefer to address in their own way any issue that may arise. The challenge is putting in place a code review process that results in better code being deployed faster without unnecessarily antagonizing the most valuable contributors to the digital success of any organization.

August 28, 2020
Eggplant Employs AI to Surface Test Recommendations

Eggplant this week announced it has extended the capabilities of its application testing platform infused with machine and deep learning algorithms to now include recommendations to help improve tests.

In addition, the latest update to the Digital Automation Intelligence (DAI) Platform makes it easier to onboard new members to an application testing team.

Finally, Eggplant is making it easier to capture snapshots of the testing environment to more easily identify areas that require additional test development.

Eggplant COO Antony Edwards said the goal is to extend a low-code platform that employs modeling tools to make testing more accessible to both developers and end users. There is simply not enough testing expertise available to keep pace with the rate at which applications are now being developed thanks mainly to the rise of DevOps, he noted.

By employing a testing platform based on a low-code platform that enables tests to be created using modeling tools, it becomes feasible to include more end users in the testing process, he said. Generally, end users have the best understanding of how an application should function, so including them in the testing process also yields higher quality applications that will be accepted sooner, Edwards added.

Eggplant, which was just acquired by Keysight Technologies, has been making a case for applying artificial intelligence (AI) to application testing to increase the overall capacity of testing teams. Using AI makes it possible to conduct more than 100 tests in parallel because most of the mundane tasks associated with building and updating tests have been automated, said Edwards.

There will always be a need to include humans in the testing process, but Edwards noted AI can augment the capabilities of testing teams in a way that not only allows more tests to be conducted faster but also ensures more resources are allocated to writing code, versus continually debugging applications.

Many IT organizations are now caught up in a vicious DevOps cycle. They can develop more applications faster, but the more applications they develop the more time they need to spend debugging them. Ultimately, organizations find themselves spending more time debugging existing applications than developing new ones. AI-infused testing tools break that cycle by identifying issues before applications are deployed, which is when they are easiest and least costly to fix.

Breaking that cycle has become more of an imperative because in the age of microservices, the overall application environment is becoming more complex, Edwards said. Most testing teams are not going to be able to create the tests needed to assure the quality of those applications unless it becomes possible to generate more sophisticated tests using modeling tools that automate much of the process.

Naturally, it may take some organizations time to trust the tests and recommendations generated by an AI platform. However, in the wake of the economic downturn brought on by the COVID-19 pandemic, it’s apparent DevOps processes need to be streamlined. Arguably, one of the best places to begin that process is by automating as much of the application testing process as possible.

August 28, 2020
Envoy Proxy Server Project Comes of Age

The momentum behind the open source Envoy proxy software project is increasing as the number of platforms it can be deployed on continues to expand.

Developed under the auspices of the Cloud Native Computing Foundation (CNCF), Envoy has most recently been extended to include support for Arm64 CI processors and Postgres databases.

Envoy provides a layer 7 edge and service proxy that can be employed to provide a layer of network abstraction for both cloud-native applications based on microservices and existing legacy applications based on virtual machines.

Initially developed by Lyft, there are now more than 180 IT vendors and more than 155 companies making contributions to Envoy. Matt Klein, a software engineer for Lyft who is also a maintainer for the Envoy project, said at that level it’s clear Envoy has achieved a critical level of mass. In fact, that level of support is why so many providers of service meshes have elected to deploy their frameworks on top of Envoy.

Klein conceded Envoy is a complex piece of software that requires a significant level of engineering expertise to master. However, as IT teams look to become more agile, the need for a layer of abstraction that reduces network complexity has become more apparent, especially as organizations move to embrace edge computing.

That agility now also can be extended to software platforms via plugins. In the case of Postgres, for example, Envoy can now decode and interpret the wire protocol format for Postgres to extract data and metrics from Postgres traffic. That capability will not only route database traffic more efficiently but it also offloads the management of that task from the Postgres server without any changes to the database platform required.

Envoy makes it possible to extend many of the infrastructure-as-code concepts first put forward to manage virtual machines in the cloud to networks. DevOps teams have been able to provision virtual machines in a matter of minutes for years. However, in on-premises IT environments it can take days, sometimes weeks, to provision networking resources. Envoy provides a layer of abstraction that makes it possible to manage network services at a higher level of abstraction.

Klein said in addition to making Envoy less complex, the team overseeing its development is working to add deeper support for WebAssembly, a platform that makes it easier to build applications on top of Envoy. The Envoy team also plans to tighten integration with multiple observability tools to enable DevOps processes to be applied more easily to networking services, added Klein.

As Envoy continues to evolve, the rate at which network operations and DevOps will converge is only going to accelerate. DevOps teams will be able to employ Envoy in concert with a service mesh to programmatically manage network services at will. It may take some time for roles within enterprise IT organizations to be redefined to reflect that new reality, but the days when networks were managed via command-line interfaces that had to separately accessed for each router, switch or appliance on the network are coming to a close. In its place, a new era of software-defined networking based on commodity processors is finally starting to rise.

August 27, 2020
Survey Surfaces Impact of COVID-19 on DevOps Teams

A survey of 500 IT and engineering professionals across software and finance industries published today by Harness, a provider of continuous integration/continuous delivery (CI/CD) platform, suggests there is significant room to improve application development processes now that most DevOps teams are working from home due to the COVID-19 pandemic.

The survey finds only 43% of developers said development velocity has increased since COVID-19, while just over half (52%) said they are happier in their roles since the pandemic began. Ravi Lachhman, a developer evangelist for Harness, said as velocity increases so does the need to automate DevOps processes across a distributed application development team. The level of automation achieved, in turn, has a direct impact on developer satisfaction.

Despite any productivity concerns, however, nearly three-fourths of developers (74%) said their organization plans to keep a remote or distributed workforce following the pandemic. That compares to only 2% who said their teams were fully remote prior to the pandemic.

On the plus side, developers are overwhelmingly satisfied (94%) with their employer’s investment in employee growth and development, and two-thirds (66%) said they value their companies even more than they did before the crisis. However, only a little more than half, (52%) said it was unlikely they would change jobs within the next year. Among developers who make less than $75,000, 54% said they are at least somewhat likely to change jobs, compared to 46% of those making more than that.

Compensation was the most-cited priority for switching jobs. Over a third (35%) said that if they were to look for a new employer, their top priority would be better pay. That compares to 27% who would prioritize more opportunities for career and professional growth and 20% who would switch for the ability to work remotely. Another 13% would leave for better benefits, while 5% cited shorter hours.

Almost two-thirds (60%) anticipated that further salary cuts or layoffs for their development teams are at least somewhat likely. So far, 28% of developers reported there have been layoffs at their organizations, with 22% saying those layoffs included members of their teams. Only 29% of developers said they have seen an increase in developer onboarding since the pandemic began, compared to 42% who saw a decrease. Another 29% saw no change.

Nearly two-thirds of respondents who earn $150,000 or more are happier in their roles since COVID-19 began (64%), while 44% of those who make $75,000 or less are happier.

Among developers over the age of 55, almost three-quarters (70%) said they are either less happy or their happiness has not changed. Interestingly, that compared with 43% of millennials. The report also noted both men and women feel they are compensated fairly, at 81% and 82%, respectively.

The so-called “new normal” that DevOps teams are adjusting to is clearly taking a toll. For the most part, the transition to working from home full-time has not been as disruptive as many initially feared. However, it’s also clear many organizations will need to revisit how their DevOps workflows are constructed to accommodate the fact that most DevOps teams won’t be returning to an office anytime soon.

August 27, 2020
GitLab Presses Case for DevSecOps Collaboration

GitLab today at its GitLab Commit Virtual event pledged to make securing its open source continuous integration/continuous delivery (CI/CD) platform a more collaborative effort.

Cindy Blake, senior security evangelist for GitLab, said the company is committed to sharing a road map developed in collaboration with DevOps teams that lets organizations see what cybersecurity issues will be addressed when.

Rather than merely sharing DevSecOps marketing collateral, Blake said the goal is to encourage engineers to participate in a feedback loop that will help prioritize what issues need to be addressed across a shared DevSecOps agenda.

Most of that effort thus far has focused on making tools for scanning for vulnerabilities a natural extension of a DevOps workflow by embedding them into GitLab Core. Most recently, GitLab acquired Peach Tech, a provider of protocol fuzz testing and dynamic application security testing (DAST) API testing tools, and Fuzzit, a continuous fuzz testing tool. The company also moved to make its CI/CD platform available as a set of hardened Docker container images.

The next priority is to enable organizations to construct DevSecOps workflows spanning DevOps and cybersecurity teams, said Blake, noting it will be easier to construct workflows at scale using a CI/CD platform that tightly integrates all the tools required.

In time, GitLab plans to extend those efforts to include machine learning algorithms that will be trained to identify and remediate cybersecurity issues, she added.

As a founding member of the Open Source Security Foundation, GitLab is also committed to working with the rest of the open source community to better secure software, Blake said.

There’s naturally a lot more focus these days on application security. A recent Global DevSecOps Survey conducted by GitLab finds developers are exercising more control over security—more than 25% of developers reported feeling solely responsible for security, compared to 33% of security team members who say they own security. A total of 29% of respondents said they believe everyone should be responsible for security.

However, cybercriminals are also making a concerted effort to compromise software supply chains by inserting malware into DevOps workflows constructed on CI/CD platforms. The challenge with securing open source platforms, of course, is the code employed to build these platforms is accessible to anyone including, of course, cybercriminals.

It’s too early to say to what degree cybersecurity concerns are influencing the selection of CI/CD platforms. However, as cybersecurity professionals become more involved in application development, it’s only a matter of time before issues involving the security of the platforms used to build applications become a higher concern.

In the meantime, the line is blurring between application development and cybersecurity. The challenge for each organization will be determining just how far left they want to shift responsibility for cybersecurity to the individuals who build their applications versus a cybersecurity team that has a less vested interest in accelerating the rate at which code is promoted into a production environment.

August 26, 2020
Kong Extends Ambitious Effort to Unify Enterprise IT

Kong Inc. today released an update to the enterprise edition of its platform that provides a foundation for connecting all the classes of applications that make up a modern IT environment.

Company CTO Marco Palladino said version 2.1 of Kong Enterprise makes it possible to deploy the data plane that integrates hybrid instances of the platform without requiring a separate instance of an open source Cassandra database in each location.

The latest update also adds support for Kong Mesh, a service mesh for integrating services that is built on top of Kuma, an open source project that Kong donated to the Cloud Native Computing Foundation (CNCF) earlier this year, which supports both containers and legacy virtual machine platforms.

Kong has also made it possible to build plugins using the Go programming language and added support for plugins to manage application programming interface (API) life cycles.

Finally, Kong has added support for the new Kong Vitals user interface for dashboards and an application registration tool within its developer portal.

Based on an API management platform that serves as the foundation for Kong Enterprise, Kong Enterprise takes connectivity within enterprise IT environments to a higher level of abstraction, Palladino said, by integrating all Layer 4 through 7 services for both monolithic and microservices-based applications. Rather than having to manage an array of networking services and associated services in isolated silos, Kong Enterprise provides a layer of abstraction for unifying the management of highly distributed IT environments running loosely coupled applications.

Thus far, more than 250 IT organizations have embraced Kong Enterprise to simplify the management of distributed IT environments, Palladino said. Those efforts will only accelerate as enterprise IT organizations move to reduce the total cost of IT by streamlining the management of IT. Most enterprise IT organizations today are managing multiple platforms using separate tools that have to be mastered by different administrators. Kong is making a case for unifying the management of more than 20 different platforms. At the core of that effort is Kong Gateway, an instance of Kubernetes through which Kong extends the reach of its framework into those environments.

Thanks mainly to the economic downturn brought on by the COVID-19 pandemic the rate at which enterprise IT organizations have been moving toward hybrid cloud computing is expected to accelerate. Most organizations will not be able to economically justify hiring and retaining separate administrators for each platform they deploy.

At the same time, digital business transformation initiatives require a more agile approach to enterprise IT enable by managing API-based services at a higher level of abstraction.

Naturally, it may take for organizations to exercise the political will need to bring all the fiefdoms that make up an enterprise IT environment together. However, as organizations continue to move down the path toward DevOps the need to ruthlessly automate distributed IT processes across an extended enterprise has never been more apparent.

August 26, 2020
CircleCI Looks to Advance DevOps Best Practices Adoption

CircleCI today launched a developer hub as part of an effort to further advance adoption of DevOps best practices.

Company CEO Jim Rose said CircleCI developer hub provides access to a set of resources for optimizing continuous integration/continuous delivery (CI/CD) pipelines. In many cases prior to the onslaught of the COVID-19 pandemic, organizations had only partially implemented best DevOps practices, noted Rose. The CircleCI developer hub will make it easier for those DevOps teams to extend the scope of those processes while serving to educate IT teams that have had limited exposure to DevOps workflows, he said.

The CircleCI developer hub provides access to, among other things, CI/CD configuration optimizations and pre-built packages, migration guides and documentation. There are also reusable snippets of CircleCI configuration code, customized open source Docker images that can be used on any platform, and command-line interfaces (CLIs) and application programming interfaces (APIs).

Later phases of the developer hub will include an orb development kit to help users quickly create new orbs using CircleCI CLI commands. An orb is a software development kit (SDK) that simplifies integrating the CircleCI CI/CD platform with third-party tools and platforms. CircleCI also plans to provide a much more streamlined search engine experience for CircleCI documentation.

The CircleCI developer hub was built in part using $100 million in additional funding the company raised earlier this year. Overall, CircleCI claims it processes nearly 2 million jobs across 100,000 projects per day.

In general, Rose said DevOps is becoming more challenging as organizations attempt to orchestrate the building and deployment of monolithic and microservices-based applications constructed using container and serverless computing frameworks. There are plenty of instances in which deploying a monolithic application is still more appropriate, so IT organizations need to be able to create DevOps workflows capable of spanning multiple classes of frameworks and platforms, he noted.

Making matters more complex is the testing of microservices now needs to be integrated within production environments, added Rose. It’s not really possible to completely test a microservices until IT organizations see what impact it might have on other microservices that have already been deployed in a production environment, said Rose.

It’s not clear what impact COVID-19 has had on the productivity of DevOps teams. It might be easier for developers to generate more code working from home than in the office. However, generating quality code at the right time during a project requires a lot more orchestration when members of the application development team are highly distributed. More than a few organizations either already have or soon will be tuning DevOps workflows to align with the so-called new normal.

Of course, there are plenty of places to find tips and advice when it comes to building and extending DevOps workflows. The challenge is creating workflows that are not only flexible and resilient but also simple to implement and maintain.

August 25, 2020
Gitpod Open Sources Automated Local Development Environment

Gitpod announced today it is making available an instance of its Gitpod integrated development environment (IDE) available under an open source license.

Company CEO Sven Efftinge said DevOps teams will now have the option of deploying Gitpod themselves as an alternative to the existing cloud service provided by the company.

Designed to be used in conjunction with repositories such as GitLab, GitHub Enterprise and Bitbucket, Gitpod automates the building of development environments on-premises, Efftinge said. It continuously monitors updates to any Git repository.

Gitpod will then configure the right tools for a specific git branch, automatically compile machine-executable code and download all the dependencies required to allow developers to build code on-premises. Rather than having to manually configure a desktop environment, developers can start coding in seconds, Efftinge said.

Gitpod borrows concepts from continuous integration platforms and applies them to development environments, he said, noting developers today typically spend multiple hours per week setting up and maintaining local developer environments.

By automating that process, he said, the local development environment can also better align with production environments. One of the reasons why code doesn’t work in a production environment when it is first deployed is because the local development environment has been manually configured in a way that doesn’t accurately reflect the production environment, Efftinge noted.

That issue becomes even more problematic when developers are working on multiple projects involving, for example, separate microservices that are all going to run in different production environments, he added.

Gitpod also prevents drift by making sure all versions of a local desktop environment among developer teams remain consistent. Collaboration also becomes easier when the members of the teams have standardized on the same instance of a local desktop environment.

Efftinge said there are already hundreds of thousands of developers using the Gitpod cloud service. The open source edition provides an option that will enable DevOps teams to self-host their own instance, he noted.

Gitpod, formerly TypeFox, collaborated with Google, Ericsson, Arm and Red Hat to develop the open source Eclipse Theia project, which provides a foundation for building IDEs in the cloud using TypeScript, CSS and HTML. Theia, now being advanced under the auspices of the Eclipse Foundation, consists of a front-end running in a browser or in the local desktop application and a back-end running on any host. Both the front-end and back-end communicate through JSON RPC over WebSockets. Gitpod leveraged that project to build its own cloud-based IDE.

In the wake of the COVID-19 pandemic, there’s naturally a lot more focus on automation and developer productivity. Most developers will be working from home for the foreseeable future to help combat the spread of the virus. Of course, most developers have been writing code outside the office for years. The challenge now is enabling teams of developers to develop applications as efficiently as possible regardless of where they are located. Eliminating inefficient manual tasks such as maintaining local development environments could be a significant step in that direction.

It’s not clear yet whether DevOps teams are more or less productive since the onslaught of the COVID-19 pandemic. However, it is apparent the need for a consistent experience among teams of highly distributed developers is acute.

August 25, 2020