devopsy.kinsta.cloud

Category: Continuous Delivery

Continuous delivery is a software engineering discipline that allows organizations to consistently develop and release quality software at a rapid pace. It is achieved by automating the software development lifecycle, enabling tasks such as development, testing and release to occur concurrently and continuously.

Harness CD Platform Now Available via Source-Available License

Harness today announced a free community edition of its continuous delivery (CD) platform is now available under a source-available license rather than a more traditional open source license.

Scott Sanchez, chief marketing officer for Harness, said the goal is to make CD as accessible to developers and DevOps teams as their current CI platform. Harness already makes an open source edition of its continuous integration (CI) platform available.

The Harness CD platform is available under a Polyform Shield license that provides free access to the platform, including the right to use, modify and distribute it but not launch a competitive offering. A source-available license is an alternative to an open source license that allows an organization to potentially create a fork of the Harness CD that could, for example, be delivered as a cloud service. That specific issue has led to scenarios where vendors that create an open source project find themselves competing for revenue against a cloud service provider that is not necessarily as committed to contributing and maintaining the primary open source project. At this juncture, it’s not clear how many other vendors that make software available under an open source license might take a similar tack.

While most organizations that embraced DevOps have been able to successfully implement a set of CI best practices, the number of organizations that have implemented CD is not nearly as high. One reason for that is automating deployments across platforms has never been especially easy. Each platform tends to be customized to the point where it is difficult to automate application deployments. In effect, each platform is its own unique type of “snowflake.”

However, as the number of digital business transformation initiatives continues to steadily increase, the need to regularly update application experiences is going to require more organizations to implement a set of CD best practices. A source-available license should help increase the number of DevOps teams that can access and experiment with a CD platform, noted Sanchez. The challenge, of course, is that many of those applications are now based on microservices with components widely distributed across an IT environment.

Historically, CI and CD have been synonymous when viewed within the context of a CI/CD platform. It’s not clear to what degree DevOps teams will look to better leverage the CI/CD platforms they already have in place rather than deploying a dedicated CD platform for IT operations teams and letting developers choose whatever CI platform they prefer. In general, a recent survey conducted by the Continuous Delivery Foundation found that 80% of respondents that work for companies with two or more employees are involved in DevOps to some degree.

Regardless of approach, the coming year is shaping up to be one where many more organizations take DevOps initiatives to the next level. Arguably, as more organizations realize how dependent they are on software to differentiate themselves from rivals, the ability to deliver new application experiences has become critical. At the core of those efforts is a set of DevOps best practices that will ultimately determine just how competitive those organizations will be.

January 19, 2022
How AI Can Improve Software Development

Change is constant and inevitable. While some changes are welcome, others are just a necessary evil.

Continuous calls for digitalization combined with a pandemic-prompted emphasis on transforming business operations, force organizations across industries to modernize their legacy systems. The emergence of cloud computing has shrunk organizations’ on-premise IT footprints. However, with 96 of the world’s largest 100 banks, 90% of the world’s largest insurance companies, 23 of the 25 largest retailers in the U.S. and 71% of the Fortune 500 still using mainframe systems, any modernization efforts—resulting in increased risk of downtime or defects—will frequently lead to disastrous consequences.

So, while making changes can be expensive, uncertain, and lead to customer loss and/or embarrassment in the marketplace, modern business demands that systems change to continue to deliver value. The problem with mainframe modernization, however, is that today’s code search tools, linters and program analysis tools are deficient when it comes to mitigating the risks associated with maintaining and improving systems.

Modernization: A Knowledge Problem

With 10,000 mainframes in active use globally, the average organization annually spends anywhere from 60% to 80% of its IT budget just on maintenance. But, as experienced and proficient programmers retire or move on, many companies are realizing that the specialized domain knowledge of the industry and institution that those developers used to create and update these complex critical systems is walking out the door with them.

As the developers who originally wrote the code have long since moved on, and documentation leaves a lot to be desired, developers new to a system are unable to acquire comprehensive knowledge of the system’s functionality, at a detailed or fast enough level, to become adequately productive and proficient. That lack of speed comes with significant risk and cost to any organization.

Recognizing the full and true intent of the functionality written in code is not simple. Developers today spend roughly 75% of their time searching through source code, to identify the code representing the functionality that needs to be changed. Complicating matters further, understanding the code is not enough. To effectively and efficiently maintain and support any system, software developers must know precisely what an application actually does—and how altering code in one part of the system impacts it as a whole. But since the code representing a behavior that needs to be changed can be littered throughout the system, developers might think they’re making a simple tweak, when really they may be breaking the entire system (and not even knowing it). The risk of change resulting in unintended consequences is real.

All of that being said, how can developers improve their efficiency and effectiveness?

The Shortcomings of Today’s Tools

With code repositories growing unchecked over time to extraordinary sizes, those responsible for maintaining and preserving a system’s functionality say it’s becoming increasingly challenging to find bugs in code without machine assistance. And with debugging being an arduous, time-intensive endeavor, developers are increasingly turning to code search tools, linters, and static and dynamic analysis tools to analyze millions of lines of code, flag errors and suggest fixes.

Whether the tool excels at bug localization, code visualization or “error” detection, entirely too many tools are insufficient when it comes to actually identify the specific lines of code that require attention. Program analysis tools only illustrate code in ways that developers still have to interpret (potentially incorrectly) and draw their own conclusions. While code search tools might speed the rate at which developers can build a mental model of the code, such tools are notorious for false positives, and developers still have to undergo the mentally challenging endeavor of piecing that mental model together to identify the code, and then safely make changes. Worse yet, code completion tools—tools that less-experienced developers disproportionately rely on—can actually propose imprecise changes (one prominent tool offers incorrect code 71% of the time), and that introduces risk.

The cherry on top of this problematic sundae is that today’s tools don’t have any way to effectively validate the scope and accuracy of a proposed change. The influence of a suggested change can be tough to identify, so testing for comprehensive validation is frequently next to impossible.

Ideally, there has to be a better way to mitigate risk while making changes to legacy systems. Enter artificial intelligence. By employing a unique approach to AI, the code repository truly becomes a knowledge repository, enabling many capabilities unrealized before now.

A Developer’s AI-Enabled Accessible Knowledge Repository

By leveraging AI to automate the identification of the specific lines of code that require attention, developers can simply ask this AI-driven knowledge repository where behaviors are coming from—and quickly identify the code associated with that behavior. This puts AI squarely in the position of intelligence augmentation, which is key to leveraging its capabilities.

This novel approach of AI reinterprets what the computation represents and converts it into concepts, therefore “thinking” about the code in the same way humans do. The result is that software developers no longer have to unearth the intent of previous developers encoded in the software to find potential bugs. Even better, developers are able to overcome the inadequacies of automated testing by using AI to validate that they haven’t broken the system before they compile or check in the code. The AI will forward simulate the change and determine whether it’s isolated to the behavior under change. The result is the bounds of the change are confined to the behavior under change so that no unintended consequences arise.

While AI is not yet advanced enough to safely make changes on its own, it can serve as a knowledge repository with the added capability of directing developers precisely where to make any needed changes. With AI, programmers—even on their very first day on the job — can make any necessary fixes with confidence that they won’t break the entire system.

Unlike its code search, code navigation, code completion and program analysis tool counterparts, AI fundamentally empowers developers to safely, efficiently and effectively make any necessary changes. With AI — that is fortified by a knowledge repository that understands source code the same way a human does — developers can truly do more with less.

January 13, 2022
Continuous Delivery Challenge Looms Large in 2022

As 2022 kicks off, it’s becoming apparent that the major DevOps challenge organizations will face is mastering best practices that enable continuous delivery.

A global survey of more than 19,000 software development professionals conducted by the Continuous Delivery (CD) Foundation found 44% of respondents use either continuous integration or deployment. However, less than one in five (18%) use both practices to automate the entire software delivery life cycle. Continuous integration is currently the most popular DevOps-related activity (32%) followed by continuous deployment at 29%, the survey found.

The survey also found that only 14% of respondents said, on average, that they can go from code checked-in to successfully running that code in production in one day. Just 10% said they routinely deploy code to production each day. Nearly a third of respondents said they still manually approve code deployments to production.

Tracy Miranda, executive director for the CD Foundation, said it’s clear there is a need for a better understanding of continuous delivery as a distinct DevOps discipline that stands apart from continuous integration processes. While the two DevOps practices are complementary, she noted that continuous delivery requires a level of engineering expertise that many developers either lack or are simply not interested in gaining.

In fact, the survey found more than a quarter of respondents (26%) are not involved in DevOps-related activities at all. At the same time, however, the survey revealed that 80% of respondents that work for companies with two or more employees are involved in DevOps to some degree. Nearly 60% of enterprise developers that participated in the survey employ a continuous integration/continuous delivery (CI/CD) platform, as opposed to 48% of developers working in small businesses and 42% of contractors. However, it’s clear from the survey results that much of that activity is focused on continuous integration rather than continuous delivery.

The survey revealed, for example, that less than a quarter of respondents (23%) are involved in infrastructure provisioning and management at all. That compares to nearly a third (32%) that are involved in application and infrastructure monitoring. Only 20% are involved in creating automated tests, Collectively, the survey results make it clear there is still a long way to go in terms of shifting responsibility for DevOps practices further left toward developers.

One reason continuous delivery is so challenging is that automating deployments across platforms has never been especially easy. Each platform tends to be customized to the point where it is difficult to automate application deployments. In effect, each platform is its own unique type of “snowflake.”

However, with more attention focused on the integrity of software supply chains in the wake of a series of high-profile breaches, Miranda noted that automating software delivery will play a major role in achieving automation goals by reducing the need to rely on manual processes that are more easily compromised.

At the same time, more organizations are starting to embrace Kubernetes clusters—this can simplify automation of software delivery across a set of platforms that all share a common application programming interface (API), she noted.

The growing complexity of application development projects, coupled with increased reliance on software to drive digital business transformation initiatives, will also accelerate the adoption of continuous delivery best practices, added Miranda.

One way or another, application deployment in 2022 is shaping up to be a lot more automated than it was in previous years. The only thing left to be determined now is the degree to which that goal can be achieved.

January 12, 2022
Shift Left: Right or Wrong?

The term DevOps emerged into mainstream software development communities roughly 10 years ago. It represented a combination of tools and techniques which helped solve many pertinent issues of the time, namely responsibility for code in production and lack of a positive feedback loop. As of today, DevOps is a massive movement. It is considered the default way in which software development should be practiced. It has allowed even small teams to function with the same efficiency as much larger ones. Several engineering teams have adopted derivative approaches of the DevOps methods in order to suit their niche workflows.

The Dark Side Of DevOps

DevOps in practice presents a lot of tangible benefits to software engineering teams. Automation, transparency, and velocity are the chief advantages. From a business perspective, engineers are now more invested and accountable for the experience of users. The era of “Your Success Is Our Success” has finally arrived for engineers—and business teams couldn’t be happier.

For the individual developer, the number of things they were responsible for was piling up. More objects to monitor in production. As more features were getting built and deployed, alarms from internal systems kept going off more frequently. Individually, an engineer became responsible for the behavior, performance, security, compliance and so many other aspects of the application and underlying systems. Often, this became a distraction and began to defeat the very purpose of the DevOps methodology which meant to remove hurdles that were in the way of developers.

This is DevOps done wrong. Lack of maturity within software engineering teams has caused a lot of people to go down a DevOps path that is toxic and consequently forced engineers to burn out quickly. The core tenet of DevOps is to enable engineers to do more, but not to the point where they’re having to undertake enormous toil. DevOps still recognizes two distinct forms of specialized effort – software development and platform operations. The nature of services, scaling, and security may vary the degree to which dev and ops responsibilities converge.

Dev-Oops: It Goes Wrong Over Time

Interestingly, engineering teams view DevOps as development, and “everything else”. From the time a developer checks code into version control, there is a large assortment of workflows that kick in. Ideally, there is testing, integration, more testing, deployment and feedback. This is a large body of work, often underestimated and underrepresented. To pile this responsibility onto members responsible for development alone is where things begin to go wrong. Often, as a product ages, incident management, maintenance, and other engineering efforts pile up–resulting in fewer cycles being available for product development over time.

There is also the complementary case, where engineering teams are focused on product development and assign most free cycles to it. Platform operations are forced to become a watered-down effort and consist of the bare minimum it takes to keep the lights on, so to speak. This is a common observation in small teams and startups that typically have a new product release.

When implemented correctly, DevOps must allow engineers to take responsibility for all aspects of developing a feature and managing it in production. This means that programming the business logic, creating the declarative and non-declarative software, binding with services, building the artifact, and finally deploying to production all come under the gamut of a developer. In addition, a feedback loop—created by baking observability into the mix will provide insight into the health of the service (or feature)—will require the engineer to stay on top of the operational aspects of performance and reliability.

The current definition and practice of DevOps originated from Agile methodologies. They take off from the limitations of typically Agile teams which were released in “sprints”, still kept dev and ops efforts separate, and were agnostic of underlying tech. DevOps, on the other hand, is more tightly-coupled with the technology employed, for example: Microservices and automation. It facilitates continuous deployment and a high frequency of releases. Most importantly, it converges the notions of development and platform operations. We find that the interpretation of this “convergence” is what differentiates good implementations of DevOps from bad ones.

The Right Kind of Shift Left

At the heart of healthy DevOps adoption lies a technology shift complemented by a cultural shift. The fundamental principle that belies successful teams is a great developer experience. Irrespective of the infrastructure that powers the technical stack of a web application, software engineers who work on it need to be able to develop and deploy with ease. Let’s explore how.

Examples of tools that aid engineers during the development phase include IDEs, Source Control, TDD suites, documentation generators, and many more. Tremendous amounts of research and product development are currently underway in enabling developers to program better and facilitate more efficient development.

Different skill sets and focus areas are required to optimize different portions of the operations spectrum. In the case of large enterprise organizations, the dichotomy is clear. There are Platform Operations teams that are responsible for keeping all the systems running by providing automated means to set up and sustain reliable systems on the server side. Application Development teams are tasked with creating the applications, API endpoints, integrations and various other artifacts that will enable users to execute their tasks.

Because of the common underlying nature of both these functions–writing software–they tend to cross-pollinate. This coalescing is welcome, provided it is managed well. This is where abstractions come in handy. In particular, PaaS abstractions that provide in equal parts agility and intransigence. Pliant enough to accommodate custom workflows and bespoke needs of software products and teams, and yet preventing needless osmosis between Dev and Ops responsibilities, ultimately resulting in a superior developer experience as the nucleus of this atomic transformation.

PaaS With Flying Colors

The PaaS-low’s Hierarchy!

The perfect balance can only come from PaaS systems that comprise an architecture that is modular and offers the functionality of application build and deployment. The ability to work across multiple languages and frameworks combined with packaging and deployment best practices is a basic tenet. Abstracting the underlying infrastructure to favor support for container-based artifacts and orchestrating them at scale is the next most desirable component of a PaaS framework. A batteries-included approach to automating continuous delivery, security best practices, and other workflows goes a long way in ensuring that the PaaS substrate works for all kinds of developer needs. When using abstractions, the biggest opacity comes in the form of a lack of production parity. If the PaaS can be functional on all staging instances viz. dev, test, pre-production, prod, or any other remote instances, it would save tremendous toil arising from a lack of parity.

Bonus points for being open source and accommodating compliance, policy mandates, and customization to suit the dynamic needs of different organizations. An active community supporting the ecosystem and driving innovation and change management would make a more compelling case for PaaS tools.

January 12, 2022
CNCF Takes LitmusChaos Platform to the Incubation Level

The technical oversight committee (TOC) for the Cloud Native Computing Foundation (CNCF) announced today it is elevating the open source LitmusChaos application testing platform to the incubation level.

LitmusChaos is a chaos engineering platform donated to the CNCF by ChaosNative in 2020. Since then, it has seen adoption within production environments by more than 25 organizations, including Intuit, Lenskart, Orange, Red Hat and VMware.

ChaosNative CEO Uma Mukkara says LitmusChaos was developed to provide DevOps teams with a chaos engineering platform built using containers that makes it easier to scale up and down than existing monolithic alternatives. The platform itself, however, can be used to test both monolithic and microservices-based applications.

The platform consists of a Chaos Operator based on a software development kit (SDK); a ChaosHub for hosting and sharing experiments; Litmus Workflows that declaratively chains experiments (either in sequence or parallel) to build a chaos scenario; a centralized control plane, dubbed ChaosCenter, to design, schedule and monitor Litmus Workflows; Litmus Probes for creating chaos scenarios that automate steady-state validation and remediation actions and a Chaos Observability tool to export Prometheus metrics.

In total, there are now more than 400 contributors to the LitmusChaos project and more than 4,000 pull requests have been made. Mukkara said since the beginning of this year, Litmus operator installations have grown to more than 2,000 per day. Going forward, the goal for 2022 is to significantly increase the number of LitmusChaos integrations there are for a wider range of DevOps platforms, added Mukkara. Additional sets of experiments for Kubernetes and non-Kubernetes targets, improved observability and integration with other platforms—via the open source OpenTelemetry agent software also being advanced by the CNCF—is also planned.

In general, it’s still early days as far as the adoption of chaos engineering with DevOps workflows is concerned. However, Mukkara said that as more organizations embrace observability to better understand IT processes, the number of organizations that also use chaos engineering to test IT resiliency should also increase. In fact, as IT environments become more complex—thanks mainly to the rise of microservices-based applications—chaos engineering approaches to testing may someday be required, noted Mukkara.

In the meantime, there are still a lot of IT professionals that are uncomfortable with chaos engineering approaches to testing that deliberately break underlying services to test the resiliency of an application. The core idea, however, is that no application should have a single point of failure that could lead to it becoming unavailable. In theory, a microservices-based application won’t fail because when a service becomes unavailable calls are automatically rerouted to another service. The performance of that application will degrade, but it won’t go offline. The challenge is that building an application that achieves that level of resiliency using microservices requires a significant amount of DevOps expertise.

Regardless of how an application is constructed, the one thing that is clear is how dependent organizations are on applications to drive revenue. As such, the tolerance for application failures—or performance degradation—has never been lower.

January 11, 2022
GitLab Revamps Observability Strategy After Acquiring Opstrace

GitLab plans to make observability a more integrated element of its continuous integration/continuous delivery (CI/CD) platform after acquiring Opstrace.

Opstrace’s open source project exposes a horizontally scalable application programming interface (API) created by Cortex for Prometheus, an open source monitoring platform being advanced under the auspices of the Cloud Native Computing Foundation (CNCF). It then makes use of a Loki API to collect log data.

DevOps teams can also point an existing instance of Prometheus to Opstrace or any instance of Fluentd or Promtail software they are using to collect log data, with support for a wide variety of other APIs, such as the Datadog agent, planned. A command line interface (CLI) is provided to set an instance of Opstrace that creates a controller through which the collecting of log data is managed.

Kenny Johnston, senior director for product management for GitLab, said as part of an ongoing effort to make observability more accessible, GitLab plans to integrate Opstrace by default with GitLab Monitor for organizations that use either the self-managed version of GitLab or the software-as-a-service (SaaS) platform.

Opstrace currently collects logs and metrics and support for distributed traces is planned. The goal is to extend the overall scope of observability capabilities that GitLab already provides within its CI/CD platform, noted Johnston. The goal is to make it simpler for DevOps teams to employ observability capabilities and interrogate data to optimize workflows without acquiring a separate observability platform, he added.

GitLab is betting that, as Prometheus continues to gain traction, more DevOps teams will prefer to extend its reach into the realm of observability rather than having to manage a separate platform.

Although observability has always been a core tenet of DevOps best practices, achieving it has proven to be a challenge. At best, most organizations can continuously monitor IT environments based on predefined metrics that identify when a specific platform or application is performing within expectations. Observability combines metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshot issues without having to rely solely on a limited set of pre-defined metrics that have been created to monitor a specific process or function.

The rate at which DevOps teams will embrace observability will naturally vary. In 2022, the biggest obstacle might not be the technology as much as it is simply understanding what queries can help DevOps teams better understand the root cause of an IT issue before there is a major disruption. In the long term, it’s expected that machine learning algorithms will leverage the data collected by observability platforms to automatically identify issues that might lead to a disruption long before it actually occurs.

One way or another, the ability to discern events even within the most complex IT environment is steadily improving. Whether that’s accomplished as an extension of monitoring or by embracing true observability may not be all that relevant—as long as the ultimate goal is achieved.

January 10, 2022
LogDNA Survey Surfaces Observability ROI Concerns

A Harris Poll of 200 senior engineering professionals suggests it may be a while before most DevOps teams are able to achieve true observability in their IT environments.

The survey was conducted on behalf of LogDNA, a provider of an observability platform, and found nearly three-quarters of respondents (74%) are struggling to achieve their observability goals despite investing hundreds of thousands of dollars. In fact, well over a third (38%) said they are already investing $300,000 annually on observability tools.

Despite those investments, less than half of respondents are very satisfied with their ability to use log data. Key areas of frustration include the challenges of collaboration with colleagues spanning multiple teams (67%), the fact that the tools are not easy to use (66%) and the routing of security events (58%).

Nevertheless, the bulk of survey respondents said they are optimistic about observability. A full 85% of participants said they believe true observability is possible.

LogDNA CEO Tucker Callaway said the core issue that many organizations fail to recognize is that observability is really a data literacy and management challenge. Organizations are collecting logs, metrics and traces from a wide range of tools and platforms without thinking through optimal data pipeline construction. The result is an aggregation of data that has not been normalized to simplify surfacing actionable intelligence, he said. In fact, many organizations quickly discover that all they’ve really accomplished is increase their data storage costs, said Callaway.

In the absence of that data normalization, Callaway noted it becomes exceedingly difficult for cross-functional teams to query all the data being collected within a repository.

At its core, observability promised to make it easier to identify IT issues before they cause a disruption. Observability in one form or another has always been a core tenet of any DevOps best practice. Initially, DevOps teams focused on continuous monitoring as the most effective way to proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue. Observability platforms make it possible to correlate events in a way that makes it easier for analytics tools to identify anomalous behavior that might be indicative of the root cause of an IT issue.

In contrast, traditional IT monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. Observability platforms combine metrics, logs and traces—a specialized form of logging—to make it easier for IT professionals to interrogate data being generated by a wide range of DevOps tools and platforms.

Of course, the assumption is that DevOps teams actually know what queries to launch to determine what the root cause of an IT issue might be. In the longer term, machine learning algorithms should make it easier to identify anomalies within IT environments. In the meantime, however, achieving observability still requires a fair amount of DevOps expertise that is not always readily available.

Eventually, observability will become much more automated than it generally is today. As that transition occurs, IT teams should find the ROI of observability is steadily improving.

January 4, 2022
Best of 2021 – 6 Signs You’re Doing DevOps Correctly

As we close out 2021, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the twenty-first in our series of the Best of 2021.

A lot of organizations claim to be DevOps-focused because they use specific tools and loosely follow certain practices. But to really be a DevOps-centric organization, there are specific DevOps principles and processes that ensure your organization is as optimized and automated as possible. To ensure that you’re not just using the DevOps title without following the underlying principles as an organization, here are six signs you’re actually doing it right.

Deployment Is Fully Automated

Having deployment completely automated from start to finish is one of the more apparent signs that you’re following the DevOps principles correctly. Automated deployment is a huge time saver, especially when things go wrong and you need to revert to a previous build version quickly. Manual deployments can lead to problems often caused by human error. Having a strong focus on automating processes is a core belief DevOps-centric organizations should hold dearly. Automation increases an organization’s efficiency and reduces errors.

You Have Frequent and Rapid Release Cycles

As soon as software is ready to go, it should be deployed and delivered to the customers. We live in a world where competitors are building better software daily. It’s to an organization’s benefit that new features and bug fixes are continuously deployed to stay ahead of the competition. DevOps is all about continuous delivery, so having developers produce new code and quickly release it into production is one of DevOps’ core features.

You’re Using The Right Tools and Platforms

The tools you use to deploy features and fixes should make building applications easy and testing straightforward. You should generally be able to build, test and deploy to production with no hiccups along the way. If you’re spending a lot of time fixing any issues that occur, then you might not have the right tools or processes in place. Fortunately, today there are extensive continuous integration software options that can adapt to any current existing infrastructure. Although tools are an essential piece of DevOps, it’s important to remember that while tools make DevOps possible, it’s the practices that truly make the changes happen.

You Have a Continuous Feedback Loop

Nothing is perfect in its first iteration, and errors are bound to occur along the way. You must have a continuous feedback loop system to detect when and where things go wrong. Having software tools that monitor processes and quickly inform you when errors occur will make fixing issues that pop up easier and faster.

If you have a system set up to identify issues, fix them and improve upon them, you’re doing DevOps correctly. If you run into problems and simply fix them without taking the time to analyze why they occurred or how to prevent them from appearing again, you might want to consider making some changes.

Development and Operations Groups Work Together

Communication between development, testing and IT operations groups is vital for DevOps success. Cooperation between these different groups ensures that the pipeline will run smoothly. Any problems that do occur will be faster to fix when developers and IT operations personnel can easily coordinate with one another.

Goals and Objectives Are Clearly Established

While communication between different teams is essential, having clearly defined goals and objectives that everyone is striving toward is just as important. Ultimately, all the teams have the same goal: making the organization more efficient; however, the individual practices they follow to make that happen might differ. Team members should be aware of precise requirements that need to be met for a goal to be reached, instead of making assumptions or leaving it to chance.

December 30, 2021
Best of 2021 – From Agile to DevOps to DevSecOps: The Next Evolution
As we close out 2021, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the ninth in our series of the Best of 2021.

As the workforce shifts to remote locations, networks are becoming more diffuse as the edge gets farther away. At the same time, the number of devices and applications on those networks continues to rise. Some estimates suggest that, by 2030, there will be 15 connected devices for every person on the planet.

The rapid expansion of the remote workforce and their associated application needs has created a significant challenge for both application developers and the IT professionals tasked with deployment and maintenance of applications. Finding the right balance between speed of release and stability and reliability of applications – never an easy task – is paramount for the success of any organization.

Agile and DevOps evolved to help organizations strike this balance, providing the best possible customer experience in the shortest realistic time frame. But have Agile and DevOps fulfilled that promise?

The Promise of the Agile Manifesto

While Agile as a methodology has been known and followed for nearly two decades, the ways in which organizations have applied Agile have been as varied as the organizations themselves.

The Agile manifesto set out four basic values when it was first written in 2001:
- Individuals and interactions take precedence over processes and tools.
- Working software takes precedence over comprehensive documentation.
- Customer collaboration takes precedence over contract negotiation.
- Response to change takes precedence over mindlessly following plans.
Agile processes are applied across a range of development activities, from development of standalone applications to development of cloud-native business applications. The Agile Manifesto also contains several more principles that further explain what the values mean when put in practice.

These Agile principles can be condensed into a few areas of focus:

Prioritize the customer over the product

Agile development requires continuous customer engagement throughout the entire development process, with the end goal being frequent product delivery based on specific customer feedback. While this can present challenges for an Agile team, particularly when customers demand changes late in the development cycle, Agile maintains that the best way to elevate customer experience is to make the customer part of the development process.

Trust Agile Teams to Self-Organize and Self-Manage

Agile focuses on smaller teams that remain in constant communication with customers, as well as with each other. These teams should include both developers and those with knowledge of the larger business requirements being met through development. For these teams to work effectively, trust is key. Agile posits that self-organized teams, given leeway to pursue their projects in the way they see fit, result in the shortest development cycles.

Keep it Simple

Agile teams require simplicity. This correlates with Agile’s disdain for overuse of process. Agile seeks to minimize the total amount of work required, thus minimizing overall development time.

Agility Leads to Agility

Just as Agile teams continuously improve their product and the customer experience, they also constantly improve themselves. Constant evolution of teams and the processes they employ (remember, teams before processes) improves team agility itself.

While the Agile values and principles seem simple, effective application of the values and principles is another story entirely. Indeed, at times, the implementation of Agile development has violated the very first value of the manifesto by focusing on which tools and processes should be used. But, in an environment where more users are clamoring for more products and services, implementation of Agile methodologies is not just important, it is imperative.

Building on Agile with DevOps

The precise meaning of the term itself and the scope of DevOps remain a source of debate. At a high level, DevOps is the marriage of the development cycle with post-development activities, such as deployment and maintenance.

According to open source contributor Barbara Ericson of Cloud Defense, “Instead of IT operations and software development being siloed off from each other, DevOps breaks down the traditional boundaries that previously existed between them in order to achieve continuous integration and continuous delivery (CI/CD) of quality software features and applications to end users.”

Just as with Agile, the effective implementation of DevOps requires cultivating a DevOps culture within the organization. However, Agile and DevOps are also not mutually exclusive. That is to say, DevOps is not merely a replacement for or an evolution of Agile. Although Agile and DevOps differ in their structures and approaches, they can work in unison to achieve the ultimate goals of speedy deployment and enhanced customer experience.

While Agile and DevOps share common goals, they have not always agreed on how to achieve those goals. DevOps differs in many respects from Agile, but, at its best, DevOps applies Agile methodologies, along with lean manufacturing principles, to speed up software deployment.

One area of particular tension between Agile and DevOps is that the latter relies heavily on tools; in particular, when it comes to the automation of testing and deployment processes. But DevOps can overcome the resistance of Agile developers to tool usage simply by applying Agile principles themselves.

Effectively, DevOps proponents must convince Agile teams that dogmatic adherence to the underlying principles of Agile is actually inconsistent with Agile in the first place. Ironically, Agile developers who insist that the process is always bad actually violate Agile principles by refusing to acknowledge the benefits offered through change, another Agile principle.

The challenge is to have the Agile development teams trust in the automation efforts of DevOps, while at the same time encouraging the DevOps team to consider the business goals of deployment rather than pursuing speed of deployment for its own sake.

With constant communication between the Agile team and DevOps team (another Agile principle), development can achieve a degree of comfort with DevOps tasks and processes. This means that testing and deployment automation can proceed quickly, resulting in a decreased time to market, overall.

Ultimately, Agile and DevOps can work together most effectively when they recall their shared goals: smaller change increments based on feedback resulting in faster delivery cycles.

Continued Evolution: DevSecOps

As networks expand and businesses continue to integrate cloud-based services (in many cases, building multi-cloud environments), security issues become more and more complex. One area in which Agile is lacking is in integrating security into the development process. Unfortunately, in an Agile environment, application security is often an issue addressed after development, rather than as part of the process. Enter the next iteration of DevOps: DevSecOps.

DevSecOps revolves around three basic principles: ensuring data security while minimizing inconvenience for users in accessing data, using development tools that identify risks as early as possible in the development process and ensuring data encryption using technologies such as secure sockets layer (SSL) and virtual private networks (VPNs).

While DevSecOps and Agile may also disagree about the priority of tools, security must become an integral part of any development project. This is why Agile, DevOps and DevSecOps must all work together to ensure not only rapid deployment but secure deployment.

The Next Evolution

Agile and DevOps were both important evolutions of the software development process, but DevSecOps is the next evolutionary step. Despite a few disparate characteristics, Agile, DevOps and DevSecOps can and should work in unison to achieve their shared goal: the best possible customer experience and short deployment cycles.
December 24, 2021
Best of 2021 – The Next Evolutions of DevOps

As we close out 2021, we at DevOps.com wanted to highlight the most popular articles of the year. Following is the seventh in our series of the Best of 2021.

As society continues to recover from the global COVID-19 pandemic and some semblance of normal starts to return, technology continues to move forward. The pandemic certainly changed what normal means and people, processes and technologies all have adapted and evolved to meet those challenges. The narrative that the pandemic caused of your organization’s digital transformation is true for many practitioners. With 2022 just around the corner, DevOps is on the cusp of maturation of previously bleeding-edge paradigms and a focus on engineering efficiency. Here are five DevOps evolutions you should keep an eye on going forward.

Engineering Efficiency is Front and Center

Engineering efficiency is a particularly broad umbrella term. At the foundational level, engineering efficiency focuses on making someone (for example, an engineer) more productive. There are a multitude of disciplines that intersect, from organizational design to engineering/developer experience. Engineering efficiency is increasingly critical for organizations today. The pandemic had two major impacts on the technology sector in terms of resources and staffing. The first, at the start of the pandemic, was the unprecedented and unpredictable availability of resources; physical locations were being shut down because of the medical severity of the pandemic and the “getting hit by a bus” factor was playing out in real life. Second, during the recovery period, the Great Resignation began, and with it, a fight for resources that has become increasingly intense. Both have meant that the availability of resources, especially ramped resources, is scarce.

The Spotify model of tribes/guilds of engineering resources is a promising evolution; folks can frequently move around, reducing the toil that engineering resources face and allowing for engineering resources to not only ramp up more quickly but also increasing engagement and retention. The industry continues to march toward standardization in areas that were typically bespoke, such as the release process, and I think we’ll see that continue to evolve, as well.

Git is Ubiquitous

Leveraging a source code management (SCM) solution used to be reserved for software engineers. But as the proliferation of “something-as-code” has touched multiple levels of the technology stack from networking, storage and, ultimately, a development pipeline, the march of operations-focused engineers adopting many of the traits of software engineers continues. Having iteration in your infrastructure stack with more ephemeral/disposable infrastructure is becoming the norm. Saving and packaging the multitude of configurations for the “something-as-code” source control is a natural area of evolution.

Building off of source control are the package managers—your Docker registries, Helm chart repositories, etc.—for more deployable artifacts; that said, getting those artifacts to production is a process. Because Git is becoming ubiquitous across the technology landscape, this leads to the continued adoption of GitOps. Since the seminal piece was written by Weaveworks in 2017 defining the GitOps paradigm, leveraging GitOps is seen as a viable paradigm for many organizations five years later, especially when starting with greenfield initiatives.

Kubernetes is No Longer Bleeding Edge

In 2022, Kubernetes will celebrate the eighth birthday of its first commits on GitHub. Taking a jog down technology memory lane, eight years before Kubernetes (in 2006), VMware was still a private company and it would be a few years before vSphere was even released (remember how it felt to run a workload on a virtual machine in 2014?). The Kubernetes ecosystem still moves quickly, but placing workloads on Kubernetes is not a novel concept anymore. As application infrastructure and architecture have adopted the Kubernetes way (in other words, being idempotent and ephemeral), there is maturity around running a suitable workload on Kubernetes.

Kubernetes by design is highly pluggable. If you do not like the opinion or implementation of something inside Kubernetes, you can replace that opinion. Don’t like how Kubernetes handles ingress traffic? Then choose from one of the many ingress controllers available today. That is just one example of dozens of pluggable areas. Because of this, Kubernetes is viewed as a dynamic, non-static resource. Using Kubernetes takes trial and error and getting your cluster to be robust, performant and reliable is an ongoing journey that requires ongoing iteration.

Authors Can Be the Enforcers

There is no question that in a modern software stack, we are inundated with data. The metrics soup and the continued science of and exploration of observability (metrics, traces, logs) provide us with a lot to work with. Making decisions, even automated ones, based on that data continues to be critical. With the rise of site reliability engineering (SRE) practices, reacting or anticipating spikes in usage, failures and even security-related items are par for the course for modern teams.

These decisions can be authored into policies that specific platforms understand, such as autoscaling rules for a cloud vendor or something like Open Policy Agent in the Kubernetes ecosystem. With so many items shifting left toward the development team, finding the right resource, skillset or institutional knowledge to author these rules can involve the input of several teams. Because of the rise of “something-as-code,” regardless of whether the author of these rules is a software engineer, DevOps engineer, platform engineer or anyone in between, that author has the ability to enforce.

Shifting Left … But Complexity Shifts Left, Too

The old adage is that complexity is like an abacus: You can shift complexity around, but it never really goes away. With the movement to shift responsibility left to development teams, this also means that associated complexity is shifting to the development teams. Modern platform engineering teams provide the infrastructure (compliant Kubernetes clusters) to teams and any workload that is run on those clusters is up to the development team that owns it. Typically, development teams then focus on features and functionality. Managing lots of non-functional requirements—and even core infrastructure requirements such as networking—can be a burden; think about how your organization would handle a service mesh.

If you are a DevOps or platform engineer, making your internal customers—your development teams—successful is a great goal to work toward. Crucial to this is disseminating expertise. This can be in the form of automation and education. A common practice with the DevSecOps movement is to have some sort of scanning step as part of the build or deployment process, disseminating the internals as far as how the scan is performed, what happens if something is found, etc. Gaining internal adoption is a journey and having a good developer experience built around clarity and stability are important.

Evolution in 2022

Even with all of the challenges, bumps and learnings that the last year has given us, technology continues to evolve and advance and even lower the barriers of entry to continually become more inclusive. Focusing on improving engineering efficiency and reducing toil will allow for more participation and lower the barriers to entry.

December 23, 2021