devopsy.kinsta.cloud

Category: DevOps Culture

Can Improving Developer Experience Actually Reduce Costs?

From platform engineering to AI, many new DevOps initiatives aim to improve internal developer experience (DX). Developer experience is essentially like user experience but considers the experience a developer has with a framework or tool. The promise is that quality DX creates a more satisfied workforce, reducing burnout and avoiding employee churn. But can improving developer experience also reduce costs?

I took to LinkedIn with this question, and the response was a resounding “yes.” In today’s lean tech climate, you might think that making the case to invest in employee satisfaction is a tough sell. But, it turns out that developer experience has significant ripple effects that benefit the business in diverse ways, making it a lucrative opportunity for technical leadership.

Developer experience contributes to outcomes like better efficiency, easier onboarding, less re-engineering, reduced overhead and more, which all — indirectly, granted — can reduce costs. Below, I’ll outline some of the ways developer experience can affect the bottom line, featuring viewpoints from various sources across the field of tech.

Improved Efficiency = Dollar Savings

The most obvious impact of enhancing developer experience is that improved efficiency can equate to dollar savings. For a concrete example, consider an API that provides useful error code responses and explains them within a developer portal. This saves developers time and money, said Brandon Boyd, product manager at Tekion. “Otherwise, they will have to guess about error codes and implement improper code, which requires rework,” he said.

With solid DevOps practices, a release that might have taken three months with manual processes could take a single sprint, said Anna Daugherty, director of product marketing, Opsera. “This leads to expensive development time saved while simultaneously getting the release to market faster to improve customer satisfaction and reduce churn,” she said.

Better Cycle Times Reduce Cost

Reduced friction can also lead to reduced cost. Specifically, better developer experience around areas like provisioning, scaling, and security can help things run more smoothly, equating to development and tooling efficiency, added Daugherty. “Taking out the guesswork and human error from development can save money on infrastructure, redundant tool costs and time costs,” she said.

Improving developer workflows, a practice some call developer productivity engineering, can have tangible outcomes that reduce costs. “Developers need flexibility and support from the right technology stack to be successful and deliver on quality work for overall business success,” said Dana Lawson, senior vice president of engineering, Netlify. “Investing in a cohesive workflow using composable web architecture streamlines processes to enable developers to focus on the higher-level thinking their projects need, and therefore cut time and budget spent on busywork.”

Good developer experiences also have minimal context switching, which can further reduce cycle time. “Forcing developers to toggle between various screens, applications, and docs inhibits deep work and focus and directly impacts how long projects can take and even the quality of the work,” said Isaac Nassimi, SVP of product, Nylas. Removing friction in the developer workflow can help launch projects faster, “creating longer revenue-generating timelines and opportunities,” he said.

Less Turnover and Onboarding

Developer turnover and onboarding can be expensive. LinkedIn reports that for each employee lost, the cost to the company could be up to 250% of their annual salary. Better internal developer experience can improve employee retention, which saves significant expenses. “It costs to hire a developer, and it costs to bring a new one up to speed,” said Andrew Cornwall, senior analyst, Forrester. “Happier developers are less likely to leave.”

But instilling a good reputation amongst developers has other side-benefits, Cornwall explained, such as attracting more first-tier, productive engineers. Developer retention also leads to better familiarity with the code, aiding debugging efforts and reducing human error. “Happy developers who have been there for a while start to understand what the goals of the team and the business actually are,” he said. “They can align what they do to satisfy a greater outcome.”

Reduced Overhead For Support

Another outcome for software products with good developer experience is that they are more self-service, meaning developers are more empowered to find the information they require to do their work. This, in turn, lowers the need for manual support and reduces labor costs.

Daan Stolk, co-founder, Alphadoc.io, recounted how his team at a previous company tracked whether support tickets could have been prevented with the right information architecture up front. He estimated that over 50% of cases could have been solved with the proper documentation and resources. As you can imagine, this finding led to a big overhaul of their developer support materials. “Ultimately, this resulted in customers not having to wait for responses to their questions and becoming more self-sufficient,” he said.

Less Re-Engineering

Developers make up an increasingly high expense on the balance sheet, and forcing them to re-engineer code is wasteful. One way to avoid re-engineering is to have better practices around reusable interfaces. For instance, GraphQL is proving to be a helpful layer to aggregate disparate data sources and federate a graph across an organization, making data retrieval more efficient.

“Reduce the time it takes [developers] to do recurring tasks by 30% or 40%, and you are talking about millions of dollars in savings annually,” said George Snowflack, director of central U.S. & LATAM, Apollo GraphQL. “This is why so many of the Fortune 500 have moved to a federated GraphQL architecture, because of its ability to compose a modern data access layer just one time, and then reuse it infinitely across any apps or clients on the front end.”

More Accountability and Responsibility

When employee needs are respected, they tend to feel more kinship with their internal community and take on additional accountability for their actions. “When a company’s developer experience promotes developer ownership, developers are more likely to take responsibility for their cloud environments, including cost optimization,” said Brandon Turner, VP of engineering, Kion. “Developers that feel empowered are much more willing to take accountability for their actions, including optimizing costs.”

Access to a Wider Talent Pool

When software is more usable, more people can participate, thus lowering the barrier to access and reducing the need for high-cost resources. “For some, this could include analysts moving into the developer role creating new efficiencies,” said Rob Koplowitz, former vice president, principal analyst, Forrester. “For the more intrepid, it could enable business developers that could change the paradigm significantly,” he said. However, a citizen developer culture must involve the proper controls around areas like change management, governance, and guardrails.

Furthermore, internal education costs are reduced with good developer experience. “The cost-savings value of prioritizing the developer experience is that companies do not have to make investments in continued education or hiring new individuals to help build upon new skills,” said Neil Kanungo, VP, product-led growth, KX. “Conducting in-person or costly training exercises can be viewed as an expansion if organizations have the time and budget, rather than a necessity.”

So, How Do We Get There?

It’s clear that improving DX can reduce expenses, especially regarding reduced labor costs. But how do you go about building out experiences that developers love? Well, it’s important for technical leadership to clearly understand what positive developer experience improvements actually look like. “These improvements should include introducing self-service mechanisms, removing friction, making educational training simplified and more accessible, and providing free access to community resources and support,” said Kanungo.

Lastly, leaders shouldn’t expect developers to optimize their costs without the proper support. Instead, cost savings will hinge on giving developers more visibility into their applications, especially for complex shared cloud environments. “If a development team can’t tell what cost inefficiencies are related to their application, they are less likely to optimize those costs,” said Kion’s Turner. “Simplifying the developer experience by giving development teams isolated environments that they own promotes responsibility for the health of those environments,” he added.

Furthermore, helpful tools will be required to move developer experience initiatives forward. Jon Peck, senior manager, developer relations and enterprise advocacy at GitHub, explained how DevOps practices, such as AI pair programmers, cloud IDEs, automation, asynchronous communication formats and shift-left security tools can work together to improve developer experience and indirectly bring cost-savings. The latter, developer-first security, could have real tangible monetary benefits since fewer vulnerabilities in production means fewer costly security incidents.

Lastly, as leaders seek to optimize their IT departments, much thought has been placed on ways to measure software developer productivity lately. Yet, most savvy executives view this as a red herring. While taking account of some development metrics is helpful, relying too heavily on minute analytics, such as individual commits or bug fixes, can negatively affect developer experience.

Final Thoughts: Good DX = Less Humanpower = Cost Savings

The most obvious outcome of improved developer experience is that less energy and time are expended. Developer cost per hour doesn’t have as much of an economic impact. This impact is apparent in both bustling and lean economies. “In times of expansion, improved developer experience can improve new-hire onboarding costs,” said Kevin Swiber, API strategy lead at Postman. “In times of contraction, developer experience can help improve cycle times as folks do more with less.”

It’s also evident that developer experience is especially important for API-first strategies. Better usability around API gateway management, for example, can increase simplicity and enable teams to allocate resources elsewhere, explained Abdallah Abedraba, developer relations, Zuplo. Mark Rovetta, senior programming writer at F5 Networks, also added that improving the quality of API documentation can reduce costs substantially since it enables deployment fluidity and aids learnability.

So, with all that said, listen to your developers and their needs. You might learn a thing or two. And save a buck or two.

January 29, 2024
Five Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

SimplyHired.com

D2 Consulting
Springfield, Virginia
DevOps Engineer
$150,000 to $180,000

Indeed.com

Mariner Finance
Remote
DevOps Engineer
$120,000 to $150,000

ZipRecruiter

Cogility Software
Irvine, California
DevOps Engineer
$110,000 to $140,000

CareerBuilder.com

BAE Systems
Manassas, Virginia
DevOps Engineer
$92,290 to $156,860

Dice.com

Rackner
Falls Church, Virginia
Software DevOps Engineer
$90,000 to $140,000

January 29, 2024
New Relic Adds Ability to Correlate IT Events to Business Processes

New Relic this week added an ability to enable DevOps teams to better understand the potential business impact any change to an IT environment might have.

Manav Khurana, chief product officer for New Relic, said New Relic Pathpoint adds a dashboard to the company’s observability platform through which business metrics can be collected alongside telemetry data. That data is collected either via connectors provided by New Relic or via a metrics application programming interface (API) that IT teams can invoke.

IT teams can use that API to connect to custom applications or an end-to-end business process, noted Khurana.

All that data is then modeled alongside DevOps telemetry data to provide insights into the impact any change might have on the business in real-time, he added. That approach provides the added benefit of making it easier to strategically align the management of applications, services and infrastructure by business function, noted Khurana.

In the near future, New Relic will also extend the natural language capabilities it is currently previewing to New Relic Pathpoint, he said.

Longer term, New Relic is working toward making it possible to model how proposed changes to business processes will impact IT environments, added Khurana.

Business and IT leaders intuitively understand that every business event creates a corresponding IT event, but most organizations manage those processes in isolation. New Relic Pathpoint presents an opportunity to close a divide between IT teams and the rest of the business that has plagued organizations for decades.

It may be a while before advances in data analytics and AI eliminate that divide completely, but creating a common set of metrics for managing business and IT processes is a crucial first step.

In the meantime, DevOps teams, in addition to accelerating the pace at which applications can be built and deployed, are being asked to employ observability platforms such as New Relic One to minimize disruption to the business. Observability has always been a core DevOps tenet, but achieving and maintaining it is challenging. Most DevOps teams today aspire to be able to maintain some level of continuous monitoring. However, as it becomes easier and less costly to instrument applications using open source agent software, interest is rising in observability platforms that simplify the investigation of anomalies indicative of an issue that could disrupt an application environment. As that shift occurs, more organizations will seek to correlate IT and business events.

It’s still early days as far as the adoption of observability platforms is concerned, but it’s apparent that as application environments become more complex, the plethora of monitoring tools that DevOps teams rely on will need to be streamlined. It’s not uncommon for DevOps teams to have multiple monitoring tools that surface a set of predefined but often conflicting metrics. Observability platforms, however, unify logs, metrics and traces in a way that makes it simpler to launch queries to identify the root cause of an issue, hopefully before any catastrophic disruption can occur.

January 25, 2024
5 Security Threats DevOps Teams Should Know

DevOps security, also known as DevSecOps, is a philosophy in which security practices are integrated into the DevOps process. DevOps security involves creating a ‘security-as-code’ culture with ongoing, flexible collaboration between release engineers and security teams. The DevSecOps movement, much like DevOps itself, is focused on creating new solutions for complex software development processes within an agile framework.

DevOps security is about breaking down silos and promoting open collaboration across teams. It’s about making everyone accountable for security with the ultimate aim of improving the quality and speed of code releases. The adoption of a DevOps security model necessitates a cultural shift across the organization. It requires the integration of security into every aspect of the development and operations processes.

DevOps Security is not just about implementing tools and technologies; it’s about changing the way we look at security. Rather than being an afterthought, security should be a key part of the ongoing conversation from the start of the project to the end.

5 Security Threats DevOps Teams Must Know About

As a DevOps professional, it’s crucial to be aware of the potential threats that could compromise your systems, applications, and data. Let’s take a look at the top five security threats that DevOps teams must be aware of.

Phishing Attacks

Phishing attacks are a common security threat that can lead to data breaches. These attacks involve cybercriminals impersonating legitimate organizations to trick individuals into revealing sensitive information, such as passwords and credit card numbers.

Phishing attacks can be particularly damaging for DevOps teams. They can lead to unauthorized access to development systems and data, which can lead to devastating supply chain attacks. For example, an attacker could trick a team member into revealing their login credentials for a critical system, allowing the attacker to gain access and inject malware into a CI/CD pipeline.

Code Injection

Code injection is another common security threat that DevOps teams must be aware of. This involves the exploitation of an application by injecting malicious code. The attacker can then execute the malicious code to compromise the system or steal sensitive data.

For DevOps teams, code injection attacks can be particularly harmful. These attacks can lead to compromised systems and applications, resulting in data breaches and significant downtime.

Man-in-the-Middle Attacks

Man-in-the-middle attacks are a type of security threat where the attacker intercepts and potentially alters the communication between two parties without their knowledge. This can lead to a range of damaging outcomes, including data theft, session hijacking, and even identity theft.

For DevOps teams, man-in-the-middle attacks can be particularly problematic. These attacks can lead to compromised systems and data, and even the unauthorized alteration of code.

Container Vulnerabilities

Container vulnerabilities are a significant security threat for DevOps teams. Containers are a key technology in the DevOps world, allowing for the rapid and reliable deployment of applications. However, like any technology, they come with their own set of security risks.

These vulnerabilities can range from use of insecure images, to misconfigurations and software bugs, to more serious issues like kernel exploits. If an attacker is able to exploit a container vulnerability, they could potentially gain control over the entire system.

DDoS Attacks

Last but not least, distributed denial-of-service (DDoS) attacks are a significant security threat that DevOps teams must be aware of. These attacks involve overwhelming a production environment with traffic, causing it to become unavailable to users.

For DevOps teams, DDoS attacks can be particularly damaging. Not only can they lead to significant downtime, but they can also result in lost revenue and damage to the organization’s reputation.

How to Mitigate These DevOps Security Threats

While these security threats are concerning, there are effective strategies and techniques that can be used to mitigate them.

Implement Email Security Practices and Tools

To combat phishing attacks, it’s crucial to implement email security tools that can detect and block phishing attempts. These tools can help to filter out phishing emails, reducing the likelihood that team members will fall victim to these attacks.

In addition to implementing email security, it’s also important to provide security awareness training for team members. This training should cover the basics of cybersecurity, including how to recognize and respond to phishing attempts.

Use Input Validation and Sanitization in Applications

To protect against code injection attacks, it’s important to use input validation and sanitization in applications. Input validation involves checking that the data provided by a user meets certain criteria before it’s processed by the application.

Input sanitization, on the other hand, involves cleaning up the data to remove any potentially harmful elements. This can help to prevent malicious code from being executed within the application.

Employ HTTPS and SSL/TLS Encryption for Data in Transit

To guard against man-in-the-middle attacks, it’s important to employ HTTPS and SSL/TLS encryption for data in transit. This ensures that any data being sent between the user and the application is encrypted, making it much more difficult for an attacker to intercept and read the data.

Regularly Scan Containers for Vulnerabilities

To protect against container vulnerabilities, it’s crucial to regularly scan containers for vulnerabilities. This involves using tools that can analyze the containers and identify any potential security risks.

By regularly scanning containers, DevOps teams can identify and address vulnerabilities before they can be exploited by an attacker.

Implement Rate Limiting and Network Filtering

Finally, to mitigate the risk of DDoS attacks, it’s important to implement rate limiting and network filtering. Rate limiting involves restricting the number of requests that a user can make to the application within a certain period of time.

Network filtering, on the other hand, involves blocking traffic from certain IP addresses or ranges that are known to be associated with DDoS attacks. This can help to reduce the impact of a DDoS attack and protect the availability of the application.

In conclusion, DevOps security is a crucial aspect of the DevOps process. By being aware of potential security threats and taking steps to mitigate them, DevOps teams can significantly reduce the risk of a security breach and ensure the ongoing success of their projects.

January 25, 2024
The Evolution of DevOps
In the bustling corridors of the tech world, a revolution known as DevOps reshaped the way software is built and delivered. It’s not just about bridging the gap between development and operations; it’s about fostering a culture of collaboration, innovation and continuous improvement.

Today, DevOps is recognized as a culture of beliefs, practices and principles. As DevOps increases in popularity as the preferred methodology of software development, it’s important to understand how it evolved and what it requires of today’s DevOps engineers.

Evolution of DevOps

Before DevOps became common practice, development and operations existed as two separate components of the application release process. The development team would write code and pass it to the operations team, who would then deploy it to the production environment.

Part of the problem with this process is the tension between what the developers are trying to achieve versus the goals of the operations team.
- Application development pushes for continuous updates to add features, fix defects and make other necessary changes, often as fast as possible.
- IT operations, on the other hand, tries to limit the number of releases as they are responsible for running a stable system with the highest percentage of uptime.
To further complicate matters, there was often no clearly defined or automated process for handing over applications. This would result in miscommunications and misalignments, such as:
- Developers writing code without considering where or how the code would be deployed or passing along poorly-documented deployment guides.
- Operations not fully understanding what they were deploying and why, resulting in inefficiencies. Additionally, any time they would discover issues with the application they would have to send it back to the developers with improvement suggestions.
These situations would cause release delays as operations would struggle with unclear directions or try to resolve unforeseen interactions between the new code and the servers. Server errors could also result in blame games between developers and operations, which would negatively impact company morale. And as this was a continuous process, the development of updates and fixes would continue well beyond the initial launch of an application.

That process began to change in 2007 when consultant Patrick Debois, facing frustrations over the barriers between development and operations teams, looked for a different approach. A meeting in 2008 with software developer Andrew Shafer over “Agile Infrastructure” led to the formation of the Agile Systems Administration Group.

Then, in 2009 Debois watched a livestream of the “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” presentation from the O’Reilly Velocity Conference. Inspired to host his own event, he launched the first DevOps Days conference in Belgium, which inspired similar conferences around the world.

DevOps quickly gained popularity as companies realized how it could increase efficiencies by:
- Making software development and related operations more reliable
- Reducing the time needed to bring software products to market
- Quickly and effectively delivering software updates
- Improving the culture of work within the organization
DevOps Today

The DevOps mindset changed the culture of software development organizations and evolved into a comprehensive set of practices and principles. DevOps aims to bring both development and operations together by:
1. Shortening the software development life cycle (SDLC) and increasing the velocity with which applications move through the release pipeline.
2. Providing a high level of quality control and ensuring that each update is well tested with a minimal number of bugs or errors.
Today, DevOps engineers have their own role, which often requires a range of technical skills and the ability to work effectively with other teams and departments. Many different methods of rapid deployment and various DevOps toolchains are used across an organization’s development team. These approaches often have different points of emphasis, such as the differing emphases of speed of release, security considerations and user experience between DevOps, DevSecOps and site reliability engineering. For instance, the adoption of microservices architecture. By decomposing applications into smaller, independently deployable services, organizations can enhance deployment efficiency. This allows for quicker iterations and updates and minimizes downtime during deployments, ensuring a seamless user experience. Furthermore, the collaborative culture inherent in DevOps paves the way for faster problem resolution and innovation, fostering an environment where continuous improvement it’s a daily reality. These benefits translate into a competitive edge in the market and drive business growth through technological excellence.

No matter the approach, currently, there are a few consistent sets of best practices for all modern DevOps teams:
- Automation. Automation is a cornerstone of DevOps and affects every stage of the SDLC from infrastructure provisioning to configuration management and testing. By eliminating much redundant manual work, automation can drive down costs and human error.
- Cloud computing. Cloud services provide scalable and on-demand resources, facilitating more flexible and dynamic infrastructure. Use of the cloud speeds developer productivity and efficiency by providing a unified platform for deploying, testing and releasing applications.
- Containerization. Containerization simplifies the process of packaging applications into a single, self-contained unit. Utilizing technologies such as Docker and Kubernetes enables developers to create standardized and portable containers that are deployable across different environments.
- Microservices. The adoption of microservices architecture breaks down large applications into smaller, self-contained services, enabling DevOps teams to develop, test and deploy more efficiently, significantly streamlining the overall process.
- Continuous Everything. DevOps goes beyond the standard practice of continuous integration and continuous delivery (CI/CD) as it also includes continuous testing, monitoring, feedback and operations. By seamlessly sharing insights across various stages, this approach cultivates an ecosystem of relentless enhancement and perpetual progress.
Aspiring DevOps Engineers

There are many professional skills needed to work in DevOps, including expertise in environmental testing, application monitoring and version control. However, true success within DevOps is less about programming skills or managing computing resources but instead comes from the ability to manage extremely complex systems.

In the dynamic landscape of technology, DevOps stands as a beacon of efficiency, collaboration and continuous improvement. DevOps has proven to be a transformative methodology that propels organizations toward a future where innovation, resilience and customer satisfaction are paramount.
January 25, 2024
Will the Rise of Generative AI Increase Technical Debt?

In the last year and a half, we’ve seen a glut of new generative AI-related tools reach the market. Various AI pair programmers powered by large language models (LLMs) are becoming integrated into software development environments. The number of new generative AI startups is staggering, and generative AI seems to be the new “catch-all” to accelerate software development and operations.

In this new paradigm, AI tooling accumulation seems inevitable. Yet, the explosion of new AI APIs, libraries and frameworks has the very real potential to increase technical debt and, in the process, overwhelm DevOps teams. “These new applications and capabilities will require many new MLOps processes and tools that could overwhelm any existing, already overloaded DevOps team,” said Bob Quillin, vFunction chief ecosystem officer.

Below, we’ll consider if and how new generative AI tooling has the potential to exacerbate technical debt. Notably, the rush to adopt AI, shifts in the AI tooling landscape and accelerated code output could all be producers of technical debt. Choosing your tools correctly, iterating thoughtfully and upskilling teams will be important guidelines to mitigate negative outcomes. That, and creating unique solutions to ‘flip the coin’ — using AI to consolidate code and reduce existing technical debt.

How AI Will Contribute to Technical Debt

The rise of generative AI-related tools will likely increase technical debt, both due to the rush to hastily adopt new capabilities and the need to mold AI models to suit specific requirements. “New LLMs and generative AI applications will undoubtedly increase technical debt in the future, or at a minimum, greatly increase the need to manage that debt proactively,” said Quillin. “It starts with new requirements to continually manage, maintain, and nurture these models from a broad range of new KPIs from bias, concept drift, and shifting business, consumer, and environmental inputs and goals,” he said.

Incorporating AI may require a significant upfront commitment, leading to additional technical debt. “It won’t be just a build-and-maintain scenario, but rather, the first of many steps on a long road ahead,” said Prince Kohli, CTO of Automation Anywhere. Product companies with a generative AI focus must invest in creating a data and model strategy, a data architecture to work with AI, controls for the AI and more. “Technology disruptions and pivots such as this always lead to this kind of technical debt that must be continually paid down, but it’s the price of admittance,” he said.

AI is a tumultuous space, and some degree of technical debt will likely be produced by future organizational shifts. “The pace of change and innovation in the AI field itself is exceedingly rapid,” said Juan Orlandini, CTO of Insight North America, citing the staggering emergence of new AI startups and tools. “There will be consolidation, and also, unfortunately, a number of the startups will cease to exist.” When this occurs, teams must grapple with unsupported packages or tools.

Another type of debt is the nature of AI and how it accelerates development across the board. “One of the great values of these tools is the acceleration they bring to development,” said Orlandini. “Simply put, we can put out more features faster.” The other side of this double-edged sword is that AI accelerates the creation of all programming endeavors, even those that become technical debt.

Generative AI-related tools are most likely to accelerate the pace of change and deliver new code development outcomes more quickly. “In the process of consuming their output, there is a possibility of accumulating technical debt if the usage is broad, rapid, and pursued indiscriminately,” said Ajay Sabhlok, CIO and CDO, Rubrik. “Developers, in this case, may be too focused on program execution and forget to pay attention to code complexity, repetition or use of non-standard logic.”

Ways to Mitigate AI Technical Debt

Thankfully, there are specific methods organizations can adopt to manage new technical debt. For starters, you should focus on creating smaller proof of concepts and then iterating as you learn, said Kohli. “When leaders like the head of technology or IT are deploying these processes, they should first create small agile teams that learn fast and disseminate those learnings efficiently,” he said.

Also, technical debt is more likely to accrue when the wrong platform or tool is chosen. Therefore, “organizations should be very deliberate in choosing what products to leverage,” said Orlandini. He encourages IT leaders to avoid relying on companies or products with flashy presentations and poor tech or great tech with poor business acumen. “Our industry is rife with stories of great products with terrible business plans — which ultimately leads to their demise,” he said.

Other experts similarly foresee upcoming shifts that may affect internal operations. “Companies should keep their options open when considering AI model vendors or tool providers,” said Kohli. “They must keep their ‘hands on the wheel’ and be prepared for turns.”

Just like any new technological adoption, generative AI requires forethought. Therefore, putting the same amount of planning and preparation into deploying generative AI as we do for developing coding standards around manual coding could help minimize technical debt, said Sabhlok. “It will help to ease into areas of usage by adopting a tiered approach to expansion and, at each step of the way, review the results against standards, compare notes across the development organization, and course correct where necessary. This can avoid the accumulation of technical debt in the long run.”

“IT leaders should first ensure that additional resourcing and training are in place to both upskill existing teams and hire new talent as required,” said Quillin. “This includes adding observability tools that can identify new and existing forms of architectural and LLM drift. Finally, to always be ready for any new, rapidly emerging future requirements and opportunities, IT leaders should invest in best practices and tools that will proactively keep their app architectures clean and ready for the ‘next big thing.’”

How AI Could Decrease Technical Debt

On the other hand, some believe that if used properly, the contrary is true — that new AI technologies will actually reduce technical debt. “The rise of LLMs, frameworks, and APIs, when applied properly together, will actually reduce technical debt,” said Ashwin Rajeeva, CTO and co-founder of Acceldata. “Code discovery and reuse is more easily accomplished with LLMs.” He points to the ability of this technology to identify duplicative code and consolidate it into a more streamlined interface.

Automated documentation generation is another area where LLMs could help improve code discovery and reduce duplicative efforts. In addition, LLMs could also help reduce the prevalence of code that doesn’t follow company standards, said Rajeeva. “It is much easier for LLMs to understand the company standards by learning from good coding practices and applying it to newly written code.”

Others also see the power of using generative AI to enhance automation across the board. “AI tooling is probably going to help the most in automation, velocity to transform old codebases, and empowering non-coders to accelerate the adoption of new business processes into their workflows,” said Orlandini.

Final Thoughts

AI is a multifaceted subject, and its use could go one of two ways. On one hand, if not used correctly, generative AI could only add to the surmounting technical debt pile. To avoid this outcome, organizations will likely need guardrails on when and how to use LLMs, said Rajeeva. He also advocates “making it a best practice within the corporation with sufficient training for newer developers.”

Then, on the other hand, generative AI could be utilized to respond to technical debt. It has the awesome potential to unlock automation gains and enhance employee satisfaction and retention, especially within older industries that may be behind the technological curve, said Kohli. “With AI and now generative AI, intelligent automation can deliver on all those promises such that it is fast becoming a new pillar of enterprise architecture.”

So, take the new wave of AI with a grain of salt. It will open up immense solutions but will also come with a significant degree of technical debt. Thankfully, there are ways to manage debt caused by AI in a way that can benefit operations and push development objectives forward.

January 25, 2024
Legit Security Applies AI to Detect Vulnerable Application Secrets

Legit Security today announced it has expanded the scope of its application security posture management (ASPM) platform to make use of artificial intelligence (AI) to discover secrets more accurately in applications that cybercriminals can actually exploit.

Scanners typically generate high false positive rates to the point where developers often start to ignore them. As a result, it’s not uncommon for secrets to be discovered in an application even though an alert was generated.

Legit Security CTO Liav Caspi said the company is now applying a large language model (LLM) specifically trained for source code along with prompt engineering techniques and heuristics to reduce the level of alert noise that developers would otherwise see when scanning for secrets. The overall goal is to correctly identify access keys, passwords, application programming interface (API) keys and other personally identifiable information (PII) that should not be exposed once an application is deployed in a production environment, he noted.

That capability can be applied to detect secrets across all development assets, including code repositories, source code management (SCM) tools, build tools and logs, artifacts and private and public documentation. In addition, this approach also uncovers toxic combinations of secrets or ones that might lie buried within assets such as source code history or modified Confluence pages that developers often forget exist, noted Caspi.

Legit Security also enables DevSecOps teams to apply preventive guardrails on developer endpoints using the Legit command line interface (CLI) to stop secrets from being exposed before code is pushed. In addition, automated workflows can reach developers as part of a pull request or create Slack messages or Jira tickets to streamline remediation. DevSecOps teams can also continuously triage results to create a baseline that includes exceptions.

As part of a larger effort to secure software supply chains, Legit Security has developed a software-as-a-service (SaaS) platform designed to apply security policies across an entire software development life cycle. The company is now extending the capabilities of that platform using AI technologies to detect secrets in both source code and all the components that make up a development pipeline as they are added.

In the wake of an executive order requiring federal agencies to better secure software supply chains, there is now more focus than ever on application security. Unfortunately, many developers are a little too cavalier when it comes to managing secrets during the application development process. They often leave secrets in plain text to facilitate workflows, but more often than many care to admit, those secrets are not removed before an application is deployed. Knowing this, cybercriminals have learned to scan applications for those exposed secrets. Once gained, they then appear to be just another legitimate user of an application that might not be discovered until malware is activated.

It may be a while before the cultural and technical issues that conspire to make the software supply chain insecure are fully addressed. In fact, the problem may get worse before it gets better—simply because the pace at which distributed applications are being built and deployed only continues to accelerate. At the same, the cost of making a secrets mistake is also only going to increase as regulations become more stringent. In fact, it’s now only a matter of time before stiff fines intended to make an example of one or more unfortunate organizations are eventually going to be levied.

January 23, 2024
Chronosphere Acquires Calyptia to Process Log Data More Efficiently

Chronosphere today announced it has acquired Calyptia to add additional pipeline management capabilities to its observability platform.

Calyptia created Fluent Bit and Fluentd, a set of open source log processors that are now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).

Chronosphere CEO Martin Mao said these log processors will be embedded into the company’s namesake observability platform to enable the routing, transformation and optimization of log data at scale. That issue has become problematic because cloud-native applications based on microservices create exponentially more telemetry data than legacy monolithic applications. If it becomes too costly to analyze that data, then DevOps teams will begin to limit the number of applications they will observe, noted Mao.

The Calyptia platform achieves that goal using filters at the data source to reduce the volume of data that might otherwise be shared by 30% or more.

The shared data can then be analyzed in real time rather than waiting for it to be stored and indexed. In addition, DevOps teams can enrich that data in flight to include additional context.

Calyptia cofounder Anurag Gupta added that the primary mission for the company has always been to make log data more accessible to developers as they build and deploy applications.

The acquisition of Calyptia comes on the heels of a log management alliance between Chronosphere and CrowdStrike through which a log storage and virtualization capability, dubbed Logs powered by CrowdStrike, has been added to the observability platform. The overall goal is to simplify the management and analysis of logs via a single click of a graphical tool or within the context of a larger DevOps workflow based on a command line interface (CLI) without requiring data engineering expertise, said Mao.

It’s not clear how fully DevOps teams have embraced observability, but it’s apparent the legacy monitoring tools that collect a set of predefined metrics are not able to keep pace with the complexity of cloud-native application environments. An observability platform makes it simpler to query log data and discover the root cause of an issue—hopefully before a catastrophic event occurs. The challenge, of course, is providing that capability without incurring massive increased storage costs for log data.

Despite those concerns, the need for observability is becoming a more pressing issue with each passing day. Organizations have never been more dependent on applications to drive revenue. A degradation in performance is usually an early indicator of a disruption that often has a direct impact on revenue and profitability.

There is, of course, already no shortage of observability platforms. The issue most DevOps teams encounter is finding the funding needed to employ one at a time when organizations have become much more sensitive to IT costs. There may be an opportunity to rationalize some monitoring tools to help pay for the cost of a more modern observability platform, but regardless of approach, an ounce of prevention is always going to be worth a pound of proverbial cure.

January 22, 2024
Five Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

CareerBuilder.com

Nasscom, Inc.
Remote
Technical Architect (DevOps)
$160,000 to $170,000

SimplyHired.com

Phillips Auctioneers
New York, New York
DevOps Engineer
$150,000 to $170,000

Indeed.com

Tesla
Fremont, California
DevOps/Platform Engineer
$133,000 to $169,000

LinkedIn

University of Wisconsin: School of Medicine and Public Health
Madison, Wisconsin
Cloud Infrastructure Manager
$120,000

Dice.com

Maximus
Arlington, Virginia
Senior DevOps Cloud Engineer
$62,500 to $158,066

Image source: Carles Rabada (Unsplash license)

January 22, 2024
SREs Say There’s Plenty of Room to Improve Incident Management

A global survey of 423 site reliability engineers (SREs) found diagnosing issues is the most difficult aspect of incident management for more than half of respondents (53%).

Conducted by Catchpoint, a provider of a network monitoring platform, the survey found SREs respond to hundreds of ticketed (84%) and non-ticketed incidents (71%) per month. However, 42% also admitted their organization is not spending as much time as they should learning from major incidents, the survey found. Nearly half (47%) said learning from incidents has the most room for improvement in overall incident management activities. More than half of respondents (55%) are included in on-call rotations, but only 42% of SREs actually lead post-incident efforts.

Leo Vasiliou, director of product marketing for Catchpoint, said the survey makes it clear that SREs still don’t have enough visibility into their IT environments. In fact, 64% of respondents said they should be monitoring endpoints that, despite being beyond their control, could impact application environments.

Overall, two-thirds of respondents (66%) are using between two to five monitoring or observability tools to manage their IT environments. The most widely tracked metrics are uptime/availability (78%), followed by performance/response times (71%), latency (64%) and error rates (64%). A full 81% of respondents had two or more types of telemetry feeding their observability frameworks, with 43% having four or more.

Just under a quarter (24%), however, also acknowledged their organization breached a contractual service level agreement (SLA) in the last 12 months.

On the plus side, just over half (53%) anticipated advances in artificial intelligence (AI) would make their work easier, with only 4% believing AI will replace them.

There is no doubt organizations of all sizes are more dependent on IT than ever, but as application environments become increasingly complex, the probability an IT issue will disrupt business processes only increases. Hopefully, AI will soon make it simpler to respond to incidents by, for example, providing summarizations of incidents that make it easier to onboard new members to the response team as an incident escalates.

In the meantime, Murphy’s Law will continue to dictate that if something can go wrong, it will. At this point, it’s not possible to prevent every incident, so the focus needs to be on discovery and containment. The only way to achieve that goal is to provide IT teams with the tools needed to mitigate issues before they become catastrophic events. Achieving that goal requires having an incident management plan in place that enables the IT teams to respond adroitly versus, for example, wasting time trying to figure out who on the IT team might be needed to address it.

Clearly, SREs can play a bigger role in incident management. Most of their focus is rightly on preventing issues from occurring in the first place. After all, an ounce of prevention is still worth more than a pound of cure. The challenge, as always, is finding the time needed to prevent issues from occurring when so much of it is being consumed resolving incidents that, unfortunately, often continue to occur at a pace that is too fast and furious for IT teams to manage effectively.

January 22, 2024