devopsy.kinsta.cloud

Category: DevOps Practice

Tips for Increasing Developer Velocity in the AI Age

These days, everyone is trying to improve release frequencies. Under mounting pressure to deliver more, many software teams are considering new strategies to enhance their developer velocity, like AI-assisted code development and platform engineering. AI has been a boon for automatic code generation, but all the code in the world doesn’t matter if you can’t deploy it into production efficiently.

Another point is that velocity isn’t always measured productively, said Dori Exterman, chief evangelist of Incredibuild and a self-proclaimed ‘productivity freak.’ Too often, managers place value on quantitative elements, such as points per sprint and unrealistic metrics, when they really should be considering the quality of these components and how easy they are to maintain in the long run.

I recently chatted with Exterman and other experts about tactics organizations can use to enhance their development velocity. Below, we’ll consider some helpful methods, from optimizing testing automation to creating internal development platforms (IDPs) and evolving your CI/CD processes. Essentially, optimizing the plumbing will be necessary to match the productivity of instant code generation—otherwise, the vision of continual release will remain stunted.

New AI, Same Plumbing…

No tech article today can mention developer productivity enhancements without a nod to AI. Just take the interest in Copilot, for example. Among GitHub Copilot users, 40% of the code they’re checking in is now AI-generated and unmodified, according to a Microsoft executive. And GitHub reported that 88% of Copilot users feel more productive when using the tool.

In contrast to past hype cycles, the latest AI wave is not just a passing fad—it will have long-lasting effects, said Exterman. However, most of our current DevOps processes were constructed for manual code writing, not for the era of AI-assisted development.

If you quadruple the amount of water you use but use the same plumbing system, you can’t push more water through, said Exterman. Similarly, if you amplify the amount of code you’re producing but don’t evolve your build processes, staging and testing, then DevOps suddenly becomes the new blocker.

Ways to Accelerate Developer Velocity

So, how can we improve development efficiency and productivity in the age of AI? Here are some actionable strategies for overcoming these bottlenecks and improving developer velocity.

Accelerating the Plumbing

Exterman stresses the importance of leveraging AI to enhance the foundational processes of development. This involves using techniques such as distributed computing and smart test selection to streamline build and testing processes. By optimizing these foundational steps, development teams can ensure that their coding efforts translate into efficient and rapid progress.

Optimizing Testing through AI

AI’s potential not only lies in code generation—it extends to optimizing testing processes. Exterman suggests tools like Launchable that allow AI to prioritize tests that matter most. This strategic approach to testing enables teams to focus on high-impact areas while maintaining comprehensive test coverage.

Embracing Test Automation

Exterman underscores the importance of comprehensive test automation. Rapid iteration loops are essential for maintaining velocity, and automation facilitates fast feedback cycles. This approach ensures that each code change undergoes rigorous quality checks, preventing errors from propagating downstream.

Keeping an Eye on Technical Debt

One of the potential downsides of increasing programming velocity is the accumulation of more technical debt. As more development occurs, the temptation to compromise on documentation, unit testing and best practices grows. Exterman emphasizes that velocity must be pursued in tandem with managing technical debt to ensure the long-term health of projects.

Investing in Platform Engineering

Recently, we’ve witnessed a paradigm shift toward platform engineering. Just over half of IT organizations have already embraced platform engineering, according to the 2023 State of DevOps Report. Exterman emphasizes the need for a robust internal developer platform that supports the seamless transition of code from development to staging and production. For instance, tools like Terraform and Earthly play a pivotal role in establishing testable environments and reproducible builds.

“Bring more data services closer to the developers,” said Venkat Ramakrishnan, VP, product management and engineering, Portworx by Pure Storage. “Choose platforms that provide easy-to-deploy services that developers can consume and provide visibility and control into the underlying costs.”

Enabling Observability

Observability, according to Exterman, is also important for gauging efficiency. The ability to continuously monitor and log processes enhances the overall development cycle. One report from Splunk even found that 64% report their investment into observability has exceeded their expectations. Tools from New Relic and Datadog, for example, offer insights that tie quality, delivery and productivity together, enabling teams to make informed decisions.

Evolving CI/CD With Continuous Merge

One method to progressing CI/CD is continuous merge, a new concept that can be implemented with tools like gitStream that aims to decrease the window between a pull request, review and merge.

“CI/CD has done wonders to improve the quality and efficiency of the software development toolchain, but it mainly focuses on things that happen after code has been written and reviewed,” said Ori Keren, CEO and co-founder of LinearB. “Continuous merge is a mindset that extends CI/CD practices further upstream to improve the quality and efficiency of the code review process. We’ve reached a point where code review times frequently cause the most significant delays in the software delivery process; continuous merge is how we fix that.”

Using Cloud-Agnostic Platforms

Cloud-native development has become more popular as organizations seek to design software architectures bred for the natural scalability of the cloud. Simultaneously, both technology and business motivators are resulting in a multi-cloud condition. In this scenario, IT really needs agnostic, open platforms to retain flexibility and reduce bottlenecks.

“Enable developers to build their apps in a cloud-agnostic way so they can scale their apps globally,” said Ramakrishnan. “Much of developer and app momentum is lost when scaling apps. Helping the application teams architect it right by providing cloud-agnostic application platforms like Kubernetes enables developers to move faster, scale faster and reach their customers everywhere.”

Measuring Developer Velocity

The above tips should help improve velocity, but how exactly should you measure success? Well, measuring developer velocity goes beyond simply tracking the number of points completed in a sprint, reiterated Exterman. He emphasizes the importance of focusing on the ability to deliver high-quality products to the market swiftly. He underscores the need for a fast feedback loop for addressing bugs and iterating quickly. AI plays a role here, too, aiding in test writing and enabling continuous delivery.

End Benefits of Improved Developer Velocity

Exterman concludes by highlighting the real-world benefits of increased developer velocity. Faster bug-fixing and release cycles, particularly crucial in industries like gaming, result in improved user experiences and customer satisfaction. He emphasized the significance of aligning key performance indicators (KPIs) and objectives and key results (OKRs) with the aspects that truly matter, promoting a culture of measured and sustainable progress.

September 11, 2023
Five Great DevOps Job Opportunities

DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

CareerBuilder.com

Cognizant Technology Solutions
Phoenix, Arizona
Senior DevOps Architect
$130,000 to $180,000

Indeed.com

IBM
Research Triangle Park, North Carolina
SAP DevSecOps SME
$126,000 to $235,000

Dice.com

DGN Technologies
Dallas, Texas
Azure DevOps Engineer
$110,000 to $115,000

SimplyHired.com

Sabre Systems
Lexington Park, Maryland
Cloud Engineer – DevOps
$106,000 to $134,000

LinkedIn

Deloitte
Miami – Fort Lauderdale, Florida
Junior Cloud Infrastructure Architect
$65,000 to $97,000

September 11, 2023
Operational Intelligence: AI-Powered SRE Measurements and Observability

In my recent blog, Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated AI can help analyze vast amounts of data from monitoring and observability systems, identifying patterns and correlations that may be difficult for humans to detect. In this blog, I explain, in more detail, how AI-engineered tools can be used to improve the site reliability engineering (SRE) measurements and observability pillar of practice.

AI Use Cases for SRE Measurements and Observability

Making Applications Observable: AI tools like Dynatrace can provide real-time observability of applications, identifying trends and anomalies that might not be evident from raw data. Challenges here include the potential for overwhelming amounts of data, which can be mitigated by AI’s capacity to analyze and prioritize this data effectively.

Making Infrastructure Observable: Tools like New Relic and Datadog use AI to provide insights into the health and performance of infrastructure. The challenge here can be the complexity of modern, cloud-based infrastructures, but AI tools can model these systems and analyze their behavior to provide actionable insights.

Monitoring Applications in Production: AI can be used to monitor applications in real-time, identifying potential issues before they impact users. Tools like Splunk use AI to monitor application logs, detect anomalies and automatically alert relevant teams. The key challenge here is filtering the signal from the noise, which AI can help by learning which anomalies are significant.

Monitoring Production Infrastructure: AI can monitor infrastructure health and performance, predict maintenance needs and identify potential bottlenecks. Tools like Zenoss use machine learning to predict infrastructure failures before they occur. One of the main challenges here is the diversity of infrastructure components, but AI can handle multiple data sources and types and integrate this data into a comprehensive overview.

Tracing and Logging: AI can help manage the vast amounts of data generated by modern logging and tracing systems, identifying patterns and correlations that might be difficult for humans to detect. Tools like Logz.io use AI to analyze log data, detect anomalies and provide actionable insights. One challenge can be the sheer volume of log data, but AI can handle this by focusing on patterns and anomalies and learning over time which issues are significant.

Production Dashboard Design: AI can help design more effective dashboards by identifying which metrics are most relevant and should be highlighted. Tools like Grafana use machine learning to optimize dashboard layouts based on usage data. The challenge here is determining what information is most relevant, but AI can learn this from usage patterns and user feedback.

Performance Forecasting: AI can help with predicting system performance based on current and historical data, helping to prevent performance issues before they arise. Tools like Virtana use AI to provide accurate forecasts of system performance, helping teams to plan for future capacity needs. The challenge here is the variability of performance data, but AI can handle this by identifying patterns and trends in the data.

Automated Root Cause Analysis: AI can be instrumental in quickly identifying the root cause of issues in complex, distributed systems. Tools like RunWhen and Moogsoft use AI to correlate events across different systems and tools, significantly reducing the time to identify the root cause. The challenge here is the complexity of modern systems, but AI can handle this by analyzing data from multiple sources and identifying correlations that might not be evident to humans.

RoadMap for SREs to Transform Measurements and Observability With AI

Here is a practical roadmap that organizations can use to transition to AI tools for SRE measurements and observability.

Understand the Current State: First, an organization needs to fully comprehend its existing monitoring and observability practices. This includes understanding what tools are currently in use, what metrics are being monitored and where the gaps in visibility exist.

Define Goals and Objectives: The next step is to define what the organization hopes to achieve by implementing AI in its monitoring and observability practices. This could include goals like improving system performance, reducing downtime or predicting system failures before they occur.

Identify the Right Tools: Evaluate different AI tools to help achieve these goals. This includes looking at AI-powered monitoring tools like Dynatrace, New Relic and Datadog, log analysis tools like Logz.io and Splunk and AI-driven observability platforms like Moogsoft and Zenoss.

Test and Iterate: Implement the chosen tools on a small scale initially, using a test environment if possible. Monitor the results and adjust your approach as necessary. This step might involve tweaking the AI models, adjusting the data inputs or even trying a different tool if the initial choice doesn’t provide the expected results.

Training and Upskilling: Ensure your teams are well-versed in using the chosen AI tools. This could include formal training sessions, workshops, or even bringing in external experts to provide guidance. This will help your team to effectively use these tools and get the maximum benefit from them.

Full-Scale Deployment: Once testing has been successful and the team is comfortable with the tools, deploy them across your production environment. Continue to monitor their performance and make adjustments as necessary.

Continual Evaluation and Adjustment: AI tools learn and improve over time. It’s important to regularly evaluate their performance and make adjustments as needed. This could involve feeding them new data types, adjusting their models or retraining them based on the latest performance data.

Summary

Harnessing the power of AI to revolutionize your SRE practices can propel your organization to new heights. AI tools can significantly enhance the measurements and observability pillar of SRE, providing unparalleled insights into application and infrastructure performance, enabling more efficient monitoring practices and ultimately driving improved system reliability and performance. These tools, such as Dynatrace, New Relic, Datadog, Logz.io and Moogsoft, can transform how your organization perceives and interacts with its technological environment, providing crucial visibility and actionable intelligence.

Imagine a world where your monitoring systems not only alert you about a problem but also predict potential system failures before they occur and even recommend preventive actions. With the implementation of AI in your SRE practices, this can become a reality. By following a well-defined roadmap that begins with understanding your current state, defining your objectives, selecting the right AI tools, iteratively testing and upskilling your teams, you can pave the way for a future where improved system performance and reduced downtime are the norms. AI-powered SRE practices today can unlock a new era of operational efficiency and system reliability.

September 8, 2023
The Great CentOS Linux Migration: How We Got Here and What’s Next

The recent news regarding Red Hat’s decision to limit access to the source code of their Red Hat Enterprise Linux (RHEL) distribution has garnered considerable attention. In order to understand the ramifications and subsequent changes, it is important to look back at CentOS, its historical alignment with RHEL and the options moving forward.

CentOS Then and Now

Starting in 2004, right from its inception, every iteration of the commercial RHEL offering triggered a corresponding release of CentOS. These iterations mirrored RHEL, albeit with modifications to eliminate the Red Hat branding and other minor updates. The presence of open source CentOS Linux facilitated swift installation of a free RHEL variant, rendering it accessible for diverse purposes spanning from development to production environments.

In January 2014, Red Hat joined the CentOS project. Under a new CentOS governing board, they became the main sponsor and driver of the CentOS project. They continued to be an open source downstream alternative to RHEL.

On December 8, 2020, the CentOS project and Red Hat set a new end-of-life (EOL) date for CentOS Linux 8. They shifted focus and investment into a new Linux distribution, CentOS Stream, with a rolling release model. EOL means that the community stops releases regardless of the severity of a bug or a vulnerability. Today, CentOS versions 6 and 8 are already EOL, and version 7 is the last, with EOL set for June 30, 2024. This announcement immediately sparked new projects to fill the CentOS gap. New open source projects such as Rocky Linux and AlmaLinux took off quickly, gaining contributors, sponsors and users.

Fast forward to June 21, 2023, when Red Hat announced it would no longer publicly publish RHEL on git.centos.org.

Previously, the source code was accessible to all, and it was used by various RHEL-compatible alternative open source projects such as Rocky Linux and AlmaLinux. It also affected other distributions, including Oracle Linux, which was a RHEL derivative for many years. All of these open source Linux distributions offered an alternative for applications; they were equally compatible with both free open source Linux and the commercial RHEL offering.

According to the 2023 State of Open Source Report, 15% of all organizations and 20% of large organizations globally still have CentOS Linux deployed in their environments. Organizations have to move off EOL CentOS 6 and 8 and, in less than a year, off CentOS 7. Running EOL versions of any software can pose a substantial risk and cause failure in IT internal and external compliance. New vulnerabilities are constantly disclosed, and without updates or patches to CentOS, it is critical to plan for the “Great CentOS Linux Migration.”

CentOS Migration Planning and Challenges

Migration can be challenging. If organizations have legacy applications, they might encounter significant challenges making them work properly on a different Linux distribution. Even a small change in the individual packages that form the Linux distribution can impact and delay testing. While every application is different, and many are not dependent on low-level Linux functions, others could be significantly impacted. The number of CentOS deployments and applications has an exponential impact on the migration efforts. Evaluating different Linux distributions also takes time and will depend on the use cases, the skills developers have within each organization and the results of testing. The actual installation of a new Linux distribution is the easy part—the testing of all the layers on top of the operating system is what will take time. Some might be straightforward while others will require significant changes. Those are the reasons why it is helpful to stay with compatible distributions that will minimize the migration challenges.

As organizations navigate the CentOS EOL and the migration period, it will be helpful to find options to extend CentOS long-term support. This means finding the expertise to generate patches that address newly disclosed high-severity vulnerabilities post-EOL. In other words, organizations must expand the community’s long-term support to address security.

Wait for What’s Next

The biggest question about alternative distributions is how closely they resemble RHEL. Are they going to continue to be bug-for-bug compatible? Or are there enough differences to no longer consider them application-compatible? The Rocky Linux Enterprise Software Foundation announced its plan to access RHEL source code and its intentions to continue to be bug-for-bug compatible. AlmaLinux OS Foundation announced a different approach. It is working with the CentOS Stream community while maintaining binary compatibility. This means not strictly bug-for-bug but focusing on applications to continue to be compatible.

It is possible we’ll see other potential changes from Red Hat with regard to the access to RHEL’s source code. Today, it’s only available to customers under a commercial license agreement.

The big test will be the next minor version, RHEL 9.3. It will be interesting to review the corresponding 9.3 versions of Rocky Linux, AlmaLinux, Oracle and others, including SUSE, who announced their intention to offer an open source compatible Linux distribution. The analysis and comparison of how applications run with the new minor version will reveal how close they are to RHEL.

To mitigate uneasiness, organizations have to plan and test alternatives. Finding post-EOL support and patches will allow organizations to have a runway for migrating CentOS to different Linux distributions.

As everyone in the industry with CentOS deployments waits for the next release or the next few releases, extending long-term support becomes a very viable solution. It is wise to make sure security and IT compliance aspects are addressed. Perhaps more open source Linux distributions will appear in the near future and offer another alternative to CentOS Linux.

September 8, 2023
Oracle Bill is 5x Client’s Budget ¦ Toyota Out of Space

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Birmingham looks like the Detroit of the UK—is it Oracle’s fault?
Plus: Was Toyota’s factory failure caused by running out of disk space? (more…)

September 7, 2023
3 Ways to Achieve Faster App Development Without Compromising Security

Bringing products to market quickly, efficiently and safely is key to brand success, and elite DevOps teams minimize the lead time for changes by quickly and efficiently moving new products and software updates from concept to deployment. DevOps teams often measure lead time in hours, not days, weeks or months, which underscores the centrality of speed in today’s hyper-competitive online environment. Customer experience and brand loyalty are constantly changing as new products and services come to market, and DevOps teams are under immense pressure to stay as agile and efficient as possible. Increased technology development velocity can lead to heightened security risks. As of early 2023, as many as nine out of 10 companies have detected security risks in their software supply chain. This is why companies cannot ignore the rising number of attacks targeting software development teams.

High-profile incidents at SolarWinds, NetBeans IDE, Kaseya and Codecov demonstrate the risks facing software development teams at every company.

These threats exploit the continuous integration and continuous delivery (CI/CD) process to sneak harmful software into authentic applications and install risky and harmful components whenever possible.

The risk of code injections makes safeguarding internal and external dependencies crucial. If not appropriately protected, compromised software pipelines can lead to data leaks, code tampering and system exploitations, putting the entire business at a heightened risk.

Simply put, DevOps teams must maximize development speed without compromising security. This is an undoubtedly tall order, but one that’s achievable with the right processes in place. Here are three strategies DevOps teams can implement immediately to reach this goal.

Streamline Privileged Access Management

Streamlining privileged access management is vital for enhancing cybersecurity in DevOps teams, particularly with the growing threats to software supply chains. This requires adopting stringent security practices, including temporary access permissions that can be granted and revoked as needed—a strategy ideal for managing temporary services in DevOps workflows.

This approach facilitates automatic management of access privileges, including revoking password permissions when team members leave the organization or project and thorough scrutiny of who has access to what.

At the same time, enforcing least privilege access, a principle that aims to minimize the attack surface by eliminating unnecessary permissions and adjusting overly broad ones, is also essential, reducing the chances of account hijacking or theft of static API keys.

Similarly, zero standing privileges are also essential, relying on just-in-time access management that ensures minimal active permissions at any given moment, enhancing security by reducing potential vulnerabilities.

Leverage AI Solutions for Security

The cybersecurity AI sector is soaring alongside escalating cybersecurity threats to DevOps teams.

Deloitte expects cybersecurity AI solutions to become a $19 billion market by 2025, as the technology’s “ability to adaptively learn and detect novel patterns can accelerate detection, containment and response, easing the burden on SOC analysts and allowing them to be more proactive.”

AI can help DevOps teams detect and prevent cybersecurity incidents by quickly identifying and monitoring applications at risk, ensuring greater resilience in their DevOps processes through CI/CD, applying analytics to streamline data challenges, and identifying potential hot spots before they escalate and present real risks.

AI can also help combat alert fatigue within security teams, reduce analyst workloads and cut down on false positives, saving valuable time. According to a 2022 IT industry study, 60% of respondents receive more than 500 cloud security alerts every day, and alert fatigue is “causing critical alerts to be missed at an alarming rate.”

By leveraging AI for cybersecurity, DevOps teams can enhance their defensive posture without diluting their team’s impact, allowing them to best balance speed and security in the app development process.

Enable Communication and Collaboration in DevOps Teams

In the realm of DevOps, success relies not only on implementing the right tools and technology but also on fostering an adaptive culture that encourages seamless communication and collaboration among team members.

Effective communication forms the bedrock of this cultural shift. By establishing clear lines of communication, development and operations teams can better understand each other’s challenges and perspectives. Open dialogues can foster mutual respect and create a shared understanding of common goals, thereby encouraging a collaborative problem-solving approach.

Meanwhile, fostering a collaborative culture is equally vital. When teams collaborate effectively, they can identify and resolve issues more quickly, improve the quality of the software, and accelerate delivery times.

Embracing Speed and Security in App Development

Achieving faster app development without compromising security is a complex yet achievable goal in today’s competitive environment.

By investing in streamlining privileged access management, leveraging AI and fostering a culture of effective communication and collaboration, DevOps teams can enhance both speed and security.

To be sure, in the face of escalating cyberthreats, speed and security are not mutually exclusive but, rather, two sides of the same coin. By implementing these strategies, DevOps teams can navigate this delicate balance, accelerating development timelines while ensuring robust security.

September 7, 2023
Despite Concerns, AI Increasingly Used to Build Software

A survey of 1,001 senior technology executives, developers, and security and operations professionals published this week by GitLab finds 67% are working for organizations already using artificial intelligence (AI) to build software, with 60% of those respondents reporting their organizations use it every day. A full 90% said their organizations are either already using or plan to use AI to build software.

More than half (51%) have already identified productivity as a key benefit, but 32% are also “very” or “extremely” concerned about introducing AI into the software development life cycle (SDLC).

Those concerns included AI-generated code introducing security vulnerabilities (39%) and whether that code may not be subject to the same level of copyright protection as human-generated code (48%). A full 95% of senior technology executives said they prioritized privacy and intellectual property protection when selecting an AI tool.

Most organizations will continue to move forward with AI, but nearly two-thirds of respondents (65%) who use or are planning to use AI for software development said their organization hired or will hire new talent to manage its implementation.

A full 81% cited a need for training to successfully use AI in their daily work, with 75% of respondents saying their organization provides training and resources for using AI. However, a roughly equal percentage also said they are finding resources on their own.

David DeSanto, chief product officer for GitLab, said it’s already clear that AI will have a major impact on productivity simply because 75% of a developer’s time is typically spent on activities that have nothing to do with writing actual code.

Less clear is the amount of expertise required to use AI to build software. In some cases, development teams may need to master, for example, prompt engineering to automate DevOps workflows at scale, but in many more instances, it’s likely the DevOps platform will automatically execute the commands required.

Regardless of approach, it’s clear that as the amount of code being created by developers starts to steadily increase, the need to leverage AI to manage DevOps workflows will become more acute. The challenge is the pace at which developers can leverage AI to write code is, for the moment, exceeding the pace at which it is being embedded into DevOps platforms.

In addition, organizations must determine whether to converge DevOps workflows with the machine learning operations (MLOps) workflows used to build AI models that are integrated into applications. As more MLOps capabilities are added to the GitLab platform, it will become easier to achieve that goal and enable data scientists to work more collaboratively with DevOps professionals, noted DeSanto. Today, a cultural divide between two distinct IT cultures is slowing down the pace of AI innovation.

One way or another, DevOps workflows are about to be transformed. The challenge now is determining which tasks and bottlenecks are likely to be eliminated by AI to enable DevOps teams to focus on managing applications at an unprecedented scale.

September 6, 2023
Stop Leaking Secrets!

Merriam-Webster defines a secret as something kept from others or shared only confidentially with a few. Modern software development teams use software secrets such as passwords, tokens and API keys to properly set up, protect and maintain development environments and delivery pipelines and to enable programmatic access to services such as AWS Buckets or Azure Blob Storage for cloud-native apps. Yet, while secrets are critical for modern software to function, they are difficult to manage across delivery stages. All too often, software teams trip over the complexities and inadvertently leave these secrets exposed in both public and private software repositories.

Potential exposures can stem from many factors: Plain text passwords, weak cryptography, flawed CI/CD or packaging automation and compromised developer accounts or malicious insiders. Whatever the cause, bad actors are regularly scouring repositories and application binaries for hard-coded secrets they can exploit. Look no further than the CircleCI and CodeCov incidents, where exposed secrets left these organizations vulnerable to adversaries intent on attacking software supply chains.

Rethink Secrets Prioritization

According to GitGuardian analysis, 10 million new secrets occurrences were exposed on GitHub in 2022, a 67% increase compared to the company’s 2021 findings. As software becomes larger and more complex–with open source, third-party and commercial components–the number of detected secrets will only increase, creating a different challenge: Prioritization for remediation.

Regrettably, teams lack a scalable means to understand which detected secrets developers can’t or shouldn’t remediate. Today, teams must manually sift through potentially thousands of detections and understand their context to triage these threats. Gaining context means discovering where secrets reside (in your code, in a software dependency, in the documentation included with your software) and understanding their purpose (to access resources, to enable user testing, to enable intrusion detection).

As teams consider how to simplify and automate the prioritization of secrets, important questions to consider include:

● What secrets are your developers responsible for protecting? For example, secrets used to access internal CI/CD pipeline technologies. Depending on permissions, leaking these secrets enables malicious actors to change automated build processes with no one being the wiser. Similarly, API credentials to access data or services needed for their code to function. Leaking these secrets gives malicious actors programmatic access to those APIs, which can be a gold mine of customer data or further compromise the environment.
● What secrets are worth protecting? There are some you may be responsible for but which are not worth protecting. For example, to facilitate automated software testing, developers typically publish API keys and certificates in public repositories alongside their applications or in documentation bundled into release packages or software containers. Since these testing “secrets” are designed to be commonly shared, they should not be encrypted or protected.
● Which detected secrets are not actionable? For example, developers typically cannot manage secrets in third-party or commercial components and dependencies within software. Instead, these exposures represent a third-party risk management problem. Another example is canary secrets added to software by security operations; remediating these secrets hampers intrusion detection.

Once teams have established their secrets definitions, building a process to identify and prioritize them comes next.

Managing Secrets

Any organization involved in software development should have clear rules for handling and incorporating secrets into code and for communicating that process and those rules to developers. Here are secrets best practices every organization should take into consideration:

● Promote developer best practices — and monitor compliance: Secrets leaks are more likely to occur in development organizations that lack clear guidelines for properly managing secrets in their code and easy-to-use tools to implement those guidelines. DevOps teams should formulate policies based on an understanding of how secrets are generated within their development environment, how those secrets are protected once deployed and how (hopefully) they are eventually cycled out of use.

● Revamp how applications use secrets: Design applications in a way that eliminates the reliance on access tokens that reside in code. Use technologies like OAuth OpenID Connect (OIDC) tokens that allow developers to authenticate users without directly exposing user credentials in code. Reduce the risk of abuse by using multi-factor authentication, leveraging identity and access management platforms and setting IP ranges that limit inbound connections for APIs to known addresses or address ranges.

● Establish end-to-end security: Misconfigurations in CI/CD tooling or release oversight can result in the inclusion of secrets during final builds that might otherwise escape notice during a code audit. Consider implementing binary analysis to extend visibility into leaked secrets beyond raw code and into software packages and containers ready for release. This capability can detect exposed secrets resulting from build or packaging automation.

● Streamline secrets prioritization: Noisy scanning results create a ton of triage work just to weed out false positives (e.g., widely distributed testing keys or network credential placeholders) before remediation can start. Automating some of this triage with relevant file intelligence saves countless hours of effort and ensures developers focus on addressing actual risks.

No Silver Bullet

Unfortunately, there is no “silver bullet” fix for software supply chain risks and attacks, nor for leaks and exposures. Instead, organizations need a holistic approach that includes scoping the problem, addressing short-term risks posed by leaked credentials and then extending changes in all processes with an eye toward taking the toil out of identifying, prioritizing and remediating secrets leaks—and therefore preventing potential supply chain attacks before they can happen.

September 6, 2023
Five Great DevOps Job Opportunities

DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

LinkedIn

Icertis
Phoenix, Arizona
Principal Azure DevOps Engineer
$132,000 to $199,000

SimplyHired.com

Decision Sciences International
Remote
DevOps Engineer
$130,000+

Indeed.com

Hewlett-Packard Enterprise
Durham, North Carolina
DevOps Engineer
$124,500 to $286,300

CareerBuilder.com

CACI
Oklahoma City, Oklahoma
DevOps Platform Engineer
$104,200 to $229,200

Dice.com

SPECTRUM
Greenwood Village, Colorado
DevOps Engineer
$78,900 to $139,500

September 5, 2023
Internal Developer Platform (IDP) Reference Architectures
If you’ve been following the DevOps space recently, you’ve probably noticed the excitement brewing around platform engineering. Despite uncertainty in other areas of tech, it seems like platform engineering is poised for growth—Gartner even expects that by 2026, 80% of software engineering organizations will have established platform teams.

The platform engineering idea hinges on an internal tooling system known as an internal developer platform (IDP), which platform teams own and operate and is exposed to developers to improve their efficiency, satisfaction and release fluidity. IDPs have been the talk of the town. But what does an IDP actually look like? What sort of components are they usually composed of?

Well, at PlatformCon 2023, McKinsey representatives shared a reference architecture for building out IDPs. This framework was modeled after hundreds of internal platforms and could be a helpful benchmark for other organizations as they begin developing and standardizing their internal developer platforms.

Below, we’ll examine the sample IDP reference architecture and its components. We’ll also have a refresher on the platform engineering concept and offer some expert tips on overcoming potential obstacles while implementing platform engineering and IDPs.

Why Engineer Internal Platforms, Why Now?

First, for those new to the idea, let’s explain what platform engineering is. I’ve previously defined platform engineering as “maintaining an internal self-service platform that serves the needs of developer consumers.” The practice doesn’t necessarily supplant DevOps, rather, the experts I’ve spoken to view it as the next evolution.

This trend is essentially a response to developers overseeing more and more aspects of software life cycle management. Complexity in enterprise-level software delivery workflows has escalated in the last decade, leading to what The State of Platform Engineering Report describes as “increased cognitive load on engineers.” This is negative, as it could lead to shadow operations and burnout.

The fact is, most developers require the same sort of technologies, which could be standardized across the board. For instance, take the consolidation around containers and Kubernetes, infrastructure-as-code and the need to implement similar build configurations and security policies across applications.

By outsourcing the maintenance of tools for shipping and running cloud-native applications to a common platform, you take the burden off developers and free them up to code. Also, by centralizing DevOps tooling, you can enforce more governance and build out internal standards. Which should help reusability and overall deployment agility … at least, that’s the promise of it all. 😉

Sample IDP Reference Architectures

So, let’s turn to some real-world examples to see how this is being implemented. Mike Gatto, senior DevOps engineer, McKinsey, shared an internal platform reference architecture for AWS-specific environments:

McKinsey’s Internal Developer Platform reference architecture

As you can see, the architecture is divided into these subgroups. At a high level, they perform the following functions:
- Developer control plane: This is the version control, where developers publish their code. They provide a workload specification (in McKinsey’s case, a Score file) that codifies specific configurations.
- Integration and delivery: This includes the CI pipeline to build application code into a container and publish it to a registry. This notifies the platform orchestrator to initiate a build. Then, resources are provisioned. This area also includes a CD pipeline and infrastructure control plane.
- Monitoring and logging: This plane includes tools for observability, analytics, and application performance monitoring.
- Security plane: This area includes tools for secrets management, policies, and security vetting.
- Resource plane: This plane encompasses cluster management, database tools, networking, and messaging architectures.
“By separating developer and platform team concerns and allowing them to control the respective areas they’re experienced in separately, without requiring too much overlap,” said Gatto, “we’re making deployments more compliant and better aligned with client initiatives.”

Humanitec has also created internal developer platform reference architectures for GCP and Azure, displayed below.

IDP Reference Architecture for GCP

IDP Reference Architecture for Azure

PlatformEngineering.org has also expanded McKinsey’s original IDP reference architecture with a tooling landscape that adopts the same umbrella subgroupings but includes other pluggable tools.

The Internal Developer Platform tooling landscape

Roadblocks to Developer Platforms

Starting from a blueprint could be a big help to rapidly construct IDPs. Yet, there will still be some potential roadblocks associated with adopting an internal developer experience platform. One of them is creating common platforms that actually cater well to all environments.

“You can not and should not try to please everybody,” said Kaspar von Grünberg, CEO and co-founder of Humanitech. “Platform engineering is an 80/20 game.”

“Platform organizations often face a lot of pressure when it comes to meeting demands for building more services that are both agile and agnostic to cloud and data center environments,” said Venkat Ramakrishnan, VP, product management and engineering, Portworx by Pure Storage. “At the same time, almost all platform engineering teams are understaffed while facing the need to support large application development initiatives and developer teams twice their size.”

Tips to Get Platform Engineering On Track

Reference architectures for various clouds could help cater IDPs to different scenarios. But outside of that, it will take a product mindset approach to strategically accept feedback and iterate the platform. This will involve doubling down on developer experience and adopting cloud-native foundations that can scale.

“Platform engineering teams must consider adopting Kubernetes as it enables them to manage and deploy applications in production at scale, without having to manage applications at the individual level,” added Ramakrishnan. “Additionally, it may not be realistic to expand platform engineering teams alongside growing platforms, so teams should also consider leveraging automation and self-service capabilities to help them better manage projects at scale.”

Is ‘You Build It, You Run It’ Dead?

The idea of ‘you build it, you run it’ is starting to look a little antiquated next to the new and shiny platform model. It’s tough to say, but I’ll be keeping an eye on this area to see if all the effort to standardize the internal engineering approach pays off!
September 5, 2023