devopsy.kinsta.cloud

Category: Enterprise DevOps

Five Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

Indeed.com

MedStudy
Remote
Senior DevOps Engineer
$125,000 to $165,000

SimplyHired.com

Raytheon
Tewksbury, Massachusetts
Technical Fellow – DevSecOps/Cloud Native
$122,000 to $155,000

CareerBuilder.com

State Street Corp.
Quincy, Massachusetts
Azure DevOps Engineer
$120,000 to $202,500

Dice.com

USEReady
New York, New York
Senior Azure DevOps Engineer
$120,000 to $160,000

LinkedIn

Northrop Grumman
Dulles, Virginia
Senior Principal DevOps Engineer
$107,300 to $199,600

December 18, 2023
The Future of Jenkins in 2024

As software development practices evolve, the tools that support them must also adapt to stay relevant. Jenkins, the venerable automation server used by many DevOps teams, is at such a crossroads as it faces the changing landscape of cloud-native applications and Kubernetes adoption.

This article examines Jenkins’ current utility, particularly its features that have made it a staple in CI/CD practices, and addresses the question of whether it will maintain its relevance in 2024 and beyond.

How Jenkins is Used by DevOps Teams

CI and CD

Jenkins has become an integral part of many DevOps teams due to its CI/CD capabilities. Continuous integration (CI) is a software development practice where developers regularly merge their code changes into a central repository. Continuous delivery (CD) is the practice of delivering the software in short cycles to ensure that it can be reliably released at any time.

Jenkins supports the CI/CD approach in a number of ways. First, it allows developers to automate code integration. This means that whenever code changes are pushed to the repository, Jenkins can automatically build and test the project. Second, Jenkins allows for the continuous delivery of software by automating simple deployment processes (although it lacks full deployment orchestration capabilities).

Build Pipeline Visualization

Another key feature of Jenkins is its build pipeline visualization. This feature allows DevOps teams to visually represent the process of software delivery from code integration to deployment. The pipeline visualization provides a clear and intuitive representation of the software delivery process, making it easier for teams to understand and manage the process.

Build pipeline visualization in Jenkins is customizable, allowing teams to tailor it to their specific needs. For example, it can be customized to show the status of each stage of the delivery process, from code integration to testing to deployment. This allows teams to quickly identify any issues or bottlenecks in the process and address them.

Automated Testing

Automated testing is another area where Jenkins shines. With Jenkins, DevOps teams can automate the process of testing their code by running scripts and integrating with automated testing tools as part of the build process.

Jenkins supports a wide range of testing frameworks and tools, allowing teams to choose the ones that best fit their needs. Moreover, Jenkins can be configured to automatically run tests whenever changes are made to the code. This ensures that any issues or defects in the code are detected early and can be addressed promptly.

Workflow Scripting

Workflow scripting is another key feature of Jenkins in DevOps teams. With Jenkins, teams can script their workflows, automating the process of software delivery. This allows teams to standardize their workflows, ensuring that each delivery process is consistent and predictable.

Jenkins supports the Groovy scripting language for workflow scripting. This means that teams can use the power and flexibility of Groovy to script their workflows. Moreover, Jenkins provides a user-friendly interface for scripting workflows, making the process easy even for those who are not familiar with Groovy.

Current Usage and Adoption Statistics

As of 2023, Jenkins remains a leader in the CI/CD space. With its extensive plugin architecture and large user base, it is used by thousands of organizations worldwide. According to the 2022 DevOps Survey by the Cloud Native Computing Foundation (CNCF), Jenkins was used by over 50% of respondents for automation and CI/CD, maintaining a significant presence in the market.

According to the Jenkins website, there are over 1 million active Jenkins users and more than 200,000 active installations worldwide, with the platform handling 1 million+ jobs per day. It had about 44% of the global market share for CI/CD in 2023.

The growth trajectory of Jenkins, however, has shown signs of plateauing when compared to newer, more cloud-native-focused CI/CD tools. This observation is backed by Google Trends data, reflecting a steady but not sharply increasing interest in Jenkins over recent years.

Despite facing competition from emerging tools designed for cloud-native development, such as GitHub Actions, GitLab CI and CircleCI, Jenkins has responded by evolving its ecosystem. The creation of Jenkins X is a strategic move to cater to Kubernetes users and address some of the criticisms related to cloud-native development. However, as we’ll discuss, Jenkins X has not been readily adopted by the industry as of yet.

Limitations of Jenkins in Cloud-Native Environments

While Jenkins is a powerful tool for DevOps teams, it has serious limitations in cloud-native environments. These limitations stem from the fact that Jenkins was originally designed for traditional, monolithic applications, not for the dynamic and distributed nature of cloud-native applications.

Statefulness

One of the critical limitations of Jenkins in cloud-native environments is its inherent statefulness. Cloud-native architectures typically favor stateless applications, which are easier to scale and manage in dynamic environments like Kubernetes.

Jenkins, however, relies on maintaining state—it stores configuration, build history and user data, making it challenging to scale and manage efficiently in a cloud-native setup. This statefulness requires persistent storage, which complicates deployments in distributed systems where resiliency and horizontal scaling are key. Moreover, the risk of data loss or inconsistency increases in cloud-native environments, particularly during scaling operations or when recovering from failures.

Configuration Complexity

Jenkins is often criticized for its complex configuration requirements. Setting up Jenkins in a cloud environment involves intricate configurations that are not only time-consuming but also prone to errors. This complexity is exacerbated when integrating with modern cloud-native tools and services, requiring extensive customization and maintenance of plugins.

In addition, Jenkins’ traditional master-slave architecture was not designed for dynamic cloud environments, where resources need to be provisioned and deprovisioned rapidly. This leads to a significant operational overhead for DevOps teams.

Immutable Infrastructure

Adopting an immutable infrastructure approach is a cornerstone of cloud-native environments, promoting the idea of replacing or rebuilding servers rather than modifying them.

However, Jenkins’ traditional model does not align well with this principle. It typically requires modifications to its runtime environment for updates or changes, contrasting with the cloud-native approach of immutable deployments. This can make it challenging to maintain consistency and reliability in cloud-native systems where automation and minimal manual intervention are paramount.

Microservices Limitations

Jenkins was originally designed for monolithic architectures and doesn’t naturally fit into microservice-based applications, which are prevalent in cloud-native environments. In microservices architectures, applications are broken down into smaller, independently deployable services.

Jenkins, however, tends to treat applications as monolithic units, making it less efficient for handling multiple, independent microservices. This limitation affects its ability to provide fine-grained control over individual services’ build, test and deployment processes. Consequently, Jenkins may struggle with the high degree of automation, flexibility and scalability required in microservices environments.

What is Jenkins X?

Jenkins X is an open source project that provides pipeline automation, built-in GitOps, and preview environments to help teams collaborate and accelerate their software delivery at any scale. It is an improved version of the original project, designed specifically for Kubernetes, a popular container orchestration platform.

Jenkins X streamlines the technical aspects of building, testing, and deploying software. It automates the installation of popular tools like Helm and Draft, making it easier for developers to get started. It also provides pre-configured pipelines so teams can start delivering software right out of the box.

Jenkins X also introduces a new Jenkinsfile format, Jenkinsfile.yaml, which provides a more straightforward and human-readable way to describe pipelines. This new format, combined with the automation provided by Jenkins X, helps reduce the complexity of pipeline configuration and management in Kubernetes.

Limitations and Criticisms

Jenkins X has not been widely adopted by the software industry and has not achieved the respect and prominence of its predecessor, Jenkins. Let’s review the main limitations and criticisms of the tool.

Lack of Maturity

Jenkins X’s relatively recent entry into the market means it has not had the time to mature like its predecessor. This lack of maturity can manifest in several ways, including fewer features compared to established tools, fewer community-tested plugins, and potentially more bugs or unresolved issues. Due to the fast-paced evolution of cloud-native technologies, tools need time to adapt and stabilize; Jenkins X is still in this phase of its lifecycle.

Moreover, a mature tool typically has extensive documentation and a large community of users who can offer support and share best practices. While Jenkins X has seen growth in its user base, it still lags behind more mature CI/CD solutions in these areas. This can lead to challenges when teams are looking to solve complex problems or when they require advanced support.

Limited to Git Projects

Another criticism of Jenkins X is that it’s limited to Git projects. While Git is the most popular version control system, it’s not the only one. Some teams may use other systems like Mercurial or SVN, and these teams would be unable to use Jenkins X.

Even within the world of Git, Jenkins X has limitations. For example, it doesn’t support all Git hosting platforms. While it works with popular platforms like GitHub and Bitbucket, it doesn’t support others like GitLab. This can be a significant limitation for teams that use unsupported platforms.

Community Support

Finally, there are concerns about community support for Jenkins X. While the project is open source and theoretically has the potential for wide community involvement, some critics argue that it’s primary source of support is CloudBees, the company behind Jenkins.

These critics worry that if CloudBees were to withdraw its support, the project could falter. They also argue that the project’s focus on certain tools and platforms (like Kubernetes and Git) reflects CloudBees’ business interests more than the needs of the broader DevOps community.

Conclusion

In conclusion, Jenkins’ role in the CI/CD landscape has been pivotal, and its continued evolution with projects like Jenkins X demonstrates an effort to adapt to the needs of modern, cloud-native environments. While Jenkins X has yet to reach the widespread adoption and maturity of its predecessor, its specialized focus on Kubernetes and GitOps reflects the shifting priorities in software development.

Jenkins, despite showing signs of plateauing in growth, still possesses a strong user base and an established presence in the market, supported by a vast plugin ecosystem and a robust community. Its limitations in cloud-native contexts—such as managing statefulness, configuration complexity, and supporting immutable infrastructure—pose challenges that Jenkins X seeks to address.

As we approach 2024, the relevance of Jenkins will likely hinge on its ability to integrate more seamlessly with cloud-native technologies and the willingness of the community to evolve its ecosystem. The breadth of its adoption and the depth of its integration in existing development pipelines mean Jenkins will continue to be a significant player in the DevOps toolchain. However, its long-term relevance will depend on the project’s ability to innovate and remain adaptable in a landscape that is rapidly embracing cloud-native solutions.

December 15, 2023
Copado Expands Beta of Generative AI Tools for Salesforce DevOps Platform

Copado today announced it has expanded a beta program through which it provides application development teams building applications for the Salesforce software-as-a-service (SaaS) platform access to generative artificial intelligence (AI) capabilities enabled by ChatGPT.

David Brooks, SVP of Evangelism for Copado, said the company, via the usage of vector databases, has exposed instances of ChatGPT to everything in its knowledge base that has been collected over the last 10 years. That approach enables Copado to extend the large language model (LLM) used to train ChatGPT without exposing any data that might be used to one day train the next version of the generative AI platform.

The goal is to enable DevOps teams to generate code and tests faster while simultaneously reducing the time required to handle support calls, said Brooks.

In addition, Copado is using ChatGPT to enable impact analysis and conflict detection to reduce the level of friction encountered when deploying applications in a production environment. That latter capability is critical because, in many instances today, DevOps teams don’t have a firm understanding of how code needs to be deployed, resulting in unnecessary rollbacks of application deployments, noted Brooks.

Generative AI will also automate the creation of documentation, such as release notes for every application deployed, to reduce the overall level of toil DevOps teams experience, noted Brooks.

Copado also plans to provide governance tools that will enable DevOps teams to ensure the code written by an LLM is not based on a hallucination that results when trying to reconcile conflicting data, added Brooks. There will, of course, always be a need for a human to review code before it is deployed in a production environment, he added.

In the longer term, Copado will make it possible for organizations to invoke multiple LLMs to address various functions as they best see fit, said Brooks. In fact, DevOps teams should expect to soon be employing multiple copilots trained to address specific tasks such as writing code, deploying applications, assessing cybersecurity and addressing compliance mandates, he added.
Ultimately, application development and deployment efforts that today require weeks to complete will soon be accomplished within the same day, said Brooks.

At the same time, organizations will also be able to better manage application environments that today operate at levels of scale few IT teams can manually track. Generative AI capabilities, in effect, will make it simpler for DevOps teams to effectively manage application environments at much higher levels of scale, said Brooks.

Each organization will naturally need to determine to what degree they are prepared to absorb that level of innovation, but it’s clear the pace at which applications are built and deployed is about to exponentially increase. The issue organizations will increasingly encounter is that business processes will not be able to absorb the rate at which software can be built and updated.

It’s not clear just yet to what extent generative AI will transform DevOps, but as advances continue to be made, many of the manual tasks and bottlenecks that today conspire to slow down application development and deployment are about to be eliminated in ways that few would have thought possible a little more than a year ago.

December 13, 2023
Things To Do During Slow Times
We all know it happens in North America; once the U.S. Thanksgiving holiday hits, things start to slow down in IT—from vendor salespeople busy with family to coworkers out of the office in larger numbers. Since all regions and most religions have holidays clustered around the new year, the worldwide slowdown goes on for a good month and a half, sometimes—and in some locales—even longer.

And how we experience that slow time varies pretty widely. For some, it’s time to be completely off, like those coworkers. For others, it is time to finish those tasks that have been simmering on the back burner. For still more, it is a time of frustration as the things they’ve been working on grind to a halt. Back in the network computing days, we would do large, complex technical evaluations of the gear the vendors sold to enterprises. I covered mostly storage and servers with some security at the end, but my lab had more security, some software and some networking. We needed vendors to participate, and participation mostly meant traveling to our labs for any physical product, to set it up and let us test it. Needless to say, this slowed to a (sometimes frustrating) crawl as people reduced travel closer to the new year. I’ve felt that same frustration as an architect—when we couldn’t get vendors to respond and you really can’t blame them for doing the best they can in a time of seriously reduced staffing.

So, what can we do that doesn’t tie us to others who may or may not be available?

You know the list specific to you far better than I do, but here are some general suggestions to ponder. Most of these I have done during slowdowns.
- Learn that new language or framework. We all have one that the org is interested in, but in which we are weaker. Pick it up in those quiet hours; you can slap on headphones and focus with fewer interruptions.
- Same with new tools like test generators, DLP, GitOps or whatever your org is considering implementing. Heads down; figure it out in a test bed, and become at least versed and, at best, an SME.
- Clean up all that cruft you know you left behind over the past year. We’ve all got dead-end branches, stray text files, outdated bash scripts … Dynamic environments spew extraneous artifacts more than historically was the case. Take that time to go clean it up. I had a large script a couple of years ago that spun up a bunch of containers and ran tests. When I went to clean up, there were three different versions of the script and several orphaned containers still running. Good to get taken care of.
- Research, research, research. Whether it is something like reading a product’s site, checking crowdsourcing tools like G2 or looking at what some of the better/smarter analysts out there have to say, pick the tool you think would benefit the org, validate that it will and maybe even start documenting the use case and ROI justification (if you use ROI for consideration of new products; benefit analysis if not).
- Catch up. While generally the least fun use of this extra time on the job, it is often the most practical. We all have a ton of work, and sometimes the peace of mind that cranking out backlog offers is worth not doing the other things on this list.
- Take the time off. Seriously. Most of us (though no longer all of us) got into IT because we loved the technology and the promise. Walk away, take a breath and don’t even think about it for a week or two. Then, come back; I bet a bit of that wonder will return, making the job more fun and you more adaptable. It works if you can manage it. As others have said for many years, walking away and rediscovering a hobby can be invigorating, too.
Don’t sweat it, and don’t totally waste it. Figure out the best use of that time and move on it. There are a few spaces that simply do not slow down for holidays, and for those people, this blog is unlikely to be useful now, but at some point, slowdowns will arrive. I was a team lead at a tax software company. We slammed the holidays prepping for tax season, then got short weeks all summer. The mental checkout of teammates was similar to the end of year but delayed until early May. So we did all of this then.

But no matter what or when slowdowns happen, you’ve knocked out another year of keeping the org running. So, celebrate in some way; you deserve it.
December 13, 2023
DevOps Strategies for Connected Car Development

The concept of smart, connected vehicles is no longer science fiction but a day-to-day reality. These vehicles, empowered by internet connectivity and a suite of digital technologies, offer unprecedented convenience, safety and entertainment features.

But as the cars on our roads become smarter and more interconnected, the complexity of developing their software systems multiplies. DevOps is emerging as an indispensable strategy for organizations in the connected car industry, bringing together software development and IT operations to streamline, optimize and accelerate the development life cycle.

This article briefly reviews connected car technology and explores several DevOps strategies that can support connected car development. We’ll discuss how DevOps can manage the complexity of automotive software, accelerate development speed without compromising safety and provide the essential tools for efficient building, testing and deployment in a highly regulated environment.

What Are Connected Cars?

Connected cars are vehicles equipped with internet access, allowing them to share data with devices both inside and outside the car. Essentially, they offer a multitude of features that make the driving experience more convenient, safe and enjoyable. While this technology is still in its nascent stages, it’s rapidly evolving, and soon, connected cars will become a staple on our roads.

These hyper-connected vehicles are part of a broader ecosystem that encompasses a network of physical objects—everything from other vehicles and traffic lights to pedestrian crosswalks and parking spaces—all connected by technology.

The potential of connected cars is immense. They promise to transform our lives in numerous ways, from reducing traffic congestion to improving road safety and revolutionizing the way we travel.

Key Features and Systems of Connected Cars

Infotainment Systems

An integral part of connected cars is the infotainment system. This comprehensive system provides both entertainment and information tailored to the driver’s needs. From navigation and music streaming to voice commands and hands-free phone calls, these systems are designed to enhance the driving experience while ensuring safety.

Infotainment systems also offer a wide range of services, such as real-time traffic updates, weather forecasts, and points of interest. They can connect with a smartphone, allowing users to access apps, make calls and send messages without taking their eyes off the road.

Telematics

Telematics uses GPS and onboard diagnostics to record and transmit data about the car’s performance, location and usage. It allows for real-time tracking of the vehicle, making it particularly useful in fleet management and stolen vehicle recovery.

Telematics data can also be used to assess driver behavior, monitor fuel efficiency, and predict maintenance needs. For instance, if the car’s engine starts showing signs of trouble, the telematics system can alert the driver and recommend a nearby service center.

Advanced Driver-Assistance Systems (ADAS)

Advanced driver-assistance systems (ADAS) are systems that help the driver in the driving process. These systems use sensors, cameras and radar to monitor the vehicle’s surroundings and assist the driver in various ways.

ADAS can provide features like adaptive cruise control, lane-keeping assist and automatic emergency braking. These systems not only enhance driver comfort but also significantly improve road safety by reducing the risk of accidents.

Vehicle-to-Everything (V2X) Communication

Vehicle-to-everything (V2X) communication is a key technology in connected cars. It enables vehicles to communicate with any entity that may affect the vehicle and vice versa. This includes other vehicles (V2V), infrastructure (V2I), pedestrians (V2P) and networks (V2N).

V2X communication can improve traffic flow, reduce accidents and enhance fuel efficiency. For instance, connected cars can receive information about road conditions, traffic jams or accidents ahead in real-time, allowing the driver to take appropriate action.

Over-the-Air (OTA) Updates

Over-the-air (OTA) updates are another key feature of connected cars. Similar to how you receive updates on your smartphone, OTA updates allow car manufacturers to update software, fix bugs or add new features remotely.

OTA updates can improve vehicle performance, enhance security and extend the vehicle’s lifespan. They can also add new functionality, allowing the car to stay up-to-date with the latest technology without the need for a physical visit to a service center.

Why DevOps is Essential for Connected Car Development

Addressing the Complexity of Car Software Systems

The connected car is a complex ecosystem of software systems. These vehicles have numerous systems that communicate with each other, the driver and the outside world. Managing the development of these systems can be a daunting task, and this is where DevOps strategies come in.

DevOps aims to shorten the system development life cycle and provide continuous delivery with high software quality. This methodology is particularly suited to the complex software systems of connected cars, as it encourages a holistic view of the development process, ensuring that all components work together seamlessly.

Moreover, DevOps helps to manage the complexity of car software systems by automating tasks, reducing errors and improving efficiency. The use of automated tools for configuration management, deployment and monitoring means less manual work, fewer mistakes, and quicker problem resolution.

Enhancing Development Speed Without Compromising Safety

One of the greatest challenges in connected car development is the need for speed. In this fast-paced industry, companies are under pressure to develop and deploy new features quickly to stay competitive. However, this must not come at the expense of safety, as any faults in the system can have dire consequences.

With DevOps strategies for connected car development, it’s possible to enhance speed without compromising safety. The DevOps approach emphasizes a culture of rapid and frequent deployments, coupled with rigorous testing to ensure quality. By integrating development and operations, it enables teams to work together more effectively, leading to faster development cycles.

Furthermore, the use of automation in DevOps allows for quicker detection and resolution of issues. Automated testing tools can run thousands of tests in a fraction of the time it would take a human, ensuring that any faults are identified and fixed before they can cause harm.

Building, Testing and Deploying Efficiently in a Regulated Environment

The automotive industry is heavily regulated, with strict safety and emission standards that must be met. These regulations add an extra layer of complexity to the development process, as they must be considered at every stage of design, testing, and deployment.

DevOps strategies for connected car development are well-equipped to handle this challenge. By implementing a continuous integration/continuous deployment (CI/CD) pipeline, teams can ensure that all changes are tested and validated against regulatory requirements before they are deployed. This not only ensures compliance but also saves time and resources by preventing issues from reaching production.

Additionally, DevOps encourages transparency and visibility across the development process, making it easier to demonstrate compliance to regulators. Through the use of version control systems, every change can be tracked and traced, providing a clear audit trail.

DevOps Strategies for Connected Car Development

Version Control for Connected Car Development

In any software development project, version control is vital. It allows developers to track and manage changes to their code, making it easier to collaborate and preventing conflicts. In the context of connected car development, version control becomes even more critical due to the complex and interconnected nature of automotive software systems.

Version control enables multiple teams to collaborate on projects without stepping on each other’s toes, using techniques like branching and merging. Furthermore, version control systems provide a history of changes, which is invaluable for debugging and understanding how the system has evolved over time. This can be particularly useful in a regulated industry like automotive, where there may be a need to provide an audit trail.

Continuous Integration and Continuous Deployment (CI/CD)

CI/CD is a core DevOps practice that involves integrating changes to the project regularly and automatically testing those changes to detect and fix problems early. The aim is to provide rapid feedback so that if a defect is introduced into the code base, it can be identified and corrected as soon as possible.

For connected car development, a robust CI/CD pipeline is essential. It allows for fast-paced development without compromising quality or safety. By automating the build, testing and deployment processes, teams can ensure that new features and updates are delivered quickly and reliably.

Automated Testing and Quality Assurance

Given the complexity of connected car systems and the potential consequences of a malfunction, rigorous testing is of paramount importance. However, manual testing is time-consuming and prone to error. That’s where automated testing comes in.

Automated testing is another key DevOps strategy for connected car development. By writing test scripts that can be run automatically, teams can save time, reduce the likelihood of human error and ensure comprehensive coverage of the system.

Moreover, automated testing tools can be integrated into the CI/CD pipeline, allowing tests to be run every time changes are made. This provides immediate feedback on the impact of those changes and ensures that any issues are identified and addressed as early as possible.

Infrastructure-as-Code (IaC) for Connected Car Environments

The final piece of the DevOps puzzle is infrastructure-as-code (IaC). IaC is the practice of managing and provisioning computing infrastructure through machine-readable definition files rather than physical hardware configuration or interactive configuration tools.

In the context of connected car development, IaC can be a powerful tool. It allows teams to create and manage their development, testing and production environments in a consistent and repeatable way. This not only enhances efficiency but also ensures that the environments are configured correctly and securely.

Moreover, by using IaC, teams can automate the creation and teardown of environments, saving time and reducing the risk of human error. IaC also supports version control, enabling teams to track changes and roll back to a previous state if something goes wrong.

Conclusion

In conclusion, connected cars represent a significant advancement in the automotive industry, but they also bring a host of development challenges. By adopting DevOps strategies for connected car development, companies can navigate these challenges and unlock the full potential of this exciting technology. From managing complex software systems to ensuring rapid and safe development, DevOps offers a robust and flexible approach that is tailored to the needs of the connected car industry.

December 8, 2023
2024: The Year of Testing

Caveat: I currently cover DevOps – including test – and security – including the *AST. This does give me a viewpoint that others may not have, and also potentially gives me blinders that others may not have. I am not generally a predictor but a reader of direction. Proclamations like this are rare for me and are shorter-term/more tactical than the grab bag of predictions usually seen this time of year.

I was thinking about this one this morning; the direction of travel of traditional and security testing is very similar at the moment. We are seeing the vast complexity of the overall space and the rapid growth of automation come together to once again make all testing a topic worthy of consideration.

I loved test-driven development (TDD). I can’t speak for others, but RPC/API is just a cool way to implement centralized functionality, and TDD is almost mandatory to develop APIs and the clients that call them. There is no UI to distract or have separate requirements in API development; there is a stated function and correct or incorrect behavior. To a computer scientist, this is predictable beauty. TDD made it more than predictable—it made it a trail through the woods. “We know it needs to do X, Y and Z, with the only side effects being A,” can be detailed with tests to prove that is what it does, and then look to prove the “It doesn’t have other side effects” part. And you can do it all while developing—no need for dev/test iterations. Like any IT solution, it isn’t a great fit for everything, but even if it isn’t called TDD, it is nearly impossible to develop API-based solutions without some level of TDD.

The thing is, TDD is absolutely a developer’s tool. Developers and DevOps pretty much must use it to turn out usable APIs. The test team only has a use for a subset of those tests. Because if the dev team did it right, they passed all the TDD tests, and the test team’s role in that scenario is to look closely at what the developers might have missed. Integration is often beyond the control of an individual dev and/or API, for example, because many different clients may be integrating with it. Test teams, in a true TDD shop, fill the gaps with meaningful tests that evaluate the overall application quality.

Security testing—at least static application security testing (SAST) and increasingly dynamic application security testing (DAST)—are also becoming a core part of the development process. This is as it always should have been. It is far easier to fix a security flaw (or any flaw) at the design/early code stage than in a testing process after deployment. The quality of security testing inputs is also increasing, making it far more viable. And finally, the tools are being effectively integrated. Being a developer who did a lot of security work because we often needed it and had no one to do it, I will say I get a little rush about being told right in my IDE about the security risks in my code. I write, I see, I revise where necessary, and I turn out better code. All without having to pause for security or bang heads over policy.

To date, all this testing is separate, but I suspect, having vaguely the same location in the process of development, that testing will all occur together and the markets may even start to merge a bit—though merger of security testing and functional/performance testing tools is a bit of a stretch, it is more possible than in the past. In most orgs, even though developers have an increasing stake in both sets of testing, the target purchasers are still two different groups—security versus test—with different priorities. That’s why I think they aren’t likely to merge any time soon.

While AI in most technology spaces is currently experiencing the same level of FUD as XML, Java, JavaScript or low-code was in the past, all super-hyped tools in tech do have a sweet spot. Security and testing are in that sweet spot for AI. Both in test generation and in results evaluation, it does it better, faster and with more coverage while giving staff a concrete set of results to pore over and improve security posture and/or code quality. I cannot say loudly enough that in terms of enabling the steps in the SDLC that have suffered from a lack of resources and timeline to write, execute and evaluate tests, AI is a game-changer.

So, all of these things together mean that 2024 is a good time to take a hard look at your testing environment(s) and architecture, both traditional testing and security testing. Now that AI has made advanced automation a fact, it is time to consider implementing the level of testing we always knew we should have but never had the resources. Build it right into the build process and make it part of the organizational DNA—just like Agile is.

If you could hire a specialist who would paw through your code and give you a list of the 25 things you need to fix first, you would do it. That is, essentially, what modern testing tools are. They can do more tests, evaluate more issues and side effects and filter down to the “No—this is really important!” list without costing the team a ton of time.

And keep rocking it. We’re all sitting on bugs and vulnerabilities. I do hope we are way past denying that fact. Apps are big old complex monsters; these days, sometimes built from generations of technologies cobbled together. Of course they have issues. Even yours. So let’s take steps to clean it up, just a little bit. And keep clipping along.

December 6, 2023
Five Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

SimplyHired.com

Selby Jennings
Chicago, Illinois
AWS DevOps Architect
$191,000 to $242,000

Indeed.com

Verizon
Alpharetta, Georgia
DevOps Architect
$159,000 to $201,000

LinkedIn

JetBlue
New York, New York
Senior Principal DevOps Engineer
$134,600 to $210,100

Dice

Northrop Grumman
Dulles, Virginia
Senior Principal DevOps Engineer
$120,900 to $181,300

CareerBuilder.com

Octo
Washington, D.C.
Platform DevOps Cloud Engineer
$110,000 to $170,000

December 4, 2023
It’s All About Communication

Things get crazy, don’t they? I mean, IT, even with Agile and DevOps, is a very interruptive endeavor. When something in production breaks, whatever you were working on suddenly just got a lot less important. Agile is aimed at making more stable code, and DevOps is focused on shifting troubleshooting left, but inevitably, IT systems – be they ones we wrote or ones someone else did and we manage/maintain – are complex systems, and they break unexpectedly. As time goes on, things like test-driven development (TDD) or continuous testing have steadily improved quality and reduced interruptions, but we still say things like “Oh, the container crashed, spin up a new one and keep moving,” when that crash should be a work-stopping event. If it crashed once, it will again, and it only takes a few minutes to find some of the horror stories out there caused by doing exactly what I just described. There is a line between “resilient application infrastructure” and “We ignore critical failures because we can.” And one side of that line is pretty ugly.

So, with all of our advances, we are still facing unexpected issues cropping up – sometimes caused by coworkers or vendors – that we need to deal with.

One thing to remember is that this, too, is part of your job. Start there. While most of us aspire not to be “the fix-it guy,” if you wrote the code or managed the infrastructure, you are the right person to fix the unexpected problem. So, no stress, but others (like management) might be stressed themselves, so communicate clearly. A five-minute email or thirty-second Slack message can save a ton of frustration and meetings later. A simple “This came up and is a priority, so that thing I was set to deliver tomorrow will be delayed by at least X amount of time,” will do a lot to help others understand the impacts on your daily flow.

I’ve worked a fair amount of my life as a contractor, and I admit that I honed the skills in the above paragraph living that life. While contractors are generally paid by the hour, ownership is a bigger deal, but in the end, contractors are there to do what needs doing, and the client decides exactly what that is. The same applies when not a contractor, though also, and in my experience, it applies double or triple if you are in charge of a team – you have an entire team, and you better communicate if an issue so big it side-tracks deliverables occurs. We had one of seven inputs to a massive data pool that was the source of billing fail on one team I managed. I mailed stakeholders in our other projects with a simple, “The company likes to get paid, so our attention is 100% on restoring that feed; your project is delayed.” This example was one (of two when I led that team) that was pretty extreme, but it’s a good example of effective communication; just let them know. Most business leaders are there for the org while focused on their piece of it, and if they have an issue they can hash out the business’s priorities and come back to you.

And keep nailing it. Errors and issues and vendor relations failures happen. You’re the ones minimizing the disruption and keeping the org rocking. Don’t stop or get discouraged when it sets you off your timelines; just communicate … and keep being the star.

November 22, 2023
CISA, NSA Issue Supply Chain Security Guidance Report

The National Security Agency (NSA), Office of the Director of National Intelligence (ODNI) and Cybersecurity and Infrastructure Security Agency (CISA) have issued guidance to assist software developers and suppliers to shore up software integrity and security across various stages of the software lifecycle.

The report, Securing the Software Supply Chain: Recommended Practices for Software Bill of Materials Consumption, was developed by the Enduring Security Framework (ESF) Software Supply Chain Working Group.

The report offered recommended practices as a framework for assessing and improving security practices concerning software development, release, updates, vulnerability notifications and mitigation.

The guidance also addressed the escalating cybersecurity threats targeting software supply chains, which have exposed vulnerabilities susceptible to exploitation by state adversaries.

It highlighted potential risks such as design flaws, integration of vulnerable third-party components, infiltration of supplier networks with malicious code and malware injections in deployed software.

To combat these threats, the report underscored industry best practices, emphasizing open source software management and software bills of materials (SBOM).

Specific areas covered include SBOM consumption, life cycle, risk assessment and operational implementation, ultimately promoting enhanced transparency and risk management throughout software life cycles for organizations.

Alex Kreilein, vice president of product security at Qualys, said while the concept of an SBOM is not new, organizations now have concrete and foundationally important guidance on how to make operational use of SBOMs while developing, managing or consuming them.

“The guidance supplied by the Enduring Security Framework reduces the complexity of SBOM management, making it accessible to everyone for the first time,” he noted. “SBOMs are part of a healthy software ecosystem, but are by no means the only factor.”

He explained that while the SBOM is an effective output, it cannot be trustworthy if the system that created it cannot validate the provenance of the product.

“It’s critical for organizations to understand the whole supply chain from code through pipeline to production,” Kreilein said.

He added that this is why The Linux Foundation and Google started the Supply-chain Levels for Software Artifacts, or SLSA framework, as a practical set of methods to action NIST recommendations in the Secure Software Development Framework (SSDF).

John Gallagher, vice president of Viakoo Labs at Viakoo, called the report a “major step forward” in providing an overall framework and transparency for how SBOMs are constructed, maintained and used.

“One of the key ways this helps relieve concerns over software supply chain security is by providing clear guidance on the risk scoring mechanisms,” he said.

This augments other software supply chain efforts (though it doesn’t replace them) in areas like the use of SBOMs in SaaS environments or version control mechanisms for SBOMs.

He noted two significant points on implementation that come out of this framework; first is that it gives guidance to departments like procurement as to what they should be requiring from their vendors, and second it gives a framework for solution providers to base their consumption and generation of SBOM and Vulnerability-Exploitability eXchange (VEX) data on.

“While this document does not address the methods of remediation, it does help to more quickly identify software supply chain vulnerabilities so that they can be addressed early on and ideally before the widespread use of that vulnerable code,” Gallagher said.

He noted that the more vulnerabilities that are caught early or before they are broadly used the later stage of remediation is either much easier or not needed.

Anthony Tam, manager of security engineering at Tigera, said when vulnerabilities have been identified, it is important to prioritize them based on severity and potential impact.

“This allows organizations to allocate their resources more effectively and focus on the most critical vulnerabilities first,” he explained.

Prioritization can be done by using a risk-based approach, considering factors such as the likelihood and potential impact of a vulnerability being exploited.

“By prioritizing vulnerabilities, organizations can ensure that they are addressing the most critical security risks first and reducing the overall risk to their software system,” Tam noted.

November 21, 2023
It’s Funny How We Forgot About Container Sprawl

…Because it sure as heck did not forget about us.

We are (thankfully) past the days when everything was a container. At least, most of us are. Just like anything else, some orgs will continue that approach for reasons both great and terrible. And all of us still use far more containers than we need, I would say. “One container, one function,” works if you are talking about one whole application, and maybe sometimes for busy parts of an application, but it certainly isn’t needed to host a couple of low-use APIs, for example. My favorite example is separate instances of the same exact application for different teams (not business units – that is a good method to separate business units).

Thankfully, technology is once again saving us from ourselves and our misdeeds. Processing power is no longer an issue, meaning we can spin up a million containers and keep them running until our public cloud bill arrives. So, it’s not about processing power. I cannot remember the last time that an org was bandwidth-limited, so it’s not about the network. The biggest issue is still security. More containers means a bigger footprint, which inevitably creates more vulnerabilities. Right behind that is container inventory management. What is running where, what images are available and what state are they in, which running containers have drifted from their definition, etc.

I’m a developer that has sometimes done support for the apps I develop. That means that I laugh at your “Our images do not drift because we’re agile and build-driven.” Of course they do, because troubleshooting is real. The real question is, can you get them back into alignment when the ticket closes? The DevOps toolchain is pretty darned good at that if you let it. Changing a running image and telling no one is a terrible idea that still happens nearly every day across our industry. But technology has brought us drift monitoring. We can be notified of changes as they happen. For those in highly sensitive spaces like FinTech, changes can be blocked in production, but for most of us, we want to know when it is happening and who made the change. And we can get that today.

Technology has also brought us some wonderful scanning tools from SBOM to SCA to Registry. These tools can give us a list of things for security to worry about in our images. In order to reliably generate these lists, we still have to know what images we are keeping and where—but choose a reliable registry or two, fit them into your DevOps processes and scan on commit. Simple and thorough—as long as everyone uses them.

Thanks to policy and regulation, container management tools can offer you a detailed list of not just what instances they’re hosting, but metadata like what image they came from. That helps manage inventory, and by now we are all building hierarchies of images from base to application specific.

So, we can tell what is running, what images are available to run, how far running images have drifted, what known vulnerabilities exist in both our images and our instances and how compliant we are WRT containers. Dunno about you, but it seems technology has our back yet again.

You are cranking it out, keeping the lights on, and dreaming of more you could be doing. Use the tools we already have to make it easier and the resulting systems more secure and reliable. You won’t regret it. And once again for those of you whose users never say it: Thank you.

November 15, 2023