devopsy.kinsta.cloud

Tag: CI/CD pipelines

Infrastructure-as-Code (IaC): The Key to Agile and Automated Cloud Deployments
My early years as a technical support engineer at Divox International provided me with the opportunity to gain hands-on experience with the difficulties encountered in manual infrastructure management. One incident stands out — we spent over 48 hours recreating a production environment for a client. Although having detailed documentation was helpful, the 48-hour crunch involved the added pressure of working with manual processes, which introduced inconsistencies that eventually led to significant failures. This experience provided me with a realization that automation and consistency are paramount for infrastructure management. My years of experience encompass a journey from a TechOps engineer at Mesame Technologies to a DevOps specialist at Woolworths, closely mirroring the industry’s move toward automation and agility. Infrastructure-as-code (IaC) has been at the forefront of this transformation, allowing teams to define, version and deploy infrastructure with the same ease and reliability as application code. Deployments are not just swifter under this new regime — they are also more reliable and scalable, which is vital in today’s helter-skelter digital economy. So, is IaC a panacea for poor cloud deployments? I will share my thoughts on the subject.

Evolution of Infrastructure Management

While I was working as a TechOps Engineer at Mesame Technologies, the provisioning of infrastructures was a tedious manual process that required long runbooks and conferring with multiple console-based configuration commands. These methods were highly prone to human errors, leading to unnecessary outages and serious inefficiencies in the company’s infrastructure.

As a DevOps engineer at RCG Global Services, I observed the ever-increasing intricacy of infrastructure demands. It was becoming untenable to rely on manual infrastructure configuration, with project timelines often hanging in the balance. We had to find a way to provision infrastructure that was not only consistent but also reliable and efficient at a large scale. In this context, undemanding simplicity made IaC a natural choice.

The infrastructure teams can define it declaratively. Hence, these setups have reproducibility, and there is a risk minimization associated with manual setups. Using Terraform and CloudFormation, one can provision environments and scale them within minutes. You can do this with a kind of speed that eliminates configuration drift and minimizes deployment time.

Core Benefits of Infrastructure as Code

As a senior system programmer at FIS, I led a project that overhauled our processes for deploying a complicated payment-processing system. Previously, setting up new environments manually took days, and the result was a series of inconsistently configured systems. Implementing IaC changed all that with:
- Dependability and Consistency: Each environment was defined in code, guaranteeing uniform setups spanning development, testing and production.
- Efficiency and Speed: We reduced the deployment time from days to hours, which, in turn, improved our time to market.
- Cost Optimization: Infrastructure expenses were reduced by 30% due to automated scaling policies and resource provisioning.
- Improved Teamwork: Storing infrastructure specifications in Git boosted teamwork for development, operations and QA. Visibility and governance increased and everyone was on the same page.
Essential IaC Tools and Practices

Throughout my career, I have made extensive use of Terraform and AWS CloudFormation to control my infrastructure.
- Terraform: Its declarative syntax and multi-cloud support made it our tool of choice for efficiently managing our AWS resources, Kubernetes clusters and hybrid cloud environments.
- CloudFormation: At TSYS, we leveraged CloudFormation for workloads centered on AWS, owing to its efficient integration with AWS services and its strength in handling rollback scenarios.
- Kubernetes and GitOps: At Workoi, we extended IaC principles to Kubernetes, using Helm and Argo CD and ensuring version-controlled, automated deployments.
Implementation Strategy

At ITC Infotech, I created a framework to assist organizations in their transition to IaC.
1. Begin Increments: Establish the fundamentals in a non-critical environment where it is safe to take risks and where final practices can be honed to perfection.
2. Team Training: At Woolworths, Terraform and CloudFormation workshops were held to upskill teams before full-scale adoption.
3. Management of State and Secrets: Secure handling of Terraform state files and secrets with AWS S3, Parameter Store and HashiCorp Vault prevented configuration drift and security leaks.
4. Automated Testing: Integrating tfsec, checkov and kube-bench into CI/CD pipelines guaranteed security and compliance.
5. Iterate and Expand: After validation, broaden the application of IaC to include additional services, ensuring the use of standardized, reusable modules throughout.
Best Practices and Patterns

My extensive experience across several organizations has helped me pinpoint vital best practices that guarantee successful IaC adoption:
- Modular Design: At FIS, we succeeded in creating reusable Terraform modules for VPCs, IAM roles and EC2 instances — virtually cutting our IaC redundancy in half and making our IaC maintainable.
- Security by Default: We embedded security policies within our IaC templates, ensuring compliance right from the get-go.
- Collaboration and Documentation: We maintained copious documentation and enforced pull requests for all infrastructure changes, inducing a culture of shared ownership.
- Testing and Validation: We implemented syntax validation, got good at running unit tests and even did some chaos engineering — all to better ensure that our deployments were resilient and fault-tolerant.
Future Trends and Innovations

The field of IaC is developing consistently, eliciting thrilling progress:
- AI/ML Integration: At Woolworths, we looked into machine learning models that would analyze infrastructure usage patterns dynamically to improve our deployments.
- Serverless IaC: With serverless computing emerging, managing AWS Lambda and Google Cloud Functions with IaC tools is becoming crucial.
- Fully Autonomous Infrastructure: The future points to infrastructure that can heal itself, drive itself with policies and optimize itself using real-time system metrics.
Conclusion

My work has given me firsthand experience of how IaC alters cloud infrastructure. The move to using code for automated, reliable cloud deployments is a necessity in modern cloud computing.

But today, building programs that control, provision and manage cloud infrastructures requires more than just coding skills — it demands strategic vision, stakeholder alignment and a well-planned change management process to upskill and motivate the people who will work in the new cloud-native model. And honestly, it takes some chalk smoke, using a few old-time and some new tools to get folks over the finish line and across the moat to the next tier in the cloud — higher agility and resilience. The payoffs are better cost efficiency, rapid scalability and a cloud that works more like a digital garden and less like a smoke signal.
May 8, 2025
Optimizing CI/CD Pipelines for Developer Happiness and High Performance

As organizations strive for faster, more reliable software delivery, continuous integration and continuous deployment (CI/CD) pipelines function as the critical arteries of modern development.

However, when these essential pathways become slow, unreliable or require excessive manual intervention, they transform from conduits of innovation into significant sources of friction. This inefficiency does not just delay builds; it actively erodes the developer experience (DevEx), hindering productivity, lowering morale, contributing to burnout and ultimately stifling the pace of progress and innovation.

“The #1 way you can optimize your CI/CD pipelines is to identify and leverage tools that reduce the amount of work that your developers have to invest in the building and maintaining your CI/CD pipelines,” says Kai Tillman, senior engineering manager at Ambassador, an API development company focused on accelerating API development, testing and delivery. “Tools that replace manual steps for environment creation, deployment and testing with simple commands can have an outsized impact on the overall experience building and maintaining the pipelines, allowing for more time to be focused on other tasks.”

Focusing on DevEx through strategic CI/CD optimization can transform the development lifecycle and significantly improve developer productivity. However, it is even more important to first examine the costs of slow pipelines and understand the disruptive nature of flaky tests. Additionally, consider the impact of rapid feedback loops, intelligent automation and streamlined workflows. These factors collectively emphasize how a well-optimized dev loop reduces friction, accelerates iteration cycles and fuels innovation and business success.

I spoke with notable engineering leaders to understand the detrimental effects of inefficient CI/CD on DevEx, as well as the best engineering practices to streamline workflows, reduce wait times and accelerate feedback loops — learning from them why optimizing CI/CD is a fundamental investment in developer happiness and high performance.

How Sluggish CI/CD Drains Developer Experience

Mudit Singh, VP of Product and head of growth at LambdaTest, shares insights from his company’s recent velocity tours — meeting engineering leaders across organizations of all sizes, from

entry-level developers to executive leadership. “Nearly all of them agree that a sluggish CI/CD pipeline does more than delay build times or slow deployment frequency — it erodes the very fabric of a team’s morale and productivity.”

Slow or bottlenecked pipelines create a negative feedback loop. “Overall productivity takes a huge hit when teams have to wait for development feedback on individual commits,” Mudit highlights. “Issues that could be quickly resolved instead take longer to debug, leading to delayed fixes and compounding stress across team members, especially when a breakdown happens just before a critical deployment. That means extended work hours for all teams, sometimes even wasted weekends.” But the impact extends beyond individual frustration. “A sluggish CI/CD process does not just delay deployments; it disrupts the developer feedback loop and creates an environment of stress and uncertainty that permeates the entire organization,” he notes.

Beyond productivity and morale, a poorly functioning CI/CD pipeline can also erode trust across the wider business. Steve Fenton, director of developer relations at Octopus Deploy, describes this as a trust-loss spiral. “When developers struggle to get changes quickly and reliably through the CI/CD pipeline, it doesn’t just slow feedback. A more damaging effect is the loss of trust. Though developers are doing their best to deliver features without regressions or deployment failures, when changes are delayed or cause customer-impacting issues, the business loses confidence in their ability to deliver.” This often leads to increased bureaucracy and slower processes, further exacerbating the problem.

Mary Moore-Simmons, VP of Engineering at Keebo, highlights the toll on engineering happiness and efficiency. “Engineers are typically the most expensive people in a company and making them wait for builds to finish or forcing them to manually fix flaky tests is a major productivity killer,” she explains. “Long pipeline runtimes also force engineers to context switch between different tasks, and context switching is one of the biggest drains on developer velocity.” When engineers are frustrated with inefficient processes, their productivity suffers — and, even worse, they are more likely to leave for companies that prioritize a better DevEx.

Practical Strategies for CI/CD Improvement

Improving CI/CD pipelines and, by extension, DevEx requires a focus on identifying and eliminating bottlenecks while implementing smart automation. Fenton emphasizes the importance of looking for queues. “The crucial insight for improving CI/CD pipelines and developer experience is to search for queues. Code review times, manual testing and heavyweight change approval processes are common choke points that slow work. Delays between development, completion and deployment to production are like cracks in a rock — water seeps in, freezes and breaks the rock apart. Developers are frustrated as they start new work but get interrupted by previous work that is still stuck in the system. And so, context switching increases, reducing developer effectiveness and increasing the chance of errors.”

Moore-Simmons advocates a strong testing strategy and thoughtful pipeline design. “Having a strong testing strategy can be an underutilized technique to optimize CI/CD. When building any product or feature, engineers should create a plan for what should be tested and think deeply about what is most critical to test at each step in the pipeline (local dev, staging, etc.). This allows the team to test the right things at the right time, which can increase the speed of delivery and reduce CI/CD friction. I recommend thoughtfully splitting CI/CD pipelines for your services so teams can operate quickly and more independently. This can be a great way to speed up testing and deployment while empowering each team to fine-tune their testing and deployment processes to maximize both quality and speed for their unique services.”

“We strive to embody the very principles we advocate at LambdaTest when it comes to intelligent test automation,” Mudit shares. “We know faster feedback loops are important, but there is also a significant need to invest in richer, context-aware feedback. That means better reporting and using the right tools to get actionable insights at every stage, especially in testing. Integrating automated test intelligence tools can analyze test failures and generate targeted root cause analyses and even remediation suggestions,” he continues. “This approach not only tells developers that something went wrong but also helps them understand the root cause quickly — whether it is an environmental issue, a dependency failure or even a flaky test. By embedding this intelligence into the pipeline, teams can address problems more efficiently and reduce the cognitive overhead of sifting through raw logs and error messages.”

Automation That Makes Developers More Productive

Automation is the key to a high-performing CI/CD pipeline, but it is crucial to automate the right things in the right way to truly benefit DevEx. Tillman recommends the use of

developer-focused tools. “By leveraging the Blackbird-hosted environments, you can reduce the overhead in infrastructure development needed to get test clusters going. Blackbird’s deployment commands provide an easy way to get code under test and evaluation into this environment with a simple command. Blackbird can be leveraged in your pipeline to keep resources like Mocks up-to-date when changes are made to an API’s OpenAPI spec. All these together can help simplify your pipelines by handling much of the time-consuming overhead for you.”

Moore-Simmons provides an example of smart automation that fosters collaboration: “One smart automation in a CI/CD pipeline is using a linter to drive alignment and collaboration on coding standards and best practices. When teams get together and agree on code quality and standards, it can be a great opportunity to build strong communication and collaboration practices.

Memorializing those standards in a linter, which is regularly reviewed and updated by the team, allows you to drive continuous alignment. This sets clear expectations for engineers and creates a sense of shared goals and purpose on the team, which reduces conflict during code reviews.”

“Manual deployment is the most common trigger for the trust-loss spiral, where mistakes in sequencing or parameters result in broken software,” Fenton emphasizes. “Similarly, scripted deployments creak under the weight of complexity, and nobody feels confident editing them to introduce improvements. Deployments are too fundamental to continuous delivery to be performed by hand, with checklists or using homegrown scripts.” He advocates for a highly automated process based on Lean principles.

Prioritizing DevEx in Your CI/CD Journey

Improving CI/CD and DevEx requires a conscious and ongoing effort, driven by leadership. Tillman, from his engineering purview at Ambassador, highlights the need to reduce developer toil through smart tooling. He sees Blackbird improving the CI/CD journey by “replacing multiple manual steps with a sequence of simple commands” and “leveraging the Blackbird-hosted environments,” which reduces infrastructure overhead. His simplest advice for developers is to find tools that eliminate as much bespoke work as possible within CI/CD pipelines.

“DevOps research highlights the importance of automation and also reminds us that Lean principles can guide improvement efforts,” Fenton stresses. “When a developer has finished coding, the CI/CD pipeline should be a streamlined, high-automation process that validates the new software version and gets it deployed.”

Improving CI/CD pipelines and DevEx is an iterative process that never ends. Moore-Simmons advises against trying to do everything at once: “Gather regular feedback from your engineers, use that to determine the highest-priority and highest-impact work, scope it down as much as practical, tackle the top one or two items and then repeat. Improving CI/CD pipelines and developer experience is like peeling back the layers of an onion: Once you solve your biggest problem, you will find more things to fix.” She encourages teams to “find joy in the continuous improvement process and do not forget to look back and reflect on how far you have come since you started.”

“Conversations with engineering leaders during my recent velocity tours revealed that many of the challenges we assumed were solved years ago — like integrating automated tests directly into CI/CD pipelines — are still very much there,” Mudit notes. “And in many enterprises, automation is not fully integrated into the build processes, with teams still manually triggering tests after every build. The automation of automation is still missing in a lot of companies, and that needs urgent attention.”

Tillman reminds, “as dev teams face increasing code complexity, security concerns and the demand for faster feature releases, optimizing CI/CD ensures stable and automated deployments that keep pace with growing demands. CI/CD optimization is no longer optional, it is about survival.” Investing in CI/CD optimization is ultimately an investment in the success and well-being of your developers, which in turn drives innovation and business growth.

April 16, 2025
DevOps and Hybrid Cloud: Life in the Fast Lane?

Scream if you want to go faster! When it comes to a dual hybrid-cloud-and-DevOps rollout, most organizations would admit they’re eager to get going a little bit faster. But what started so enthusiastically can turn to frustration (and a different kind of screaming starts) when they’ve accelerated into this major transformation without considering all the necessary elements.

The benefits are clear. Hybrid cloud backed up by DevOps is an efficient vehicle for change. A robust DevOps platform can be a make-or-break factor in developing a hybrid cloud strategy that goes the distance. Together, they are a force for transformation, and if organizations aren’t already implementing both DevOps and hybrid cloud, they should be asking themselves why not.

It is true that the move to hybrid cloud is a different journey for cloud-enabled organizations than it is for cloud-native ones. The trip is harder for cloud-enabled, particularly for larger enterprises. Increased release frequency requires tighter alignment and collaboration than is traditionally seen between lines of business, development and IT operations, and this, in turn, drives the need for enhanced collaboration, automation and information transparency.

But although the transition might be harder, it must be seen as inevitable. Organizations could wait another year, maybe a little more, but the transformation of technology and business is going to happen—so why not get started? In the first of two articles, I’m going to take a look at what needs to change so that organizations can start living life in the hybrid cloud and DevOps fast lane.

A Cultural Phenomenon

As has been covered in these pages in some detail already, the move to hybrid cloud backed by DevOps is doomed to fail if organizations think all they need to worry about is tooling and hiring. It’s not. Underlying both of these transformations is a sweeping cultural change. DevOps is about agility, trust and autonomy, and so is hybrid cloud.

Most of all, though, DevOps and hybrid cloud are a commitment to enhancing an organization’s developer experience (DevX), and that’s not something that can be achieved with simply a ticketing tool or by adopting a GitOps approach. DevX means autonomous, unbound movement. It means empowerment, user experience and self-service. These are changes that can only happen with a successful change in business culture, not a tool. And this is an important starting point that many organizations are still struggling to understand.

The People Problem

People are another well-worn topic, but one that continues to present a major challenge. The cultural transition required to successfully implement DevOps can, theoretically, start with just one person. But for a successful outcome, it actually requires long-term change as deeply-ingrained beliefs are disbanded and tribal loyalties are broken down.

For instance, many believe that implementing DevOps requires hiring new people. That belief embraces two fallacies: One, that organizations will be able to hire the teams they need and two, that DevOps is just a job title. As we know, a good DevOps team is not hired, it is developed through the melding of development and operations. The old silos, where devs developed and ops operated, is a relic from less-enlightened times.

Today, it requires learning how to build automation, often open source, while managing on-premises, private and public cloud infrastructure. DevOps is a philosophy that seeks to help others, and with that comes the need for collaboration, encouragement, persuasion and other soft skills.

Security and Privacy Are not Straightforward

Moving to a hybrid cloud means that more care needs to be taken with regard to security and privacy. One of the main problems is that it’s no longer enough for security teams to reign from on high. With control moving from the server room to the cloud and from highly-siloed ops engineers to the whole team, security policies also need to change. Competent developers will make the right decisions, but they need the right bottom-up tooling that takes their decisions from theory to reality and safeguards those who aren’t as aware.

These must be a catalyst to make organizations sit up and take a long look at what security in the age of cloud, containers and microservices really means. If the simultaneous move to DevOps and hybrid cloud is that catalyst, then so be it. Just as DevOps is revolutionizing the way teams approach their work, DevSecOps is going to revolutionize the way they look at security.

Security shows its face at various stages of infrastructure: When a cloud organization is created and structured; when a vault is integrated to ensure that key passwords are secured and even when a pipeline is launched that triggers dedicated security software. This isn’t an exhaustive list, but it illustrates how organizations need to start taking a methodical, step-by-step look at security in their hybrid cloud.

What About GitOps?

Gartner recently found that the average company uses 28 infrastructure tools. That’s simply too many to be handled and understood by one person. Even specialist DevOps engineers can’t be experts in all the cloud providers, OpenStack and VMware. Other studies showed that a developer generally only uses 5% of their Git repositories or an average of five repositories.

The mistake lots of companies make is thinking that suddenly shifting to using APIs with a GitOps approach is adequate to handle the change to hybrid cloud. It’s not, and can actually be detrimental to DevX. Why? There’s too big a gap—it’s like just doing a long weekly run the day before a marathon. You might start, but you’ll be severely compromising your ability to meet your goal. Even DevOps engineers struggle to maintain a single source of truth and well-maintained reusable repositories.

The solution is to plan for the long term. It’s not realistic to believe that one or two DevOps engineers can be the saviors of a hybrid cloud approach. There are far too many tools, best practices and options—between Ansible, Terraform, Kubernetes and Concourse CI/CD to name just a few—to be managed by a handful of devs, no matter how talented they are.

That’s why the true way to hybrid cloud mastery lies in the ability to get the whole tech team pulling in the same direction, collaborating, upskilling and leveraging their strengths to support the parts that make the whole. This might sound like a pipe dream, but there are companies out there talking the talk and walking the walk.

Start Your Engines

It is futile to tell an organization exactly how to get started since many are already taking the first steps and every organization’s situation is unique. There is more to cover; in the next installment, I’ll be taking a look at areas like FinOps, automation and cloud waste. Hopefully, you’ll see the fast lane is clear up ahead.

May 23, 2022
Continuous Delivery Foundation Adds Interoperability Project

At a CDEventscon event this week, the Continuous Delivery Foundation (CDF) announced it is hosting a CDEvents project through which it hopes to create a vendor-neutral specification for defining the format of event data across multiple services, platforms and systems.

Andrea Frittoli, open source developer advocate at IBM, co-creator of the CDEvents project and member of the CDF technical oversight committee, said the open source consortium is launching this initiative to foster interoperability across continuous delivery (CD) platforms. In the context of the CDF, that mission spans both continuous integration and continuous deployment platforms that are subsets of CD.

CDEvents itself is based on an existing CloudEvent specification created by the Serverless Working Group within the Cloud Native Computing Foundation (CNCF). As part of the effort, the CDF has already built a proof-of-concept implementation that enables Tekton pipelines and Keptn orchestration tools for managing application life cycles to communicate over a common event protocol.

Collecting metrics from multiple DevOps platforms is a challenge because there is no standard for sharing event data. Analytics providers and value stream management platforms that collect metrics need to create connectors for each platform they support. CDEvents would provide a standard interface for collecting metrics from any developer tool or platform. That data would also make it easier to visualize workflows that describe artifacts and their metadata, he added.

That lack of a common way to describe events means developers not only must constantly re-learn how to consume events, it also limits the potential for libraries, tooling and infrastructure to facilitate the delivery of event data across environments ranging from routers and tracing systems to software development kits (SDKs).

Naturally, it may be a while before this continuous delivery specification results in increased interoperability across DevOps environments, but the need for one across what has become a patchwork of fragmented tools and platforms is self-evident. Many organizations are already investing in an assortment of analytics tools in the hopes of improving the rate at which applications can be developed and deployed. In too many instances, DevOps teams have inadvertently created bottlenecks that slow down the rate of development. The challenge is that, as new tools and platforms are added to a DevOps workflow, they are not likely to be able to share data with those applications until a connector is created.

Not everyone, of course, is a fan of having more metrics. The goal should be to coach teams on how to become more efficient rather than implementing a set of draconian mandates. Arguably, such mandates are only going to drive away developers that are trying to strike a balance between coding as an art form and application development as a craft.

Nevertheless, as organizations better appreciate the critical role software plays in driving business processes, the greater the desire to measure the pace and efficacy of their application development efforts—to justify the levels of investment in software development they are making.

May 20, 2022
How DevOps is Revolutionizing Embedded Development

There’s something curious happening on the desktops of embedded developers around the world. From home offices to cubes to labs, in lamp-lit rooms late at night and sun-lit rooms early in the morning, developers are doing something different. They’re beginning to implement DevOps practices in their embedded development.

Something quite serendipitous has occurred to make this possible: The parallel rise of cloud-native computing and the explosion of internet of things (IoT) devices and edge technologies. A world with billions of embedded devices connected to the internet all the time, every minute of every day, requires a new way of designing, testing, deploying and maintaining these devices. At the same time, cloud-native computing has introduced continuous delivery (CD) and continuous integration (CI), among other methods, to enable automated software builds that are reproducible. This is creating a revolution in the embedded world.

Until now, embedded software was written and deployed and then remained static in perpetuity because it was isolated. Not anymore.

The Slog, Security Demand New Thinking

Embedded software developers have traditionally faced a monumental slog to build projects. Typically, embedded developers have had to build an entire platform to build new software, including the kernel, the distribution and more. And all of that software would need to get (re)built anew every single time something changed. Everything had to be checked in. Everything had to be rolled out. There was a version control infrastructure team, a CI team, a QA team—and they all had to work together and be aligned to meet the production schedule.

This approach to developing was incredibly slow, costly and clumsy and in no way architected for today’s world of connected devices, IoT and the edge. Consumer product makers (home appliances, for example) simply don’t have the resources to provide frequent updates. Additionally, consumers are not used to frequent updates on consumer devices that are not their phones or laptops. With the rise of modern applications, it is just no longer acceptable to build static software that is never changed or updated.

With billions of devices connected to an always-on network, large-scale security vulnerabilities have never been a bigger concern or consideration among companies that have millions of nodes on a network. Today, companies need to be continuously connected to all of their nodes and devices to properly maintain and secure them.

Enter DevOps and Embedded Innovation

DevOps has already transformed the way cloud-native applications are developed. Software gets built often, but it also gets deployed often. That’s why web apps are constantly being upgraded. Now, embedded is poised to take advantage of this software version control system that automates software builds and that is reproducible. Bringing DevOps to the embedded software ecosystem is about providing a set of tools or a platform that does the version control, CI and CD all in a single package—a platform to simplify the workflows for companies that typically have to build all of this themselves now can deliver opportunities for tremendous efficiencies and innovation.

Consider the airline industry as it was 60+ years ago. At that time, the airlines built their own planes from start to finish. Today, Boeing and Airbus build all airlines’ planes. Of course, each airline can customize their fleet from the overall flying experience to the interior branding, but everybody is leveraging the same core technology because airframes are really hard to build and the differentiation happens on top of that foundation.

Similarly, embedded software developers can focus on the customization of their product, not the time-consuming build that provides no differentiation. This is the concept behind the cultural shift we see in embedded software: Buy the airframe and customize it rather than build it yourself. Also inherent in the DevOps model is a distributed development practice, where different team members in different roles can be anywhere in the world and still work together seamlessly.

The pandemic certainly created more demand for this model, but distributed teams aren’t going away and are more prevalent than ever in technology development. Many embedded software developers are still working in the same way they were in the 1980s and 1990s. With DevOps practices and platforms in place today, embedded software developers can focus on the value-add of their application. By taking all the complexity out of building it yourself, companies today can simply leverage easily consumable and reusable tools to achieve what cloud developers have been doing for years.

March 24, 2022
How AI Can Improve Software Development

Change is constant and inevitable. While some changes are welcome, others are just a necessary evil.

Continuous calls for digitalization combined with a pandemic-prompted emphasis on transforming business operations, force organizations across industries to modernize their legacy systems. The emergence of cloud computing has shrunk organizations’ on-premise IT footprints. However, with 96 of the world’s largest 100 banks, 90% of the world’s largest insurance companies, 23 of the 25 largest retailers in the U.S. and 71% of the Fortune 500 still using mainframe systems, any modernization efforts—resulting in increased risk of downtime or defects—will frequently lead to disastrous consequences.

So, while making changes can be expensive, uncertain, and lead to customer loss and/or embarrassment in the marketplace, modern business demands that systems change to continue to deliver value. The problem with mainframe modernization, however, is that today’s code search tools, linters and program analysis tools are deficient when it comes to mitigating the risks associated with maintaining and improving systems.

Modernization: A Knowledge Problem

With 10,000 mainframes in active use globally, the average organization annually spends anywhere from 60% to 80% of its IT budget just on maintenance. But, as experienced and proficient programmers retire or move on, many companies are realizing that the specialized domain knowledge of the industry and institution that those developers used to create and update these complex critical systems is walking out the door with them.

As the developers who originally wrote the code have long since moved on, and documentation leaves a lot to be desired, developers new to a system are unable to acquire comprehensive knowledge of the system’s functionality, at a detailed or fast enough level, to become adequately productive and proficient. That lack of speed comes with significant risk and cost to any organization.

Recognizing the full and true intent of the functionality written in code is not simple. Developers today spend roughly 75% of their time searching through source code, to identify the code representing the functionality that needs to be changed. Complicating matters further, understanding the code is not enough. To effectively and efficiently maintain and support any system, software developers must know precisely what an application actually does—and how altering code in one part of the system impacts it as a whole. But since the code representing a behavior that needs to be changed can be littered throughout the system, developers might think they’re making a simple tweak, when really they may be breaking the entire system (and not even knowing it). The risk of change resulting in unintended consequences is real.

All of that being said, how can developers improve their efficiency and effectiveness?

The Shortcomings of Today’s Tools

With code repositories growing unchecked over time to extraordinary sizes, those responsible for maintaining and preserving a system’s functionality say it’s becoming increasingly challenging to find bugs in code without machine assistance. And with debugging being an arduous, time-intensive endeavor, developers are increasingly turning to code search tools, linters, and static and dynamic analysis tools to analyze millions of lines of code, flag errors and suggest fixes.

Whether the tool excels at bug localization, code visualization or “error” detection, entirely too many tools are insufficient when it comes to actually identify the specific lines of code that require attention. Program analysis tools only illustrate code in ways that developers still have to interpret (potentially incorrectly) and draw their own conclusions. While code search tools might speed the rate at which developers can build a mental model of the code, such tools are notorious for false positives, and developers still have to undergo the mentally challenging endeavor of piecing that mental model together to identify the code, and then safely make changes. Worse yet, code completion tools—tools that less-experienced developers disproportionately rely on—can actually propose imprecise changes (one prominent tool offers incorrect code 71% of the time), and that introduces risk.

The cherry on top of this problematic sundae is that today’s tools don’t have any way to effectively validate the scope and accuracy of a proposed change. The influence of a suggested change can be tough to identify, so testing for comprehensive validation is frequently next to impossible.

Ideally, there has to be a better way to mitigate risk while making changes to legacy systems. Enter artificial intelligence. By employing a unique approach to AI, the code repository truly becomes a knowledge repository, enabling many capabilities unrealized before now.

A Developer’s AI-Enabled Accessible Knowledge Repository

By leveraging AI to automate the identification of the specific lines of code that require attention, developers can simply ask this AI-driven knowledge repository where behaviors are coming from—and quickly identify the code associated with that behavior. This puts AI squarely in the position of intelligence augmentation, which is key to leveraging its capabilities.

This novel approach of AI reinterprets what the computation represents and converts it into concepts, therefore “thinking” about the code in the same way humans do. The result is that software developers no longer have to unearth the intent of previous developers encoded in the software to find potential bugs. Even better, developers are able to overcome the inadequacies of automated testing by using AI to validate that they haven’t broken the system before they compile or check in the code. The AI will forward simulate the change and determine whether it’s isolated to the behavior under change. The result is the bounds of the change are confined to the behavior under change so that no unintended consequences arise.

While AI is not yet advanced enough to safely make changes on its own, it can serve as a knowledge repository with the added capability of directing developers precisely where to make any needed changes. With AI, programmers—even on their very first day on the job — can make any necessary fixes with confidence that they won’t break the entire system.

Unlike its code search, code navigation, code completion and program analysis tool counterparts, AI fundamentally empowers developers to safely, efficiently and effectively make any necessary changes. With AI — that is fortified by a knowledge repository that understands source code the same way a human does — developers can truly do more with less.

January 13, 2022
DevOps Database Platforms Slash DX Bottlenecks
An enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. For example, test data provisioning is a challenge today, particularly in pre-release testing of database updates. Standard practices involve shared database environments that need to be maintained from one test to the next, slowing test cycles and generally not fitting into an automated CI pipeline. As I explained in A New DevOps Database Platform is Needed, at best, it makes things chaotic; at worst, problems with database integrations with DevOps is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

A recent whitepaper, A New Kind of DevOps Database Platform, indicates that there are multiple reasons why a new kind of DevOps database platform is needed, which are:
- Compatibility requirements
- Integration with current CI/CD pipelines
- Integration with portfolios of value stream management
- Infrastructure cost efficiency
- Productivity for developers, testers and operations
- Security and governance
- DevOps performance
The paper explained that a DevOps database platform requires the following capabilities to satisfy these requirements.
- Automated database provisioning: the ability to deploy a database using automated procedures.
- Database cloning: writable database environments are delivered on demand, based on a built versioned image.
- Distributed repository: database images can reside on Linux and Windows servers, as well as storage systems.
- Simple declarative builds: a simple, predefined method to make the creation of versioned data images simple, while enabling user/group access controls, data masking and other data engineering functions.
- Scalable data cloning: database clones are served on demand, with many simultaneous clones served from a single image.
- Comprehensive database support: the repo supports all common relational, NoSQL, machine learning and file-based data stores.
- Versioned databases: each image becomes a versioned, immutable DevOps artifact.
- Deploys equally to all cloud environments: DevOps databases can be deployed and operated on any on-premises infrastructure or cloud.
- Cross-platform: DevOps databases can be deployed and operated on common operating system environments such as Windows and Linux.
- Security features including access management, data masking and encryption: database images easily incorporate access control, data masking and synthetic test data via SQL scripts, batch files or command line tools.
- Compatible with CI/CD platforms: DevOps databases can be deployed and operated equally well in multiple CI/CD platform tools such as Azure DevOps, Jenkins and CloudBees.
- Compatible with containers and container orchestrators: DevOps databases can be deployed and operated equally well in multiple container environments such as Docker, Kubernetes and Red Hat OpenShift.
- Deliver writeable data at the speed of DevOps: Deliver writable databases in seconds.
- API for easy integration with DevOps tools: An API is available to enable integration of the DevOps database control plane with other tools.
- Audit logging: The DevOps database includes a complete audit log of operations.
Transforming to the New Database DevOps Platform

A DevOps database platform in an enterprise requires more than simply buying and installing software and telling everyone to use it. With multiple entrenched database platforms already in place, there are practical and political considerations that must be overcome. Introducing any change to a large group of applications in an enterprise can be chaotic, and will often be more costly and take too much time if the changes are not compelling.

To be sufficiently compelling, the database DevOps platform needs to be affordable, easy to understand, easy to set up and use with a wide variety of CI/CD pipelines. The following will be sufficiently compelling benefits to drive adoption:
- Versioned, secure database environments or images managed as a DevOps artifact.
- Images can be built and managed by the DevOps team, without requiring new infrastructure.
- Multi-DB and multi-TB data environments in seconds.
- Database environments can be served on demand for a CI/CD pipeline.
- Lower-level database environment infrastructure is streamlined, with a single VM capable of supporting many simultaneous database environments which drives an immediate cost savings and positive ROI.
- Data security and governance benefits with versioned database environments, with built-in and auditable security policies.
What This Means

Digital transformation is imperative for modern enterprises and organizations to remain competitive. However, the transformations of many enterprises and organizations are seriously impeded by the challenges of integrating databases into their DevOps application pipelines and value streams. A new DevOps database platform can allow the databases to be treated like any other DevOps artifacts that are orchestrated and automated within the DevOps toolchains.

Suggested resources for automating database environment delivery include Windocks, Cohesity and other storage appliances that provide virtualized (cloned) database environments that can be adapted for DevOps purposes. I want to particularly acknowledge input from Windocks, who has introduced a DevOps data repo that features database virtualization on standard Linux and Windows servers, for versioned data environments and CI pipelines and cloud-native support for data delivery for containers, Kubernetes and conventional instances.
July 27, 2021
A New DevOps Database Platform is Needed

The diverse number of database platforms, deployment environments and data variants needed to support the plethora of applications and services required for rapid DevOps continuous delivery lead times, coupled with the challenges of integrating database changes for application pipelines, is a growing challenge for many organizations as they strive to accelerate their DevOps transformations.

When you stand back and look at the bigger picture, an enterprise with many databases and various data requirements for DevOps continuous delivery presents a problem. At best, it makes things chaotic; at worst, it is a gross impediment to overall organization agility—which undermines the very purpose that DevOps is meant to serve.

In my book, Engineering DevOps, I explained that databases offer several challenges for CI/CD pipelines.

In the simplest case, when a database is used solely by one application or microservice, the database pipeline can be integrated with the application pipeline to ensure its changes use the same version management system and change controls to ensure the database changes are kept in sync with the application. However, many database platforms are not as responsive to changes as the applications themselves and become a bottleneck for the pipeline.

If a database is used by several applications, each with its own DevOps pipeline, then changes to the database must be coordinated with all applications that depend on it. Maintaining a separate database pipeline linked to the application pipelines through CI/CD toolchains, as a strategy, is overly complicated and can often introduce additional bottlenecks and synchronization issues.

Another challenge with database pipelines is the management of data in the database during the CI and CD stages. Data in production needs to be persistent and is ever-changing, even while the underlying database code is changing. During the CI process, sample data from production needs to be used to ensure testing is realistic, and new test data must be introduced to verify new database capabilities are working.

During the CD stages, rollbacks to prior versions of database code must avoid rolling back to versions that are no longer compatible with production data. This is a key reason why database changes must follow a disciplined approach that fully tests database changes with applications prior to full deployment. Rolling deployments, in which applications and database changes are deployed gradually in separate clusters, can minimize the blast radius in case of problems. The required coordination of separate systems adds complexity and bottlenecks.

So, why not simply integrate database changes as artifacts within the same DevOps pipeline as the application?

Gilad David Maayan’s September 2020 article DevOps Database Strategies and Challenges: Automation, Scalability, and More explained some of the specific challenges that arise with integration of traditional databases with DevOps.

Database tools were not designed for integration with DevOps tools. These tools tend to be database- and environment-specific, which can make it difficult to incorporate into pipelines designed for flexibility.

Database performance and ease of management often do not perform well in DevOps environments where multiple applications need to use the database in a short time, such as in the accelerated lead times for DevOps continuous delivery. This is especially true with larger-scale databases. If an application change must sync with a database schema, changing the pipeline can become complicated especially if there is a need to sync across multiple application pipelines to meet continuous delivery goals.

While DevOps processes are proven for continuous delivery of applications, incorporating database changes into the DevOps pipeline is hindered by large data sets, limited database automation and security concerns. As a result, database development and testing continue to rely on outdated processes, with fixed and often shared instances refreshed from backups. A new DevOps Database platform is needed to provide highly dynamic and efficient cloud-native database solutions for modern enterprise digital transformations, and that platform must include DataOps use cases for DevOps, DevSecOps, continuous testing and governance. This new kind of data repository must be designed to integrate seamlessly with DevOps CI/CD pipelines through flexible APIs and cloud-native architectures.

Given the points above, traditional database solutions and parallel data pipeline strategies are not sufficient to meet the challenges of enterprise DevOps transformations. What is needed is a new kind of DevOps database platform that allows the database, and database changes, to be as agile as any other artifact in the DevOps pipeline.

Paul Stanton, VP of product management at Windocks, in his June 2021 YouTube video Windock DevOps Data Repo, said, “Imagine terabyte class database environments managed as DevOps artifacts with entire versioned databases available in seconds.” That’s what we need.

The following are examples of the capabilities required for such a new DevOps database platform.

• Automated database provisioning
• Distributed repositories support
• Simple declarative builds
• Dynamically scalable
• Virtual hard drive images for a wide variety of DBs – Linux, SQL, Oracle and others
• Distributed versioned database environments working on standard file systems and storage systems delivery writeable data at the speed of DevOps – in other words, in seconds.
• Deploys equally to all operating systems and cloud environments, Docker containers and Kubernetes.
• Many clones per virtual machine (VM)
• Security features including access management, data masking and encryption
• Compatible with and easily integrates with CI/CD platforms
• Compatible with multi-cloud (AWS, Azure, GCS)
• Built-in data security and governance into DevOps data.

I will explain these more completely in a subsequent article.

What This Means

Digital transformations are imperative for modern enterprises and organizations if they are to remain competitive. However, the transformations of many enterprises and organizations are being seriously impeded by having too many database platforms and tools that do not integrate well and that bottleneck continuous delivery goals. A new kind of DevOps database platform, in which the database is treated as any other artifact, is needed to resolve technical problems with database integration to DevOps application pipelines. A new DevOps database platform solution is needed to accelerate time-to-value and realize efficient, safe operation for digitally transforming enterprises and organizations.

July 8, 2021
GitLab Updates Approach to Integrating DevOps Workflows

GitLab today took the wraps off a major update to its continuous integration/continuous delivery (CI/CD) platform that embeds analytics tools in workflows to identify the root cause of inefficiencies, among other additional capabilities.

David DeSanto, senior director for product management at GitLab, said that capability is part of an ongoing effort to include value stream analytics within the core GitLab 14 platform as an alternative to acquiring and deploying a separate platform. There are also additional charts and dashboards that enable DevOps teams to uncover bottlenecks over time.

In addition, GitLab announced it is extending the security capabilities of the platform by adding integrations with open source Semgrep and Trivy vulnerability scanning tools as part of an effort to enable organizations to embrace DevSecOps best practices within workflows. GitLab has also added support for the fuzz testing tool it gained via the acquisitions of Fuzzit and Peach Tech to enable DevOps teams to test application security more thoroughly along with making available additional dashboards to track security metrics.

As part of an effort to enable organizations to better secure their software supply chains, GitLab has also added support for immutable pipeline definitions that can’t be altered without specific levels of permissions being granted. That capability also enables DevOps teams to more easily achieve compliance with any number of regulatory frameworks, noted DeSanto.

The company is also adding a pipeline editor that provides access to a visual tool for creating, managing and visualizing pipelines. That tool is designed to both make DevOps more accessible to novices and provide experienced DevOps teams with an alternative tool for managing pipelines in a way that doesn’t require them to write additional code.

Finally, as part of its comprehensive approach to DevOps, GitLab is also making available agent software for Kubernetes clusters to eliminate the need for organizations to acquire a separate DevOps platform to build and deploy cloud-native applications based on microservices constructed using containers. The company now provides IT organizations with the option of crafting a GitOps workflow for Kubernetes using either an agentless or agent-based approach.

Gitlab is betting that, as DevOps continues to evolve and mature, more organizations will prefer to replace the various tools they need to integrate themselves with a platform that lowers the total cost of DevOps. Gartner is forecasting that, by 2023, 40% of organizations will have switched from multiple point solutions to a DevOps value stream delivery platform to streamline application delivery, versus less than 10% using such a platform in 2020. Today, only 10% of organizations are employing such a platform, according to Gartner.

There should also be many more organizations employing DevOps workflows to build applications. Organizations of all sizes now realize how dependent they are on software to drive a wide range of digital business transformation initiatives. The rate at which that software can now be securely developed and updated, in many cases, goes to the heart of their ability to compete. The issue is finding a way to achieve that goal without constructing a platform that ultimately takes time and resources away from building secure software.

June 22, 2021
How Containers Simplify DevOps Workflows and CI/CD Pipelines

DevOps enables organizations to capitalize on the flexibility of cloud resources in real-time, which is a huge game changer. Just as the adoption of Agile methodology can make a development team more responsive, DevOps revolutionizes implementation of IT objectives from end to end, from a waterfall-style process gating company productivity to a dynamic process that makes better-quality software fast and easy.

Organizations have been taking advantage of increasingly flexible resources and moving away from conveyor belt-style project delivery with hardware and software resource planning, ordering and integration. DevOps provides a way to address the question, “If we can provision resources quickly and easily, how can we complete entire projects with similar responsiveness?”

As DevOps lowers the wall between development and operations, staying sane and accountable doesn’t have to bog down processes. Structured communication still takes place, but in an iterative, incremental fashion, much like polishing a jewel.

Instead of lofty goals set in the somewhat-distant future, practical solutions can be created, deployed and adjusted. The process gets applications in the hands of end users far sooner, smooths any rough edges using actual user feedback, and helps organizations not only become more responsive to changing needs but make much more efficient use of valuable software development and operations resources.

Why Dev and Ops Teams Need to Collaborate

It can be hard for an organization’s culture to adapt to this level of change. But both development and operations teams have some pain points that help motivate them to work together more dynamically.

The development (Dev) side of the house wants to deliver more timely solutions, ensure greater acceptance of new applications and use its resources more efficiently. Operations (Ops) has challenges with hardware resource planning; they have maintenance and cost-reduction projects that are delayed. Ops also has insights into new features, efficiencies and integrations, but with no way to implement them.

These and other pain points arise when traditional, waterfall-style processes constrain communication and implementation. Creating a new flow between the two teams helps to reduce these pain points.

Rather than being tied to a specific technology, DevOps is a methodology aimed at making software development efficient. Containerization narrows the gap between development and IT Ops to a minimum. Containers are a great tool for enabling DevOps workflows and simplifying the development and delivery pipelines. Containers hold the code and dependencies required to run an application and reside in isolation. This enables teams to develop, test and deploy apps inside these closed environments. And this doesn’t affect different parts of the delivery, making the lives of testers and developers a lot easier.

Enabling CI/CD

DevOps has created a way to automate processes to build, test and code faster and more reliably. Continuous integration/continuous delivery (CI/CD) isn’t a novel concept, but tools like Jenkins have done much to define what a CI/CD pipeline should look like. While DevOps represents a cultural change in the organization, CI/CD is the core engine that drives the success of DevOps.

With CI, teams must implement smaller changes more often, but they check the code with the version control repositories. Therefore, there is a lot more consistency in the building, packing and testing of apps, leading to better collaboration and software quality. CD begins where CI ends. Since teams work on several environments (prod, dev, test, etc.), the role of CD is to automate code deployment to these environments and execute service calls to databases and servers.

The CI/CD concept isn’t entirely new, but it’s only now that we have the right tools to fully reap the benefits of CI/CD. Containers make it extremely easy to implement a CI/CD pipeline and enable a much more collaborative culture. Containers are very lightweight and can scale endlessly, run on any environment and are very flexible. It doesn’t get easier than this.

Now, you can move code across containers–or container clusters, as is the case with Kubernetes–instead of moving code among various virtual machines (VMs) in different environments. VMs are static and leverage a monolithic application architecture, whereas containers use a distributed microservices model. This opens doors to new benefits when it comes to elasticity, high availability and resource use.

As part of a DevOps team-oriented approach, team members can reach out to their peers and share innovation, feeding their knowledge and experiences into the evolving process. Development teams can put the new resources to use, taking advantages of technologies such as containers and Kubernetes to architect solutions. Working together, this cooperation lets Ops run with the solution, scale it, distribute and upgrade efficiently without delay.

Putting it All Together

DevOps has helped organizations reimagine their entire development lifecycle. And while DevOps brings cultural change, CI/CD is the core engine that makes DevOps successful. New tools like containers now make it easier for organizations to make the most of CI/CD. However, data storage must be taken into consideration, because it undergirds a dynamic DevOps environment. It also provides failover, disaster recovery and other essential availability and reliability features. Software-defined storage offers the flexibility that a dynamic DevOps environment needs. It’s a crucial element in the success of your DevOps program.

June 22, 2021