devopsy.kinsta.cloud

Category: Enterprise DevOps

Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools

In my book “Engineering DevOps,” published a few years ago, I classified DevOps into “nine pillars of DevOps best practices” – a comprehensive framework that covered the full scope of DevOps practices. Each of these pillars represents a vital area of DevOps implementation and, together, they ensure a holistic approach to integrating development and operations for effective collaboration and continuous delivery.

The current generation of AI-engineered tools is now being applied to each of the DevOps pillars and has the potential to significantly increase the efficiency, reliability and speed of the software delivery process. By automating repetitive tasks, improving decision-making with predictive insights and facilitating proactive issue resolution, AI can drive a higher level of quality, security and performance in the software development life cycle. Furthermore, AI’s capacity to learn and adapt over time means continuous improvement, fostering a truly data-driven, learning-oriented DevOps culture.

AI Applications for the Nine Pillars of DevOps

The following paragraphs briefly explain how the current generation of AI-engineered large language model (LLM) tools are being applied or could be practically applied to improve each of the nine pillars of DevOps.

Leadership Practices: Leadership is vital to drive cultural changes, set vision and goals, encourage collaboration and ensure resources are allocated properly. Strong leadership fosters a successful DevOps environment by empowering teams and supporting innovation. AI can assist leaders in decision-making by analyzing large datasets to identify trends and predict outcomes, providing valuable insights to guide strategic planning.

Collaborative Culture Practices: DevOps thrives in a culture of openness, transparency and shared responsibility. It’s about breaking down the silos that can exist between different teams (like development, operations and QA) and promoting effective communication and collaboration. AI-powered tools can improve collaboration through smart recommendations, fostering more effective communication and knowledge sharing.

Design-for-DevOps Practices: This involves designing software in a way that supports the DevOps model. This can include aspects like microservices architecture, modular design and considering operability and deployability from the earliest stages of design. AI can be used in the software design phase to automatically suggest improvements in code quality or to identify potential performance issues early.

Continuous Integration Practices: This is about developers frequently merging code changes into a central repository. After that, automated builds and tests are run. The main objectives are to detect problems early and reduce integration issues. AI can analyze patterns in past build failures to predict potential issues with new builds, leading to proactive error mitigation.

Continuous Testing Practices: Continuous testing involves automating tests to provide instant feedback on business risks and allowing for faster, more reliable releases. This approach enhances the quality of the software and reduces the time it takes to resolve defects. AI can be used to generate and optimize test cases automatically, reducing manual effort and speeding up the testing process.

Elastic Infrastructure Practices: Elastic infrastructure involves automated provisioning and management of computing resources, often leveraging cloud-based solutions for scalability and resilience. This includes practices like infrastructure-as-code (IaC). AI can help manage cloud resources by predicting future needs based on usage trends and automatically adjusting resources to optimize cost and performance.

Continuous Delivery Practices: This is an approach where code changes are automatically built, tested and prepared for release to production. It aims to make releases less disruptive and more frequent, increasing speed and efficiency. AI can help predict the potential impact of new releases on the system, aiding in decision-making related to deployment.

Continuous Security Practices: Also known as DevSecOps, this involves integrating security practices into the DevOps workflow. The goal is to introduce security considerations early and often in the life cycle rather than leaving them as an afterthought. AI can be used in anomaly detection to identify potential security threats and automate security policy enforcement.

Continuous Monitoring Practices: Monitoring systems and application performance in real-time allows teams to detect and react to problems quickly, often before they impact users. It’s crucial for maintaining application health and informing future improvement efforts. AI can analyze logs and metrics to predict potential system failures or performance degradation, allowing for proactive maintenance and issue resolution.

Pitfalls and Challenges

Applying AI to DevOps is a complex process with certain challenges. Here are some potential pitfalls, along with ways to address them:

Data Quality and Availability: AI relies heavily on data for training and making accurate predictions. If the data is inaccurate, incomplete or biased, it can lead to misleading outcomes. Ensure you’re collecting high-quality, relevant and unbiased data for training AI models. Regular data audits can help maintain data integrity.

Over-Reliance on Automation: While automation is beneficial, over-reliance on it without human oversight can result in overlooked errors or system failures. Maintain a balance between automation and human intervention. Implement periodic manual checks and reviews to validate AI-generated results and actions.

Security and Privacy Concerns: As AI tools are more integrated into the DevOps pipeline, they may become targets for security threats. Also, the handling of sensitive data used for AI training can raise privacy issues. Implement strong security practices, conduct regular security audits and consider privacy requirements during the data collection process.

Black Box Problem: AI, especially complex models like neural networks, often operate as a “black box,” meaning it’s difficult to understand why they made a certain decision. This lack of transparency can make troubleshooting difficult. Consider using explainable AI (XAI) techniques, which aim to create more understandable AI systems, providing transparency into how decisions are made.

Skills Gap: The application of AI in DevOps requires a certain level of expertise and understanding of both domains, which can be hard to find. Invest in training and upskilling existing staff, or consider bringing in external expertise through hiring or consultancy. Sharing knowledge and encouraging continuous learning can also help bridge this gap.

Summary

AI-engineered tools can enhance all of the nine pillars of DevOps by increasing efficiency, speeding up processes and making predictions to aid decision-making. However, implementing AI in DevOps comes with challenges such as ensuring data quality, maintaining a balance between automation and human intervention, addressing security and privacy concerns, managing the “black box” problem of AI and overcoming skills gaps. Each of these pitfalls can be addressed through proactive strategies like maintaining data integrity, implementing manual checks, bolstering security practices, using explainable AI techniques and investing in training and skills development.

June 2, 2023
Why You Need a Multi-Cloud and Multi-Region Deployment Strategy

As I began writing this article, I encountered some technical challenges in my attempt to provision three cloud providers in three separate regions to simulate multi-cloud and multi-region deployments.

I couldn’t get one of the regions to behave as I expected when provisioning the environment. As it turned out, the region (and by association, cloud) was down due to a cut undersea cable, and it would take several weeks until the outage was mitigated.

Fortunately, there are plenty of clouds in the sky to choose from, but this experience emphasized the importance of multi-cloud redundancy, especially in a production environment.

This article examines the benefits of adopting a multi-region and multi-cloud deployment strategy for better resiliency, regulatory compliance and optimal user experience.

Resilient Reasons to Choose Multi-Cloud and Multi-Region Architectures

The first reason to choose a multi-cloud or multi-region environment is for data resilience. When our data is in a single region or on a single cluster, there is a good chance that you will experience data failure. You need a mitigation strategy if you want to keep the data lights on 24/7. These strategies fall into two buckets: Multi-region and multi-cloud.

We first need to define the differences between multi-cloud and multi-region. The architectures are simply the implementation details of how we plan for one versus the other. However, depending on your database provider, those details can represent a significant time investment, which we’ll look at later.

Multi-Region

With multi-region, we typically deploy our applications to globally distributed regions. However, regional deploys are subjected to physical faults. Inclement weather, political unrest, acts of foreign or domestic terrorism and other physical variables have all led to regional outages in the past.

Regardless of the cloud, the regions are exposed to similar physical faults. Instead of losing your data, we can switch to another region to handle the workload. Even though much as our industry hates latency, it is always preferred over loss.

Deploying to multiple regions allows for failover handling. If the Sydney region goes down, Singapore may still be online. It would be a globally bad day if the same physical faults simultaneously impacted Frankfurt, Sydney and Seattle, for example.

Multi-Cloud

Physical faults are only one of the problems that public and private clouds encounter. User error, poorly configured networks and even banal errors can have substantial consequences that impact entire clouds, either from the provider or the user.

In these circumstances, regional failover won’t help you. An expired certificate, accidental deletion or disabling the wrong port will take you offline, regardless of whether you’re in Seattle or Sydney. If that happens, someone somewhere is most definitely awake and is likely very unhappy.

Choosing a multi-cloud configuration protects you from these types of faults so that if Google, for example, has interruptions, AWS is still online—or IBM, Deutsche Telekom, OVH or Hetzner.

The Correct Answer: Deploy Both!

Multi-region is a go-to method to mitigate against mostly physical faults. Multi-node clusters in each of those regions handle general data corruption or failures on a more granular level.

Multi-cloud is a go-to method to mitigate configuration or software-level faults, which are not that uncommon.

The correct answer is to blend the two, working within operational cost constraints and general survivable requirements from regulatory and business objective use cases. For example, I do want access to my heart-rate data from two years ago, but the order of browsing across Amazon shopping is less important.

Regulatory Reasons for Multi-Region Architecture

Apart from data resiliency, most companies serving a global market are subject to a number of geopolitical influences, as well. Most users will be familiar with the GDPR regulations ensuring European data stays on European soil. Further regulations require the ability for government oversight to request data at will from service providers, which requires its own set of reasonable isolation from those not in those jurisdictions.

Using a multi-region database provider allows companies to ship single binaries of software without creating isolated applications for different countries—a practice that was the norm not too many years ago.

Performant Reasons for Multi-Region Architecture

Co-locating data and access provides the optimal pathway for getting a user’s data to the end destination. It goes without saying, but the less copper your data has to traverse, the faster the experience will be for your users. Retention and regulation aside, in many cases, you need to mitigate latency caused by region separation.

It’s not just the individual user’s data. Imagine a sales enablement platform where an account executive needs to manage all of APAC or EMEA. If these accounts are optimally stored for reads on geo boundaries, you can remove seconds of delay between page navigations for your account manager. We all know how much account executives like to click!

Choosing a DB Provider With Multi-Region and Multi-Cloud

At this point, you should be convinced of the benefits of multi-region and multi-cloud deployments. But at what cost? Creating replication, failover, logging and synchronization mechanisms is a big, expensive ask. Even with deployment tooling, coordinating all the pieces for multi-region deploys is still a significant operational lift, let alone multi-cloud, where variables in APIs come into play.

Choosing a single provider that deploys across regions and clouds means you get to benefit from their collective experience while reducing a significant burden.

Having a database provider offer one-click multi-region deploys is the way to go, but you still need to handle the data access layer. The role-based access, federation and other settings that enable your developers still need to be distributed along with the underlying data sources.

You need a provider that makes it simple to deploy the same configuration for horizontally scaled instances, leaving you to pick your geo router of choice to distribute the request to the nearest point of presence.

Closing

Multi-cloud and multi-region deployments provide significant redundancy, scalability and flexibility. Not only does this enable faster access times and improved reliability, but also provides the flexibility required to adapt to disruptions or unpredictable increases in traffic.

Deploying multi-cloud and multi-region is no small feat–but the peace of mind and security it provides make it worth the effort.

June 1, 2023
Linux Foundation Europe to Host RISE Open Source Project

The Linux Foundation Europe, in collaboration with RISC-V International, announced today it will host a RISC-V Software Ecosystem (RISE) Project. Through the project, multiple organizations will contribute engineering and financial resources to harden critical open source software for use in commercial applications and enterprise IT environments.

Founding members of the project include Andes, Google, Intel, Imagination Technologies, MediaTek, Nvidia, Qualcomm Technologies, Red Hat, Rivos, Samsung, SiFive, T-Head and Ventana.

RISC-V International is defining a set of application processors that includes software development tools, virtualization support, language runtimes, Linux distribution integration and system firmware that is intended to be used primarily in edge computing environments.

Amber Huffman, chair of the RISE Project, said at a minimum, each of the founding members contributing €80,000 and additional engineering resources. The RISE Project deliverables will be prioritized by the RISE Technical Steering Committee (TSC) as the project looks to address performance, security and reliability issues involving open source software, she added.

The TSC will also be able to shift skills from one initiative to another as priorities continue to evolve, Huffman added.

The overall goal is to use a more holistic approach to eliminate duplicate work that each member of the RISC-V community would otherwise do separately to create an application processor, noted Huffman.

The Linux Foundation Europe is an arm of the Linux Foundation that has been created to host projects that are of particular interest to organizations investing in open source software based in Europe. The RISE project is being launched at a time when the European Union debates a proposed Cyber Resilience Act that requires organizations that sell hardware platforms that connect to the internet to ensure that the devices their software runs on comply with cybersecurity best practices.

The Linux Foundation also provides the funding for the Open Source Security Foundation (OpenSSF) that is marshaling resources to enable maintainers of open source software to address vulnerabilities and adopt DevSecOps best practices to better ensure the security of downstream applications that incorporate that code. In addition, the OpenSSF is building out an incident management capability that will assist maintainers in the event a zero-day vulnerability is discovered.

It’s not clear when all these investments might eventually lead to higher-quality open source software. However, it’s clear that vendors that benefit from open source software are rallying to address a range of quality assurance issues. Less clear is the degree to which enterprise IT organizations that benefit from open source software are making similar commitments.

In the meantime, it’s not likely any of these issues will have a material impact on organizations’ reliance on open source software. The benefits of reduced costs and increasingly faster pace of innovation enabled by open source software development far outweigh the risks. However, as cybercriminals become more adept at embedding malware within software supply chains, there’s a clear need for increased vigilance.

May 31, 2023
Forget Change, Embrace Stability

DevOps is about change. It was introduced to accelerate change, and it has opened up a whole new world of tools and possibilities that are all based on the premise of changing IT. Starting with Agile, we now have DevOps variants and subprocesses that run the entire software development life cycle (SDLC) process.

Want to make more frequent updates that can regularly show demonstrable progress to business owners? Agile and CI! Need to get the app settled in its inevitable home with automation? You need CD! Want testing and zero-day protections? Start with DevSecOps built into the CI/CD process and add in application and API security! Want to fully automate deployment in the whay builds are automated? Grab a GitOps tool and let’s goooo! The list goes on and on. Literally. We’ve almost got more tools than we have names for spaces at this point.

And all of those tools introduce change. Most of them introduce a massive amount of change. You don’t get to GitOps without breaking some of your existing processes and reworking them. We have fallen in love with change in IT.

But there is a funny thing that’s happened along the way; I’ve mentioned it before, but never from this perspective. All great changes require time for consolidation. Be it stock market swings or revolutions, afterward, there is a period of consolidation. In IT, containers are a good example. When they became capable of actually running workloads, IT experienced a period of time when making all the various machines run in VMs was the thing. People weren’t (generally) looking for the new big change in the process, they were just glad that those one-off machines that had custom hardware could now be moved to something more manageable and that spinning up servers could be done in days or weeks instead of the months it used to take when you had to order hardware. But that consolidation created organizational standards and stability that the organization enjoyed for years – even after containers rose to challenge VMs and some organizations even today.

So every once in a while, take a breather, consolidate gains, create standards that have meaning for your organization, then continue on with whatever the shiny new change is. Make sure that once you have identified, for example, the need for code scanners that can work within or that are complementary to the integrated development environment (IDE), that you have one scanner—or one scanner per programming language—to maintain and monitor instead of one tool per project. That saves time and creates stability, both of which will serve the organization moving forward.

Change is here to stay; it’s the only constant. IT is caught in a cycle of change followed by change, with the next wave driven by generative AI. But in times of change, seek stability and, in times of stability, seek changes that can have a positive impact. So go standardize and organize before the next wave hits.

Mostly, the changes will serve the organization by freeing up your time and letting you spend more of your time rocking the IT world. Those systems stay up because of you. More time is more innovation, and more stability means less burnout. All of which is good for the organization.

May 31, 2023
US DoJ Makes PyPI Give Up User Data ¦ Tape Storage: Not Dead

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: PyPI complies with a “string of subpoenas,” and LTO continues to grow, despite predictions of its demise. (more…)

May 25, 2023
Dell Looks to Expand Pool of Available DevOps Expertise

Dell Technologies has quietly launched a training and education initiative to significantly expand the number of IT professionals that have DevOps skills.

Brad Maltz, senior director for DevOps and developer relations ecosystems for Dell, said the next era of IT will require IT professionals that understand how to automate the management of application development and deployment. Given its long infrastructure history, Dell is especially interested in helping IT teams to use a mix of graphical tools and command line interfaces (CLIs) to programmatically deploy software at much faster rates and at scale, he added.

Dell has already hired more than 200 specialists to provide that training as part of a DevOps Transformation Services consulting services engagement, said Maltz. The primary area of focus will be working with IT organizations that are embracing platform engineering to centralize the management of DevOps and GitOps workflows, he noted. Many of those teams are now relying on declarative automation tools that ultimately make DevOps more accessible to a wider range of IT professionals, added Maltz.

The number of IT professionals that employ graphical tools based on IT management frameworks such as ITIL still far outnumber the number of IT professionals that have DevOps skills. The ongoing debate between these two camps essentially comes down to the need to be able to consistently manage IT at scale versus being able to adroitly keep pace with the needs of organizations that are increasingly dependent on software. In the digital transformation era, it’s become clear that more of the organizations that hire IT professionals are placing a higher value on agility, said Maltz.

Bridging the divide between ITIL and DevOps to address that requirement is largely a psychological challenge that requires IT teams to address a range of cultural issues in addition to mastering different types of tooling, he added.

Regardless of whether organizations continue to manage IT infrastructure internally or rely more heavily on managed services provided by Dell, the need to continuously deliver secure software faster has become a business issue. The trouble is the level of DevOps expertise within organizations tends to vary widely. A recent survey published by the Continuous Delivery (CD) Foundation found only 22% of respondents are using both continuous integration (CI) and continuous delivery (CD) to automate all building, testing and deployment of code in a production environment. Most organizations that have adopted DevOps make extensive use of CI to build applications, but CD has proven to be more challenging to master. In fact, the CD Foundation survey found only 7% of respondents described themselves as DevOps engineers or specialists.

The bulk of the managed services that Dell provides are focused on managing infrastructure rather than deploying software, but the company does have a managed DevOps offering based on the application development and deployment platform that an internal Dell DevOps team built to construct and deploy applications. Naturally, Dell has a vested interest in accelerating the rate at which software can be deployed on IT infrastructure it provides and manages.

It may be a while before the overall size of the DevOps community expands, but the training initiatives such as the one launched by Dell will help. However, it may become a lot easier to find and retain that talent in the months and years ahead. In the meantime, organizations would be well-advised to invest in upskilling their own IT teams. After all, it’s less expensive to train someone that already works within an IT organization than it is to compete against thousands of other organizations willing to pay top dollar for the best DevOps expertise money can buy.

May 25, 2023
Is AI the Answer to Devs’ Documentation Problem?

I have written this blog half a dozen times over the years, but every once in a while something comes up that makes me realize I should revisit it once more for those in the back. This time, it was a social call with a group of friends, some of whom are in high-tech, all of whom are smart and often insightful. One member of the group, a tech lead for a company that you do business with, was talking (in vague terms, of course; this person is a professional) about work he is doing, and the group started to make fun of documentation.

Let’s face it—since the very first line of code was written, we have sucked at documentation. It is a rare project that (a) gets just enough documentation and (b) keeps that documentation up-to-date. What we most often get is no documentation or documentation that records the initial design but, by the end, bears little resemblance to the code’s actual workings. The worst of all types of documentation is the “It’s self-documenting!” kind, which has gotten another boost based on the idea that OpenAPI/Swagger will be a tell-all about what is actually happening with the API. Like far too much API testing, it does a good job of telling you about inputs and outputs and, in most cases, absolutely nothing about what is done with the inputs or what magic happens to get you the outputs.

After our call, I got to thinking. What can we do to adequately document the code? I mean, with AI becoming all the rage and the set of lines of source code available for training being so massive, you would think we could do better. Maybe we can—but right now, the language processors that we currently pass off as AI are too busy being conversational or generating the code itself. Both of those spaces are going to take a while to get “good enough” to rely upon without review (not as long as many think, but certainly not tomorrow). Maybe that point will be when some enterprising engineer will turn it on source. Though I think that it will take a documentation specialist and not a coder—after all, we coders created this problem and haven’t fixed it in decades.

Many of us truly believe the “self-documenting” part when that is and absolute lie. It may be self-documenting to someone who wrote it or, in the best cases, someone with deep domain knowledge, but certainly not to the person who has to pick it up. We all have horror stories of working on that application—or porting/rewriting an application—that claimed to be documented or self-documented and was not even close. The saving grace in all of the cases that I worked on was that the code actually worked and performed the job required. Given some extra time to step through it, it could be rewritten even without good comments and docs. (Hat tip to the dev that had to pick up one massive porting project from me because I left to go write about storage—the project was huge and was a blast, but having to hop in halfway through probably made him crazy.)

If you are planning for the future, I recommend a ton of things, but for documentation, only two. First, write design docs, discuss decisions and put them in Git with the source code. Require that they be reviewed and updated after each sprint. Then, make sure that there are comments in the code. Using naming conventions to document code is helpful, to be sure. But more is needed—particularly as code morphs into whatever it is going to be in long-term final form. We have all seen that method or function that has a name … and then doesn’t do what the name implies. Or worse, that does what the name implies but also applies half a dozen side effects because devs needed somewhere to put new functionality. And we haven’t even talked about complexity like feature flags.

So do both, and require that devs spend a bit of time on each during each sprint—not to be a jerk, but to pave the way for dev #327 that inherits this code and has to make sense of it. Help them get productive quickly.

And keep rocking it. We have documentation problems because we solved business problems rapidly. That’s a good thing, we just have to think about the next gen of devs that will be on this project, and give them that little boost into productivity that documentation offers. And maybe you’ll help the compliance team while you’re at it, but that’s a different blog.

May 24, 2023
8 Hot Takes: Will We See a Monolithic Renaissance?

Is the DevOps world slowly backing away from microservices and to a monolithic renaissance? After years of proselytization about the benefits (Flexibility! Resilience! Support faster release speeds!) of microservices and serverless architecture, there are signs of a potential backlash against the singularly fragmented microservices architectural approach.

The argument in favor of rolling back to monolithic architectures feels similar to the cloud repatriation debate. Just as with a lot of cloud-first transformations—often in lockstep with them, actually—many organizations moved headlong toward microservices without a case-by-case business or technical rationalization for the change. As the shiny newness of microservices has dulled under that approach’s daily exposure to real-world deployment and business challenges, the monolith isn’t looking so bad in a lot of situations.

The monolithic versus microservices debate boiled over the last few weeks in the wake of the publication of a case study earlier this month from the Amazon Prime Video team. The analysis from Marcin Kolny, senior software development engineer for Prime Video, reported that his team was experiencing scaling bottlenecks in its audio/video quality inspection tooling, which was architected as a distributed system using serverless components. By rearchitecting the tool as a monolith, they were not only able to increase scaling capabilities but drastically reduced the infrastructure costs around that tool.

“Moving our service to a monolith reduced our infrastructure cost by over 90%,” Kolny wrote.

The announcement elicited a fair degree of hyperbole on the topic, but it also spurred a very organic airing of the pros and cons of monolithic, microservices and serverless architectures.

The following are 10 different perspectives spurred on by this online debate from a range of CTO and engineering experts. Some of them are diametrically opposed to one another, many are overlapping in their opinions. But as a body, they point to the general (and very pragmatic) consensus that there simply will never be a one-size-fits-all approach to software architectures.

Microservices a Siren Song for Complication

“It’s clear that in practice, microservices pose perhaps the biggest siren song for needlessly complicating your system. And serverless only makes it worse. Replacing method calls and module separations with network invocations and service partitioning within a single, coherent team and application is madness in almost all cases.”

—David Heinemeier Hansson, creator of Ruby on Rails and CTO at 37signals (maker of Basecamp and HEY)

Architectures Aren’t Good or Bad

“Application architectures are neither good nor bad, nor are they appropriate for every use case. Organizations need to step back and consider carefully what it is they’re trying to achieve and which architecture might best serve that need instead of choosing the most “modern” architecture because, well, it’s in vogue.”

—Lori MacVittie, distinguished engineer in office of CTO at F5Networks

Distributed Architectures Will Suit AI-Driven Future

“In the ongoing debate between monolithic and serverless architectures, it is important to acknowledge that both approaches have their merits and considerations. Monolithic architectures have been prevalent in large human-centric organizations due to their management advantages and historical use cases. However, as the world continues its rapid progression towards a more AI-driven future, the landscape is shifting towards architectures that embrace fluidity, autonomy, and distribution.”

— Reuven Cohen, veteran CTO and entrepreneur, mentor for TechStars

Monolithic and Serverless Each Have Pros And Cons

“Over the last 15 years, everyone has wanted to move to serverless microservices-based horizontally-scaled architecture – but it is not ideal for every application’s specific needs. As with any choice in life and in business, there are always tradeoffs; you must evaluate and choose the appropriate architecture based on your business and technology needs. Another important dimension to this is workforce skills for application architecture.

—Hemang Davé, client technical leader and CTO at Kyndryl

The Amazon Rearchitect Wasn’t Even Truly a Monolith

“The problem is that they called this refactoring a microservice to monolith transition when it’s clearly a microservice refactoring step, and it is exactly what I recommend people do in my talks about serverless first. I don’t advocate “serverless only”, and I recommended that if you need sustained high traffic, low latency and higher efficiency, then you should re-implement your rapid prototype as a continuously running autoscaled container as part of a larger serverless event-driven architecture, which is what they did. If you built it as a microservice to start with, it would probably take longer (especially as you have to make lots of decisions about how to build and run it) and be less able to iterate as you figure out exactly what you are trying to build.”

—Adrian Cockcroft, longtime DevOps advocate and veteran architect with stints at eBay, Netflix, Amazon, and Battery Ventures, among others

One Tool as a Monolith, Maybe, But Not All of Amazon

“Microservices, while duplicative and inefficient from a compute standpoint, allow you to release stuff really quickly, which is a tradeoff that most organizations at scale are willing to make.

Microservices exist to reduce communication pathways—not to solve technical scaling challenges. At a certain point, teams stop functioning due to the communication overhead. A seven-person team has 21 communication pathways, whereas a 100-person team has 4,950. I work with large retailers selling billions of dollars a year online, and putting everything into a single monolith just doesn’t work. Could the entirety of Amazon.com be deployed as a single monolith? No way.”

—Kelly Goetsch, chief strategy officer at commercetools

Architectural Tension Always Exists Between Speed and Maintainability

“Speed to market and speed to develop does not always equate to long-term scalability and maintainability. You must actively balance your investments across these two critical pillars to build a viable product. Active rebalancing and reconsideration of the stack and technology mix for your FinOps and BizOps context is both art and science.

Cloud-native distributed systems paradigms are here to stay. The Prime Video folks have mistakenly labeled a properly factored, highly coupled data-intensive processing step a monolith—it’s at best a rocky outcropping in their distributed microservices forest. “

—Ajish George, managing director of cyber data & analytics for State Street

The Shiny Thing is Not Always the Best Thing

“The news that AWS discovered that serverless and microservices aren’t for everything is surprising. Not because it didn’t work out for that use case; but because the wrong use cases have been so well-known for the serverless/microservices pattern and should have been obvious to Prime Video. It’s worth remembering: That shiny new thing isn’t necessarily a better way of trapping mice. Right tool for the right job.”

—Doug Vanderweide, director of cloud architecture for Avanade

May 23, 2023
Five Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

LinkedIn

Samsung Electronics America
Mountain View, California
Lead Site Reliability Engineer (SRE) – DevOps Specialist
$180,000 to $210,000

Indeed.com

Federal Reserve Bank of New York
New York, New York
DevOps Engineer
$178,100 to $207,900

CareerBuilder.com

SAIC
Chantilly, Virginia
DevOps Advisor
$130,000 to $200,000

Dice.com

Saicon Consultants
Alpharetta, Georgia
DevOps Engineer
$110,000 to $140,000

Simply Hired.com

Hewlett-Packard Enterprise
Fort Collins, Colorado
DevOps Automation Engineer
$109,400 – $251,500

May 22, 2023
How Test-Driven Methodologies Reduce CI/CD Lead Time
Modern development environments, in which rapid continuous delivery is facilitated by automated continuous integration/continuous development (CI/CD) pipelines, require thorough and automated testing in development prior to integration. This article explains how test-driven development (TDD) and acceptance test-driven development (ATDD) methodologies reduce CI/CD lead times.

Lack of test automation in development impacts CI/CD lead time for subsequent integration, delivery and deployment stages of a CI/CD pipeline in the following ways:
- Increased defect rate: Without automated testing during the development stage, more defects and issues are likely to be introduced into the codebase, increasing the time spent on debugging and fixing issues in later stages.
- Slower feedback loop: Without test automation, the feedback loop between code changes and test results is slower, making it more challenging to identify and resolve issues quickly. This delay can impact the overall lead time in the CI/CD pipeline.
- Reduced confidence in releases: When testing is not performed adequately or isn’t automated, there is less confidence in the quality of the software being released. This may lead to more manual testing and validation, further increasing lead times.
- Inefficient use of resources: Without test automation, developers and testers need to spend more time on manual testing, reducing the time available for developing new features or fixing existing issues.
- Increased merge conflicts: When test automation is not performed early in the development stage, the risk of merge conflicts increases as different developers work on the same codebase. Resolving merge conflicts can be time-consuming and may cause delays in the integration, delivery, and deployment stages.
- Decreased stability and reliability: Inadequate test automation during the development stage can lead to less stable and reliable software. This may require additional time and effort to address issues and ensure that the software is production-ready.
- Inconsistent test environments: Manual testing may not always be conducted in a consistent environment, leading to potential discrepancies between development, staging, and production environments. This can cause unexpected issues and delays in the pipeline.
- Hindered collaboration: Lack of test automation may reduce visibility into the quality of the codebase, making it more difficult for team members from development, QA, security and operations to collaborate effectively and identify issues early.
Although CI/CD pipelines help catch issues and ensure code quality through automated builds and tests, having a strong foundation of test automation during the development stage helps minimize the potential for issues in the pipeline and ensures a smoother CI/CD process.

There are several reasons why many development teams do not automate tests prior to integration, despite the benefits of doing so.
- Lack of awareness or understanding: Some development teams may not be aware of the benefits of test automation or may not fully understand how to implement it effectively in their development process.
- Insufficient skills or experience: Test automation often requires specialized skills and knowledge, and some development teams may lack the necessary expertise to implement it effectively.
- Time constraints: Developing and maintaining automated tests can be time-consuming. Teams working under tight deadlines might prioritize feature development over creating automated tests, believing that it will save time in the short term.
- Limited resources: Test automation tools and infrastructure can require significant investments in terms of cost and resources. Some organizations may not have the necessary budget or resources to invest in test automation, especially for smaller projects or teams.
- Resistance to change: Some teams may be resistant to adopting new practices, particularly if they have been successful with their existing processes. Changing established workflows and learning new tools can be challenging and may encounter resistance from team members.
- Incomplete or rapidly changing requirements: In situations where requirements are unclear or frequently changing, development teams may struggle to create meaningful automated tests, as they may require continuous updates to stay relevant.
- Overemphasis on manual testing: Some teams may place a greater emphasis on manual testing due to a belief that it provides more accurate or thorough results. This perception can lead to a reluctance to invest time and effort into developing automated tests.
- Fear of increased maintenance: Automated tests, like any other code, need maintenance and updates as the application evolves. Some teams may be concerned about the additional effort required to maintain a test suite, particularly if they have limited resources.
The amount of test automation that should be done in development prior to the integration stage of a CI/CD pipeline depends on several factors, including the complexity of the application, team resources and the specific project requirements. However, here are some general guidelines to help you determine the right balance:
- Unit tests: Developers should aim to automate most, if not all, unit tests during the development stage. They are typically fast to execute and provide quick feedback, allowing developers to catch issues early and minimize the number of defects that make it to the integration stage.
- Component and integration tests: While unit tests focus on individual components, component and integration tests verify the interaction between different components or services. These tests should be automated as much as possible to ensure that any issues arising from component interactions are identified and resolved quickly.
- Static code analysis: Automating static code analysis checks during development helps identify code quality issues, security vulnerabilities, and potential performance bottlenecks early in the process.
- Code coverage: Strive for a high level of code coverage with your automated tests. However, it’s important to remember that code coverage is just one indicator of test quality and should be complemented by other testing strategies.
Striking the right balance between automated and manual testing is important. Automate tests that are repetitive, time-consuming or prone to human error. Manual testing should be reserved for exploratory testing, usability testing and other areas where human intuition and judgment are necessary.

Test-driven development (TDD) and acceptance test-driven development (ATDD) are development methodologies that emphasize creating automated tests before writing the actual code, helping teams ensure code quality and catch issues early prior to integration.

Test-driven development (TDD) is a software development methodology that emphasizes writing tests before writing the actual code. The primary goal of TDD is to create clean, maintainable and bug-free code by catching issues early in the development process. TDD follows a short iterative cycle consisting of three main steps: Write a test, make the test pass and refactor the code. The developer starts by writing a test for a specific feature or function, which initially fails. Next, the developer writes the minimal amount of code necessary to make the test pass. Finally, the developer refactors the code to improve its structure and readability while ensuring that all tests still pass.

Acceptance test-driven development (ATDD) is an extension of TDD that focuses on defining and validating the high-level requirements and acceptance criteria for a feature or functionality before development begins. ATDD involves collaboration among developers, testers, and business stakeholders to ensure that the application meets the needs of its users. The process begins with writing acceptance tests, which are derived from the user’s perspective and based on the acceptance criteria. These tests are written in a human-readable language to facilitate clear communication among all team members.

Once the acceptance tests are defined, the development team follows the TDD process to implement the feature, using the acceptance tests as a guide. As the team develops the code, they also create unit tests and other lower-level tests to validate the implementation. The acceptance tests are executed as part of the development process and serve as a measure of progress. When all acceptance tests pass, it indicates that the feature or functionality meets the defined acceptance criteria, and the development task is considered complete. By using ATDD, teams can improve collaboration, ensure that the application meets user expectations and reduce the risk of miscommunication or misunderstanding requirements.

Hurdles to overcome when transitioning to TDD and ATDD include:
- Mindset shift: Adopting TDD and ATDD requires a change in mindset, where testing is considered an integral part of the development process rather than an afterthought.
- Skill development: TDD and ATDD require specialized skills in writing effective tests and implementing test automation. Development teams may need to invest time and resources in acquiring these skills.
- Time and resource allocation: Transitioning to TDD and ATDD might initially lead to a perceived increase in development time, as writing tests before code can seem slower. However, the long-term benefits of improved code quality and reduced debugging time should be considered.
- Collaboration and communication: ATDD relies on effective collaboration and communication among developers, testers, security, operations and business stakeholders. Establishing a culture of open communication and collaboration is crucial.
- Tooling and infrastructure: Implementing TDD and ATDD may require investments in new tools or changes to the existing development infrastructure to support test automation and continuous integration.
In the context of TDD and ATDD, the roles of quality assurance (QA), security and operations teams are crucial for ensuring a well-rounded and robust software development process. Their involvement helps to address various aspects of the application, leading to a more reliable and secure product.
- Quality Assurance (QA): QA team members collaborate closely with developers in both TDD and ATDD methodologies. They can help define test cases, create test plans and provide input on acceptance criteria. In ATDD, they play a particularly significant role in developing and validating acceptance tests, ensuring that the application meets the defined requirements and user expectations. QA team members may also be involved in setting up and maintaining test automation infrastructure, analyzing test results and identifying areas for improvement.
- Security: Security professionals contribute to TDD and ATDD by identifying potential security vulnerabilities and ensuring that the application is developed with security best practices in mind. They can help define security-related test cases and help to embed security into the application, reducing the risk of vulnerabilities and improving the overall security posture.
- Operations: Operations teams are responsible for managing the infrastructure and deployment processes that support the application. In TDD and ATDD, they can collaborate with developers to ensure that the application is designed and developed with operational concerns in mind, such as scalability, performance and maintainability. Their involvement also helps to promote a DevOps culture where development and operations teams work together to ensure the application’s success in production environments.
A practical roadmap for teams to transition to TDD and ATDD while overcoming the hurdles mentioned earlier could look like this:

Build awareness and buy-in:
Provide training and skill development:
Start small and iterate:
Foster a culture of collaboration:
Set up the necessary tooling and infrastructure:
Monitor progress with metrics:
Continuously improve and adapt:
Summary

TDD and ATDD methodologies contribute to reducing CI/CD lead times by automating tests during development. This improves code quality, minimizes defects and promotes collaboration among team members from the outset of the development process. By writing tests before the actual code and focusing on meeting acceptance criteria, teams can catch and resolve issues early, leading to fewer delays and reduced time spent on debugging and fixing errors during the integration, delivery and deployment stages. These practices also help ensure that the application aligns with user expectations and operational requirements, resulting in a smoother and more efficient CI/CD pipeline.

Anyone who is passionate about modern quality engineering processes for building high-quality software to participate is encouraged to join new ShiftSync Community from Tricentis to engage in discussions about this topic and other development methodologies.
May 19, 2023