devopsy.kinsta.cloud

Tag: stability

Agile is no Excuse

Did you ever hop in a car and just start driving with no destination? I know someone who used to do that for relaxation, but while I do enjoy a good road trip, the destination is far more important for me than the journey. My Mom lives nine hours away, and I love to go visit her. I know the routes like the back of my hand and just get to enjoy the ride—precisely because I know where I’m going.

Most people wouldn’t sign up for a cruise that refused to tell them what stops were planned or even where the ship was going. But those same people are currently in a boat built by either IT or vendors and they have no clue where that boat is going. This is an issue with the software industry, not with the people in question. I recall a couple of memorable cases (not naming names here, but you no doubt were a victim of or heard of at least one of the examples I can think of off the top of my head) where a vendor suddenly announced they were changing everything, and customers would have to deal with it (moving platform, changing licensing, suddenly becoming a product in a different market …)

That’s crazy. Honestly. You could have a piece of software central to your DevOps operation suddenly change to be less useful or even unsuitable. A company you have a contract with. The advent of “commodity” virtual hardware and everything-as-a-service has made us lose sight of the fact that customers have a need for stability and are not in the least wrong to demand it.

And that brings us to today’s actual topic. More and more vendors respond to “What’s on your roadmap?” questions with, “We’re agile, so we are constantly improving.” That is a non-answer. I am a human, so I’m constantly improving, too. Weaning from a bottle at 18 or 24 months says nothing about the roadmap from today for the next 18 months. As an analyst, I generally start advising users that this stance is bad for them and they should demand any vendor they do business with have a plan that they can count on for at least the next 12 months. And that they should leave vendors that don’t have that, and not sign with ones that won’t if not already a customer. Agile means you are making constant changes. We didn’t ask how often you update; we need to know where you are steering your product. If you can’t answer that question, no one can, and the product is in jeopardy.

The problem is, because we, too, are agile, a lot of IT shops are doing the same thing to their customers. While the feedback loop is shorter and more direct, the fact is that most of you have surprised business users with some new thing. The fact that they often are thrilled to have it doesn’t change the fact that they should have known it was coming long before implementation. Agile doesn’t excuse bad communications, and DevOps is not short for ‘random change on a whim.’

So, demand your vendors offer you predictability – even in a fast-changing market – and offer predictability to your users. It only takes a second to update (automated or via email, Slack, whatever) customers that a change is in the works but not assigned yet. Or even just got assigned; just offer some warning. Be good to your users, and demand vendors be good to you. Agile, DevOps and SaaS do not require unannounced changes (good or bad), so stop acting like they are acceptable when notification systems are already set up.

In short: Stop saying, “We don’t know where we are going, but we’re Agile, so we are getting there super fast!” And stop letting vendors say it to you. You are knocking it out of the park every day. Systems run and people use them and all works as intended… Because of you. Now consider letting users know when changes are coming – not just the big ones, but the ones hidden in “We’re Agile!” Make your vendors do the same to give you peace of mind, too.

January 24, 2024
DevOps: Driving Innovation With Old Habits

DevOps can help drive cost-efficiency, greater productivity and innovation

When software development and IT operations merge and work in a state of harmony, they create a flow that we call continuous product delivery. Ideal for project managers, this flow is the elixir to innovation and enables the agility needed to respond to changing market requirements. It also creates currents of speed, proactiveness and integration within enterprises. However, asynchronous code updates, inadequate IT infrastructure, siloed functioning and inefficient processes can hinder this flow.

In 2009, Patrick Debois and Andrew Clay Shafer devised a framework to break such barriers and coined it “DevOps.” Over the last decade, DevOps has garnered many followers and fans, including myself. Product managers worldwide rely on DevOps to enhance performance, accelerate time to market and innovate with agility. DevOps is more critical today than ever before—when nature mandates a new order of remote working, product managers must rely on DevOps to catalyze innovation and sustain continuous product deployments and updates.

Yet, not all enterprises that have adopted DevOps have unearthed innovation like that of Dyson, which created a new type of ventilator in a mere 10 days to help the UK government scale COVID-19 treatments. Nor have they been able to match the frontrunners of innovation such as Google and Amazon that can deploy code thousands of times per day, compared to low performers that deploy code only about four times per day. The obvious question on everyone’s mind is, “What are these heroic innovators doing differently with DevOps?”

Driving Success Through DevOps

While business executives are aware that implementing DevOps drives business innovation, most are not aware of the old habits that remain, causing unnecessary hiccups within the application cycle and business transformation. As the way forward, organizations must tweak their approach to DevOps to be able to reap more significant benefits of business innovation. They must:

Ensure speed does not outpace stability

Today’s digital-first world has swept through every enterprise. However, to keep up with the rapid dynamics of the digital tide, enterprises need to weed out monolithic architectures and manual processes to turbocharge software development and deployment. Automating the deployment process with the help of microservices and cloud can help enterprises gain speed in software deployments and swim the tide of customer experiences. Speaking from experience, this is a privilege that has allowed me to witness how DevOps practices have enabled enterprises to crunch deployment times from weeks and days to just a few minutes. Needless to say, innovation depends on speed: The faster an enterprise deploys a novel solution, the higher its chance to claim the prize of innovation.

A word of caution here: Many enterprises that have neglected quality and stability while accelerating speed drown in a vortex of defects often. When pushing out a product or an application 30 times faster than the traditional model, enterprises must consider its stability, too. Cloud infrastructure plays a vital role in helping enterprises achieve both stability and speed. On-demand self-service, networks that cater to multiple devices, high scalability and analytics are some of the primary advantages that enable enterprises to deploy faster without impacting the core.

Losing stability for the sake of speed can cause organizations to regress instead of leaping forward. In recent times, British Airways has been in the news for repeated issues in its IT infrastructure’s stability. It has had to cancel several flights and inconvenience passengers because software upgrades and incidents have caused its infrastructure to fail. Enterprises must consider these four metrics—lead time for changes, deployment frequency, time to restore service and change failure rate—to receive the value of their DevOps investment.

Make security equal to agility

The pandemic-driven remote working has seen the video communication solutions provider Zoom soar to 300 million active monthly users last month. The brand showed great agility in meeting the surge in demand but became vulnerable to cyberattacks—500,000 Zoom passwords were available for sale in the dark web’s crime forums last month. This is an excellent example of how security is usually an afterthought when compared to agility; not prioritizing security in the DevOps cycle can place teams and their organizations at significant risk, creating vulnerabilities and exposing organizations to bad actors.

In my experience, putting security checks in place early in the DevOps process can help enterprises shift left more effectively. As a first step, security and development teams need to communicate better through the continuous delivery flow. Security teams need to play more proactive roles in foreseeing risks in innovative solutions and routine software updates. Leaving security for the last will only force organizations to be reactive versus proactive.

Build collaboration, not silos

Gartner predicts that through 2022, 75% of DevOps initiatives will fail to meet expectations due to issues around organizational learning and change. DevOps is a process that provides tools but it is also a philosophy that promotes collaboration among teams in software development. It catalyzes an ecosystem that encourages high collaboration and information-sharing. In a DevOps culture, failure is not shot down but leads to inquiry that drives solutions. Seamless information flow is the critical outcome of a DevOps-driven organization. Proper information flow is timely, answers questions and is understandable while needing the support of tools and technologies to drive success.

The General Motors Freemont Assembly line has been a great case study of how poor workforce culture can cause an enterprise to shut down. As one of the largest automobile factories in the United States, the manufacturer had to shut shop in 1982 due to rising losses attributed to an unruly workforce that was protected by a strong union. Within the next two years, the same factory was re-opened in collaboration with Toyota Motors, and it was called NUMMI.

NUMMI hired up to 85% of the old workforce of GM’s Freemont assembly. The difference was the Toyota culture–one that focused on learning and improvement and set a framework of how leadership interacted with and inspired teams. This change was fundamental in driving success for NUMMI, which reduced the time to manufacture a car to 20.8 hours in 1986—a drastic improvement over 43.1 hours in 1978.

At a time when we are witnessing unprecedented disruptions to supply chains, manufacturing and employment, enterprises need to deep dive into functioning to explore new opportunities for cost-efficiency, greater productivity and innovation. DevOps can help achieve these and build the foundations for continuous innovation. Investments in automation, cloud and culture will help enterprises realize the continuous flow of product delivery and innovation. The wave of innovation in the new normal will be propelled by the forces of time-tested practices and new next-generation approaches to enable enterprises to shift-left.

September 1, 2020
How Mainframes Live On

TL/DR: Highly controlled change is sometimes the best answer.

For much of my career, I’ve worked in mainframe shops. I have never worked on a mainframe team, but I have worked closely with them. I’m in Northeast Wisconsin, and at one point we had the highest per-capita number of mainframes in the world, so pretty much anyone from this area can say the same. They were the “stable core” of the system for which I was the “new shiny”—and at times, that was painful.

If you’re an established enterprise, even after years of trying to eliminate the mainframe, you probably still have one. They don’t go away so easily because, despite frustrations with rate of change, they do host core systems in established organizations and are reliable. I have seen similar in client server systems that are core to the business.

We are reaching a point where there are some great products out there to help modernize the mainframe and/or split up client/server applications. Some of them have been around a while (Linux) and are coming into wider adoption, some of them are building on that bit of more familiar interface and many are simply making the UI more familiar to developers like me that only ever developed in a mainframe terminal for college, if at all.

One of the main reasons why mainframes and future iteration’s applications are “sticky” is because they, and the systems they host, are generally reliable. No matter what you think of the timeline for updates, the data coming out of the mainframe and more controlled distributed applications is considered accurate in most organizations, over most other sources.

To be clear, there are groups like this from each successive wave of computing. The client/server app that replaced your mainframe customer information system has learned over time that a fault in their system impacts the core of the business, for example.

The goal is to increase the rate of change, without undermining the reliability. You hear a lot about throw-away instances and “fail fast,” but dropping your billing system isn’t really an option, so these need to be tempered with the knowledge of what failure implies in core systems.

Make the change, do the modernization, but listen to lessons learned on core systems. Take into account that your informational website being down is a serious problem for marketing, but the billing system being down is a disaster for the company. There is a degree of difference, and the approach should be less about continuous deployment and more about stable deployment. Yes, it can be a scale, and you can start the motion from one to the other slowly to make sure you’re covering your bases.

In short, assess the risk levels of these systems before deciding how to proceed. This is pretty much common-sense advice—but better to read it and not need it, than to find out when corporate revenues are impacted.

I worked at one shop where a certain core application was using technology that was a couple of generations old and no longer supported. The pool of available developers was shrinking, but it was an important application. This type of thing is where modernization is critical; having someone say, “We don’t touch that,” is no longer acceptable. That story ends in disaster when the system fails, and no one knows how to troubleshoot—let alone fix—it.

As with so many things, the balancing act is up to you; determine your organization’s level of risk acceptance, then determine the impacts of this application being unavailable and proceed from there.

Each step of mainframe development is a little different from what developers coming out of college expect, but there are new tools and mechanisms available of which these developers can take advantage.

— Don Macvittie

March 1, 2019
DevOps: The Innovation Power Couple
If “Dev” and “Ops” are the power couple of tech, “DevOps” is the holy matrimony that takes innovation from trainwreck to fast and repeatable. At most businesses, however, Dev and Ops still sleep in separate rooms and agree only on their mutual contempt.

Dev (Development) and Ops (IT Operations) usually are divorced or ready to serve the papers because their incentives clash. In Ops’ opinion, Dev is an idiot—always out late drinking with friends, he comes up with wild ideas and wants to implement them immediately. Dev can’t make anything work on real servers and can’t write scalable, stable code.

To Dev, Ops is a total code-blocker—she makes Dev go through “change management boards” and slows down progress. She lets the competition get to market first. So every day, Dev and Ops have an angry shouting match that ends in tears.

I wish I were exaggerating. My coworker Brandon Cipes once walked into an interview for a product development position. The interviewer broke down crying because the relationship between Dev and Ops had gotten that nasty. They refused to meet with each other, let alone be in the same room.

When Dev and Ops have a dysfunctional relationship, they struggle to communicate, collaborate and get results. Their organization will fail at innovation because it lacks the cultural, technological and procedural base to do otherwise.

The missing base is DevOps. It’s a unified strategy for Development and IT Operations, and implementing it can have the following effects:
1. Communication Becomes Unavoidable
Normally, Dev and Ops play “telephone” because they’re physically separated and have too many points of communication. Emails, IMs, grudgingly accepted meetings and passing conversations ensure Dev and Ops never get the full message.

DevOps fixes communication by assigning Dev and Ops personnel to the same teams. They sit in one room and work in a single squad (a.k.a. a pod or team) of no more than eight to 10 people. If they turn their heads, they see each other’s beautiful faces. Until promotions or career change do them apart, they can’t avoid talking and working toward common goals.

In essence, DevOps puts “cross-functional” teams together and thereby eliminates hand-offs. Fewer points of contact, less telephone, better communications.
1. Collaboration Gets Automated
For many Dev and Ops teams, “collaboration” is another word for bureaucracy. It’s TPS report-ish stuff characterized by change review board meetings and laborious documents that tell Ops how to deploy new code. Dev and Ops might accidentally (or intentionally) withhold information from each other.

In DevOps, collaboration is automated. Instead of change review board meetings, you use version-controlled software. The hand-offs from Dev to QA and QA to Ops are done by machine. A script deploys the code for Ops so there’s no risk of misfiring.

“Collaboration” is no longer an ordeal or choice. Documenting and sharing the squad’s collective knowledge is wired into the process.
1. Fast and Stable Releases
Normally, a software deployment is like a sloth trying to cross US 101: slow, awkward and soon to be roadkill. That’s all thanks to bad communication and bad collaboration.

With DevOps, you catch problems before they go live. Test automation prevents any need for manual checks. One person corrects the problem in a matter of minutes whereas before, a team of five might spend eight hours tracking down and fixing one bug. Ops feels confident that tested code won’t cause havoc.

Thus, the upshot of DevOps is speed and stability. Before, Dev and Ops delivered maybe a few times a year and faced regular breakdowns. DevOps can deliver multiple times daily without a hitch.

Say ‘I Do’

DevOps is a cultural, procedural and technological transformation. As in a marriage, both partners are responsible for good outcomes. Linked as they are on one squad, Dev learns to care about stability and Ops begins to value innovation.

If “agile” is about making businesses nimble and quick, DevOps is about applying agile principles to technological development and operations. Ultimately, it’s about beating your competition to the next big thing.

Dev, easy on the drinking escapades. Ops, live a little. Tie the DevOps knot and make some innovation babies.

About the Author / Zubin Irani

Zubin Irani is co-founder and CEO of cPrime, a full-service consultancy that focuses on agile transformations and project delivery solutions for hundreds of software teams, including more than 50 of the Fortune 100. Connect with him on LinkedIn.
April 21, 2017
The 3 Secrets to Overcoming the Agility-Stability Paradox

Regardless of whether an organization subscribes to a bimodal IT approach, one thing is clear: It’s more important, and more difficult, than ever to fight the agility-stability paradox. Today’s businesses must build on a delicate balance of being compliant, plan-driven and keeping the lights on while also seeking ways to become more fluid and agile with high customer touch. The barriers standing between these two key initiatives are numerous and complex; whether it’s mismatched technology or a lack of mutual understanding, the agility-stability paradox remains.

In a lot of ways, this paradox comes down to a difference between traditional and new forms of IT. Traditional IT can be described as predictable–it’s compliant and plan-driven, with conventional processes and long-term cycles. But as the next generation of IT starts to take hold, today’s businesses must find ways to also fold in a new IT that’s dynamic, fluid, agile and continuous, with short cycles and high customer touchpoints. In this way, traditional IT and new IT are polar opposites, and many companies struggle to find a balance between the two. Competing attitudes, too much change at once and tooling challenges all stand in the way of harmony between the reliability of the traditional approach and the need for agility in the new way of doing things.

Balance is Possible

While there is no magic tool or wand that can solve for the incongruities so many organizations face, balance is possible. Enterprises don’t have to choose reliability, customer experience, uptime and performance over agility, release velocity, new markets and competitive response. The real secret to overcoming the agility/stability paradox lies in establishing who, what and how. What this means is that with the right people, processes and tools in place, any company can achieve the balance between agility and stability needed for success.

Who? People

People serve as the “who” of finding balance between reliability and agility. This is the chance to integrate your culture into solving the agility/stability paradox. Start by identifying how each member of the team needs to adapt and shift to accommodate changes. The real trick to getting the people aspect of things right is communication. Define a communication path and be really clear about why your organization needs to transform, how it will be transformed and when everyone can start expecting to see some real adjustments take shape. Orchestrate a response mechanism across your teams and enable social operations collaboration to facilitate execution. Be sure to provide ongoing updates and encourage feedback.

What? Process

Now that you’ve established who needs to play a role in overcoming the agility/stability paradox, it’s time to talk about what exactly needs to change. Process is absolutely crucial here. There are a few keys to establishing the right process for balancing traditional and new IT models. The first is to identify, and not dictate. Dictating change from the top down is never going to be effective. Instead, work together to determine areas of weakness and highlight them. Use your observations as a guide, not a mandate.

From a more tactical perspective, be sure to streamline workflows wherever you can. Define instead of inheriting urgencies and alerts. Don’t be afraid to rely on automation wherever possible, either. By automating processes that help out on the people side of things, you can free up your team to spend less time on tedious, manual tasks and focus more on innovation.

How? Tools

Tools are your best friend when it comes down to actually making all this change happen. Considering the vast number of tools in the market, choosing a toolset may appear daunting at first. But if you start with strong integration across the operational modes already in place, then you’re already in good shape. In particular, to bring new, agile IT into the fold, you’ll want to incorporate IT service management (ITSM) solutions and establish a DevOps tool chain.

Don’t Boil the Ocean

The most important thing to remember is to start small—you don’t have to boil the ocean. Build a culture where traditional and new IT environments are encouraged to work closely together, where you can build and run software that addresses both agility and reliability to respond to the pressures on the business. Start by defining what is is right for your team and organization and acknowledge the existing culture and people. Don’t be afraid to experiment and create isolated learning environments, either. Just make sure to maintain a clear connection to customer value and use this as your North Star. Finally, don’t forget to celebrate. Celebrate your failures and learn quickly from them.

About the Author / Manish Kalra

Manish Kalra is director of Product Marketing at PagerDuty. He is responsible for driving product marketing strategy, product positioning, launches and collateral development for the PagerDuty incident management platform. Prior to PagerDuty, he served as Director of Product Marketing at companies like Sumo Logic and Splunk.

September 22, 2016
The business benefits of DevOps

As DevOps has become more of a mainstream concept it seems like there’s less need to define what it is or explain what it does. As prevalent as DevOps is for those of us who work in the world of DevOps or organizations that have already embraced DevOps there are still many businesses out there muddling along the old-fashioned way and wondering why they should care about DevOps.

The simple answer is that not caring about DevOps may be suicidal. The 2014 State of DevOps report found that organizations successfully implementing DevOps tools and practices lead competitors in almost every relevant metric. Two statistics that stand out from that report are that companies using DevOps are twice as likely to exceed profitability, market share, and productivity goals, and had 50 percent higher market capitalization and growth than rival businesses.

Why is that, though? What is it about DevOps that leads an organization to be more profitable and productive? It’s a combination of three things, really: faster delivery times, greater stability / reliability, and freeing up time to focus on value.

Let’s break those down some. DevOps is an evolution of the Agile development methodology. In fact, I’ve asserted in the past that DevOps is really just a matter of taking Agile principles and applying them to the rest of the company outside of software development. Whether Agile or DevOps the result when it comes to developing software is faster, more frequent delivery.

Next we have greater stability and reliability. It might seem that the traditional linear method of developing software and releasing it on a pre-defined schedule is more stable. Developers and the rest of the organization have plenty of time to refine the code before it’s deployed, they know exactly when it will be deployed, and they know it basically won’t change for another year or so when the next major release is scheduled. The reality is the complete opposite in many cases.

Developing with a rigid process methodology like waterfall means that only the changes and features identified at the beginning of the project are included in the release. As the need for improvements or additional features is identified during development those things are sidelined and added to a list to be pursued in the next release cycle. By the time the software is developed, tested, and deployed it is already out of date to some extent and bugs or issues that are identified take longer to address or get pushed to be handled in the next major release.

The whole process is slow and results in less stability and reliability. DevOps yields greater stability and reliability because the faster development and more frequent release schedule enables organizations to identify and resolve issues immediately.

Those two things combined mean that developers and others throughout the organization have more time available to focus on improvements and innovations that contribute to the bottom line. Organizations that have embraced DevOps also generally have tools in place to automate routine tasks which enables individuals to dedicate more time to growing the business and less time to pissing on fires and maintaining the status quo.

DevOps has come a long way in just a few short years. There may still be companies out there trying to figure out why they need DevOps, but those companies won’t be around long. They’ll either figure it out and jump on the DevOps bandwagon or they will no longer exist to worry about it.

September 29, 2015
Continuous Testing System Stability
The effort to speed up continuous testing will be wasted if the underlying system that continuous testing depends on is not stable. In my prior blog
Continuous Test Results Analysis – at the Speed of DevOps, I discussed the importance and practices for speeding up continuous testing results analysis. However, unless the continuous test system that the test results are derived from is rock solid, the results will at best be unreliable and at worst – results may not be generated at all!

One of the first rules of DevOps is to “keep going and don’t stop”. Stopping is the equivalent of pushing the red button on a factory assembly line effectively causing the entire line to shut down. Then, panic ensues because every second that the line is down affects all of production leading to increasing costs not to mention restarting the line is substantial also.

Many techniques are employed to minimize the need to stop testing despite reported product test failures. However if the underlying continuous testing system itself is not stable, then the entire DevOps system including continuous testing will immediately be interrupted until the system recovers. Development team members are looking for results, so the time savings for speeding up the availability and the total work associated with results analysis is huge for every minute of results analysis time saved on each CI/CT cycle. When aggregated over multiple code branches, a few minutes saved per test result can be man-years of time saved over a year!

So what can be done to ensure the continuous test system itself is stable and produces reliable results even in the face of inevitable occasional system failures? After all, there is no such thing as a truly perpetual machine that never ever breaks down. Below are some suggestions in a checklist format that have proven useful in high performance DevOps deployments.
1. Build a fast-response multi-discipline team: There are a wide variety of ways any system of interconnected computers and applications can fail whether they are virtualized, internal, or running on a cloud used remotely from a data center in the cloud. The following list of suggestions for the team will assure fast recovery when the system does experience failure.
  • Assign a continuous testing system architect with responsibility to design system stability SLAs, design high-reliability system architectures and define sufficient team workflows suitable for disaster prevention and recovery consistent with recovery time SLAs.
  • While the team members may be in-or-out sourced, full or part-time, there needs to be clearly responsible first responders and 2nd line escalation team members with definitive workflows that take care of failure events 24/7 for every hardware, software, tools, network and firewall component that the system depends on.
  • Assign product team leads to plan changes needed for new products and services.
  • 24/7 availability system designs are pricey to build, expand, and maintain, so finance and accounting team members need to be part of the team to keep clear, current cost data relevant to the continuous testing system.
  • A senior budget manager approves expenses needed for stability without delay.
2. Anticipate failures – prepare, prepare, prepare!: Don’t wait for a failure before testing system recovery procedures! Documented disaster prevention and recovery plans or high-availability tools won’t save the continuous test environment if the team is not ready to react during a failure event.
  • Run failure simulations for each defined failure scenario at a frequency that matches the expected probability of each failure.
  • Cross-train team members on system recovery best practices so the skills will be ready when most needed – ensure no one person has essential unique recovery skills.
3. Choose test tools designed with high stability system administration capabilities: Look for the following features designed into in test tools to ensure the continuous test environment can be configured and managed for high reliability.
  • Tools are implemented as autonomous services, controlled through RESTful APIs, that can be separately invoked and managed
  • Test system service monitoring is built-in including up-time, wait-time and usage metrics for each use case of each test system service.
  • Diagnostic capabilities are built-in to each service. When a failure occurs the failure log includes helpful descriptions including causal information.
4. Engineer the system for stability: Typical continuous testing systems have many “moving parts” and many of them have to work in concert for the system to perform properly. It is common engineering knowledge that the reliability of a system is determined by the reliability and configuration of its parts. Here are some suggestions to improve the overall system reliability.
  • Choose powerful servers and virtual machines (VMs) with ample processor speed, fast I/O, large and fast memory and data storage. This is important to minimize the chance of system failures due to possible overloading of these resources.
  • Deploy multiple servers and VMs for each system component with enough redundancy so that no one component failure will interrupt the entire continuous testing system.
5. Configure the test workflow to avoid false negatives: There is no point in reporting product test failures to users if the reason for the failure was an underlying continuous testing system failure instead of a failure of the product itself. Such “false negatives” are a big waste of time. Here are some suggestions to reduce the chance of false-negative test reports.
  • Assign an application to continuously verify, in-real time, that all the system components and interconnections are operating and report these results to the continuous testing report aggregation tool. These results are called CT system environment test results.
  • Use the CT system environment test results within the continuous test results analysis tool to determine if any test failure coincides with an environment failure. Mark each verdict “inconclusive” instead of “fail” if it does and include a description of the environment failure in the verdict log.
6. Maintain the system to assure long-term stability: Every system needs maintenance at some point. Hardware components wear out and software components become obsolete. Each time a system component is replaced or upgraded to new versions there is a risk of instability. Here are some recommendations to reduce the risk of introducing instabilities to the continuous testing system when changes are implemented.
  • Create a maintenance environment that mirrors the test and production environments.
  • Define a test suite that will exercise all system components and topologies.
  • Test changes in the maintenance environment before deployment.
  • Occasionally run failure/recovery simulations in each production environment.
The above is a partial list of suggestions for continuous test system stability that have been proven to yield good results for DevOps. At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

What do you think of these suggestions and do you have others that should be mentioned?
April 1, 2015