devopsy.kinsta.cloud

Tag: systems

Why You Shouldn’t Forget Workflows With Agentic AI Systems

As Matthew McConaughey runs around an airport wishing he had used an AI agent to book his flight, we could be forgiven for assuming that the technology is now a well-understood and integral part of everyday life for most people.

Of course, that’s not true. In reality, it’s quite far from the truth, even for businesses, but that is not to say that things aren’t going that way. In the future, many of our decisions will be handled by agents, or at least heavily informed by them. But in areas where we want more control, we will likely retain decision-making power, following a programmatic approach or workflow.

When booking a vacation, for example, we could consider different countries, cities and situations, the cuisine, the weather and the culture. Or we could just get agents to put together some options based on our preferences. The third option would be a mix of the two, such as choosing where you want to go and allowing agents to choose and book activities — again, based on our preferences and hobbies.

The choice will come down to how much autonomy we want to retain. And it’s the same with AI systems. In this article, I’ll explain the move from one decision-making method to another and show why both are still valid and complementary.

How Models Think

Traditional models have limited knowledge. They are trained on a finite amount of data, which puts some restrictions on the kinds of problems they can solve. For example, the first public iteration of ChatGPT could only deliver answers to prompts based on the information available up to 2022, as its underlying LLM, GPT-3.5, was trained on data up to that point. This is also why you will often hear or read about models that have been fine-tuned for specific use cases, such as HR or legal, as these use cases require additional steps and data to tailor them to the use case.

These models can be used to underpin generative AI applications within an organisation. They can therefore broadly answer queries relating to the models that have been fine-tuned, such as general HR policies or legal practices, but cannot answer specific questions employees might have that relate to their circumstances or organisational data. To unlock this ability, we can create a workflow that enables models to access and retrieve data from proprietary systems and internal corpora of data. This is what is known as a Retrieval-augmented generation (RAG) system, but one that is programmatic, i.e., a human has built in the capabilities that enable the model to access systems outside of its training data to extend its capabilities.

With tool-augmented LLMs, it is therefore possible to create advanced workflows that are limited only by the modular elements developers choose to include, meaning the control logic is determined by a human.

An agentic system uses an LLM’s ability to reason to determine the steps required to complete a task or fulfill a request. The difference between this approach and programmatic workflows boils down to the amount of autonomy we want our AI systems to have.

Structured workflows involve a linear progression to deal with problems. In many instances, this is the most appropriate approach, as it guarantees that the system won’t go beyond what it has been programmed to do. However, as the reasoning capabilities of LLMs have continued to develop, it is now possible to hand the control logic of compound AI systems over to the LLMs themselves. This is thanks to “chain-of-thought” reasoning that enables an LLM to break down problems, e.g., prompts, into logical steps, taking a little longer to consider each stage so that a more reliable and accurate output is delivered.

Orchestrating Agentic Autonomy

Orchestration frameworks, such as LangGraph, are used to build in the stages of reasoning that agents use to complete their tasks. This can include a stage in which the agent considers its initial answer and determines whether to proceed with or change its output. Orchestration frameworks also build in the tools that agents can use to complete their tasks. A simple example might be providing a specific tool that retrieves live meteorological information. By giving the agent access to this tool, it can bring in this live information for the use case that is being served, such as booking activities on vacation. No one wants to be struggling through outdoor activities when there’s a storm raging, which is why an agent who can reason on the information they get from calling a weather tool is particularly useful.

These agents are built in line with a framework known as ReAct (‘Re’ for ‘reasoning’ and ‘Act’ for ‘acting’), which ensures the agent follows the Thought-Action-Observation reasoning paradigm. These stages mean the agent will first consider the tools it has at its disposal to return a satisfactory response, such as a weather app, then take action by retrieving the information, observing the situation from the raw data and then presenting the information in a user-friendly format following a further reasoning stage.

Agents are the new kid on the block, but it doesn’t mean they’re here to usurp traditional workflows. The two work rather well in conjunction. Workflows can hand off to an agent, or team of agents, to complete tasks, just as with the use case I outlined at the beginning of this article — and it’s no different in a business context. The reverse can also be true, with agents handing off to pre-defined workflows once a course of action has been suggested. This can then lead to further feedback loops to iterate and ensure accuracy and better decision making.

Ultimately, the amount of autonomy we build into a system will always be determined by the use case, which will, in turn, be determined by the exact amount of control that is required.

May 21, 2025
Implementing Threat Modeling in a DevOps Workflow
Security remains paramount in the contemporary digital landscape, especially within the DevOps workflow. DevOps practices aim to streamline the software development lifecycle (SDLC) by fostering collaboration between development and operations teams, ensuring a secure SDLC. However, the rapid pace of DevOps can sometimes overlook critical security concerns. Integrating threat modeling into the DevOps workflow is essential to identify and mitigate potential security threats. This comprehensive guide explores how threat modeling can be effectively implemented within a DevOps framework to enhance security.

Understanding Threat Modeling

Threat modeling is a structured approach to identifying, evaluating and mitigating security threats to a system. It involves anticipating potential attacks, understanding their impact and devising strategies to defend against them. This proactive approach to security ensures that vulnerabilities are identified early in the development process, reducing the risk of security breaches post-deployment.

Key Elements of Threat Modeling
- Assets Identification: Determining what needs to be protected, including data, systems and services
- Threat Identification: Recognizing potential threats that could exploit vulnerabilities
- Vulnerability Identification: Identifying weaknesses in the system that could be exploited
- Mitigation Strategies: Developing countermeasures to address identified threats and vulnerabilities.
Integrating Threat Modeling into the DevOps Workflow

Integrating threat modeling into a DevOps workflow involves embedding security practices throughout the development and operations lifecycle. This approach ensures continuous security assessment and improvement, aligning with the DevOps principles of continuous integration and continuous deployment (CI/CD).

1. Shift-Left Security: Integrating Early in the SDLC

One of the fundamental principles of DevOps is the concept of ‘shift-left’ security, which means incorporating security practices early in the SDLC. By integrating threat modeling at the initial stages, potential security threats can be identified before they become deeply embedded in the codebase. This early integration supports a secure SDLC, ensuring that security measures are considered from the beginning and throughout the development process.

Benefits of Shift-Left Security
- Early Detection of Vulnerabilities: Identifying security issues early reduces the cost and effort required to fix them later
- Improved Code Quality: Encouraging developers to write secure code from the beginning
- Reduced Time-to-Market: Minimizing the risk of delays caused by security issues discovered late in the development process.
2. Continuous Threat Modeling

Continuous threat modeling aligns with the DevOps principle of continuous integration and deployment. This practice involves regularly updating threat models to reflect changes in the application, infrastructure and threat landscape. Automated tools can assist in maintaining up-to-date threat models, ensuring that security assessments are conducted consistently.

Key Aspects of Continuous Threat Modeling
- Automated Security Scans: Integrating security scanning tools into the CI/CD pipeline to detect vulnerabilities continuously
- Regular Updates: Revisiting and updating threat models as the application evolves
- Monitoring and Feedback: Utilizing feedback loops to improve threat models based on new information and detected threats.
3. Collaboration Between Development, Operations and Security Teams

Effective threat modeling in a DevOps environment requires close collaboration between development, operations and security teams. This cross-functional collaboration ensures that security is considered at every stage of the development lifecycle.

Strategies for Enhancing Collaboration
- Security Champions: Appointing security champions within development and operations teams to advocate for security best practices
- Regular Security Training: Providing ongoing security training to developers and operations personnel to raise awareness and build skills
- Shared Responsibility: Promoting a culture of shared responsibility for security across all teams.
4. Integrating Threat Modeling Tools

Numerous tools are available to facilitate threat modeling within a DevOps workflow. These tools can automate various aspects of threat modeling, making it easier to identify and mitigate potential security threats.

Popular Threat Modeling Tools
- OWASP Threat Dragon: An open-source tool for creating threat models and visualizing potential threats
- Microsoft Threat Modeling Tool: Provides a systematic approach to threat modeling with pre-defined templates
- IriusRisk: A platform that integrates with CI/CD pipelines to automate threat modeling and risk assessment
5. Case Studies and Statistics

Integrating threat modeling into a DevOps workflow is essential for identifying and mitigating potential security threats before they become significant issues. Real-world case studies and statistical evidence can illuminate the tangible benefits of this approach. This section delves into notable case studies, such as the Capital One and Equifax data breaches, to underscore the critical importance of threat modeling and continuous monitoring in DevOps practices.

Capital One Data Breach

In 2019, Capital One experienced one of the most significant data breaches in recent history, exposing the personal information of over 100 million customers. The breach was caused by a misconfigured web application firewall (WAF), which allowed an attacker to access Capital One’s Amazon Web Services (AWS) cloud environment. This incident highlights several critical points about the importance of continuous threat modeling and monitoring in a DevOps workflow.

Capital One’s breach underscores the necessity of rigorous configuration management and monitoring. A misconfiguration, such as the one that occurred in this case, can have devastating consequences if not detected and corrected promptly. Continuous threat modeling could have identified this misconfiguration early, allowing Capital One to address the issue before it was exploited. Furthermore, regular security assessments and automated monitoring tools could have alerted the security team to the vulnerability in time to prevent the breach.

The Capital One breach also illustrates the importance of a comprehensive approach to security that includes not only technical measures but also organizational practices. By fostering a culture of security awareness and implementing robust security practices throughout the DevOps lifecycle, organizations can reduce the risk of similar incidents. Continuous threat modeling ensures that security is an ongoing consideration, rather than a one-time task, and helps to create an environment where potential threats are identified and addressed proactively.

Equifax Data Breach

The 2017 Equifax data breach exposed the personal information of approximately 147 million people, making it one of the largest and most damaging breaches in history. The root cause of the breach was an unpatched vulnerability in Apache Struts, a widely used web application framework. This incident highlights the critical need for integrating automated security scanning tools into the CI/CD pipeline, a key aspect of threat modeling in DevOps.

The Equifax breach could have been prevented if the vulnerability had been identified and patched in a timely manner. Automated security scanning tools, integrated into the CI/CD pipeline, can continuously monitor for known vulnerabilities and alert developers to the need for patches. By integrating such tools, organizations can ensure that their applications are always up to date with the latest security fixes, reducing the risk of exploitation by attackers.

Moreover, the Equifax breach emphasizes the importance of a proactive security posture. In a DevOps environment, where rapid development and deployment are the norms, it is crucial to have automated systems in place that can keep up with the pace of change. Continuous threat modeling and automated security scanning provide a means to do this, enabling organizations to identify and mitigate vulnerabilities as they arise, rather than after they have been exploited.

Statistics Supporting Threat Modeling in DevOps

Statistical evidence further reinforces the value of integrating threat modeling into DevOps practices. According to Gartner, “By 2022, 90% of software development projects will claim to be following DevSecOps practices, up from 40% in 2019.” This significant increase reflects a growing recognition of the importance of integrating security into the DevOps workflow. Organizations that adopt DevSecOps practices, including continuous threat modeling, are better positioned to identify and address security threats early, reducing the risk of breaches and other security incidents.

The State of DevOps Report 2021 by Puppet provides additional insights. The report indicates that organizations integrating security into their DevOps practices are 2.4 times more likely to detect security incidents before they cause significant damage. This finding underscores the effectiveness of proactive security measures, such as continuous threat modeling, in identifying and mitigating threats before they can have a detrimental impact.

A survey by the SANS Institute revealed that 64% of respondents experienced improved security posture after implementing DevSecOps practices. This improvement is likely due to the continuous nature of threat modeling and security assessments in a DevOps environment, which helps organizations stay ahead of emerging threats. By continuously updating threat models and integrating automated security tools, organizations can maintain a robust security posture that adapts to new challenges and vulnerabilities.

Benefits of Integrating Threat Modeling into DevOps

The case studies and statistics discussed highlight several key benefits of integrating threat modeling into a DevOps workflow:
- Early Detection of Vulnerabilities: Continuous threat modeling and automated security scanning tools enable organizations to identify vulnerabilities early in the development process, reducing the risk of exploitation.
- Proactive Security Posture: By continuously assessing and updating threat models, organizations can maintain a proactive security posture that adapts to new threats and vulnerabilities.
- Improved Security Awareness: Integrating threat modeling into DevOps fosters a culture of security awareness, encouraging all team members to prioritize security in their work.
- Reduced Risk of Breaches: Continuous threat modeling helps organizations identify and address security threats before they become significant issues, reducing the risk of data breaches and other security incidents.
Statistics
- According to a report by Gartner, “By 2022, 90% of software development projects will claim to be following DevSecOps practices, up from 40% in 2019.”
- The State of DevOps Report 2021 by Puppet indicates that organizations that integrate security into their DevOps practices are 2.4 times more likely to detect security incidents before they cause significant damage.
- A SANS Institute survey revealed that 64% of respondents experienced improved security posture after implementing DevSecOps practices.
6. Best Practices for Implementing Threat Modeling in DevOps

Implementing threat modeling within a DevOps workflow is essential to ensuring that security is embedded into the SDLC from the outset. By following best practices, organizations can maximize the effectiveness of their threat modeling efforts, leading to more secure and resilient applications. The following sections expand on the best practices for implementing threat modeling in a DevOps environment.

Early and Continuous Integration

Integrate Threat Modeling Early

The concept of ‘shift-left’ security emphasizes the importance of integrating security measures at the earliest stages of the SDLC. By incorporating threat modeling during the initial phases of development, potential security issues can be identified before they become deeply ingrained in the codebase. Early integration of threat modeling ensures that security considerations are a fundamental part of the design and architecture process.

Continuous Updating of Threat Models

Threat models should not be static documents that are created once and forgotten. Instead, they need to be living artifacts that are continuously updated to reflect changes in the application, infrastructure and the evolving threat landscape. Continuous updating ensures that the threat models remain relevant and effective in identifying and mitigating potential security threats.

Utilize Automated Tools

Automated tools play a crucial role in facilitating continuous threat modeling and security assessments. Tools such as OWASP Threat Dragon, Microsoft Threat Modeling Tool and IriusRisk can automate various aspects of threat modeling, making it easier to integrate these practices into the CI/CD pipeline. Automation helps ensure that threat modeling is performed consistently and efficiently, reducing the burden on development and security teams.

Collaborative Approach

Foster Cross-Functional Collaboration

Effective threat modeling requires close collaboration between development, operations and security teams. This cross-functional approach ensures that security is considered from multiple perspectives and throughout the entire development lifecycle. Collaboration can be fostered through regular meetings, joint workshops and shared documentation.

Appoint Security Champions

Appointing security champions within development and operations teams can help promote a security-first mindset. These individuals act as advocates for security best practices, ensuring that security considerations are prioritized and integrated into everyday workflows. Security champions also serve as points of contact for security-related questions and issues, bridging the gap between different teams.

Provide Regular Security Training

Ongoing security training is essential to keeping all team members informed about the latest threats, vulnerabilities and best practices. Training should cover fundamental security principles, specific threats related to the technologies being used and hands-on exercises in threat modeling and mitigation. Regular trainings help build a culture of security awareness and ensure all team members have the knowledge and skills needed to contribute to threat modeling efforts.

Leverage Automated Tools

Automate Threat Identification and Mitigation

Threat modeling tools can automate the identification and mitigation of potential security threats, making the process more efficient and consistent. Tools such as OWASP Threat Dragon, Microsoft Threat Modeling Tool, and IriusRisk offer features such as visualizing threat models, suggesting mitigations and integrating with CI/CD pipelines. Automation reduces the manual effort required for threat modeling and helps ensure that security assessments are performed regularly.

Integrate Security Scanning Tools into the CI/CD Pipeline

Integrating security scanning tools into the CI/CD pipeline allows for continuous vulnerability detection. Tools such as Snyk, Veracode and Checkmarx can automatically scan code, dependencies and configurations for vulnerabilities as part of the build and deployment process. Continuous scanning helps identify security issues early, providing developers with immediate feedback and allowing them to address vulnerabilities before they reach production.

Regular Reviews and Updates

Regularly Review and Update Threat Models

Regular reviews and updates of threat models are essential to ensuring their continued effectiveness. As applications evolve and new features are added, threat models must be revisited to account for these changes. Regular reviews should be scheduled at key milestones in the SDLC, such as before major releases or after significant architectural changes.

Implement Monitoring and Feedback Loops

Monitoring and feedback loops are critical to continuously improving threat models. By collecting data on security incidents, vulnerabilities and threat model effectiveness, organizations can gain insights into areas for improvement. Feedback loops should involve regular debriefs and post-mortem analyses to understand what worked well and what could be improved. This iterative approach helps refine threat models and ensures that they remain relevant and effective.

Metrics and Reporting

Track Key Security Metrics

Tracking key security metrics is essential for measuring the effectiveness of threat modeling efforts. Metrics such as the number of vulnerabilities identified and remediated, time to resolve security issues and the impact of security incidents can provide valuable insights into the security posture of the organization. These metrics should be tracked over time to identify trends and measure progress.

Use Metrics to Identify Areas for Improvement

Metrics can be used to identify areas for improvement in threat modeling and overall security practices. For example, if certain types of vulnerabilities are consistently identified, it may indicate a need for additional training or changes to development practices. Metrics can also help prioritize security efforts, ensuring that resources are focused on the most critical areas.

Demonstrate Value to Stakeholders

Reporting on key security metrics helps demonstrate the value of threat modeling to stakeholders. By showing the tangible benefits of threat modeling, such as reduced vulnerabilities and faster resolution times, organizations can build support for continued investment in security practices. Regular reporting also helps keep stakeholders informed about the security posture of the organization and any emerging threats.

What’s Next?

Implementing threat modeling in a DevOps workflow is crucial for proactively identifying and mitigating potential security threats. By following best practices — such as early and continuous integration, fostering cross-functional collaboration, leveraging automated tools, regularly reviewing and updating threat models and tracking key security metrics — organizations can enhance their security posture and build more resilient applications. As the digital landscape continues to evolve, the importance of proactive security measures like threat modeling cannot be overstated. By embedding security into every stage of the SDLC, organizations can ensure that their applications remain secure and resilient against emerging threats.
July 18, 2024
Go Big: Scalability in the Age of DevOps

Achieving hyperscale is a work of technical art for any business. As companies grow and customers begin to rely more heavily on your service, ensuring reliability during peak business times is mission-critical. An enormous amount of work goes on behind the scenes to ensure scalability, and oftentimes it becomes necessary to redesign or rethink processes and systems to improve them as they face increased pressure.

When customer demand reaches its apex, it’s the ultimate test for any organization’s systems and processes; therefore, it’s important to prepare a comprehensive strategy that focuses on scalability, reliability and hypergrowth in a targeted, calculated and data driven methodology.

The first step on your path to hyperscale is forecasting and gathering data analytics around business and growth patterns for a full year to make smarter forecasts. Set aside time each day throughout the year to go over the allocation of your data and analytics, measuring them for accuracy to see how they stack up to your current predictions for that specific day. This may seem taxing and cumbersome when engineers have other priorities to work on, but devoting time to this in the early months of the year will help you be that much more accurate when it comes to predicting the needs of your customers and how you should go about scaling to meet those needs.

Especially during the holiday season, it’s important to have a minute-by-minute projection of how much traffic you expect and need. That’s where forecasting and an advanced data science model come into play. A common mistake made when it comes to scalability is not getting granular enough about the real pressure points for your systems. Having accurate projections and forecasts in place that feed into your data science model means you can better test your systems and production limits while gaining greater insight into meeting your scaling points. If you find fragilities ahead of time through this, you can assign your engineering team to fix them well in advance.

Another way to truly bolster your scalability efforts is by introducing an operability development model. This entails constantly deploying some controlled chaos into your systems to see how they really stack up and function under pressure, with the goal of up-leveling the capacity of both your operations and system growth. The operability development model allows you to really test the grit of your systems by squeezing the most out of them, prepares you to respond quickly and methodically should the worst case scenario occur and sets you down the path toward continued performance growth and scale.

From here, you’ll want to institute a preventative maintenance model. Think of it like getting an oil change for your car. Instead of waiting until the very moment you need your software to perform exceptionally, start taking a look at the little things that could or are causing problems ahead of time and fix them on a monthly or quarterly basis. A preventative maintenance model helps you inspect, adapt and solve these little nuisances before they become serious problems, granting you the cleanest operating environment possible.

Scalability is king, but availability and reliability are arguably just as critical. Improving availability means reducing latency and getting as close as you can to where your customers utilize your services. This can be achieved by deploying locations around the world to make connections much faster and services more independent. To maintain reliability, you should have processes in place that can quickly respond and offer support to customers 24/7 to maximize consistency and performance.

Another strategy to consider—if it makes sense for your business or if you’re evaluating cloud providers as a growth option—is to double down on your efforts by re-architecting your platform and systems as cloud native from the ground up to support greater levels of scalability and cost efficiency. By going cloud-native, you can achieve independently scalable services so that you don’t have to scale all of your services together. This means your engineers and developers don’t have to spend way too much time figuring out where issues and fragilities occur.

A platform architected as cloud-native allows you to leverage microservices and containerization, creating an environment with more reliable, self-healing, high-tenancy and auto-scaling features that take the most advantage of your hardware and services. An added benefit of a cloud native platform is that you can use your data to find the most opportune times to both scale in advance and descale based on performance and customer expectations. The efficiencies that come from re-architecting as cloud native enable engineers to spend more time on automating software and the delivery pipeline, which is a huge boon for scalability.

At the end of the day, a well-thought-out preparation strategy for mass scalability makes the whole process much more exciting and obtainable. It also greatly reduces lead-up anxiety by getting a head start on planning so that you have more time to sit back and enjoy the fruits of your labor. Lastly, a yearlong scalability strategy enables you to experiment and try new things without the fear of failure. Testing new approaches ahead of time to see what does or doesn’t work will only help you learn and make your scalability efforts that much more effective for the future as your company continues down the path of hypergrowth and hyperscale.

— Jamie Tischart

January 18, 2019
How Systems Engineering Can Help Avoid Product Development Pitfalls

Product development programs all start with great optimism regarding their potential to provide an excellent product to customers, along with excellent profits to the company. But research shows that a majority of product developments end in failure. The reasons for each failure are unique, but the overarching reason often can be described as the lack of systems engineering.

I managed a program several years ago that is a great example of how programs that have an excellent idea, sound technical basis and talented people working on them can still fall into this trap, and how systems engineering and a systems mindset can rescue these programs from the brink of failure and propel them to a successful conclusion.

Here’s the scenario: Scientists working for a Department of Defense electronics supplier devised and patented a method for doubling the frequency range of a radar amplifier in the same physical footprint. Additionally, advances in technology allowed two of the most sensitive components to be replaced by components that were substantially more reliable, increasing the amplifier’s projected life by more than 500 percent. Further, these new components were not destroyed by the absence of a vacuum, allowing easier, faster and cheaper repair of the parts. Funding was secured, and the product development process commenced. The project was not plagued by requirements issues (though many product developments are), as the performance requirements were identical to the product it was replacing, aside from the increase in bandwidth and reliability.

Unfortunately, prototypes were failing to meet the requirements in either bandwidth, power or both. As the program neared the end of its allocated development funding and the pressure on the engineering team increased, each subsequent amplifier prototype was billed as being “full spec,” with multiple changes made in each iteration to meet this promise. Each prototype failed to live up to this billing, and the program was eventually deemed a failure and scheduled for termination.

How did this program devolve to this point? The scientists and engineers involved were brilliant individuals with a history of successful projects and a long list of patents to their names. The requirements were well-understood, and the program had avoided the all-too-common pitfall of neglecting to consider how the piece would fit into the overall system and its effects on upstream and downstream components, overall system power and cooling budgets, etc., which tends to happen in projects run by component designers that focus on the design and capabilities of their particular widget. However, as the program budget and schedule for development was steadily consumed, those involved felt obliged to make each next prototype do everything, rather than taking a deep breath, seeing how far they’d come and how far they had left to go and then creating a road map to get there.

When I was brought in to the program and we had secured some additional development funding, we got down to the business of creating the road map. In addition to the electronics company’s scientists and engineers, we brought in experts from both academia and a federally funded research and development center to review the initial concept and progress to date, determine whether the program had a path forward and, if so, delineate a series of experiments that would culminate in a successful prototype.

Termination of programs is always an option and should not necessarily be viewed as a bad outcome for a program in dire trouble. In evaluating whether it the right move, look beyond the siren call of spent budget, take a stark look at the chances for success and determine if the program or project can still deliver its anticipated benefits. Many programs in trouble attempt to recover by “dumbing down” the requirements; while that may offer an “out” for a development that is in trouble and allow the program to move to the next phase, ask whether continued investment in the system worth the lesser benefits of a product that meets the revised, easier requirements.

After careful review, we determined that the program had a path to success. A series of six prototypes were planned: three experiments to evaluate the performance of individual design changes, a fourth to combine successes from the three experiments, a fifth planned to be production representative/fully specification-compliant and a sixth to show repeatability.

With this new plan in place, the project was able to proceed in a systematic fashion without pressure to produce instant success. Previously, multiple improvement ideas were incorporated into each new prototype with the hope that this would be the full-spec version, and the program could then proceed into production (and the company could reap its financial reward). Experiments were defined up front, and their results were evaluated without the uncertainty of multiple changes masking the successes (or failures) of individual changes.

Success was achieved on the fourth prototype, and the program eventually was successful in creating a significant capability and reliability enhancement for this weapon system. Much of the credit belonged to the hard work and dedication of the engineering leader for the electronics firm (who received its Product Development of the Year award). But even more so, the credit for this dramatic (and very unusual) recovery was earned by the team that had the wherewithal to take a step back and apply basic systems engineering principals to lay out a plan to recover the project from its death spiral.

All too often, companies or program offices bring in a “Cleaner,” but many of these are just tyrants who bully everyone into working long hours, hoping that if you try everything under the sun, something will work. A systems engineer, however, has the composure to take a step back and review, despite the immense schedule pressure. Rather than rushing into the next setback, a systems engineer will look at what the program or project is supposed to accomplish (high-level requirements), review the efforts underway to support those requirements and set a plan with discrete steps and measurable progress points. By not trying to do everything all at once, wherein successes and failures are indistinguishable and possibly masking each other, having a systematic (there’s that word “system” again) approach allows the system engineer to move toward a path that will result in meeting the original requirements.

About the Author/ Jay Schatz

Jay Schatz is senior program manager at Base2 Solutions. He is a high-performing technical project management and program management professional with the breadth of experience that comes from over 22 years of experience managing programs for the aerospace industry and in the US Air Force. Jay is currently providing program management and systems engineering support to the Boeing 787 program, providing expert advice to clients on program management, supplier relations and engineering solutions. Connect with him on LinkedIn.

December 8, 2017
Is Your Company’s IT a Disaster Waiting to Happen?
Recent IT disasters suggest gaps in hardware testing, backup system testing and inadequate disaster recovery plans.

Although the summer often is the most profitable season for most airlines, this past summer wasn’t great for several of the largest carriers. In July, Southwest Airlines suffered a 12-hour IT outage, triggered by a router malfunction that quickly took down many of its core operating systems like a house of cards. Worse, backup systems failed to deploy. Flight delays and cancellations continued for a few days.

A similar disaster unfolded at Delta in August, when a power control module at the airline’s central data center malfunctioned, creating a surge to the transformer and loss of power. Delta’s systems didn’t switch over to backups as planned, causing scheduling mayhem as agents couldn’t access key systems. For both companies, the outages resulted in tens of millions of dollars in lost revenues.

Sure, both of these outages affected businesses running more traditional IT infrastructure, but even if your company is running in the cloud, your business isn’t immune from a damaging outage. In September, Microsoft Azure experienced two outages lasting several hours each, one in Europe and the other affecting U.S. customers across several key services, including SQL Database, Virtual Machines, Service Bus and Visual Studio team services.

As software developers and testers, we often focus narrowly on the application and the server it is running on—but there’s a lot more to consider. Hardware and components, backup systems and third-party providers all come into play when testing applications for stability and reliability. Of course, there’s no way to prevent all bad things from happening but there are some steps you can take to cover the bases more thoroughly.

Look Beyond the Software

If there were three main lessons learned from these recent IT disasters, they would be:
1. Assess risk of the whole environment. The quality of your software is only as good as the components it runs on—the hardware, network, power, storage, and other third-party applications are often required. Understand the full picture and all of the dependencies. Categorize applications based upon business criticality and failure rate to focus attention on the pieces that really matter.
1. Backup systems and processes are not “set and forget.” Spend more time documenting them, testing them and training staff. Make sure that the processes are up to date as IT adds new components and vendors to the system or if the company goes through a merger or acquisition.
1. Invest according to your risk profile. If your company can’t operate for more than a few minutes without access to certain systems, plan on budgeting for warm or hot failover systems which enable near 100 percent uptime with always-on, parallel systems continually synchronized and ready to go. It’s too expensive to enable this failover capability for your entire environment—you will need to be picky. Some IT assets, such as an airline’s scheduling and reservation systems, deserve top priority and resources, while other assets such as the HR system, can do with less. IT directors also need to frequently investigate the need for upgrading old, unstable infrastructure technologies. Most companies also don’t spend enough time monitoring and testing backup and recovery systems.
To minimize the risk of a major IT outage bringing down your business and alienating customers, the recipe is as follows: Analyze the points of failure, determine what is reparable and what is not easily fixed (for example, installing software usually can be done much more quickly than building a new server environment) and, finally, install and test failover systems and processes that can maintain basic operations with minimal customer disruption.

Delta & Southwest: A Quick Analysis

In both cases, Delta and Southwest had backup systems that did not deploy properly—or, at least, in a timely fashion. The first point of action would include conducting thorough, regular testing of those backup systems and locations and upgrading capabilities if necessary. Second, since no disaster recovery system is foolproof, create manual workarounds for critical system outages. That would have helped tremendously so that customers weren’t completely stranded. Third, create a fully redundant second data center that switches over automatically when the primary data center goes down. Naturally, this is the most expensive option, but depending upon the outcome of the risk analysis, it may be worth the money.

With Southwest, a router failure caused a cascading effect that brought down the network altogether. It’s not clear if a secondary network was available for failover, although it appears the answer is no. Again, better, more frequent monitoring could have isolated the router problem and alerted IT to fix or replace the hardware before disaster struck.

Any company with technology from many different vendors should work closely with vendors and partners whenever possible to mitigate possible risks. Vendors often are able to provide insight into how they test their own systems and can even share test cases to assist in validating the end-to-end system. Keep in mind that aging technology is not the only problem—sometimes new technology brings untenable risks. A recent example is the Samsung Galaxy Note 7 and its overheating battery problem and recall. As well, it’s smart to understand how the local utility handles its outages and what if any help it provides to customers in the event of service disruption.

Planning Ahead

Large companies with complex IT infrastructures should take a holistic view of critical system availability, considering all of the possible ripple effects from a single point of failure. Today’s systems and networks are increasingly interconnected and interdependent. For both airlines, the effect on customers and employees lasted far beyond the actual outage, as managers struggled to get agents and planes reorganized and rescheduled. Take that risk into any spending calculations for the disaster recovery plan. Companies that have gone through mergers and acquisitions have additional complexities from IT integrations—or lack thereof. Failover strategies should ideally be incorporated upfront as part of any acquisition plans.

Comprehensive testing strategies are paramount, although sometimes overlooked, in preventing outages. Request information from core vendors, such as the networking equipment provider, on testing processes and ask to see their test cases. Though the vendor should be responsible for ensuring thorough testing of their own products, the customer is always the one to suffer the costs of a downed system. Development and testing teams should get clear on how external systems and providers handle down time and equipment failure. Strive to align the company’s disaster recovery processes with those of partners.

Finally, for companies running mission-critical systems in the cloud, there are ways to mitigate the effects of an outage. Multi-cloud management tools such as CliQr (now owned by Cisco) allow IT to quickly redeploy workloads from one cloud service to another whenever needed. Containers are also helping in this regard due to the inherent portability benefits of the technology. Companies are using containers to move applications and workloads from region to region within one cloud provider; from cloud to cloud, such as from AWS to Azure; and even from the cloud to an on-premise data center.

There are many causes of IT outages today, and as IT environments become more multifaceted, supporting more technologies, data and applications all the time, those risks grow. We don’t have an answer to the problem, but there is one strategy: leave no stone unturned. Involving testers and developers in outage prevention and disaster recovery planning is a wise move, indeed.

About the Author / Kevin Dunne

Kevin Dunne is vice president of strategy and business development at QASymphony. Prior to joining QASymphony, Dunne was a business technology analyst at Deloitte. He can be found online at Twitter and LinkedIn.
September 29, 2016
Continuous Testing System Stability
The effort to speed up continuous testing will be wasted if the underlying system that continuous testing depends on is not stable. In my prior blog
Continuous Test Results Analysis – at the Speed of DevOps, I discussed the importance and practices for speeding up continuous testing results analysis. However, unless the continuous test system that the test results are derived from is rock solid, the results will at best be unreliable and at worst – results may not be generated at all!

One of the first rules of DevOps is to “keep going and don’t stop”. Stopping is the equivalent of pushing the red button on a factory assembly line effectively causing the entire line to shut down. Then, panic ensues because every second that the line is down affects all of production leading to increasing costs not to mention restarting the line is substantial also.

Many techniques are employed to minimize the need to stop testing despite reported product test failures. However if the underlying continuous testing system itself is not stable, then the entire DevOps system including continuous testing will immediately be interrupted until the system recovers. Development team members are looking for results, so the time savings for speeding up the availability and the total work associated with results analysis is huge for every minute of results analysis time saved on each CI/CT cycle. When aggregated over multiple code branches, a few minutes saved per test result can be man-years of time saved over a year!

So what can be done to ensure the continuous test system itself is stable and produces reliable results even in the face of inevitable occasional system failures? After all, there is no such thing as a truly perpetual machine that never ever breaks down. Below are some suggestions in a checklist format that have proven useful in high performance DevOps deployments.
1. Build a fast-response multi-discipline team: There are a wide variety of ways any system of interconnected computers and applications can fail whether they are virtualized, internal, or running on a cloud used remotely from a data center in the cloud. The following list of suggestions for the team will assure fast recovery when the system does experience failure.
  • Assign a continuous testing system architect with responsibility to design system stability SLAs, design high-reliability system architectures and define sufficient team workflows suitable for disaster prevention and recovery consistent with recovery time SLAs.
  • While the team members may be in-or-out sourced, full or part-time, there needs to be clearly responsible first responders and 2nd line escalation team members with definitive workflows that take care of failure events 24/7 for every hardware, software, tools, network and firewall component that the system depends on.
  • Assign product team leads to plan changes needed for new products and services.
  • 24/7 availability system designs are pricey to build, expand, and maintain, so finance and accounting team members need to be part of the team to keep clear, current cost data relevant to the continuous testing system.
  • A senior budget manager approves expenses needed for stability without delay.
2. Anticipate failures – prepare, prepare, prepare!: Don’t wait for a failure before testing system recovery procedures! Documented disaster prevention and recovery plans or high-availability tools won’t save the continuous test environment if the team is not ready to react during a failure event.
  • Run failure simulations for each defined failure scenario at a frequency that matches the expected probability of each failure.
  • Cross-train team members on system recovery best practices so the skills will be ready when most needed – ensure no one person has essential unique recovery skills.
3. Choose test tools designed with high stability system administration capabilities: Look for the following features designed into in test tools to ensure the continuous test environment can be configured and managed for high reliability.
  • Tools are implemented as autonomous services, controlled through RESTful APIs, that can be separately invoked and managed
  • Test system service monitoring is built-in including up-time, wait-time and usage metrics for each use case of each test system service.
  • Diagnostic capabilities are built-in to each service. When a failure occurs the failure log includes helpful descriptions including causal information.
4. Engineer the system for stability: Typical continuous testing systems have many “moving parts” and many of them have to work in concert for the system to perform properly. It is common engineering knowledge that the reliability of a system is determined by the reliability and configuration of its parts. Here are some suggestions to improve the overall system reliability.
  • Choose powerful servers and virtual machines (VMs) with ample processor speed, fast I/O, large and fast memory and data storage. This is important to minimize the chance of system failures due to possible overloading of these resources.
  • Deploy multiple servers and VMs for each system component with enough redundancy so that no one component failure will interrupt the entire continuous testing system.
5. Configure the test workflow to avoid false negatives: There is no point in reporting product test failures to users if the reason for the failure was an underlying continuous testing system failure instead of a failure of the product itself. Such “false negatives” are a big waste of time. Here are some suggestions to reduce the chance of false-negative test reports.
  • Assign an application to continuously verify, in-real time, that all the system components and interconnections are operating and report these results to the continuous testing report aggregation tool. These results are called CT system environment test results.
  • Use the CT system environment test results within the continuous test results analysis tool to determine if any test failure coincides with an environment failure. Mark each verdict “inconclusive” instead of “fail” if it does and include a description of the environment failure in the verdict log.
6. Maintain the system to assure long-term stability: Every system needs maintenance at some point. Hardware components wear out and software components become obsolete. Each time a system component is replaced or upgraded to new versions there is a risk of instability. Here are some recommendations to reduce the risk of introducing instabilities to the continuous testing system when changes are implemented.
  • Create a maintenance environment that mirrors the test and production environments.
  • Define a test suite that will exercise all system components and topologies.
  • Test changes in the maintenance environment before deployment.
  • Occasionally run failure/recovery simulations in each production environment.
The above is a partial list of suggestions for continuous test system stability that have been proven to yield good results for DevOps. At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

What do you think of these suggestions and do you have others that should be mentioned?
April 1, 2015
The Internet of APIs
The Internet moves at the speed of light. Not only that, but the terminology used when describing The Internet seems to change with the tides. One of the newest phrases to start making the rounds is “The Internet of APIs.” In an attempt to understand exactly what it is, I went looking for a definition and was unable to find anything at all. As such, it’s time to put a definition out there.

There are two important words in the phrase: “Internet” and “API”. Let’s start with the last word first and work backwards. What exactly is an API? Luckily, the definition of API is pretty settled. For our purposes, I like the definition given to us by API Evangelist:

An API, or Application Programming Interface, at its most fundamental level, is how software applications talk to each other using the Internet. An API allows you to open up data and other digital resources, to public developers, businesses, or even between departments and locations within a company. APIs are increasingly the way in which companies exchange data, services and complex resources, both internally, externally with partners, and openly with the public.

And while we’re at it, let’s look at the popular definition of “Internet” given to us by Webopedia:

The Internet is a global network connecting millions of computers. More than 100 countries are linked into exchanges of data, news and opinions.

So we’re narrowing down on a definition of The Internet of APIs. But we’re not quite there yet. Let’s look back at recent history to see if we can clarify things a little bit. The modern-day web API has been around since Salesforce introduced it’s API at the turn of the century. In the year 2000, they introduced an XML-based API that kicked off the web services API era. And while there were a few other web services APIs that came out afterwards (including eBay and AWS); but things didn’t really start getting serious until Flickr launched their API in 2004. The social network revolution, sparked by companies like Friendster, MySpace and Twitter, brought about a real need to securely exchange information between multiple applications. So when Facebook and Twitter announced their own APIs in 2006, that was when the usage of APIs grew to include identity management in addition to the exchange of data it had become known for.

And then came smartphones. APIs exploded in popularity once the smartphone industry took over the continent thanks to Palm and Blackberry; then came iOS and Android. Why the popularity explosion? Well, applications needed to be made for these new (popular) form factors. However, these new form factors had very specific requirements around data usage, drastically different screen sizes than application developers were traditionally used to, along with previously unseen device capabilities. Add to that the fact that mobile applications needed to figure out a way to share data with each other securely and a perfect storm of needs presented itself.

While all of this was going on, the wireless capabilities available to the consumer also changed. In the year 1999, one year before Salesforce released their web services API, 802.11a and 802.11b hit the market. WiFi was still a growing technology, but the introduction of 802.11a and 802.11b changed everything. The prior iteration of 802.11 had a transfer rate of 2 Mbps; while a allowed for short range speed of 54 Mbps and b allowed for longer range speed of 11 Mbps. The cost of b was also a lot lower than a.

Also in 1999, the first full Internet service available on a phone was launched in Japan by NTT Docomo. This was during the days of 2G networks. In the US, around the same time, cell phones became affordable to everyday consumers. It was a time of incredible growth in the wireless industry and over the following years, cellular networks were stretched to their limits.

This is the environment that led to the growth of the web service API. There were a few things that were extremely important to the upcoming boom:
1. A speed boost in wireless networking was required; and
2. Wireless networking hardware needed to get into the hands of more people; and
3. The cost of wireless networking hardware needed to drop; and
4. Cell phones needed to get in the hands of more consumers; and
5. Internet service needed to hit the cell phone.
Add all of that to the growth of social networking and identity management via API, and a perfect storm of circumstances presented themselves. More and more APIs were built utilizing a person’s identity for personalization and security.

Thus the Internet of APIs was born.

So, based off of the existing definitions as well as recent history, what exactly is “The Internet of APIs”? The Internet of APIs is:

A global network of structured data, digital resources and commands accessible via a growing interface that allows applications and devices to exchange information using any and all communications protocols available.

Since the creation of the ARPANET (arguably, the Internet’s predecessor) packet-switching networks have been utilized as an efficient and reliable way to exchange data. So what’s different about the Internet of APIs since at its core it’s about exchanging data? The main difference lies in the fact that the data provided by the Internet of APIs is data with structure. It’s data that is being shifted around and meant to be consumed in ways that may not possibly have even been foreseen by the creator(s) of the data. The Internet of APIs allows data that may have been designed for one thing to be used for something completely different; thus facilitating the connection of two completely different applications in ways sometimes unimagined.

The Internet of APIs is the underlying foundation to the Internet of Things. Without the capability for connected devices to personalize their usage according to the individual, as well as share data, the Internet of Things could not exist. Without the improvements in specifications in both wireless and cellular hardware, the Internet of Things could not exist.

The Internet of Things is powered by the Internet of APIs; and as the Internet of APIs grows in scope and capability, the same will happen to the Internet of Things. Without regularly updated data and personalization, a hardware device is dumb. A thermostat is just a thermostat without the underlying API. But with the API, a thermostat becomes a Nest. Without an API, a light bulb is just a light bulb. But with the API, a light bulb becomes a Philips Hue light bulb. Without the Internet of APIs, a person walks in to their home, walks in to their living room, flicks a light switch and turns their heat up. With the Internet of APIs, a person walks in to a warm, well lit home because their thermostat turned on five minutes before they got home and their lights turned on one minute before they walked through the door.

Data and personalization: that is the power of the Internet of APIs.

About the Author/Kurt Collins

Kurt Collins is Director of Evangelism at built.io (link). Before that, he started his engineering career as an intern at Silicon Graphics (SGI) where he worked on debugging different parts of their UNIX-flavored operating system. Two years ago, Kurt co-founded The Hidden Genius Project along with eight other individuals after they all saw the screaming need to increase the amount of Black men in the technology industry. Mentoring young people interested in the technology industry is a personal passion of his. Today, he continues his work with tech startups and non-profits alike.
February 9, 2015