devopsy.kinsta.cloud

Tag: software bugs

  • The Bug in Production: What You Don’t Know Can – and Will – Harm You

    The Bug in Production: What You Don’t Know Can – and Will – Harm You

    Despite the risk of unplanned downtime, many organizations that develop software and services push them live without adequately testing for bugs that will manifest against production traffic. This is a huge gamble: Those bugs could lead to errors that bring down the service altogether. Pushing live without sufficient testing can be detrimental to your business.

    The Problem of Going Live With Bugs in the Code

    Software downtime leads to loss of revenue and of reputation. In fact, Gartner analysts have estimated that the average cost of downtime is $5,600 a minute—that’s well over $300,000 an hour. To provide a real-life example of what this looks like, Microsoft Azure suffered a major outage in November 2018 caused by issues introduced as part of a code update, lasting for 14 hours and affecting customers throughout Europe and beyond. With migration from legacy systems to micro-environments in the cloud, outages and downtime pose a growing and serious problem.

    As companies switch to DevOps and CI/CD models to move faster and provide application updates sooner, software developers continually release new features and often push code updates as fast as they’re written. The classic six-month development timelines of dev, quality assurance (QA) and beta testing have been compressed to days and sometimes hours. Gone is the time when teams could beta test with customers for extended periods to flag real-time bugs.

    With current quality testing tools, developers don’t know how a new software version will perform in production—or whether it will even work in production. The Cloudbleed bug is an example of this problem—in February 2017, a simple coding error in a software upgrade from security vendor Cloudflare led to a serious vulnerability discovered by a Google researcher several months later.

    Flaws can lead to serious security issues later, in addition to having the immediate impacts mentioned above. Heartbleed, a vulnerability that arose in 2014 and stemmed from a programming mistake in the OpenSSL library, left large numbers of private keys and sensitive information exposed to the internet, enabling theft which otherwise would have been protected by SSL/TLS encryption.

    Standard QA Testing Isn’t Enough: Test With Production Traffic

    The way QA testing is typically done is no longer sufficient for today’s increasingly frequent and fast development cycles. Traditionally, DevOps teams haven’t been able to do side-by-side testing of the production version and an upgrade candidate.  The QA testing used by many organizations is a set of simulated test suites, which may not give comprehensive insight into the myriad ways in which customers may actually make use of the software. Just because upgraded code works under one set of testing parameters, doesn’t mean it will work in the unpredictable world of production usage.

    As in the case of the Cloudflare incident, the error went entirely unnoticed by end users for an extended period of time and there were no system errors logged as a result of the flaw. Just as QA testing isn’t sufficient, relying on system logs and users also has a limited scope for what can be detected.

    It is estimated that fixing flaws after a software release can be five times as expensive as fixing them during design—and it can lead to even costlier development delays. Enabling software teams to identify potential bugs and security concerns prior to release can alleviate those delays. Clearly, testing with production traffic earlier in the code development process can save time, money and pain. Software and DevOps teams need a way to test quickly and accurately how new releases will perform with real (not simulated) customer traffic and while maintaining the highest standards.

    By evaluating release versions side by side, teams can quickly locate any differences or defects. In addition, they can gain real insight on network performance while also verifying the stability of upgrades and patches in a working environment. Doing this efficiently will significantly reduce the likelihood of releasing software that later needs to be rolled back. Rollbacks are expensive, as we saw in the case of the Microsoft Azure incident.

    Some organizations stage rollouts, which requires running multiple software versions in production. The software teams put a small percentage of users on the new version, while most users run the status quo. Unfortunately, this approach to testing with production traffic is cumbersome to manage and costly, and still vulnerable to rollbacks. The other problem with these kinds of rolling deployments is that while failures can be caught early in the process, they are—by design—only caught after they’ve affected end users.

    This brings more questions, including: How do you know whether the new software is causing the failures? and, How many failures does the business allow before recalling or rolling back the software, since the business does not observe side-by-side results from the same customer? This disrupts the end user experience, which ultimately affects business operations and company reputation. And staging may not provide a sufficient sample to gauge the efficacy of the new release versus the entire population of customers.

    Cost is still an issue as well. If you stage with 10% of customers on the new version and a failure costs more than $300,000 an hour, then a failure affecting 10% of users could potentially still cost more than $30,000 per hour. The impact is reduced, of course, but it’s still significant—not counting the uncertainty of when to rollback.

    Looking Ahead

    Standard QA testing is no longer enough. To reduce the risk injected into the software development life cycle by today’s rapid iterations, DevOps teams can test in production and evaluate release versions side-by-side. This will help prevent costly rollbacks or staging while still releasing a quality, secure product. The old way of doing things is not sufficient, but fortunately, there is a better way.

    — Frank Huerta

  • Network Resilience and Security from A to Z

    Network Resilience and Security from A to Z

    An observer watching a bunker shot by legendary pro golfer Gary Player was heard to say: “I’ve never seen anyone so lucky in my life.” The player retorted: “Yes, and the more I practice, the luckier I get.” Yet, when it comes to cybersecurity, nearly half of organizations are solely relying on luck to get them through a cyberattack.

    There is not enough practice or training in terms of incident response, and testing happens haphazardly depending on the developer and organization—obfuscating baselines and the context necessary to ensure security and resilience. Particularly in terms of testing, we’ve noticed that despite awareness of its importance, bugs and vulnerabilities routinely slip through. In fact, a recent Ixia survey found that 34 percent of developers have deployed products that have had a few bugs. Worse, 31 percent said products harbored significant vulnerabilities that required patching later in the cycle when shipped.

    The problem has been exacerbated by the rapidly growing normalization of agile development processes. Groups of developers are tasked to build products piecemeal, leading to application development that is often incremental and happens in iterative cadences. Testing and oversight happen at each step of the process, but the segmented nature of the development cycle means that bugs and vulnerabilities often arise when the code is assembled, and are routinely missed. For instance, the average IT web application had 32 vulnerabilities, according to recent WhiteHat Security report. It’s clear that more than a local test—a comprehensive end-to-end test—and relevant training are critical.

    A Change in Culture

    Improvement begins by changing the culture that minimizes security testing for the sake of launch timelines. Too often, the product development team will come together just to be told they need to move up their release date. The habit results in products that walk a thin line of performance, as they may contain unknown security holes due to not being fully tested. Just consider the slew of IoT devices and services that have been proven to be vulnerable because of this habit—from cars to coffee machines to cameras. It’s a major reason for security incidents, even with multiple layers of security tools.

    This also requires the right training. Having seemingly secure code and the right security measures is not enough for a strong security stance if proper training is not implemented. Organizations need to learn how to respond. Yet, recent SANS Institute research into the incident response capabilities of companies worldwide found that 43 percent of respondents did not have a formalized incident response plan, and 55 percent didn’t even have an incident response team.

    This less-than-vigilant approach to system-level security can have worse implications once a product goes live in the network. For instance, updates, patches and configuration changes applied when a new feature is added can reset an organization’s security posture, and this sometimes goes unnoticed—leaving gaps for threat actors to exploit. Or sometimes a machine fails to update and becomes and entry point, as experienced by JPMorgan Chase. Continuous testing and training from the very onset of operation and throughout a tools lifespan are imperative for all employees, especially IT and security pros—even in its simplest iterations.

    What to Look For

    Successful and continuous testing and training expose security and IT pros to anomalous activity they’d not be able to recognize otherwise. It’s necessary to be able to answer questions such as, “What does suspicious activity look like on my network?” or “What security alerts require immediate action?” The more they see, the more refined their eye becomes—it’s all about a proactive approach paired with repetition.

    Which raises the question, what is considered a good test? It starts with accurately reflecting the widest range of attack types in live operation. It is also important to create an accurate sandbox to test in so that it can be as close to reality as possible. The more accurate the environment, the better prepared IT and security teams will be. Realistic depictions better prepare employees as new malware, phishing and DDoS attack types emerge every day—not to mention it helps keep up with evolving typical application behavior.

    More specifically, function and system testing should be at the heart of any program following the development cycle test process. Quality of service, performance and resilience all benefit, along with security, as a result.

    Some companies take shortcuts, such as using internally generated attacks or crowdsourced probes to attack their networks. Just as bad, some have the development team create and run their own test scenarios. While this can give the illusion that everything is good, it creates a false sense of security—a single scenario only protects you from one type of attack.

    The key is not to stop at the minimum when it comes to security testing and training. Developers spend lots of time building great features; why not spend time training teams on how to use them? Limited or biased tests can easily overlook glaring flaws.

    Ultimately, the right approach to testing can make networks robust and prepare IT teams for real-world attack situations, helping minimize response times and the negative impact on the business. As breaches now seem to be coming from every direction, testing needs to come to the foreground, from the development stages to when products sit directly in the line of fire.

    About the Author / Jeff Harris

    jeffharrisJeff Harris is VP, Solutions at Ixia, leading solutions marketing for Ixia’s security portfolio of products and capabilities. As a former product development leader of advanced networking, communications, and surveillance products for commercial and military applications, Jeff has a deep appreciation for security implications that occur in development and operation. Jeff has led first to market product teams in personal area networks, mobile ad hoc networks, and a wide range of surveillance equipment and platforms. Connect with him on LinkedIn and Twitter.