devopsy.kinsta.cloud

Tag: continuous testing

Shift Left Without Fear: The Role of Security in Enabling DevOps
It’s the age of DevOps, and we all want to shift left, which refers to the idea of performing processes earlier in the CI/CD cycle. That includes security checks and audits. By starting security operations earlier in the delivery pipeline, it becomes easier to find problems, and teams have more time to address them before pushing code into production.

The challenge, of course, is building a shift-left security operation that allows you to perform security operations early without slowing down delivery speed. Let’s take a look at how to square that circle.

Defining Shift-Left Security

A popular term in DevOps culture is “shift left.” But what does shift left mean? Shifting left refers to the effort by a DevOps team to implement measures to ensure application quality at an early point in the software development life cycle. In the case of application security, this means implementing measures to ensure security concerns are taken into consideration while the application is being developed, rather than at the end of the process.

The overall goal of shift-left security is to identify and resolve potential vulnerabilities earlier in the development life cycle, when these issues are less expensive to fix. Doing so will help to mitigate the risk potential security concerns pose to both the delivery schedule and your end -users.

DevSecOps: Integrating Security Into the Development Process

Let’s be honest: Faster and more frequent development cycles aren’t going away. Slowing delivery speed to analyze and bulletproof the application at the end of the development process is no longer a viable format for implementing security. DevOps and CI/CD need to be embraced and enabled by the security staff.

For security teams to enable the DevOps process, collaboration is key. Through this collaboration, the process of DevOps transforms into something known as DevSecOps. In DevSecOps, security personnel work closely with the rest of the DevOps organization to help those who create and deliver the application to gain a better understanding of what is needed to address security concerns as early in the development life cycle as possible.

The following tips serve as a jumping-off point for implementing this collaboration:
- Make security the responsibility of the entire DevOps team: Impress upon the development and operations staff that security is now the responsibility of the entire organization. Simply addressing potential vulnerabilities post-development will eat up too much time and increase the risk that something will be missed. Buying into this process is an essential part of implementing successful collaboration between the security team and the rest of the organization. An example of this would be impressing upon developers to code with security in mind.
- Facilitate regular discussions about application security throughout the development process: It is the responsibility of the security folks to keep security considerations at the forefront of a developer’s mind while they develop new features. Use the agile process to your advantage. Backlog meetings can involve security folks to ensure the creation of user stories includes consideration for security concerns. Your daily standup meetings are another place to include security personnel. This will provide security staff with an opportunity to speak up when the work described by the developer requires—or might require—considerations related to security that have yet to be addressed. Doing so will help to ensure the ball is not dropped, allowing for potential issues to be rectified before they even exist.
- Use test automation and continuous integration to your advantage: When it comes to addressing security concerns throughout the development process, continuous testing is your best friend. And, in cases where it makes sense, the DevOps team should integrate automated test scripts for the detection of security and into their continuous integration process. Such tests can include developer-written scripts to test for vulnerabilities in the code base and tools for static code analysis. Another part of your continuous testing strategy could involve the use of tools for infrastructure monitoring.
The data produced by these strategies can then be analyzed by security folks. And from this analysis, recommendations can be made to the DevOps team regarding the path forward to close potential gaps in security before they ever reach production and can be exploited.

Understanding the Role of Security Personnel in the Development Process

Security folks are often not coders, and development personnel are often not security experts. And while it may be frustrating at times, the solution to managing this disconnect is in the collaboration described above. Situations will arise where developers need to develop with various technologies to build a quality product. It’s possible that these technologies will lend themselves to security concerns that require security expertise. The job of security personnel shouldn’t always be to fight back against the use of these technologies. The security team should act as facilitators—take what development and operations staff intend to do and foresee the potential impacts that innovation will have on security—then take these potential impacts and provide insight into what needs to be done to reduce the risks.

Conclusion

In the past, security often was considered as an afterthought in the development of an application and its infrastructure. The movement toward faster development cycles and more frequent application releases has forced development organizations to think about security differently. Nowadays, so much can change in an application in a short time. This has forced DevOps organizations to integrate security into the development pipeline to make security a continuous consideration in the development life cycle. With this integration, the responsibilities of security personnel have shifted. Security teams now need to be enablers of the DevOps culture, empowering DevOps processes with security in mind.

To see how Symantec CWA can help you shift left without fear, sign up for a free trial to see how it can help secure your cloud.

— Scott Fitzpatrick
June 11, 2019
How to Create Bug-Free Blockchain Apps
While all developers strive for bug-free code, it’s particularly crucial in a blockchain deployment where sensitive data or other confidential info is being exchanged, such as in health care or finance. However, some businesses have learned that lesson the hard way. Cryptocurrency exchange Binance recently revealed a devastating security breach that resulted in a loss of more than $40 million. While this is a large sum, a security breach could be even more costly for businesses that lose private messages, confidential contracts and more.

Rebuilding (and regaining customer trust) is not where organizations should be spending their time. A better way forward is for company leadership to mandate a blockchain test automation strategy from the very beginning.

While most software testers are familiar with testing web applications, they may not be sure how to approach blockchain testing. The good news is, some of the same focus areas of web testing apply to blockchain testing, too (functional, performance and security). However, blockchain involves a greater number of tests, effort and focus areas, such as infrastructure orchestration, staging and simulating scenarios in a distributed environment, which requires an even greater level of expertise.

Because of these additions, fixing a bug in a blockchain app requires several extra steps and, as a result, it is more time-consuming and costly than fixing bugs in other apps. For example, to reproduce bugs or verify code, developers should perform these steps multiple times:
1. Form a new endpoint.
2. Deploy new code to the new endpoint.
3. Migrate the current data to the new back end.
4. Suspend the old back end.
5. Update all front ends to the current version.
After that, verifying the bug-fix performed by testers or continuous integration (CI) tools will require these steps again. This complexity makes it difficult to release bug-free blockchain applications, yet doing so puts organizations and their customers at risk of financial harm. Due to the high-risk nature of deploying blockchain applications, it’s critical that testing is included in the managerial vision and strategy.

To release blockchain apps with bug-free code, businesses should follow these guidelines:

Integrate Test Automation Into Your Blockchain Testing Strategy

Test automation can relieve testers from performing thousands of orchestrating tests manually, which is both difficult and inefficient. A good system architecture should be scalable, maintainable and testable, so teams can test components independently along with mocked components. If a test requires a different environment or infrastructure, infrastructure as code (IaC, such as Puppet, Chef, Ansible and AWS is invaluable in setting up appropriate test environments. Another crucial part of the strategy is to provide the infrastructure with declarative definitions and allow developers to implement codes that support test automation.

Run Unit and Integration Tests to Cover Both Back End and Front End Code

Blockchain applications contain at least a back end (a set of smart contracts or chain codes running on a blockchain) and a front end (where users can interact with data stored in the blockchain). Performing unit tests will cover both back end and front end code, and, because of its critical role to blockchain apps, its primary focus should be on the chain codes. Integration testing, however, should cover the various types of oracles and front end applications. Performing additional tests will eliminate any holes that can threaten the back end by verifying the integration between the UI and chain codes.

Perform Tests in a Local Infrastructure

Though public blockchains (such as Bitcoin or Ethereum) have several test-nets, it’s optimal to perform the tests in a local infrastructure. Local infrastructures are lightweight, faster and more stable for unit and integration tests. For example, in a Ganache local infrastructure, it’s possible to run more than 1,000 unit tests on five Ethereum smart contracts with approximately 3,000 deployments in an hour or less.

Continuously Update Security System Test Suites

Blockchain is a multi-party infrastructure and in some cases, it is a public network. As a result, these applications will be visible to many stakeholders and may even be open-sourced to the world. This level of transparency means that hackers are able to study the programs and plan attacks based on a particular system. Continuously updating security system test suites will ensure applications are protected against all common security vulnerabilities.

Minimize Continuous Testing Pipelines for Each Phase of the Development Process

Layering integration and unit tests in a local infrastructure with continuous updates to the security system test suites creates a continuous testing pipeline that is essential to high-quality blockchain code. Continuously verifying code changes in multiple environments, such as local blockchain and test-net, ensures high-quality code is delivered to the production blockchain. These tests, however, can last hours or days for each code change, so it’s important for teams to minimize and create lean pipelines that focus on test stages, test types and test environments, and develop multiple pipelines for code at each phase of the development process.

Build Customized Tests Suites to Eliminate Prospective Cheating Holes

Test teams also need to customize test suites in the pipelines to cover both functional and security testing. Covering security cases is important, but unit test modules should also exercise smart-contract-unit potential security threats, such as overflow and re-entrancy, to eliminate as many prospective cheating holes as possible.

Due to the nature of decentralization and orchestration of underlying infrastructure, developing bug-free code for blockchain applications presents many more challenges than other applications. By taking best practices from traditional software testing, especially test automation, and customizing them for blockchain development, businesses can achieve bug-free blockchain coding and protect their most vulnerable applications.

— Thong Nguyen
May 31, 2019
Tricentis Disrupts RPA Market with Resilient Model-Based Automation
Model-based automation technology trusted by 1600+ enterprise organizations now drives a dedicated Tricentis RPA offering

Mountain View Calif., May 22, 2019 – Tricentis today announced the availability of Tricentis RPA, which is differentiated by its award-winning model-based automation technology and radically simplified bot pricing model. Tricentis’ model-based automation technology has over 12 years of research and development behind it and is currently leveraged by 1600+ enterprise organizations. It is best known for delivering highly-resilient end-to-end automation of complex business processes—supporting over 150 enterprise technologies, from legacy applications to the most modern SaaS-based apps.

“To satisfy rapidly-changing customer expectations, organizations must be able to change existing processes and applications with both agility and confidence,” explained Tricentis Founder and Chief Strategy Officer Wolfgang Platz. “They need to know that the automation will work—that it’s resilient, with the ability to recover from frequent application changes without requiring technical resources. With this degree of resiliency, organizations are able to scale automation at a much faster rate. This equates to a greater number of unattended bots.”

Tricentis is known for enabling business users to achieve unprecedented test automation rates across highly-complex and rapidly-evolving processes. The core technology that drives these high automation rates is now being featured in a broader platform that directly addresses the unique requirements of RPA, enabling organizations to achieve equally high levels of process automation across the enterprise.

Tricentis CEO Sandeep Johri noted, “Tricentis is the first vendor to provide a purpose-built enterprise RPA solution as well as a comprehensive continuous test automation platform. Organizations looking to boost operational efficiency in terms of software testing have been trusting Tricentis for years. Now, they’re also applying our industry-leading automation to many other critical business processes in the production environment. This is a strategic business benefit which accelerates enterprises on their end- to-end automation journey.”

Tricentis RPA Product Details

Tricentis RPA consists of Tricentis RPA Studio and Tricentis Orchestrator. RPA Studio is a workspace to design, implement, and test bots. RPA Orchestrator enables users to run bots on-demand, distribute bots to agents, monitor and maintain agents, and generate reports as well as audit logs. Both are designed specifically for business professionals, and do not require any scripting or coding. Tricentis RPA supports a wide array of technologies including SAP, Salesforce.com, Oracle, ServiceNow, and Pega as well as core banking applications like Finacle, Temenos, and Avaloq.

Tricentis is the first vendor to offer highly-simplified pricing for RPA. Organizations only pay for running bots. This low cost to entry helps organizations achieve operational efficiency without the barriers associated with cumbersome licensing.

Tricentis RPA Studio and Tricentis RPA Orchestrator are offered for free. Customers can download the solution on-demand and immediately start creating bots for their RPA projects without limitations or cost. Tricentis charges for bot execution on a subscription basis.
About Tricentis

With the industry’s #1 Continuous Testing platform, Tricentis is recognized for reinventing software testing for DevOps. Through agile test management and advanced test automation optimized to support 150+ technologies, we provide automated insight into the business risks of your software releases—transforming testing from a roadblock to a catalyst for innovation. The result is accelerated software delivery speed, improved cost efficiency, and reduced business risk.

Tricentis is the only vendor to achieve “leader” status in all three top analyst reports (i.e., the “Triple Crown.”) This honor is based on our technical leadership, innovation, and a Global 2000 customer base of 1600+ companies, including global enterprises such as Allianz, ANZ Bank, Cisco, Dolby, Experian, First Data, HSBC, Merck, Office Depot, Samsung, Swiss Re, Starbucks, Telstra, UBS, Vodafone, Whole Foods, and WorldPay. Customers rely on Tricentis to achieve and sustain test automation rates of over 90 percent—increasing risk coverage while accelerating testing to keep pace with Agile and DevOps.

Tricentis has a global presence in Austria, Australia, Belgium, Denmark, Germany, India, Netherlands, Singapore, Switzerland, Poland, United States and the UK. To learn more, visit https://www.tricentis.com or follow us on LinkedIn, Twitter, and Facebook.
May 29, 2019
The Essential Toolbox for DevOps Analytics

With DevOps teams under constant pressure to improve software delivery cycles while maintaining—and improving—quality, many have turned to continuous testing (CT), implementing it as part of their overall development strategy. CT offers a number of benefits to DevOps teams, but it also introduces new challenges, specifically infinite amounts of data that require analysis. As more teams integrate CT into their strategy, it is clear that analyzing, understanding and filtering the results data quickly is critical to preventing bottlenecks in the DevOps process. The problem is, teams simply don’t have the time needed to analyze data.

Current research indicates that teams spend anywhere between 50 and 72 hours per regression cycle analyzing test results. This includes filtering out noise and assessing failures that may impact their software releases. However, the amount of time DevOps teams have to review and qualify test results has shrunk dramatically in recent years—from days to hours or even minutes, in some cases—leaving teams scrambling.

With pressure to accelerate velocity and deliver consistent, high-quality experiences, DevOps team require new analytics solutions to help them manage large test-results datasets in a CI/DevOps environment.

Let’s explore five essential tools that enable DevOps teams to quickly and efficiently analyze data, triage issues and act upon failures with the best possible insights.

Executive Dashboards

Dashboards have evolved immensely over the years, allowing dev managers and QA managers to easily examine the pipeline, see CI trends related to time, build health and more. When looking for the best solution, make sure it includes quality heat maps and CI dashboards, which make it easy to spot an anomaly in the CI pipeline and allow managers, DevOps teams and practitioners to quickly drill down into the single test report and the issue.

Single Test Report Visibility With Advanced Reporting Artifacts

With a single test report, practitioners have access to the entire flow of test steps, video, logs, screenshots and Jira–even the ability to drill down into the source code. This makes it quick and easy to locate an issue. For teams, these means more time fixing and less time searching.

Cross-Platform Reports

These analytics reports show the UI/UX simultaneously across multiple screen sizes, resolutions and platforms. Since the boundaries between the digital platforms need to be seamless, whether on desktop browsers or mobile devices, having the ability to see the UI/UX on multiple form factors (and layouts) at the same time is essential to assess overall quality.

DevOps Noise Reduction Tools Powered by AI and Analytics

Today, finding which error classifications are true application bugs is critical to efficiency. Noise reduction tools that are powered by AI allow teams to filter out problems caused by various issues that are not software defects—for example, device connectivity issues and wrong scripting. Having a tool with the ability to locate each failed test execution, the root cause and the classification category is a huge productivity boost for both the test engineer and the developer.

DevOps and Actionable Insights

Analyzing data is one thing, knowing what to do it with it is another. Today, there are a number of analytics solutions on the market that provide teams with both capabilities. Once an issue is detected, classified and reported, having deep insight brings practitioners halfway toward the resolution of the issue.

To succeed in software delivery, teams not only need better automation as part of their processes, test flows and CI/CD workflows, but also an analytics platform to manage all of their test results data. Since DevOps involves a collection of team members from all parts of the software delivery lifecycle (SDLC) process, the central platform needs to meet the needs of all team members. As you work to build your next test analysis toolbox, consider the five features above to efficiently evaluate the data, act upon it and deliver iterations and features with confidence.

— Eran Kinsbruner

May 23, 2019
5 DevOps Trends Worth Knowing in 2019
DevOps today is more than the buzzword it was 10 years ago. Back then, the idea of combining development with production in your IT infrastructure was not common; the logistics were nonexistent and most of the time businesses decided it was far too much effort.

However, if the last nine years has taught us anything, it’s that DevOps is worth every penny businesses invest. It helps companies streamline production, keep customers happy and provide agile services to attract new audiences.

But as with anything in the IT field, there is a paradigm of change. This article discusses some of the top DevOps trends happening now and how to implement them in your own system.

Automation Will Be a Bigger Focus

Automation has always been the way for businesses. Whether it’s speeding up the pipeline with machines, providing better response times for angry customers with chatbots or better inventory management with tracking and logging system, automation is behind it all.

DevOps is no exception.

But as with everything, there can be too much automation. The best thing to do is break down processes and find simple, redundant tasks that can save time if automated and apply the changes.

The most important aspect of automating DevOps is to understand the six pillars in the DevOps cycle, which are:
- Continuous business planning
- Collaborative development
- Continuous testing
- Continuous release and deployment
- Continuous monitoring
- Collaborative customer feedback and optimization
DevOps Assembly Lines: The Next Big Thing

To track the development of your application, we have pipelines. These visually represent the product from its initial stage right through until final development and release.

However, continuous integration (CI) is no longer the priority focus when it comes to effective monitoring and implementation. Continuous delivery and automation are a major part that businesses have already started delving into in 2019.

This is where DevOps assembly lines enter the picture.

Assembly lines are focused on bridging the gap between manual and automated tasks, ensuring there is no redundancy while providing several tools for teams to implement, to ensure apps are hitting the production lines as reasonably quickly as possible.

Microservices Architecture Adoption Taking Off

Microservices architecture may as well be that new kid in school trying to fit in. These smaller services compound into agile, implementable tools that DevOps teams can use to scale and test software over the course of weeks—not years.

As you can imagine, this can help some businesses streamline their development and production lines, setting them way ahead of their original production estimates—which is never a bad thing.

Implementing microservices into your DevOps team can also help companies focus on the runtime and efficient delivery of new development sources, without the hassle of creating dependency errors when something decides to bug out for no apparent reason.

Security Needs to Step Up

Cybercrime is on the rise—we all know that’s old news. But as a business, cybercrime can either make or break your sales and weigh down your DevOps team.

That’s why more companies are hopping onto enterprise security options. Although there are third-party programmers for almost every situation you need, none of them have the same security levels as the tools from the enterprise version provides. Safeguarding both your company and your clientele.

Riding the Artificial Intelligence Wave

Finally, we have artificial intelligence (AI) and machine learning (ML). When you think about it, these two elements are literally the sum of everything we want in the DevOps community—the ability to crunch out hours upon hours of menial labor and saving your teams hours to focus on higher-priority tasks.

The combination of AI, used to predict algorithmic influencers, and ML’s ability to adapt to new situations will be key for DevOps teams to get to the next level in 2019.

— Gil Artmoore
April 11, 2019
Autonomous Testing’s Role in Agile DevOps

DevOps testing, as an Agile methodology, is key to delivering testing and integration on a continuous basis, including continuous feedback, to enable better collaboration and quality.

For too long, testing has progressed by small incremental steps, creating a quality assurance (QA) winter. We think it’s time for a revolution, so we have written and published our manifesto for change. It’s a future where an intelligent test agent handles all testing needs so organizations’ resources are freed up, thus saving them time and driving revenue. This future means intelligent testing is a function everyone on a given team can contribute to—and, above all, allows for quality to be embedded in everything an organization produces.

How do we get there? Apply these perspectives and view testing strategically:

Testing Shouldn’t Be an Afterthought

QA should be a C-suite responsibility. No one laughs at the idea of a chief compliance officer, so maybe you should consider appointing a chief quality officer. Successful companies are the ones that adopt the mantra of test early, test often and test in depth. This means testing becomes a key strategic requirement in your organization. If testing needs more resources, make them available. If there is tension between your test and dev teams, try merging them. If you want to release new features frequently, look into continuous testing.

Testing Must Be Cloud-First

Up until now, testing has been done in-house using old servers, begged resources and outdated infrastructure. We believe testing should be run in the cloud, allowing you to leverage an almost infinite resource pool. Testing done like this will be faster, more efficient and cheaper.

Testing should not be the poor man of the software development process. Companies that engage with their testers produce the best and most reliable software. Those that treat testing as an afterthought risk bugs, production delays and, ultimately, their reputations.

Agile DevOps Requires Autonomous Testing

DevOps has become critical to the success of many companies. Without DevOps, many of the services end users take for granted would simply fall over. Social media sites and apps such as Facebook and LinkedIn would fail if they constantly went offline. If content delivery services such as SoundCloud, Netflix and Hulu kept dying mid-track, you’d soon cancel your subscription. People also expect messaging services to be 100% reliable.

DevOps straddles the border between developing software and running it in production. It is what you get when you apply the Agile approach to software development to deploying services in production.

Autonomous testing is the outcome of intelligent test automation. With autonomous testing, you now have an intelligent test agent (ITA) augmenting the work of your test and DevOps teams. This ITA is like the perfect regression tester—focused, tireless and driven, but still intelligent. Using this ITA means changes can be tested and deployed completely automatically. Without this, the job of an agile DevOps engineer would be infinitely more difficult.

Intelligent Automation Just Works

Classic test automation is dumb, and like all dumb systems, its growth and utility are limited. In this era of AI, test automation should be intelligent, too. There are few areas of software development where it is easier to apply AI—and even fewer where the rewards are so obvious and immediate.

Put simply, well-designed, intelligent automation just works. Intelligent automation means tests become less brittle, so no more time wasted on unnecessary maintenance. Productivity increases because test failures are automatically diagnosed and classified. New tests become easier to create and test flows are easy to reuse.

— Paul Clauson

April 8, 2019
Gremlin Brings Chaos Monkey Testing to Spinnaker CD Platform

Gremlin this week announced it will integrate its hosted chaos engineering service with continuous delivery platforms starting with the open source Spinnaker project developed by Netflix.

Chaos engineering traces its roots to a resiliency testing philosophy that posits applications should be able to keep functioning regardless of any service or infrastructure failure. Most IT organizations, however, don’t have the expertise required to build the tools required to testing resiliency based on those “chaos monkey” principles that randomly turn services off to see if an application fails.

Matt Jacobs, principal software engineer for Gremlin, said the decision to integrate with Spinnaker is being driven by the need to infuse chaos monkey resiliency testing directly into the application development and deployment pipeline at a more granular level. That capability is now a bigger requirement than ever because as organizations embrace microservices to build the next generation of cloud-native applications, it’s simply not possible for DevOps teams to manually track every dependency, he said, adding resiliency testing based on chaos monkey principles is just another type of testing that should be automated as part of a larger continuous integration/continuous delivery (CI/CD) platform to ensure an application doesn’t fail completely.

Integrating with a Spinnaker project that has drawn support from Google, Microsoft and Oracle provides the easiest path to achieving that goal. Spinnaker is also now a core element of a Continuous Delivery Foundation led by Alibaba, Anchore, Armory, Autodesk, Capital One, CircleCI, CloudBees, DeployHub, GitLab, Google, Huawei, JFrog, Netflix, Puppet, Red Hat, SAP and Snyk.

The Gremlin service is accessed via a user interface and comes in two forms: a free edition and an enterprise edition, which comes with support from Gremlin. While the concept of chaos monkey testing has been around for some time among the most advanced DevOps practitioners, enterprise IT organizations, which tend to be more conservative when it comes to introducing anything that might deliberately break their applications, will need a considerable amount of hand-holding. Gremlin is designed to learn the application environment in a way that introduces chaos in a controlled fashion before an application is deployed in a production environment.

On the positive side, resiliency testing based on chaos monkey principles tend to accelerate the rate at which IT staff learn how an application has been architected, said Jacobs. That can prove especially invaluable when training new members of a DevOps team, he noted.

Ultimately, the decision to embrace resiliency testing comes down to how proactive an organization wants to be when it comes to preventing downtime. When mission-critical applications are offline, there is a direct impact on the amount of revenue an organization generates. There are plenty of functional and performance testing tools, but the real difference between having an issue and a catastrophic event that requires DevOps teams to get out of bed in the middle of the night often comes down to how truly resilient any given mission-critical application really is.

— Mike Vizard

April 5, 2019
Sauce Labs Combines Functional and Performance Testing

Sauce Labs announced it is now moving beyond functional testing, adding a front-end testing tool it has dubbed Sauce Performance to make it easier to accelerate testing within the context of a DevOps process.

Lubos Parobek, vice president of products for Sauce Labs, said the goal is to enable developers or testers to begin testing both how well application functions work and perform much earlier in the DevOps process using the same testing tool.

Parobek said that as organizations embrace modern application development architectures based on microservices, it’s become crucial to test the capabilities of any module within an application because the number of interdependencies between modules continues to increase. That approach accelerates the rate at which applications can be developed. However, any flaw that impacts how the application works or performs can have a debilitating impact on any number of microservices that make up the application, he noted.

Sauce Performance makes use of debugging and performance trace logs to identify the root cause of an issue in addition to capturing metrics such as Time to First Meaningful Paint, Time to First Interactive, Page Weight and Speed Index. Sauce Performance also employs algorithms to automatically establish a baseline that can be used for regression testing.

Parobek said Sauce Performance serves to streamline testing processes as well. Most existing front-end performance testing solutions require integration with a load/stress testing tool or a separate application performance monitoring (APM) tool. However, Sauce Performance allows developers to use their existing Selenium test scripts to capture both functional and performance test data.

In addition, Sauce Performance has been designed to integrate with multiple continuous integration/continuous deployment (CI/CD) platforms to make it possible to feed test results directly back to developers, which can then decide to prioritize issues that need to be addressed.

Sauce Performance is available as part of the Sauce Labs Continuous Testing Cloud.

Parobek said organizations want to be able to make it easier for developers to test their own code as part of their DevOps best practices. That doesn’t eliminate the need for dedicated testers for evaluating functions, but potential performance issues today are being addressed often long before applications are deployed in a production environment.

One of the reasons applications are not as thoroughly tested as they might be is that many organizations require developers to learn how to use tools that have different user interfaces and then switch between testing tools. The result is a complex testing process that makes it more tempting to short-shrift elements of the testing process as application delivery deadlines loom.

More challenging still, many organizations still don’t continuously test applications after they are deployed to ensure new functionality does not adversely impact the user experience.

For as long as applications have been built, tension has existed between the developers who create applications and the people who test them. That debate may never be resolved, but everyone agrees in this age of DevOps that it’s better to test early and often.

— Mike Vizard

April 5, 2019
The New AI Overseer Role in DevOps

Robot overlords are the stuff of science fiction fantasy. Today, with more teams implementing artificial intelligence (AI), DevOps teams need a hard, continuous look at reality. An AI overseer watching apps at work could make the difference between sadness and euphoria in your efforts.

We’re seeing AI misses more and more often. Facebook recently admitted its algorithms failed to stop uploads of a particularly heinous live-streamed video. Autonomous vehicles have struck cars and people. Jaywalkers in China have been misidentified and publicly shamed in error. It’s not for lack of effort in development. The adage “If it were easy, anyone could do it” applies.

A few weeks back, I was in a Twitter chat session on AI led by Kirk Borne, data scientist at Booz Allen Hamilton. The question came up of what organizations should consider when creating guidelines for use of data and bias-free AI. Here’s my response:

A4: DevOps teams should have someone whose role is AI oversight—actually watching what it’s doing and constantly testing/validating its findings #MITSMRChat

— Don Dingee (@StratisetDon) February 14, 2019

Common Problems

Most fails are due to fire-and-forget, wherein AI systems are trained, tested against training and turned loose. Even a well-designed AI system is subject to several problems:

Errors: This is where AI makes a factually incorrect decision. False positives and false negatives are in this category, along with miscategorization problems.

Gaps: There are situations in which AI understands conflicting options and is unable to choose. Then, there are situations in which the AI doesn’t recognize what it sees.

Bias: Maybe the system prefers green M&Ms or German shepherds because there was some subtle trend in the training data for “candy” or “dog.”

Ethics: This can be a higher level of bias, where preference is given based on age, gender or race. Sometimes it’s the classic no-win scenario in which both options are harmful.

An AI overseer guards against these types of problems in near real-time, every day, for as long as it takes to weed them out. Some problems are easier than others. An example of a more difficult problem came when autonomous vehicle researchers realized that not all traffic signs are brand new and well-lit. Shade, branches, wind, scratches, dings, bullet holes, stickers, graffiti and many more types of defects exist in the real world.

Increased Complexity

Inevitably, part of the solution the AI overseer leads is better scenario-planning and more training data. Retraining becomes a way of life and at some point, bigger data sets also lead to more complex implementations and another challenge: speed—after retraining to correct issues, the reloaded real-time classification engine runs like it’s stuck in molasses, producing correct results too late to be useful.

While the A system is running, a B version should be in development with new training and classifier boxes. For very large data sets, teams should consider designing a workload-optimized training server complex instead of relying on elastic cloud resources. This was behind NVIDIA’s recent acquisition of Mellanox—using InfiniBand for more bandwidth between GPU clusters in AI server complexes.

Where training systems often scale, real-time systems usually don’t. They are optimized for size and power consumption and can only accept so many additions. When a bottleneck emerges, it can kill the entire application. New generations of CPUs, GPUs and FPGAs keep pushing the envelope for designers tackling AI challenges.

Human-in-the-Loop

In many use cases, the job of the AI system is not to make every single decision autonomously. Instead, the system should make routine decisions and lead the AI overseer to look at exception cases. Directing humans quickly to areas needing attention is often more valuable than risking bad decisions. Human-in-the-loop is something DevOps teams should architect into systems more often.

This may all sound obvious, but it points out a big difference in developing AI versus developing code-based algorithms. If there’s a bug in code, you can debug it and find the offending lines of code to fix. But debugging AI is a lot more difficult. Digital twin implementations come into play using playback of real-work inputs to look at what happens inside the classifier black box. There are usually hints of problems developing before catastrophe strikes. Running data both good and bad can help spot issues.

Ideally, the AI overseer is a hybrid of a data scientist plus a systems architect. Look for someone who understands both data sets and hardware implementations, at least at a high level. They may have to call in other resources on the team for detailed investigations.

The last thing you want after all the hard work in developing a system is to have it run amok months or years later. It’s just Murphy’s Law. AI systems will do something entirely unexpected at the worst possible time. An AI overseer can help a DevOps team get ahead of AI problems.

March 27, 2019
How DevOps Orchestration and Feedback Loops Can Enhance Your DevOps Pipeline

DevOps orchestration and feedback loops can significantly impact and improve your organization’s delivery pipeline. In this post, we’ll define and illuminate the importance of DevOps orchestration and feedback loops, and how together they can make a difference in your pipeline.

DevOps Orchestration

Although people widely use the terms “automation” and “orchestration” interchangeably, their scopes vary with significant process differences between them. DevOps automation automates a single function or a task in a DevOps pipeline that is routine and monotonous. Orchestration describes a string of such automated tasks consolidated into a streamlined workflow. In environments where automation would not work because of barriers among the groups, orchestration can be used to help organizations solve issues despite such barriers. Hence, DevOps orchestration simply brings together already existing automation networks into a cohesive, consolidated platform.

The Need for DevOps Orchestration

DevOps teams deal with constant navigation between development and operations. It would be highly beneficial if a single platform could pull together the tools teams typically use as various automation pieces from multiple DevOps toolsets. This is where DevOps orchestration services come into the picture, as orchestration enables teams to achieve that.

The primary goal of DevOps orchestration is to ensure seamless and quick delivery of new builds into production. It also aims to achieve streamlined and error-free software releases. Different organizations use various coding standards, guidelines and policies. DevOps orchestration plays a major role in efficiently managing and ensuring adherence to such policies. DevOps orchestration also is pivotal in properly defining and maintaining third-party software and their multiple versions across different environments.

DevOps Monitoring

Effective DevOps monitoring is vital. It is the key to understanding and evaluating various performance metrics in the production environment. When done right, DevOps monitoring can provide visibility into different critical aspects such as deviations in performance, application functionality issues and service availability problems. In any DevOps environment, the ability to identify and pinpoint such issues proactively can be highly beneficial, and DevOps monitoring enables executives to do so.

Feedback Loop

Feedback loops are the essence of any process improvement and DevOps is no exception. Typically, such loops initially function with an organization’s policies and progress all the way through implementation, policy adjustments (if any) and usually ending with reimplementation. They are not limited to specific process cycles in a DevOps environment—for example, it’s applicable to CI/CD practices, infrastructure monitoring and application development

Types of Feedback Loops

Broadly, there are two categories in feedback loops: reinforcing feedback loop and balancing feedback loop. People refer to the former as “a loop of accelerating change.” In a reinforcing feedback loop, increased value in the first entity results in an increase in the second entity, which in turn further increases the value of the first entity. Conversely, in a balancing feedback loop, value increase in one entity results in a decreased value in the second entity, which consequently decreases the value of the first entity.

Significance of Feedback Loops

One of the major capabilities that differentiates and carves a unique niche for DevOps from the rest of its predecessor methodologies is its feasibility to incorporate feedback from customers as well as other systems in the pipeline, including the monitoring system. The feedback obtained helps to bridge the gap between the software’s function and customer expectations. It also provides valuable insights about what you can do to improve the build of the software and ultimately enhance its functionality and reliability.

Feedback loops are particularly crucial when it comes to continuous testing. To carry out continuous testing effectively, building automated tests is only a job half done; the rest depends on the visibility of the test results and how you can utilize them to enhance the existing process. To do that, you need comprehensive feedback of your software’s performance across various stages of the software development life cycle (SDLC), starting right from development and continuing all the way to post-production. Implementing the right feedback loops will help you gather such detailed feedback.

Thus, feedback loops serve as the difference between merely speculating how an end-user might consume a product and methodically improving the existing workflow to better suit the end-user requirements through feedback.

— Kavin Elango

March 22, 2019