devopsy.kinsta.cloud

Tag: continuous testing

  • Continuous Testing Defined

    Continuous Testing Defined

    Across the software testing industry today, everyone has started to agree that continuous testing is necessary for keeping pace with Agile and DevOps. What’s not as clear, however, is exactly what continuous testing involves.

    Wikpedia defines continuous testing as the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

    At Tricentis, we believe that continuous testing …

    • Assesses business risk coverage as its primary goal
    • Establishes a safety net that helps the team protect the user experience
    • Requires a stable test environment to be available on demand
    • Seamlessly integrates into the software delivery pipeline and DevOps toolchain
    • Delivers actionable feedback appropriate for each stage of the delivery pipeline

    But what about the software testing community at large? How do you define continuous testing and what does it really mean to you? At Accelerate 2017, the continuous testing conference, we turned the question over to our attendees. We created a What is Continuous Testing board and invited attendees to share their thoughts.

    Some of the bravest attendees got the ball rolling with entries stressing the need for enterprises to embrace change and its importance for Agile and DevOps. 

    These entries (and perhaps the opening reception as well) opened the flood gates to lots of great attendee ideas. Here’s a glimpse at what people were talking about:

    As the board started to fill up, a few key words popped out:

    Many of the obvious buzzwords stuck out, such as Agile and DevOps—not to mention both Continuous and Testing, which were repeated often—albeit separately. Numerous cards indicate that enterprises see continuous testing as a journey or a goal to strive toward rather than a technical process component like continuous integration.

    Here at Tricentis, we are passionate about the future of continuous testing and are constantly innovating to help our customers stay ahead of the curve. To learn about what we believe continuous testing requires and our vision for where it’s headed, check out our Continuous Testing Resource Center and be sure to watch Tricentis Founder Wolfgang Platz’s keynote from Accelerate 2017.

    — Jessica Wanivenhaus

  • The Impact of DevOps and Continuous Testing in 2018

    The Impact of DevOps and Continuous Testing in 2018

    At its core, DevOps is meant to touch an entire organization; however it’s heavily weighted in the software development life cycle (SDLC). That’s why DevOps is going mainstream in 2018. Companies are making the business benefits of DevOps a top priority—it’s all about delivering value to the customer faster through maximum automation of the release cycle/processes. And that begins with organizations realizing that to mature their DevOps processes, they need continuous testing as part of their continuous integration practice.

    Cross-platform Engagement and IoT Will Mean More Devices—and More Apps

    With organizations shifting processes and innovation to a mix of digital technologies—a concept known as digital transformation—consumers are engaging with different apps across various platforms 24 hours a day, seven days a week. Add in the rise of internet of things (IoT) devices, which allow for seamless transitions across smartphones, TVs, tablets and other devices. Now is the time for optimum user experience (UX).

    In 2018, industries such as financial services, health care, retail and automotive will fully embrace IoT. We may also see new types of applications replacing current ones, as consumers and developers embrace the constantly evolving innovative capabilities coming in 2018. However, until the C-suite implements the tools needed to keep up with the growing demand for cross-platform upkeep and testing, other industries will experience slow adoption.

    A critical step in delivering flawless UX is testing, and lots of it. Never has there been more to test, measure and develop than in this digital revolution we are seeing today.

    Frequent New Releases Will Require Even Faster Updates

    As the number of platforms consumers engage with rapidly grows, so does the pressure for developers to ensure apps are running as they should at all times. Same goes for the pace of cloud adoption: Cloud test labs will need more sophisticated capabilities, meaning more platforms and blind spots which require continuous optimization to ensure market readiness. This need for faster turnarounds on update releases and bug fixes will multiply in 2018.

    To maintain and grow customers and profitability, CIOs must recognize the importance of empowering developers with the tools and time to implement continuous testing across the SDLC. Tools, such as those powered by automation and the cloud, increase efficiencies and free up time developers previously spent on manual quality-checking, allowing them to ensure the apps they are producing are keeping up with consumer expectations. If users feel their app is not delivering on the experience they want, they will delete it and move on to the next competitor, which has a direct negative impact on the company’s bottom line.

    Outdated Technologies Will Make for Even Slower Updates

    There is a suite of new tools being introduced to the market for all stages of the SDLC in the hopes of keeping up with the heightened speed of testing. As many DevOps environments struggle to adapt their pipelines to these new tools, many are powered by emerging technologies including machine learning and automation. In 2018, developers should ditch outdated infrastructure and processes in favor of more modern architectures, including implementing microservices.

    Modernizing pipelines will allow developers to design in next-gen engagement methods such as artificial intelligence (AI), augmented reality (AR) and virtual reality (VR) without compromising velocity.

    AI Will Experience Its Shining Moment (for Some)

    In the coming year, AI will start to move from a theoretical concept to a commonly applied technology, especially within the DevOps space. However, developers first must understand what they want AI to help them accomplish—and how—within the SDLC and across the DevOps pipeline. One logical place to start is figuring out how they can best leverage it to analyze test automation strategies.

    The value of a well-oiled, or not-so-well-oiled, DevOps process can make or break an organization. As the new year begins, be sure to include the IT leadership in these conversations to ensure developers are getting what they need, when they need it, to keep business moving as usual. IoT, AI and the cloud are just a few things to keep in mind as the industry moves to implement the latest tools in 2018.

    About the Author / Eran Kinsbruner

    Eran Kinsbruner is the lead technical evangelist at Perfecto and the author of the “Digital Quality Handbook.” He is a software engineering professional with nearly 20 years of experience at companies such as Matrix, Qulicke & Soffa, Sun Microsystems, General Electric, Texas Instruments and NeuStar. He holds various industry certifications such as ISTQB, CMMI, and others. Eran is a recognized mobile testing influencer and thought leader. He is also a patent-holding inventor (test exclusion automated mechanism for mobile J2ME testing), public speaker, researcher and blogger. Follow him on Twitter, connect with him on LinkedIn and check out his professional mobile testing blog.

  • DevSecOps: Security Needn’t be Sacrificed for Speed

    DevSecOps: Security Needn’t be Sacrificed for Speed

    With the right practices, security needn’t be an impediment to continuous development and rapid improvement.

    The same cycle keeps repeating, according to IT security and risk management consultancy Sense of Security co-founder and CTO Murray Goldschmidt. A new technology is hyped, leading to rapid adoption, but security is left as an afterthought. It happened with WiFi and voice over IP (VoIP), he says, and is currently happening with the internet of things (IoT). But it doesn’t have to be that way.

    DevSecOps: DevOps + Security

    DevOps is good for making things better, faster. But there tends to be a culture clash between those talking about speed, velocity or agility and those concerned with issues such as control points.

    So Goldschmidt and his colleagues are showing their clients how security can be integrated into DevOps (DevSecOps) in an automated manner without affecting velocity.

    Most people with development, operations or cloud backgrounds aren’t well-versed in security, he suggests, so Sense of Security shows clients how DevSecOps means security and DevOps can run in parallel.

    “It’s not that complicated,” he says, but it’s something most people don’t think about.

    Organizations often have a vulnerability management life cycle to identify and treat issues. After all, that’s part of various industry standards.

    The typical approach is an annual penetration test to find issues, which can then be addressed.

    A better way is to move to automated testing, and there are various commercial and open-source products to do that job. Goldschmidt says he sees little resistance to this idea.

    The next step is to go from scheduling these vulnerability scans on a quarterly basis to weekly or daily—or even to run them continuously.

    Citing the Struts vulnerability that caused a big problem for Equifax, he notes the challenge is to identify vulnerabilities in your systems before an attacker does. Continuous automated scanning means the window of vulnerability is very narrow.

    This idea is simple enough, he says, but for some reason it doesn’t occur to most people until someone brings it to their attention.

    When a scan reveals a vulnerability, its significance (severe, medium, etc.) is used to prioritize remediation.

    “DevOps guys understand those triggers,” he observes: If you give them the data, they can take the necessary actions. For example, that might mean shutting down the servers in a small web farm one at a time, applying the patch, restarting and moving on to the next. They could all be patched and running by the end of the day.

    Patching Isn’t Everything

    However, “it is unfair to assume organizations can patch things quickly” every time, he says. For example, the Struts vulnerability required more than just a patch—the code running on it had to be recompiled and tested, and that process could take months in a large organization.

    So organizations should not rely on the idea that patching is the fix for a vulnerability. While patches will be applied eventually, if you can make it difficult to actually take advantage of any exploits, the risk of data breaches is very low.

    One very easy-to-implement example of these measures is using a stateful firewall to ensure inbound-only access to web servers. By disallowing outbound access—”the channel through which data is lost”—there is no path for data to leave.

    Goldschmidt also recommends using tools to examine in-house code and the open-source and commercial code that is used with it. Such tools can detect most common problems, including openings for SQL injection, cross-site scripting and token misuse.

    He describes these tools as being the equivalent of “spellcheck for developers,” and says it reduces the likelihood of putting insecure code into production.

    The market includes products from Veracode (now part of CA), Sonatype and Qualys.

    Qualys, he notes, provides differential reporting. This makes it obvious that something has changed, and “that is the key to automation.” Whether the system itself has changed or new information has been discovered about vulnerabilities or configuration issues, action is required.

    Furthermore, the ease of making configuration changes that could have security implications makes continuous scanning even more important.

    These checks should not be limited to individual modules. Rather, they should be repeated—preferably automatically—as modules are assembled to help developers avoid poor code and to check the overall product.

    RASP Away Some Attacks

    Runtime application self-protection (RASP) is another technology Goldschmidt advocates. The idea is to detect abnormal and possibly malicious activity. One example is that normal operation would see an application communicating with its application server, and the application server communicating with the database. If the application suddenly makes a direct request to the database—as happens in an SQL injection attack—action is clearly required.

    Depending on the situation, RASP might raise an alert or immediately terminate the application, among other possibilities.

    The use of these and other technologies and techniques should allow organizations to run at the pace implied by DevOps without sacrificing security.

    — Stephen Withers

  • The Future of Continuous Testing: Fail Faster

    The Future of Continuous Testing: Fail Faster

    In the early days of Agile and Lean development practices, a very noticeable gap appeared between dev teams and ops and QA. Development was accelerating at a lightning-fast pace, but testing processes could not keep up, slowing down the entire life cycle and preventing organizations from achieving continuous delivery. Traditional testing methods, which rely heavily on manual processes and frequently updated GUI tests, proved too cumbersome and inadequate for our new lifecycle timelines.

    The era of testing as an afterthought is dying out, and continuous testing has emerged as the new standard. The idea behind continuous testing is pretty straightforward: Implement a systematic approach towards improvement that incorporates not only automation, but also all of the other tools and processes that will help reduce risk before software progresses to release and production. Organizations embracing this are rethinking testing as a continuous process that starts from the sandbox, rather than a precursor to release.

    Continuous Testing: Achieving ‘Fail Faster’

    At xMatters, we strive to a testing strategy that supports our goal of committing to production multiple times per day—one that we’ve been diligently moving closer to, having progressed from releases every quarter to every month to every week to now, twice per week. Our approach is to conduct unit tests at the team level, code quality, sonar and static analysis, and then deploy to production at limited percentages that we bump up in stages. Our overarching mantra is “fail fast,” because the faster and sooner you fail, the faster and sooner you can fix what’s broken, and the less resources and time you will spend troubleshooting and resolving issues. This, of course, translates to real cost savings.

    Correlating failure to actual events is the biggest hindrance to the continuous testing dream, as pinpointing root causes is an incredibly time-consuming process. This is where automation will take us to the next level, if implemented in a smart way. The future of continuous testing will be adaptive so as to determine the causes of failures faster. We need to automate as much as possible the process of predicting where code quality issues are and identifying the precise data correlations that will enable us to fix issues rapidly.

    Part of this adaptive testing approach will include automating intelligent regression testing. We don’t need to run a black box test in every instance, for example, which can take hours every time. We’ll save incredible amounts of time, money and developer resources if we design tests that can detect new changes and only test those. Smarter regression testing will result in precious time saved that ultimately impacts—and can even make or break—the speed of innovation.

    Another critical consideration to smarter testing is making sure to account for a comprehensive understanding of risk. We need to incorporate real business considerations into our understanding of risk and bake those into our tests as well. We should hold our software to not only standards of code quality and integration success, but also to meet real business expectations for areas such as security, compliance and reliability. Testing from the bottom up must be done in tandem with top-down analysis to account for the full spectrum of potential risks that have the potential to impact the bottom line. Machine learning systems can also be extended to learn new business applications as new functionalities are added and assist in creating test cases that serve to enhance the quality of the application.

    Developing these intelligent automation tools remains our biggest barrier to achieving the continuous testing dream, and while many organizations are gunning for it, most are still mired by manual processes. A global study conducted earlier this year by CA Technologies, for instance, demonstrated that while 75 percent of organizations identify continuous testing as important, just 20 percent report a high quality of test automation coverage.

    It’s worth noting that not every aspect of testing can be successfully automated. The vast majority of functional tests will almost certainly not be manual processes—but what about user experience testing? We have yet to conceive of an approach to this that can truly be automated. Because nothing beats a real human’s assessment of experiential quality, this will be one of the areas of testing that will likely remain manual. Thus, while our goal is to automate 100 percent, realistically this won’t be the case.

    The end goal here is not only to achieve the dream of continuous delivery, but also to eliminate distractions as much as possible. The more repeatable processes you can automate, the fewer distractions your team will encounter, and the more they can focus on creativity, strategy and building new features and capabilities instead of fixing old ones. This is what will ultimately determine your ability to innovate, as you’ll no longer have to sacrifice either time, scope or quality.

    About the Author / Deepa Guna

    Deepa Guna is Senior QA Architect for xMatters. She has been working in technology since 2003. Connect with her on LinkedIn.

  • Continuous Testing: Crushing the Testing Bottleneck

    Continuous Testing: Crushing the Testing Bottleneck

    The practice of continuous delivery has been with us for many years, but it has started to see real adoption in the last couple of years. Continuous delivery is about automating and simplifying the process of delivering new features to the market at any given time, accelerating delivery and helping bring Dev and Ops together to make them jointly accountable for the release process.

    As continuous delivery is helping to break down barriers, culture and bottlenecks in the delivery pipeline, a new practice has evolved to help teams solve this most daunting challenge of the delivery pipeline: Quality.

    The practice is called continuous testing, which can be defined as:

    • Testing your change in an efficient way (so your tests are completing faster),
    • In an optimized way (so that you find issues as early as possible in the cycle),
    • Based on risk and priority (so you test what is important to your users and the business).

    Some may think of this as equivalent to test automation, but that premise would be wrong.

    Automated testing tools can execute tests repeatedly, provide reports on testing outcomes and show comparisons with previous test runs. This is a critical foundation for a continuous testing practice, but just a starting point.

    In a continuous testing practice, testing is not done just in the “testing” phase and not only by test teams. Testing is performed at the level of the build (for example, Unit Test and Security Test) and does not stop after the change has been deployed into the production, where testing can ensure great customer experiences. Here are some other key aspects of continuous testing.

    Continuous Testing Automates Test Environments

    Let me clarify what I mean by environment. Basically, you want to replicate the real-world situation much as possible for testing. A test environment contains the test cases to be run, the right test data and services available, and the right application version deployed in the right environment. The juxtaposition of these elements of an environment must come together automatically to ensure a continuous testing practice.

    Continuous Testing Amps Up Throughput

    A key premise of continuous testing is that you are continually maximizing efficiency. This is a must to keep up with the velocity and frequency of today’s modern software factory.

    You are doing testing earlier in the testing cycle (shift left) where problems are easier to fix and less expensive. You are doing more testing in parallel. You are automating the promotion and feedback during testing stages. And you are ideally able to prioritize the testing order based on the risk profile of the code so that you are testing higher risk areas first. All of this enables you to find issues quickly and roll forward if possible or roll back seamlessly.

    Continuous Testing Applies Intelligence

    Finally, continuous testing means you have the knowledge at your fingertips about the risk profile of a release so you can make more informed decisions. You should be able to see what was changed in the code and the implications. For example, was there a small change in the UI or a complex backend change that affected integrations and could break the APIs for the interface?

    You also need a detailed view of the quality level of the release, including which tests have failed, which tests have not been executed, whether you have tested the new or the risky areas. All this intelligence will help you to understand the quality profile of a release and make faster decisions about whether additional tests are needed or if a release is ready to go to production.

    Steps Toward Continuous Testing

    There is no perfectly prescribed way to put a continuous testing practice in place for your organization. For example, you may find that your biggest bottleneck of the moment is managing test data, so you should tackle that first. But here are five steps critical in your adoption of continuous testing:

    Step 1: Implement best-of-breed test automation tools—Functional, GUI, Performance, Security, etc.

    Step 2: Automate test environments with the right test data and virtual services to ensure real-world, always available testing.

    Step 3: Automate test requirements (you’ll find manual methods can no longer keep up).

    Step 4: Automate the automation—Connect your testing toolchain and stages. Automate promotion from one stage to the next. Automate feedback loops end to end.

    Step 5: Get smarter—Automatically align the right tests to the right app version. Automate testing sequence by risk profile of the code. Build release health profiles.

    About the Author / Uri Scheiner

    Uri Scheiner is a senior director of product management in the Continuous Delivery Business Unit of CA Technologies. He was previously a key architect and responsible for the product road map and vision at a leading release automation company Nolio, which was acquired by CA in 2012. Uri started his career as a software engineer at Mercury Interactive (acquired by HP in 2006) where he was responsible for developing several products for the DevOps and APM markets.

  • How an Integrated Tool System Helps Implement DevOps

    How an Integrated Tool System Helps Implement DevOps

     DevOps is the marriage of application development teams with system operations teams, their philosophies and actions.

    In an agile environment, development, testing and operations all must work together to meet frequent iterations, releases and delivery goals. This calls for a collaborative development environment that supports data orchestration with other lifecycle tools.

    The article will explain how multitool integration support can help organizations perform the DevOps life cycle, which includes continuous planning, continuous integration, continuous testing, continuous monitoring, continuous delivery and continuous feedback mechanism.

    The Drivers of DevOps

    Today, with the increasing competition and the need for an early market release, the software development industry is undergoing a paradigm shift. Agility has become an inevitable process across all organizations. In addition, shorter ROI cycle spans, frequent releases, increased CRs and different IT systems and tool technologies have made the situation critical.

    The industry demands higher transparency, maximum visibility, minimum turn-around time, lesser human intervention and efficient processes at every stage, from development to delivery.

    However, organizations that use specialized tools to manage builds, deployments and delivery processes continue to face difficulties when it comes to data orchestration and process automation across the tool chain.

    In an agile environment, developers and testers need to ensure not only that code changes work and integrate well, but also  that frequent iterations do not make the product unstable. Hence, seamless cross-tool collaboration is very important here.

    The process chain consisting of Code –> Build –> Provision –> Test –> Defect Fix –> Deploy –> Release needs to run continuously and smoothly without compromising delivery deadlines or the product quality.

    Automation is the Catch Line

    With the rising demand for continuous delivery and sprint-based development, manually setting up a test, build and deployment environment is burdensome and time-consuming.

    Software teams need to automate the entire cycle of build, provisioning and deployment of test environments, including the tools, scripts and test data to ensure rapid delivery. They also need to collaborate around the application architecture and monitor event-action-based mechanism for seamless data flow across the tool chains.

    Reports show the most important DevOps components are IT automation, agile development and collaborative teaming of personnel. Therefore, without centralized orchestration of tools, achieving DevOps is not a realistic situation. Since organizations spend around 73 percent of their DevOps budget in tool acquisition, it is important they know how to get the tools working together to overcome DevOps obstacles.

    In the process of connecting their existing and new toolsets, most organizations search for an integration methodology that is scalable, reliable and easy to maintain. Point-to-point integration between tools help, but to a limited extent. Even a single change in integration rules involves complex coding and high maintenance across development, test and deployment cycles.

    How ESB-based Integration Can Help

    An enterprise service bus (ESB)-architected integration hub connects both the existing and the new tools to support a seamless data flow between them across the DevOps life cycle.

    Once integrated, the data from both the development and the operations tools become readily available for use to every stakeholder. The benefit of using ESB-based integration is that you can add or remove any tool based on your connectivity requirements.

    In a typical scenario, DevOps teams use a series of cross-functional tools starting from project planning to delivery and operations. Neither of these tools comes with built-in integration features, which results in inconsistent flow of information among tool users.

    Project managers use application lifecycle management (ALM) or Agile planning tools to detail the scope of work and tracking progress reports. Upon unit testing, developers use a different tool to check in a code from their integrated development environment (IDE) to the source code management (SCM) repository. They also perform a code quality analysis before committing to the SCM.

    In such a scenario, continuous integration (CI) can be achieved only when developers bring these planning, unit testing, code analysis and SCM tools under one roof. That is, connecting the tools through a common integration hub. It also helps developers to manage everything from within the IDE.

    After successful check-ins, build engineers trigger the SCM tools for automated build and testing. Upon a successful build, testers engage in continuous testing using manual and test automation tools, followed by provisioning for continuous deployment. They also perform load testing, API testing and security testing using separate tools. An integration support helps to keep the real-time data flowing between the build and the testing tools uninterrupted and seamless.

    A DevOps system also includes application performance monitoring (APM) tool for continuous monitoring of memory consumption, CPU usage, database health and release tracking. Operations team use help desk/ IT service management (ITSM) tools for continuous feedback generation and customer support. And delivery managers use continuous delivery tools for defining and tracking release pipeline through graphical interface.

    When all these tools are connected to a central integration hub, product release and help desk operations become faster and efficient. The integration between development and ITSM tools facilitates continuous feedback generation, which helps to improve the product quality.

    Through an integrated tool ecosystem, project managers can create a DevOps workflow based on business logic and automate the data movement across tool systems. Business analysts, developers, testers and delivery managers can also get real-time update on development progress and keep track of actionable items. Supervisors can route cross-tool data to a central repository and generate meaningful reports and analytics before taking business decisions.

    Fig: ESB Integration Bus Facilitating Adoption of DevOps

    An ESB-based integration tool helps organizations to implement and leverage the best DevOps practices by establishing connections across all participating tools. An integrated tool ecosystem helps achieve the following:

    • Real-time collaboration between development, delivery and operations tools
    • Continuous planning from requirements capturing and review to design and code analysis
    • Cross-tool traceability for defining relationships between various data objects
    • Test strategy implementation for continuous testing
    • Continuous integration through automatic triggering of build on successful completion of code check-in
    • Continuous testing through workflow based automatic triggering of both manual and automated test cases
    • Scheduled test automation script execution enabling continuous delivery
    • Continuous monitoring of release quality through reports and dashboards
    • Automated defect identification and resolution for faster help desk response
    • End-to-end traceability providing better release predictability and change impact analysis
    • Meaningful reports, metrics and KPIs for quick decision-making
    • Continuous delivery through tracking release pipeline

    How Inter-tool Connectivity Brings Continuity

    To understand the stages of engineering practice, let us consider a sample DevOps scenario in an IT service organization.

    The organization uses an ITSM tool to manage support tickets submitted by customers. On submission of a ticket, it is automatically passed on to an ALM tool as user stories or defects for internal development.

    On allotting to a sprint, developers and testers view the user stories or defects from IDE, SCM and test tool. Once the coding is complete and codes get checked in at an SCM tool, an automatic build gets triggered using a build tool. Upon successful completion of the build, tool provisioning is done through a virtual machine.

    The successful build can be automatically deployed using a deployment tool and automated tests can be executed through a test automation tool. The test results are then captured and linked with test cases and the source requirements for traceability.

    The defects that are raised in the defect tracking tool for failed test cases can be automatically routed to developers for fixing.

    Fig: DevOps scenario achieved in a collaborative tool environment

    Once connected to an integrated tool system, users get a complete visibility of the tool records and their relationships from within their own tools. Developers, testers and help desk managers can get complete traceability view between development artifacts and change requests or defects.

    When all the tools participating in DevOps sync with each other, developers can easily view the support tickets reported by the operations team right from their own IDE.

    The help desk team also gets a real-time update whenever a change request gets implemented or a defect is resolved by the development team. It eliminates the latency in communication between the Dev and Ops teams to a great extent.

    In addition, customers get frequent updates from the help desk team, which makes them happy and confident about the product performance.

    Conclusion

    Implementation of DevOps is not a “Do once and forever” type of task. Just deploying the right set of tools to the right users will not serve the purpose. Proper connectivity between and across the DevOps tools chain is necessary if you want to create a collaborative work environment around the Dev and Ops teams.

    An integrated tool system saves considerable amount of time and effort, which would otherwise be wasted on manual execution, monitoring and reporting of data during DevOps implementation.

    About the Author / Sanat Singha

    Sanat Singha is a senior technical and web marketing writer. He takes special interest in ALM process, technologies and tools integration as an industry. Sanat does content marketing, professional blogging, social media promotion and website analytics.

  • DevOps, Continuous Delivery – It’s All About Automation

    DevOps, Continuous Delivery – It’s All About Automation

    I spent a large portion of my career in the networking industry, marketing routers, switches and other gear that makes the internet work. Networking—like many industries—is currently undergoing a revolution that is in large part driven by software. Functions that used to be delivered as a standalone device can now run in software (not unlike the apps on your smartphone); and software is also being relied on more to make decisions about how traffic is routed through the network.

    I still read up on the networking industry occasionally, and if you’re looking for an expert opinion or just good journalism in this space, you’d be hard-pressed to find a better source than Craig Matsumoto. Craig wrote an interesting opinion piece called “NFV, SDN, Big Data – It’s All About Automation.” You don’t have to know what those acronyms mean to understand the premise.

    What’s interesting is that the same premise can be applied to many of the things that promise to make software development easier, faster and cheaper. DevOps, continuous delivery, continuous testing—all rely on automation to be successful.

    DevOps, of course, requires more than just automation—as it’s also a cultural shift—but automation plays a huge role. The three main components of successful DevOps are commonly cited as tools (or technology), process and people (or culture). Automation is the primary goal of the first two: tools that enable you to automate previously manual tasks, and then establishing the processes that allow you to take advantage of this automation.

    The same can be said of continuous delivery or continuous testing. Continuous testing relies on being able to automatically generate test cases, automatically create virtualized testing environments and create automated calls for testing.

    The premise of a modern software factory also relies on automation. Any modern factory, whether it’s producing cars or toothbrushes, strives to improve efficiencies by eliminating as many manual tasks as possible.

    The fact that software development and operations are lagging behind other industries in terms of automating the production of its goods (i.e., software) is maybe a bit ironic, given that software development is typically seen as more technologically advanced.

    However, it’s also a result of the maturity of the industry and that fact that coding, by its nature, cannot be automated. But the tasks that surround the actual writing of code certainly can be automated, and that’s what the modern software factory promises. The more you can automate, the more successful you will be in delivering quality software faster, with reduced risk and costs.

    Automation seems to bring with it concerns about employment—that is, there is a not-unfounded fear that it’s possible to automate oneself out of a job. However, there was an interesting study that came out, focusing on the automobile industry—one of the most well-known examples of factory automation. The study concluded that:

    In the last six years, (2010–2015), according to the IFR (International Federation of Robotics), US industry has installed around 135,000 new industrial robots. The principal driver is automation in the car industry. During this same period, (2010–2015), the number of employees in the automotive sector increased by 230,000.

    Essentially, automation not only improved production velocity and quality, it also resulted in a net increase in employment.

    The message is clear: Automate what you can and only rely on manual tasks for what you must. If you focus on automating your software development life cycle, not only will you be able to deliver software faster, cheaper and with better quality, but you also will be well on your way to achieving continuous delivery and DevOps.

    — Brendan Hayes

  • In DevOps, Quality is a Team Effort

    In DevOps, Quality is a Team Effort

    Technology is changing every industry; we can all agree on this foundational premise. If you disagree, I believe the rest of this post is going to do nothing for you, so feel free to stop reading now. If you do agree, then you know that apps, the internet of things (IoT) and APIs are table stakes now. Businesses need strategies for each, and the forward-thinking executives are already adapting. The ones that aren’t risk alienating a channel in their overall digital strategy. It’s not all about strategies for building new types of functionality and partnerships; organizations also need to validate they will work specifically with their customers’ use cases in mind.

    When we study software development, the history books teach us that waterfall strategies followed a linear path. In the past, this pace may have worked, but the future seems bleak with the evolution of technology and its disruption of traditional, manual business processes advancing at an exponential rate. Over time, the stages in the software development life cycle (SDLC) may remain the same, but organizations need to focus specifically on becoming more efficient at completing more work in less time. With this “do more with less” mindset, it’s no wonder many organizations see a backlog of work piling up. This point is especially true for quality assurance and quality engineering teams.

    First, let’s unpack the software development stages from a traditional sense. Historically, organizations would get an idea and that idea could come from the businesses stakeholders (sales, marketing, customers, etc). The business would prioritize/stack rank all of their ideas by which carried the most merit and how much work those ideas would require for implementation. This is known as the “planning” phase. From planning, the idea would progress to be built. “Building” is the part of the life cycle where your artists/engineers live. You know, the creative people, the developers. They build the technology according to specifications that come from the planning phase. Most importantly, they are incentivized based on the project being delivered on time.

    After the technology is built and packaged, it is passed on to the team that has been traditionally responsible for quality—quality assurance (QA). QA has their time with the product to validate the product works as intended, can scale to the appropriate demand levels, is secure and meets all the use cases without error before they pass it over to the team that will deploy it to the hands of their customers. Finally, the business is measuring feedback and customer sentiment, and then rolling that feedback back to the plan stage for improvement.

    As time has passed, we have also seen another major trend develop. Organizations have developed agile/iterative SDLC strategies that are focused around bringing smaller chunks of functionality to their users/customers faster. Just like “lean manufacturing,” this concept has been focused predominantly around helping the developers get their projects completed faster. The primary challenge with waterfall development strategies is that by the time an organization brings the plan to market, the needs and priorities of the customer will have already changed. The speed of idea generation outpaces the execution of said ideas. That being written, while development has kept pace, quality has not, unfortunately.

    Now I pose a simple, albeit, extremely important question: Who owns responsibility for ensuring quality of the finished product? If your answer is QA (like stated above), and I can see why that would be the instinctual response—the process is set up for failure. Here’s the fundamental reason why: if quality is so important to the process, then why is it the third phase in the SDLC? This implies planning and building without quality in mind is an acceptable process.

    Quality should be owned by everybody in the process. QA/QE teams are your enablers of embedding quality throughout the SDLC. The distinction is more about instituting an ideology or culture based on quality.

    With a culture focused on driving quality, when the planning team works on creating a specification for the idea, they should work with development and QA/QE as they are designing the flows, behaviors of the application. This allows for QA/QE teams to understand what to validate much earlier in the process. Teams are always looking for ways to factor more automation into the process. QA teams are also looking for ways to make their test environments more closely resemble the production environment where their customers interact with the business.

    Unfortunately for QA/QE, they feel pressure from the business to execute within a constrained time frame with better results. This typically means inadequate windows of opportunity to start building automated tests or an incomplete environment that doesn’t quite resemble what the end user will face.

    The last frontier to validate is performance which answers the important question: Will the app scale for the expected user population? This is often the last step before flipping the switch and shipping the product to your customers. But what happens if the application doesn’t scale? Organizations need to drive performance validation earlier. Better yet, they should engineer the application for the desired performance, and that means—yes, you guessed it—taking care of this earlier in the life cycle.

    What if it were possible to drive more improvement?:
    1. … To start building functional test plans and test automation at the time when planning happens?
    2. … To ensure that development and test environments more accurately have access to the constrained downstream resources that cause unexpected “wait time”?
    3. … To ensure that the automated testing that will be executed in the testing phase, has the right data to power the tests?
    4. Finally, … to make sure that performance isn’t just an afterthought but a strategy to scale to the changing landscape of a digital business?

    It’s more than possible. It’s obtainable. This process is known as continuous testing. It’s about the validation that needs to occur earlier—not just within a window of allotted time within an overall life cycle. It’s also important to understand that it’s a maturity curve, and it will take work to evolve your individual process.

    Can your organization embrace continuous testing strategies? If not, what’s stopping you? If so, where can we begin to improve? Driving quality isn’t just about your company’s KPIs—it’s about your customers’ perception of your brand and what you do for them. You never get a second chance to make a first impression. Let’s build the right impression today!

    — BobbyMelwani

  • Move the Needle in DevOps with Continuous Testing

    Move the Needle in DevOps with Continuous Testing

    Shift-left testing, risk based testing, early testing and technical debt reduction are all predicted to be some of the most sought-after trends in product development life cycle. All of these can be achieved with continuous testing through DevOps, as it emphasizes quicker transitioning of quality products to production environment at a reduced business risk.

    Software Engineer in Test (SEiT) or Software Developer in Test (SDiT) was long-spoken earlier and now implemented as part of DevOps culture. As the software side of the organization moves to continuous testing practices, so, too must the product side of the organization. External market forces including competition, innovation quotient, faster time to market, scalability and lean process initiatives definitely play a role in driving the continuous testing trend.

    From a technical standpoint, test environment creation, full test coverage, regression testing and integration testing of multiple endpoints are key parameters to ensure a faultless production environment when a new feature is added.

    Adopting Testing Tools for DevOps Implementation

    Continuous testing implementation not only includes adoption of right tools and test automation, but also a cultural change to incorporate testing at each stage of the agile development life cycle. It will involve changes right from test planning, test scenarios creation, automation feasibility analysis to test execution.

    Teams need to be restructured in an agile setup to align testers closer to developers and operations teams, and have common goals for product/software delivery. This involves changing the culture and organization structure, which generally takes a longer gestation time. Test management tools such as JIRA and TestNG can help expedite this structuring, as both parties would be aware of the product features that need to be delivered and can view the progress at all times. There also are various open-source tools, including Selenium, Appium, Python robot framework/pytest and JMeter, that aptly meet the needs of agile testing.

    Regression testing generally takes up the bulk of the time, as new features need to be tested every time they are added to avoid critical failures. Hence, automating regression tests have become an imperative as part of continuous testing. JUnit, which is a unit testing framework, promotes test-driven development and early code quality improvement. Bamboo can run tests in parallel batches and support hundreds of build agents providing an early feedback to developers. Automation scripts for test data cleanup, load and execution saves the testing effort of the operations team. Selenium test automation framework for web applications is one of the most famous tools for continuous testing, as it works with a host of programming languages including Java, C#, Groovy, Perl, PHP, Python and Ruby. Appium is another tool used frequently for cross platform mobile application testing.

    Ensuring an Effective Product Release

    Testing UI components and functionality of a product through API-based test automation is another important phase in continuous testing. API test scenario creation also generally needs involvement from the ops team. Considering virtualization of devices, along with applications or services, will accelerate test setup environments. Device virtualization using Raspberry pi helps in testing end–to-end use cases in IoT scenarios. Network virtualization for performance testing ultimately helps improve uptime and reduce outages.

    The orientation of continuous testing with continuous integration and continuous delivery pipeline within an enterprise will determine the long-term success and standing of the final product in the market, from a quality perspective.

    About the Author / Urvashi Babaria

    Urvashi Babaria is a Product manager at eInfochips working in new age areas like IoT, DevOps and CloudOps. She is a techno commercial marketer with nine years of experience in product and project management, business analysis and transformation. She has keen interest in areas of data science including visualization, analytics and modeling. Connect with her on LinkedIn.

     

  • Moving from ‘Tester’ to ‘Quality Enabler’

    Moving from ‘Tester’ to ‘Quality Enabler’

    Speed versus quality. Can you really have one while the other exists?

    When Agile came on to the scene, it pushed to get development teams to create code as quickly as possible. Then DevOps came on as a practice that helped these Dev teams and the Ops side of the house to interact and collaborate more on best practices, tools to use and pinpointing issues in operations.

    But the testing practices and QA teams have had challenges keeping up with all this code that is getting thrown their way at an avalanche pace. That is probably why nearly two-thirds of self-proclaimed DevOps practitioners report that their biggest bottleneck occurred in the QA/Test phase. In fact, this cited bottleneck was more than 2 times greater than the No. 2 proclaimed bottleneck, planning (29 percent).

    It’s been said that “the consumer appetite to engage via digital channels is matched only by intolerance for any defects that might have an impact on their user experience and business outcomes.” If organizations are going to speed their delivery pipeline, they need to remember that the success of continuous delivery hinges on the effective employment of continuous testing.

    Part of the problem remains with manual testing practices. You can’t manually test millions of lines of code, then try to figure out the customer experience during integration testing with other internal or third-party systems and API, with hundreds of microservices or varying mobile devices.  The complexities of today’s composite apps make it impossible.

    And traditional, legacy testing tools don’t help, either. Gone are the days of waiting for the UI to be complete before QA teams begin their testing. And gone are the days of developers throwing code over the wall to the testing team and washing their hands of it.

    A few other facts:

    1. Ninety-one percent of QA professionals agree that the role of testing must change to meet the demands of today’s agile development needs.
    2. Only 1 in 3 organizations involve QA in planning requirements.
    3. Fewer than 1 in 5 organizations write new tests each time new code is checked in.

    The idea of continuous testing helps to solve that. Continuous testing is the practice of testing across every activity in the software development life cycle (SDLC) to uncover and fix unexpected behaviors as soon as they are injected. That can’t happen in the above environments.

    A new ebook from CA Technologies and devopsy.kinsta.cloud notes that to continuously deliver software, organizations need to make sure tests, procedures and processes around quality are designed to fit in line with the flow of development, integration and deployment. While continuous testing is dependent upon strategic automation, it is not synonymous with automated testing—a continuous testing process encompasses other important QA activities that impact overall product quality.

    As such, continuous testing depends on four major pillars:

    1. Code Quality – Did developers create the application code correctly?
    2. Pipeline Automation – Can the application code work across environments without manual intervention?
    3. Application Quality – Did developers create the correct features?
    4. Customer Experience – Are users perceiving value in the application delivered?

    These four pillars are discussed in more detail in this continuous testing ebook. You’ll learn the evolutionary journey you can take to shift testing practices left, and really make testing and quality everyone’s responsibility. You’ll even read how GM Financial was able to transform its load practices by implementing some of these practices. See how you can start using these pillars in your QA efforts.

    Now is the time to elevate QA’s new role from that of tester to quality enabler.

    — Scott Edwards