Tag: application testing

  • What Developers Should Take Away from the Iowa Caucus App Debacle

    What Developers Should Take Away from the Iowa Caucus App Debacle

    In the wake of the recent Iowa caucus, vote-counting software has fallen under suspicion after a smartphone app built by Shadow, Inc., brought chaos and delays to the first 2020 Democratic caucuses. Stopgap measures, such as Nevada officials canceling the application’s use in its upcoming caucus, are not enough to ensure vote tallies the electorate can be comfortable with. What’s needed is to correctly identify and fix the underlying problem. 

    The flaw does not lie with a lack of end-user sophistication, dataset complexity or inappropriate app security. Remember, government officials around the world are transforming urban areas into connected, responsive smart cities. Students are live-streaming interstellar space flights to Mars and Saturn. Uber drivers navigate past traffic jams, while seniors use apps to make time-sensitive healthcare decisions and adjust their retirement portfolios. People of all ages and experience levels do this and more, on the fly, with handheld mobile devices. 

    The real problem lies in the way leadership teams drive software delivery. The Iowa app failed for a number of reasons: It was released with no product testing after a short, two-month development runway conducted under a cloud of secrecy. It was not built to scale. Worse, there was little to no input, transparency or communication among the application’s most important stakeholders—the Shadow management team, the software developers and the end-users, in this case, precinct captains running the Iowa caucuses and party.

    The IT professionals in charge could have avoided these problems by following three simple principles of successful software development: 

    • Collaborate With All Stakeholders: Applications are no longer stand-alone widgets. They are digital experiences. With even the most sophisticated design, experiences that don’t serve their target audience are failures. A development process that prioritizes collaboration with end users can prevent such failures. Feedback from those with a vested interest in the outcome would have quickly raised red flags for developers. In Iowa, collaborating with intended users would have highlighted a need for accessibility training and connectivity issues, since the app was deployed in buildings with high-speed internet along with remote, off-grid locations. 
    • Build to Scale: Every developer knows to build for performance at an order of magnitude larger than you expect. This includes ensuring performance on the widest possible range of handheld devices and phones. From a developer’s perspective, the phone model will impact application performance due to different hardware specs. It’s imperative for designers to build apps that are native, so that all users, devices and performance profiles will function as intended. A useful software solution should reap success, not failure, when more people use an application than initially expected. 
    • Be Transparent: Valid concerns over security risks and election integrity unfortunately led Shadow’s team to prioritize secrecy over transparency. That didn’t work. Problems are not detected and solved when siloed or when small groups of people work on a solution in secret. Proper vetting and testing falls by the wayside. The risk that the app will not be suitable for its intended users increases. The app may have additional, exploitable security vulnerabilities that can only be found when teams ensure proper openness and transparency during the development process. Proper governance and guardrails that protect core data mitigates the security risk for election-based applications (and software in general). There’s little to be gained and much to risk by closing off input and vetting at any point during the full application development cycle. 

    Developers Move From Stressed to Empowered 

    One simple, effective way to solve or avoid these issues is by adopting an enterprise-grade, low-code software development platform. The “low-code” label refers to visual, model-driven software development, which employs an unambiguous visual language throughout the application development lifecycle, enabling nontechnical stakeholders, end users and professional developers to work together. 

    Such a collaborative platform can surface issues with senior leadership before it’s too late. Feedback from citizen developers—that is, not citizens in the voting sense, but rather, non-traditional developers—can capture leadership’s careful attention, precisely because the feedback is from nontechnical users. The right low-code platform would have enabled the Iowa caucus volunteers and stakeholders to collaborate on every step of the process without slowing it down.

    Another benefit of a robust low-code platform is having more people on deck to meet product goals with tight deadlines. Insights from citizen developers with unique backgrounds can also help when testing applications. With suitable governance controls, low-code functionality speeds up a developer’s ability to deliver software without taking away their ability to dive deeply and extend functionality with coding when needed. 

    Market analysts have evaluated the strengths and merits of different low-code platforms. In multi-experience use cases—such as Iowa’s, particularly because the eyes of the world are watching—it’s important to use a low-code platform that enables applications that are truly device native. This allows the software to take advantage of core device capabilities, typically delivering better performance and supporting the interactivity users are familiar with. Additionally, continuity online and offline would address connectivity issues in areas where signals are weak or nonexistent; the data can be properly synced when a signal is available. Lastly, a robust low-code platform provides tools to enable apps to scale automatically, avoiding the coding work required to allow an app originally built for 1,000 people to service 100,000 or a million. 

    An Urgent Role for Software Developers

    When roughly half of all eligible voters sit out general elections, talented developers should not lose sight of the many ways technology can aid and expand civic participation. Eventually, we may live in a world where election-based apps for registered voters are accessible and secure, regardless of the user’s device, age, location, physical ability or educational background. Building in redundant systems of paper-based vote confirmation and other offline systems will add integrity, inspire greater confidence and drive adoption of these innovations.

    What happened in Iowa is, unfortunately, not an isolated case. These complications are endemic to how most leadership teams drive software delivery. Development teams of all types will continue to run into similar issues until their leadership makes transparency, accessibility and scalability a priority by investing in the appropriate resources. Organizations can leverage the strength and impact of fully robust low-code software development platform to that end. 

    Since the introduction of low-code in 2004, platform capabilities have matured at an accelerated rate, moving to widespread adoption for enterprise use. During the same period, elections have increasingly relied on software-driven electronics, leading to consistent issues. Citizens casting ballots is one of the most sacred mechanisms of our republic. Any software used to tally votes must be bulletproof. Truly open and holistic low-code software development platforms can make this happen by proactively bringing people together to make sure software works for those who need it most.

    — Jon Scolamiero

  • Testim Adds SDK to AI-Based Testing Platform

    Testim Adds SDK to AI-Based Testing Platform

    Testim this week launched a beta release of a software development kit (SDK) consisting for a set of JavaScript application programming interfaces (APIs) that will enable DevOps teams to build tests that leverage machine learning algorithms from within their integrated development environment (IDE).

    Company CEO Oren Rubin said the Testim Development Kit makes it possible to call capabilities such as the ability to record user interface interactions via an application programming interface (API).

    The Testim approach to applying artificial intelligence (AI) testing differs from rivals in that Testim allows DevOps teams to fully inspect how algorithms construct a test, he noted. DevOps teams can also customize those algorithms to fit their own requirements, which is critical when it comes time for DevOps to explain how tests were conducted as part of a compliance audit.

    Testim will continue to provide access to its platform via a graphical user interface, but with the edition of the SDK, Rubin said his company expects to see an increase in the number of DevOps teams that will incorporate Testim within the context of continuous development/continuous deployment (CI/CD). Once that occurs, the number of tests those teams will be willing to run against software before it’s deployed in a production environment should also increase, especially as more developers leverage APIs to create their own tests, Rubin said. That doesn’t eliminate the need for another set of eyes to test an application, but it should reduce the number of common errors made before any piece of code is tested by someone else on the DevOps team.

    One of the most time-consuming elements of any DevOps process is the time it takes to create tests for each application. The Testim platform employs machine learning algorithms to discover the entire application environment and then begin creating appropriate tests. As DevOps teams modify those tests, the Testim platform continues to learn how to build tests the way the DevOps team prefers.

    The Testim platform doesn’t eliminate the need for testing expertise on the DevOps team, but it does enable organizations to scale their testing processes by augmenting whatever testing expertise they have on hand to reduce bottlenecks and improve overall software quality. Those algorithms also provide the added benefit of never taking a day off or quitting to take a more lucrative position elsewhere.

    When it comes to AI, there is naturally a lot of fear and loathing. On the one hand, IT professionals become concerned AI will replace them. On the other hand, they are often disappointed when machine learning algorithms don’t instantly automate a process to their satisfaction. It takes a significant amount of time for algorithms to learn how any IT environment operates, with each update to that environment requiring those algorithms to discover new functions. At this juncture, it’s not a question of whether AI will play a role in DevOps processes, but rather to what degree. Given the repetitive nature of testing, Testim is betting application testing will prove to be an area that lends itself to AI sooner than later.

    — Mike Vizard

  • Datadog Acquires App Dev Testing Platform Madumbo

    Datadog Acquires App Dev Testing Platform Madumbo

    Datadog announced it is extending its reach beyond IT monitoring to include application testing by acquiring Madumbo.

    Madumdo CEO Gabriel-James Safar said it’s become increasingly apparent that application testing needs to be more informed by the metrics gathered by IT monitoring tools. The combined Datadog/Madumbo will bring intelligence to automated testing processes using both application and infrastructure data gathered via the Datadog platform.

    The Madumbo platform already makes extensive use of machine learning algorithms and predictive analytics to inject artificial intelligence (AI) into the application testing process. The more data those algorithms can access, the more precise the AI models become over time.

    That approach made it possible for Madumbo to build a bot that checks for errors using code running on a real browser, which Safar said means DevOps teams can capture the same defects as they are experienced by end users. That’s critical because much time is wasted by IT organizations that are unable to prioritize tasks to either identify the root cause of a problem or, arguably more importantly, prevent issues in the first place, said Safar. Because of that issue most organizations lack the time required to develop best-in-class DevOps processes.

    Datadog provides a complimentary set of IT monitoring tools that are consumed as a software-as-a-service (SaaS) application. That data will then become a trusted source for Madumbo, which in turn will increase confidence in the AI recommendations being made by Madumbo, said Safar.

    How many of those customers will wind up replacing whatever application testing tools they currently employ with Madumbo remains to be seen. But if Datadog can provide a more efficient approach to testing, testing tools may wind up driving organizations to replace their monitoring tools. Most organizations spend an inordinate amount of time trying to correlate data across tens of monitoring tools that often generate either conflicting alerts or alerts about issues the IT organization has already been informed of by another tool. Most IT organizations eventually will see the rise of AI platforms that will drive consolidation across the entire DevOps tools category.

    In the meantime, the more issues that get addressed during the application development process, the fewer number of problems that will raise their ugly head in production environments. In fact, it’s arguable the whole point of investing in best-in-class DevOps processes is not just to roll out applications faster, but also make sure those applications are of a markedly higher quality than previous generations of applications. That’s not going to happen, however, unless DevOps teams can derive actionable intelligence from the millions of alerts that get generated up and down the IT stack.

    — Mike Vizard

  • Eggplant Extends Scope of AI Applied to Application Testing

    Eggplant Extends Scope of AI Applied to Application Testing

    Reliance on machine learning algorithms and other forms of artificial intelligence (AI) to not just run application tests at scale  but actually construct them may be about to increase.

    The latest release of a Digital Automation Intelligence (DAI) Suite from Eggplant enhances the AI capabilities baked into the testing tool to include the capture and analysis of runtime performance data in a way that enables a testing process to “learn” where potential bottlenecks are hidden. Eggplant CTO Antony Edwards said DAI Suite will then suggest a series of tests designed to make it easier for developers to identify the root cause of the problem.

    The goal is to reduce reliance on humans to construct formulaic tests, which Edwards said often limits the rate at which applications can be deployed and updated.

    Eggplant’s DAI Suite

    Other new capabilities being added to DAI Suite include enhancements to Eggplant Functional, which now takes advantage of AI to discover the elements of a graphical user interface (GUI) to automatically populate the automation logic in an Eggplant AI application model. Eggplant Performance, a load and performance testing product, has been integrated with JMeter, an open source performance testing framework, to enable developers and testers to reuse JMeter assets as well as creating load test scripts. Additionally, new real-time key performance indicators (KPIs) can be employed to shorten the feedback loop between testing and analysis.

    Both Eggplant Manager and Eggplant Automation Cloud are now 10 times faster and provide higher levels of testing scalability, according to the company.

    Edwards noted that, in general, application testing is too formulaic. The end goal should be on identifying the best way to delight an end user rather than merely trying to check a compliance box. That requires being able to test more application use cases at scale without slowing down the overall application development life cycle. In fact, he said the biggest hindrance to DevOps today is a reliance on manual testing processes that are driving a $32 billion testing services sector.

    It’s now only a matter of time before AI is applied more broadly to application testing. As the number of applications being developed increases, the more challenging testing becomes. The longer it takes to conduct those tests the more tempted developers are to short-shrift the testing process altogether, believing whatever issues that might arise can be addressed quickly using agile development methodologies. However, not only are there more dependencies than ever between applications, but also end users are less forgiving. Many of them will not, for example, ever give a mobile application that initially disappoints them a second look. Compounding that pressure is increased reliance on digital business processes, where the quality of the underlying application experience directly correlates to the perception of the value of the corporate brand.

    Developers may not always have the highest regard for application testing. But as IT continues to evolve, chances are high there soon will be many more developers perfectly happy to take advantage of all the AI help they can get.

    — Mike Vizard

  • Plutora Brings Order to Application Testing Chaos

    Plutora Brings Order to Application Testing Chaos

    One of the reasons application testing doesn’t occur as often or extensively as most IT organizations desire is because managing multiple simultaneous testing projects is complex. To help simplify the management of applications testing at scale, Plutora has updated Plutora Environments, its continuous delivery management platform, which now includes an ability to manage the availability, usage and configuration management involved in testing applications at scale.

    Jeff Keyes, director of marketing, says Plutora Environments now provides a single repository through which all application testing can be managed, including identifying and provisioning available IT infrastructure resources.

    Made available as a software-as-a-service (SaaS) application, Plutora Environments provides needed governance and control capabilities. Without them, Keyes says, application testing tends to get de-prioritized when developers find it too challenging to quickly access the appropriate resources.

    Capabilities added to the latest release of Plutora Environments include the ability to:

    • Visually associate new code commits with the features or fixes via a dashboard,
    • Ensure accuracy of test coverage across high-velocity tests,
    • Trace updated code to specific defects or original requirements, and
    • Automatically link change requests to associated test cases.

    Keyes notes a large amount of IT infrastructure allocated to testing is being wasted due to a lack of oversight. In many ways, what’s been missing in many DevOps environments is some of the management functionality found it traditional IT service management (ITSM) software, he contends, and the latest release of Plutora bridges that gap within the context of an application testing environment.

    Now that developers are being held more accountable for the quality of their applications, demand for access to testing resources will steadily increase, Keyes says.  IT organizations then must balance a limited amount of resources against those demands—they may want to shift responsibility for application testing to the left as part of an integrated DevOps process, but to accomplish that goal, they will need to be able to provide developers with access to infrastructure resources for testing purposes on demand. Otherwise, developers will simply opt to continue writing code, even though many of the dependencies on which that code depends has not been tested.

    Application testing, regardless of era, regularly gets the short shrift. Developers are almost always falling behind schedule. One of the easiest ways to make up for lost ground is to reduce the amount of time allocated to testing. Some developers even believe agile development methodologies enable them to respond fast enough to any flaw by delivering additional updates—many of which don’t get tested either, because of a lack of time. End users, however, are becoming less forgiving when it comes to application experiences. If an application doesn’t perform as expected the first time, customers are less apt to give that application a second chance, much less a third or a fourth.

    Because of that reality, in many organizations there’s more focus than ever on application testing. The challenge is not necessarily creating or running the test, but rather, how best to manage the overall process end to end.

    — Mike Vizard

  • Micro Focus Unfurls Functional Testing Service

    Micro Focus Unfurls Functional Testing Service

    The merging of software assets from Micro Focus and Hewlett-Packard Enterprise (HPE) that occurred earlier this year included StormRunner, a series of load- and performance-testing tools. Micro Focus recently extended that portfolio with the introduction of Micro Focus StormRunner Functional, a suite of functional testing tools delivered as service that can be employed to test applications running on any browser, operating system or mobile device.

    Shane Evans, director of product management for application delivery management for Micro Focus, says StormRunner Functional spans a variety of automated tests addressing unified functional testing (UFT), UFT Pro and Selenium browser testing tools that include elements developed by both HPE and Micro Focus. The Micro Focus StormRunner Functional Testing service also includes pre-built integration with Google Analytics, Micro Focus Octane, GIT and Jenkins to streamline the testing process.

    Interest in automated testing has increased substantially in the last few years, says Evans, because developers are under more pressure than ever to roll out new applications and associated updates faster. Unfortunately, most developers don’t have the time required to test applications themselves, or there is simply not enough money in the budget to hire quality assurance teams to test every element of an application. Micro Focus StromRunnner Functional provides an alternative approach using a set of testing tools that have been widely employed across the enterprise for decades, says Evans.

    Evans notes Micro Focus is pursuing a comprehensive approach to application delivery management spanning everything from testing tools to application lifecycle management. Micro Focus is making the contention that most DevOps teams are a lot more interested in getting access to the results of a test than managing the actual testing technologies themselves, says Evans. In addition, advanced analytics and machine learning algorithms, which now are standard elements of testing process, require access to massive amounts of data to be effective. That data is a lot easier to aggregate in the cloud using SaaS applications than in an on-premises environment. In fact, Evans says, those requirements are among the factors driving a wave of recent acquisition across the application testing sector.

    Arguably, the biggest testing challenge organizations now face is moving away from processes based on IT Information Library (ITIL) guidelines to a more DevOps-centric process, under which continuous testing is more embedded within the application development process, Evans says. Functional testing is only one element of the testing process, but it has become more critical as end users become less forgiving of substandard digital experiences inside and out of enterprise IT environments. Regardless of who inside the organization is responsible for testing applications, the expectation now is that every application experience inside the enterprise will be on par with a consumer application experience.

    — Mike Vizard

  • Survey Finds DevOps Teams Under Increasing Pressure

    Survey Finds DevOps Teams Under Increasing Pressure

    The fact that a much larger percentage of companies now recognize they are software companies that happen to specialize in a specific vertical industry is now putting a lot of pressure on developers, according to a new survey of 750 development team leaders in the United States and the UK conducted by Testplant, a provider of tools that automate application testing.

    The survey finds that 42 percent of the respondents claim they are expected to design, develop and test apps in an unrealistic amount of time. More than one-third (36 percent) also report they are not given enough time to ensure apps are properly tested before deployment, while 56 percent note that use of outdated techniques and tools is holding them back from meeting the demands of the digital world.

    Because of this combination of factors, nearly half the respondents (49 percent) say they are deploying applications before they go through an ideal testing amount of testing, and 45 percent admit to putting applications into production that will perform below initial requirements.

    Dr. John Bates, CEO of Testplant, says it’s a matter of time before IT organizations will have to rely more on tools that automate much of the testing process, as pressure on DevOps teams to roll out new applications and update existing ones is only going to increase. The survey respondents (60 percent) say most of the pressure to accelerate software development is being driven from within the organization itself. But nearly half (49 percent) say pressure comes from competitors. As more organizations realize their competitive differentiation comes from the software they develop, there’s now a lot more focus on accelerating application development, says Bates.

    The survey also notes that the applications being built are more complex than ever. Over the next 12 to 18 months, half the respondents (50 percent) say their company will develop more apps using components for an internet of things (IoT) environment. Well over half (58 percent) also say their company will develop more apps with artificial intelligence (AI) components, and 62 percent say their company will increase the amount of applications that incorporate machine and deep learning algorithms.

    When it comes to deploying applications, automation is ranked as the most critical challenge by 67 percent of the respondents, and 70 percent claim they are now more focused on automated testing today than they were in the past.

    In total, 75 percent of the respondents say that with access to better tools, the applications deployed by their team would have a greater impact on the business. A full 81 percent also say that with more time, applications deployed by their team would have a greater impact on the business.

    Unfortunately, more time for application development is luxury most business don’t have today. Not only is a wave of digital business innovation is making it easier for startups to not only enter markets, but companies that traditionally have focused in one market segment are now extending their reach to adjacent areas. The difference between surviving and being made obsolete now comes down the level of customer experience an organization can deliver by the applications it makes available to customers.

    Unfortunately, when it comes to those applications, customers tend to be unforgiving. Once bad application experience and many of them will move on to try an application developed by a rival. Bates notes that too many organizations still seem to think they can test their applications on end users on the assumption they can fix any issue that comes up rapidly. In reality, rather than participate in what amounts to an extended beta testing process, most end users will simply look for a better alternative application experience.

    In short, the difference between success and complete failure now hinges on the level of quality application testing that can be implemented in the shortest amount of time possible.

    — Mike Vizard

  • Tips for Creating Unit Tests for Mobile Apps

    Tips for Creating Unit Tests for Mobile Apps

    Mobile app testing ensures your application is working perfectly without any freezes, bugs or crashes. There are unit, auto, system and acceptance tests, and in this article we will talk about unit testing for mobile apps.

    First, let’s discuss what unit tests are in general. Unit testing checks whether certain parts of the code work correctly. In other words, it helps us produce more robust and maintainable code. Each one small test is dedicated to a small feature. All of them are connected, however, so you get a kind of chain. If any part of this chain fail, the whole chain will fail as well.

    As a part of mobile application testing unit tests may be written at any phase of development or even before development (test-driven development, or TDD, is one of the iOS unit testing best practices). With this kind of development, you let your tests drive the development process. They indicate what you do, what you should do next and what the result should be. Testing controls the design.

    Thus, unit testing is the first step to perfectly working code. But it also is the primary means for your software quality assurance. In this article we’ll explain how to do mobile apps unit testing on an iOS example and give you a small iOS unit testing tutorial.

    Do I Need to Conduct Unit Testing?

    Does everyone need to do unit testing? The short answer is no. The long answer is that it  is a very good tool, though it isn’t necessary in all situations.

    You don’t need unit testing if:

    1. your app is very small and simple. If your app is small, it will be faster to check everything manually because there’s not that much logic to be checked.
    2. you don’t plan your app to work in the long term and you need it only for a demonstration.
    3. you’re a super-human who writes perfect code without any mistakes.

    If you don’t fall into any of these categories, then unit testing is what you need. At first, this kind of testing may seem too time-consuming. Lack of time is mentioned as one of the most serious problems related to testing, but actually performing unit testing can save a lot of time and improve development speed. In addition, it can serve as a kind of documentation for your library. We at Mobindustry spend up to 30 percent of our development time on testing.

    Unit tests have two main purposes:

    1. Reduce the number of bugs by making sure that our class/module/component is working as expected with all possible inputs.
    2. Reduce the number of regressions by making sure that our new functionality, bug fixes or refactoring hasn’t broken any existing functionality.

    How Do Unit Tests Work?

    How to create iOS unit testing? Xcode has built-in support for unit tests and works as an iOS unit test framework. You can use Xcode by pressing cmd+U or via the menu by selecting Product -> Test. xCode will run all active tests and generate a report with results (passed/failed). All failed assertions are indicated. Code coverage data is also generated in this framework, which allows you to get to any class and check which lines of code were called and how many times they were called.

    Types of Unit Tests

    Unit test for iOS can either check the UI or the logic of the application. Logic ones have a few subtypes:

    • Regular ones are used to check straightforward functionality.

    func testExample() {

    let account = Account()

    account.amount = 100.0

     

    account.withdraw(20.0)

     

    XCTAssert(account.amount == 80.0, "Account should have current amount of 80.0 but has

    (account.amount)")

    }

    This is used to simply check if the withdraw() method is working correctly.

    • Asynchronous tests use XCTestExpectation to check for expected outcomes.

    func testExample() {

    let url = URL(string: "http://www.google.com")

    let expectation = XCTestExpectation(description: "expectation_description")

     

    let task = URLSession.shared.dataTask(with: url) { (data, response, error) in

    if let error = error {

    XCTFail("Error: (error.localizedDescription)")

    return

    } else if let statusCode = (response as? HTTPURLResponse)?.statusCode {

    if statusCode == 200 {

    expectation.fulfill()

    } else {

     

    XCTFail("Request failed with status code (statusCode)")

    expectation.fulfill()

    }

    }

    }

    task.resume()

     

    waitForExpectations(timeout: 10) { (error) in

    XCTFail("Expectation (expectation.description) has failed")

    }

    }

     

    This one will check if Google’s website is available (i.e. if it returns status code 200).

    XCTestExpactation is used to wait for the async task to finish. An Expectation will either finish successfully by calling the fulfill() method or will fail if the timeout time has passed but fulfill() hasn’t been called.

    • Measure ones check heavy logic:

     

    func testExample() {

    self.measure {

    let lotOfObjects = []

    for obj in lotOfObjects {

    obj.makeCalculations()

    }

    }

    }

     

    You can create this test to measure how much time and how many resources are spent on particular calculations inside the measure() block.

    Tips on Unit Testing for Mobile Apps

    • Remember the structure

    The common AAA approach to structuring is the most common and convenient way to write tests. AAA stands for:

    1. Arrange – the part that allocates all necessary objects and sets up initial parameters
    2. Act – the part that executes the logic
    3. Assert – the part that checks if the obtained results match the expected results

    Сode written this way is easy to read and use, which means it’s easier to maintain.

    • Cover only necessary things with tests

    Some people think it’s necessary to cover every single part of your code with a unit test. This approach is actually time-consuming, complex and completely useless.

    There are four types of code you may have in your project. Let’s look at them starting with the most extreme options:

    1. Simple code without any dependencies: Probably everything is already clear with this kind of code and there’s no point in assessing it.
    2. Complex code with lots of dependencies: If you have this kind of code, it’s a good idea to refactor. There’s no need to cover this code because we’ll rewrite it, which means new classes will appear and method signatures will change. Why write something that won’t be used? Of course, such code also needs quality assessment, only on a higher level.
    3. Complex code without dependencies: This code contains algorithms or business logic. This is the perfect kind of code for testing because these are very important parts of the system.
    4. Moderately complex code with dependencies: Such code links different components together. Tests are very important here; they tell how exactly components must interact.

    Usually, they cover the following areas of code:

    • Common functionality, core models, classes and their interaction
    • Main UI workflows
    • Crucial logic
    • Bug fixes

    Unit tests may be applied not only for these areas. They can actually be used for testing almost any functionality.

    Don’t cover trivial functions and system functions.

    • Test only one thing at a time

    The idea is to divide your application into different independent modules that are able to work by themselves without interacting with other parts of the application. This allows you to work on each module separately and simplify the code. If you check several things at the same time, sooner or later your code will become overloaded and too complex. It will become difficult to support it and use.

    • Name your tests logically

    It’s very important to name your tests properly and assign them to the correct parts of your project. Many projects have tests that no one is running because of inconvenient placement and naming. Here are some tips for logical structure:

    1. Choose a logical location in your version control system: They have to be part of your version control. Depending on your needs they can be organized differently, but here’s a common recommendation—if your app is monolithic, put all of them in a Tests folder; if it has many components, store relevant tests in each component’s folder.
    1. Choose a convention for naming: One of the best ways to do this is to add a test project to each project you build.
    1. Use the same convention for naming test classes that you use for naming project classes: If you have a class ProblemResolver, it’s a good idea to add a class called ProblemResolverTests. Each class should work only on one thing; otherwise, your tests will be very hard to manage.
    • Work with mocking objects

    You can use different fake objects to make the testing process easier. There’s no need to allocate and manage a huge class or component just to use a single method from it. Such fake objects can be divided into two groups:

    • Stubs
    • Mocks

    Stubs are fake methods for returning a predetermined response. The main use for stubs is when interacting with an API. The idea is not to wait for the actual connection to finish but to return predefined data or a predefined error instantly.

    Let’s assume we have an app where you can add a friend connection. There’s a FriendListViewController that displays a list of all your friends. This friend list can be downloaded from the backend, so there’s an ApiManager class with a method like this:

    ApiManager.shared.getListOfFriends(withCompletion: { (friends) in

    let friendsListVC = FriendListViewController.init(friends: friends)

     

    //Present friendsListVC

    }) { (error) in

    //Display error

     

    }

    This method has a completion handler, and we need to wait till it finishes.

    But we actually don’t need to wait because we don’t need any APIs or even an internet connection. We need to make sure FriendListViewController is working correctly, and all we need for that is some list of friends; it doesn’t matter where they come from. So we’ll introduce an ApiManagerStubs class that will duplicate methods from the actual ApiManager but will not perform any actual API requests.

    For this particular test, we’ll write a method for getting a list of friends:

    func getListOfFriends() -> [Friend] {

    let friend1 = Friend.init(name: "Bob", surname: "Barker", age:34);

    let friend2 = Friend.init(name: "John", surname: "Doe", age:30);

     

    return [friend1, friend2];

    }

    After that we can easily allocate and test FriendListViewController.

    func testExample() {

    let friends = ApiManagerStubs.shared..getListOfFriends()

     

    let friendsListVC = FriendListViewController.init(friends: friends)

     

    //Testing friendsListVC.tableView

    }

    Mocks are fake objects that partially duplicate real objects, allowing you to check if a method was called or a property was set after certain events.

    In the example above, when a friend is deleted we need to clear the chat history for this friend. We have some class like ChatManager for this functionality. But again, for this, we don’t need to actually delete any data or call any APIs; we just need to check if the corresponding callbacks are triggered. So we’ll introduce the mock object ChatManagerMock, which behaves like ChatManager, only instead of deleting data it just remembers which callbacks were triggered.

    class ChatsManagerMock {

    var deleteFriendWasCalled = false

     

    func clearChatHistoryForFriend(_ friend: Friend) {

    deleteFriendWasCalled = true

    }

    }

    Now we can implement our test as follows:

    func testExample() {

    let friends = ApiManagerStubs.shared.getListOfFriends()

     

    let friendsListVC = FriendListViewController.init(friends: friends)

     

    let mock = ChatsManagerMock()

    friendsListVC.chatsManager = mock

     

    //Trigger Delete action for first friend in the list

    friendsListVC.deleteFriendButtonAction(nil, index: 0)

     

    XCTAssert(mock.deleteFriendWasCalled, "deleteFriendWasCalled should have been called")

    }

    Conclusion

    Unit tests are a very powerful tool to help developers avoid bugs and spot them quickly when they do appear. They help you maintain your code, but at the same time they need maintenance as well. The big problem of many projects is tests that no one uses, and to avoid it you need to stick to some rules. Your tests must:

    • be reliable;
    • not depend on the environment they’re run in;
    • be easy to maintain;
    • be easy to read and understand (even a beginner QA has to be able to understand what’s going on);
    • have a standard naming convention; and
    • be launched regularly and be automated.

    Following these basic rules will help you get a working system. Each test will check only one thing. Your tests will be specifications for class method; it will tell all about what input they expect and what other system components expect as the output. There are very few systems like this. Such systems have an actual specification, and there’s usually not very much text in it—only basic features, server schemes and a “getting started” guideline. The life of such projects no longer depends on people; developers may come and go while the system is reliably tested and tells all about itself with the help of its own tests.

    About the Author / Svetlana Cherednichenko

    Svetlana Cherednichenko is a content writer. Currently she is a part of Mobindustry marketing team. She is an active member of the debate movement and she is a winner of two regional tournaments. Now she is in the process of improving her skills in graphic design and learning new frameworks. She graduated from the Dnipropetrovsk National University. Svetlana likes to travel, read about tech news and play video games. Connect with her on LinkedIn.

  • QA/Testing and DevOps: Partners, Not Adversaries

    QA/Testing and DevOps: Partners, Not Adversaries

    Software testing and QA is sometimes given low priority during development. The idea of treating software testers as adversaries by the developers seems old-fashioned and is short-sighted. It’s hard to believe that such an attitude can still exist, and yet it still persists, even in a modern DevOps team. But a DevOps-focused software team benefits when it considers QA/testing as an integral part of developing high-quality applications.

    In a business setting, the hidden costs of buggy or unsecure software are lost productivity, revenue and extra work by the development team (and even extra work by users, if the software isn’t running right). QA/testing helps test for system flaws, vulnerabilities and security issues, so why wouldn’t a DevOps team be supportive?

    QA/testing can help an enterprise avoid big mistakes. In fact, the DevOps Institute notes, “With the advent of DevOps and the movement to break down silos between developers and operations, it becomes critically important that all members of an IT team—regardless of what tools they use, or role they play—understand the essentials of testing.” And, according to Puppet, a standard for automating the delivery and operation of software, high-performing enterprises build quality testing into each stage of their software development process.

    Make QA/Testing a Priority

    Whether your DevOps-focused developers are handling the QA/testing themselves or passing it along to QA/testing specialists, it’s something that must be done. Ideally, developers themselves would handle QA/testing in an integrated DevOps environment, and that’s why some developers see it as an unwelcome chore. It’s not something they’re usually trained in, and they might struggle with it or see it as a burden. That could be the reason why some DevOps teams hit a “software testing bottleneck.” As data from a Vanson Bourne survey commissioned by Appvance shows, 73 percent of 200 enterprise IT decision makers in the United States have adopted DevOps processes, but 57 percent of those adoptees experienced a bottleneck during software testing that impacts how quickly new and updated software can be released.

    On the flip side, the developer’s company may have QA/testing professionals in-house, so the developers are hands-off in that area. Needless to say, the relationship between developer and the QA/testing pro could get a little rocky. Just think about it: The role of the QA/testing specialist is to find problems and point out issues with the developer’s code. Or, perhaps the developer or engineering staff is lagging behind schedule, so the team inevitably needs QA/testing to rush its processes, further creating a tense or adversarial relationship between the two groups.

    At one of my startups a while back, we worked for months on our first software release of a web application for an online marketplace. Late on a Wednesday night, we let the CEO of our company try it out—and the app immediately crashed. “Gee,” I said, “It was working on my computer.” Of course, it really wasn’t. We had synchronization problems, which are common in multi-threaded applications that access the same information for multiple users. And we made the mistake of letting the CEO be the first simultaneous user to try it out (creating a bad impression), instead of making organized QA/testing a priority at our company.

    QA/testing in a DevOps environment can help your team avoid mistakes, so passion for app functionality doesn’t supersede quality and robust bug-free software. In the end, what you need from your team is passion for both functionality and quality/testing. Otherwise, you’ll deliver a faulty product and lose customers or users—unnecessarily—because of a bad first impression by your software.

    Two Approaches for Integrating QA/Testing with DevOps

    When you’re integrating QA/testing into your DevOps environment, there are two different approaches or procedures to ponder. First is the more traditional test of the full system, sometimes called integration testing. The second is test driven development (TDD), which is popular in Agile and Scrum methodologies. The main difference between the two is in how software code is tested. During integration testing, developers review and test the code. Then, for critical facets of the software, a peer review of code is completed. Afterward, testers create scripts and deploy an array of integration and system tests. These could include scalability testing, load testing, stress testing and very importantly, security testing. Code vulnerability, SQL injections and malicious code would be things to think about.

    Under TDD, the order in which testing appears in the cycle is reversed. Each developer first writes “unit tests” and then writes the code. This is done iteratively in such a way that every discrete section of code is automatically tested. When new code is sent to the integration server, the tests go with it. The server itself runs unit tests for all the code that developers deposit in the server. On top of this, testers design system and integration tests to run automatically by the integration server. Finally, testers perform any manual tests that are impossible to automate. In this way, the integrity and quality of the code is constantly guaranteed.

    Academic studies and field experience suggest that TDD increases development time by approximately 15 percent … but hold on. Experience also shows that total cost of ownership of TDD-developed software is lower than software using integration testing only. It’s worth noting that by using TDD and integration testing, a company can deploy software of world-class quality that exhibits less than 0.3 errors per 1,000 lines of code during 18 months of use.

    QA/Testing: A Partner in Your Software’s Success

    If you’re hesitant about integrating QA/testing into your DevOps team and instead tempted to outsource it, you may want to push pause. It’s actually quite challenging if you do development in-house and expect to simply outsource what you think is independent QA with cheap testers. Outsourcing QA/testing and assuming that piece is done is not really outsourcing—it’s staff augmentation at a distance and it usually doesn’t work. You don’t want a “siloed team” to handle a critical component such as QA/testing. Instead, outsource QA/testing and bring your in-house and outsourced teams together around a common DevOps culture and mindset.

    DevOps experts Bob Aiello and Leslie Sachs, in an IBM report, say quality software “… requires not only a shift in behavior, but a dramatic shift in the organizational culture as well.” So whether you outsource or have an in-house squad, your developers and the entire team should take ownership of QA/testing, and remain open to new ways to test code efficiently.

    Ultimately, QA/testing should not be viewed as a time-consuming, disruptive chore by anyone on your team or in your enterprise. Think of it as a partner in your software’s success—not an adversary.

    About the Author / Steve Mezak

    Steve Mezak is founder and CEO of Accelerance, a leader in global software development outsourcing, as well as the author of “Software Without Borders,” co-author of, “Outsource or Else: How a VP of Software Saved His Company,” and a CIO columnist.

     

  • 10 Rules for Writing Automated Tests

    10 Rules for Writing Automated Tests

    Over the recent months many of our customers asked for our guidance and building their test suites properly. Performance, ease of management and scalability were key priorities for them. We decided to share our suggestions with the broader audience.

    We came up with the list while building our own test suite. Since we strongly believe in dogfooding, Testim is tested using Testim. We also learned a lot working with a variety of companies from small startups to large enterprises.

    Rule 1: Prioritize

    Most apps include thousands of scenarios. Testing the functionality and correctness is merely one step of the way to shipping high-quality software and meeting users expectations. There’s performance testing (speed and responsiveness in a normal state or under load/stress), visual testing (checking that 1+1 = 2 is one thing, but many visual bugs can make it a very bad experience), and so on.

    Start by listing the most important user flows, for example login, add to cart, checkout, etc. Order the list by those more vital for user conversion.

    Try to sort by first adding those that must succeed e.g. create account, checkout and payment. You can use your analytics tools (e.g. Google Analytics, Mixpanel, Kissmetrics, Woopra) to find which are the common scenarios. We also found Segment.io to be useful.

    Rule 2: Reduce, Recycle, Reuse

    The next step is to break the user’s scenarios down to simple, single purpose, almost atomic flows. e.g. “Create user,” “Login,”  “Send email,” etc. Create a test for each of these flows. When completed, compare this to the the user stories.

    Naming convention – Defining a good naming convention is an important part. For example, including the component name in the flow name, which can result in flow structure (e.g  “Account.login“ and “Account.logout”).

    Reuse components. Don’t Copy/paste – for non-experienced developers/testers, a copy/paste seems like a reuse. However, the challenge is to update those steps when the flow changes. Even a single step/line that repeats in hundreds of tests is a huge hassle.

    Rule 3: Create Structured, Single-Purpose Tests

    Single purpose tests verify one thing only!

    Single purpose tests are easier to create and maintain, they will also run faster compared to longer and complex ones. Ideally, they should only fail if the feature they test is broken or one of its dependencies (which we’ll try to minimize). If you test several features and the first one fails, you know nothing about the other features, since you haven’t reached them.

    A test should consist of three parts:

    1. Setup – e.g. login
      Those steps gets the AUT (application under test) to the state required for testing. This could vary by the test: sometimes you only require a login. you require a complex structure to pre-exist.
      Note: Premature optimization is not a good practice. Implementing the login flow (user/pass fields) is an easy way to start, and if you reuse the same login in all your tests, it wouldn’t be difficult to change the implementation later on to a quick implementation (e.g. by using cookies), so there’s no need to start with the speed optimization if you know it’s easily fixed.
    2. Actions – the flow you want to test.
      Continuing with our aforementioned “login” example above, unlike the setup steps, which require the login to be completed successfully (don’t care in which way), testing the “login” itself is a different task, and should not be reused.
    3. Validations – validate UI state, validate DB – the result you expect.
      There are different validations.his depends greatly on whether you’re testing data which is being updated from production on a weekly basis, or whether it’s the same Test Data.
      In UI testing, the basic validations are done on elements in the UI.

      • Validate an element is visible.
      • Validate an element comprises some text.
      • Validate an element is in a specific state.
      • Visual Validation – validate the is comprising the expected text, image and has the correct styling (e.g. font, layout).

    Using a predefined dataset has the advantage of making the validations easier, as when we know the exact setup state, you also know the precise expected state after the stimuli steps were completed.

    If the dataset is updated, you get different scenarios which you didn’t expect (as different user use your application differently), making it harder to debug failures (as the scenario might not be reproducible).

    Example: A test for a login could look like:

    • Set initial state:
      • Open AUT
      • Navigate to the signup page
    • Actions:
      • Choose a random username and it to a variable named username
        g. “user” + Date.now() + “@mailinator.com”.
      • Sign up for new account by using username.
        • Set username to username
        • Set random password
        • Click submit
      • Validations:
        • Open mailinator.com
        • Navigate to “inbox”
        • Look for email to username
        • Check that there is one email with the expected link.
        • Click on the link
        • Login with username and pass
        • Validate login was successful

    Note: One of the biggest differences between unit tests and end-to-end tests is the overhead. Opening a browser and logging-in to the AUT used to be extremely slow, pushing testers to perform many tests after reaching the Setup state, saving valuable time. Today, with docker, it’s much faster to boot a browser up, the speed of the browser help developers to write applications that load quickly, minimizing the  overhead of writing end-to-end tests.

    Rule 4: Tests’ Initial State Should Always be Consistent

    Since automated tests always repeat the same set of steps, they should always start from the same initial state. The most common maintenance issues when running automated tests are ensuring the integrity of the initial state. Since automated tests depend on that state, if it is not consistent test results won’t be, either. The initial state is usually derived from user previous actions.

    For example:

    1. Adding an item to a list that already contains it (from previous test run).
    2. Views that behave differently for new users and returning users.
    3. Application redirect automatically if user already logged in.

    Possible solutions:

    • Create new user for each test run
    • Use a dedicated app environment for test automation, as opposed to using your production one.
    • Seed your application with initial data before each suite run, also known as fixtures.
    • Use conditions on steps/group to handle dual cases in test which are hard to set.
    • Eliminate dependencies between tests. The AUT’s state after “Test_1” shouldn’t affect the flow of “Test_2”.

    Rule 5: Compose Complex Tests from Simple Steps

    Complex tests should emulate real user scenarios, you should prefer composing those test from simple tests parts (shared steps) instead of recording the entire scenario directly. Since you already tested all the simple actions in the simple tests, the complex tests should only fail on integration issues (this is where Rule No. 2 comes in handy).

    Make sure to order your suite to run the simple tests first. It will give you indication when something is wrong in the complex tests when running the whole suite.

    Rule 6: Add Validations in Turnover Points

    Validations are usually used at the end of the test to signal the test is in a passed/failed state. It is also best to add them at points of major changes as a checkpoint to stop the test if an action failed. Since validation will run until its timeout reached, like all steps, it is also a mechanism to wait for a long async operations in your app (e.g. fetching data from server, continuous view rendering, etc.) before running the next steps.

    Examples:

    1. On action completion – navigation, form submission, modal display, etc.
    2. After major DOM changes add validations to ensure you are in the new state before trying to play next steps.
    3. Following a client-server communication which change app state – e.g. data submitted successfully message, etc.

    Rule 7: No Sleep to Improve Stability

    Sleep with random duration (what I call magic numbers) is one of the main sources of flaky tests. Although all apps are asynchronous in nature, there are rare cases where a human doesn’t know when the page is ready and perform the next action. The same should be true for automated tests.

    The reason to avoid static sleep is you rarely know the load of machine upon you run the test on. Only in performance testing do you have the machine to yourself (knowing for sure all CPU and memory are for you). Since you probably use virtual machines or Docker you will get timeouts if you set the sleep too slow, or slow tests if you set it too high.

    In most cases you add a wait-for-condition step, which waits just like a human would, or sometimes waits for some internal state (e.g. javascript value).

    At Testim we provide all kinds of built-in wait-fors, ranging from wait for element to be visible to wait for a specific text (you use also variables and regex), wait for some visual change (e.g. color change) or code-based changes.

    Rule 8: Use a Minimum Two Levels of Abstractions

    If your test is composed mostly of user interactions, such as clicks and set-text actions, then you’re probably doing something wrong. If you used low-level frameworks (e.g. Selenium), you might have heard of the PageObject Design Pattern, which recommends to separate the business logic (e.g. login) from the low-level implementation (e.g. set username, set password and click login button). This is true for Testim as well. Having layers contribute to reuse reduces the maintenance and allows other members of the organization to quickly understand what happen, as sometimes the testers write the tests but the developers run them (before merging new code). The developers in this case need to understand quickly what the test does, and what is the bug.

    Rule 9: Reduce the Occurrences of Conditions

    A test should have as few as possible conditions (if-statements). Those that have many are usually unpredictable (don’t know the exact state you’re in) or complex (you might even see loops, too). Try to simplify your tests by:

    • Start your test at a predefined state;
    • Disable random popups; and
    • Disable random A/B testing, and choose a specific flow.

    Rule 10: Write Independent and Isolated Tests

    An important methodology of test authoring is creating self contained, independent flows. This allows to run tests in high parallelism, which is crucial for scaling the test suites. If, for example, you have 1,000 tests that run for a minute each, running them one by one will take more than 16 hours. Minutes, at full concurrency, can cut this testing time to 1 minute.

    Having a fast build reduces the feedback time between the moment that developers commit their code and the moment they knows their feature is production ready and can lead to true agile development cycle.

    In Summary

    Proper planning of your tests can make a big difference in your development process. The ability to get fast feedback, adapt tests to new user flows quickly and quickly analyze the results, identify bugs and get enough information to quickly fix them. Proper planning can also save a lot of time and expenses. Suites that are not properly planned require high maintenance and keep the R&D team away from building new functionality and increasing coverage. Finally proper planning can help expand the responsibility for quality beyond the Q&A organization and let others contribute.

    About the Author / Oren Rubin

    Oren RubinOren Rubin is CEO of testim.io. He has over 20 years of experience in the software industry, building mostly test-related products for developers at IBM, Wix, Cadence, Applitools, and Testim.io. In addition to being a busy entrepreneur, Oren is a community activist and and the co-organizer of the Selenium-Israel meetup and the Israeli Google Developer Group meetup. He has taught at Technion University, and mentored at the Google Launchpad Accelerator. Connect with him on LinkedIn and Twitter.