devopsy.kinsta.cloud

Author: Frank Moyer

4 Mobile Software Testing Use Cases Where Device Clouds Don’t Work

A device cloud, which provides on-demand access to a variety of different types of mobile devices, may seem like the most efficient and effective way to run mobile tests–and it is, in many scenarios.

But depending on which types of mobile tests you need to run or how you need to run them, they aren’t always the best solution. If you blindly turn to device clouds by default to meet your test infrastructure needs for all use cases, you could end up shooting yourself in the foot.

To prove the point, I’d like to walk through four mobile testing use cases where you either fundamentally can’t use a device cloud or where the disadvantages of cloud-based mobile testing outweigh any benefits of the strategy.

What is a Device Cloud?

Before diving into a discussion of mobile testing scenarios where device clouds don’t make sense, let me make clear what I mean by “device cloud.”

I’m referring to a cloud-based mobile testing environment where developers and QA engineers can run software tests on remote devices. In other words, a device cloud is a “cloud” of mobile devices you can access on demand for executing tests. Device clouds are an alternative to running tests on mobile devices that you have to set up locally and manage yourself.

They have become a popular solution for mobile testing because they offer several benefits compared to testing with local devices:

● Testers can easily test applications across hundreds or thousands of different mobile devices without having to purchase and set up each device themselves.
● You typically pay only for the time during which your tests run, making device clouds a more cost-effective solution in many cases than testing on devices that you have to acquire locally.
● Because most providers add new types of mobile devices to their clouds on a regular basis, testing teams always have access to up-to-date mobile hardware.

A variety of vendors (including Kobiton, where I work) offer device clouds that deliver these benefits.

Why Device Clouds Don’t Always Work for Mobile Software Testing

Device clouds are great for use cases where teams need to test mobile apps across many mobile devices and don’t face special requirements. However, depending on your needs and priorities, they may not make sense for your testing routine.

Here are four scenarios where you can’t or shouldn’t use a device cloud for mobile testing.

1. Testing for infrastructure disruptions

Devices hosted in a device cloud operate under best-case conditions. They are fully charged, professionally managed devices that are connected to very stable, high-speed networks.

That’s good in the respect that it means the devices can run tests reliably. But it’s bad for teams who want to understand how their mobile software performs under less-than-ideal circumstances. If you want to know how Bluetooth interference or a spotty cellular connection could impact mobile app performance, for example, you won’t be able to simulate those conditions using a device cloud.

You can simulate these conditions by deploying devices locally in what QA professionals call an isolation chamber, where testers can introduce “chaos” like weak network connections to evaluate how the software responds.

2. Testing mobile apps with strict security requirements

Testing software on a device cloud requires you to upload your applications, along with any data that you need to feed into them for testing purposes, onto mobile devices owned by a third party.

In most cases, this isn’t a big deal. Providers secure their infrastructure and network connections to protect sensitive data.

But if you’re dealing with exceptionally rigid security requirements–as you might in highly regulated industries or in situations where you’re testing apps that manage very sensitive data–a device cloud might create too many security risks. You’re better off in that case testing on-premises, where you have full control over data and software.

3. Minimizing test execution time

Mobile software tests that take place on remote devices hosted in the cloud almost always take longer to complete than tests that run locally. The main reason why is that cloud-based tests require data to move over the Internet, which is slower than local network connections.

To be clear, we’re talking about delays of perhaps only 200 or 300 milliseconds. In most cases, that’s not a big deal. But if you require ultra-fast test execution, which is beneficial in scenarios where you need to run a large number of tests and you don’t want them to delay your software delivery pipeline, cloud-based testing might slow you down too much.

4. Testing on obscure devices

Major device clouds cover most of today’s most popular types of mobile devices, which makes them a great solution if you need to test on devices that are widely used by consumers.

But if you need to ensure that your apps run on more obscure devices–such as specialized IoT devices that you might use in your company’s facilities–you’re less likely to find them in a device cloud. You’ll need to acquire the specialized hardware and run tests on it locally.

Conclusion

Device clouds are a great way to meet garden-variety mobile testing requirements. However, not every mobile testing use case is simple enough to support using a device cloud. Sometimes, tests require a level of security, performance and control that only local mobile test environments can deliver.

October 12, 2023
Turning Software Testers Into 10xers

How do you make technical processes faster and more scalable? If your answer is simply to automate them, you’re stuck in the 2010s. Today, automation is widespread, and automating processes in a basic sense is no longer enough on its own to maximize efficiency and velocity. Instead, teams who want to excel–who want to become 10xers, so to speak–need to look beyond automation in its simplest form. They must take automation to the next level in a way that makes processes not just a little faster but orders of magnitude faster. Allow me to explain by discussing how software testers can put this principle into practice.

The Limitations of Basic Test Automation

When I talk about “automation in its simplest form,” I’m referring to automation that helps to execute core processes.

In the context of software testing, basic automation means using frameworks like Selenium or Appium, which provide a way for engineers to execute tests automatically. That saves a lot of time and effort compared to executing each test manually.

To be sure, there’s nothing wrong with simple automation. But the fact is that by now, most teams have caught on to the value that test automation provides. Automatically executing most of your tests no longer makes you much faster than average. It might have been a big deal a decade ago, back when test automation was less widespread, but it’s hardly enough to make you a 10xer today.

Going Beyond Simple Automated Testing

What does set you apart? What can testers do to supercharge their efficiency and velocity?

The answer, again, is to take automation to the next level. Rather than simply using test automation frameworks to execute tests, they can also use automated tools to generate the tests in the first place.

For example, modern automated testing solutions can automatically create test scripts based on actions that testers perform manually, making it possible to generate tests without having to write any code. That means you can create many more tests in much less time.

As another example, next-generation automation can supercharge the interpretation of test results. If engineers want to know, for instance, when a screen finishes rendering in an application or when a crash occurred, they don’t have to parse test results manually, or write custom scripts, to find that information. They can leverage modern automation tools that collect that data on their own.

In a similar vein, modern test automation tools can accelerate the process of troubleshooting tests that fail to run. For example, imagine that an automated test fails because it is supposed to test an element that developers relabeled in the newest version of an application. Traditionally, fixing that issue would have required engineers to review the test, sort through application code and then update the test with the new element ID. But this process could instead be automated using tools that can identify the application elements that a test should assess and update test scripts with the appropriate element IDs, all without requiring manual intervention.

A Copilot-Inspired Approach to Software Testing

You could draw analogies between the types of opportunities that modern test automation software makes available to testers and those that AI-powered tools like GitHub Copilot have opened up for software developers.

If you’re a software developer today, you don’t have to settle for basic automation tools, like continuous integration servers (which automatically integrate code into larger codebases) or source code management platforms (which automatically track changes to code over time). You can also benefit from a new breed of automation tools, like Copilot, which can automatically generate code, unit tests and much more.

For developers, Copilot is a prime example of a next-generation tool that can dramatically accelerate software development. Developers who learn to use Copilot effectively can work potentially ten times faster than their counterparts.

The types of next-generation test automation solutions I’ve described above do the same thing for software testers. Instead of relying just on basic automation tools to run tests, modern testers have a host of additional, more powerful automation at their disposal to help create, manage, interpret and troubleshoot tests. Those who learn to use those tools will become the 10xers of today’s generation of software testers.

April 14, 2023
5 Unique Challenges of Mobile App Testing

At first glance, testing a mobile app may not seem to be very different from testing a conventional desktop app. Mobile and desktop apps are often written in the same languages and hosted on the same servers. They must also meet the same basic user expectations in areas like loading speed and accessibility.

But when you dive into the details, you realize that mobile apps are fundamentally different beasts from desktop apps–and that, by extension, mobile testing requires a unique approach. You can’t simply take a software testing strategy that works for desktop apps and graft it onto your mobile apps.

This article breaks down five of the main reasons why mobile testing necessitates a different strategy from desktop application testing and which unique needs QA engineers should consider when testing apps.

1. Mobile Configuration Variability

Perhaps the biggest difference between mobile and desktop testing is that with mobile apps, engineers have a much wider set of possible configurations to contend with.

In the desktop world, there are just two main operating systems – Windows and macOS – and a relatively small number of OS versions for each family. And although there are many types of PCs and laptops, they all conform to the same basic hardware standards.

In contrast, there are over 24,000 distinct types of Android mobile devices out there – to say nothing of iOS devices, which add to the diversity of hardware. There is also a wider selection of mobile operating systems and versions.

What this means for QA teams is that there are many more variables to test for with mobile devices. It also means testing must be more efficient so that engineers can test for as many potential configurations as possible without delaying software release cycles.

2. Lack of Mobile Testing Standards

With a traditional, web-based desktop app, there is a consistent set of standards that the app is supposed to comply with when rendering content – specifically, the standards set by the W3C, a consortium that advocates for a standards-based, interoperable World Wide Web.

In the mobile realm, however, there is nothing equivalent to the W3C standards. Apps can render content in widely varying ways, many of which are device-specific.

Here again, this increases the need for specific testing teams to account for more variations and edge cases. With desktop apps, ensuring that the app complies with the W3C standards is often sufficient, but mobile testing is not so simple.

3. Unique Mobile Accessibility Requirements

Accessibility testing, which ensures that accessibility features like the ability to increase text size function properly, is important for delivering a great experience to all users, whether they access apps using desktops or mobile devices.

But with mobile devices, accessibility testing is harder, because there is more room for error when implementing accessibility features. For instance, the smaller screen sizes of devices – and the greater variation in average screen size – could mean that an increase in text size will cause the app to render some text off the screen. Or, a “nighttime mode” feature on a screen could result in lower-than-expected contrast between text and backgrounds, causing accessibility challenges for some users.

A mobile testing strategy needs to be able to accommodate risks like these, which aren’t as pronounced for desktop apps.

4. Mobile Environment Differences

By definition, mobile devices may be used in a wide range of physical settings. Depending on the location from which a user accesses a mobile app, application performance could be impacted by a variety of environmental factors that don’t typically apply for desktop apps.

For instance, limited network connectivity might undercut application performance when users who travel too far from cell towers. Or, energy-saving features on mobile devices that are running on low battery could decrease the speed at which apps render content.

Once again, these factors create additional risks that QA teams need to address when planning testing routines.

5. Higher Stakes for Mobile Testing

It’s a best practice to strive to deliver the best possible experience to all users, whether they are on desktop or not. But the reality is that poor user experience in mobile apps tends to have a more negative impact on a company’s brand.

The reason why is that it’s easy for users to highlight poor app performance by giving apps low ratings or leaving negative comments in marketplaces. They can’t do these things for most desktop apps, because, unlike mobile apps, most desktop apps are not downloaded through centralized marketplaces with user-rating features.

This difference doesn’t make mobile testing more technically challenging, and to be clear, I’m not saying you can skimp on desktop testing because users will abandon your desktop apps if they don’t work as they should, but it does increase the stakes of perfecting a testing strategy. Businesses simply have more to lose, reputationally speaking, from delivering buggy apps than they do from low-quality desktop apps.

Conclusion: The Need for Purpose-Built Testing Solutions

All of the above is why we need more testing strategies and solutions that are designed specifically for mobile.

Historically, QA teams have frequently tried to extend desktop testing strategies to address mobile as they’ve added apps to their catalogs, but that approach simply doesn’t work. Mobile apps and devices are too different in fundamental respects to be shoehorned into a desktop-centric testing routine. The sooner QA engineers realize this, the sooner they’ll optimize their user experiences.

February 1, 2023
4 Best Practices for Entertainment App Testing

Entertainment apps represent a broad category of software, ranging from mobile games, to social media platforms, to streaming video apps and beyond.

However, all entertainment apps share a few key traits in common:

They typically host dynamic content that is unique for each user. This makes entertainment apps different from apps where most or all of what each user sees is the same.

They host rich multimedia content including, in many cases, streaming audio and video.

They are often created by vendors operating in highly competitive markets, where a poor user experience can send users flocking to competitors’ apps. There is no shortage of apps out there for gaming, video streaming and so on, and if your app fails to perform, your users can easily convert to a competing solution.

For these reasons, teams responsible for testing entertainment apps must develop a testing strategy tailored to the unique challenges of these applications. Traditional testing tools and methodologies aren’t enough for running tests that support an excellent end-user experience for entertainment software.

With that reality in mind, let’s take a look at four best practices to follow when testing entertainment apps. These strategies go a long way toward ensuring that entertainment apps deliver the end-user experience that is necessary for maximizing user engagement and retaining users, no matter how complex or dynamic the apps happen to be.

Testing for Video and Audio Syncing

An easy – and all too common – way to shred the user experience in an entertainment app is to stream videos where the visual and audio streams go out of sync with each other. Videos need to be perfectly in sync with audio to keep users happy.

Of course, the challenge that testers face is that most testing frameworks don’t offer an easy means of evaluating whether visual and audio content are in sync with each other, especially when the content is streamed. Frameworks that let you evaluate when content initially loads or when it updates aren’t enough for analyzing sync between visual and audio streams. As a result, teams have traditionally resorted to testing for sync manually, or not testing it at all.

Fortunately, there’s a better way. Using AI, you can run tests that automatically identify lag between video and audio. In turn, you can detect problems that may cause streams to go out of sync, then fix them before your users experience sync issues.

Minimize Test Latency

When you have an entertainment app where content changes in real-time, your tests need to be able to execute and display results in real-time, too. But that’s difficult to do if network latency issues prevent data from moving between your test cloud and your testing tools quickly enough to provide real-time results.

There are two ways to solve this challenge. First, you could run tests on on-premises devices in order to avoid network latency problems. Alternatively, if you want to take advantage of a test cloud while still achieving real-time results, you can optimize network traffic in a way that minimizes latency and allows test data to flow between your test cloud and your testing tools in a few dozen milliseconds–as opposed to the hundreds of milliseconds traditionally required for packets to flow from tools to the device cloud, and then back again.

Test Across Geographies

The way a user based in, say, Barcelona experiences an entertainment app might be quite different from the way one located in New York or Tokyo experiences the same app. The reason why is that the network hops over which data needs to travel, as well as the efficiency of the carrier network that delivers the data, which can have a major impact on how quickly content loads and is streamed. For media-rich entertainment apps, latency and bandwidth limitations can be particularly problematic.

For this reason, you should be testing your entertainment apps from the perspective of users spread across the world. Doing so allows you to represent multiple geographies and carrier networks in your tests, ensuring that your app performs properly not just from the location where your testing team is located, but from all of the locations where your actual users reside.

Test for Multiple Screen Resolutions

Variations in screen resolution can have a significant impact on how content is rendered. Attempting to display content at a screen resolution that software developers didn’t plan for can result in pixelated images, distorted images or images that aren’t fully visible.

That’s why you should be testing how content on your entertainment app is rendered under different screen resolutions. Testing for just one or a handful of resolutions isn’t enough.

You can perform these tests manually, by reviewing content rendering by hand. A better approach, however, is to take advantage of AI tools that can automatically compare renderings of the same content under different screen resolutions to detect inconsistencies. That’s a much faster, more efficient, and more scalable way to ensure your content appears as users expect it to.

Conclusion

Entertainment apps are a unique category of software, and they require a unique testing strategy. Testing teams need to test for factors that aren’t important for most other types of apps, such as evaluating whether video and audio streams are in sync. They also need to minimize latency during testing and test across multiple geographic locations and screen resolutions. Practices like these are the only means of ensuring that entertainment apps deliver the user experience necessary to succeed in highly competitive niches.

November 28, 2022