Improving Mobile Testing with Device Access

Posted Jul 10th, 2017

Setting up and maintaining your own test infrastructure and device lab in-house is getting harder and harder to do. That’s why using an on-demand testing service for the bulk of your testing is the way to go for DevOps teams.

Why, you ask? Well, let us count the ways. Here’s a look at the main reasons why you’ll derive the greatest value by using an on-demand testing service.

  1. You run a lot of test cases
    Gone are the days when testing was done just before release, and only a few test cases were enough to approve deployment. Testing is now earlier in the development lifecycle. As part of the continuous integration (CI) process, testing has become more frequent. Netflix estimates they run a total of 20,000 tests per day on their applications.

    If you run very few test cases per day (enough to count them on your fingers), your QA process has a long way to go to mature. On the other hand, if you run a lot of tests daily, you’ll want all your attention and time devoted to creating, tweaking, and analyzing these tests. Time spent on configuring and maintaining test infrastructure is time taken from actually running tests. To support this frantic pace of testing, an on-demand testing service is the best solution.

  2. You need to scale fast
    Scaling your in-house test infrastructure means adding more VMs. You typically overprovision to ensure you don’t run out of resources in the middle of a test, and end up paying for unused resources.

    For your device lab, buying a new device, setting it up and running your first test on it can take days to weeks. With an on-demand testing service, all it takes is a couple of clicks, and you’re all set to run your first test in minutes. That can’t be beat.

    Speed is often the most underrated benefit of a cloud testing lab. With the effortless provisioning of additional test servers, and instant availability of real testing devices, an on-demand testing service is the way to go for DevOps teams.

  3. You need a wide range of devices
    It’s not just the number of test cases that are growing exponentially, but the number of mobile devices, too. When testing mobile apps, simulators can go only so far. They’re good at spotting errors, but aren’t of much help when troubleshooting performance issues. What you need is a real device to get to the root cause of the bottleneck.

    Facebook recently disclosed that their device lab has 2,000 devices, and they plan to upgrade it to 4,000 devices soon. These devices play a key role in refining user experience, as apps once released can’t be rolled back. This means the stakes are higher with mobile apps, and all bugs should be resolved before release.

    While Facebook is one of a handful of organizations with such a big device lab, most companies make do with 3-5 devices. However, with a real device testing service, any company can have access to a device lab as large as Facebook’s at a fraction of the cost.

  4. Device management becomes hard
    The hardest part of maintaining a device lab is preparing the devices for every test run. Devices need to be rebooted after every test, and if necessary, updated and flashed. You can automate some steps of this process, but the bulk of the process needs to be done manually.

    With a device testing service, you can define how you’d like your device to be configured for each test, and it’s all done for you automatically.

  5. Your teams are located globally
    An in-house device lab is not necessarily accessible to everyone in the organization. For example, Facebook’s device lab is only accessible to developers in that location, while other global offices need to make do with fewer devices or simulators. This is not ideal for a globally distributed DevOps team. On the other hand, if two developers sitting halfway across the world are able to identify an issue affecting a particular device type, and troubleshoot it together using the same device testing platform based in the cloud, that’s a big win for collaboration.

As your testing evolves, you need a capable testing service that can scale along with your testing efforts. This on-demand service should feature a robust test automation framework, and a well-equipped real device lab. It should provide advanced automation based on Selenium and let you perform manual tests whenever required. As mobile apps pose new challenges to DevOps teams, a mobile testing service that is on-demand and fully equipped will make all the difference between a high quality app and a buggy app.

P.S. Sign up for an upcoming webinar where you can learn how to use your Selenium skills for mobile testing with Appium. The webinar will take place on July 13th, 2017. Register today.

Twain Taylor began his career at Google, where, among other things, he was involved in technical support for the AdWords team. His work involved reviewing stack traces, and resolving issues affecting both customers and the Support team, and handling escalations. Later, he built branded social media applications, and automation scripts to help startups better manage their marketing operations. Today, as a technology journalist he helps IT magazines, and startups change the way teams build and ship applications.

Written by

Twain Taylor


Mobile TestingDevOps