There are a lot of discussions in the software testing community discussing the benefits and drawbacks to use an instrumented browser approach to run mobile web automation. As I’m working with a number of customers on this exact subject, I thought I would share my opinion on the subject.
I’ve seen solutions that drives Apple UIAutomation (or Android UIAutomator) to perform mobile web automation. Unfortunately, at least for now, they all fall short of expectations. UIAutomation doesn’t provide any precision, so good luck automating mobile web apps such as Google Map. It is also tight to Instruments that can only run one instance per Mac OS. This is not realistic in a Continuous Integration scenario. These solutions require also a lot of coding and personally I’m not a big fan of coding for functional testing. First it takes forever to write the tests but you end up quickly in a maintenance nightmare. I love these type of tests for unit tests, not for functional testing. Finally, these solutions have problem supporting the latest OS and on iOS it’s a big problem since adoption is very, very fast: In less than a week, 50% of people had upgraded to iOS7. We’re at close to 80% today. If the automation framework you’re relying on doesn’t support the new OS from day 1, you’re in big trouble!
Yes, it would be awesome to be able to record tests on native mobile browsers on real devices, be able to perform complex gestures, be able to replay them on other mobile native browsers and add meaningful object based validations. It’s just not possible today due to too many restrictions. Your best bet is still today to use an instrumented browser. I personally didn’t find one customer having problems with this approach.