Appium is not ready for prime time (Part 2)

appiumpartii Appium is not ready for prime time (Part 2)

I decided to write a second part to my “4 reasons why Appium is not ready for prime time” article. There was an interesting comment posted by Satyajit Malugu which I thought required a post by itself because it raises some important questions. Hopefully, some additional comments will keep coming as I’d like to inform as much as possible testers and developers about the current state of Appium.

Here are the 4 comments made by Satyajit

1. No implicit waits – Appium has good support for implicit waits. Ruby_lib is using them and my tests rarely have to wait on something. Let me know I can provide my logs of this works.

I’m not a big fan of implicit waits as they don’t offer flexibility for specific elements you want to wait on. You basically define a global wait-for-element for the entire duration of the test. Typically I woud want to use different type of waits wether I’m dealing with screen-flow for my app or if I need to wait for the return of a back-end call. Implicit waits don’t give me this flexibility. Additionally, I can’t specify the element I want to wait on. Finally, I didn’t see any capabilities to wait for other element’s charactertics such as count, color, enable/disable, etc. Explicit waits are much more powerful and flexible. That’s why we spent a lot of time implementing them in our product and we add new ones almost every week based on the feedback we receive from our community.

TouchTestWaits Appium is not ready for prime time (Part 2)

2. No parallel runs – Appium/sauce labs philosophy is to use emulators in their cloud and the support parallel tests, its in their road map but not there yet

I have 2 problems with this comment:

  • I can’t recommend an open source mobile test automation that rely on a commercial vendor to run tests. I like sauce labs, I like what they’ve done for web browser testing but if I want to use an open source solution, I want the benefit of open-source all the way. So if I’m not using SauceLabs, I’m left with buying 500 Mac Mini if I want to run 500 iOS tests in parallel. Not too sure this is a good investment.
  • Sauce Labs executes tests on simulators. So if my tests pass on a simulator, am I good to go? Can I ignore testing on real-device? Of course not and I still don’t know why testers still run their regression tests on simulator. I understand why developers use them: It’s cheap, it’s integrated within their IDE, it’s great to debug their apps. But testers? Let’s get serious. Simulators are not the right platform to run tests in my opinion: Way too many false-positive (and negative!), they come with stock OS which is a problem on Android where there are plenty of OEM customizations, network connectivity comes from the client (PC or MAC) and is very different from what you get on a real devices and finally the CPU/Memory they simulate is not fit to run any serious performance testing.

3. Gestures – there is support for gestures, not in the most easiest of the ways but I am able to flick, swipe, long tap etc. Now tell me which other framework provides you even that?

Mobile apps rely heavily on gestures and support for ALL OF THEM in a mobile test automation product is critical. It shouldn’t require any coding and should be easy to use. Appium is not there yet and somehow I find it perfectly normal. I’ve seen the SOASTA engineering team working endless hours to support ALL OF THEM across native, hybrid and Mobile Web apps on both iOS and Android. It’s a LOT OF WORK but this is the price to pay to offer ease of use and full support to testers and developers on all platforms. We today support 60+ standard gestures on both iOS and Android, so yes it is possible.

TouchTestGestures Appium is not ready for prime time (Part 2)

4. Limited support for android < 4.1. Now is it appium’s fault that google decided to come up with new framework 17+? Also there are couple of projects underway to merge selendroid into appium, or using espresso. If you are motivated enough, you could create implement a page object design pattern and use selendroid and appium in the same test suite.

When you build an automation framework, you look at your market and you’re trying to support the majority of your users. Today, Appium supports 18.2% of the market. If I’m a developer or a tester, is this sufficient? Can I wait for Appium to come up with support for < Android 4.1? Or should they not start their automation project until carriers and manufacturers decide to upgrade the phones of their users? Should they mess with Selendroid and read pages of support forums to make their solution work for Android < 4.1? Appium decided to invest in the future and that’s fine. But it means that testers will use the solution … in the future.

Keep the comments coming!

There, and back again!

sfo There, and back again!

A quick personal note to let you know that after 6 great years spent in Europe, my family and I are moving back to the Bay Area! This is the third time I’m packing my bags for the West Coast! Back in 1989, I spent one year in Seattle (Go Seahawks!!) at the university. I’ve spent 8 years between 1998 and 2006, working for IBM. I’m now heading back for god knows how long to work more closely with my colleague at SOASTA. I guess there is something special about this country … icon smile There, and back again!