Appium is not ready for prime time (Part 2)

appiumpartii Appium is not ready for prime time (Part 2)

I decided to write a second part to my “4 reasons why Appium is not ready for prime time” article. There was an interesting comment posted by Satyajit Malugu which I thought required a post by itself because it raises some important questions. Hopefully, some additional comments will keep coming as I’d like to inform as much as possible testers and developers about the current state of Appium.

Here are the 4 comments made by Satyajit

1. No implicit waits – Appium has good support for implicit waits. Ruby_lib is using them and my tests rarely have to wait on something. Let me know I can provide my logs of this works.

I’m not a big fan of implicit waits as they don’t offer flexibility for specific elements you want to wait on. You basically define a global wait-for-element for the entire duration of the test. Typically I woud want to use different type of waits wether I’m dealing with screen-flow for my app or if I need to wait for the return of a back-end call. Implicit waits don’t give me this flexibility. Additionally, I can’t specify the element I want to wait on. Finally, I didn’t see any capabilities to wait for other element’s charactertics such as count, color, enable/disable, etc. Explicit waits are much more powerful and flexible. That’s why we spent a lot of time implementing them in our product and we add new ones almost every week based on the feedback we receive from our community.

TouchTestWaits Appium is not ready for prime time (Part 2)

2. No parallel runs – Appium/sauce labs philosophy is to use emulators in their cloud and the support parallel tests, its in their road map but not there yet

I have 2 problems with this comment:

  • I can’t recommend an open source mobile test automation that rely on a commercial vendor to run tests. I like sauce labs, I like what they’ve done for web browser testing but if I want to use an open source solution, I want the benefit of open-source all the way. So if I’m not using SauceLabs, I’m left with buying 500 Mac Mini if I want to run 500 iOS tests in parallel. Not too sure this is a good investment.
  • Sauce Labs executes tests on simulators. So if my tests pass on a simulator, am I good to go? Can I ignore testing on real-device? Of course not and I still don’t know why testers still run their regression tests on simulator. I understand why developers use them: It’s cheap, it’s integrated within their IDE, it’s great to debug their apps. But testers? Let’s get serious. Simulators are not the right platform to run tests in my opinion: Way too many false-positive (and negative!), they come with stock OS which is a problem on Android where there are plenty of OEM customizations, network connectivity comes from the client (PC or MAC) and is very different from what you get on a real devices and finally the CPU/Memory they simulate is not fit to run any serious performance testing.

3. Gestures – there is support for gestures, not in the most easiest of the ways but I am able to flick, swipe, long tap etc. Now tell me which other framework provides you even that?

Mobile apps rely heavily on gestures and support for ALL OF THEM in a mobile test automation product is critical. It shouldn’t require any coding and should be easy to use. Appium is not there yet and somehow I find it perfectly normal. I’ve seen the SOASTA engineering team working endless hours to support ALL OF THEM across native, hybrid and Mobile Web apps on both iOS and Android. It’s a LOT OF WORK but this is the price to pay to offer ease of use and full support to testers and developers on all platforms. We today support 60+ standard gestures on both iOS and Android, so yes it is possible.

TouchTestGestures Appium is not ready for prime time (Part 2)

4. Limited support for android < 4.1. Now is it appium’s fault that google decided to come up with new framework 17+? Also there are couple of projects underway to merge selendroid into appium, or using espresso. If you are motivated enough, you could create implement a page object design pattern and use selendroid and appium in the same test suite.

When you build an automation framework, you look at your market and you’re trying to support the majority of your users. Today, Appium supports 18.2% of the market. If I’m a developer or a tester, is this sufficient? Can I wait for Appium to come up with support for < Android 4.1? Or should they not start their automation project until carriers and manufacturers decide to upgrade the phones of their users? Should they mess with Selendroid and read pages of support forums to make their solution work for Android < 4.1? Appium decided to invest in the future and that’s fine. But it means that testers will use the solution … in the future.

Keep the comments coming!

10 thoughts on “Appium is not ready for prime time (Part 2)

  1. Yeah but as someone who has used both your product and Appium I can truthfully say that while it might not be there yet, it’s definitely getting there, and it doesn’t have the completely outrageous price tag that SOASTA has.

    • Ha, the great “too expensive” argument! :-) First of all, TouchTest is one of the least expensive commercial product out there. Of course, it doesn’t beat “free” when it comes to price (there is however a free version that you can use forever http://www.soasta.com/free/). But what good is a “free” product when it doesn’t deliver on the requirements? Would you not rather pay for something that does EVERYTHING you need for your testing TODAY. Or do you want to invest time on a product that can’t deliver on all of your requirement, not knowing that he will ever get there? There is no “free” lunch in my opinion.

  2. what alternatives do you suggest to appium? I’m primarly concerned on running tests concurrently on many devices and having good support for android too.

  3. Explicit waits: When you write test script with Appium, the script is a sequence of statements, one of those statements can be the implicit wait for a UI element. I am not sure how to use explicit wait and how it improve the test script. Please provide some test scenarios to illustrate the benefits of Explicit wait.

    Gestures: I agree the support is limited but the list of gestures you posted above, are they not just a implementation using the limited gesture functions Appium is supporting ?

    No parallel runs: No comments.

    Limited support for android = 4.1 but selendroid supports any Android version (also android < 4.1).

    • Hi Chi,

      My understanding is that Appium works pretty much like webDriver when it comes to waits meaning that implicit waits apply to all elements in your script. Basically you setup a internal timeout that will be used for all element searches. Explicit waits are more intelligent and can be applied to a particular web element. This is particulary useful as I don’t wait all waits to be the same, especially when dealing with back-end calls. Does it make sense?

      For gestures, I don’t quite get your questions. With appium, if I want to support all standard gestures, I will have to write some code myself to support it as the support for gestures in Appium is limited (it will probably grow!). I’m not a big fan of writing too much code to support my requirement. I would rather spend time writing new tests ;-)

      As for Selendroid, I understand that this is a way of having support for Android < 4.1. I’ve taken a look at it and I’ve seen a lot of people struggling to make it work. Test automation is hard, no doubt. But I shouldn’t have to spend extra time to support my requirement. Especially when it comes to OS.

  4. Type error:

    Limited support for android less than 4.1: This statement is not true, since Appium has two implementations for Android OS, namely uiautomator and selendroid. UIAutomator requires 4.1 or higher but selendroid supports any Android version (also android less than 4.1).

  5. With Appium, there’s a second reason why one would have to use Selendroid: testing a hybrid app. A test that works on a native Android app sends one kind of JS to do mobile gestures, but has to send another kind of JS (for selendroid) to do the same gestures if a developer makes the app a hybrid… Which means someone on the testing side has to accommodate this in their test framework architecture.

  6. One huge benefit for Appium is that it doesn’t require modification to the source code of our app. SOASTA requires developer to recompile the project with their testable utility, means that we are not testing the same app that are going to be deployed to the market. From developer perspective, it may not be a big deal. But for QA point of view, they have concern.

    I agree that Appium is still very young and has lots of bugs. I hope that Appium could become better and stable very soon. I’m also keeping my eyes open for the upcoming Selenium 3 which will have mobile support.

    Last but not least, one of the feature i really like in SOASTA is the support of multi-touch, which is not being supported by many commercial tools.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>