Appium is not ready for prime time (Part 2)

appiumpartii Appium is not ready for prime time (Part 2)

I decided to write a second part to my “4 reasons why Appium is not ready for prime time” article. There was an interesting comment posted by Satyajit Malugu which I thought required a post by itself because it raises some important questions. Hopefully, some additional comments will keep coming as I’d like to inform as much as possible testers and developers about the current state of Appium.

Here are the 4 comments made by Satyajit

1. No implicit waits – Appium has good support for implicit waits. Ruby_lib is using them and my tests rarely have to wait on something. Let me know I can provide my logs of this works.

I’m not a big fan of implicit waits as they don’t offer flexibility for specific elements you want to wait on. You basically define a global wait-for-element for the entire duration of the test. Typically I woud want to use different type of waits wether I’m dealing with screen-flow for my app or if I need to wait for the return of a back-end call. Implicit waits don’t give me this flexibility. Additionally, I can’t specify the element I want to wait on. Finally, I didn’t see any capabilities to wait for other element’s charactertics such as count, color, enable/disable, etc. Explicit waits are much more powerful and flexible. That’s why we spent a lot of time implementing them in our product and we add new ones almost every week based on the feedback we receive from our community.

TouchTestWaits Appium is not ready for prime time (Part 2)

2. No parallel runs – Appium/sauce labs philosophy is to use emulators in their cloud and the support parallel tests, its in their road map but not there yet

I have 2 problems with this comment:

  • I can’t recommend an open source mobile test automation that rely on a commercial vendor to run tests. I like sauce labs, I like what they’ve done for web browser testing but if I want to use an open source solution, I want the benefit of open-source all the way. So if I’m not using SauceLabs, I’m left with buying 500 Mac Mini if I want to run 500 iOS tests in parallel. Not too sure this is a good investment.
  • Sauce Labs executes tests on simulators. So if my tests pass on a simulator, am I good to go? Can I ignore testing on real-device? Of course not and I still don’t know why testers still run their regression tests on simulator. I understand why developers use them: It’s cheap, it’s integrated within their IDE, it’s great to debug their apps. But testers? Let’s get serious. Simulators are not the right platform to run tests in my opinion: Way too many false-positive (and negative!), they come with stock OS which is a problem on Android where there are plenty of OEM customizations, network connectivity comes from the client (PC or MAC) and is very different from what you get on a real devices and finally the CPU/Memory they simulate is not fit to run any serious performance testing.

3. Gestures – there is support for gestures, not in the most easiest of the ways but I am able to flick, swipe, long tap etc. Now tell me which other framework provides you even that?

Mobile apps rely heavily on gestures and support for ALL OF THEM in a mobile test automation product is critical. It shouldn’t require any coding and should be easy to use. Appium is not there yet and somehow I find it perfectly normal. I’ve seen the SOASTA engineering team working endless hours to support ALL OF THEM across native, hybrid and Mobile Web apps on both iOS and Android. It’s a LOT OF WORK but this is the price to pay to offer ease of use and full support to testers and developers on all platforms. We today support 60+ standard gestures on both iOS and Android, so yes it is possible.

TouchTestGestures Appium is not ready for prime time (Part 2)

4. Limited support for android < 4.1. Now is it appium’s fault that google decided to come up with new framework 17+? Also there are couple of projects underway to merge selendroid into appium, or using espresso. If you are motivated enough, you could create implement a page object design pattern and use selendroid and appium in the same test suite.

When you build an automation framework, you look at your market and you’re trying to support the majority of your users. Today, Appium supports 18.2% of the market. If I’m a developer or a tester, is this sufficient? Can I wait for Appium to come up with support for < Android 4.1? Or should they not start their automation project until carriers and manufacturers decide to upgrade the phones of their users? Should they mess with Selendroid and read pages of support forums to make their solution work for Android < 4.1? Appium decided to invest in the future and that’s fine. But it means that testers will use the solution … in the future.

Keep the comments coming!

State of Testing Survey 2013

There is a cool initiative going on from the folks at Tea Time With Testers and QA Intelligence. They’re launching a survey to help us understand the state of testing! What are the main challenges in the profession? Where is the profession headed? I’m definitely supporting the initiative and interested in the results! I’m following the software testing trends via analysts but a survey coming from the community could be quite insightful!

You can take the survey here.

Note: I have no affiliation with the survey.

4 reasons why Appium is not ready for prime time

appium 4 reasons why Appium is not ready for prime time

As someone responsible for a mobile automation product, I spend a reasonable amount of time looking at the competition. I particularly like when I see some innovation, something that forces us to do even better and stay ahead of everyone else. Usually, this innovation comes from small company or from the open source community. Appium has been on my radar ever since it came out, but I waited for Appium to get some maturity before trying it out. My conclusion is very simple: Appium is not ready for prime time. It reminds me of Selenium and webDriver in its infancy. Not something you want to invest in for serious work. Maybe fine for individual developers to build some smoke tests, but if you have an enterprise app for example, with complex gestures and a lot of tests to build and maintain, Appium is not a good candidate, yet.

These are the main shortcomings I’ve noted. Some are really showstoppers for me.

#1 – No support for intelligent waits. 

notimedelays.jpg 4 reasons why Appium is not ready for prime time

This is the biggest shortcoming and a big red flag. You can code some time delays (polling loops) and that’s it. A time delay is the WORST mechanism to manage your page flow or address back-end response time variability. Reasons being:

  • How long should I wait after a back-end call? Yeah, let’s put 10 seconds, it should be way enough. It might work for 90% of the time and you end up spending time investigating errors for 10% of your test cases. The goal of automation is to find regression issues in your app! Not problems with your environment.
  • How long should I wait to account for device performance? Transition between pages depends a lot on the performance of your device. Might be almost instantaneous on the latest ipad Air for example with their new CPU and could be crawling on older iOS devices. So should you add 2 secs each time you transition between screens in your app? Should be enough right? Sure, it should be fine. Multiply these 2 seconds by your number of transitions, number of tests, number of devices and you end up with a lot of hours spent WAITING! When you have 10000 tests to run during the night for example, that’s not something you can afford. Especially when developers need feedback FAST!

Without being able to wait on a UI element characteristics (visibility, value, etc.) to manage page flow or back-end call, an automation framework for mobile is pretty much worthless in my book.

#2 – On iOS, you can only execute one test at a time per Mac.

This is a limitation coming from Apple Instruments that Appium uses for execution. I’m working with customers running 5000+ tests per night, on 30+ devices, all in parallel. It would take them many days to run all their tests sequentially. This is a SERIOUS limitation and I don’t know how Appium is going to solve this since they have no control over Apple Instruments. You can always buy 1000 Mac Minis! Yeah right.

#3 – Limited support for gestures.

If your app only use simple gestures, like tap, you could be fine. But this is 2013 and gestures are the name of the game to improve the user experience (double-tap and pan for Google map for example. It allows you to zoom with your thumb for example. Pretty neat!). Appium allows you to write Javascript wrappers to support these gestures. But c’mon .. Do you really want to spend time writing support for gestures or would you rather write new tests for your app?

#4 – Limited support for Android < 4.1

androidversion 4 reasons why Appium is not ready for prime time

Appium only supports API Level >=17. If you look at the number of devices running the different versions of Android (via the Android Dashboard) you realize that Appium can only support a very limited share of the market. Do you want to invest seriously in a framework that only allows you to support 18.2% of your market? I don’t think so. I thought it was pretty crazy to have such limited support and I dug a bit. Apparently, you can support Android < 4.1 by using a Selendroid library but it’s not for the faint of heart by judging by the number of people struggling to make it work.

These are my 4 reasons why I think Appium is not ready for serious work. They’re showstoppers in my book. I’m also not a big fan of the approach Appium is taking, which is fairly similar to Selenium/Webdriver. If you like to write tons of code for your functional tests, be my guest. Of all the WebDriver implementations I’ve seen in my career, a lot required a big investment, especially on the maintenance side. But I haven’t seen all projects in the world.

Appium is young and it might grow to something fit for mobile. And honestly, I hope it will! I’m a big fan of open source and the innovation it brings. But mobile is a big business and the product you pick to test your app is an important choice and investment, and you want to make sure it covers all your requirements. I just don’t think Appium is there yet.

Taking Mobile Test Automation on the Road!

topbestpracticesformobiletestautomation Taking Mobile Test Automation on the Road!

Spring is right around the corner and it’s going to be fairly busy for me! I’m very exciting to take my talk arond the United States. If you’re getting started with mobile test automation, this is a great talk to attend. I’ve worked with dozens of customers to help them avoid typical pitfalls and get them on the right track. Mobile test automation is an exciting journey that will make a BIG difference for the quality of your mobile apps. But it is a true software project by itself that needs careful planning, dedication and some best practices to follow. This is what my talk is all about!

If you attend “Top Best Practices for Successful Mobile Test Automation“, you will learn:

  • How to set the right goals and expectations for your automation project
  • How to track success and ROI
  • How to automate your mobile automation
  • When to start automating mobile tests
  • What mobile tests to automate
  • Where to run mobile tests
  • How to build testability in your mobile app
  • How to quickly increase your mobile test coverage
  • How to best manage mobile flows in your test
  • How to leverage your functional tests in your performance tests

This is my current engagements for the begining of the year. If you hear of any other great software testing conferences to attend, let me know!

Why use an instrumented browser for Mobile Web Test Automation?

amazon Why use an instrumented browser for Mobile Web Test Automation?

There are a lot of discussions in the software testing community discussing the benefits and drawbacks to use an instrumented browser approach to run mobile web automation. As I’m working with a number of customers on this exact subject, I thought I would share my opinion on the subject.

If you want support for precision, complex gestures (multi-touch) while being able to access DOM elements as objects, there are no best way today than to use an instrumented browser. Continue reading

Join me at the Jenkins User Conference October 23rd 2013

jenkins Join me at the Jenkins User Conference October 23rd 2013

It’s been some VERY busy 6 months for me getting back to the United States but things are settling down nicely! So expect more updates from me on this blog! Aren’t you some lucky readers? 🙂

At SOASTA, we’ve always been big fan of CloudBees and this is why last year we had partnered with them to offer mobile developers and testers a way to continuously build and test mobile apps.

One of our goal is to compress the build-test-feedback process to a minimum. We want to provide developers a continuous feedback loop so they can assess the quality of their code as soon as they’re done writing it. Traditionally this feedback is given to them within days by traditional testing organization using outdated products. With Jenkins and TouchTest, we want them to give them this feedback in minutes while extending the scope of their tests.

Our plugin makes it easier for them to setup this feedback loop and adds new functionalities to build a more robust regression test environment.

  • Ability to play a composition, make the App Touchtestable automatically, Install App on device and capture screenshots.
  • Reboot iOS devices
  • This is a testing best practice that allow to run a set of tests under a “clean” state. After a while, a mobile device can become bloated and unstable. Being able to automatically reboot devices reduce the number of false-negatives and developers can spend time fixing real problems. This capability is unique on the market today for iOS (This capability comes free on Android)
  • Wake up iOS device
  • This functionality can be used to make the test environment more robust by waking up any iOS devices and put them in a “ready-to-test” state. A more robust test environment allows developers and testers to focus on more important activities ie. Building new tests!

Stockfish SVN Demo This Config Jenkins Join me at the Jenkins User Conference October 23rd 2013

You can learn how to get started with the plugin here.

Next week, I will be in Palo Alto at the Jenkins User Conference where I hope we can meet you so I can show you the great stuffs we’re cooking for developers. I especially want to demonstrate some end-to-end scenario I’ve been working on allowing to combine, via Jenkins, performance tests and functional tests of mobile apps. A lot of customers I’m working with want to understand the impact of back-end load on the performance of their mobile app front-end. Being able to do these type of tests continuously in order to find performance problems early in the development cycle, really help flush out the type of bottlenecks you don’t want to find when you run your 1M+ load test later on. The sooner you find these problems, the easier and cheaper it is to fix them! 

See you there!


mobileperformance Join me at the Jenkins User Conference October 23rd 2013




There, and back again!

sfo There, and back again!

A quick personal note to let you know that after 6 great years spent in Europe, my family and I are moving back to the Bay Area! This is the third time I’m packing my bags for the West Coast! Back in 1989, I spent one year in Seattle (Go Seahawks!!) at the university. I’ve spent 8 years between 1998 and 2006, working for IBM. I’m now heading back for god knows how long to work more closely with my colleague at SOASTA. I guess there is something special about this country … 🙂


It’s time to get real! Introducing mPulse

soastampulse Its time to get real! Introducing mPulse

I can’t resist to show you the latest video we’ve sent to our customers on Valentine’s day. This is real footage of our latest product mPulse, born soon after the logNormal acquisition only a few months ago. I’m so amazed by the work done by our engineering team in such a short time! In a nutshell, mPulse redefines the Real User Measurement space, allowing customers to understand in real-time how their web and mobile apps are behaving in the field. To help deal with bad user experience of mobile and web apps, mPulse comes to the rescue with Big Data, Real-time Processing, Actionable Intelligence and amazing 3D visualization!!

Get plenty of info and FREE RUM!

Enjoy the show!