4 reasons why Appium is not ready for prime time

appium 4 reasons why Appium is not ready for prime time

As someone responsible for a mobile automation product, I spend a reasonable amount of time looking at the competition. I particularly like when I see some innovation, something that forces us to do even better and stay ahead of everyone else. Usually, this innovation comes from small company or from the open source community. Appium has been on my radar ever since it came out, but I waited for Appium to get some maturity before trying it out. My conclusion is very simple: Appium is not ready for prime time. It reminds me of Selenium and webDriver in its infancy. Not something you want to invest in for serious work. Maybe fine for individual developers to build some smoke tests, but if you have an enterprise app for example, with complex gestures and a lot of tests to build and maintain, Appium is not a good candidate, yet.

These are the main shortcomings I’ve noted. Some are really showstoppers for me.

#1 – No support for intelligent waits. 

notimedelays.jpg 4 reasons why Appium is not ready for prime time

This is the biggest shortcoming and a big red flag. You can code some time delays (polling loops) and that’s it. A time delay is the WORST mechanism to manage your page flow or address back-end response time variability. Reasons being:

  • How long should I wait after a back-end call? Yeah, let’s put 10 seconds, it should be way enough. It might work for 90% of the time and you end up spending time investigating errors for 10% of your test cases. The goal of automation is to find regression issues in your app! Not problems with your environment.
  • How long should I wait to account for device performance? Transition between pages depends a lot on the performance of your device. Might be almost instantaneous on the latest ipad Air for example with their new CPU and could be crawling on older iOS devices. So should you add 2 secs each time you transition between screens in your app? Should be enough right? Sure, it should be fine. Multiply these 2 seconds by your number of transitions, number of tests, number of devices and you end up with a lot of hours spent WAITING! When you have 10000 tests to run during the night for example, that’s not something you can afford. Especially when developers need feedback FAST!

Without being able to wait on a UI element characteristics (visibility, value, etc.) to manage page flow or back-end call, an automation framework for mobile is pretty much worthless in my book.

#2 – On iOS, you can only execute one test at a time per Mac.

This is a limitation coming from Apple Instruments that Appium uses for execution. I’m working with customers running 5000+ tests per night, on 30+ devices, all in parallel. It would take them many days to run all their tests sequentially. This is a SERIOUS limitation and I don’t know how Appium is going to solve this since they have no control over Apple Instruments. You can always buy 1000 Mac Minis! Yeah right.

#3 – Limited support for gestures.

If your app only use simple gestures, like tap, you could be fine. But this is 2013 and gestures are the name of the game to improve the user experience (double-tap and pan for Google map for example. It allows you to zoom with your thumb for example. Pretty neat!). Appium allows you to write Javascript wrappers to support these gestures. But c’mon .. Do you really want to spend time writing support for gestures or would you rather write new tests for your app?

#4 – Limited support for Android < 4.1

androidversion 4 reasons why Appium is not ready for prime time

Appium only supports API Level >=17. If you look at the number of devices running the different versions of Android (via the Android Dashboard) you realize that Appium can only support a very limited share of the market. Do you want to invest seriously in a framework that only allows you to support 18.2% of your market? I don’t think so. I thought it was pretty crazy to have such limited support and I dug a bit. Apparently, you can support Android < 4.1 by using a Selendroid library but it’s not for the faint of heart by judging by the number of people struggling to make it work.

These are my 4 reasons why I think Appium is not ready for serious work. They’re showstoppers in my book. I’m also not a big fan of the approach Appium is taking, which is fairly similar to Selenium/Webdriver. If you like to write tons of code for your functional tests, be my guest. Of all the WebDriver implementations I’ve seen in my career, a lot required a big investment, especially on the maintenance side. But I haven’t seen all projects in the world.

Appium is young and it might grow to something fit for mobile. And honestly, I hope it will! I’m a big fan of open source and the innovation it brings. But mobile is a big business and the product you pick to test your app is an important choice and investment, and you want to make sure it covers all your requirements. I just don’t think Appium is there yet.

Taking Mobile Test Automation on the Road!

topbestpracticesformobiletestautomation Taking Mobile Test Automation on the Road!

Spring is right around the corner and it’s going to be fairly busy for me! I’m very exciting to take my talk arond the United States. If you’re getting started with mobile test automation, this is a great talk to attend. I’ve worked with dozens of customers to help them avoid typical pitfalls and get them on the right track. Mobile test automation is an exciting journey that will make a BIG difference for the quality of your mobile apps. But it is a true software project by itself that needs careful planning, dedication and some best practices to follow. This is what my talk is all about!

If you attend “Top Best Practices for Successful Mobile Test Automation“, you will learn:

  • How to set the right goals and expectations for your automation project
  • How to track success and ROI
  • How to automate your mobile automation
  • When to start automating mobile tests
  • What mobile tests to automate
  • Where to run mobile tests
  • How to build testability in your mobile app
  • How to quickly increase your mobile test coverage
  • How to best manage mobile flows in your test
  • How to leverage your functional tests in your performance tests

This is my current engagements for the begining of the year. If you hear of any other great software testing conferences to attend, let me know!

Why use an instrumented browser for Mobile Web Test Automation?

amazon Why use an instrumented browser for Mobile Web Test Automation?

There are a lot of discussions in the software testing community discussing the benefits and drawbacks to use an instrumented browser approach to run mobile web automation. As I’m working with a number of customers on this exact subject, I thought I would share my opinion on the subject.

If you want support for precision, complex gestures (multi-touch) while being able to access DOM elements as objects, there are no best way today than to use an instrumented browser. Continue reading

Join me at the Jenkins User Conference October 23rd 2013

jenkins Join me at the Jenkins User Conference October 23rd 2013

It’s been some VERY busy 6 months for me getting back to the United States but things are settling down nicely! So expect more updates from me on this blog! Aren’t you some lucky readers? icon smile Join me at the Jenkins User Conference October 23rd 2013

At SOASTA, we’ve always been big fan of CloudBees and this is why last year we had partnered with them to offer mobile developers and testers a way to continuously build and test mobile apps.

One of our goal is to compress the build-test-feedback process to a minimum. We want to provide developers a continuous feedback loop so they can assess the quality of their code as soon as they’re done writing it. Traditionally this feedback is given to them within days by traditional testing organization using outdated products. With Jenkins and TouchTest, we want them to give them this feedback in minutes while extending the scope of their tests.

Our plugin makes it easier for them to setup this feedback loop and adds new functionalities to build a more robust regression test environment.

  • Ability to play a composition, make the App Touchtestable automatically, Install App on device and capture screenshots.
  • Reboot iOS devices
  • This is a testing best practice that allow to run a set of tests under a “clean” state. After a while, a mobile device can become bloated and unstable. Being able to automatically reboot devices reduce the number of false-negatives and developers can spend time fixing real problems. This capability is unique on the market today for iOS (This capability comes free on Android)
  • Wake up iOS device
  • This functionality can be used to make the test environment more robust by waking up any iOS devices and put them in a “ready-to-test” state. A more robust test environment allows developers and testers to focus on more important activities ie. Building new tests!

Stockfish SVN Demo This Config Jenkins Join me at the Jenkins User Conference October 23rd 2013

You can learn how to get started with the plugin here.

Next week, I will be in Palo Alto at the Jenkins User Conference where I hope we can meet you so I can show you the great stuffs we’re cooking for developers. I especially want to demonstrate some end-to-end scenario I’ve been working on allowing to combine, via Jenkins, performance tests and functional tests of mobile apps. A lot of customers I’m working with want to understand the impact of back-end load on the performance of their mobile app front-end. Being able to do these type of tests continuously in order to find performance problems early in the development cycle, really help flush out the type of bottlenecks you don’t want to find when you run your 1M+ load test later on. The sooner you find these problems, the easier and cheaper it is to fix them! 

See you there!

 

mobileperformance Join me at the Jenkins User Conference October 23rd 2013

 

 

 

QTP, leave them mobile testers alone

the wall1 QTP, leave them mobile testers alone

I’m having more and more discussions with customers looking at investing into a mobile test automation solution. It’s a great way for me to listen to their problems and discuss potential solutions. One recurring requirement I’m hearing is the capability for the automation solution to integrate with HP QTP. And I have a big problem with the rationale that makes them request such integration.

I have nothing against HP products (or any testing products) as long as they’re used for what they’re designed for. As an example, Loadrunner is probably best fit for load and performance testing of client-server applications rather than large scale testing of web or mobile applications. Same goes with QTP in my opinion. It’s probably king for functional testing of desktop applications in a waterfall environment but is completely unfit for mobile applications which require dev and test high velocity. Do we still need a rudimentary script language (vbscript! Come on, this is 2013!) for fast test creation? Do we want to write code or rather record directly on an actual mobile device? Do we want to use a product that was built for mobile or a product built for desktop apps, developed 12 years ago? Do we want developers to use … QTP? Really? Continue reading

Could developers be the future of software testing?

mobileautomation2 Could developers be the future of software testing?

According to different reports, the global software testing market is around $30B and is looking at growing at about $50B in 2020. Not too shabby for the bastard child of IT! It is estimated that around 30% to 40% of this market is taken by offshore testing services (these days located in India, Eastern Europe and South America for the most part). I have nothing against outsourcing testing services as long as they bring the best value to the development organization. The problem with these services (from my own experience) is the fact that they still operate the same way than 20 years ago. Basically performing manual testing at the end of the whole development phase and have still a lot of difficulties to embed within an agile environment. I’m thinking that a lot of testing services company (especially the largest one) are doing everything they can to keep software testing in a status quo, in a dormant state for their own benefit. I don’t see a lot of these companies trying to invest in automation offering … How long will they keep fooling customers?

Continue reading

Get involved with SOASTA CloudTest Mobile Development!

iwantyou Get involved with SOASTA CloudTest Mobile Development!

I just realized it’s been more than 2 months without writing anything on my blog. Nope I didn’t take an extended break but been very busy with our mobile automation product CloudTest Mobile! And today I’m looking at getting you busy as well!

If you remember, back in February of this year, SOASTA introduced a brand new capability to CloudTest allowing developers and testers around the world to perform functional test automation for mobile devices. We also introduced our Early Adopter Program, allowing participants early access to new product features before the general release of the product. We received great feedback from our users and since then our engineering team has worked tirelessly to integrate this feedback into our product.

A couple of weeks ago, we were very excited to announce the latest major release of CloudTest Mobile which brings to market unprecedented features, allowing you to deliver even better quality mobile apps. As with our initial release, we’re opening up an Early Adopter Program dedicated to you developers and testers! Being part of this new program will allow you to:

Continue reading

CloudTest Mobile – A dream come true!

SOASTACloudTestMobile CloudTest Mobile   A dream come true!

If you’re in software development (and if you are reading this blog, I’m assuming this is the case. If you’re looking for cooking lessons, this is not the right place. Really.) you know the feeling. During long months of a release cycle, it is a roller coaster. From the initial idea which gets everyone in the company excited (if you don’t get pumped up by what your engineering team is cooking, maybe you’re in the wrong company …), to the early mock-up, the first demo (I remember the first demo we had a chance to see back in Feburary. I was speechless. Literally.), the beta-phase and the first feedback from actual customers, seeing the competition trying to get in the bandwagon, all the technical breakthrough the engineering team is able to crack … Everything leads to this day. When engineering, marketing, sales and operation cross the finish line (which is also the starting line) together. It is a feeling that will never get old for me. NEVER.

Continue reading