4 reasons why Appium is not ready for prime time

appium 4 reasons why Appium is not ready for prime time

As someone responsible for a mobile automation product, I spend a reasonable amount of time looking at the competition. I particularly like when I see some innovation, something that forces us to do even better and stay ahead of everyone else. Usually, this innovation comes from small company or from the open source community. Appium has been on my radar ever since it came out, but I waited for Appium to get some maturity before trying it out. My conclusion is very simple: Appium is not ready for prime time. It reminds me of Selenium and webDriver in its infancy. Not something you want to invest in for serious work. Maybe fine for individual developers to build some smoke tests, but if you have an enterprise app for example, with complex gestures and a lot of tests to build and maintain, Appium is not a good candidate, yet.

These are the main shortcomings I’ve noted. Some are really showstoppers for me.

#1 – No support for intelligent waits. 

notimedelays.jpg 4 reasons why Appium is not ready for prime time

This is the biggest shortcoming and a big red flag. You can code some time delays (polling loops) and that’s it. A time delay is the WORST mechanism to manage your page flow or address back-end response time variability. Reasons being:

  • How long should I wait after a back-end call? Yeah, let’s put 10 seconds, it should be way enough. It might work for 90% of the time and you end up spending time investigating errors for 10% of your test cases. The goal of automation is to find regression issues in your app! Not problems with your environment.
  • How long should I wait to account for device performance? Transition between pages depends a lot on the performance of your device. Might be almost instantaneous on the latest ipad Air for example with their new CPU and could be crawling on older iOS devices. So should you add 2 secs each time you transition between screens in your app? Should be enough right? Sure, it should be fine. Multiply these 2 seconds by your number of transitions, number of tests, number of devices and you end up with a lot of hours spent WAITING! When you have 10000 tests to run during the night for example, that’s not something you can afford. Especially when developers need feedback FAST!

Without being able to wait on a UI element characteristics (visibility, value, etc.) to manage page flow or back-end call, an automation framework for mobile is pretty much worthless in my book.

#2 – On iOS, you can only execute one test at a time per Mac.

This is a limitation coming from Apple Instruments that Appium uses for execution. I’m working with customers running 5000+ tests per night, on 30+ devices, all in parallel. It would take them many days to run all their tests sequentially. This is a SERIOUS limitation and I don’t know how Appium is going to solve this since they have no control over Apple Instruments. You can always buy 1000 Mac Minis! Yeah right.

#3 – Limited support for gestures.

If your app only use simple gestures, like tap, you could be fine. But this is 2013 and gestures are the name of the game to improve the user experience (double-tap and pan for Google map for example. It allows you to zoom with your thumb for example. Pretty neat!). Appium allows you to write Javascript wrappers to support these gestures. But c’mon .. Do you really want to spend time writing support for gestures or would you rather write new tests for your app?

#4 – Limited support for Android < 4.1

androidversion 4 reasons why Appium is not ready for prime time

Appium only supports API Level >=17. If you look at the number of devices running the different versions of Android (via the Android Dashboard) you realize that Appium can only support a very limited share of the market. Do you want to invest seriously in a framework that only allows you to support 18.2% of your market? I don’t think so. I thought it was pretty crazy to have such limited support and I dug a bit. Apparently, you can support Android < 4.1 by using a Selendroid library but it’s not for the faint of heart by judging by the number of people struggling to make it work.

These are my 4 reasons why I think Appium is not ready for serious work. They’re showstoppers in my book. I’m also not a big fan of the approach Appium is taking, which is fairly similar to Selenium/Webdriver. If you like to write tons of code for your functional tests, be my guest. Of all the WebDriver implementations I’ve seen in my career, a lot required a big investment, especially on the maintenance side. But I haven’t seen all projects in the world.

Appium is young and it might grow to something fit for mobile. And honestly, I hope it will! I’m a big fan of open source and the innovation it brings. But mobile is a big business and the product you pick to test your app is an important choice and investment, and you want to make sure it covers all your requirements. I just don’t think Appium is there yet.

Taking Mobile Test Automation on the Road!

topbestpracticesformobiletestautomation Taking Mobile Test Automation on the Road!

Spring is right around the corner and it’s going to be fairly busy for me! I’m very exciting to take my talk arond the United States. If you’re getting started with mobile test automation, this is a great talk to attend. I’ve worked with dozens of customers to help them avoid typical pitfalls and get them on the right track. Mobile test automation is an exciting journey that will make a BIG difference for the quality of your mobile apps. But it is a true software project by itself that needs careful planning, dedication and some best practices to follow. This is what my talk is all about!

If you attend “Top Best Practices for Successful Mobile Test Automation“, you will learn:

  • How to set the right goals and expectations for your automation project
  • How to track success and ROI
  • How to automate your mobile automation
  • When to start automating mobile tests
  • What mobile tests to automate
  • Where to run mobile tests
  • How to build testability in your mobile app
  • How to quickly increase your mobile test coverage
  • How to best manage mobile flows in your test
  • How to leverage your functional tests in your performance tests

This is my current engagements for the begining of the year. If you hear of any other great software testing conferences to attend, let me know!

Why use an instrumented browser for Mobile Web Test Automation?

amazon Why use an instrumented browser for Mobile Web Test Automation?

There are a lot of discussions in the software testing community discussing the benefits and drawbacks to use an instrumented browser approach to run mobile web automation. As I’m working with a number of customers on this exact subject, I thought I would share my opinion on the subject.

If you want support for precision, complex gestures (multi-touch) while being able to access DOM elements as objects, there are no best way today than to use an instrumented browser. Continue reading

It’s time to get real! Introducing mPulse

soastampulse Its time to get real! Introducing mPulse

I can’t resist to show you the latest video we’ve sent to our customers on Valentine’s day. This is real footage of our latest product mPulse, born soon after the logNormal acquisition only a few months ago. I’m so amazed by the work done by our engineering team in such a short time! In a nutshell, mPulse redefines the Real User Measurement space, allowing customers to understand in real-time how their web and mobile apps are behaving in the field. To help deal with bad user experience of mobile and web apps, mPulse comes to the rescue with Big Data, Real-time Processing, Actionable Intelligence and amazing 3D visualization!!

Get plenty of info and FREE RUM!

Enjoy the show!

Could developers be the future of software testing?

mobileautomation2 Could developers be the future of software testing?

According to different reports, the global software testing market is around $30B and is looking at growing at about $50B in 2020. Not too shabby for the bastard child of IT! It is estimated that around 30% to 40% of this market is taken by offshore testing services (these days located in India, Eastern Europe and South America for the most part). I have nothing against outsourcing testing services as long as they bring the best value to the development organization. The problem with these services (from my own experience) is the fact that they still operate the same way than 20 years ago. Basically performing manual testing at the end of the whole development phase and have still a lot of difficulties to embed within an agile environment. I’m thinking that a lot of testing services company (especially the largest one) are doing everything they can to keep software testing in a status quo, in a dormant state for their own benefit. I don’t see a lot of these companies trying to invest in automation offering … How long will they keep fooling customers?

Continue reading

Get involved with SOASTA CloudTest Mobile Development!

iwantyou Get involved with SOASTA CloudTest Mobile Development!

I just realized it’s been more than 2 months without writing anything on my blog. Nope I didn’t take an extended break but been very busy with our mobile automation product CloudTest Mobile! And today I’m looking at getting you busy as well!

If you remember, back in February of this year, SOASTA introduced a brand new capability to CloudTest allowing developers and testers around the world to perform functional test automation for mobile devices. We also introduced our Early Adopter Program, allowing participants early access to new product features before the general release of the product. We received great feedback from our users and since then our engineering team has worked tirelessly to integrate this feedback into our product.

A couple of weeks ago, we were very excited to announce the latest major release of CloudTest Mobile which brings to market unprecedented features, allowing you to deliver even better quality mobile apps. As with our initial release, we’re opening up an Early Adopter Program dedicated to you developers and testers! Being part of this new program will allow you to:

Continue reading

CloudTest Mobile – A dream come true!

SOASTACloudTestMobile CloudTest Mobile   A dream come true!

If you’re in software development (and if you are reading this blog, I’m assuming this is the case. If you’re looking for cooking lessons, this is not the right place. Really.) you know the feeling. During long months of a release cycle, it is a roller coaster. From the initial idea which gets everyone in the company excited (if you don’t get pumped up by what your engineering team is cooking, maybe you’re in the wrong company …), to the early mock-up, the first demo (I remember the first demo we had a chance to see back in Feburary. I was speechless. Literally.), the beta-phase and the first feedback from actual customers, seeing the competition trying to get in the bandwagon, all the technical breakthrough the engineering team is able to crack … Everything leads to this day. When engineering, marketing, sales and operation cross the finish line (which is also the starting line) together. It is a feeling that will never get old for me. NEVER.

Continue reading

Mobile, Cloud, PaaS … Crazy times at SOASTA!

Not much update from me in the last few months as we’ve been EXTREMELY busy icon smile Mobile, Cloud, PaaS ... Crazy times at SOASTA! We’ve been taken over by the whole mobile movement as our TouchTest Beta is going strong all across the world. We’re currently working with 100′s of companies who have a chance to be part of this beta to test their native application. The feedback were receiving is incredible and I’m helping a bunch of companies and partners to get started with our technology. If you’re into the mobile business (as you should be!) and are struggling with functional test automation, register on our website and I’ll be happy to get you started. If you’re still not familiar with TouchTest you can watch the webinar I ran in EMEA last month which will tell you why we came up with this new way of doing functional test automation for native applications. It has a lot to do with what customers were struggling with icon smile Mobile, Cloud, PaaS ... Crazy times at SOASTA!

 

You can also watch this video where Robert Scoble discovers TouchTest for the first time! He looks very impressed! Ken Gardner, our founder and executive chairman, and Tal Broda our VP of engineering, gives him an in-depth view of our product.

 

We’re also very busy on the partner front. In the past few weeks, we’ve announced some very important relationships with market leaders. Believe it or not, we’re partnering with HP as we’re able today to leverage their new public Cloud. We’ve also announced yesterday our support to the next-generation cloud from RackSpace and with OpenStack! As we’re working with 17 Cloud APIs today, supporting an open source API is very important for us and will help us integrate quickly OpenStack compatible cloud providers (if you’re interested to understand how we leverage Cloud APIs, there is a very good article from Dave Murphy on our blog). During the Rackspace event, Tom Lounibos and Dave Murphy gave their impression on the announcement and what it will mean for SOASTA in the future:

 

Last but not least, we’ve got more and more testers using CloudTest Lite (we’re seeing hundreds of download every day!) But we also want to influence developers to run more tests! As more developers are working using a PaaS, the best way for us to influence them is to be part of their development platform. That’s why we announced support for VMWare CloudFoundry last week. It will allow every developers using CloudFoundry to use CloudTest Lite for free during their development cycle. We’re the official test automation framework for CloudFoundry. That’s our present to them for their first birthday!

I’m writing this short note while attending the Selenium Conference in London where we have a chance to demo CloudTest and TouchTest to the Selenium community. I’m very excited to be in direct contact with true testers from all over the world and who are investing into the latest software testing trends. There are a few companies really innovative in this hot software testing market and we’re really glad to be recognized as such by the Selenium community. The Selenium movement has done a lot to help testers over the years and being part of this event is our way to say THANK YOU!