State of Testing Survey 2013

There is a cool initiative going on from the folks at Tea Time With Testers and QA Intelligence. They’re launching a survey to help us understand the state of testing! What are the main challenges in the profession? Where is the profession headed? I’m definitely supporting the initiative and interested in the results! I’m following the software testing trends via analysts but a survey coming from the community could be quite insightful!

You can take the survey here.

Note: I have no affiliation with the survey.

4 reasons why Appium is not ready for prime time

appium 4 reasons why Appium is not ready for prime time

As someone responsible for a mobile automation product, I spend a reasonable amount of time looking at the competition. I particularly like when I see some innovation, something that forces us to do even better and stay ahead of everyone else. Usually, this innovation comes from small company or from the open source community. Appium has been on my radar ever since it came out, but I waited for Appium to get some maturity before trying it out. My conclusion is very simple: Appium is not ready for prime time. It reminds me of Selenium and webDriver in its infancy. Not something you want to invest in for serious work. Maybe fine for individual developers to build some smoke tests, but if you have an enterprise app for example, with complex gestures and a lot of tests to build and maintain, Appium is not a good candidate, yet.

These are the main shortcomings I’ve noted. Some are really showstoppers for me.

#1 – No support for intelligent waits. 

notimedelays.jpg 4 reasons why Appium is not ready for prime time

This is the biggest shortcoming and a big red flag. You can code some time delays (polling loops) and that’s it. A time delay is the WORST mechanism to manage your page flow or address back-end response time variability. Reasons being:

  • How long should I wait after a back-end call? Yeah, let’s put 10 seconds, it should be way enough. It might work for 90% of the time and you end up spending time investigating errors for 10% of your test cases. The goal of automation is to find regression issues in your app! Not problems with your environment.
  • How long should I wait to account for device performance? Transition between pages depends a lot on the performance of your device. Might be almost instantaneous on the latest ipad Air for example with their new CPU and could be crawling on older iOS devices. So should you add 2 secs each time you transition between screens in your app? Should be enough right? Sure, it should be fine. Multiply these 2 seconds by your number of transitions, number of tests, number of devices and you end up with a lot of hours spent WAITING! When you have 10000 tests to run during the night for example, that’s not something you can afford. Especially when developers need feedback FAST!

Without being able to wait on a UI element characteristics (visibility, value, etc.) to manage page flow or back-end call, an automation framework for mobile is pretty much worthless in my book.

#2 – On iOS, you can only execute one test at a time per Mac.

This is a limitation coming from Apple Instruments that Appium uses for execution. I’m working with customers running 5000+ tests per night, on 30+ devices, all in parallel. It would take them many days to run all their tests sequentially. This is a SERIOUS limitation and I don’t know how Appium is going to solve this since they have no control over Apple Instruments. You can always buy 1000 Mac Minis! Yeah right.

#3 – Limited support for gestures.

If your app only use simple gestures, like tap, you could be fine. But this is 2013 and gestures are the name of the game to improve the user experience (double-tap and pan for Google map for example. It allows you to zoom with your thumb for example. Pretty neat!). Appium allows you to write Javascript wrappers to support these gestures. But c’mon .. Do you really want to spend time writing support for gestures or would you rather write new tests for your app?

#4 – Limited support for Android < 4.1

androidversion 4 reasons why Appium is not ready for prime time

Appium only supports API Level >=17. If you look at the number of devices running the different versions of Android (via the Android Dashboard) you realize that Appium can only support a very limited share of the market. Do you want to invest seriously in a framework that only allows you to support 18.2% of your market? I don’t think so. I thought it was pretty crazy to have such limited support and I dug a bit. Apparently, you can support Android < 4.1 by using a Selendroid library but it’s not for the faint of heart by judging by the number of people struggling to make it work.

These are my 4 reasons why I think Appium is not ready for serious work. They’re showstoppers in my book. I’m also not a big fan of the approach Appium is taking, which is fairly similar to Selenium/Webdriver. If you like to write tons of code for your functional tests, be my guest. Of all the WebDriver implementations I’ve seen in my career, a lot required a big investment, especially on the maintenance side. But I haven’t seen all projects in the world.

Appium is young and it might grow to something fit for mobile. And honestly, I hope it will! I’m a big fan of open source and the innovation it brings. But mobile is a big business and the product you pick to test your app is an important choice and investment, and you want to make sure it covers all your requirements. I just don’t think Appium is there yet.

Join me at the Jenkins User Conference October 23rd 2013

jenkins Join me at the Jenkins User Conference October 23rd 2013

It’s been some VERY busy 6 months for me getting back to the United States but things are settling down nicely! So expect more updates from me on this blog! Aren’t you some lucky readers? icon smile Join me at the Jenkins User Conference October 23rd 2013

At SOASTA, we’ve always been big fan of CloudBees and this is why last year we had partnered with them to offer mobile developers and testers a way to continuously build and test mobile apps.

One of our goal is to compress the build-test-feedback process to a minimum. We want to provide developers a continuous feedback loop so they can assess the quality of their code as soon as they’re done writing it. Traditionally this feedback is given to them within days by traditional testing organization using outdated products. With Jenkins and TouchTest, we want them to give them this feedback in minutes while extending the scope of their tests.

Our plugin makes it easier for them to setup this feedback loop and adds new functionalities to build a more robust regression test environment.

  • Ability to play a composition, make the App Touchtestable automatically, Install App on device and capture screenshots.
  • Reboot iOS devices
  • This is a testing best practice that allow to run a set of tests under a “clean” state. After a while, a mobile device can become bloated and unstable. Being able to automatically reboot devices reduce the number of false-negatives and developers can spend time fixing real problems. This capability is unique on the market today for iOS (This capability comes free on Android)
  • Wake up iOS device
  • This functionality can be used to make the test environment more robust by waking up any iOS devices and put them in a “ready-to-test” state. A more robust test environment allows developers and testers to focus on more important activities ie. Building new tests!

Stockfish SVN Demo This Config Jenkins Join me at the Jenkins User Conference October 23rd 2013

You can learn how to get started with the plugin here.

Next week, I will be in Palo Alto at the Jenkins User Conference where I hope we can meet you so I can show you the great stuffs we’re cooking for developers. I especially want to demonstrate some end-to-end scenario I’ve been working on allowing to combine, via Jenkins, performance tests and functional tests of mobile apps. A lot of customers I’m working with want to understand the impact of back-end load on the performance of their mobile app front-end. Being able to do these type of tests continuously in order to find performance problems early in the development cycle, really help flush out the type of bottlenecks you don’t want to find when you run your 1M+ load test later on. The sooner you find these problems, the easier and cheaper it is to fix them! 

See you there!


mobileperformance Join me at the Jenkins User Conference October 23rd 2013




QTP, leave them mobile testers alone

the wall1 QTP, leave them mobile testers alone

I’m having more and more discussions with customers looking at investing into a mobile test automation solution. It’s a great way for me to listen to their problems and discuss potential solutions. One recurring requirement I’m hearing is the capability for the automation solution to integrate with HP QTP. And I have a big problem with the rationale that makes them request such integration.

I have nothing against HP products (or any testing products) as long as they’re used for what they’re designed for. As an example, Loadrunner is probably best fit for load and performance testing of client-server applications rather than large scale testing of web or mobile applications. Same goes with QTP in my opinion. It’s probably king for functional testing of desktop applications in a waterfall environment but is completely unfit for mobile applications which require dev and test high velocity. Do we still need a rudimentary script language (vbscript! Come on, this is 2013!) for fast test creation? Do we want to write code or rather record directly on an actual mobile device? Do we want to use a product that was built for mobile or a product built for desktop apps, developed 12 years ago? Do we want developers to use … QTP? Really? Continue reading

Long live Instagram!

instagram logo Long live Instagram!

I’ve been a hobbyist photographer for the past 10 years. Pretty much synched with the birth of my daughter who is today happy to enjoy 15,000 pictures of her first week in life icon smile Long live Instagram! The time was 2002 with the first decent digital camera out there. My first one was a Kodac DC3200 and was a great tool for me to learn the basics of photography. Then I moved to a Canon G2 before switching to a Canon SLR as Digital SLRs were costing an arm, a leg and everything in between. I made the mistake to rent a Leica M6 (At Keeble and Suchat in Palo Alto) so I could understand all the fuss about these cameras and feel confident that it was for snob. Huge mistake … I had to have one the following week, and another one, and another one … Oh and how about a Rolleiflex, oh and a Hasselblad, oh and look at those nice Russian Camera. At some point I probably had more than 20 working camera at home … And a few lenses. I was processing my own black and white and really, really enjoyed everything about photography. I was carrying a huge bag with many camera inside at all time. Then life catched up. Lots of traveling, different priorities. Sold most of the camera (Kept the Leica. Come on!), still have 300 rolls of film in my basement, and settled on a mid-range Canon digital camera for family events.

Beginning of the year, I’ve purchased an iPhone 4S to replace my Android HTC Desire (which takes crappy pictures) and started playing with it. I “discovered” Instagram back in May during a photo trip to Seattle with one of my good friend. We decided to shoot only with our iPhone for 2 days. I fell in love with the simplicity of the combo iPhone+Instagram. Reminded me of the time of my Leica + 50mm lens. No zoom (zoom with your feet!) and I’ve always liked the square format (Makes you think differently). Instagram allows me to give some “pop” to my photos (mainly contrast) and I’m able to share my pictures with a very private group of friends around the world. When I travel, it’s also a great way to keep in touch with the family as well (FaceTime does help too).

clouds Long live Instagram!seattle Long live Instagram!

sea Long live Instagram!goldengate Long live Instagram!

You can see some of my favorites pictures of 2012 on Flickr

Continue reading

CloudTest Mobile – A dream come true!

SOASTACloudTestMobile CloudTest Mobile   A dream come true!

If you’re in software development (and if you are reading this blog, I’m assuming this is the case. If you’re looking for cooking lessons, this is not the right place. Really.) you know the feeling. During long months of a release cycle, it is a roller coaster. From the initial idea which gets everyone in the company excited (if you don’t get pumped up by what your engineering team is cooking, maybe you’re in the wrong company …), to the early mock-up, the first demo (I remember the first demo we had a chance to see back in Feburary. I was speechless. Literally.), the beta-phase and the first feedback from actual customers, seeing the competition trying to get in the bandwagon, all the technical breakthrough the engineering team is able to crack … Everything leads to this day. When engineering, marketing, sales and operation cross the finish line (which is also the starting line) together. It is a feeling that will never get old for me. NEVER.

Continue reading

Join the SOASTA TouchTest Movement next week!

touchtestwebinar Join the SOASTA TouchTest Movement next week!

It’s been 2 weeks now that we’ve announced TouchTest and the feedback is really amazing! I think we’re really on to something here. The Beta is going extremely well with great excitement from engineering team across the world really looking at making our platform the best solution for mobile test automation.

I’ve been demoing our technology at least 3 times a day for the past 2 weeks with my agenda filling quickly! So I thought I’d organize a TouchTest gathering for everyone involved into mobile apps development and testing in Europe and beyond!

During this webinar I will discuss and demonstrate the new CloudTest capabilities including:

  • Precision capture and playback of ALL continuous touch gestures including pan, pinch, zoom, and scroll on iPhones, iPads, iPods and Android mobile devices
  • Touch-based UI testing from INSIDE the app, replacing brittle optical recognition approaches, enabling validations based on variable values and internal app state changes
  • Control of test devices via IP address, eliminating the need for tethering, jailbreaking, or contracting with expensive 3rd party services
  • Private Device Clouds that support affordable testing with real devices in your lab or across the globe

Do you want to join the TouchTest Movement? Join me February 23rd 2012, 12pm-1pm GMT