If you’re in software development (and if you are reading this blog, I’m assuming this is the case. If you’re looking for cooking lessons, this is not the right place. Really.) you know the feeling. During long months of a release cycle, it is a roller coaster. From the initial idea which gets everyone in the company excited (if you don’t get pumped up by what your engineering team is cooking, maybe you’re in the wrong company …), to the early mock-up, the first demo (I remember the first demo we had a chance to see back in Feburary. I was speechless. Literally.), the beta-phase and the first feedback from actual customers, seeing the competition trying to get in the bandwagon, all the technical breakthrough the engineering team is able to crack … Everything leads to this day. When engineering, marketing, sales and operation cross the finish line (which is also the starting line) together. It is a feeling that will never get old for me. NEVER.
The excitement gets to higher level when you know for a fact that you’re the first company to release a product so unique. Mobile test automation is a green field (80% of all mobile apps are tested manually! Shocking.) and when you’re the first to reach this field with a product loaded with so much innovation, the feeling is very, very special.
Back in 2008, SOASTA has revolutionized the load and performance space by offering the first ever cloud-based testing platform. Today this platform is used daily by more than 400 customers worldwide, giving them access to 400,000 servers able to generate up to millions of virtual users, bringing them real-time analytic (probably as important as scale) at a very reasonable cost (no later than yesterday, I was talking to a customer who had an offer from a very well known competitor offering him a license for 5000 VUs for $150k! It is a very hefty price for a 20 years old technology … They decided to invest in the future instead without breaking the bank.). Last year, we made our technology available for all with the release of CloudTest Lite, allowing every testers in the world to run load and performance testing for free, forever. We have today ten of thousands happy testers using the product.
Today, we’re once again introducing something never seen before. A product that every single customer we’ve been talking to in the past 6 months have been waiting for to start investing in mobile app test automation. They’ve all tried what was out there but couldn’t find the right product that could meet all their requirements. It doesn’t mean other solution are useless as they probably all offer something interesting. I’ve tried most of them the past few months and I’ve found some interesting ideas. But none of them could cover the full scope of requirements a mobile app deserves today. We’re at the beginning of a new era that will lead to 80% of all application development being driven by mobile by 2015 and the need for new development and testing solution is becoming increasingly important. Release cycle in the app world is shrinking and automation is going to be mandatory and will be a real differentiator between a successful app (whether consumer or enterprise) and a failure (usually, a costly failure!)
So what’s so special about what we’re offering today? Well, before I discuss some important points, you can take a look at this short videos which introduce CloudTest Mobile, shot by Dan “The Man” Bartow who you probably know if you attend software testing conference.
In today’s app world, there are key requirements a product need to meet and looks like none of the existing one has been able to meet all of them since 80% of all apps are tested manually today. I’ve tried to gather these requirements into 3 main groups.
#1 – Meet the agility of app development life cycle
There are multiple problems I’ve seen with competitor solution I’ve been trying these past few months. I think the most concerning one is around their overall approach for recording. A lot of them still rely only on optical recognition. It reminds me of the 90′s when we had to deal with screen coordinates with tests breaking everytime the UI was different by a few pixels. We all know this is too fragile. The real cost in automation is not so much in the initial investment (as writing even a few automated tests brings loads of value) but in the maintenance of these tests release after release. If you have to go back to your tests, deal with cryptic scripting/code, sometime you better off with manual testing!
A few solution followed our tracks after we released our beta back in January and are today offering object based validation by being able to access the internal of the application. We were the first one to offer this approach and the fact that a lot of competitor are jumping on the bandwagon tells me that this is the best approach today. It is particularity useful for validations as you validate against the objects itself and are less prone to test breakage when you make change to your UI.
The third factor is a very important one. Yesterday, Apple has announced their new OS, iOS 6 which should be available this fall. Just like in the desktop world, whenever there is a new version of an OS, big or small, you have to ensure your application runs without any issue on the new OS. You just can’t take the risk not to launch a set of regression tests on the new OS. With CloudTest Mobile, as soon as iOS 6 is available, it will be a matter of upgrading your device and then run your set of tests on it. If you find issues related to the new OS, you can fix them quickly and release a new version on the app store. You will be able to validate quickly apps on the new OS because we don’t require jailbreak. Any customers using solution requiring to Jailbreak devices don’t have this luxury. Because they have to jailbreak device for their automation to work, they will have to wait for some hacker to come up with the proper jailbreak procedure to be able to upgrade to iOS 6 and validate their app on the new OS. It usually takes a few weeks to months for this procedure to be available. Apple makes it more and more difficult for people to jailbreak their device so it takes longer and longer. iOS 5 jailbreak was very long if I recall. Any customer using test automation relying on jailbroken devices will have to wait weeks or month before they can feel confident their apps run without any issues on the new OS. They will have to rely on manual testing (expensive) to validate all their app until someone come up with the proper jailbreaking procedure. This is an extra cost they have to take into account for each OS upgrade. With CT Mobile, this cost is zero.
#2 – Meet the requirement of the mobile world
Record real gestures on real devices
Mobile apps can be very simple (and this is a good thing) but can also be very complex from a gestures perspective. Gaming brings the most complex gestures but a lot of consumer apps today bring also their share of pinchOpen, pinchClose, swipe, rotate, landscape change, etc. If your solution is relying on a simulator or emulator, forget about recording these gestures. If the device you’re testing against is 5000km away from you (in a datacenter), forget about it as well. If you want to support real gestures, you have to record on REAL DEVICES. You want to record the gestures’ speed and velocity (critical for gaming) on the device and replay these on real devices as well. If you want to record precise tests, this is the only way. No wonder customers goes back to manual testing when the testing solution don’t support all generic gestures the OS brings but also customized gestures developers might come up with.
This is today the iOS gestures we support. Oh yeah, they’re all there.
Record and replay on untethered devices
When you test for the mobile world, you have to be … mobile. In the real world, devices are on carrier or wifi networks and applications are not used while it is tethered to a computer. How are you going to understand how your application behave when you have low signal, or on slow network if your device has to be connected to your computer? How are you going to validate against multiple devices (like all iOS devices) if all of them are connected to a computer? Build an expensive mobile rack? Come on, we can do better than that. With CloudTest Mobile you can test on any number of devices, connected on wifi or 3G/4G network. They can be close to you or anywhere in the world!
The fact that you can run your tests on untethered devices is not only very convenient but also opens up a realm of opportunity. One of them is what we call a Private Device Cloud. It allows you to create a pool of available devices for testing you can leverage when you need to run regression tests. These devices can in a lab somewhere in the world, but they can also be your own employees’ smartphone! Do you want to invest in 200 different devices when you can possibly leverage spare time of anyone smartphone? Sure you don’t.
The second benefit goes beyond functional testing but has more to do with performance testing. Because our technology is inside the app itself, we have access to all the internal of the device including CPU usage, memory consumption, signal, battery information, etc. We can also wrap transactions around your app business processes and time them to bring you a performance view of your app. How does it perform when your device is on wifi? On a particular carrier? How much memory do it consume? How fast is your login or search process? Add to the fact that you can also use CloudTest to load the backend of the mobile app, and you have a complete mobile performance testing solution. This has never been seen before!
Support all kind of apps
Again, a lot of customers came back from solution relying on simulator/emulator or mobile datacenter as they can’t support apps making use of GPS, accelerometer, camera, etc. How can you automate tests for a game today if you can’t move your device? Rotate it? How do you test a social app based on location if you can’t access GPS information? Duh. CloudTest Mobile supports any kind of apps, and trust me, we’ve tried a lot of them!
#3 – Support for the entire app life cycle
For developers, testers and business users
As I was writing a bit earlier, we’ve tried to make CloudTest Mobile very intuitive and simple to use. The visual and drag and drop environment is very, very convenient and could be used by any business users willing to add a few tests they find important. Developers will be happy to find an integration with their favorite Continuous Integration framework, whether they’re using Jenkins or Bamboo or any other CI. They can create simple or complex tests and launch them via our plugin and get results as part of their CI dashboard. This is a mobile app world and you want to put in place continuous testing, for every single build, every single code check-in made by developer. That way developers get continuous feedback and can fix problem on the spot. If you only test a few builds, you have to investigate which changes or addition of code cause the problem and this investigation is costly as it can be related to some code developers added days ago.The sooner you fix bugs, the better and the least expensive.
During development and support phase
Support is not the most fun activity for a developer. Trust me, it’s not fun at all for a test engineer either. Everyone wants to work on the new and cool stuff! Who wants to fix bugs and maintain existing automation tests? We’ve tried to make the maintenance and addition of new tests as painless as possible for testers. The fact that we rely on object recognition (vs fragile optical recognition) eliminate most of the maintenance you’ll have to make on your existing tests when there are new changes. The fact that we’re very visual will help you to quickly add the necessary new validation you’ll have to add based on some new bug fixes. It is extremely difficult to add new tests when relying only on test code and we’re confident a visual environment will alleviate this challenge.
Before the deployment phase
A lot of very successful app are released and used globally. A lot of them struggle from a performance perspective, especially at launch. I’ve seen way too many pop-up on my ipad telling me that some servers are too busy for the app to run at this time or bad performing app it’s almost impossible to use. Because we’re able to test on devices connected on wifi and 3G/4G and the fact that we can load the app backend at the same time, we can recreate a launch day before it happens and make sure all app users across the world will have a good user experience.
if you’re an existing users of CloudTest Lite, our free version: Log in your instance, click About, select the latest major release and click on update to selected release. Easy enough? We thought so
If you’re not a CloudTest Lite users, head to this page and register for your free version. Install, enjoy!
When you first log in to this new version of CloudTest Lite, you will notice some change to our welcome page. We’ve added link to our top CloudTest Videos coming from our community website CloudLink and there are also training videos specifically focused on CloudTest Mobile. I will recommend you watching them along with our new TouchTest Tutorial and Developer Guide. This should get you started in no time!
This is a very special day for us at SOASTA and I hope you’ll get as excited as us when you have a chance to try our new product. Please, feel free to reach out to me directly (fberinger at the soasta domain name) or in the comments below. I can’t wait to hear your feedback!!