I was giving a demo to a partner last week using SOASTA CloudTest Lite and I had included some monitoring in the demo. People on the call who are doing performance/load testing using open-source products were very impressed by our ability to setup monitoring for any kind of infrastructure while being able to combine and correlate the infrastructure performance counters with the application performance data. And everything in real-time! So I though I would record a quick video to show everyone how it’s done. If you’re interested, you can find more than 50 in-depth CloudTest Lite videos on our CloudLink website.
However, there is one slight problem which needs to get fixed before reaching this stage: Performance. I guess you’ve noticed, performance on mobile is not all that great compared to the performance you get on your desktop’s browser. As this infography illustrates nicely, users have high expectation and mobile performance is still disappointing to most of them (I’m with them on that one! )
I’ll be flying out to London next week and participate to 2 testing events.
First, I will meet our new partner Testhouse and hold a joint webinar: Load and Performance Testing in Today’s Web and Mobile World – A revolution with SOASTA CloudTest. This is an introduction to cloud testing where I’ll have an opportunity to share some of the good stuff we’re doing with SOASTA. If you’re into load and performance testing, you’ll quickly realize that there is no turning back: Testing from the Cloud is your best choice today if you truly care about the performance of your web or mobile application. I love games (big chess fan, not very good though. Ping me if you want to play on redhotpawn.com) and there will be a game with a very nice prize for the winner. I can’t tell you more …
On Friday, I will participate to the SIGIST Conference and yes, I will talk about … Cloud Testing. Hopefully, my talk will lead to some very interesting discussions! While I love webinar, face-to-face discussion are the best! Conference topics will range from test automation, agile testing, test requirements and of course performance testing! I can’t wait to hear Mieke Gevers talk: Ever been fooled by performance testing results. I can definitely relate to the topic and it should lead to a good discussion. The complete agenda for the conference is here [PDF]
Before talking about why I’m so excited today, let me tell you a story. 18 months ago, as I was investigating SOASTA CloudTest for my own Software Testing organization, I had a chance to get a demo from Dan Bartow and a chat with Tom Lounibos about the product. Slavy Slavov who was heading our team in Sofia, Bulgaria was also part of the call. We were litteraly blown away by what we were seeing! My organization was using HP Loadrunner at the time (and you should know what I think about LoadRunner …) and CloudTest seemed way ahead of the competition: Leveraging public and private clouds, real-time analytics, a very innovative visual approach to test creation and execution etc. Slavy and myself were sold on first sight! Beside features and technical questions, I’ve asked the rather obvious question about an evaluation version of CloudTest, something my team could play around for a while before making any purchasing decision. Myself couldn’t wait to have a chance to play with it! Well, to my upmost disapointment there was no trial version available. Tom offered a proof-of-concept for whatever projects we were working on but the timing was too tight for us. We left it at that unfortunately …
I’m done watching most of the videos from this year’s Velocity conference and I couldn’t resist but to share this one with you. John Rauser’s keynote “Look at your data” was definitely one of the most popular. A very good reminder to dig into actual data to find performance problems. Summary statistics never tell you the complete story!
Next week, I am heading to Rome to present at the International SOA Conference – Moving SOA into the Cloud. I’ll be giving the presentation I gave at DataCenter 2011, but with additional content. Of course there will be some specifics about how to approach testing SOA in the Cloud but also content on the type of problems cloud testing allows to find across all environments we usually test in ie. Lab, staging and production environment via real customers’ story.
For example, how easy it is for a badly designed page to cap out your bandwidth (yes, we’re at 238mbits/s!) and max out your webservers with a low volume test in a staging environment.
How a fix in a load balancer firmware can help you go from this (horrible) situation ie. 6% error rate, response rate to the roof after 15min, bandwidth capping out.
To this much better situation ie. 0.001% error rate (can’t get better than that!) with 172K virtual users.
A few Observations on structure in testing Over the years, it’s been common for people in our community to mention exploratory testing, only to have someone reply, “Oh, so that’s like unstructured testing, right?” That’s a little like someone refer to a cadenza or a musical solo as “unstructured music”; to improv theatre as “unstructured theatre”; to hiking without a map or a precise schedule as “unstructured walking”; to all forms of education outside of a school as “unstructured learning”. When someone says “Exploratory testing is unstructured,” I immediately hear, “I have a very limited view of what ‘structure’ means.”
Scale Fail Hilarious video from Josh Berkus at MySqlConf 2011
uTest partners with Mozilla to build a test case management system Test Case Management can be trickier than herding cats. If you work for a small to mid-sized company and everyone’s counting on you to QA the apps to glowing perfection (no pressure, right?), then you’ve clearly demonstrated you’re a supremely organized person. It’s in your DNA.