Fred Beringer » Software Testing

Hola Barcelona! Meet SOASTA at the Mobile World Congress!

Fred — Mon, 06 Feb 2012 14:33:46 +0000

I’m kicking off 2012 conference season with what should be one of the top conference I will have the opportunity to attend this year. The Mobile World Congress is THE place to be if you’re involved in the mobile world. The annual attendance is generally between 50,000 and 60,000 people! Theme for this year is perfect: Redefining Mobile.

As mobile takes more and more importance into the life of consumers and become a primary focus for Enterprise business, we at SOASTA are trying to stay ahead of the game and provide the best testing products to address this ever growing market. As you know, we’ve announced our new mobile test automation solution a couple of weeks ago and so far the feedback is INCREDIBLE! We’ve held our biggest webinar in our history for the occasion and we’re looking for a repeat 2 weeks from now at a convenient time for our European customers. I’ve been doing demo of our new technology all last week and it really opens up some very interesting discussions and open a realm of opportunity for developers and testers. The fact that we don’t rely neither on simulator/emulator or OCR makes complete sense for them. They’ve been abandoning these inappropriate approaches for a while now and are looking at investing into what looks to them the only way to properly test their apps. Being able to test INSIDE the app, record every single events, actions, gestures received by the application is very, very appealing. Being able to use validations they’re familiar with because they’re CloudTest users is also a definite plus. They’re also very excited about the concept of Private Device Cloud (and by its possible public extension!). As you know, when testing with TouchTest technology, your devices don’t have to be tethered. It means that I can run a test on a device located on the other side of the world for example. Now imagine, a company having 3000 employees and developing mobile apps. Before TouchTest, to validate your application on a broad number of devices, you had to either buy the actual hardware or rent some VERY expensive lab provided by third party relying on OCR. Wouldn’t it be nice to leverage my employee smartphones to run my tests? When you think about it, they all have different smartphones you app should support (whether it’s iOS, Android, Windows Mobile, etc.), they’re all connected to different carriers … They’re a very good representation of your market! Because TouchTest doesn’t require your devices to be tethered, you can tap into this pool to not only run functional tests on all of them but also capture performance information during execution! Oh, and you don’t have to mess up your employee’ smartphones because TouchTest doesn’t require jailbreaking. How about that for redefining mobile test automation? Definitely a game changer the industry was expecting.

What’s also very interesting to notice is that every single customer we talk to about mobile test automation, is also interested to hear about our solution for load and performance testing. Validating functionally the application running on an actual device is one thing. Making sure that the full ecosystem the application rely on, whether it is the customer’s backend/infrastructure or third-party, is probably as important in 2012. Being able to reproduce actual traffic whether it is coming from desktop’s browser or actual mobile devices is today a reality every customer wants to take advantage of. Having one platform covering the whole end-to-end test requirement is key to them.

My goal at the World Congress will be to discuss our new technology with application developers and testers to get their feedback and ideas. Meeting with companies to understand their challenges testing their apps and show them that they don’t have to go back to the dark ages (aka manual testing) to ensure the robustness of their deliverable. I also want to discuss with potential partners who wants to provide the best technology to customers and are willing to be part of the best ecosystem today for software testing. In the past few months, Correlsense, AppDynamics and OpTier have join our rank to provide the best services to customers and I’m confident I’ll have some great conversation with partner with a compatible DNA so we can strengthen our platform.

I will also formally meet a number of Spanish customers during a business breakfast and roundtable organized by our partner Testabil. There are still a few seats left so if you’re interested to participate, drop me a note at fberinger [at] SOASTA [dot] com! It would be great to meet you!

This is going to be an AWESOME week!

Get Shareaholic

SOASTA TouchTest – The day after

Fred — Wed, 25 Jan 2012 12:06:19 +0000

Wow! What a week! We’ve announced our new AWESOME product TouchTest this Monday and things got crazy really quickly! I’ve been talking about it yesterday all day long with testers and potential customers, and the feedback is truly AMAZING! Then when I had finally some time to catch up with what was said about TouchTest, I ended up spending quite a bit of time reading everything that was being said about TouchTest. Check this out!

Fast Company: There’s an App for that App

When apps were simple, only accessed by users numbering in the hundred or thousand and not particularly critical (in terms of usage, whether it was business use or for more sensitive situations like in medical environments) then the way the interface worked wasn’t incredibly vital. But as the app economy grows up, people are relying on apps more and more to help them communicate, to run their businesses, to interact with customers, to diagnose medical conditions, and other much more significant uses. In these situations, having a precise and responsive UI is vital to helping the app work, or to keep and delight a customer, and so on.

Information Week: Cloud Platform Stress Tests Mobile Apps

Other companies have been providing similar software testing and management services for enterprises, notably Mercury and Rational Software. And the acquisition of these companies for significant sums in years past, by HP and IBM respectively, underscores the value of such services to businesses.

ReadWriteWeb: App Testing Catches Up With the New Era of Gesture-Based Input

This can all be done without tethering, rooting or jailbreaking a device. TouchTest controls the device through software and accesses them through their IP address. Basically, TouchTest is Internet driven. Organizations can build a “private device cloud” with devices inside their data centers or in remote locations. That means that the number of devices needed to perform large scale testing is greatly reduced.

TabTimes: Pinch, zoom, touch: CloudTest release gives devs insight into mobile app performance

Aberdeen Group analyst Jim Rapoza said a growing number of companies are looking for better tools to test mobile app performance.

The Droid Guy: Developer’s Soasta Offering Mobile App Performance And Functional Testing Solutions

Through a multi-track, almost Garage Band like interface, TouchTest is able to log every input function of an application, every gesture, every reaction, save it in tracks to compare and recall with other results. While in testing mode the user can watch the test results in real time or play them back and compare them.

This is really, really exciting for us at SOASTA. We really have the feeling that we’re changing the way people will test mobile apps and it is a FANTASTIC feeling.

I’ve been playing with TouchTest the past couple of days and I’ve promised you some additional demo of the product. So here it is! A test for a mobile application I call the “dancing squares”!

If you want to try TouchTest for yourself, please apply to our beta program! And if you want to learn more about the technology behind TouchTest, I’d recommend you join our webinar on January 26th!

Get Shareaholic

Mobile Test Automation the SOASTA way with TouchTest

Fred — Mon, 23 Jan 2012 15:20:47 +0000

2 weeks ago I’ve spent a week at SOASTA HQ for our 2012 kick-off. A good opportunity to meet old friends, and make new ones! As you know, in December we’ve raised an additional $12M to expand our business and awesome people are joining the revolution every day! On the menu was a recap of 2011, plan and objectives for 2012 and of course a little bit of partying and dancing (first ankle tear for me in 2012! But I think I’ve scored some major points on the dance floor )

But during this week there was a true seminal moment, not only for me but for the whole team at SOASTA.

The engineering team led by Tal Broda unveiled what they had been working on for the last 2 months, day and night. I knew they were working on something yet again very disruptive but the secret was well kept and I couldn’t wait to see it for my own eyes. Imagine the scene: On the stage, a combination of ipad and iphone (iPad 1 and 2, iPhone 3G, 4 and 4S). The lead developer for this project (Tana Jackson, 23 Ironman under her belt!) had an ipad in her hand, untethered. She launched a demo application with colorful cubes displayed on the screen and started playing with them: Moving them, making circles, making them bigger, smaller … All the typical gestures you can expect from an iOS app. And in CloudTest, the magic happened: All these gestures were recorded in real-time in a typical CloudTest clip. But best of all, all the recorded gestures could be then replayed perfectly on the 5 iOS devices sitting on the stage. MIND BLOWING!

How in hell was that working? If you’re familiar with mobile test automation, you know that the market is today split into 2 types of products: Those relying on simulator and emulator and those on optical recognition. Both approaches are far from perfect: With simulator, you’re very far from reproducing the true user experience as it should occur on a real device. And with optical recognition, you have not only to invest a good part of your budget to afford it but the maintenance of test cases is a nightmare as you’re working at the pixel level … 4 pixels off and your test is broken. Most of the customers we’ve talked to are facing the same challenges when it comes to mobile test automation: They’ve tried a lot of open-source and commercial products before going back to … manual testing.

So just like in 2008 when Tom Lounibos, SOASTA CEO, addressed the challenges of load and performance testing to allow company to test at scale their web application, today SOASTA wipe away the challenges of test automation for native mobile apps. No more simulator or emulator, no more crazy expensive optical recognition mechanism. Nope, you’re going to be able today to test INSIDE THE APP. Record events from the app as they occur on the device. All possible gestures: Tap, doubleTap, swipe, scroll, pinchOpen, pinchClosed, etc. You will record their speed, velocity and acceleration and replay them on your mobile devices. And since the new mobile capabilities are part of the CloudTest platform, you take advantage of everything you’ve been using when executing load testing or functional test for web application ie. All the visual programming language, creating transaction, validating every steps of your test cases etc. If you’re familiar with CloudTest, then you’re all set to test mobile apps!

This is a short demo I’ve recorded so you can get a better feel of what CloudTest is:

This is what we’re announcing today:

TouchTest technology – this is the core of the announcement. TouchTest is our unique capability to record and replay ALL mobile gestures with full precision from inside iOS and Android mobile applications without tethering to or jailbreaking real mobile devices for functional and performance testing of mobile apps (Windows & others to come). There is nothing like this – all other approaches are incomplete, brittle, and unstable due to their external OCR and physically hardwired approaches. As we operate inside the app, we support native mobile, mobile web and hybrid mobile apps.
Private Device Clouds – because there is no need to tie down or physically or electronically “break” into a device, any employee or developer’s mobile device, anywhere, can be used as part of a private device cloud used for automated functional tests during development or as user experience devices for performance testing. Our customers with 5 or 500 developers all over the world can now be part of a private mobile test cloud when and as needed. Our platform connects and manages all the devices.
iOS and Android labs – we’re packaging our offering for mobile functional testing into labs targeting mobile development shops and orgs focusing on iOS and Android app development (more mobile labs will follow)

2008 marked a giant leap for load and performance testing when SOASTA first introduced CloudTest. 2012 is as big of a leap applied to mobile test automation. I feel so honored to be part of this adventure … I feel like the guys working at Apple in 1983 on the Macintosh … We’re trying to make a dent in the software testing universe and it looks like we’re making a BIG one!

We have a webinar coming up on Thursday dedicated to TouchTest. DON’T MISS IT! You’ll get more explanation on how the whole technology works.

To wrap up this post, this is what Tom tweeted last week. This is what the SOASTA team is all about … Kudos to all who makes it happen!

Get Shareaholic

Future UIs will bring challenges to Software Testing!

Fred — Wed, 18 Jan 2012 12:12:23 +0000

This video was taken at the TED conference in 2010 and presents the future of User Interfaces. The presenter, John UnderKoffler, is the guy who built the technology behind “Minority Report” 10 years ago. You can find emerging application for these kind of UIs from logistics, Financial distribution, military, etc.
Today’s UIs rely on Basic Touch (button), multi-touch (several actions) or gesture based (swipe, pinch, etc). But soon we’ll see interface based on voice (Siri like), 3D navigation, bendable phones (Nokia) and hovering gestures (Seen in this awesome demo)

If you think about it for a minute, you can easily imagine what kind of challenges these new UIs will bring to testers. Today, we’re still in the stone age when it comes to automating the test of apps on our smartphones or tablets. Think fingers and eye balls with a strong dependency on emulator, simulator and optical recognition which is definitely not the way to do it! Heck I don’t want to update all my test cases when one of my UI elements move 4 pixels … This is both crazy and scary! Are you already involved in testing mobile apps? Are you already struggling with automation? Watch this video and start shouting for help!

A video well worth 15 minutes of your time.

Get Shareaholic

Enterprise Mobility, a new era of opportunity!

Fred — Tue, 17 Jan 2012 09:17:50 +0000

This is an interesting infographic pulled by [x]Cube Labs showing the state of mobility in the Enterprise. Reducing cost is the major reason for mobility adoption (63%) to put in parallel with the growth of Enterprise apps in the Apple App Store (+186% in 2010). Interesting to see the revenue projection as well: $40B for developers in 2016! That’s a HUGE Market!

This is definitely a new era of opportunity for software development and of course software testing!

Get Shareaholic

Installing SOASTA CloudTest Lite – Even my mum can do it!

Fred — Thu, 22 Dec 2011 11:59:21 +0000

Ok, the title is a bit misleading as my mum is technology challenged but I’m sure my sister could do it! Seriously, yesterday I was describing the CloudTest Lite installation process to a software tester as he was not used to receive an installation package as a virtual image. The first time I installed CloudTest Lite, I was not used to it either! And actually wondered if it was the right packaging option for such product. Yes, I questioned my own engineering team! Shame on me. And as always, they blew me away as I’ve never looked back.

I can’t today imagine installing this kind of product in any other way. Let me explain: This is a web based product so it needs an application server to run (JBoss), a web server (Apache) and a database to store the results (PostGreSQL). Do I really want to install all these individual components on my local machine? Probably not. Testing is all about finding bugs, not spending time installing and configuring a product. You don’t want to spend time understanding why your database is not starting, or why you can’t access your web server etc. You want to push a button, be done with it and start testing. That’s exactly with this idea that CloudTest Lite was packaged. We’ve included everything you need to start testing in less than 10 minutes in a virtual image. You don’t have to mess up your local environment or read a bunch of installation and configuration document.

You don’t believe me? Watch this video and if you’re still not convinced, go through the process yourself and get your own opinion! I’d be interested to hear your feedback!

Get Shareaholic

What apps are in every developer toolkit?

Fred — Wed, 14 Dec 2011 16:17:23 +0000

This is an interesting survey done by bestvendor.com. They’ve interviewed 500 developers from companies of less than 100 people, asking them their choice of tools for their development environment. It’s interesting to notice that some categories are clearly dominated by one product: Eclipse is killing everyone else, MySQL is the obvious choice, Git has become in no time a strong leader … I’m very sad not to see any software testing tools/frameworks in this developer’s toolkit. They seem more important than website analytics for developers … I would also have been interested to see the preferred choice for continuous integration … My guess would be Jenkins but you never know.

It would be interesting to see similar survey for testers! I can bet we would start seeing CloudTest Lite on such infographic. And you know what? I’m 100% confident we’ll see it appear as #1 in one category for 2012. I can’t say more but our development team is cooking something truly awesome … !

Click on the picture for the larger version.

Get Shareaholic

Introduction to cloud provisionning with SOASTA CloudTest

Fred — Tue, 13 Dec 2011 15:28:30 +0000

Christmas is approaching fast and I thought I would treat you with another cool CloudTest Video Hopefully you’ve enjoyed the previous one which show you how easy it is to setup and use monitoring during a load and performance test when using SOASTA CloudTest Lite. Today I wanted to show you the power of CloudTest when it comes to testing from the Cloud.

You might have experience leveraging the cloud with some open-source performance testing product, struggling to manually provision your cloud environment from the EC2 console for example. Don’t get me wrong, the console is very useful, but definitely not fit for us in performance testing who might need more than 300 servers on any given day. (one of) The cool thing with CloudTest is that the whole provisioning aspect is entirely automated. For any type of Cloud providers! Whether you’re using EC2, Microsoft Azure, GoGrid, rackspace, IBM Cloud, etc. you don’t have to worry spending time on their management console, pulling your hair because you can’t get your servers up and running FAST. Performance testing is all about finding performance bottlenecks, not spending time getting an environment up and running. So today, I thought I would show you how it’s done within CloudTest. Enjoy!

Here a transcript for this video:

Hello and thank you for watching this video.

Today I’m going to show you how easy it is to run a performance or load test from a public cloud using CloudTest – how cool is that!

So what you’re looking at here is CloudTest. You’re familiar with CloudTest Lite, which is our free version of our offering allowing you to run load and performance test with up to 100 virtual users on your local machine. This is CloudTest, same web based interface and real time analytics – everything you’ve got in CloudTest Lite obviously you get in Cloud Test but what you get in addition is the access to Cloud Environment whether its public or private. And today I’m going to show you how easy it is to interact with that Cloud Environment for your performance tests. So we are going to test our Soasta Store which is our ecommerce demo website – very simple, people browse for a book, they interact on the forum – very simple application. I already have a composition ready, showing you user behaviour. so people are going to browse on that web site, they’re going to hit the home page, they’re going to search for a book and they’re going to post on the forum.

So what I’m going to show you is how to run that particular test on the Cloud. The first thing I need to do is to set up a Cloud Environment which is called a “Grid” in CloudTest world. I’m going to give it a name “Grid for Video” and I’m going to click on next. Here I’m going to tell CloudTest how many servers I need, the location of the servers and the type of servers. So for that particular test I want to run a disparate test from four locations in the world. The US West Coast, East Coast, Europe and Asia.

So I’m going to have one server on the East Coast, one server on the West Coast, other location – we are going to add Europe, and we’re going to add Singapore. Automatically we are also going to spawn one result server, and that result server – that’s our analytics server, responsible to gather aggregate, combine all the performance data coming from the application and the infrastructure. So I’ve declared my eight servers – four test servers, four results servers. I can change the server type if I want by default here it’s EC2 Large, but it could be High Memory XL, Quadruple, Double, CPU XL, all the server types Amazon offers. So I’m done with my Grid, it’s not deployed yet, I only deploy a grid when I need it. So here I’m actually ready to run my test so I’m going to deploy those instances – 4 locations, 4 test servers and 4 result servers. The cool thing about CloudTest is here – what you are seeing is an automatic deployment. So we have integrated all Cloud provider API within the product such as EC2, Microsoft Azure, IBM Cloud, Rackspace, Go grid so that you don’t have to deal with the manual processing of your server. I don’t have to go to Amazon console to manually launch my server which can be quite a difficult task when you want to spawn 20 servers, 50 servers or 100 servers. You don’t want to do that manually, you want a mechanism, an automatic mechanism and CloudTest offers you that here. It’s allocating all the servers I need for that particular environment. What’s cool about it is that if you get an error – connection time out, connection error, if anything goes wrong, CloudTest is automatically going to re-try until you get those servers up and running in good health for you to run your test. So I don’t have to spend time on that particular thing, we’ll go back a bit later.

Next, what I need to do on my composition is pick and choose the location for these clips to be executed. As you can see here, I have a drop down that allows me to choose the location I want to use. So here I have a lot of locations that I can use – Rackspace, Azure, IMB Cloud, EC2, but I have an EC2 grid so I’m going to use EC2. East Coast, West Coast, Europe and Singapore, ok, so I’m going to run those four tests on those four locations. I think I’m done on my composition so I’m going to save it. I’m going to go back to my grid; it’s not ready yet so as you can see here as I was saying, I’m getting errors here on some of them. So CloudTest is re-trying until you get all these servers up and running. Here I have eight so I only have one problem now with one of the servers – connection refused. I don’t really care as I know that CloudTest is going to re-try until it gets those servers up and running for me which is cool. I can get some metrics on these servers that are up – allocation time, pending time, busy time – all types of statistics. I can change some settings for those servers – allocation block size, re-try count, re-try delay, there’s a whole bunch of advanced settings that you can set. I’m not going to do that here because I want to keep it simple.

So here I’m actually done allocating, CloudTest is done allocating all those servers.

What it’s doing now is its starting the monitoring for those eight servers because you might want to check on those servers as you generate load or get results, you want to check on those servers to make sure everything is fine and when it’s done it tells you the grid is ready. Here my grid is up and running. I have one server, actually 2 servers in all locations – one test server and one result server.

So what I’m going to do here, I’m going to start my test.

What happens when I start my test here, it’s just the same thing as when you run your test on CloudTest Lite. On CloudTest Lite, all the clips are going to be executed locally, here CloudTest is going to deploy that composition and those clips on those specific locations. So the browsing scenario is going to be executed from the East Coast and the home page scenario is going to be executed from the West Coast etc. So it’s the same thing as CloudTest Lite but here you are able to leverage public files and specifically here EC2. So that’s your dashboard that your familiar with. If you use CloudTest Lite, same thing – real time analysis that you get in front of you. What you’ve also got is the monitoring summary which is what I’ve shown in a previous video which is very simple to set up and put in place so it tells me the CPU percentage of the server hosting the website, per process CPU percentage and the process count. All the performance matrix coming back from the infrastructure which is very important.

I’ve added a dashboard which gives you a graphical interface of that test so here the target is our website “Soasta Store”, it’s located in Seattle, Washington state. Here we can see our four servers, west coast, east coast, Europe and Singapore hitting the website with the particular traffic that we’ve defined in the composition. I can click on each of these servers and get individual response time for these locations. That’s very important for some of our customers that want to replicate, not only do they want to replicate global traffic but they want to understand the response time and the performance metrics when people hit the website coming from these locations because they might want to optimise a specific location for a particular event. For example, an ecommerce site might want to have a special offer for east coast customers and so on the D-day they want to make sure the performance of the website is optimised for east coast customers so that’s why they want to test for example 80% of the traffic is going to come from the east coast and the rest from the rest of the world and they want to make sure that these response times are really good for the east coast customer. I have the same type of information here on my summary dashboard.

So that’s my average response time, aggregated view, I would say from all locations and I can split by location here and get a detailed view. I can filter them out so if I’m only interested by monitoring Europe and US East I can do that and as you can see obviously we can see that the response time from Europe is spiking a little bit and the east coast is obviously better because its closer to the website in that particular case and there’s no CDNs in so it’s a very simple web environment.

So that’s it, we’re testing from the cloud from multiple location, I’ve got four locations, I could test from 28 locations, that’s what Soasta offers today.

So very easy to set up you just declare your grid, you spawn your server automatically and you use them. What you want to do is when you’re done with your test campaign, I would say test for a day on those multiple locations. I’m going to stop my test here. What you want to make sure that you do is that you tear down your environment. Again its completely automatic here, I just tick on tear down instances and Cloud test is going to kill those servers for me and its very fast and here I’m done for my grid. If I want to use it again I just deploy the grid and that’s super easy. So what’s interesting is that we are seeing more and more customers creating their tests and testing up to 100 virtual users using CloudTest Lite on their local machine and when they’re ready to scale, when they want to see how their website is going to scale with the same type of scenario, they give the scenario to us and we execute the scenario on our global cloud. So if you are interested in doing that just send me an email fberinger@soasta.com and I’ll be happy to help you look at your scenario and help you run those tests on our global cloud. Whether it’s EC2 or a Azure or Go Grid or Rackspace, whatever you choose, some customers have a preference. That’s up to you!

So I hope you enjoyed the video and I hope that it was informative and hopefully I’ll see you for my next video. I’ll see you soon!

Bye!

Get Shareaholic

Early Christmas – SOASTA secures another $12M in funding!

Fred — Mon, 12 Dec 2011 12:35:19 +0000

I took my Monday’s coffee with the best read ever! Tom Lounibos, our fearless leader, announced internally a fantastic news to all SOASTA employees. We all knew this was in the work but it feels so good to finally be able to announce it to the world! We’re announcing today a $12 million round of funding! How about that for a Christmas’ present?

It does wrap up a FANTASTIC year for SOASTA:

In 2011, SOASTA’s customer base topped 3,000, including six of the top 10 retailers and well-known leaders in the financial services, media, telecommunications, consumer packaged goods, and technology industries.
SOASTA’s Global Test Cloud expanded dramatically in 2011 to support generating load from across the world, including U.S., Canada, China, UK, Germany, Amsterdam, Ireland, Dubai, Switzerland, Australia, Singapore, Japan and Hong Kong
SOASTA delivered its first free edition in 2011, CloudTest Lite, which offers functional testing and performance testing for up to 100 virtual users at no cost.
The company was awarded a patent on CloudTest’s innovative visual test creation environment, which delivers a familiar GarageBand-style process for building complex performance and functional tests in a fraction of the time.
SOASTA was named one of the “Top 50 Venture-Backed Companies” by The Wall Street Journal, one of the OnDemand top 100 emerging Internet companies by AlwaysOn, and one of the “10 Emerging Cloud Vendors You Need to Know” by CRN.

This new round of funding is a clear indication that the company is on the right tracks, ready to take the world of software testing to the next level. And unless you’ve been living in a dark cave this last year, it means MOBILE testing. I’ve been lucky to see some internal demo of what’s coming next from our engineering team and it blew my mind. Literally. SOASTA has been disrupting the load and performance testing world ever since 2008 with the introduction of Cloud Testing … What’s coming next for mobile is at least as big! Stay tuned …

Other noteworthy news under my Christmas tree?

We’ve announced a partnership with AppDynamic. It results in the integration of the AppDynamics real-time application monitoring solution with our CloudTest Platform. It allows you to quickly set up and run full-scale performance tests of production systems using CloudTest with Java and .NET performance analytics collected through AppDynamics. Real-Time testing and production monitoring converge! We’ve ran a VERY successful webinar a couple of weeks ago which goes through the new possibilities this partnership offers. Check it out!

Speaking of monitoring, users of CloudTest Lite can now monitor datasources from CloudWatch and NewRelic. How cool is that?
Closer to home, we’ve opened up a new SOASTA office in the UK! Windsor here we come!
On a more personal note, I’m celebrating my first anniversary with SOASTA this month. As you can imagine, I couldn’t be happier!

Wow, this is only Monday and so much adrenaline already!

Get Shareaholic

Introduction to monitoring using SOASTA CloudTest Lite

Fred — Thu, 08 Dec 2011 21:17:34 +0000

I was giving a demo to a partner last week using SOASTA CloudTest Lite and I had included some monitoring in the demo. People on the call who are doing performance/load testing using open-source products were very impressed by our ability to setup monitoring for any kind of infrastructure while being able to combine and correlate the infrastructure performance counters with the application performance data. And everything in real-time! So I though I would record a quick video to show everyone how it’s done. If you’re interested, you can find more than 50 in-depth CloudTest Lite videos on our CloudLink website.

Enjoy!

Here is an audio transcript of the video:

Hi there and thank you for watching this short video.

Today I wanted to show you how easy it is to start monitoring your application infrastructure when you are using Soasta CloudTest Lite. The whole point of doing load and performance testing is to optimise the performance of your app and finding bottlenecks that could be in the application itself or in the infrastructure where the application is running. And so today we are going to focus on that, on the infrastructure problem we could find.

For that demo, I’m going to hit our demo site which is called Soasta Store. It’s a very small wordpress application running on EC2, and that mimics ecommerce web sites so people can shop for books, they can interact on forum, post topic and reply etc. so a very simple ecommerce site. I won’t go through the scenario recording, creation and how to create a clip, and how to prioritise the scenario and all that good stuff. Maybe in subsequent videos I will go through that but today it’s all about monitoring. So I’ve prepared everything and I’ve created a composition. A composition is a representation of my end to end test and the traffic that I’m going to use for that test. So during the test a virtual user will browse the ecommerce site, they’re gonna hit the home page, they’re gonna search for a book etc. I have some seed data there too you know, randomised the book and they’re gonna interact on the forum. So very simple scenario that’s just enough for what we want. We’ll use a hundred virtual users again – enough, and this is what you get with Soasta CloudTest Lite, you can test up until hundred virtual users. So I get my composition that I’m gonna play. Now I’m gonna set up the monitoring part. The first thing you wanna do is to create a monitoring server group and that’s gonna tell CloudTest Lite the servers that you want to monitor. So I’m gonna click next here, I want to create a monitoring server group, I’m gonna give it a name “Soasta Store Demo Server Group” and I’m gonna start adding all the servers that I want to monitor here. I only have 1 server, it’s a very small application so I only have 1 server but it could have multiple web servers, I could have multiple app servers, some database servers as well but here I only have 1 server. I’m gonna click on next, I’m gonna select the type of resources I want to monitor, it could be the CPU memory, the system resource in that box, the app server running in the box, it could be a network device so SNMP where I’m gonna be able to monitor switches, load balancer and all that kind of stuff, Databases and Cold Fusion. Here I only want to click on that box – monitor the system resource.

I’m gonna click on next, here I’m gonna tell CloudTest Lite how to access that box. On my example, this is a red hat box so that’s perfect on port 22 I’m gonna connect using SSH and authenticate using username and password – I’m gonna enter it here. I’m gonna test the connection just to make sure I didn’t do any typo – I’m good. So here I’m running red hat but I could pick any, Linux distro, we can monitor AIX, we can also monitor Windows. For windows XP2003 and Vista you’re gonna have to install a conductor like an agent that will allow us to retrieve performance counter on those Windows box. If you’re using windows server 2008 you can connect using WMI but on our example we’re talking about red hat Linux. So I’m good here. Before finishing I’m gonna create a monitor on that server group and I’m gonna finish. So that’s my actual monitor that I’m creating, that’s gonna run on that server group so “Soasta Store Demo Monitor” and it’s sitting on that server group, perfect. Next – here I’m gonna select the resources I want to monitor so this is all my choices and well I might as well monitor everything so the IO stuff, the memory, the network activity, all the per process activity that’s specific for apache – my web server, and actually argument, I get apache 2 process running, so I’m gonna enter that here, and the process count. I can add custom object if I have custom monitor running but I’m not going to do that here. I don’t actually have custom monitor. And I’m good to go, finish, I go on monitor, I add my monitor here. I can start it right from here but there’s 1 thing I prefer because I don’t like to think about starting my monitor and stopping my monitor, what I usually like to do is directly on the composition I can tell CloudTest Lite to start the monitor. Every time I run a test it’s gonna start to monitor automatically and it’s gonna stop it when a test stops so it’s just convenient, it’s easier. I’m just gonna verify 1 thing which is my ramp up time – 5 minutes, so it’s 5 minute ramp up for 100 virtual users should be good enough to get a fair understanding of the performance of the application and the infrastructure. I’m gonna start the test so it’s loading the composition right now and really quickly we’re gonna start seeing some performance results coming back at us. So if you’re not familiar with CloudTest Lite, when you run the test you’re gonna watch dashboard as everything is dashboard and widget based. This is my typical dashboard that I use for demo that I’ve created. With all the high level performance information so I’m gonna have my fundamentals, data – elapsed time, message sent, my average response time which is an aggregation of all the response time of all the pages and all the assets. The throughput, my current virtual user and I have a widget like the typical average response time, the virtual users, my bandwidth, my send rate – it’s very important, and error count, bytes received and that’s my dashboard. But you get to customise it the way you want with as many widgets as you want. I’m actually gonna add 1 chart which is the error analysis. Just if you hit somewhere we wanna see, I forgot to add this one so I’m gonna add it here. So in real time, I can, you know, add widget, remove widget, so here I don’t have any error but maybe later we’ll see if we get something. It’s always useful to have this widget on your high level view.

So we’re 1 minute and a half with the test, we’re doing OK, we got a spike here but nothing out of the ordinary. The response time is fairly flat, we got 36 virtual users, everything looks fairly good. Let’s jump on the monitoring side to see what’s going on in that box. So again that’s my typical monitoring summary I would say, so I’ve got the CPU percentage on the box and we’re seeing that it’s increasing very well. Can’t say that it’s looking really good. I got my memory usage on top of the virtual user; I can have both of them on top of each other so I can clearly see what going on. It’s definitely increasing but it’s normal. I got my process count, per process CPU percentage and all the process count you see increasing – that’s all apache process spanning as the load is increasing. I got, you know, obviously I’m spanning new apache process and this is my top window. So things are looking well, not that good because I’m starting to hit 100% on my box in 2 minute time frame. Let’s go see on my summary what’s going on. Yes, so definitely I can see that there is something going on here. It is fairly flat but here it’s increasing. I’m starting to see some errors here. Not that much but it’s getting there. I got an error, I can see what’s going on so OK 3 forbidden – not gonna care too much about that for now. But definitely there is something going on. What I’m going to do is, I’m gonna put my CPU percentage on top of the average response time. That’s the cool thing about that UI, you can take 2 widgets and combine them, correlate them which is very useful to see – to have a better view and that allow you to explain, I would say, the performance, what you observe. It’s gonna help you get a better explanation I would say. You’re gonna see, I’m not very clear here right, but you’re gonna see what I’m talking about. So here I’m combining both charts, I’m gonna remove that, I’m gonna make it a bit clearer and there’s plenty of options to customise your dashboard. It’s actually awesome let me tell you that. So CPU percentage, I want to see an area – that’s always a bit better and I wanna have that the other way around so I wanna see that here and I’m gonna apply and I’m gonna say OK. Alright, so here you can clearly see the fact that you can combine widget allows you to correlate data so here there is a clear correlation between my CPU kind of maxing out here and increasing response time. It’s obvious that my CPU maxed out and I actually know that because the box we are using for that demo is really small. I think its one of the smallest EC2 instance, with only 4 CPU I think so it can’t handle a lot of load. So there’s an obvious correlation between what’s going on in the box and what we’re seeing as far as response time. Here at the same time were seeing the error count, and yeah, we’re seeing time out this is why we are having an increase of response time. So very easily just by setting up a resource monitor on the box I’m hitting I can already tell you that that box is not fit for you for production, for production. But just to show setting up monitor, server group monitor and using dashboard and widget, you can already start to get some answer about the performance of your application.

And that’s really cool, I mean here I’m only monitoring 1 server but I can show you the result of the same wordpress app but on a bigger machine with multiple web servers handling 10000 virtual users. So here you can see CPU percentage for all the boxes we have and we have a lot of web servers. Here you can see them. Right, so that’s a more realistic test – that’s my memory usage for all the servers, CPU percentage on top of virtual users so that you can see that actually it was really good – look at that, 10000 virtual users and we only have CPU percentage 40% so plenty of room to grow. But you know that environment was ready to handle production type load which is good. But my demo is only 1 server and definitely not enough to handle that traffic so here you can see its obvious that we’re having issues very quickly actually.

So if you’re using CloudTest Lite and you’re not monitoring today, please start whether you get a windows box, or Linux box, or AIX box or whatever box, try to get credentials to the server and to start really get the whole story because if you don’t monitor your infrastructure I think, then you’re actually missing a big part of the story.

Download SOASTA CloudTest Lite today and start monitoring!

Get Shareaholic