I was giving a demo to a partner last week using SOASTA CloudTest Lite and I had included some monitoring in the demo. People on the call who are doing performance/load testing using open-source products were very impressed by our ability to setup monitoring for any kind of infrastructure while being able to combine and correlate the infrastructure performance counters with the application performance data. And everything in real-time! So I though I would record a quick video to show everyone how it’s done. If you’re interested, you can find more than 50 in-depth CloudTest Lite videos on our CloudLink website.
Here is an audio transcript of the video:Hi there and thank you for watching this short video. Today I wanted to show you how easy it is to start monitoring your application infrastructure when you are using Soasta CloudTest Lite. The whole point of doing load and performance testing is to optimise the performance of your app and finding bottlenecks that could be in the application itself or in the infrastructure where the application is running. And so today we are going to focus on that, on the infrastructure problem we could find. For that demo, I’m going to hit our demo site which is called Soasta Store. It’s a very small wordpress application running on EC2, and that mimics ecommerce web sites so people can shop for books, they can interact on forum, post topic and reply etc. so a very simple ecommerce site. I won’t go through the scenario recording, creation and how to create a clip, and how to prioritise the scenario and all that good stuff. Maybe in subsequent videos I will go through that but today it’s all about monitoring. So I’ve prepared everything and I’ve created a composition. A composition is a representation of my end to end test and the traffic that I’m going to use for that test. So during the test a virtual user will browse the ecommerce site, they’re gonna hit the home page, they’re gonna search for a book etc. I have some seed data there too you know, randomised the book and they’re gonna interact on the forum. So very simple scenario that’s just enough for what we want. We’ll use a hundred virtual users again – enough, and this is what you get with Soasta CloudTest Lite, you can test up until hundred virtual users. So I get my composition that I’m gonna play. Now I’m gonna set up the monitoring part. The first thing you wanna do is to create a monitoring server group and that’s gonna tell CloudTest Lite the servers that you want to monitor. So I’m gonna click next here, I want to create a monitoring server group, I’m gonna give it a name “Soasta Store Demo Server Group” and I’m gonna start adding all the servers that I want to monitor here. I only have 1 server, it’s a very small application so I only have 1 server but it could have multiple web servers, I could have multiple app servers, some database servers as well but here I only have 1 server. I’m gonna click on next, I’m gonna select the type of resources I want to monitor, it could be the CPU memory, the system resource in that box, the app server running in the box, it could be a network device so SNMP where I’m gonna be able to monitor switches, load balancer and all that kind of stuff, Databases and Cold Fusion. Here I only want to click on that box – monitor the system resource. I’m gonna click on next, here I’m gonna tell CloudTest Lite how to access that box. On my example, this is a red hat box so that’s perfect on port 22 I’m gonna connect using SSH and authenticate using username and password – I’m gonna enter it here. I’m gonna test the connection just to make sure I didn’t do any typo – I’m good. So here I’m running red hat but I could pick any, Linux distro, we can monitor AIX, we can also monitor Windows. For windows XP2003 and Vista you’re gonna have to install a conductor like an agent that will allow us to retrieve performance counter on those Windows box. If you’re using windows server 2008 you can connect using WMI but on our example we’re talking about red hat Linux. So I’m good here. Before finishing I’m gonna create a monitor on that server group and I’m gonna finish. So that’s my actual monitor that I’m creating, that’s gonna run on that server group so “Soasta Store Demo Monitor” and it’s sitting on that server group, perfect. Next – here I’m gonna select the resources I want to monitor so this is all my choices and well I might as well monitor everything so the IO stuff, the memory, the network activity, all the per process activity that’s specific for apache – my web server, and actually argument, I get apache 2 process running, so I’m gonna enter that here, and the process count. I can add custom object if I have custom monitor running but I’m not going to do that here. I don’t actually have custom monitor. And I’m good to go, finish, I go on monitor, I add my monitor here. I can start it right from here but there’s 1 thing I prefer because I don’t like to think about starting my monitor and stopping my monitor, what I usually like to do is directly on the composition I can tell CloudTest Lite to start the monitor. Every time I run a test it’s gonna start to monitor automatically and it’s gonna stop it when a test stops so it’s just convenient, it’s easier. I’m just gonna verify 1 thing which is my ramp up time – 5 minutes, so it’s 5 minute ramp up for 100 virtual users should be good enough to get a fair understanding of the performance of the application and the infrastructure. I’m gonna start the test so it’s loading the composition right now and really quickly we’re gonna start seeing some performance results coming back at us. So if you’re not familiar with CloudTest Lite, when you run the test you’re gonna watch dashboard as everything is dashboard and widget based. This is my typical dashboard that I use for demo that I’ve created. With all the high level performance information so I’m gonna have my fundamentals, data – elapsed time, message sent, my average response time which is an aggregation of all the response time of all the pages and all the assets. The throughput, my current virtual user and I have a widget like the typical average response time, the virtual users, my bandwidth, my send rate – it’s very important, and error count, bytes received and that’s my dashboard. But you get to customise it the way you want with as many widgets as you want. I’m actually gonna add 1 chart which is the error analysis. Just if you hit somewhere we wanna see, I forgot to add this one so I’m gonna add it here. So in real time, I can, you know, add widget, remove widget, so here I don’t have any error but maybe later we’ll see if we get something. It’s always useful to have this widget on your high level view. So we’re 1 minute and a half with the test, we’re doing OK, we got a spike here but nothing out of the ordinary. The response time is fairly flat, we got 36 virtual users, everything looks fairly good. Let’s jump on the monitoring side to see what’s going on in that box. So again that’s my typical monitoring summary I would say, so I’ve got the CPU percentage on the box and we’re seeing that it’s increasing very well. Can’t say that it’s looking really good. I got my memory usage on top of the virtual user; I can have both of them on top of each other so I can clearly see what going on. It’s definitely increasing but it’s normal. I got my process count, per process CPU percentage and all the process count you see increasing – that’s all apache process spanning as the load is increasing. I got, you know, obviously I’m spanning new apache process and this is my top window. So things are looking well, not that good because I’m starting to hit 100% on my box in 2 minute time frame. Let’s go see on my summary what’s going on. Yes, so definitely I can see that there is something going on here. It is fairly flat but here it’s increasing. I’m starting to see some errors here. Not that much but it’s getting there. I got an error, I can see what’s going on so OK 3 forbidden – not gonna care too much about that for now. But definitely there is something going on. What I’m going to do is, I’m gonna put my CPU percentage on top of the average response time. That’s the cool thing about that UI, you can take 2 widgets and combine them, correlate them which is very useful to see – to have a better view and that allow you to explain, I would say, the performance, what you observe. It’s gonna help you get a better explanation I would say. You’re gonna see, I’m not very clear here right, but you’re gonna see what I’m talking about. So here I’m combining both charts, I’m gonna remove that, I’m gonna make it a bit clearer and there’s plenty of options to customise your dashboard. It’s actually awesome let me tell you that. So CPU percentage, I want to see an area – that’s always a bit better and I wanna have that the other way around so I wanna see that here and I’m gonna apply and I’m gonna say OK. Alright, so here you can clearly see the fact that you can combine widget allows you to correlate data so here there is a clear correlation between my CPU kind of maxing out here and increasing response time. It’s obvious that my CPU maxed out and I actually know that because the box we are using for that demo is really small. I think its one of the smallest EC2 instance, with only 4 CPU I think so it can’t handle a lot of load. So there’s an obvious correlation between what’s going on in the box and what we’re seeing as far as response time. Here at the same time were seeing the error count, and yeah, we’re seeing time out this is why we are having an increase of response time. So very easily just by setting up a resource monitor on the box I’m hitting I can already tell you that that box is not fit for you for production, for production. But just to show setting up monitor, server group monitor and using dashboard and widget, you can already start to get some answer about the performance of your application. And that’s really cool, I mean here I’m only monitoring 1 server but I can show you the result of the same wordpress app but on a bigger machine with multiple web servers handling 10000 virtual users. So here you can see CPU percentage for all the boxes we have and we have a lot of web servers. Here you can see them. Right, so that’s a more realistic test – that’s my memory usage for all the servers, CPU percentage on top of virtual users so that you can see that actually it was really good – look at that, 10000 virtual users and we only have CPU percentage 40% so plenty of room to grow. But you know that environment was ready to handle production type load which is good. But my demo is only 1 server and definitely not enough to handle that traffic so here you can see its obvious that we’re having issues very quickly actually. So if you’re using CloudTest Lite and you’re not monitoring today, please start whether you get a windows box, or Linux box, or AIX box or whatever box, try to get credentials to the server and to start really get the whole story because if you don’t monitor your infrastructure I think, then you’re actually missing a big part of the story. Download SOASTA CloudTest Lite today and start monitoring!