Managing a lab for performance testing … Feel the pain !

server room Managing a lab for performance testing ... Feel the pain !

I’ve always wanted to write a post about an important aspect of performance testing: Budgeting your hardware environment for Performance Testing. While managing a QA organization at Experian, it’s been one of the least enjoyable part of my job. Managing and growing people, sure ! Anytime ! Managing a large hardware environment? mmm .. Not so much. But it’s necessary to build it and manage it seriously if you want to run proper performance testing. The fact that I had to manage a large hardware budget during an economic downturn was actually a good thing as I had to be creative to find alternative and I became pretty damn good at rolling on the floor to get what I need from our finance guy … icon wink Managing a lab for performance testing ... Feel the pain !

There are multiple problems you face when you handle a large hardware lab dedicated to performance testing:

  • It’s godamn expensive! – I should work in the hardware business. amazing margins ! Lock-in ! 3 years and you can throw it away. Damn …
  • You NEVER know how much you need – You never know what type of customers your going to have to test for next. You have pretty good results with one set of hardware and then you have a large financial institution asking your performance results on the latest and greatest IBM Box. They don’t want to buy your software (nor the hardware) before knowing what to expect. Extrapolation of results? BLACK MAGIC at best. You end up in very uncomfortable situation …
  • You NEVER have enough – If your QA organization start doing performance testing, and doing a good job at it, you will never have enough hardware. Maybe the thing is to never start icon wink Managing a lab for performance testing ... Feel the pain ! (and unfortunately a lot of organization decide not to invest in performance testing). You start doing performance testing for one product, two … Doing a good job, need more hardware … Budget cycle come. It’s tight … You need to prioritize amongst all the products you’re responsible for. Sales and marketing scream at you because they don’t get numbers or figures they need for pre-sales. People above you shift the strategy (with good reasons) and you have to re-prioritize all your efforts, capacity … In any case, you NEVER have enough hardware to cover everything you want to cover. It’s an interesting excerise to be able to predict, plan … And can get frustrating because you want to do so much more.
  • It’s NEVER utilized 100% -A box dedicated to performance testing is never utilized 100%. NEVER. Especially the largest one. So you think you’re smart and you’re going to put in place a server planning process where you can use spare cycle for something else. Functional testing for example ! Not a bad idea all together but can lead to major nightmares especially if you’re using this box for hard commitment. My recommendation is to never plan these box as part of resources available for testing (functionally) the critical path of a major software release. NEVER.
  • Everyone outside your organization want to use your environment - You’ve done a good job building a reasonable environment, getting some good results out of it, good feedback from customers … You can be sure that some organization with the same kind of need would want to use it at some point. You’re nice, we’re all a big family ! Sure why not ! Go ahead ! We have our server planning process, you can use it during spare cycle. Ooopss … they end up using it for a major delivery, for a customer you can’t say no to, hard commitment etc. You’re stuck. There is a customer down the line and there is nothing you can do … So you start playing tough. People throw rocks at you icon smile Managing a lab for performance testing ... Feel the pain ! Seriously, my recommendation is to help other organization to get ready for the unexpected, sit with them to understand their need and show them their option. If you’ve been there, done a good job, share your knowledge to help. “Give a man a fish and you have fed him for today. Teach a man to fish and you have fed him for a lifetime” (an alternative: “Give a man a fish and you have fed him for today. Teach a man to fish and you can sell him fishing equipment” icon wink Managing a lab for performance testing ... Feel the pain ! )

Cloud Computing change the game of course as you don’t need to know how much you need as you can have all you need (theoretically), you can bring it up and down as needed, it’s wayyyyy cheaper and other organization can quickly create their own environment. Perfect world !! But … What do you do with your existing hardware? After all, it’s pretty powerful, would be good to be able to use it as part of your new environment as added power to simulate virtual users, deploy your tests …

Guess what ??


<shameless plug>
You’ve got “bare metal”? Private clouds? (vCloud? Eucalyptus?) Using EC2? Azure? IBM Cloud? SHAKE IT REALLY WELL !! You’ve got a seamless Performance Testing platform !! Well, you’ve got it if you’re using our latest product SOASTA CloudTest PRO !! It brings you the WORLD’s LARGEST TEST PLATFORM ! Cherry on the cake, it integrate with uTest community of 34,000 testers !! Now, that’s what I call innovation !!!
</shameless plug>

Building and managing a hardware environment dedicated to performance testing is critical and might not be fun everyday. But in 2011, you actually have plenty of new options to help you in your task and make the whole process a bit more enjoyable !

One thought on “Managing a lab for performance testing … Feel the pain !

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>