Performance Testing – When will they learn?

error500 Performance Testing   When will they learn?

If you’re a C-Level executive and care about your brand, please read on. You probably cares a lot about your revenue, your customer satisfaction, retention and conversion rate. If your business model relies on online advertisement, you watch your click-through, page view, number of visitor and hits. Of course you’re cautious about some of your spending, your infrastructure, service and R&D cost.

Well, from what I’m reading lately, it looks like some of you guys (yes you, the C-Executives !) are over looking the importance of your website stability or responsiveness.

benchmark Performance Testing   When will they learn?

These are some article from the past 2 months related to website crashes and/or slowdown.

Is the world cup killing Twitter? (my own article)
Clegg crowdsourcing site crashes in heavy traffic
ESPN’s Server Crashes Due To NBA Free Agent Frenzy
TravelRepublic blames Fabio Capello and England team for website crash
BBC investigates website slowdown
iPhone 4 Pre-Orders Started Today: Expect Long Lines, Website Crashes
Tax deadline extended after county website crashes
Sites crash under World Cup strain
Fedex Website Crashes Due To iPhone Tracking
France’s official website crashes on Bastille Day launch
Flipboard launches ‘social magazine’ for iPad, struggles with demand

We’re not talking small business here ! ESPN, The BBC, Fedex, the French Government (!). You would think they have enough cash and in-house skills to avoid these type of problems. You’d think they’re used to high volume, understanding the impact of a world cup, the launch of a new iPhone or the importance of a tax day on their volume. Nope. Looks like they need a good reminder.

Reasons for these failures ? They’re always the same: Lack of a proper operational infrastructure and monitoring, badly designed website and application layer, volume expectation are not well understood, lack of fail-over solution, untested new features etc. There are a lot of remedies for these potential problems and guess what? A consistent performance testing strategy is one of them. Something that is not only a process but a mindset. Something which should be completely embedded within the company DNA.

Dan Bartow, VP Engineering at Soasta was writting recently:

Performance testing is arguably the most important part of the software development life cycle, yet it’s usually the first part to get cut off the tail end of a project or sprint. This is if performance testing is even being done at all. With experience from working with dozens of the biggest names in high traffic online applications, I can tell you that the percentage of companies out there doing performance testing is staggeringly low.

Dan is preaching his own church here of course, but he’s got a point. Performance testing is still seen pretty much a nice to have or something you’d find at the bottom of the budget sheet. It’s an after thought rather than something very much ingrained in the company strategy.CEOs and Product Management only see the functional aspect of what their company offers and still don’t take into account the potential impact of slow performance or instability.

In 2008, the Aberdeen Group released a study: “The Performance of Web Application: Customers are won or lost in one second“. They’ve found that inadequate performance could impact revenue by up to 9%. Take the average revenues of organizations that participated in the study: $1.3 billion. Performance problems could account for an average of $117 million annually !

They highlight the fact that an additional second of waiting on a website significantly impact customer satisfaction and visitor conversions. Page viewsm conversions and customer satisfaction drops 11%, 7% and 16% respectively ! Who is hiding these figures to CEOs? Do they have visibility on this impact so that they can make a difference and start making the right decisions?

impactofoneseconddelay Performance Testing   When will they learn?
The study identify some important steps to be taken by organization to minimize the occurrence of performance problems:

  • Develop capabilities for a job-role specific view into application performance.
  • Develop capabilities for monitoring and managing the performance of web applications across multiple browsers.
  • Develop tools for load testing of web application.
  • Deploy capabilities for monitoring geographical performance and distribution of content demands.
  • Deploy tools for balancing content demand across devices dynamically.
  • Develop capabilities for synthetic monitoring of web application.
  • Deploy intelligent agents for adjusting caching decisions with traffic load.

From my perspective, this can only happen if a real performance-focused mindset is pushed from the top of the organization. CEOs, are you really listening?

 Performance Testing   When will they learn?

4 thoughts on “Performance Testing – When will they learn?

  1. Hi Fred,

    I’m not a C-Level but I’ve read your post ;-)

    Scalabilty, availability, volume and performance (SLA, etc) should be part of requirements given by products managers
    With this developer team will know what they have to achieve.
    After it will be responsibility to QA team to ensure that all those requirements are reach.

    When your taking in example twitter, “Is the world cup killing Twitter?”.
    You certainly know that twitter answer to more than 3 000 000 000 per “normal” day.
    Being able to answer to 3 billions req/day with “mostly realtime” messages is clearly good performance.

    When I was working in Experian, we faced an issue (reported by delivery !) where Experian product has to be restart every 4 days due to segfault during months
    The best append after… When I was in front of customer when they do a security audit on our software and the guy comes and show me how to be remotely root in 5 min on the server we sold them $$$$ !

    Who is responsible in this situation ?
    Dev team ? Ofcourse they have done big mistakes, but in a such huge compagny what is a QA team for ?

    So, I came to ask to QA Team how do they let those issues in a production ready software, I got this answer:
           “We do not had time to test it a lot because the delivery date was already over”

    When QA team sign production versions they agree to take responsabilities.

    When you are talking about performance and quality, I think that your are preaching your own church too :-)

    Having real results in house isn’t the same game.

  2. Thanks for your comment Julien. What you describe is exactly what I’m trying to highlight. I’m not judging developers and testers who are squeezed to fit 1000 new features in a product with the resulting impact of not having enough time to address important area such as performance. What I am concerned about is the lack of focus (and investment) toward performance there is in a lot of company. It’s a constant fight to squeeze these non-functional requirement into a product backlog or into a company wide strategy.


  3. Well done Fred!

    Thank you for sharing your excellent thoughts about performance testing. I agree completely that C-levels don’t usually understand the business ramifications of the speed of web applications. They tend to classify all types of testing as a matter for the IT department, and unless the site crashes during the middle of their expensive television advertisement, they don’t give it much attention.

    I believe that the senior executive that needs to pay the most attention to web performance is the head of marketing. Their bonus is usually tied to metrics that reflect their campaigns (clicks, conversion rates, etc.), and certainly the availability or responsiveness of their site has a tremendous affect on the resulting income. The Aberdeen study is a great illustration of the value of performance engineering.

    There are plenty of other empirical data that substantiate the impact on profitability of companies with an online presence. E-commerce speed improvements have resulted in 25% more page views, 7-12% more revenue, 16% higher conversion rate, and 5% higher order values. Many other statistics are presented in my blog post entitled “Web Performance Tuning = 10% Profit Increase” at

    Thanks again. I will link to this article from my blog tomorrow.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>