QA is always the bottleneck. What do we do about it ?

bottleneck QA is always the bottleneck. What do we do about it ?
As I was reading this excellent article from Matt Heusser, I couldn’t refrain from chuckling as I could relate a lot from what I was reading as an ex-software engineer, tester and now in a management position. As I’ve experienced most aspect of the spectrum, I can claim some objectivity and wanted to add my 2 cents to the article.

So yes, QA is always the bottleneck (at least the last one …). It is simply due to its positioning in the development life cycle. If you’re in a company selling on the shelf software, QA can be saved by packaging but it’s rare (I’ve had one occurrence while working for IBM as part of the DB2 team. Packaging team screwed up … Allowing more time for testing icon smile QA is always the bottleneck. What do we do about it ? )

So yes, it is a confession. But what do we do about it?

Matt is suggesting that it’s not all about optimizing the testing. It’s about optimizing release. I would say it’s about optimizing every organization involved in the delivery of a software and make sure they deliver their share of quality on time. Matt is only looking at the question from a QA perspective while I think we need to look at the bigger picture. Making sure QA doesn’t become the bottleneck is a team responsibility and by team I mean Product Management, Development, QA, Project Management, management and executives. You need to engage the whole product organization. QA can’t absorb every organization shortfall, including its own. Every step of the development cycle is critical if you’re looking at delivering top quality products.

Product/program management

They hold the roadmap of the software and should define clear business requirements as well as functional requirement. In some organization (or country, UK vs US for example), there is a distinction between product manager (business) and program manager (functional). It might be embedded within the same organization. In any case, they need to provide the business direction and the functional direction (If you’re lacking someone dedicated to functional specification you leave this to your super smart developer who is going to come up with a UI which makes perfect sense if you’re a Vulcan (Ref: Joel from Joel on Software). Something like this.

If these requirements are not clear, incomplete (they will be, but you need some base to work with) or come too late, it will impact the development team and the whole product cycle.

According to new research, success in 68 percent of technology projects is “improbable”. Poor requirements analysis causes many of these failures, meaning projects are doomed right from the start. These are staggering numbers, hitting the high end of the Standish Chaos Report (source) In my mind good management of requirement should be at the top of the priority list in every software organization.


Executives are usually great leaders with great vision. They usually have a good understanding of the market and customer requirement. They ought to provide clear strategic direction for our product. These direction will change based on market opportunity, economic climate, competition. What they need to understand is the impact of ever changing direction on teams. If they don’t have software development background (trust me, some don’t) they need to educate themselves or get help from expert in their organization so they can appreciate change in timing, budget and scope when they request a strategical change. Building trust is key.

Project Management

One of the key responsibilities of the Project Management Office is to first ensure proper planning, organization and management or resources in order to help bring a given project to successful completion. From my perspective one of the possible shortfall coming from PMO and potentially impacting the release is not being able to gather all necessary information from the different organization to make informed decision about the scope, timing and budget of a given project. They want to find the right balance between too much information and way too few. It’s a very difficult function of the development group as they have to rely on all the other organization to do a good job. I’ve seen so many failed projects based on poor planning, poor or non-existent risk management because of low performing PMO function ran by people with no software development background not being able to appreciate risks with the right priority.

Development team

A lot, and I mean A LOT, can go wrong at that level of the development cycle and impact the whole project and make the job of the QA team much harder. In no particular order:

  • Lack of technical specification: Unless you’re involved in the NASA program, the QA team doesn’t need all the details around the implementation but they need to get the right level, especially important for white box testing. Heck, it could be self-documented code (java doc, dioxygene …) or high-level specification. But give me something guys !
  • Non-testable software: Give me access to your API. Document it so I can use it to do some amount of low level testing and automation at that level.
  • Stable build: Getting a crash while starting the software is unacceptable. Getting an incomplete build is unacceptable. There should be some amount of consequence for developers checking code breaking a build. Back in my days at IBM, developers were paying $5 per broken-build. Build stability improved quite drastically very quickly. Having weekly stats about build broker was helping as well. Nobody wants his name on the top of the list. Sorry guys, seems harsh but in my book it is a big NO-NO.
  • Unit Tests, static analysis, Code Review: It’s not all about writting code, it’s about writting good code and development is responsible for this code. Let’s start with a good base.
  • Communication of what’s been developed and fixed: If you don’t automate this part, you’re looking for trouble. I won’t rely on a developer to write report on what was developed and fixed.
  • Missed deadline: Occurs usually when the sizing exercise was rushed or scope was not well understood. To the defense of developers, yes sometime shit happens.
  • Hidden features: That’s my favorite. Developers taking initiative to code a new feature or enhancement without making anyone aware about it. While we should promote initiative, we also want to promote communication.

qabottleneck QA is always the bottleneck. What do we do about it ?

QA Team

So every other organization screwed up, how are we going to save the release? Well, let’s look into our own backyard before saving the release, as a team. What do we find in the QA backyard?

  • Testers testing everything but the critical path leaving it for the end.
  • Testers finding severity 1 problem the day before the release. The bug being there for the past 3 months.
  • Testers testing functionality in isolation without end-to-end scenario.
  • Testers spending days to document their testing for the sake of being audit ready without thinking of the actual value of this document.
  • Testers investing in automation (yeah!) but focusing on the wrong area (booh !) and without any clue around ROI.
  • Testers writing sloppy bug reports, not giving enough information to reproduce a problem.
  • Testers not optimizing their test environment, taking forever to put in place, clean and not investing in automation for setup and tear down.
  • Testers not being able to communicate with developers.
  • Testers not being able to grasp risk based testing and priority.
  • Testers not being able to articulate capacity need to management, commercial tool requirement etc.
  • Testers without enough business knowledge to be able to test the software with a customer perspective.
  • Testers without the right technical level to cover the full necessary scope or needing support from the development team to grasp technical concept.
  • Testers not being able to size their testing effort.

So yes, a lot can go wrong within a test team (well, unless you have a high performing QA team !) leading to all kind of problems with your release. But improving the way your QA team operates is only a small part of the solution. Having a united software organization working as a real team, handling problems with a teamwork approach is the actual target. Matt mentions Challenge=opportunity and I agree 100%. Having a high performing software organization is definitely a challenge but it is one hell of an opportunity to make a difference in the software business. QA is always the bottleneck. What do we do about it ?

 QA is always the bottleneck. What do we do about it ?