• The tail of available variety is far longer than we realize.
• It’s now within reach economically.
• All the niches, when aggregated, can make up a significant market.

In his book, Anderson backs up his statement with example taken from today’s various industry: iTunes, Ebay, NetFlix, Amazon, Google, etc. All these company have quickly grasped the concept of “market of multitudes” as Anderson calls it.

Following the article and the original book, a lot of people are seeing the Long Tail everywhere ! There is a long tail in TV ads, guitare tablature (!), college sports, sports in general etc. A hit market in one region of the world could be an unexplored  niche market in another area. The Long Tail is all about the economics of abundance and how to turn unprofitable customers, products, services and overall market into profitable ones.

Bellow a quick overview of the long tail for music.

I strongly recommend you to read the original article and of course the book !

As a little mind game, I’ve tried to apply the Long Tail concept to Software Testing and particularly bugs. Anderson demonstrates that the Pareto principle is not necessarily valid today in retail. Basically, the market of niches could be of greater value than the mainstream market. 20% of the products don’t necessarily account today for 80% of the revenue.

If you’ve been in Software Development and Software Testing, you have realized that the Pareto principle is very much alive. You quickly realize that roughly 80% of high priority issues comes from 20% key causes.

Because these high priority problems are usually highly visible or translate into high financial impact, a software development team will focus a lot of their attention on these. Software Testers will determine areas of high-risk to dictate their approach:

• New functionalities.
• Features with complex business logic.
• Area with security or safety impact.
• Area with financial impact.
• Features identified as high priority for stakeholders.
• Features developed under great time constraint, using new technologies or with lots of code refactoring.
• etc.

Note that I consider high priority bugs and not high severity bugs. A crash is always severe (your software is on the floor damn it !) but depending on how you end up with the crash, it might not be the most important crash to fix. You might want to read the previous sentence a second time and start screaming. Done? Good Feel better ? Seriously, would you rather fix a crash which occur when you start your software on any platform or fix a crash which only occur when you’re running your software on AIX 5.3 Technical Level 2 Patch Level 27 with a DB2 8.2 Service Pack 1 patch 35 and with character set ISO 8859-14 (Celtic Languages)? The answer is obvious but everything could change if your largest customer is actually running under the second environment (you still want to fix the first crash though).

What do you find in your long tail of bugs? Well, it really depends on what is considered a priority for your specific customers or market but generally:

• Bugs occurring on very odd situation and what could be considered as unrealistic path of discovery.
• Bugs occurring on certain combination of environment.
• Bugs occurring on non-supported environment.
• Bugs occurring when testing the software with limits beyond requirement.
• Low priority usability bugs.
• Low priority maintainability bugs.
• Internationalization bugs.
• etc.

Again, the content of your long tail of bugs varies depending on the type of software you’re involved in, your market, your type of customers. There is not one rule.

At this stage, you might ask yourself the following question:

What depth of the tail needs to be addressed? In other words: When do you stop fixing bugs? (In a sense it is related to the “when do we stop testing” question. I recommend you to read this excellent article from Michael Bolton). Again, there is not one single answer (it would be kind of boring wouldn’t it ?). It really depends on a number of factor:

• You’re in startup mode and you want to show your software as soon as possible to get feedback.
• You’re working on a service pack version of a software and you’ve addressed all high priority problems of your highly visible (or high revenue generator) customers.
• You’re working on a brand new product and your capacity and timing allows you to fix as much bugs as possible (this strategy will compete with the possibility to implement new features of course).
• You’ve reached code freeze and your product needs to enter its packaging cycle (if you’re shipping CDs for example)
• You’re insane and you think there is an end to the long tail of bugs
• etc.

To answer the original question: Should the long tail of bugs be revisited? Of course ! Your niche bugs (yeah, you know that one with the Celtic character sets, running on this improbable environment) could become your hits bugs of tomorrow ! This is especially true if you’re working on global products where internationalization could redefine quickly your priority! If you have a product which can easily adapt to different target audience, your priority could move quickly as well !

Having visibility on your complete tail of (known) bugs should help you with the longer strategy of your products as you quickly assess if you can easily tackle a new market with the same product.

On top of this long tail of bugs, you can easily imagine a parallel long tail of test cases but maybe more importantly a long tail of features. Do you think your most popular features generate the most revenue or do you think the game is changing and you should instead focus on niche features to grab a potential larger audience?

Related posts:

1. QA is always the bottleneck. What do we do about it ?
2. Measuring Software Quality
3. Code review is a good investment !
4. uTest.com – Can crowdsourced testing help my organization ?
5. Software quality metrics dashboard

I’ve started my career as a typical Software Engineer, messing up with C++ and Java code on B2B software (good time !). During that same time, I had the opportunity to see Lou Gerstner on stage as he was visiting our IBM campus. It was right in the middle of the e-Business period and Lou was here to explain us what e-Business was about and what it meant for us. It was one hour which changed my life. I was so impressed by the message, the clarity, the energy, the motivation. I knew from that point on that this was something I wanted to do: Being able to have this type of impact on people. being able to share a vision, drive a business, etc. I guess I would have had the same conclusion if I had the opportunity to see Jack Welsh on a stage. He’s also one of my role model.

Setting a target is the easy part, hitting it requires a bit more work ! Especially when you’re a software engineer and you want to become someone like Gerstner. There is not one way to reach this type of objectives and I could have chosen a lot of different way. I was in IBM, looked like it was a good opportunity to learn and experience management so I stuck with it.

I had my first management role about 10 years ago. And boy didn’t I know anything at that time! Between the project pressure and the learning process, I was a bit overwhelmed. I hadn’t received formal training yet and at that time and was looking for some clues. I was fortunate enough to have my manager close by (pretty much next door) and who was very hands-on on the day to day activities of his team. I thought that if that guy was at that position, he was probably doing something right (If you’re fortunate enough, that’s usually the case). So I started observing him, picking up what I thought was good practices, tried to understand the impact he had with some of the behaviors he was demonstrating. The overall team was fairly successful, so he became a good first model. Stepping back, there is one thing he taught me which stands up from the rest: learn as early as possible to manage the emotional bank accounts you have with people you work with.

The term was coined by Stephen Covey while he was giving course on organizational behavior. He also wrote about it in his acclaimed best seller, The seven habits of highly effective people, In a nutshell, Covey tells us that we’re keeping an emotional bank account with everyone we have a relationship with. It is true for the one you love, your spouse, children, family, your friends but also your co-workers and your employees. In a sense, this account represent the level of trust and confidence we have with people.

Any account we have with people will start at a neutral level. As we make deposits in this account, the level of trust and confidence go higher, the relationship grows. If we start making withdrawal from the account, the relationship and the trust suffer. The goal is to have a positive account with people at all time, deposit often so that the relationship will have a greater tolerance to mistakes we might make.

Covey describes 6 major ways of making deposit into all your emotional bank accounts you have with people.

Understand the individual

With employees or co-workers, you need to be able to listen in order to understand what’s important to them, their aspiration, what make them happy. Being able to listen is a very good way to make large deposit in their emotional bank account. Listening is not as easy as it sounds! You need to put yourself in their shoes, understand their thought process, empathize. Remove yourself from the picture. It’s all about them.Listening to people is a good way to understand them. Being there, visible and observing is also another way. I encourage you during the day to walk around the office to observe, touch base with people, try to get the temperature, encourage, make small personal connection, laugh ! It shows people that you’re here and that you care.

Keep commitments

Breaking a promise is one of the largest withdrawal you can make so you better do what you promise to people. You basically need to walk the talk so that everything that comes out from your mouth means something to people.

Clarify expectations

If you want to avoid major withdrawal from people emotional bank account, you want to make sure that expectations are clear. You need to invest time and effort to make sure roles and goals are clear. Some people needs general direction. Some needs greater detail. But in any case, you need to make sure people understand what you expect from them.

Remember the little attention

In a way, this is an area where you can make the largest deposit over time but it is the most forgotten. Smiling, thanking people, small encouragement, kind words etc. These are all small things we should all do as individual. It shows that we care, that we’re aware of others. All these deposits are small but numerous. You can really load an account with these. Think about it as the long tail of emotional bank accounts ! (I’m currently reading the Long Tail … ! )

Show personal integrity

You could do perfectly all the previous points, if you don’t demonstrate integrity your accounts are doomed. Since they represent trust and the strength of a relationship, if you lack integrity your balance will become overdrawn. Don’t talk behind people, don’t criticizes or complain (be constructive), don’t be rude or aggressive, don’t use people mistakes as weapon, don’t hold grudges, don’t take offense too quickly, forgive, be loyal etc. Be whole and complete and people will trust and respect you.

Apologize with sincerity

Being a good person every day is difficult. We make mistakes. Sometime we’re going to offend someone, break a  promise, forget to thank someone, we’re going to be angry, not clear enough etc. We’re human, we screw up, and it’s ok as long as we’re able to recognize our mistakes and apology. Sometime, an apology completely offset a withdrawal and makes it a deposit in an account !

All these concepts are kind of obvious when managing a relationship but applying them requires awareness and effort. Integrating them into your management best practices is definitely a good investment !

Related posts:

1. Jack Welch: Straight from the Gut
2. Time management: Who’s got the monkey ?
3. Building high performing Software Testing team – 90 days plan (part 1)
4. Building high performing Software Testing team – Part 1 – Organizational climate
5. Behind the cloud is a great read !

Almost a year ago (wow, times really fly !) I’ve shared with you an email management system I was starting to use. During this year, I’ve shared this system with as many people as possible as I think it is really excellent. I don’t see myself today not using it as it did help me being 10 times more efficient when dealing with the pile of email I receive everyday. Some close colleague were as excited as I was and made a few improvement. In this quick post, I wanted to show you how I was configured and hopefully it will encourage you to start using it !

We all know that all email don’t have the same priority. In my opinion, if you can read and answer an email in less than 30 seconds, it might be useful to treat it on the spot. However, most of them need be treated differently based on their urgency or maybe some require more thoughts from you to answer. For me, priority goes from: I need to answer this email now, today, tomorrow, this week or it can wait next week. For some it can wait even longer. That’s the first important objective of PIFEM: Organize your mailbox so you know at any time during the day which email requires your attention. You don’t have to scan through your inbox to remind your brain what you need to take care of. PIFEM does it for you. I really don’t want my brain to be used for this hence the automation.

The whole system makes use of search folders in Outlook. Every new email will fall into the To Be Checked search folder you had created previously with the following settings:

Then you need to create search folders for email you need to take care of today, tomorrow, this week, next week or later. Below are the search folders settings I’m using for each:

Today

Tomorrow

This week

Next Week

Later

Whenever a new email arrive on your To be Checked search folders, decide what you want to do with it.

With PIFEM, there is no way you can forget to answer an email. That’s the beauty of it! The problem is that not everyone is using it and some people tend to forget you had asked an important question which require an answer. You can setup PIFEM to make sure you follow-up on people who are a bit disorganized.

You simply need to create 2 (or more, depending on how granular you want to be) new categories and the associated search folders.

New categories

Search folder for the answer expected this week.

Create another one for answers expected next week with the associated category.

When you write an email with some questions and really want to follow-up, go to your send items and set the appropriate category. The email will go directly to either Reply expected this week or reply expected next week. Then you can chase people for your answer !

Last advice, you can set your quick click to complete so that when you go through your email you can quickly set them to complete. You can also setup a keyboard shortcut to make it even more efficient ! (mouse is a waste of time sometime)

I’m sure this email management system can be optimized a bit more and adapted to your own need. If you have some suggestion, send them my way and I will definitely update this post.

Related posts:

1. Pay It Forward Email Management (PIFEM)
2. Managing your workday: Quality over Quantity
3. Time management: Who’s got the monkey ?
4. Software testers and the nine forgettings

Damn ! I just realized that my latest article is almost 1 month old ! This is unacceptable for my trusted readers and as I spend a night in London I decided to share one of my latest read. I did read during the past months but for once, I indulged myself with some great science-fiction books I had in my stack: The forever war from Joe Hadelman (always trust a classic. This is an awesome read !), the Ophiuchi Hotline from John Varley and The Currents of Space from the man himself, Isaac Asimov (I’m a huge fan …).

But I finally took some time 2 days ago to pick up my copy of “Behind the Cloud: The untold story of how salesforce.com went from idea to billion-dollar company and revoluzioned and infustry” from Marc Benioff, Salesforce.com CEO, and I actually couldn’t stop reading it ! I’ve just finished it on the plane to London and I have to say that I had a blast !

I’ve always been interested about Salesforce.com as I was reading a lot of articles about the agile transformation they made some years ago. As my organization is going throught such transformation I always been impressed and inspired by what they did. By picking up this book, I wanted to understand the bigger picture and how Marc Benioff pulled it off 10 years ago. He was the first one to have this vision around Saas and then Paas. With the strong emergence of cloud computing the past few years, I wanted to understand why this guy got it right way ahead of everybody else.

The story is a bit different from the Google and Facebook story. Marc was already a successful executive at Oracle when he built the company. He seeded the company with $6 millions which he had saved from working at Oracle and various successful technology investment he made. That’s pretty impressive if you ask me and it does help implement a great idea in the first place. The actual great idea is actually priceless.

The book is divided into playbooks and gave me some great insights into the whole process. I’m not at this stage into a startup mode (I’m working for Experian Decision Analytics, part of the Experian group, a 15k+ people company. Not what I would call a startup!) but there are some really nice ideas which can be applied to the corporate world. I’m a strong believer that it doesn’t matter if you work for a large company or a very small one. Entrepreneurship spirit can and should be encouraged in either one of them.

The playbooks are the following

The startup playbook: How to turn a simple idea into a high-growth company

This playbook really apply to startup but can also be applied to new products your company is building. I mean, do the following ring a bell whether you work in a startup or in a large company?

• Have a big dream.
• Work only on what is important.
• Listen to your prospective customers.
• Defy convention.
• Hire the best players you know.
• Take risk.
• Think bigger.

From my perspective, these plays are a given in any size company.

The marketing playbook: How to cut through the noise and pitch the bigger picture
The events playbook: How to use events to build buzz and drive business

I combine both playbooks as they go along very well. I’m not, by far, the marketing specialist. But reading about it is really appealing. When you work in software development, you’re proud about what you design and build. But who really care if you’re not able to market it? To go against your competitor? To tell a great story about what you’ve built? To be innovative to separate yourself from the pack? In these 2 playbooks, there are some great advices for any startup to build up their brand. Some can without a doubt be applied to larger company.

The sales playbook: How to energize your customer into a million-member sales team

This was an interesting part of the book, but nothing really new under the sun. I don’t think Benioff really revolutionized the sales process but applied best practices to engage customers, make them Salesforce’s partners and grow the seeds. I’m not a sales expert but from my perspective what I read was sales by the book.

The technology playbook: How to develop products users love

I consider myself as technology driven so you can bet that I was particularly interested by this chapter. But more than technology, this part is all about innovation and users. It reminds us to  have the courage to pursue innovation, to set a strong (and simple) foundation (can you spell KISS?), to reuse as much as possible (Can you say open-source?), to transcend technical paradigm, and most important of all in my opinion: Harness customers’ ideas.

The corporate philanthropy playbook: How to make your company about more than just the bottom line

This part is perfectly placed within the book and remind us all than a company is more than revenue and profit but also about what it can give back to the community. I was actually astonished by what Salesforce.com, a fairly young company, is able to bring to the community. This is a great example of corporate citizenship.

The global playbook: How to launch your product and introduce your model to new markets

This is one of the best part I would say. It tells you how Salesforce.com was able to scale globally without overspending, to adapt to different market (either through partnership or by replicating what they did in the US), how they’ve handled some IP dispute (especially in Australia) and how not to use a seagull approach (Swooping in, messing up the place, and flying away) but commit yourself fully to your new reach.

The finance playbook: How to raise capital, create a return, and never sell your soul

There are some great concepts Benioff reminds us in this part:

• Everything takes twice as long and cost twice as much as you’d expect.
• “Revenue is vanity; profit is sanity” only apply to well established company. Not startup (duh !)
• Be innovative and edgy in everything you do. Except when it comes to your finance. This is a very interesting part giving some details around the IPO process and why they chose to go to the NYSE vs the NASDAQ (”We need the credibility and the panache of the NYSE brand; it’s traditional, old-line, well established; it’s the antithesis of salesforce.com“)

The leadership playbook: How to create alignment, the key to organizational success

I was very eager to read about this part as this is maybe my top priority in my current position. I’m always open to new idea s !

I have to say that I’m not disappointed with what Bernioff has to share with us as leadership was a key foundation to his success (how else could it be?)

He shares with us what he’s used to formalized his management process. He’s called it V2MOM which stands for Vision, Values, Methods, Obstacles and Measures. This is something he had used to guide every discussion at salesforce.com. From 1999 to today ! As an example, here is what V2MOM looked like back in 1999:

Vision (What do you want?)

• Rapidly create a world-class internet company/site for Sales Force Automation

Values (what’s important about it?)

• World-class Organization
• Time to market
• Functional
• Usability (Amazon quality)
• Value-added partnerships

Methods (how do you get it?)

• Hire the team
• Finalize product specification and technical architecture
• Rapidly develop the product specification to beta and production stages
• Build partnerships with big e-commerce, content, and hosting companies
• build a launch plan
• Develop an exit strategy; IPO/acquisition

Obstacles (what might stand in the way?)

• Developers (I did chuckle with that one )
• Product manager/Business development person

Measures (How will you know when you have it?)

• Prototype is state-of-the-art
• High-quality functional system
• Partnerships are online and integrated
• Salesforce.com is regarded as leader and visionary
• We are all rich

Benioff rewrites and communicate the V2MOM every 6 months.

The rest of the chapter reminds us about some great concepts to implement when building a great leadership team:,

• Build a recruiting culture: Hire a HR manager right away. Hiring is a top priority. That’s true in a startup and in a large company.
• Use aggressive interviews: 360-degree interviews, have your candidate presents so you can check how they perform on the fly, how they handle curve balls.
• Make sure you hire people who fit the culture you’re trying to grow. Typically at Salesforce.com they had trouble hiring people coming from the client-server world.
• Make sure everyone in the company understands the importance of Mahalo: Mahalo is the Hawaiian spirit of gratitude and praise. I can’t emphasize enough the importance of telling someone “Thank You” in any corporation, large or small. I’ve been astonished in my career by some people not understanding the value of such a simple concept. This is particularly important in a tough economy when monetary rewards  are a bit more difficult to give.

“Behind the Cloud: The untold story of how salesforce.com went from idea to billion-dollar company and revoluzioned and infustry” from Marc Benioff is a fantastic read I recommend everyone to read ! Whether you work for a small startup or a very large company !

Related posts:

1. The Art of War – Sun Tzu
2. Jack Welch: Straight from the Gut
3. Founders at Work: Stories of Startups’ Early Days
4. Richard Branson: Losing my virginity
5. Building high performing Software Testing team – 90 days plan (part 1)

As the web is becoming more ’social’, we’ve seen a lot of change in the data space:

• Data volume is becoming larger and larger: Google manipulate 20 petabyte of data every day. Facebook handles 20 petabyte for their 10 billions photos !
• Data is getting more and more connected due to its social trend. It means a huge number of joins (I don’t have to remind you that ‘joins’ is the ennemy in the DB performance world)
• Data is becoming less structured.
• Need for scalability and fault tolerance is exploding. Geographical distribution is also key if you want to be global.

While relational databases bring a lot of benefit especially around Atomicity, consistency, isolation and durability (ACID), is well proven and the industry is very familiar with it (programers, DBA), they unfortunately hit a wall when dealing with Web 2.0 requirement. Their schema is rigid (lack of flexibility), joins are really slow, it’s difficult to distribute the data across nodes, the optimization you can introduce break the benefit of normalization ie. data integrity and downtime is not acceptable. But the major issue today is scalability. RDBMS scale reasonably well vertically but there is so much you can do with one huge system. If you want to meet the social web requirement, you will have to scale horizontally. It does offer flexibility but is definitely more complex: You need to group your data by function and spread your functional groups across databases. You then split your data within functional areas across multiple database (sharding).

It’s been demonstrated by Eric Brewer, who is a professor at the university of California that distributed system can have only 2 of the following characteristics:

• Consistency: Perception from the users that a set of operations occurs all at once.
• Availability. All operation must be performed with an appropriate response time.
• Partition tolerance. All operation must complete, even if one component/node is down/broken.

These characteristics are known as the CAP theorem. Since horizontal scaling is based on data partitioning, there is a trade-off remaining between consistency and availability. ACID database transaction doesn’t allow this tradeoff but non-relational databases address this limitation by introducing a new principle: BASE which trades some amount of consistency for availability.

• Basically Available: Appears to work all the time.
• Soft state: It doesn’t have to be consistent all the time.
• Eventually consistent: At some stage it will reach consistency !

Don’t you love it? While ACID is pessimistic and forces consistency for all operation, BASE has an optimistic view and assumes that inconsistent operation will occur (hell, shit happens !) but will reach a consistent state at some point. It seems a bit loose and difficult to manage but this is why non-relational database come to the rescue to implement smart consistency pattern and help you reach scalability you couldn’t dream about a few years ago !

There are today 4 main trends in the non-relational database world which dominate the space:

Key-Value databases
Entries are stored as key-value pairs in large hash tables. Domains (possible values of an attribute) are similar to those found in table but no specific schema is defined. Keys are arbitrary while values are blobs. There are no explicit relationships between domains. You access keys and values through API (SOAP, RESTful). Integrity is guarantee by the application itself.

Major open-source and commercial Key-values databases:

• Dynamo (Amazon)
• SimpleDB (Amazon web services). Written in Erlang !
• Voldemort (LinkedIn)
• Memcached: In  memory key-value store. All the major web players are using it: Facebook, Twitter, YouTube etc.

Column-oriented databases
Entries are stored by column versus row. It brings you a big performance uplift when you need to query many rows for smaller sets of data (not all columns) and it maximizes disk performance (read scans). It’s definitely not the right choice if you need to query all columns of a single row or need to write a new row with all column data supplied.

Major open-source and commercial column oriented databases:

• Cassandra: Facebook
• Big Table: Google
• Hypertable: Open-source implementation of BigTable.
• Hbase: Open-source implementation of BigTable.

Document databases

This was inspired by Lotus Notes and very similar to key-values stores. Each DB record is stored as a document (JSON). DB is schema-less and highly denormalized.

Major open-source and commercial column oriented databases:

• CouchDB
• MongoDB

Graph databases
In this model, entities are stored as nodes and edges. Nodes represents entities while edges represent relationships. It’s basically a key-value store with full support for relationship.

Major open-source and commercial column oriented databases:

• Neo4j

the whole noSQL (ie. Not Only SQL)  hype is picking up a lot of steam right now with the acceleration of the social web. The big relational DB players are already playing with it (IBM’s M2 corrals massive data sets with Hadoop). What will happen to your favorite RDBMS? Should they leverage Memcached or JBoss cache to get an uplift in scalability and performance while relying on their RDBMS engine to maintain ACID properties? is this the right balance if you don’t have Facebook or YouTube performance requirement?

If you’re interested to read more about noSQL (I know I’m hungry for more !), I definitely recommend the following read:

• Amazon Dynamo paper a classic.
• Bigtable paper. From Google.
• Lots of video presentation and Slides from the 2009 NoSQL conference.
• If you’re scratching your head around eventual consistent (I know I am !!), this is the article to read.
• The CAP Theorem.
• One Twitter for the latest news about NoSQL: @nosqlupdate is a must.

Related posts:

1. High performance at massive scale – Lessons learned at Facebook
2. Grand Central Dispatch – Future of thread programming?
3. Oracle-Sun hearing starts today while Sun ships new Java EE version
4. Google launches an open source programming language
5. State of the economy is helping Agile development adoption

The European Commission has started to listen to Oracle argument for its planned acquisition of Sun Microsystems. All eyes will be on Sun’s MySQL business unit as the talk held last month ended up in an impasse as the EC is concerned that the acquisition will seriously reduce competition in the hot databases market. Oracle will be helped by customers such as Vodafone, The UK National Health Service, BBV etc. Against this merger, you can find Microsoft and SAP.

Interestingly enough, IBM seems to not care that much, according to Steve Mills, head of IBM software division who doesn’t consider MySQL as a competitor to Oracle (or IBM DB2). When you know that MySQL is the back-end behind Facebook and Youtube, you’re wondering about Mill’s comment:

It’s certainly not the most sophisticated database in the marketplace,” Mr Mills said. “It’s not used for sophisticated query and analysis things.

All Java developers wonder also about the fate of NetBeans and it is currently very difficult to predict. Personally, I’m optimistic in a short-term perspective. But longer term, I can’t see Oracle investing heavily on NetBeans when they’ve been pushing so much their JDevloper environment in the past and their Eclipse-based plug-ins. Could JavaFX save NetBeans as it is today the only IDE for building JavaFX application? More likely, they will add support for JavaFX into Eclipse … That is assuming JavaFX become a viable alternative to Ajax. That’s a lot of if …

Interestingly enough, Sun has just shipped the latest version of their key Java platform products: Java EE 6, GlassFish V3 and NetBeans 6.8.

Java EE 6 brings a new features called profiles that allows web application deployment scenarios with developing a custom stack. I see this as a generic stack for specific application scenarios you can extend. It gives you a starting point and speed up your deployment.

GlassFish V3 supports Java EE 6 profiles of course and improve its startup time and add monitoring capacibilities.

Related posts:

1. Google launches an open source programming language
2. Grand Central Dispatch – Future of thread programming?
3. High performance at massive scale – Lessons learned at Facebook
4. Musings on NoSQL
5. Software testing under NetBeans environment

“Oh, and by the way we’re launching our own programming language …” This is the latest news coming from Google today after a tornado of announcement last week: Acquisition of Admob and Gizmo5, new personal dashboards, free WiFi in US airport during holidays (Hey, Yahoo gives free Wifi too !), lower price of online storage etc. It looks like Go comes from an 20% time project R&D project and is becoming today a project opened to a broad community with formal support from Google. Here are its major attributes:

• Compiled language very close to C but with some inheritance possibilities.
• Use dynamic types close to what you would find in Python and JavaScript.
• Offer a better way to program for multicore processors based on CSP (Communicating Sequential Processes). It’s something you can find in language like Occom and Erlang. It’s probably very different from Grand Central Dispatch but offering a better way to take advantage of multicore is definitely a hot topic right now.
• Will offer faster compilation time by handling dependency between modules in a very simple way (can’t be worse than header file in ‘C’ …)
• Interestingly enough, they’ve provided compiler support for ARM processors. It’s worth noticing that ARM processors are the dominant kind in the mobile-phone market and as we know, Google is becoming a clear player with Android. Go softwares will be able to run on Android.

Pike, who is one of the lead software engineer for the project declared that: “I don’t think we’ll replace anything, we’re just putting another player into the arena.” … You will draw your own conclusion but I think Go is the tree which hides the forest. And another smart move from Google … Where will they stop ? As usual, Google has release a one hour long presentation about Go. You can watch it bellow. If you want to write your first “hello world” program in Go, you can check out their Go Tutorial. ArsTechnica has also a good article about Go.

Related posts:

1. Grand Central Dispatch – Future of thread programming?
2. Google outage: A broken cloud !
3. Oracle-Sun hearing starts today while Sun ships new Java EE version
4. Another good one from Joel ! Stackoverflow.com
5. Musings on NoSQL

Performance and scalability are high on the list of top requirement from the customer we deal with. The largest financial institution use our products at a global scale with access through the web. They expect low latency and of course a solution that can cope with future volume of utilization.
There are today 2 Internet giants who are facing scalability challenges every single day: Google and Facebook. I’ve had a chance to touched on some of these challenges talking to some of the Google engineers last week during GTAC 2009. Google is approaching the crazy number of one million servers so you can bet they have to be creative to handle their data. And deal with the carbon tax.

I’ve just finished watching a very informative presentation from Jeff Rothschild who is VP Technology at Facebook. He gave this presentation at the University of California this month. The actual presentation (75 minutes) can be watched here.

Here are some of the key things I was interested in.

Photo storage

This is of course one of the most storage consuming application on Facebook. There are about 850 millions photos uploaded to the site each months in various resolution ! So yes, storage is a challenge. Facebook former system was based heavily on CDNs from Akamai and Limelight as well as file handle cache placed in front of NetApp. A memcache storage layer was introduced to reduce the load on the NetApp filers.

This system was still generating way too many IOs (3 per each photo read which is was you get when you rely havily on a file system) and couldn’t handle the ever increasing volume of data. Facebook decided then to develop their own solution called Haystack which can be seen as a blob store.

Haystack stores photo data inside 10 GB bucket with 1 MB of metadata for every GB stored. Metadata is guaranteed to be memory-resident, leading to only one disk seek for each photo. Haystack servers are built from commodity servers and disks assembled by Facebook to reduce costs associated with proprietary systems.The Haystack index stores metadata about the one needle it needs to find within the Haystack. Incoming requests for a given photo asset are interpreted as before, but now contain a direct reference to the storage offset containing the appropriate data.Cachr remains a first line-of-defense to Haystack lookups, quickly processing requests and loading images from memcached where appropriate. Haystack provides a fast and reliable file backing for these specialized requests.

If you’re interested in Haystack, here is a good starting point from Facebook.

Access to data and caching

Facebook relies heavily on memcached to minimize their database load. They of course use in memory hash table running on 64 bits machines, efficient serialization, multi-threading, compression and polling drivers. What’s interesting with this approach is that the switches connected to all the memcached server become the bottleneck ! To alleviate this problem, they perform client-side throttling.

Database

Facebook uses MySQL servers distributed in multiple data centers (In case anyone doubt the scalability of MySQL. This is a clear demonstration.). They avoid like the plague shared architecture so they can manage failures more efficiently (basically avoiding storing non-static  and heavily referenced data in a central database). They use services and memcached for global queries. In order to scale across multiple data centers, they replicate as well their memcached data to deal with race condition (This is I believe the most challenging aspect of social networks).

Hardware

Difficult to get a clear number but iit looks like they have around 30k machines (800 memcached servers !). The typical hardware specification for a Haystack server is a 2xquad-core, 32 GO RAM, 512 MO of NVRAM cache and 12 1TB SATA drives in RAID6. That’s 10TB in XFS. They don’t rely on external provider to manage their environment which is farily typical with the large internet company.

Open-Source

Facebook makes good use of open-source software: Tornado, Scribe (for log aggregation. Nice !), Cassandra, Thrift, xhprof, Hive, MySQL, Memcached etc. and might release their Haystack architecture as open source as well.

More on the open-source stack facebook is using here.

The presentation is definitely worth the 75 minutes. If you’re passionate about web scalability, this is a good start ! I also would like to recommend a good stack of slides I’ve been reading a while back. A lot of good best practices to take into consideration if you’re in that space.
Real World Web: Performance & Scalability

Related posts:

1. Musings on NoSQL
2. Oracle-Sun hearing starts today while Sun ships new Java EE version
3. Testing in the cloud – Amazon EC2
4. Google launches an open source programming language
5. Cloud computing services are hot in 2009 !

GTAC 2009 is over. Hail to GTAC 2010 (I’m hearing Hyderabad, India … hint hint !). Let me tell you that GTAC 2009 was quite a ride for me (and for most participant I would say) and one of the best conference I’ve ever attended. Ok the best one. There, I’ve said it. The talk themselves were of high quality with excellent content and great presenters but everything else was on par, from the great discussions I’ve had with attendees, the lightning talks, the roundtable during lunch and the fantastic overall organization from the GTAC organizational comitee. So, Kudos to Google !

You’ll find bellow some comment and observation about the conference and the talks.

Let me talk briefly about the Zurich office. Google has a tradition of having some of the best working environment in the world and Zurich follow this tradition. It’s definitely on par with the Mountain View Googleplex I had an opportunity to visit some while back. To give you a quick idea, this is what I’m talking about (Official pictures from Google. It was forbidden to take any pictures in the premises. Trust me, it does look like this )

Some detail about the conference room. When I first came in the room I thought “what the hell were they thinking” ? Instead of having a typical large vertically deep conference room, they have it organized horizontally with the speaker in the middle and many large screens on both side and only a few row of chairs. After 15 minutes I was convinced that this is a much better space organization. Everyone is close to the speaker, can see him and interactions are much easier. Loved it.

Each GTAC participant was given a Google Wave account so they could comment live during talks on a dedicated wave. A nice way to try this new Google Application with a crowd of testers. I was really impressed by the capabilities it offers during conference such as GTAC. Kind like a real time wiki. From what I’ve seen and tried, it is still a bit unstable but very promising nevertheless. You can find all comments written during the conference on Google Wave and searching for: group:gtacgroup@googlegroups.com

The roundtable during lunch was a real good idea. Each table had a theme and you could sit during your lunch to discuss various subjects: metrics (You can bet I was really interested by this one !), testing in the cloud, testing education, functional testing etc. This setup is a nice way to break the ice between participants and of course share knowledge on these subjects.

Right before dinner on the first day were organized some lightning talk. 8 participants had an opportunity to talk about any subject during 5 minutes top. An interesting exercise I had never tried. My talk was a bit too formal to my taste, compare to some excellent one !! I will post my favorite when Google make the recording available (I believe they have recorded the lightning talk as well).

There was around 100 participants which is my opinion is a perfect number to have good interaction and good networking opportunity. From what I understand, previous editions of GTAC were much larger. I think they will keep this smaller format for future edition.

Now to the talks themselves. I won’t describe each of them but would encourage everybody to watch all of them as they were all excellent ! All the presentations slide are available here. Videos will be posted here.

Keynote by Professor Nicklaus Wirth

Professor Wirth is the designer for the Pascal language, so definitely one of my hero for sure ! He gave us a nice overview of the history of computer science and how things have evolved throughout the years. It reminded me that learning about our history was one of the nine forgettings from Lee Copeland. A very nice touch and always good to listen. For him, the focus is still too much on finding errors (test the code) rather than avoiding them (write good code). He also opened up a fairly controversial subject by stating that university don’t know how to teach programming since teacher were not programmers themselves. A large debate by itself.

Fighting Layout Bugs – Michael Tamm

Probably one of the top presentation. Not only the content and techniques described during the talk were quite excellent, but Michael is definitely a very good presenter with a very fun and entertaining style and some very subtle facial gimmicks ! Definitely worth seeing.
His technique isolate text and border from the HTML by doing bitwise operation to identify overlapping elements. Simple yet very effective ! Michael gave a quick but very convincing demo and received a round of applause right in the middle of the talk. A first ! You can find a prototype for Michael work here.
Another suggestion made by Michael was to validate both your HTML and CSS during your continuous integration by using the W3C HTML validation service and W3C CSS validation service.

Testing Applications on Mobile Devices – Doron Reuven, uTest CEO

uTest business model was not a surprise for me as I’ve gotten interested in the space some while back but it was very interesting to hear it described first hand by Doron himself. As the mobile market is exploding day after day, the crowdsourced testing model does make more and more sense and Doron did a good job at presenting us the challenges of testing these new application but also the benefits of having access to a broad capacity of professional testers all over the world to handle all the device model, carrier, languages etc.

Selenium: to 2.0 to beyond – Jason Huggins and Simon Steward

There was a lot of expectation from the assistance for this presentation. All of us are familiar with Selenium and Webdriver and were expecting some exciting announcement. We were not disappointed ! Jason (ex-Googler, now owner of SauceLabs) and Simon (current Googler and probably stand up comedian in another life. The man behind WebDriver!). They did a good job at presenting us the pros and cons of both tools and lead us to a clear conclusion: Let’s merge WebDriver into Selenium to keep all pros from both tool and get rid of the cons. Not only will they merge the code base (into a Uber framework) but also will try to get better at socializing both community ie. Common developer mailing list, common bug tracking, co-hosting of the code etc. More information about the initiative here.

Score one for Quality – Joshua Williamd and Ross Smith, Microsoft

It was very refreshing to have Microsoft presenting at a Google conference. Microsoft and Google fight hard in the field but share quite a bit at a research and development level as well as toward quality. We, the users, benefit from this collaboration.
Josh and Ross both worked hard on Windows 7 and described to us some of the games they’ve introduced for some of their test activity to better engage developers and testers, improve morale and motivation.
One interesting one was the one they’ve used for their localization testing, well at least the translation piece. They basically had a tool presenting screenshot to candidate who were asked to say yeah or nay based on the accurateness of the translated text. They kept a leader board and the only reward was to be at the top of the leader board.

It was an interesting talk for me as I had given a lightning talk the day before, specifying that bug challenge was a dangerous game if not managed properly. And I actually stand with this opinion. I get the motivation and engagement piece but as Josh and Ross mention, it needs to be a very short timeframe with clear reward mechanism ie. basically some bragging right at most. If you don’t do it properly, you end up with meaningless bugs all over the place with a heavy atmosphere of competition between your participant. Plan a lot of trial and error.

GTAC 2009 was a great edition and is definitely becoming one of best conference on Software Testing everyone need to try to attend !

Related posts:

1. GTAC 2009 in Zurich
2. GTAC 2009 – Call for attendance
3. Cloud computing services are hot in 2009 !
4. My favorite software testing experts on Twitter
5. Software visualization for better testing

We’ve recently entered a pivotal era where our networked applications and other collaboration platforms becomes completely embedded into our work environment. Forrester refers to this era as “IT everywhere”.

The author of the study has identified 15 trends that will accompany this new wave of innovation.

Social Computing

I’m particularly interested in this area as we’re working in a distributed and agile environment (SCRUM). Everything aiming at reducing distance and bring people together gets my attention.

• Collaboration platform will move from a model based on documents (Notes, Sharepoint) to a model based on individuals (such as SocialText or blueKiwi).
• Customer community will be gradually integrated into business applications either to centralize exchange on different medium (ie. unified messaging system) or/and to allow collaboration across business (ie. Extraprise)
• Telepresence will go one step beyond: HD video conferencing, virtual workspace etc.

Data and Analytics

I’m working for Decision Analytics so you can bet this is an area I’m watching closely. I’m especially keen to understand how data movement and exploitation can help with some of the service we provide (Credit Bureau Data, Marketing data etc.)

• Real time information feeds and collaboration (enterprise microblogging for example)
• Real time Business Intelligence: Discussion monitoring, Audience analysis etc.
• Universal data model to facilitate their movement (feed) or exploitation (mashup).

Next generation of tools and software

• SaaS integrated into critical business appplication (Financial application? That’s something I’m looking forward to)
• Data on the cloud and PaaS to accelerate deployment and allow scalability on demand. If you follow this blog, you know that this is something I’ve been working on. I’m glad to let you know that we today use Amazon EC2 to complement our hardware environment. I’ll probably update you in a while with our findings.
• Client virtualization to reduce cost and increase performance (or so they say …). I believe IBM is one of the first to explore this space commercially.

Mobility

This is something I want to see going mainstream. I don’t think the challenge is around technology or sofware but more a shift in mentality. This is especially true in Europe where I don’t see a lot done to promote and allow remote work. Definitely a difference with the United States where this is fairly common.

• Mobile version of business application and process. It’s not so much about replicating all functionality but to offer a functional degraded version.
• Access to work environment from everywhere for everyone needing it. This “luxury” (It’s definitely a double edge sword …) shouldn’t be only offerred to executives.

There are additional trends but this article focus on Enterprise 2.0 direction. I wish I’ll be able to update this post with additional information. I’ve not been able to put my hand on the actual report. I do accept donation to buy the report ($500)

This article is largely inspired by a post from Fred Cavazza who is a well recognized French blogger writting about Enterprise 2.0. He has kindly accepted for me to adapt his original post.

Related posts:

1. Grand Central Dispatch – Future of thread programming?
2. Musings on NoSQL
3. High performance at massive scale – Lessons learned at Facebook
4. Behind the cloud is a great read !
5. uTest.com – Can crowdsourced testing help my organization ?