It would be interesting to see similar survey for testers! I can bet we would start seeing CloudTest Lite on such infographic. And you know what? I’m 100% confident we’ll see it appear as #1 in one category for 2012. I can’t say more but our development team is cooking something truly awesome … !

Click on the picture for the larger version.

Related posts:

1. The 10 Golden Principles of Successful Web Apps
2. Grand Central Dispatch – Future of thread programming?
3. Software quality metrics dashboard
4. Is the world cup killing Twitter?
5. Thanks to Game Changers, Software Testing is flying high !

As you know, mobile web is hot, hot, hot and mobile might very well replace desktop in the next few years (the jury is out on that one!)

Some latest stats to support this trend:

• There will be more mobile internet users than wireline users in the US by 2015.
• 6B mobile connections by the end of 2011.

However, there is one slight problem which needs to get fixed before reaching this stage: Performance. I guess you’ve noticed, performance on mobile is not all that great compared to the performance you get on your desktop’s browser. As this infography illustrates nicely, users have high expectation and mobile performance is still disappointing to most of them (I’m with them on that one! )

A lot of companies are taking this problem very seriously. At SOASTA, through our partnership with DeviceAnywhere, we’re able to let company understand the impact of load on application running on real devices, connected to real carriers (shameless plug: You can watch how we do it by watching this webinar! This is REALLY cool!). There are good reasons why mobile performance is harder to reach than on desktop and I wanted to point you to 2 fantastic resources:

1/ Use Page Speed to optimize your web site for mobile

The presentation walks you through the network -> parsing -> rendering flow and the unique differences in mobile. It also gives recommendations on how to optimize for mobile. In a nutshell:

• Use an Application Cache
• Defer parsing of JavaScript
• Make landing page redirects cacheable
• Prefer touch events
• Enable Keep-Alive

You can find the slide deck here.

2/ Website acceleration with page speed technologies

Slides only can be found here. The presentation goes through:

• Modern vs older browser optimization
• Key difference between mobile and desktop
• New Page speed suggestions

Suggestions includes:

• Use an Application Cache
• Eliminate unnecessary reflows
• Avoid excessive serialization
• Avoid long-running scripts

If you want additional resources to help you started on mobile performance, I also recommend going at the bottom of this page for great links to articles, tools, etc.

Related posts:

1. The Cloud: A game changer to test, at scale and in production, SOA based web and mobile applications.
2. Hola Barcelona! Meet SOASTA at the Mobile World Congress!
3. Hot Software and Performance Testing Links – Week of April 18th
4. Mobile Test Automation the SOASTA way with TouchTest
5. Why you should join the Cloud Performance Summit

Last year, I’ve watched the talk Fred Wilson gave at the annual Future of Web Apps Miami conference in February 2010.  I’ve stumbled upon it today and treated myself with another watch. This is so inspiring to me that I thought I would share it here. If you have 30 minutes during the day, treat yourself with a really great piece!

The full transcript is available here but here is the gist of it:

• Speed – Speed is the most important feature. If your application is slow, people won’t use it.
• Instant Utility – The application should be instantly useful to its user, not after lengthy configuration.
• Software is Media – Your application should have a personality and a voice people can connect to.
• Less is more – Do one thing really well when you launch. Then you can grow.
• Make it Programmable – Others should be able to build, connect or add value to your application. Think read/write API’s.
• Make it Personal – Users and programmers should be able to personalize their application.
• RESTful – Everything in the application has a URL, and ideally, a very clean and comprehensible URL.
• Discoverabilty – Your application needs to be built from the ground up to be discovered and optimized for social media and search.
• Clean – Make your application visually simple. Embrace the KISS principle to your design.
• Playful – You can turn anything in your application into a game. Anything.

Here is the video

The 10 Golden Principles of Successful Web Apps from Carsonified on Vimeo.

Here are the associated slides

Related posts:

1. What apps are in every developer toolkit?
2. Google outage: A broken cloud !
3. Musings on the Future of Data and Predictive Analytics
4. Innovation at Google
5. Grand Central Dispatch – Future of thread programming?

This week in software and performance testing

A few Observations on structure in testing
Over the years, it’s been common for people in our community to mention exploratory testing, only to have someone reply, “Oh, so that’s like unstructured testing, right?” That’s a little like someone refer to a cadenza or a musical solo as “unstructured music”; to improv theatre as “unstructured theatre”; to hiking without a map or a precise schedule as  “unstructured walking”; to all forms of education outside of a school as “unstructured learning”. When someone says “Exploratory testing is  unstructured,” I immediately hear, “I have a very limited view of what ‘structure’ means.”

Scale Fail
Hilarious video from Josh Berkus at MySqlConf 2011

What is Facebook’s architecture
Quick summary of Facebook’s architecture

uTest partners with Mozilla to build a test case management system
Test Case Management can be trickier than herding cats.  If you work for a small to mid-sized company and everyone’s counting on you to QA the apps to glowing perfection (no pressure, right?), then you’ve clearly demonstrated you’re a supremely organized person.  It’s in your DNA.

Selecting the right CDN for your website
A Content Delivery Network (CDN) shortens that distance between your static content and the end-user.  While the text on most web pages is dynamic, most images, JavaScript, and CSS are static.  These static objects make up a large percentage of the total bytes downloaded for each page view.  By using a CDN, you place static content as close to the end-user as possible.  In turn this decreases the page load time a end-user experiences by leaps and bounds.

Some technical details on IBM’s Watson
IBM Watson’s stellar performance in the Jeopardy! show captured the world’s imagination. The first real world application for Watson involves healthcare. How does Watson address issues that previous generations of tools have not been able to address?

The black team
The computing world was different in the 1960′s. Computers were massive, expensive, and required full-time staff just to keep them running. Product cycles were scheduled in years not months. Tasks modern programming tools do in seconds took weeks. And whenever a new computer model was developed, the operating system and all applications had to be developed from scratch.

What is testing?
I think it’s easy to blur the line between what testing is, and what testers do. I’m not convinced it’s correct to blur the line (or incorrect for that matter), but I do think it’s a frequent cause of confusion among testers

Can your programming language do this?
One day, you’re browsing through your code, and you notice two big blocks that look almost exactly the same. In fact, they’re exactly the same, except that one block refers to “Spaghetti” and one block refers to “Chocolate Moose.”

Related posts:

1. Hot Software and Performance Testing Links – Week of April 4th
2. Hot Software and Performance Testing Links – Week of April 11th
3. Hot Software and Performance Testing Links – Week of March 28th
4. Software Testing conference links
5. Performance Testing – When will they learn?

That’s the way I’ve learned bubble sort for example (yeah that’s the easiest one and not the most optimized)

Well, it looks like there are more creative ways nowadays! Look at these Hungarian performing a bubble-sort! Brilliant. They also have insert, shell and select so far.

Related posts:

1. Code review is a good investment !
2. Another good one from Joel ! Stackoverflow.com
3. Design and usability – A lot to learn from the best
4. Learn to manage your emotional bank accounts and build trust
5. Performance Testing – When will they learn?

As we’re getting close to enter soon the Exabyte age (especially when we’ll have millions of sensor everywhere !), I wanted to get my head around the state of data today and wanted to share with you my opinion about where I think data is headed.

The 90′s have seen software companies competing against each other to come up with the killer apps. Whether it was desktop application, back-office, supply chain management, data management etc. Business Intelligence was already in the place with company such as Business Objects, MicroStrategy, Oracle and they were already helping business to make sense of all their internal data but it was mostly for reporting purposes rather than predictive analysis. The 00′s have seen the explosion of the Internet with its loads of new online services, new business models (with only a few of them which are actually viable), new mechanism to deliver information, better collaboration model, allowing a reduction of cost and improving relationship between business partners. As in the 90′s, competition is fierce with new players coming every day to compete on the same global market. It’s a huge market and everyone would like a piece of the pie. But we’re seeing today a lot of company, large or small, offering the same kind of products or service with no real competitive advantage. They all have access to low cost labors, same technologies, same global knowledge and we’re seeing (in my opinion) a stage of slow innovation from a software or service perspective. The real main differentiator today between 2 similar business is not so much at the feature level but around the efficiency to better understand data pertaining to this business, predicting future trend to be able to trigger actionable data for better profit. As an example, it helps business better manage their supply chain (reduce inventory and stock-outs), optimize customer selection and increase their loyalty, optimize pricing, increase product and service quality by detecting problems early, optimize marketing campaign for stronger return etc. Business is of course one of the main driver helping innovation in the analytics world. Science is an obvious contributor and beneficiary as well, from astrophysics, health care, biology, physics, meteorology etc. They all need improved data-collection technology and analysis.

As Joe Hellerstein mentions, we’ve entered the industrial revolution of data and it’s a brand new ball game!

The Petabyte age

My first computer back in 1984, an Apple IIe, had 128KB of memory (I was one of the lucky one !) and I was able to store a whooping 140KB on a single 5.25 inch floppy. Then came the time of the Atari ST and Commodore Amiga years with their 512KB memory and 3.5 inch floppy with 720KB storage. In the 90′s, my first PC, a 386sx with 1MB memory and a 20MB hard-drive. Then it went ballistic and I end up today with 4GB of memory of my primary machine and multiple terabyte of data (mainly pictures. You can see some of them on my photo blog !)
I couldn’t dream of a terabyte of storage 5 years ago and now it’s becoming the norm for consumers. Digital Data consumption has exploded for consumers between the music, pictures, movies, books etc. Business and science are of course following this trend

To give you an idea:

• Wall-Mart handles 1 Million customer transaction per hour, feeding databases estimated at more than 2.5 petabyte.
• It’s difficult to evaluate the amount of data Google is currently storing and handling everyday but a white paper released in 2008 mentions that between the indexing, processing and the serve ups of ads, Google was processing 20,000 terabytes of data (20 petabytes !) a day.
• Ebay stores 8.5 petabyes of data.
• AMSTAR a digital library used by the National Center for Atmospheric Research (NCAR) will preserve and protect 30 petabytes of data.
• The amount of information consumed by Americans in 2008 totaled 3.6 zettabytes based on a study from researchers in the University of California in San Diego (UCSD)
• More than 20 hours of  YouTube videos are uploaded… every minute (source)
• Over 200 billion emails are sent… every day (source)
• In 2005, mankind created 150 exabytes of data… this year it is estimated we will create 1,200 exabytes (source)
• In 2020 we expect to create 35 Zettabytes of data, or 35,000 exabytes.  (source)

If you’re lost with all these bytes, here is refresher !

From Data to Wisdom

The data part is only the tip of the iceberg and actually it is actually pretty pointless to gather them without doing anything meaningful with them. In the late 1970s the DIKW model was built to allow data mangers to treat their data as a strategic resource and establish a relationship between data and knowledge allowing them to feed decisions based on this knowledge.

The  model assumes the following chain of action:

• Data comes in the form of raw observations and measurements. Raw webserver logs, Databases Transaction logs, mobile phone call logs etc. There is nothing you can get from these data unless you do some crunching on them.
• Information is created by analyzing relationships and connections between the data. It is capable of answering simple “who/what/where/when/why” style questions. Information is a message, and consider the audience and the purpose.Your data starts to make some sense and become relevant to your business: Where do your visitors navigate on your website, where do they click, how long do they stay on a specific page, Where do they come from, where do they after their visit, what do they buy, who have you called from your mobile, where were you, what did you buy after being in X or Y store etc.
• Knowledge is created by using the information for action. Knowledge answers the question “how”. Basically we want to understand based on the information we’ve aggregated the conditions as to what makes people behave in the way they do. If we know that a 25 years old man, living in New York, married, working for Microsoft, with a credit score of 650 has bought a Laptop last month, can we make a prediction of what he will be likely buying in the next 2 weeks, how much he will spend, what store he will chose and how will he react if we send him a coupon for X or Y products?

• Wisdom deals with the future, as it takes implications and lagged effects into account. We now know the who, what, where, when and hopefully why. We need to test our predictions, execute them and monitor the results and probably adapt. This is where you end up with your A/B or Champion/Challenger strategy (Smashing magazine has just released its ultimate guide to A/B Testing ! How convenient.)

Today, a lot of company stops at the Information stage an analyze their metrics retroactively to understand what has happened. The successful company will go all the way to the wisdom stage to develop models to predicts optimal action for the future. Predictive analytics is a booming market where companies such as SPSS (acquired by IBM last year), Prediction Impact and of course Experian mine historical behavioral data, transactional and demographic data to develop prediction model for future behavior. Earlier this year, Google has acquire Recorded Future which tries to analyze the past and present to predict the future with technology that extracts event and time information for the web to predict stock market events or even terrorists attack.

Predictive analytics is all over all business verticals: Marriott International has developed a company wide program called Total Hotel Optimization Program for establishing the optimal price for guest rooms (revenue management). The software integrate a system to optimize offerings to frequent customers and calculate the risk for these customers to defect to competitor. UPS has deployed a software solution which is able to accurately predict customer defections by examining usage patterns and complaints. Banks of course are not the last one in this bandwagon with Capital One with their Information-based strategy and Barclays with their information-based customer management. Other example include sports where statistical analysis is used to select the best player for a particular game or phase within a game. The new England Patriots, Oakland A’s and the Boston Red Sox all use software to make decision before and during the game. A book by Michael Lewis MoneyBall gives us some insight on the A’s overall approach to analytics.

Leveraging free data

In May 2010, the Open Data Study written by Becky Hogge was published by the Open Society Institute. It explores the benefits of opening government data in the US and the UK and tries to give clues on whether similar efforts should be pushed outside of Western democracies. The origin of this approach is the belief that the opening of public data can provide important economic and social advantages. For Becky Hogge, available spacial, demographic, budgetary and social data can be used to improves services and create economic growth.

The United States and United Kingdom have recently adopted proactive approaches on OpenData by launching 2 portals http://www.data.gov and http://www.data.gov.uk.

Data.gov.uk was officially launched in January 2010. The site has been seen as a victory for pro-OpenData community. United Kingdom now see various websites and applications based on data released, especially map data related to postal codes, about the last general election in May 2010.

Data.gov is a U.S. Government portal providing access to databases created by the U.S. federal government and its agencies. It was launched in 2009 with two objectives. Firstly, the desire to encourage a bottom-up communication and the emergence of new ideas of governance, enhancing transparency of public services, citizen participation and collaboration between the state and its citizens. The opening of government data has also been conceived as a means of improving the efficiency of government agencies.

It originally contained 76 databases from 11 government agencies. Fearing that the momentum towards OpenData falling back and that too little data are publicly released, Obama has adopted a decree, December 8, 2009, requiring each government agency to publish at least three databases of quality.

The British portal already offers three times more data, while its American counterpart has six months advance. And data.gov.uk chose standardized format to foster the development of semantic web, unlike data.gov.

Please refer to the articles of Flowing Data and RWW for further comparison between these two platforms.

There are other initiatives toward opening data. Earlier this year, the World Bank released their World Development Indicators 2010

The WDI provides a valuable statistical picture of the world and how far we’ve come in advancing development,” said Justin Yifu
Lin, the World Bank’s Chief Economist and the Senior Vice President for Development Economics. “Making this comprehensive data free for all is a dream come true.

They’ve also announced the launch of freely available online database and public API to 1,000+ indicators. These data used to be very expensive and I can’t start to imagine all the mashups we’re going to see flourishing in the next months.

Any company serious about analytics needs to keep an eye on Open Data. Whether they want to contribute with their own data or leverage other’s data. There is a real opportunity here to opens new collaboration between the public and the private sector.

A need for new tools and computing environment

This deluge of data brings new challenges from a software and computing environment. Traditional RDBMS typically show their limit when dealing with vast amount of data and supercomputer are still not accessible to most company.

Cloud computing is the obvious choice to collect, store and process data. With an illusion of infinite computing resources available on-demand, the elimination of up-front commitment and the ability to pay for use of computing resources on a short-term, as-needed basis you have the perfect environment to process your data. No wonder that by 2012, customers are expected to spend $42 billion on cloud computing !

The next trends fueling these new uses of data are open-source innovative framework that supports data-intensive distributed application enabling to work with thousand of nodes and petabytes of data. Hadoop is the most popular framework today and include the following sub-projects:

• Hadoop Common:The common utilities that support the other Hadoop subprojects.
• Chukwa: A data collection system for managing large distributed systems.
• HBase: A scalable, distributed database that supports structured data storage for large tables.
• HDFS: A distributed file system that provides high throughput access to application data.
• Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying.
• MapReduce:A software framework for distributed processing of large data sets on compute clusters.
• Pig: A high-level data-flow language and execution framework for parallel computation.
• ZooKeeper:A high-performance coordination service for distributed applications.

As I have discussed a while back, the NoSQL movement is taking more and more importance and is seeing an acceleration in usage against the traditional RDBMS. Parallel database management system are good at working with structured data. the noSQL approach gives the user much more control over how unstructured data is retrieved making it more suitable for processing petabytes of data.

Last but not least, a free programming language, R, brings a very powerful software environment for statistical computing and graphics and is today taking a lot of share from the more expensive SAS.

Create an analytics mindset in your organization

Pushing for an analytics culture can create some tension between people with entrepreneurial and visionary mind and those being more pragmatic requiring a lot of proof before implementing a switch of strategy. The goal here is to find the right balance, being able to validate the strategy at a small scale and find a way to interpolate the results for a larger scale. There is always a risk to interpolate these results but that’s why successful organization hire the best mind to lead this effort. At Google, Procter and Gamble, UPS, Harrah’s, Capital One, Barclays, Amazon, Ebay etc. all organization driving an analytics strategy have leading authorities in maths, statistical analysis, optimization etc. Companies are competing for top talent and statistician is becoming THE sexy job for the future. A lot of the big players are getting really serious to attract the best talent. An example would be IBM and their plan to open a specialized data analytics center.

Because this strategy might impact the culture, processes, approach to business and skills, it needs to be push strongly from the top of the organization. You need charismatic leaders who can have both a blue-sky approach to empower their people (especially R&D) as well pragmatic, business and down to earth in order to maximize returns. Gary Loveman, Jeff Bezos, Rich Fairbanks are of this kind with the clout, perspective and cross-functional scope to change the culture.

There are 2 important additional topics I will cover in subsequent posts

Visualization of Data

Privacy and the danger of an open society

Related posts:

1. Musings on NoSQL
2. Grand Central Dispatch – Future of thread programming?
3. Jumping into the future of Software Testing – The reasons I’m joining SOASTA
4. Can We Find The Future of Web Performance in Berlin’s Pubs?
5. Is the world cup killing Twitter?

After 4 years, the wait is finally over ! The World Cup started last Friday. Beside bringing lots of excitement all over the world, lots of buzzing ears due to too many Vuvuzela, it also seems that the World Cup is brining a lot of pain to the Twitter engineering team. It might not be related, but the fact is that the tweetersphere is experiencing high level of frustration ever since the event started. Performance degradation have been seen 3 days before the start and have been increasing ever since.

This is not the first occurrence for major problems but as far as I can recall it is the first time it lasts so long. From my perspective, it definitely demonstrate that Twitter might be running on thin ice and might not have that much room for growth.

As Ed Ceaser and Nick Kallen explains in their anatomy of a whale article and as most of you know, there can be many factors affecting the performance of a high volume service running on the web. Major issues arise when you end up with spike of volume and your hardware infrastructure is not able to cope. Unfortunately throwing more servers, with more CPUs and memory is only a temporary measure and definitely not a cure. Understanding which component of your system is the bottleneck is a nightmare, especially when you have millions of users screaming at you . You’ve got to think fast and have a way to analyze your data very, very quickly. We have today a lot of tools allowing us to gather data from our servers, our network infrastructure, our load balancer, our cache etc. but the aggregation and correlation of massive amount of live data is still very difficult to perform today. And this is a real struggle for the Twitter team and for everyone else running high volume. They candidly admit it in the article:

Debugging performance issues is really hard. But it’s not hard due to a lack of data; in fact, the difficulty arises because there is too much data. We measure dozens of metrics per individual site request, which, when multiplied by the overall site traffic, is a massive amount of information about Twitter’s performance at any given moment. Investigating performance problems in this world is more of an art than a science. It’s easy to confuse causes with symptoms and even the data recording software itself is untrustworthy.

This is why Performance Intelligence is becoming more and more critical with lots of vendors jumping on this bandwagon from CA with their Real-Time maps for Performance Intelligence, Soasta with their OLTP engine, Confio, Hyperic.

What I don’t quite understand from the recent Twitter outages is the strategy the Twitter engineering team use to test their fix or enhancement. With today access to cheap cloud environment, way to generate loads and loads of virtual users you would think that a staging environment is in place to deploy new chunk of code. When I look at the latest status from their status page, I’m wondering.

Update: We’re currently experiencing site availability issues resulting from the failed enhancement of a new approach to timeline caching. Our infrastructure and operations engineers are currently working to resolve this. We’ll update you soon with an ETA.
Update: 10:17 PM PDT:  Users may temporarily experience missing tweets from their timelines. They will be restored shortly.
Update: 11:51 PM PDT: We’re  recovering from the site outage. Remember, users may temporarily experience missing or duplicate tweets from their timelines. Normal timelines will be restored shortly.

What I do appreciate though, is the level of visibility the Twitter team gives us. They screwed up, they were probably not ready to handle an event such as the world cup (Especially the USA vs England game !) but they’re honest about it:

What happened?
In brief, we made three mistakes:
* We put two critical, fast-growing, high-bandwith components on the same segment of our internal network.
* Our internal network wasn’t appropriately being monitored.
* Our internal network was temporarily misconfigured.

There are a lot of information shared by the engineering team on their blog, unfortunately I can’t find anything regarding testing, how they do it, who is doing it (devs vs dedicated Software Engineer in Test), how they cope with the hardware challenge etc. Google is very good at reaching out to the test community but I haven’t seen much coming from Twitter. Maybe this is the right opportunity for them to do so?

Related posts:

1. Why is HP killing LoadRunner?
2. My favorite software testing experts on Twitter
3. Hola Barcelona! Meet SOASTA at the Mobile World Congress!
4. Performance Testing – When will they learn?
5. Hot Software and Performance Testing Links – Week of April 11th

As you might know, I tweet quite a bit of interesting posts, stories or article I find on the web that can be of interest to everyone involved in Software Development, Software Testing and Leadership. The best way to keep track of my tweets is simply to follow me on Twitter. However, in case you missed some, here’s a quick and useful compilation of the best tweets I’ve sent out the past 2 weeks.

Note that this is only a very small selection of the links that I tweeted about, so don’t miss out. to keep up to date with all the cool links, simply follow me @fredberinger

• How Mozilla and LinkedIn use Selenium

Mozilla did more than just host the event; they offered up QA engineer Stephen Donner and QA intern Raymond Etornam to share how Mozilla benefits from using Selenium Grid and Hudson to test their web applications using continuous integration. A great Q&A ensued, with Stephen explaining how and why Mozilla has come to adopt this more structured system over time.

This was followed by a superb presentation from Wade Catron, QA Automation Engineer at LinkedIn. He showed how the page object pattern, which LinkedIn implemented six months ago, has helped to create cleaner, more readable test scripts for the QA team, and he also shared some cool tricks!

• Altassian – A different kind of Software Company

• “Being” Agile versus “doing” Agile

Why don’t agile development methods when applied to BI always yield success? It comes down to understanding that agile is really about behaviors, attitudes, and values.

• FIFA World Cup Web Site against Performance Best Practices

My analysis of the FIFA site shows that – once the World Cup starts next week and the site gets really hit by millions of users around the globe – there is a big chance the site will run into performance and scalability issues due to several best practices that my analysis shows the site does not follow. This failure causes load times of the initial page take more than 8 secondsdownloads of more than 200 elements. These problems can easily be fixed by following the recommendations I highlight in this blog.

• Mary Meeker: Innovation is Back. Internet trends analysis & graphs w link to full slideshow

At the CM Summit today, Mary Meeker from Morgan Stanley did another one of her trademark quickfire, but densely packed, presentations on Internet Trends. There were several new themes in her June update, including that there has been an “unusually high level of innovation” recently from big Web companies. She mentioned Apple, Google, Facebook and others.

• The Genius Behind Minority Report’s Interfaces Resurfaces

It’s a cliche to say that Minority Report-style interfaces are just around the corner. But not when John Underkoffler is involved. As tech advisor on the film, he was the guy whose work actually inspired the interfaces that Tom Cruise used. The real-life system he’s been developing, called g-speak, is unbelievable.

Related posts:

1. My favorite cheat sheets for Software Testing
2. My favorite Software Testing blogs
3. My favorite software testing experts on Twitter
4. Hot Software and Performance Testing Links – Week of April 18th
5. Is the world cup killing Twitter?

As the web is becoming more ‘social’, we’ve seen a lot of change in the data space:

• Data volume is becoming larger and larger: Google manipulate 20 petabyte of data every day. Facebook handles 20 petabyte for their 10 billions photos !
• Data is getting more and more connected due to its social trend. It means a huge number of joins (I don’t have to remind you that ‘joins’ is the ennemy in the DB performance world)
• Data is becoming less structured.
• Need for scalability and fault tolerance is exploding. Geographical distribution is also key if you want to be global.

While relational databases bring a lot of benefit especially around Atomicity, consistency, isolation and durability (ACID), is well proven and the industry is very familiar with it (programers, DBA), they unfortunately hit a wall when dealing with Web 2.0 requirement. Their schema is rigid (lack of flexibility), joins are really slow, it’s difficult to distribute the data across nodes, the optimization you can introduce break the benefit of normalization ie. data integrity and downtime is not acceptable. But the major issue today is scalability. RDBMS scale reasonably well vertically but there is so much you can do with one huge system. If you want to meet the social web requirement, you will have to scale horizontally. It does offer flexibility but is definitely more complex: You need to group your data by function and spread your functional groups across databases. You then split your data within functional areas across multiple database (sharding).

It’s been demonstrated by Eric Brewer, who is a professor at the university of California that distributed system can have only 2 of the following characteristics:

• Consistency: Perception from the users that a set of operations occurs all at once.
• Availability. All operation must be performed with an appropriate response time.
• Partition tolerance. All operation must complete, even if one component/node is down/broken.

These characteristics are known as the CAP theorem. Since horizontal scaling is based on data partitioning, there is a trade-off remaining between consistency and availability. ACID database transaction doesn’t allow this tradeoff but non-relational databases address this limitation by introducing a new principle: BASE which trades some amount of consistency for availability.

• Basically Available: Appears to work all the time.
• Soft state: It doesn’t have to be consistent all the time.
• Eventually consistent: At some stage it will reach consistency !

Don’t you love it? While ACID is pessimistic and forces consistency for all operation, BASE has an optimistic view and assumes that inconsistent operation will occur (hell, shit happens !) but will reach a consistent state at some point. It seems a bit loose and difficult to manage but this is why non-relational database come to the rescue to implement smart consistency pattern and help you reach scalability you couldn’t dream about a few years ago !

There are today 4 main trends in the non-relational database world which dominate the space:

Key-Value databases
Entries are stored as key-value pairs in large hash tables. Domains (possible values of an attribute) are similar to those found in table but no specific schema is defined. Keys are arbitrary while values are blobs. There are no explicit relationships between domains. You access keys and values through API (SOAP, RESTful). Integrity is guarantee by the application itself.

Major open-source and commercial Key-values databases:

• Dynamo (Amazon)
• SimpleDB (Amazon web services). Written in Erlang !
• Voldemort (LinkedIn)
• Memcached: In  memory key-value store. All the major web players are using it: Facebook, Twitter, YouTube etc.

Column-oriented databases
Entries are stored by column versus row. It brings you a big performance uplift when you need to query many rows for smaller sets of data (not all columns) and it maximizes disk performance (read scans). It’s definitely not the right choice if you need to query all columns of a single row or need to write a new row with all column data supplied.

Major open-source and commercial column oriented databases:

• Cassandra: Facebook
• Big Table: Google
• Hypertable: Open-source implementation of BigTable.
• Hbase: Open-source implementation of BigTable.

Document databases

This was inspired by Lotus Notes and very similar to key-values stores. Each DB record is stored as a document (JSON). DB is schema-less and highly denormalized.

Major open-source and commercial column oriented databases:

• CouchDB
• MongoDB

Graph databases
In this model, entities are stored as nodes and edges. Nodes represents entities while edges represent relationships. It’s basically a key-value store with full support for relationship.

Major open-source and commercial column oriented databases:

• Neo4j

the whole noSQL (ie. Not Only SQL)  hype is picking up a lot of steam right now with the acceleration of the social web. The big relational DB players are already playing with it (IBM’s M2 corrals massive data sets with Hadoop). What will happen to your favorite RDBMS? Should they leverage Memcached or JBoss cache to get an uplift in scalability and performance while relying on their RDBMS engine to maintain ACID properties? is this the right balance if you don’t have Facebook or YouTube performance requirement?

If you’re interested to read more about noSQL (I know I’m hungry for more !), I definitely recommend the following read:

• Amazon Dynamo paper a classic.
• Bigtable paper. From Google.
• Lots of video presentation and Slides from the 2009 NoSQL conference.
• If you’re scratching your head around eventual consistent (I know I am !!), this is the article to read.
• The CAP Theorem.
• One Twitter for the latest news about NoSQL: @nosqlupdate is a must.

Related posts:

1. Musings on the Future of Data and Predictive Analytics
2. Grand Central Dispatch – Future of thread programming?
3. Oracle-Sun hearing starts today while Sun ships new Java EE version
4. High performance at massive scale – Lessons learned at Facebook
5. State of the economy is helping Agile development adoption

The European Commission has started to listen to Oracle argument for its planned acquisition of Sun Microsystems. All eyes will be on Sun’s MySQL business unit as the talk held last month ended up in an impasse as the EC is concerned that the acquisition will seriously reduce competition in the hot databases market. Oracle will be helped by customers such as Vodafone, The UK National Health Service, BBV etc. Against this merger, you can find Microsoft and SAP.

Interestingly enough, IBM seems to not care that much, according to Steve Mills, head of IBM software division who doesn’t consider MySQL as a competitor to Oracle (or IBM DB2). When you know that MySQL is the back-end behind Facebook and Youtube, you’re wondering about Mill’s comment:

It’s certainly not the most sophisticated database in the marketplace,” Mr Mills said. “It’s not used for sophisticated query and analysis things.

All Java developers wonder also about the fate of NetBeans and it is currently very difficult to predict. Personally, I’m optimistic in a short-term perspective. But longer term, I can’t see Oracle investing heavily on NetBeans when they’ve been pushing so much their JDevloper environment in the past and their Eclipse-based plug-ins. Could JavaFX save NetBeans as it is today the only IDE for building JavaFX application? More likely, they will add support for JavaFX into Eclipse … That is assuming JavaFX become a viable alternative to Ajax. That’s a lot of if …

Interestingly enough, Sun has just shipped the latest version of their key Java platform products: Java EE 6, GlassFish V3 and NetBeans 6.8.

Java EE 6 brings a new features called profiles that allows web application deployment scenarios with developing a custom stack. I see this as a generic stack for specific application scenarios you can extend. It gives you a starting point and speed up your deployment.

GlassFish V3 supports Java EE 6 profiles of course and improve its startup time and add monitoring capacibilities.

Related posts:

1. Software testing under NetBeans environment
2. Grand Central Dispatch – Future of thread programming?
3. Google launches an open source programming language
4. Musings on NoSQL
5. State of the economy is helping Agile development adoption