Everyone is talking about Big Data, the tsunami of information which has been steadily growing and growing as a result of our increasingly digital world. Everything you do these days, from doing driving your car to shopping to watching tv and surfing the web, is now being captured as digital information and used by a whole range of organisations to try and find out more about you and what you do.

The information from how and when you drive your car can be used to decide your insurance premiums, and wether or not you’re a ‘good risk’. The information from how and when you shop can be used to make you feel like valued customer and get the ‘right rewards’ and ‘incentives’ from a particular supermarket. The information about when and what you watch on TV can be used by broadcasters and advertisers to make sure you see the commercials they want you to see, and to make sure you aren’t skipping them on your cable box.

And I dread to think how the information on how we use the Web is being used.

At the end of the day, Big Data is just going to keep getting bigger, and more organisation are going to be looking to find out more about what we do.

This is really good news for solutions like SAND and frameworks like Hadoop that deal with Big Data. We’ve developed easily scalable methods for storing and searching Big Data using simple parallel processing and fully shared storage. That’s really bad news for anyone who’s invested in a ‘Data Appliance’ in the hope that it can cope with the growth, but great news for anyone looking for [Hadoop consulting](https://www.sandtechnology.com/services/hadoopconsulting/) or SAND solutions.

We all know Data Appliances are good at doing certain things, they typically take an massively parallel approach to processing data but make a fundamental mistake of building a hard link between the processing power and storage which means you can’t scale the one without scaling the other, you can’t do either independently and doing both means you need to rip-and-replace you existing data appliance with a shiny new one at who know how many time the original cost.

That’s why the key to Big Data is simple scalability.