As the data storage market in Australia surpasses $2 billion, the term Big Data is becoming increasingly irrelevant. It's all big data now.
The 'big' part has lost a further degree of relevancy as what it really translates to is the ever-growing complexity and massive amounts of processing power needed to cope with it, which would have been hard to imagine even a few years' ago.
The Big Data revolution, if we should still be calling it that, carries more issues than just its incredible need for heavy-duty storage and processing. Our growing ability to understand and make use of this massive resource is proving incredibly disruptive and is changing our entire approach to IT, even if it is still early days.
Stats clashing with compute
Size isn't everything. A connection between advanced statistical models and new compute technologies to better understand and benefit from data resources is the real driving force behind this revolution. The ability to generate rules - or algorithms - to look for patterns in data and solve problems in a fraction of the time using traditional computing methods is at the centre.
It's not a new concept, just one that's finding its way into more and more aspects of our daily lives. Google has had our search patterns figured out for a long time now and built them into its algorithm-driven search engine. Now Netflix uses algorithms to figure out what we want to watch next. Driverless cars, voice-driven assistants, next-day deliveries, high-speed trading - we owe these and more services and technologies every day to algorithms.
Just as apps have changed the way we interact with computers, algorithms are enabling a quantum leap in machine learning, enabling us to do some incredible things with big data.
While the size of this data and its ability to be processed so quickly are incredible, the real value lies firstly in the vast intelligence algorithms can provide, and secondly in how enterprises can monetise this. Gartner calls it the 'algorithm economy', and claims it will be the next big thing in big data. In the research firm's own words: "data is dumb, algorithms are where the real value lies."
More pressure on storage and processing
This next big thing poses even more pressure on data storage and processing resources. CIOs have been increasingly turning to public cloud resources to handle this demand, but in the same way that many organisations prefer to keep business-critical data in-house and away from the public cloud, the same logic is likely to be applied to algorithms. As their use becomes more important, so will the want to keep it safe in-house and away from the cloud.
Sign up for CIO Asia eNewsletters.