Big Data Is the 800-Pound Gorilla in the Room

The beauty of transactional data is its transient nature, which can help augment more conventional static profiling capabilities to create high-quality dynamic profiles. It is still early days but clearly, the advantage the large networks have – social or otherwise – are centered on their access to large data sets. Their ability to analyze these datasets and link the analysis to services delivered in as close to real time as possible is truly priceless.

While these value-added services may be either related to enhancing fraud prevention in the background or offering seamless experiences in the foreground, big data is now the 800-pound gorilla in the room.

From a click on a product at Amazon, to watching a video on YouTube, to posting a like on Facebook, to changing the setting on your Nest thermostat, to tapping your Apple phone to pay – we are constantly creating a digital trail. This trail will now be further enriched, and sooner rather than later, dwarfed by data generated by sensors embedded in the homes we live in, the appliances we use, the cars we drive, the offices we work in, and practically, most public spaces in the civilized world. Some refer to this trail as human exhaust. Marketers will beg to differ; for them, this is a treasure trove.

Einstein famously remarked that not everything that can be counted counts, and not everything that counts can be counted. We are already living in a world of information overload. By some estimates, from the advent of civilization till the start of the 21st century, mankind generated five exabytes of data. Now, we produce five exabytes every two days, and that pace is constantly increasing. It is safe to say we are now living in a world of data overload.

Every existing business needs to have a data analytics angle; every new entrepreneur needs to have a data analytics angle. All of a sudden, it seems that all these years, we have been shooting completely in the dark. How did we ever take any decisions without exabytes and exabytes of data to mine and analyze?

It is interesting to note that the world wars set the stage for pioneering work in the fields of statistical analysis and sampling models. Looking back at the advances in profiling and data analytics, will the generations to come attribute these to a peacetime developmental mindset or the third world war on terror? Philosophical debates aside, given the current hype and euphoria around data analytics, we will soon be moving from data overload to a big data hangover.

It is already becoming obvious that the select few who will make it to the promised land will not only figure out how to collect and analyze data, but create richer profiles from multiple vectors, and more importantly provide a tangible benefit to the user – underscore multiple vectors and tangible benefits. With this perspective, it makes perfect sense for every social network to move into the world of payments, and, for every payment network to collect and collate as many different data sets.

It is safe to conclude that while in the past, data sampling and statistical analysis models carried the day; moving forward, everything from when to water the lawn to pricing the water for the lawn will be driven by big data.

It is also safe to conclude that while there will be a tremendous amount of data collected, not all of it will be meaningful and as a matter of fact, some of it will even be a distraction.

It is also safe to conclude that transactional data will be the icing on the cake, the exclamation mark of the data age, providing relevance to the other data feeds, and guiding the analytics towards delivering tangible and meaningful services.

Check out Mehul Desai’s August of Money.