Big Data continues to hold on to the headlines, although I think it’s being over-hyped. Businesses have started to take notice, and some are making investments already and some have even referred to it as the ‘new oil‘.
I don’t consider big data the ‘new oil’, but it does have consequences – good and bad – for the future of society and business, so I thought I’d dedicate some time on the topic.
What is big data?
There is no standard definition of what big data is, but the general consensus is that big data should conform to the 3V’s:
- Volume – There is a large amount of information amounting to Terabytes, or Petabytes of data.
- Velocity – The data is not only coming in very quickly, but should also be processed very quickly too.
- Variety – the information is coming from several different data sources, often in differing formats.
There are other V’s that have been suggested, but these are the ones that I choose to stick with. It’s important to note, is that big data requires different tools than what has normally been used (such as relational databases) for analysis.
Big data has always been around, with companies having these large data warehouses and now wanted to figure out how to use them to their advantage. This is further complicated this time by the addition of new sources of data – mobile and smart devices.
The Opportunities
The greatest opportunity available from big data is possibility of solving many of the biggest problems we have in the world.
Present and historical medical data coupled with genome mapping can help us find solutions to diseases or genetic disorders. Environmental data can help us predict climate changes, and used to improve farming methods.
Businesses can use big data to help make better business decisions. Business may be better able to predict market conditions and how well a product is doing, and decide to scale up or scale down production. It may be able to find defects much faster within their production line. Big data has even helped make better hiring decisions.
Big data can help us fight crime by gathers intelligence and evidence against criminal activity. It may help us to also take a proactive approach to security threats by picking up trends that indicate illegal or terrorist activity.
At a social level, big data along with crowd sourcing can help solve everyday problems. Waze, for instance, uses data coming from thousands of mobile devices to better predict traffic and help choose the best driving routes.
The Threats
The main threat is privacy. Lots of these data are personally identifiable information (PII). While the data may not have your name exactly, with proper analytics, you can create an eerily accurate profile of someone with it.
When the NSA surveillance news was broken earlier this year, the NSA stated that only phone metadata was being recorded. Phone metadata has no names, but many people went on to show how that data can tell a lot more that you think.
Target was one of the first companies that used big data and data analytics for marketing; in particular, it used its data to determine if a customer was pregnant and then sent targeted marketing to that customer about baby products. There were many questions about ethics at that time, particularly, whether customers would be okay with Target knowing about their reproductive cycle. Nowadays, you can hardly find a large retailer not using any time of data analytics on their customer data.
And while Target and other retailers may have your data with your permission, what about cases where you have not explicitly given permission? Facebook’s tagging feature, for instance, allows a friend to either take a photo of you and tag you, or check in to a location, and tag you as being with them. While you may not even have your phone with you, your image, and location data is being stored and mined.
One of the greatest threats, which is now gaining greater recognition, is the threat of theft of data. Going back to Target, only this week, they announced that they were hacked, and that the accounts of about 40 million customers were compromised. While they concentrated on credit card information, who knows what else they got. What if this was a medical or insurance institution; would you be comfortable if your medical data was stolen?
The Verdict
Big data on its own is nothing; it’s what we do with it. We can use the data to help us solve the world’s problems, or use it instead to further greed and destruction. There are endless possibilities as to how big data can be used.
Considering the good that can come from analysing big data, it would be better to implement proper security and governance frameworks to ensure that only good things happen.
