The Next Frontier of Big Data

The last decade has already born witness to two major waves in Big Data with the rise of smart devices and social media, but the current wave driven by sensor devices and the Internet of Things is utterly unprecedented in size, scope, and magnitude. The next frontier of Big Data will necessitate the birth of new technologies to process and detect streams of sensor data in real-time to fully capture the advantages of IoT.

Big Data as we know it first started gaining prominence in the early 2000s with Apple’s release of the iPhone. Suddenly, being online was no longer limited by the constraints of being on a
desktop computer. While the greater public was steadily adopting the Internet for its own uses from its original purpose as a communication tool for academics since the 1990s, the relevancy
of the Internet and the services attached to it grew exponentially with increased accessibility. The iPhone and other smart phones allowed users to be continuously plugged in to any number of web-based applications.

Little wonder then that social media services became the next wave of Big Data. While big players like Facebook, Youtube, and Twitter were founded around the same time that the iPhone was released, it wouldn’t be until a few years later that they truly became major players in producing, tracking, and storing masses of personal data for masses of indefinitely online users.

That brings us to today, where the rapid development of the Internet as a driver of modern life has led to the Internet of Things, the logical conclusion of ever-increasing connectivity and data creation. Unlike the Big Data waves generated by the iPhone or social media, IoT data is mostly machine-generated sensor data that vastly outpaces a human’s capacity for producing data. Further, only a fraction of the data continuously emitted by sensor devices is useful, and even then, that data is only useful for a short window of time. Streaming data is perishable, which means missing the window of time to make use of relevant data renders it effectively useless.

This means that one must have the ability to identify significant events as they happen in real-time from a continuous stream of data in order to harness the true potential of IoT data. This is a highly sophisticated undertaking, but fortunately, enables users to define events, layer data, and build pattern recogition processes in real-time.