Big Data has obviously been big news for business and enterprises over the past 6 years. We’ve heard plenty about the size and enormity and scale of all that information, and this has become compounded by the sky-rocketing of the Internet of Things market as well. Interestingly, as hype in IoT has risen, the realities of Big Data have settled in and led to a lessening of some hype and more focus on substantive implementation. Speaking last fall, Nick Heudecker, research director at Gartner, noted this observation: “This year begins the shift of big data away from a topic unto itself, and toward standard practices.”
The real issue with Big Data is this: despite all the promises of the ever-expanding growth of new capabilities and digital technologies, Big Data is still frankly a “big” challenge for most enterprises to scale well. In fact, studies are showing that most companies are failing at Big Data. Smaller firms without as much overhead may fair somewhat better. But managing Big Data is just plain hard for most organizations. Reaching an impasse, many companies have begun to realize the only way they’re ever going to make any sense of the ginormous data within the digital universe is through better tools, processes, and smarter strategies for capturing and analyzing real-time insights.
In many ways, we’ve reached a new staging ground in the Big Data life-cycle as more and more organizations are pausing, taking stock, and reassessing their goals for capturing and analyzing the vast ocean of information out there. If you’re in this camp then it’s probably a good idea to stop and take an inventory of your processes and decide what has worked and what hasn’t, before pushing on. For other organizations that have delayed their Big Data strategy altogether, you’ll have some catching up to do but it’s never too late to get started. There are plenty of new and exciting technologies that will keep everyone – newbies and more experienced firms alike – extra busy in the years ahead.
In order to pave the way forward, it’s helpful to map out some of the major trends happening in the Big Data space and identify why now is the time to get onboard this train if you haven’t already.
Machine Learning Fueled by Big Data Will Generate Huge ROI
Data scientists have become quite adept at analyzing historical data and trends, but the next big shift is with applying machine learning algorithms to “infer” or predict new types of knowledge. The new trends in the integration of machine learning and Big Data are leading to major shifts in how companies are scaling out these efforts. The synergies and connections are nicely spelled out by one blogger as follows:
Think of big data and machine learning as three steps (and phases of companies that have come out of this space): collect, analyze, and predict. These steps have been disconnected until now, because we’ve been building the ecosystem from the bottom up — experimenting with various architectural and tool choices — and building a set of practices around that.
Businesses that can crack the predictive nut from the masses of data available today will have a veritable cash machine at their disposal to drive new types of customer value and service for their verticals.
Hadoop Will Become Big Data’s Standard Operating System
Hadoop can best be described as “a parallel distributed processing middleware technology which is applied across various industry verticals to perform Big Data analytics.” Despite a lot of early hype and then some degree of reassessment, Hadoop is fast becoming the de facto operating system for Big Data and the killer app of the Internet of Things. Notwithstanding some early limitations as a batch-only processing platform, Hadoop is keeping up with the market very nicely and has gone through several major overhauls in the past several years. 2013 was a big breakthrough as YARN was introduced and “converted Hadoop from a batch-only file-based stack into a set of interactive capabilities with multiple optional databases.” And in 2014 Apache Storm and Spark gave the platform real-time processing features that are making it Internet of Things-ready. As one source well states,
Real-time Hadoop has the ability to handle the Big Data demands of the Internet of Things along with the performance needed to create valuable applications that leverage the data. As it continues to grow beyond its batch processing roots, it is well positioned to be the modern platform of choice for powering data-intensive operational and analytic applications that can tap into the 30 billion connected devices of the near future.
Companies like Wal-Mart, Facebook, and Netflix use Hadoop to mine data to derive customer insights, and many more are quickly getting on the band-wagon.
Big Data + IoT Equal Exciting New Business Opportunities
The emerging nexus between Big Data and Internet of Things is posing a whole new set of business challenges, and opportunities. In a 2014 report Goldman Sachs called IoT the third wave in the development of the Internet with a potential market share of $2 Trillion dollars by the year 2020. The market is driven by the confluence of factors such as cheap sensors, cheap Wi-Fi, ubiquitous wireless coverage, Big Data, smartphones, IPV6, and more. According to Goldman Sachs, this confluence will generate “new business models that more seamlessly integrate hardware and software offerings, which support better recurring revenue streams and greater customer stickiness.” Big Data is the crux and core of new services generated by the Internet of Things, and much of the innovation in this space will be overseen by third party providers that develop the software channels for customer access to the long trail of devices and sensors connected to the internet. Cloud based service models built on top of these channels will generate Big Data-on-demand to customers and business alike. In other words, the new era of Big Data and IoT will spawn all kinds of revenue streams that make data more intelligible, accessible, and actionable from a business standpoint; the opportunities here will be almost endless!