Hype About Hadoop

Why All The Hype About Hadoop?

In recent years there has been a lot of hype about Hadoop. Some have questioned whether it has actually lived up to the hype. Sure it has been useful for some companies, but mostly to massive corporations that have the money to spend on developing “data teams” that can optimize the use of the software. Furthermore, the product-as-a-service has been slow, and rather high-maintenance. Critiques of Hadoop are sprouting up everywhere.

This is because many companies simply purchase Hadoop software because it is a shiny new toy. This means that many of these companies may not know how to use it properly, and therefore not reaping all the benefits from it. Read this short blog to see why spending money on software and technology is you don’t know how to use is a waste. Furthermore, many are overloading it, by attempting to analyze a combination of structured and unstructured data; making the software respond slowly.

But even then, the demand is still increasing. The demand for data services, as a whole, is increasing, and rapidly. People and companies are beginning to see their competitors use smart data to increase their ROI’s. They see their competitors saving great deals of time and money using data analytics. They see innovated product development as a result of smart data. The demand for products and services in this space has driven improvements, and it is likely that these improvements and increased accessibility will continue to drive demand, resulting in a virtuous circle.

A Plethora of Data Starts the Trend

But where did this all spark? What catalyzed the demand? It may be the Internet of Things that is responsible for driving us to improve these software products and services.

Why?

1. Great Amounts of Data
2. Easy Access to Data
3. Slow Discovery of the Value in Analyzing Data

Greater and greater amounts of data are being generated each day. Today, we think the amount of data that we have accessible is astounding…just wait five years (and prepared to be amazed).

Simultaneously, companies are finding it to be rather easy to collect the data. They can easily and regularly collect data from various devices such as industrial sensors, mobile phones and cars. This allows them to constantly have up-to-date data.

Companies used to rely on surveys and historical data, to generate reports. This took a great deal of time, was not up-to-date, sometimes prone to human error (in collection of data and generation of reports), and did not answer all of the questions CEO’s may have had. But now, decision makers in companies can sleep soundly at night. By analyzing data, we get real-time, detailed, accurate information that allows us to base decisions off of science and facts, rather than speculation and educated guesses.

A Plethora of Data Causes Problems

Decision makers can be more confident that the decisions they make are based on accurate, and up-to-date information. As noted, this all began with an increase in the amount of data generated. However, it rapidly became too much data.

As more and more data was generated, companies began collecting more and more data. They began gathering data from many different sources. The more the merrier.

But does the saying go, “Quantity over quality?”

They soon realized that there was so much data, they didn’t know what to do with it. It was overwhelming.

To truly comprehend this, I imagine myself wandering a corn maze.  In the actual maze, all you see is the corn around you. You turn to the left, and all you see is corn. You turn to the right and all you see is corn.  You can turn 360 degrees, but still all you see is corn. You decide to continue going straight. But again, all you see is the corn. You will likely find yourself getting frustrated, and you will most certainly question if continuing straight was the best decision. You tried to analyze the situation better, but all you had to work with was the corn.

 Fly an airplane over the corn maze though, and you will see that the maze is in the shape of a beautiful picture. And you can now see that if you had gone left instead, you would have found the exit.

Solve This Issue and You Gain Insights

A combination of software, and data analytics have allowed companies to not just see the corn, but to see the shape of the maze. They have gained real-time visibility and insight. Thus allowing companies to act in real-time, and not have to rely on historical data. When companies are having problems in the supply chain, are experiencing customer dissatisfaction, or are experiencing anything but optimal growth, they can change data into valuable information that they can use to make the best decisions that will drive growth and prosperity. Companies certainly are beginning to realize and comprehend this. If you are new to the data hype, check out this podcast from a talk show about big data, in two minutes, it shares 5 lessons with you about using your data.

So Why the Lack of Historical Hype?

It is true that the hype of Hadoop and data was not as great as expected. I think that is because people have had a hard time understanding how data can be used. Furthermore, many have not understood the optimal context for the use of Hadoop.

Technology is something that is relatively new in the course of our history. It is expanding and innovating at such a rapid rate, it is hard to keep up with. For example, my parents just bought their first iPhones last month.

As soon as you start to understand a technological concept or a smart product, there is a new one out. I get used to my iPhone, and then within a month there is a new upgrade. Within a year, there is a new phone.

When the iPhone first came out, I remember not quite understanding the concept. I couldn’t find the value in the product. I thought, “Well I have my iPod with my music, and I have my phone. I have everything I need, why do I need them to be combined? And why the heck would I need to use the internet on my phone? That is what computers are for. Oh, and what the heck are apps?”

This is the stage many companies have been in. “I have surveys and reports, why do I need all this data? Things are working fine. If I have issues with a customer, they call customer care, and we deal with the problem. Oh, and what the heck is a cloud?”

Well, now I fully understand the value of my iPhone and can’t imagine my life without it at this point. Soon, companies won’t be able to imagine their companies functioning without using data.

The Demand Today, The Demand Tomorrow

This is already starting to happen. More and more companies are gathering data from devices, and more and more companies are buying software (or buying services from software companies) to consolidate their data. They are employing data scientists, purchasing data analytics tools and utilizing data services and data visualizations to access the insight that can be gathered from their data.

The Demand is Resulting in Improvement

This demand of companies have led to people trying to better the software. Recently, companies have innovated and helped to make Hadoop better suited to the Internet of Things. They have also made the processing of data much faster. Soon after this, companies started buying more and more software. I predict that this will only continue to increase.

However, companies that are trying to better analyze their structured data simply need to understand that software, like Hadoop, may not be the most useful tool for reading their data. Hadoop is more useful when it comes to measuring unstructured data.

Some companies certainly need to measure unstructured data; credit card companies, Amazon, Netflix, and perhaps the marketing silo of most companies. However, not everyone needs to purchase an expensive software to do this. It is more economical for most companies to hire a data analytics firm that will consolidate their data from many sources, store it, and spit it back into a readable format.

As a result, data management and data analytics companies are sprouting up everywhere.