In our business analytics project work, we have often come in after several big data project failures of one kind or another. There are many reasons for this. They generally are not because of unproven technologies that were used because we have found that many new projects involving well-developed technologies fail. Why is this? Most surveys are quick to blame the scope, changing business requirements, lack of adequate skills etc. Based on our experience to date, we find that there are key attributes leading to successful big data initiatives that need to be carefully considered before you start a project. The understanding of these key attributes, below, will hopefully help you to avoid the most common pitfalls of big data projects.

There is often a misconception of just what big data is about. Big data refers not just to the data but also the methodologies and technologies used to store and analyze the data. It is not simply “a lot of data”. It’s also not the size that counts but what you do with it. Understanding the definition and total scope of big data for your company is key to avoiding some of the most common errors that could occur.

Avoid choosing bad use cases by selecting specific and well defined use cases that solve real business problems and that your team already understand well. For example, a good use case could be that you want to improve the segmentation and targeting of specific marketing offers.

Make sure that the data you’re collecting is the right data. Launching into a big data initiative with the idea that “We’ll just collect all the data that we can, and work out what to do with it later” often leads to disaster. Start with the data you already understand and flow that source of data into your data lake instead of flowing every possible source of data to the data lake.

Then next layer in one or two additional sources to enrich your analysis of web clickstream data or call centre text. Your cross-functional team can meet quarterly to prioritize and select the right use cases for implementation. Realize that it takes a lot of effort to import, clean and organize each data source.

Non-data science SMEs are the ones who understand their fields inside and out. They provide a context that allows you to understand what the data is saying. These SMEs are what frequently holds big data projects together. By offering on-the-job data science training to analysts in your organization interested in working in big data science, you will be able to far more efficiently fill project roles internally over hiring externally.

Big data projects need buy-in at every level, including senior leadership, middle management, nuts and bolts techies who will be carrying out the analytics and the workers themselves whose tasks will be affected by the results of the big data project. Everyone needs to understand what the big data project is doing and why? Not everyone needs to understand the ins and outs of the technical algorithms which may be running across the distributed, unstructured data that is analyzed in real time. But there should always be a logical, common-sense reason for what you are asking each member of the project team to do in the project. Good communication makes this happen.

All team members, data scientists and SMEs alike, must be able to trust each other. This is all about psychological safety and feeling empowered to contribute.

Big data initiatives executed well delivers significant and quantifiable business value to companies that take the extra time to plan, implement and roll out. Big data changes the strategy for data-driven businesses by overcoming barriers to analyzing large amounts of data, different types of unstructured and semi-structured data, and data that requires quick turnaround on results.

Being aware of the attributes of success above for big data projects would be a good start to making sure your big data project, whether it is your first or next one, delivers real business value and performance improvements to your organization.