CIOs can harness dark data to boost innovation

Perhaps one of the most misunderstood terms in enterprise technology is ‘dark data’ and it’s something no CIO would want to keep in its server system. The term dark data originally coined by Gartner can be defined as “information assets which organizations collect, process, and store during regular business activities, but generally fail to use for other purposes.” But despite its ominous name, it is actually a highly-valuable asset, and storing dark data and correctly mining it can provide huge benefits to businesses and help CIOs foster innovation.

The ‘fairer’ side to dark data

Dark data can include anything from old files to content on devices and clouds that are outside IT’s immediate control and management. Dark data can appear in both structured and unstructured data, with majority of data in the unstructured segment being dark and less than 0.5% being analyzed. Majority of business data is structured data whereas unstructured data includes human and machine data. Unstructured data is not only significantly larger than structured data but also growing many times faster. This type of data is mostly retained by enterprises by deploying huge storage, backup and management infrastructure, added to a large IT budget being spent without any business outcome.

While this data explosion is putting pressure on IT to pump in more resources to store, protect and manage the data, companies across industries are yet to understand how this data can be leveraged to achieve key business insights and avoid business risks. The bottom line being – IT is struggling to know what data they have and how their data can be leveraged for business decision making.

Enterprises dealing with their customer’s personal data have another challenge of ensuring data compliance. For example, GDPR (General Data Protection Regulation) expects enterprises to ensure compliance like data protection, retention, right to forget, etc. With some part of the customer’s data likely to be present in dark data, job of CIOs becomes even more challenging to ensure compliance when they have limited insights into dark data as well as limited control to apply data policies like data retention.

However, this data can be an important asset if one knows how to use it. This data could be the key to new revenue streams, better customer experiences, and lower business costs, waiting to be discovered.

What is interesting about these dark data sets is that the problems that surround them are almost always human (organizational culture or process), rather than being a specific technology challenge. Some of the key challenges enterprises face today with dark data include the ability to find effective ways to extract value from data clutters, illuminating opportunities hidden within these hidden treasure troves, implementing effective data management mechanisms and establishing active risk mitigation practices.

In a business climate where data is competitive currency, these challenges can be potential threats and pose risks to any organization’s continued business health and well-being.

Business Impact of Dark Data

Traditionally enterprises analyzed transactional business data to make business decisions but today differentiated customer experience and new business models are possible by looking at unstructured human and machine data that are related to interactions, sentiments, online behaviour, preferences, locations frequently visited, etc. For example, just sentiment analysis has given direction to enterprises for improved product and marketing strategy.

Much higher business benefits are available when enterprises start blending their human and machine data with business data dynamically that gives 360-degree view of customers. This helps knowing customers even better, create better offers and eventually more business with higher customer satisfaction. In healthcare industry, an initiative called Patient360, enables doctors get a complete unified view of all the test images, medical reports, patient profile, prescriptions, etc. that helps doctors do accurate as well as quick diagnosis, resulting in significant patient satisfaction. With such initiatives, hospitals are launching various patient services to increase the business further.

Few enterprises observed that analysis of their huge unstructured data in Hadoop systems has not resulted in desired business value. We have seen true business value become visible when CIOs start integrating their unstructured data with structured one.

The diverse mix of content from disparate sources, such as audio, video, PDFs, social feeds, IVRs and emails needs to be curated in a secure repository to improve data quality that is essential for proper analysis that can be accessed across multiple users, applications and workloads on premise or cloud. Lack of data quality of unstructured data has been one of the reasons limiting analysis of such data for many enterprises.

A recent IDC survey revealed that 77% of surveyed Indian enterprises are storing data with the hope that in the next two years they will be able to use analytics to gain business insights from this data. However, according to an analysis by Harvard Business Review, less than half of an organization’s structured data is used in making business decisions, and less than 1% of unstructured data is used in any way at all.

In the earlier days, banks used to create their customer’s profile by looking at all the business transactions across their product lines and delivery channels. Today, banks are embarking on a journey wherein customer profiles are not only created from the business that their customers do with banks, but also from their daily interactions, sentiments, preferences, online behavior, etc.

This new process of analyzing and storing relevant data leads to achieving competitive differentiation, increased customer loyalty, deriving valuable business insights by bringing structure to data and eventually helps banks take more informed decisions in areas, such as customer retention, offers, etc. that was previously hidden in the pools of dark data that resided in the system.

Thus, combating the challenges put forth by dark data and help illuminate the data at the end of the tunnel.

(As told to Sohini Bagchi)

Runners Point