Shedding light on dark data

• Data that is historically unaccounted for, unmanaged, and undervalued
• Modern-day analytics is bringing the problem into the spotlight

DATA is the lifeblood of any organisation. With data located in multiple repositories, organisations today struggle to find information when they need it.
 
In fact, information managers often tend to have little to no insight into what data is being created, a limited control over how and where it is being stored, and almost no understanding of its business value.
 
Not only is this content taking up valuable and expensive space, but it can also become a liability if not properly managed.
 
Almost all organisations have volumes of ‘dark data’ stored away in dusty vaults and off-site storage facilities. This data is historically unaccounted for, unmanaged, and undervalued.
 
A 2012 survey by the Governance and Risk Oversight Council found that 70% of organisations reported that dark data will have a negative impact on storage and 53% said it would increase risk.
 
Modern-day analytics is bringing the problem of dark data into the spotlight and causing forward-thinking enterprises to address this issue – not only to curb mounting storage costs, but also to gain a true understanding of unstructured data.
 
Control the growth of data
 
The first step to controlling dark data is to manage its sheer volume. Dark data is not hype. In fact, classification and valuation continue to be the most pressing issues when it comes to aging data, according to Gartner.
 
The adage ‘What you don’t know can’t hurt you’ no longer holds true in today’s business environment, and this is what makes the concept of dark data so vexing.
 
Dark data can include legacy file shares, backup tapes, archives and former employee email stores that are predominantly unclassified and not visible or accessible.
 
The challenge of dark data becomes to balance its liability with its potential profit gains from using information more strategically.
 
To stay ahead of the competition, organisations have to be able to gain actionable insights from large data volumes. They should be able to identify and select data that is important to the business.
 
Modern archive solutions should allow organisations to keep only content with business value, which can in turn significantly reduce retention costs.
 
Capture data holistically

The next key to unveiling dark data is to implement a comprehensive archive strategy. Data produced on edge and mobile devices can be exceptionally valuable, just as much as information stored in virtualised servers, a corporate cloud, and everything in between.
 
To ensure that the organisation’s data intelligence efforts are successful, they should collect data and metadata everywhere it is being created. An effective archive tool can significantly simplify management of data across on-premises, cloud and hybrid environments.
 
There are now back-end repositories that enable organisations to align policy-driven rules with deep retention. Rules aligned to business objectives help reduce cost, storage footprint, business risk and administrative overhead.
 
Ideally these rules would cross multiple filesystems, applications and hypervisors with a minimal amount of custom scripting or data transformation.
 
Deliver self-service access
 
Workers across any organisation find different value within the same subsets of data. Sales may need pricing trend information by quarter, while operations may need data on facility usage, and HR (Human Resource) may need details on workplace satisfaction.
 
All of that data resides in the organisation’s archived information, but it isn’t useful unless it can be retrieved in a logical format.
 
Instead of setting up a costly data mining and analytics department inside the IT organisation, companies should enable users to search and access the data themselves and enable them to quickly discover, access and analyse information.
 
This, ultimately, maximises workforce productivity with transformational business insight into stored data.
 
Automate the data lifecycle

Organisations can help themselves by defining full lifecycles for specific classifications of data, and then running rules against those data stores.
 
How long should a user be able to store an email message? What about an email attachment? Perhaps it depends upon the type of user, or the size or type of attachment.
 
With so much data growth, and volumes of stored content that is not necessarily valuable to the business, consideration must be given to assess the full data lifecycle from creation through to disposition.
 
In fact, establishing set governance policies for defensible content deletion can be a significant key in the reduction of dark data.
 
Organisations should be able to classify, organise, retain and delete information easily so that they can streamline the entire information management lifecycle.
 
This will not only help them automate records retention, but also manage the data more strategically for improved compliance and litigation readiness.
 
Assure compliance and discovery
 
Compliance audits and litigation are two events no company wants to face, yet being realistic, every company knows it needs to prepare for. Compliance regulations touch every industry and every organisation understands the risk of the legal actions that can follow.
 
Yet, the rapidly increasing volume, complexity and disparity of data creates a challenge for organisations as they try to discover information needed to address corporate litigation, internal investigations, public information and audit requests.
 
The best way to minimise the cost of these queries is to be proactive with comprehensive archive, search and e-discovery capabilities.
 
How organisations handle the dark data challenge will become a key foundation for corporate competitiveness, business intelligence and innovation.
 
Yet, simply archiving data won’t deliver the intelligence organisations need to transform their business. To truly illuminate dark data, organisations need a modern archive with intelligence.
 
It is that intelligence that will manage storage growth, capture data comprehensively and provide users with simple self-service access, all while managing it throughout its lifecycle for lower compliance and litigation risk and cost.
 
Mark Bentkower is the director of Systems Engineering Asean, CommVault Systems. You can follow him on Twitter or LinkedIn.
 
Related Stories:
 
How file analysis tools can unleash the light in ‘dark data’
 
‘Thick data’ and how CIOs can become Code Halo heroes
 
Data stories … at home, in school, and within the enterprise
 
 
For more technology news and the latest updates, follow us on Twitter, LinkedIn or Like us on Facebook.