How cold storage is redefining the new data era

With the digital transformation of almost everything, we are experiencing a so-called “data explosion.” The amount of unstructured data being generated is growing at a rate of up to 60% per year and is expected to make up 80 to 90% of all data by 2025. 

As the amount of unstructured data generated globally increases, organisations are faced with new data storage challenges. This data is mostly unstructured and mostly cold – inactive data that is accessed infrequently. According to a recent survey conducted by Forbes, 95% of businesses expressed the need to manage unstructured data, with over 40% of businesses saying they must do so regularly. Here are some examples of the sheer volume of data being collected and processed: 

• A semiconductor manufacturer annually produces over a billion image scans documenting the manufacturing process of 4000 wafers every week, resulting in petabytes of data that must be retained for 6 months to multiple years.
• An autonomous car produces up to 2 TBs of data every hour of use that must be maintained for several years for potential safety analysis or remodeling.
• Every human genome sequenced is unique, consuming 100 GBs of storage. Each contains immense, untapped value to current and future medical research, thus, the very real motivation to retain this data indefinitely, not only for years, but decades.

As the gap between data growth and storage budgets widens, the problem of unstructured and inactive data cannot be overlooked. Step in cold storage is a well-known but often overlooked hero of archival storage – but that is about to change. 

First things first, let’s define what cold storage is

Accessing lots of data in an efficient and timely manner is key for efficient production workloads. Like many products and trends, data also has a life cycle. Depending on where the data is in its life cycle and its value to a project, there are different ‘temperatures’ used to categorise the different stages of the cycle. Data that is actively being used for a project, is known as ‘hot’ data, like financial transactions which needs to be accessed frequently. Then there is ‘warm’ data, which is still regularly accessed over a certain period. ‘Cold’ data or inactive data however, is data that is no longer or very occasionally used but still makes up over 60% of all stored data. 

Traditionally, cold data was data that was preserved by companies primarily to comply with regulatory or internal policies. However, this is not always the case anymore. Increasingly, companies are choosing to hold onto cold data not because they are obliged to do so, but because they recognise its long-term strategic value, the potential of the data to be re-used for future projects, and the opportunity to enrich the data over time in ways that increase its value. 

Traditionally, cold data was written to tape media, transferred offline, and taken to a storage facility in case it needed to be accessed again in the future. This solution is valued even today as a highly durable, secure, and inexpensive vehicle to preserve data over the long term. But more and more, the emergence of new methods of data analysis, increasing computing power, and the growing recognition that organisations across every industry must digitally transform, are imposing new requirements for data to be retained online and easily accessible. At the same time, concerns around long-term durability, data security, and low cost still need to be considered. 

Is a cold storage strategy really necessary? 

The overwhelming surge in data generation shows no signs of slowing down. It is predicted that data created during the next three years will amount to more than all of the data created over the past 30 years. So, as organisations come to realise the value of cold data, combined with the rapid increase in data generation, a cold storage strategy is vital. Companies must proactively architect meaningful solutions to harvest data-driven innovations and opportunities while mitigating the issues intrinsic to massive data accumulation and continued data growth. 

What are the benefits of cold storage? 

It’s no secret that storing data can be costly for organisations. However, cold data storage is considerably cheaper than the NVMe and solid-state disk technologies employed for high performance access to hot data. The main reason for this is that cold data benefits from trade-offs between the performance and cost of various storage technologies. Cold data can be stored on lower performing and less expensive storage infrastructure, either in-house or in the cloud, to cost-effectively allow organisations to store more of their growing sets of data. 

Modern cold storage archives were pioneered by some of the world’s largest cloud solution providers, but with emerging architectures and services, cold storage solutions are now deployable within an organisation’s data centre, colocation facility or hosted IT environment. Recent innovations simultaneously maximise performance, data durability, and storage efficiency, raising the bar on performance and affordability expectations. Data is more easily accessible with no access fees, and is retrievable within minutes, not hours or days. New erasure coding algorithms are now optimised specifically for cold storage, reducing the storage overhead compared to the outdated practice of storing multiple copies. Data can also now be maintained within in-house security perimeters to meet data sovereignty and data residency requirements.

Immense value in future archiving and preservation of data 

Cold storage not only provides a way of storing growing data sets but also helps organisations to securely maintain in-house control of these valuable assets and easily access this data to unlock and enhance its value. In this new data era, it is becoming increasingly apparent that data needs to be preserved for its historic relevance and be online and accessible to exploit its future value. Thanks to new advances in cold storage there is no need to sacrifice inactive data for active data. Whether an organisation is looking to reduce costs or increase their data’s value, cold storage should not be overlooked.