Data centres struggle with on-site staffing
Data centre managers have been forced to operate with lower staffing levels due to the global pandemic and remote workforce policies. But the situation is sounding alarm bells for facility executives as human error is widely recognised as the biggest cause of data centre outage time.
According to a recent survey, 70% of data centre managers took action and reduced on-site staffing levels when the pandemic was announced in March. Since then, many of them have been operating with a fraction of their usual staffing levels.
However, fewer employees on hand raise the risk of failure or maintenance problems being missed and even rushed working practices which lead to mistakes and oversights. This is especially true when emergencies occur and when less experienced staff man the facility.
According to Uptime Institute’s 2018 annual global survey of data centre operators, facilities have struggled to hire and retain qualified staff. Over 50% of respondents said they’d had difficulties finding candidates to fill open jobs or were having trouble retaining employees.
Some experts suggest that better training could be the answer to lowering risk levels and improving employee retention when minimal staffing levels are exacerbating the situation.
, Vice President of Marketing for iland commented, “As we have seen in 2020, it’s often the unplanned events that most challenge our preparation for change. Technology has helped businesses adapt. However, that doesn’t mean that IT professionals haven't had to overcome their own challenges."
He continued, “2020 may go down as the year in which the good and bad of IT have both been amplified. For example, IT organisations using cloud services before the pandemic were able to lean on their provider to support their changing business environment. However, organisations that managed their own infrastructure, and were burdened with a talent shortage prior to the lockdowns, likely saw that risk become more pronounced.”
However, it’s not just a question of inadequate training. Data centres have evolved considerably over the past decade, from storage facilities to high powered computing centres with AI tools and services as well as advanced cybersecurity and data aggregation features.
Roles have changed while demands have risen, particularly during COVID-19, with many enterprises and organisations requiring greater data storage capacity and bandwidth.
, International Product Marketing Senior Director at StorageCraft explained, “Data storage used to be a question of simply managing structured data in known, secure environments. This has changed, and with it, the skill set required from those managing it. Driven by explosive data growth, especially the growth of unstructured data, businesses need storage infrastructure that is highly scalable and flexible.”
Malecki continued, “In some ways, these are also the skills that are needed from storage professionals; a person that is able to quickly adapt to frequently changing demands from the business, understand what is needed, where it is needed, how the data can be accessed and protected, and propose a solution that meets all these requirements.”
He added, “It is no longer only about capacity, but about deploying storage infrastructure that safeguards data and ensures its constant availability and recoverability. As well as this, it is important to also consider threats from other factors, such as app failure, hardware failure or human error.”
He explains, “The digital transformation’s acceleration across organisations due to the pandemic has exposed skill shortages. This has especially been the case for data centre operators, requested to provide the best and most stable service while facing a drastic and sudden increase in load.”
El-Khayat advised facilities management to use a skills matrix tool to create an overview of skills available, and then work on filling the gaps with strategic recruitment. He also recommended working proactively with recruitment teams and giving team leaders the chance to build a long-term talent pipeline that fits the assessed needs.
He added that partnering with colleges and university programmes not only resulted in a more dynamic workforce but created opportunities for a demographic keen to learn. “University programmes are essential for building future capacity and specially to create incentives for a more diverse workforce – for example encouraging more female students to join the IT sector,” El-Khayat said.
Data deluge: the impact of data warehouse automation
As organisations focus more than ever on data strategy, they encounter a range of opportunities to take control of the factors that influence success, as the insight available from the effective data analysis helps improve decision making and builds competitive advantage. The transformational potential of data has not been lost on business leaders, who have tasked their technical teams with harnessing its power to deliver bottom line benefits.
As a result, organisations increasingly rely on data warehouse technologies to store, manage and analyse datasets that are often growing at an accelerating rate. By offering a curated repository of data, data warehouses are valued by users who need access to the right information in a usable format.
This is distinct from other approaches such as data lakes that act as huge collections of data, ranging from raw data that has not been organised or processed, through to varying levels of curated data sets. Ideal for some of the newer use cases such as Data Science, AI and machine learning, for more traditional analytics, data lakes can, however, be unwieldy and confusing.
As a result, many organisations opt for data warehouse solutions to manage essential data in more structured environments. However, working out how to speed up the rollout and management of these practices using technologies such as automation is essential if organisations are going to minimise the time to value and succeed in the data-driven business landscape.
Focusing On Data Warehouse Automation
In practical terms, as data enters the data warehouse environment, it is cleansed, transformed, categorised and tagged – making it easier to manage, use and monitor from a compliance perspective, which is where automation comes in. The problem is that the volume and velocity of data encountered by organisations today means that manually ingesting, processing and storing it in an accessible way that also meets compliance requirements within a data warehouse is increasingly unfeasible in the modern world.
However, with businesses constantly looking to data as the source of both reports and forecasts, a data warehouse is invaluable. As a result, data warehouse automation can help accelerate data ingestion and processing to boost time to value with data-driven decision-making in a data warehouse.
For example, Data Warehouse Automation (DWA) tools orchestrate the data warehousing process end-to-end, rather than being one of many tools that solve niche problems as in the traditional data warehousing lifecycle. This means companies don’t need teams of specialists at each stage of the process with manual handoffs between them, which can often lead to miscommunication and makes it harder to get a holistic view of the process.
Instead, implementing an automated template approach allows users to add their own data sources into and model the data to suit its needs, ensuring data structures are built quickly by automating all repetitive tasks whilst keeping IT teams in full control. As explained by Gartner in a recently published report ‘Assessing the Capabilities of Data Warehouse Automation (DWA)’, “The template-driven approach for data warehouse development reduces operational and compliance risks and is a disciplined process for delivering quality data warehouses incorporating all the best practices.”
Similarly, automation can help organisations manage the increasing levels of complexity that can blight their attempts to maximise the value of their data assets. As each new innovation builds ways to access the right data at the right time, so it can increase complexity for those tasked with designing the data ecosystem, and the problem is, many data teams still rely on 90s ETL tools and hand-coding to create and control a modern data fabric. As Gartner puts it, in the ‘Assessing the Capabilities of Data Warehouse Automation (DWA), “Automating these elements’ design plays a critical and essential role in data warehouse modernization and agile data warehousing.”
Across many teams, data warehouses now also play an important role in their efforts to implement and optimise DevOps, DataOps and other Agile methodologies. But, with automation handling the complexity, data teams can focus on strategic goals, such as delivering infrastructure and/or completing projects to Agile timeframes. For instance, teams who switch to DWA more readily adopt Agile, transformative frameworks such as DevOps or DataOps, and as a result, are in a stronger position to transform the way data is available to and used by the entire organisation.
Automation can also improve the ability of businesses to increase collaboration between IT and the business and speed critical processes, such as prototyping. Employing data-driven design to enable developers to create prototypes with actual company data, for instance, can demonstrate how requirements will behave in the final data warehouse. As the Gartner report explains: “The data-driven approach focuses on organizing the data models to align them closer to source systems. Business users and developers can collectively look at the data to gather inputs and feedback before creating the model. Using an iterative approach, data warehouse developers can rapidly build several prototypes before implementing the solution that meets the business user’s requirements. The method provides flexibility for deployment as well as management of changes to the data with flexible updates.”
While organisations the world over increase their commitment to becoming data driven, those who also automate key processes across their data warehouse strategy will be well placed to see a rapid return. In doing so, the most successful will benefit from a culture where data enhances their all-round abilities to innovate and deliver on key objectives.
Simon Spring, Account Director EMEA, at WhereScape, joined the company nearly ten years ago and throughout this time has worked effectively with hundreds of organisations looking to utilise data analytics and data warehouse automation to transform their business.