Cabling: the nervous system of the data centre
The modern data centre is a vast, deeply complex collection of interconnected parts, each working in harmony. Power and cooling systems support huge halls of serried racks and cabinets, each one home to dozens of servers, all working to efficiently gather, store and transmit a near-incomprehensible amount of data every day. That data - the fuel that powers the modern world - is pumped across fast networks of fibre like blood through the body of a great beast. Responsible for the transmission of this vital information is a sprawling network of fibre optic and copper cables, the veins and nerves of a data centre.
Cabling and cable management is an essential part of the puzzle that makes up a modern data centre. A poor cable management strategy can create inefficiencies, demand constant maintenance, and ultimately hamper the operator’s ability to deliver value to its customers. “Some data center operators try to optimise the cost of every element and build a ‘Frankenstein’ system consisting of components from multiple vendors,” says Brad Smith, director of marketing for networking interconnects at Nvidia.
”This moves all the networking specifications to the lowest level of interoperability across all the vendors.” The perils of a poorly optimised system are growing even greater, as the technology behind data centre cabling accelerates and the industry faces rising demand for super fast, stable connections.
Data centre cabling is undergoing something of a revolution, as the industry prepares to make yet another generational leap, from 100 Gb per second fibre - which became the industry standard speed in 2016 - to 400 Gb per second. The life cycle of cabling hardware is shortening as well. “While most high-performance computing (HPC) and hyperscale installations are designed around a three year upgrade cycle and are highly flexible, many enterprise data centers are still rooted in the 1G/10Gb per second BASE-T copper era with an eight year upgrade cycle,” he adds. We sat down with Smith to discuss the challenges and opportunities that the changing data centre cabling market presents, and how data centre operators can build an effective cabling strategy to suit both their facility’s demands and their budget.
Copper vs fibre
Traditionally, data centre cabling has been accomplished using copper wire. However, in recent years, as data centres have gotten bigger, more efficient and are required to handle larger and larger amounts of information, fibre optic cable is emerging as a viable alternative. As with most new technologies, fibre has its limitations.
Copper cabling, used to transmit information for more than 100 years, still maintains several advantages. Firstly, they’re cheap. Compared to fibre optics, copper can be two to five times less expensive, potentially allowing companies to save millions of dollars in installation costs when spinning up an enterprise scale facility. Another cost-saver is copper’s reliability. Passive coaxial copper cabling has a mean time between failures (MTBF) of roughly 50mn hours, markedly higher than fibre optics. In terms of speed, over shorter distances, copper performs every bit as well as fibre while consuming less power.
However, the technology has a few drawbacks. “The basic issue with cables is that, the higher the frequency of the signal, the more that the electrons move to the outside of the cable, and then end up radiating into space,” explains Smith “At slow speeds you can go a long distance, which means that older, smaller data centres can use copper without any problems.” In larger enterprise, colocation and hyperscale facilities, however, copper cabling struggles to perform over the long distances between racks, especially as the IT load these cables are expected to handle increases year over year.
“A one gig copper cable could go maybe 500 metres. A 10Gb cable could go maybe 100 metres, but things start to fall apart when you get up to 25 Gb,” Smith explains. “The power usage, cost, you name it - it all just gets very ugly. So the industry moved to using direct attach copper (DAC) cables, which use a foil shield to extend the range and capacity of copper cabling. However, once you get up to 100Gb capacity, the range of these cables is only about a metre.”
In an enterprise facility with thousands of square metres of floor space, the limitations become clear. “If you’re a small, enterprise facility you can probably wire the whole thing up with copper. But if you’re Facebook and your data centre has two ZIP codes, then you're going to have to use single mode fibre optics, which are the same type they use in the telecom industry,” Smith explains.
Copper also has a wider bend radius than fibre, and the need for copper wiring to be insulated means that fibre optic cables tend to be a lot less bulky - a fact worth considering in modern, hyper-dense work spaces, HPC facilities or micro edge data centres. The benefits of Fibre optic cabling mostly lie in the range they can cover, as well as their less bulky profile. While fibre optic cabling is still more expensive to install and maintain than copper cables, widespread use by the telecom and hyperscale data centre industries has driven the price down considerably in recent years.
In reality, both copper and fibre optic cabling solutions have a place in the modern data centre, and operators should examine carefully where each one fits within their overall cabling strategy.
Nvidia: Conquering the data centre business
Nvidia rose to prominence as a GPU manufacturer in the gaming industry, but the company has come a long way since then. This summer, the company’s sales to the data centre industry exceeded its gaming revenue for the first time in its 30 year history. Data centre sales rose by 126% in the first half of 2020, following high-profile acquisitions of companies like Arm and Mellanox, which Nvidia purchased in late 2019 for $6.9bn.
Mellanox has since brought its industry-leading connectivity solutions to Nvidia’s ecosystem, which has had spectacular results. Mellanox’s portfolio of InfiniBand and Ethernet products has prompted a strong sales boost for Nvidia, further expanding its portfolio of data centre solutions.
Cabling best practice
The secret to a successful cabling strategy, according to Smith, is understanding the needs of your facility and appropriate preparation. “You have to plan out your cabling strategy for the next five-to-eight years, not just the next two,” he explains. “Current cabling technologies that may be less expensive today may require rip and replace upgrades tomorrow. Re-cabling is very expensive, and you are guaranteed to need the bandwidth and reach of single-mode fiber in the future. Making plans now for the very quickly evolving networking systems landscape, and the use of new GPUs for artificial intelligence applications, is a wise move.” He concludes by emphasising that a successful cabling strategy should centre around identifying the lowest-cost technology that is capable of transmitting across the distance that needs to be covered.