Experimental investigation of a data-centre cooling system using a new single-phase immersion/liquid technique

This paper presents an experimental investigation conducted on a new single-phase immersion/ liquid-cooling technique developed at the OVHcloud laboratory, combining a direct-to-chip water-cooling system with passive single-immersion cooling technology. The experimental setup tested the impact of three dielectric fluids (S5X, SmartCoolant and ThermaSafe R), the effect of the water circuit configuration, and the server power/profile. Results suggest that the system cooling demand depends on the fluid ’ s viscosity. As the viscosity increased from 4.6 to 9.8 mPa s, the cooling performance decreased by approximately 6%. Moreover, all the IT server profiles were validated at various water inlet temperatures and flow rates. The energy performance of the proposed technique and the liquid-cooling system previously used by OVHcloud was compared at a 600 kW data centre (DC). The proposed technique showed a reduction in the DC electrical power consumption by at least 20.7% compared to the liquid-cooling system. Finally, the cooling performance of the air-and liquid-cooled systems and the proposed solution was compared


Introduction
Large data centres (DCs) and other large computing infrastructures comprise thousands of racks, supporting more than tens of thousands of servers and other electronic equipment.These pieces of electronic equipment consume large amounts of electric power and generate significant amounts of heat.DCs consumed approximately 3% of global electricity in 2019.This figure is expected to increase at a rate of 12% annually owing to the significant demand for DCs [1,2].Cooling systems account for up to 40% of the total energy used in a typical DC [3].The generated heat may lead to server overheating, which threatens its safe operation and may cause server failure if it is not dissipated efficiently and immediately [4].Moreover, the CPU's temperature has a direct bearing on the power consumption of a server as demonstrated by Jin et al. [5] and Sarkinen et al. [6].Thus, a low CPU temperature can provide significant power savings.
Air-and liquid-cooling systems are the most commonly used systems for cooling DCs.Traditional air-cooled DC systems cannot efficiently manage heat owing to their high dissipation rates; therefore, their high energy consumption and cooling capacity loss constitute their main drawbacks [7].Furthermore, air cooling requires the use of powerful fans and the provision of space between pieces of electronic equipment for placing bulky heat sinks and allowing sufficient airflow.
Moreover, the increasing power density of processors, such as central processing units (CPUs) and graphics processing units (GPUs), has made the cooling apparatus for these devices challenging to develop [8].Modern IT equipment has a power density of 20 kW or more per rack [9].AC/DC systems can meet cooling demands of up to 7 kW per rack, which is not sufficient for rapidly developing DC facilities used, e.g., those used for big data and cloud computing.Moreover, the electronic equipment used for computing is not only becoming more performant, it also has a greater associated thermal design power (TDP), which refers to the power consumption under the maximum theoretical load.Thus, there is an urgent need to improve cooling solutions.
Liquid-cooling methods have been introduced recently as additions or alternatives to the conventional fan-forced air-cooling of electronic equipment racks.Liquid-cooling DC systems have recently become important and popular for IT equipment, owing to their energy-efficient heat-removal capacity of up to 100 W cm − 2 .A liquid-cooling system has better transfer characteristics and is therefore more economical.Chi et al. [10] demonstrated that a liquid-cooling system allows a power of approximately 96 kW and cooling energy savings of approximately 88% relative to an air-cooling system.

Data-centre key performance indicators (KPIs)
Recently, the International Organization for Standardization (ISO) determined the overall resource effectiveness or efficiency of a DC in terms of key performance indicators (KPIs), e.g., the power-usage effectiveness (PUE) [11] and the water-usage effectiveness (WUE) [12].

Power-usage effectiveness (PUE) and partial power-usage effectiveness (pPUE)
The PUE is widely used to determine the efficient utilisation and distribution of energy resources within a DC.It is the ratio of the total energy used by the DC to the energy used to operate IT equipment over a coincident period of 12 months: where E DC is the total annual DC energy consumption (kWh), and E IT is the annual IT equipment energy consumption (kWh).Three PUE categories are defined in Table 1.
The pPUE is determined based on the energy use of particular and specified subsystems of the DC infrastructure.The main purpose of the pPUE is to evaluate the potential annual energy savings of the infrastructure subsystems.It is given by [11].
where E sub is the annual energy consumption of the subsystem (kWh).
Cooling systems are typical subsystems that apply to most current DCs.The cooling pPUE becomes [11].where E Cooling is the annual energy consumption (kWh) of the cooling systems on the DC cooling line-up.

Water-usage effectiveness (WUE)
The water-usage effectiveness (WUE) is a KPI that quantifies the DC sustainability.It helps to understand the performance of its cooling installation and allows comparison with similar systems.It is defined as [12].
where U W = I W − O W , U W is the annual water usage, I W is the total annual water input (m 3 ) from outside the DC boundaries (including the water used for humidification and that evaporated on-site for the energy production or cooling of a DC and its support systems), and O W is the total annual water (m 3 ) returned from the DC boundaries.E IT is the annual IT equipment energy consumption (kWh).WUE categories are defined in Table 2.

Liquid cooling
One of the most important liquid cooling schemes is the direct-to-chip water-cooling (WC) system, which uses a heat sink (water block (WB)).This system offers the maximum power density per rack and can function at high operating temperatures.Although the recent developments noted above have shown some benefits, there remains scope for improving the efficiency and overall cooling performance of electronic assemblies.
Immersion cooling (IC) is an attractive liquid-cooling approach because the coolant makes direct mechanical contact-and hence adequate thermal contact-with the IT equipment [13,14].The heat generated by the equipment is thus directly transferred to the coolant.IC involves submerging electronic devices in a thermally and nonelectrically conductive liquid.This technique can be easily adapted to different cooling loads, as highlighted by Kanbur et al. [15].It enhances computational performance and efficiency compared to air cooling [16].IC systems may be either single-or two-phase systems.
Single-phase IC often involves using pumps to circulate heated fluid to the heat exchanger.The coolant is cooled and returned to the enclosure, transferring heat to the facility water flowing outside the DC, as illustrated in Fig. 1(a) and (b).This cooling method relies on the principle of heat transfer by convection, one of the dominant mechanisms of heat and mass transfer in fluids.Convective heat transfer can be further divided into two main mechanisms: natural and forced convection.In forced convection, heat transfer is driven by external forces produced by fans, pumps, or other devices that generate suction to drive fluid flow.Large amounts of heat energy are thus transferred.Bansode et al. [17] established that a server fully immersed in a single-phase dielectric fluid can be operated under extreme weather conditions (with temperatures ranging from − 20 to 10 • C at a relative humidity of 100%, and from 20 to 55 • C at a constant relative humidity of 50% for extended durations).Pires et al. [18] examined the cooling mechanism of a power electronic device immersed in oil with natural convection as the heat flux mechanism.They experimentally demonstrated that the heat dissipated using oil immersion was 2.5 times more than that using natural air convection (Fig. 1(a)).In natural convection, heat is transferred without any external forces and flow is driven by temperature gradients.Thus, heat is transferred only by virtue of the differences in the densities of both medium fluids (Fig. 1(b)).Matsuoka et al. [19] evaluated the cooling performance of a single-phase IC system with natural convection designed for high-power server boards used in DCs.Although they reported a PUE of approximately 1.04 (implying the potential suitability of this system for practical cooling technology in DCs), the main disadvantage is that the footprint (W/m 2 ) is reduced by the installation of large heat exchangers (n-m servers instead of n servers, as shown in Fig. 1(b)).Eiland et al. [20] experimentally studied a single-phase IC system and reported a pPUE of approximately 1.027.Matsuoka et al. [19] performed computational fluid dynamics (CFD) simulations to evaluate the cooling performance of different coolant fluids in a liquid-IC system with natural convection for high-power server boards used in DCs.Smooth refrigerants were compared to muddy/sticky refrigerants.The simulation results showed the superiority of smoother refrigerants relative to muddy or sticky ones.This suggests that the cooling effect of natural convection depends strongly on the refrigerant viscosity.
In two-phase IC, the fluid boils and condenses, increasing the heat-transfer efficiency as shown in Fig. 1(c).The heat absorbed from electronic components causes the fluid to boil, producing vapour that rises from the liquid.The vapour condenses on a heat exchanger (condenser) located inside the enclosure and transfers heat to the facility water.A tank of height z (cm) (z > y) was designed by installing a condenser, as shown in Fig. 1(c).Through latent heat transfer, two-phase IC provides better temperature uniformity across the surface of the heating equipment [21].Wu et al. [22] performed several studies on the performance of two-phase IC DC systems.They concluded that a coefficient of performance of 6.67 and a PUE of 1.15 are achievable using this system, while the PUE of conventional air-cooled DC systems ranges from 1.7 to 2.7.Liu and Yu [23] conducted a laboratory experiment and developed a CFD model to investigate the performance of a two-phase cooling system.The obtained pPUE showed an inverse trend, decreasing with increasing server power and ranging from 1.037 to 1.053.Recently, Kanbur et al. [24] conducted a comparative study between single-and two-phase IC systems by considering thermodynamic, economic, and thermoeconomic aspects.Their results suggest that the higher power consumption rates of pumps and energy costs are the main drawbacks of single-phase IC systems, whereas capital expenditures and replacement costs represent the main disadvantages of two-phase IC.They also concluded that both systems are prohibitively costly for server power rates below 5 kW, whereas they become more affordable for higher server power rates.
The above review highlights the importance of finding an energy-saving and cost-effective liquid immersion solution, and of developing new solutions that combine different liquid-cooling techniques using two different coolant fluids.The ultimate challenge full-scale implementation, as in an indirect free cooling (IFC) DC system (where external air is used to cool the liquid pumped around different cooling components installed in the DC without applying a refrigeration process) operating under extreme climatic conditions.
In this study, we laid out the detailed system architecture required for a novel combination of new single-phase immersion/liquidcooling technology, in which a direct-to-chip WC system is combined with a passive natural single-phase IC system.Using this architecture as a guideline, we demonstrate for the first time the feasibility of implementing the IC methodology for both high and low server power rates.Our system design not only aims to reduce the complexity and improve the suitability for both independent cooling of each server and retrofitting to large-scale implementation but also allows the server to function under severe climatic conditions.This was achieved by taking advantage of the highly efficient heat-transfer mechanisms of direct-to-chip WC for microprocessors (CPUs and GPUs).To demonstrate the functionality of our single-phase immersion/liquid-cooling technique and its robustness for realworld applications, we performed a complete system characterisation to determine its thermal performance and operating conditions at various water inlet temperatures, flow rates, cooling circuit configurations, server profiles, and electric power.We also compared the impact of three different coolants on the thermal performance of the system.This paradigm shift in cooling strategy not only opens possibilities for potentially reducing the cooling energy consumption (i.e., achieving low pPUE values); it also highlights the advantage of this system for heat reuse, in cases where the DC outlet temperature reaches 65 • C.

Single-phase immersion/liquid-cooling technique
The single-phase immersion/liquid-cooling technique presented in Fig. 2 comprises a direct-to-chip WC system combined with passive natural single-phase IC technology.It involves two fluids.
• Water coolant: a cold plate (water block (WB)) is placed on CPUs and GPUs connected to a pumping substation (PSS) and a dry cooler to evacuate heat outside the DC.• IC fluid contained in an IC tank and cooling all IT equipment components rather than CPUs and GPUs.
The serpentine convection coil (30 mm in diameter and 5 m in linear length) forms part of the server chassis and communicates with the WBs through a closed-loop fluid distribution arrangement.Water can circulate on the server in two main configurations: serpentine to WBs, or WBs to serpentine.As the temperature of the water supplied to the server is lower than the IC fluid temperature (even at the WBs outlet), the serpentine coil cools the IC fluid by natural convection.
CPUs and GPUs are the most energy-consuming components in servers.Thus, liquid cooling via direct chip cooling using cold plates (WBs) is an effective cooling technology for these components.In addition, several server components besides CPUs and GPUs (for example motherboards, and especially their voltage regulator module and capacitors, RAMs, and hard disks) generate much heat during operation, as shown in the infrared image in Fig. 2.These components must be cooled to ensure the safe operation of the server.For this purpose, IC is one of promising solutions to respond to latter issue.
Within the framework of the single-phase immersion/liquid-cooling technique, each server is submerged in its own tank and cooled independently, allowing easy large-scale deployment.In an analogous manner to a library with books, the rack becomes an IT library comprising 48 or 24 IC tanks (books), thus containing 48 one-unit servers (1U) or 24 2U servers, respectively, as shown in Fig. 3(a).Fig. 3(b) shows an IC container representing a DC of up to 600 kW with eight racks each with 75 kW and a footprint of up to 37 U/m 2 .
Table 3 compares different main IC systems described in the literature and the studied immersion/liquid techniques.The singlephase immersion/liquid-cooling technique has several advantages that make it efficient and ecological.In terms of energy efficiency, it allows the cooling of racks up to 48 kW, where a facility inlet temperature up to 45 • C can be used, as will be proven in Section 2.3.A DC water-temperature difference of 20 K can be attained that leads to a lower power consumption (Operating expenses: OPEX → PUE decreases) of the infrastructure (dry cooler) and less water usage (WUE decreases).An affordable heat recovery can be established with a DC outlet temperature of 65 • C. A very good footprint of up to 37 U/m 2 can be achieved without requiring an additional pumping (condensers) system.This explains the low capital expenditure (CAPEX).

Experimental setup
An IC lab was designed and built within the OVHcloud experimental DC in Croix-France to validate the impact of this technology on different operational servers.An insulated shelter was used, containing an IC tank of 600 × 500 × 50 mm 3 filled with a dielectric fluid.Layers of rock-wool insulation panels (Rockmur: thickness e = 100 mm and thermal resistance R = 2.85 m 2 K.W − 1 [26]) were installed on all shelter walls to limit heat dissipation outside the IC tank.
The hydraulic circuit of the IC lab consisted of two pumping substations: PSS-1 and PSS-2.Water circulation on the test servers was assured by PSS-1.Facility water, which cooled PSS-1 via a Plate Heat Exchanger (PHEX), was assured via PSS-2.The return water from PSS-2 was cooled using a dry cooler placed outside the DC.A high-accuracy solenoid valve was installed on the cooling water supplied by the PSS-2.The solenoid valve was equipped with a temperature probe installed at the IC lab water inlet to ensure a smooth regulation of the supplied temperature.
The experimental setup, shown in Fig. 4, uses no pumps or fans to produce forced convection inside the IC tank, where the dielectric fluid circulates around the IT equipment by natural convection.
Fig. 4(a) shows the distribution of sensors on the system.The ambient temperature of the IC container was measured using a thermocouple.The shelter wall temperatures were measured using several thermocouples installed on the wall surfaces.Accordingly, heat leakage was estimated to be negligible.Two thermocouples measuring the water temperature were also installed at the inlet and outlet of the PSS-1 circuit, which feeds the IC tank with water.Seven thermocouples were installed inside the tank (book) to measure the temperature of the dielectric fluid.These thermocouples formed a mesh for fluid temperature measurements.A thermocouple for measuring the water temperature was installed on the inlet and outlet pipelines of each component (serpentine, water block, and dry cooler).
Two Kobold flow-rate sensors (0.05-10 l min − 1 ) were installed on PSS-2 and PSS-1 to measure the cold water flow rate feeding PSS-1 and the IC tank.Two Kobold pressure sensors (0-6 bar) were installed at the IC tank inlet and outlet.A Kobold differential pressure sensor (3.73-373 mbar) was installed on the IC tank circuit inlet and outlet to measure the pressure drops across the serpentine and WBs.
All the temperature sensors were type-K thermocouples with an accuracy of ±0.210 • C after calibration.The National Instruments data-acquisition system NI cDAQ-9174, a compact DAQ system, was used to record all the temperatures, pressures, and flow rates throughout each experimental test.Two NI 9214 cards were used to measure and record temperature data.NI 9214 has a typical measurement accuracy of 0.37 • C at 100 • C when using type-K thermocouple.Two NI 9205 cards were used to measure and record the pressure and flow rate data.The data-sampling frequency was one measurement per second (1 Hz).Data acquisition was entirely automated using LabVIEW software.In addition, the solenoid valve was controlled by a Carel DN33B9MR20 (Model B) integral and derivative (PID) temperature controller.The control is set to be linear, and the operating cycle is an automatic program that can have a maximum of 5 set points to be reached in the 5 respective time intervals.The controller reaches the set temperature in the shortest possible time, after which it will go to the next step.Fig. 4(b) shows a photograph of the IC experimental setup.

Experimental procedures and uncertainties
Before starting the tests, the experimental loop circuits were fully filled with pure water.Air was then vented from the high points on the circuit, ensuring that the entire circuit was free of air.The supplied flow rate of the IC tank was manually adjusted based on the required server load and temperature difference.The water temperature supplied to the IC tank was controlled via the solenoid valve, ensuring that the boundary conditions were well attained.Each server chassis was adapted to the IC and submerged in an insulated IC tank filled with dielectric fluid.Tests were conducted to analyse the impact of the IC/liquid cooling technology on the performance of different servers.During the experiments, various thermal conditions (TCs) of inlet water temperatures, ranging from 30 to 45 • C, were experienced.The heating process of the working fluid was accompanied by temperature, flow-rate, and pressure measurements.The fluid temperature and flow rate were changed when all the measurements reached a steady state.Furthermore, the server's full thermal heat load was compared to the electrical consumption.
The thermal heat load was estimated as where ṁ (kg.s − 1 ) is the water flow rate, Cp (kJ.kg − 1 .• C − 1 ) is the specific heat of water, and ΔT ( • C) is the water temperature difference in the server water circuit.Absolute pressure sensors, differential pressure sensors, thermocouples, and flow-rate sensor calibrations were performed by comparing the response measured by each component to those measured by high-precision sensor probes.Uncertainties were evaluated using the method described by Kline and McClintock [27].For example, the uncertainty of the thermal heat load (Q) was evaluated as where ṁ is the water flow rate, T 1 is the inlet temperature, T 2 is the outlet temperature, and Δ ṁ, ΔT 1 , and ΔT 2 denote their respective absolute uncertainties.Table 4 lists the uncertainties of the parameters involved in the measurements.A flexible I/O (fio) tester script was used to stress the memory components of the servers, e.g., hard disks.CPUs and RAMs were jointly stressed using a script that allocated a quantity of RAM to each logical core of the CPUs to be used by a memtester process.Different Intelligent Platform Management Interface (IPMI) temperature measurements were also collected on the servers (e.g CPU junction temperature …).These data can be used to extract chip to ambient thermal resistance as explained in Appendix A at the end of this paper.Different IT components such as CPUs, GPUs, RAMs, and motherboards were analysed during each test.Other tools such as an Intel® Performance Counter Monitor and AMD uProf were used to check the stability of the server performance under different applied TCs [28].The CPU frequency (MHz) stabilities were monitored in real time and exported at a rate of 1 value every 30 s throughout each test.In addition, the rate at which data could be read from or stored in semiconductor memory by a processor, as expressed by the memory bandwidth (bytes/second) stability, was supervised.The server power was measured continuously with a time step of 1 s using a power distribution unit (PDU).The newly proposed cooling technology was tested via three series of tests conducted on the IC fluid type (1), the WC circuit configuration (2), and the server profiles (3).The results are summarised in Table 5.Each evaluation test was experimentally characterised by varying the water flow rate and the inlet water temperature from 30 to 45 • C in 5 • C increments.
Conventionally used IC fluids comprise dielectric liquids such as hydrocarbons or fluorocarbons.Three types of dielectric fluid were investigated: S5 X, SmartCoolant, and ThermaSafe R. Their chemical and physical properties are listed in Table 6.S5 X is a stabilised hydrocarbon fluid [29], SmartCoolant is a synthetic proprietary fluid [30], and ThermaSafe R is a synthesised hydrocarbon mixed with synthetic polyalphaolefins [31].They have approximately the same thermal characteristics because specific heats and thermal conductivities are not significantly different.However, the kinematic viscosity of S5 X is almost double that of SmartCoolant and Ther-maSafe R.

Immersion cooling fluid comparison
An Intel-based 1U open computing server, composed of two CPUs, sixteen RAMs, and three Non-Volatile Memory Express (NVME) disks with a total power of 620 W, was tested to analyse the thermal behaviour of the three dielectric fluids (S5 X, SmartCoolant, and ThermaSafe R).The water flow rate was varied from 0.35 to 0.55 l min − 1 under different TCs ranging from 30 to 45 • C. Fig. 5(a) shows the average temperature of the IC fluids in the lower and upper parts of the tank.The fluids show globally similar thermal behaviour with relatively similar temperature curves for all TCs.A temperature difference of 12% between the lower and upper parts is observed for all TCs and flow rates.Furthermore, the fluid temperature decreases with increasing water flow rate, whereas it increases with increasing inlet water temperature.The fluid temperature in the lower and upper parts increases by approximately 6 and 4.6%, respectively, when the water inlet temperature increases by 5 • C.Moreover, the fluid temperature decreases by approximately 0.6% when the water flow rate increases by 0.05 l min − 1 .
Fig. 5(b) shows the variations in IT (CPUs and RAMs) temperatures at different TCs and flow rates when using S5 X, SmartCoolant, and ThermaSafe R. The CPUs and RAMs exhibit approximately the same thermal behaviour in each case.Their temperatures increase by approximately 5.8 and 3.3%, respectively, when the water inlet temperature increases by 5 • C.Moreover, their temperatures decrease by approximately 1.6 and 1.0%, respectively, when the water flow rate increases by 0.05 l min − 1 .
Table 7 shows the impact of using different dielectric fluids on the cooling demand repartition between serpentine (fluid cooling demand) and WBs, expressed as a percentage.For a given water flow rate on the WBs and serpentine inside the server, the cooling demand is the ratio of the temperature difference per cooling portion (WB or serpentine) with respect to the system's total temperature difference.The water temperature differences between the inlet and outlet of WB (dT WB) and serpentine (dT S) were computed as dT WB = T WB out − T WB in and dT S = T serpentine out − T serpentine in respectively.
The low-viscosity SmartCoolant and ThermaSafe R fluids are compared to the higher-viscosity fluid S5 X.The fluid cooling demand decreases from 25% (S5 X) to 19% (ThermaSafe R).This indicates that the cooling demand for natural convection depends strongly on the fluid viscosity.This result confirms the computational fluid dynamics (CFD) simulation results of Matsuoka et al. [19], which indicated that smoother refrigerants are better than sticky ones.

Water-cooling circuit configuration comparison
This section compares four possible WC circuit configurations in terms of the thermal behaviour and the cooling system efficiency of IT equipment.These WC circuits are defined by the arrangement of the serpentine and WB in the hydraulic circuit and the direction of water flow in the serpentine with respect to natural convection (counter-current flow or co-current flow), as shown in Fig. 6(a).The fluid used was S5 X, which has a higher flash point.In the first configuration, the facility water entering the server flows directly through the WBs (globally positioned on top of CPUs or GPUs).Water then flows through the serpentine (from top to bottom representing counter-current flow).In the second configuration, the facility water entering the server flows directly through the WBs and then through the serpentine channel (from bottom to top representing co-current flow).In the third configuration, the facility water  entering the server flows directly through the serpentine (from bottom to top representing co-current flow), and then through the WBs.
In the fourth configuration, the facility water entering the server flows directly through the serpentine (from top to bottom representing counter-current flow) and then through the WBs.Fig. 6(b) and (c) show the maximal temperatures of the CPUs and RAMs, respectively, for different water circuit configurations under different TCs and flow rates (0.35-0.55 l min − 1 ).As shown in Fig. 6(a), the CPU temperatures are always below the 95 • C limit regardless of the TC, flow rate, or configuration.Nevertheless, and as expected, Configurations 1 and 2, where cold water enters the WB directly, show the best CPU temperature, with a difference of approximately 5-8 • C relative to Configurations 3 and 4.
Fig. 6(b) shows that Configurations 3 and 4 have the best RAM temperatures, with differences of approximately 6-8 • C compared to Configurations 1 and 2. It is likely that the RAM temperatures exceed the limit at TC-45 • C in Configurations 1 and 2. Accordingly, Configuration 3 shows the best cooling behaviour for all the pieces of IT equipment.
The cooling efficiency (heat dissipation through the cooling system) was determined as the ratio of the thermal heat load estimated via Equation ( 5) to the electrical power measured by the PDU.For the optimal configuration and operating conditions (TC-30 • C and a flow rate of 0.4 l min − 1 ) it has a value of 94%.

Server profile evaluation
This study considers the newest server profiles dedicated to cloud computing.In addition to the 620 W server used in Sections 2.3  (a) and (b), three other servers with different profiles (1000, 550, and 810 W) were tested and characterised.Table 8 shows the total power consumption and its distribution among different pieces of IT equipment.Server #2, a 1U server with a total power consumption of 1 kW, contains three GPUs, one CPU, and six RAMs.Server #3 is similar to Server #1, but contains no hard disks.Server #4, a 2U server with two CPUs, 12 RAMs, and 24 NVME disks, is one of the most consuming servers in terms of hard disks.Fig. 7(a) shows the temperature behaviour of the IT equipment of Server #2 under different operating conditions.Unlike the increase in the inlet water temperature, the CPU, GPU, and RAM temperatures decrease with increasing water flow rate under all thermal conditions.The maximal temperatures remain below the temperature limit under TC-30, 35, and 40 • C working conditions.Yet, operation under TC-45 • C and 0.4 l min − 1 is considered unfeasible since the server throttles and its frequency decreases.For TC-45 • C and flow rates above 0.5 l min − 1 , the server operates correctly with a stable frequency and temperature.Fig. 7(d) shows the server power measurement extracted from the PDU and the thermal heat load calculated using Equation ( 5).The optimum operating conditions are always achieved with a flow rate of 0.85 l min − 1 under the four thermal conditions.Thermal efficiencies of 99, 98, 96, and 94% were obtained for TC-30, 35, 40, and 45 • C, respectively.
The optimal operating conditions were determined for Servers #3 and #4 by following the same procedure adopted with Server #2.Fig. 8 From the previous experiment, the optimal flow rate per watt, determined based on the water-cooled equipment (CPUs and GPUs), varies from 0.05 to 0.1 l h − 1 W − 1 .

pPUE and WUE analysis
This section compares the energy performance analyses of the single-phase IC/liquid technique presented here and the previous liquid-cooling system used by OVHcloud [32].Two configurations were examined: an Indirect Free cooling (IFC) system equipped M. Hnayno et al. with an evaporative cooling (EC) system (IFC + EC) and an IFC + EC system with an increase in the temperature of water supplied to the DC by 5 K (IFC+5K + EC).A 600 kW DC located in Roubaix (Northern France) was studied for this analysis.The electrical power consumption of each heat-rejection system depends on the external conditions of the DC [33].In this study, it was defined as a function of the external temperature.For example, to provide a reasonable water-return temperature for the DC, the speed of the dry-cooler fans increases as the external temperature increases the power consumption.It is likely that EC pumps operate when the external temperature exceeds a certain limit.The pinch temperature on the dry-cooler heat exchanger was 3 K.For example, the dry-cooler fans operate at maximum speed when the external temperature reaches 24 • C to guarantee a DC inlet temperature of 27 • C. It is likely that the EC pumps were operated when the external temperature exceeded 24 • C.
The total power consumption of an IFC + EC cooling system is defined as the sum of the power consumed by the substation pumps, dry-cooler fans, and EC pumps:  (7) where o and p are constants that represent the number of dry coolers and EC pumps, respectively, installed within the DC infrastructure.
An approach similar to that used in previous studies was adopted for the power consumption of a dry-cooler fan [34].The fan affinity laws indicate that the power consumed by a fan increases as the cube of the airflow rate ( V): The airflow rate, pressure drop, and outlet air temperature, based on the outdoor conditions, were provided by the manufacturer.The power consumption of the fan is determined from the physical parameters at the operating point of the fan: where i and a i are constants and v is the airflow rate (m 3 /h) generated by fans.
The method for calculating the power consumption, given as a function of the external temperature of the dry cooler, was presented previously [32].
The EC water consumption was calculated as a function of the water demand of the EC system using cooling pads.It was determined based on the technical data provided by the manufacturer including the airflow rate, external air temperature, and humidity conditions.It is defined as the sum of the evaporated water and bleed-off quantities required per hour to operate a dry cooler equipped with an EC system.In addition, the WUE can be extracted as the total water required per kilowatt-hour (kWh) [32].Temperature and relative humidity data for the past 40 years were collected from the weather underground community site [36] and analysed (one record every 30 min).Based on these data, we selected 2018 as the warmest year in this 40-year range.
Fig. 9(a) shows the hourly temperature frequency for the climatic data from Roubaix during 2018.The average temperature ranged from − 7 to 41 • C, with the bulk of the distribution ranging from 5 to 20 • C. The annual energy consumption of the cooling system, E cooling (kWh), is the sum of the power consumption at each outlet temperature multiplied by the corresponding number of hours during the year: (10) Fig. 9(b) shows the electricity consumption of the DC cooling system with respect to the external temperature of the DC in Roubaix for the three heat-rejection systems: IFC + EC, IFC+5K + EC, and IC/liquid-cooling.The results show that the IFC + EC cooling system consumes higher levels of energy by approximately 34660 kWh.Moreover, increasing the temperature of the water supplied to the DC by 5 K reduces the electrical power consumption to 27885 kWh.However, the new single-phase IC/liquid technique reduces the electrical power consumption by 36.2 and 20.7% compared with IFC + EC and IFC+5K + EC, respectively.
Table 9 shows the impact of these cooling systems on the annual average cooling pPUE values at the Roubaix site, calculated as the ratio E cooling / E IT .(The IT load was measured at the IT equipment input level as represented in ISO 30134-2: E IT Category 3.) The pPUE decreases from 1.0066 to 1.0053 for IFC + EC and IFC+5K + EC, respectively, to 1.0042 for the new single-phase IC/liquid-cooling technique.
The reduction in the WUE values of the DCs affects their annual water consumption.Table 9 also shows the impact of the different cooling systems on the WUE of a 600 kW DC at the Roubaix site.An increase in the supplied water temperature by 5 K using the IFC + EC system reduces the WUE from 0.09 to 0.04 l/kWh.Accordingly, zero WUE can be achieved at the Roubaix site using the new singlephase IC/liquid technique, where a 45 • C temperature of DC supply-water is acceptable with maximal external temperature not exceeding 42 • C, as proven before in this work.

Energy consumption: server case study
To highlight the benefits of the immersion/liquid cooling system on the yearly energy consumption at the server level, Table 10 presents a comparison of the cooling performance of air-cooled, liquid-cooled [32], and single-phase IC/liquid-cooled servers used within OVHcloud.Server #4 was used for this purpose.The fan and pump powers required at the rack level were computed for the airand liquid-cooled servers; the cooling power per server (rack level) was calculated accordingly.The pPUE computed in Section 3.1 was used to calculate the cooling power of the infrastructure.The total cooling power is the sum of this cooling power and the cooling power per server at the rack level: P Total cooling (W) = P cooling per server (rack level) (W) + P cooling infrastructure (W) Yearly energy Consumption (kW.h) = Server total power (W) × 24 h × 365(days) The immersion/liquid-cooling technique reduces the server energy consumption per year by at least 20 or 7% relative to the air-or liquid-cooling system, respectively.

Conclusion
This paper presents a new single-phase immersion/liquid-cooling technology for IT servers, combining a direct-to-chip WC system with passive natural single-phase IC technology.Direct-to-chip cooling is used to cool CPUs and GPUs with water blocks, while IC is used to cool all the other components present on the server with a serpentine.Experiments were performed using an immersed server to identify the best useable dielectric fluid.Low-viscosity SmartCoolant and ThermaSafe R fluids were compared with the more viscous S5 X fluid at different inlet-water temperatures and flow rates.We examined four possible water circuit configurations with different directions of water flow (serpentine and water blocks) and studied four server profiles.The following conclusions were drawn.
• The cooling demand of a passive single-phase IC system depends on the fluid viscosity.When the viscosity almost doubled from 4.6 to 9.8 mPa s, the cooling performance decreased by approximately 6%. • Concerning the water circuit configuration, co-current flow from the serpentine to the water blocks yielded optimal results.The CPU and RAM temperatures were always below their safe operating limits, regardless of the inlet water temperature and flow rate, and lower than that of the counter-current flow.It is unlikely that the RAMs exceed their temperature limit when the water inlet temperature is 45 • C and when water flows from the water blocks to serpentine with both co-current and counter-current flows.• The cooling technology was tested for four different server profiles at water inlet temperatures ranging from 30 to 45 • C. In addition, an optimum operating condition at a water inlet temperature of 30 • C, maximising the yield, was determined for the four servers as 94% at 0.4 l min − 1 for the 620 W server, 99% at 0.85 l min − 1 for the 1 kW server, 97% at 0.40 l min − 1 for the 550 W server, and 97% at 0.53 l min − 1 for the 810 W server.Finally, the optimal flow rate per watt, determined based on the water-cooled equipment (CPUs and GPUs), varies from 0.05 to 0.1 l h − 1 W − 1 .
• The new single-phase IC/liquid-cooling technique reduces the electrical power consumption by at least 20.7% compared with the liquid-cooling system.• The new single-phase IC/liquid-cooling technique reduces the cooling partial power usage effectiveness (cooling pPUE) to 1.0042 compared with the liquid-cooling system, where a water usage effectiveness (WUE) of zero can be achieved at the Roubaix site.
Finally, we compared the cooling performances, at the server level, of the air-, liquid-, and single-phase IC/liquid-cooled servers.The single-phase IC/liquid technique reduces the server energy consumption per year by at least 20% compared to the air-cooling system and 7% compared to the liquid-cooling system.
However, for safety reasons, a higher dielectric fluid flash point is recommended.Thus, the development of a new dielectric fluid with a high flash point and a low viscosity may present a future challenge.In addition, the compatibility of IT equipment with dielectric fluids under single-phase immersion/liquid technology for long-term use will be investigated.Future work will also focus on investigating the relationship between the server power consumption in different operating modes and the server's inlet water temperature.

Declaration of competing interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Mohamad Hnayno reports financial support was provided by Association nationale de la recherche et de la technologie Mohamad Hnayno has patent #US20220322575 pending to OVH. that can be individually calculated and then combined to estimate any temperature for a given heat load as presented in Fig 10(a).It is defined as the sum of several thermal resistances determined by: R junction− water = R CPU + R TIM2 (CPU→WB) + R WB→water (14) Where R CPU is the CPU thermal resistance, R TIM2 (CPU→WB) is the thermal paste (Thermal Interface Material: TIM 2) thermal resistance existing between the CPU's casing and water block, and R WB→water is the thermal resistance between the WB and water (copper thermal resistance and surface resistance due to the convection).CPU thermal resistance results from a series of three principal resistances: casing thermal resistance (R Casing ), I/O Die thermal resistance (R I/ODie ) and core dies thermal resistance (R core Dies ).It is determined by: R CPU = 1

Fig. 4 . 4
Fig. 4. (a) Experimental setup scheme of the IC lab, (b) Photo of the IC experimental setup.

Fig. 5 .
Fig. 5. Variations in (a) the IC fluid temperature and (b) the IT (CPUs and RAMs) temperatures, under different TCs and flow rates, using S5 X, SmartCoolant, and ThermaSafe R fluids.

Fig. 6 .
Fig. 6.(a) Four possible configurations of the WC circuit, and temperature variations of (b) CPUs and (c) RAMs for the different WC configurations, TCs, and flow rates.

Fig. 7 (
b) shows the temperature behaviour of the IC fluid under different operating conditions.Higher values are observed in the upper than in the lower part of the IC tank as a result of natural convection for a temperature difference of 8 K.The average temperature of the fluid in the tank is approximately 53, 55, 57, and 61 • C for water inlet temperatures of 30, 35, 40, and 45 • C, respectively.Fig. 7(c) shows the water temperature difference under different operating conditions.It decreases with increasing water flow rate and inlet water temperature.A maximum water temperature difference of 29 K is reached for TC-30 • C and a flow rate of 0.4 l min − 1 .
(a) and (b) show the electrical power measurements of Servers #3 and #4, respectively, extracted from the PDU and their thermal heat load.The optimum operating conditions are reached for Server #3 with a flow rate of 0.50 l min − 1 under TC-35 and 45 • C, whereas, under TC-30 • C and TC-40 • C, the optimal water flow rates are 0.4 and 0.35 l min − 1 , respectively.Therefore, the best yield of 97% was achieved under TC-30 • C and with a water flow rate of 0.40 l min − 1 .The optimum operating conditions for Server #4 are reached with a flow rate of 0.53 l min − 1 under TC-30 • C, 0.5 l min − 1 under TC-35 and 45 • C, and 0.55 l min − 1 under TC-40 • C. Therefore, the best yield of 97% was achieved under TC-30 • C with a water flow rate 0.53 l min − 1 .

Fig. 7 .
Fig. 7. Server #2: (a) IT equipment temperature, (b) IC fluid temperature, (c) water temperature difference, and (d) heat and IT power under different operating conditions.

Fig. 8 .
Fig. 8. IT electrical power and thermal heat load under different operating conditions for (a) Server #3 and (b) Server #4.

Fig. 9 .
Fig. 9. (a) Temperature frequency in hours for the climatic data of Roubaix during 2018; (b) electrical consumption of the DC cooling system with respect to the external temperature of the DC in Roubaix for the three heat rejection systems: IFC + EC, IFC+5K + EC, and IC.
/ODie = R TIM1(I/ODie) + R Die(I/ODie material) R core Dies = R TIM1(core Dies) + R Dies(core Dies material) x and y represent the repartition of power density as a function of surface area between the CPU's core dies and I/O die.R TIM1(I/ODie) is the thermal resistance of Thermal Interface Material: TIM 1 existing between the I/O die and CPU's casing.R TIM1( core Dies) is the thermal resistance of Thermal Interface Material: TIM 1 existing between the core dies and CPU's casing.R Die( I/ODie material) and R Dies( core Dies material) is the thermal resistance of I/O die and core dies material respectively.Calculation example: AMD socket SP3 Processor is one of the CPU used during the experiments.Fig. 10(b) shows the AMD socket SP3 Processor geometry.Table 11 presents all processor details and the methodology to calculate R CPU and R TIM2 (CPU− WB) .Based on these thermal resistances and knowing the Junction temperature T j and water temperatute T w and CPU power, T s and T C are computed as well as R WB− water .For T w = 34 • C; T j = 75 • C and CPU power = 205 W gives R junction− water = 0.2 K/W.

Fig. 10 .
Fig. 10.(a) Electrical analogy for the heat transfer within the CPU junction point-water configuration, (b) AMD socket SP3 Processor and water block geometry view.
IT equipment input read from the metered rack (e.g., plug strips) that monitors an aggregate set of IT systems M.Hnayno et al.

Table 3
Comparison between the studied technique and the IC systems described in the literature.

Table 5
Study parameters vs tests.

Table 6
Chemical and physical properties of the dielectric fluids used in the experiments.

Table 7
Impact of different dielectric fluids on cooling demand repartition.

Table 8
Distribution of power consumption of IT equipment for the studied servers.