1. Introduction
The worldwide demand for storage and processing of data has increased rapidly during the last several years and as a result, the quantity and the size of data centers are increasing [
1]. The development of sustainable facilities is at the same time very important since their energy consumption is extremely large. In 2012, 1.4% of the total energy consumption in the world originated from data centers [
2].
Uninterrupted operation is the most crucial requirement for data centers [
3]. Adequate cooling of the servers is therefore very important, while over cooling is a waste of energy. In average 40% of the total energy consumption in data centers is related to the cooling process [
4]. Energy efficient cooling systems are required in order to both keep the energy consumption as low as possible and to satisfy the temperature requirements in data centers. The general guidelines that specify allowable and recommended rack intake temperatures are at present 15–32 °C and 18–27 °C, respectively [
5].
Two alternative setups are commonly used in air cooled data centers; hard floor and raised floor configurations. Cold air is supplied directly to the room from the Computer Room Air Conditioner (CRAC) units if a hard floor configuration is used. In a raised floor setup, the CRAC units deliver cold air into an under-floor space instead, whereas the air then enters the data center through perforated floor tiles. It is common practice to place the perforated tiles right beneath the rack intakes in order to make it more likely that the cold air finds its way to the servers. Cold air bypass is one problem that may occur as a result of poor airflow management [
6]. The cold air then immediately returns to the outlet of the CRAC units without cooling the servers [
7], leading to loss of cooling capacity. To compensate for this, the supply flow rate must be increased to make sure that enough cold air is available at the server racks resulting in over-dimensioned cooling systems. Another example of poor air management is when the exhausted hot air from a server enters the intake side of a server rack, rather than the outlet of the CRAC unit. This phenomenon is referred to as hot air recirculation.
In order to efficiently manage the airflow, it is common to form hot and cold aisles by placing the server racks in parallel rows. When the front sides are facing each other cold aisles are formed, while back sides facing each other form hot ailses. Aisle containment is a strategy to minimize hot air recirculation and cold air bypass. Hot aisle containment prevents the hot air from mixing with cold air by using panels as a physical barrier to enclose the hot aisles. A ceiling void space can be used to isolate the hot air when moving towards the CRAC units. Cold aisle containment prevents the cold air supplied by the perforated tiles to mix with hot air by using panels to enclose the cold aisles. For an ideal cold aisle containment, the total flow rate through the perforated tiles and the total flow rate demanded by the server racks would be equal. However, there is often air leakage either from or to the cold aisles in existing data centers. The air leakage makes up for the excess or deficit in the supply flow rate, compared with the flow rate through the server racks. The aisles can also be partially contained by using panels on either side or on top of it. The potential to increase the thermal performance of a server room by the use of partial or full aisle containment has been verified in several studies, but mainly for cold aisles only. Srinarayana et al. [
8] numerically compared room and ceiling return strategies in both hard and raised floor configurations. Arghode et al. [
9] compared numerical and experimental results of a contained cold aisle to an open cold aisle for both the under- and over-provisioning of supply air. Sundaralingam et al. [
10] experimentally investigated the performance of the cooling system in different setups with either an open, partially contained or fully contained cold aisle. Alkharabsheh et al. [
11] used an experimentally validated numerical model to study both open and contained cold aisle setups. Nada et al. [
12] numerically studied partial cold aisle containment and also compared different arrangements of CRAC units and server racks with respect to airflow management. Values of flow rates through the perforated tiles and rack intake temperatures were used for validation. In several of the previous studies, computational fluid dynamics (CFD) was used to provide details of the airflow [
8,
9,
11,
12]. Further studies of CFD for airflow management can be found in the reviews by Chu & Wang [
13] and Lu et al. [
14]. In this context, it is crucial that the simulations are trustworthy. Boundary conditions, mesh and application of turbulence models should for example be carefully addressed to ensure reliable results [
11,
15]. Different levels of details are furthermore needed depending on whether the component level, rack level or room level is in focus [
13].
Detailed flow analysis of hard floor and raised floor setups in the SICS ICE data center in Luleå, Sweden, has previously been performed with numerical simulations [
7,
15,
16] and measurements [
17,
18]. The influence of a turbulence model on the fluid flow in a hard floor server room was investigated by Wibron et al. [
15], recommending the Reynolds stress model (RSM) for modeling the flow on a server room scale. Local details of the steady state and transient flow in hard floor and raised floor configurations were examined by Wibron et al. [
7]. The aim of this study is to investigate the effects of using partial aisle containment for the different floor types, i.e., considering both cool aisle and hot aisle containments. The software ANSYS CFX was used to model the flow and temperature fields inside the server room. Velocity measurements were used for validation of both floor types with open aisles. Uneven load distribution between servers and the RSM turbulence model were employed in all simulations. A combination of the performance metrics Rack Cooling Index (RCI), Return Temperature Index (RTI) and Capture Index (CI) were used to evaluate the performance of the cooling system for both hard- and raised floor configurations. The setups based on the conditions in the existing data center are very well cooled since the flow rate supplied by the computer room air handler (CRAH) units or perforated tiles exceeds the flow through the server racks. Therefore, the same setups with only 50% of the original supply flow rate are also investigated to study the possibility of increasing the energy efficiency through decreased supply flow rate.
2. Method
The geometry and boundary conditions retrieved from the experimental facility is first presented. Strategies for validation of the numerical simulations are then addressed, and finally the grid, simulation settings and performance metrics are discussed.
2.1. Geometry
A server room (Module 2) at RISE SICS North in Luleå, Sweden, was chosen for the study. In this data center it is possible to choose between hard or raised floor setups, which makes it suitable for the present study. In
Figure 1, the geometries for both floor types with open aisles are illustrated. The dimensions of the data center were 6.484 × 7.000 × 3.150 m
3. In
Figure 2, the main components and measurement locations are illustrated. The measurement locations are the same for both configurations. The locations were chosen from flow field considerations and were intended to capture the different flow conditions in the server room. The velocity was measured at the heights of 0.6, 1.0, 1.4, 1.8 and 2.2 m for each location to enable validation of the digital model with open aisles. Partial aisle containment was then imposed in the simulations by using either a side or top cover to partially enclose the aisle between the two rows of server racks. The side cover was located at the end of the rows and at the end of the perforated tiles for the hard and raised floor configurations, respectively. The top cover was located on top of the server racks and reached from the wall out to the end of the rows for both configurations.
In the hard floor configuration, the main components were 10 server racks composed of 0.8 m wide cabinets from Siemon and four computer room air handler (CRAH) units of the label SEE Cooler HDZ-2. The servers inside the racks were not limited to one manufacturer, and temperatures of the inlets and outlets of the server racks were therefore measured for input to the numerical model. The location of the inlets and outlets of the servers were furthermore retrieved from measurements in the server room. In the raised floor configuration, there were fourteen perforated tiles located in the cold aisle. Covers on the front sides of the CRAH units were used to direct the cold air under the floor. In the experiments, there were two different kinds of perforated tiles with percentage open areas of 56% and 77%, respectively. The under-floor space was not taken into account in the simulations.
An uninterruptible power supply (UPS) was also present in the data center. To include the effect of flow blockage the UPS was modeled without heat loads, while smaller obstructions, i.e., cables and pipes, were ignored.
2.2. Boundary Conditions
Flow through the servers were not resolved in this study. Boundary conditions were instead applied to the inlet and outlets of the servers and the CRAH units, i.e., the so-called black box model [
19] was used. The supply temperature and the flow rates from both the CRAH units and the perforated tiles were determined from measurements. The hotwire thermo-anemometer VTlO0 from Kimo was used to measure the temperatures, while flow rates were measured with the DP-CALC Micromanometer Model 8715 and the Velocity Matrix add-on kit from TSI. The Velocity Matrix provides area-averaged multi-point face velocity measurements. It calculates the average velocity based on the difference between total and static pressures [
20]. The accuracy of the velocity and temperature measurements are ±3% of reading ±0.04 m/s for the velocity range 0.25–12.5 m/s and ±0.4% of reading ±0.3 °C for temperatures from −20 °C to 80 °C. The supply temperature was measured to 18.3 °C for both configurations and hence independent of the measurement location. For the hard floor configuration, the flow rates were measured through the front sides of all the CRAH units. Each front side was divided into 15 measurement zones and the inlet conditions were determined from average velocities. The measured values for the CRAH units are presented in
Table 1, showing only a small difference in velocity between the units.
For the raised floor configuration, the flow rates were measured through all the perforated tiles (
Table 2). Each perforated tile was divided into five regions and the average velocities were used to calculate the inlet boundary conditions. To resolve the perforated tiles a very fine mesh is required when using CFD. A fully open tile model is however regarded to be adequate for tiles if the percentage of open area is 50% or more [
21]. Therefore, the perforated tiles were modeled as open in this work. As observed in
Table 2, there was a variation in velocity through the individual tiles. A possible reason for this was the different loads of the server racks presented in
Table 3 and the geometrical effects of the underfloor spacing. It was furthermore assumed that the mass flow rates supplied into the room where equal to the mass flow rate out of the room. The outlets were located on top of the CRAH units.
The heat load produced by the servers results in a temperature difference across the servers which can be obtained from:
where
q is the heat load,
m is the mass flow rate and
cp is the specific heat capacity. A constant value of
cp = 1004.4 J/KgK was used in all simulations. The flow rates were estimated by measuring the temperature difference across each server rack with temperature sensors from Raritan called DPX2-T1H1. The sensor has an accuracy of ±1 °C for temperatures from −25 °C to 75 °C. The sensors were placed 1.08 m above the floor on both sides of all server racks. The temperature difference was used to compute the mass flow rate through each server rack. The load of each server rack as well as the corresponding mass flow rates are presented in
Table 3 for the raised floor configuration, and in
Table 4 for the hard floor configuration. The total heat load in the server room were thus similar for the two cases, while a distinct variation of heat load was present for the different racks. All walls were assumed adiabatic; hence, no heat losses from the server room were accounted for. A no-slip boundary condition implying zero velocity at all surfaces was furthermore applied. Leakage of air from the server room was not included.
2.3. Validation
Velocity was measured for both raised and hard floor configurations with open aisles to validate the simulations. The experimental approach was similar to the one presented in Wibron et al. [
15]. The anemometer ComfortSense Mini from Dantec with the omnidirectional 54T33 Draught probe was used in the experiments. The accuracy is ±2% of reading ±0.02 m/s for the velocity range 0.05–1 m/s, and ±5% of reading for the velocity range 1–5 m/s. Velocity was measured in the locations L1-L5 presented in
Figure 2, at five heights (0.6 m, 1.0 m, 1.4 m, 1.8 m and 2.2 m). The chosen duration, 60 s, and sampling frequency, 10 Hz, of the measurements was after evaluation considered sufficient for the time-averaged velocities.
2.4. Grid Study
The raised floor configuration with an open aisle was chosen for the grid study. Following the methods described in [
15], Grid Convergence Index (GCI) and Richardson’s extrapolation [
22] was utilized to evaluate the discretization error. The solutions of three tetrahedral grids created in ANSYS Meshing were compared in the study. The refinement factor of the grids was determined from:
The number of elements are represented by
N and the number of dimensions by
D. Error bars for the computed profiles was determined using the GCI with an average value of
p =
Pave as a measure of the global order of accuracy [
23]. For a constant refinement factor, the order of
p was calculated as:
where
ε32 =
ϕ3–
ϕ2,
ε21 =
ϕ2–
ϕ1.
ϕ3,
ϕ2 and
ϕ1 are the solutions on the coarse, medium and fine grid, respectively. The relative error was approximated from:
and the convergence index of the fine grid was:
The extrapolated values were:
2.5. Simulation Settings
Time-averaged flow fields from transient simulations were used in the analysis since steady-state simulations did not converge due to fluctuations in the flow field. Values of velocity and temperature were logged at locations L1-L5 to ascertain that the simulations reached a statistically steady state. After a start up simulation period of 60 s, transient averages where recorded for 360 s to obtain time-independent results. A constant time-step equal to 0.05 s was decided on the basis of convergence. The time-step corresponded to an RSM Courant number of around 2 at operating conditions. The convergence criteria were set to 10−5 for each time-step.
A common choice of turbulence model for data center applications is the
k-
ε model. However, the
k-
ε model is less accurate in regions of low velocity and the Reynolds stress model (RSM) is therefore a better choice [
15]. Therefore, the RSM was used to model turbulence in this study. The RSM does not assume isotropic turbulence. Instead, it solves additional transport equations for the six individual Reynolds stresses in order to obtain closure of the Reynolds-averaged Navier–Stokes (RANS) equations that govern the transport of the time-averaged flow quantities. A wall function approach was used in ANSYS CFX to model the boundary layer where the velocity approaches zero close to solid walls.
Buoyancy effects should also be taken into account when modeling the airflow in data centers [
24]. The Archimedes number represents the ratio of natural and forced convection and is expressed by:
where
L is a vertical length scale, Δ
T is the temperature difference between hot and cold regions and
V is a characteristic velocity. An Archimedes number of order 1 or larger indicates that natural convection dominates. In this study,
Ar ≈ 2 and buoyancy is therefore included with the full buoyancy model in ANSYS CFX.
2.6. Performance Metrics
The Rack Cooling Index (RCI) is a measure of how well the servers are cooled and maintained within thermal guidelines [
25,
26]. Over-temperature conditions exist when the intake temperature at some server racks exceed the maximum recommended temperature. A summation of the over-temperatures of each rack gives the total over-temperature. RCI can be used to evaluate under-temperature conditions as well, but only the high side of the temperature range is analyzed here. The RCI was defined as:
where the maximum allowable over-temperature is a summation of the difference between the maximum allowable and maximum recommended temperature across all the server racks. The RCI reaches its ideal value of 100% if all rack intake temperatures are below the maximum recommended temperature. Values below 100% mean that the intake temperature for one or more server rack exceeds the maximum recommended temperature.
The RCI can be improved by increasing the flow rate or decreasing the temperature. However, these actions would increase the energy consumption. The Return Temperature Index (RTI) is a measure of the energy performance of the cooling system [
27]. It is defined as:
where
Treturn,
Tsupply and Δ
Tequip are weighted averages of the return temperature, supply temperature and temperature difference across the server racks, respectively. Deviations from 100% indicate declining performance. Values above 100% indicate mainly hot air recirculation and values below 100% indicate mainly cold air bypass.
The Capture Index (CI) is based on the airflow patterns associated with either the supply of cold air or the removal of hot air [
28]. The CI based on the supply of cold air is considered here and it is defined as the fraction of air that originates from the CRAH units. The ideal value is 100% which means that there is no hot air recirculation. Otherwise, the value is lower.
4. Conclusions
The effects of using partial aisle containment in the design of data centers were investigated in this study using an experimentally validated CFD model. In general, good agreement was found between simulations and experimental values. Hard and raised floor configurations with either open or partially contained aisles were compared with respect to airflow management. The setups based on the conditions in the existing data center were very well cooled since the supply flow rate exceeded the flow rate required by the server racks, and the values of RTI were relatively low, about 40% and 50% for the hard and raised floor configurations, respectively. Hence, the indices indicate that the servers are cooled and maintained within the thermal guidelines but the energy performance is poor. Therefore, the same setups with only 50% of the original supply flow rate were also investigated. The supply flow rate was then approximately the same as the flow rate through the server racks. Reducing the supply flow rate gave more apparent hotspots and the values of RTI were raised to over 80% in all setups, but the same tendencies as in the setups with 100% supply flow rate were found.
Based on the combination of performance metrics where both rack intake temperatures and airflow patterns were taken into account, the airflow management was significantly improved in the raised floor configurations. For the hard floor configurations, there were hotspots close to the end of the rows. The side cover reduced the extent of these. However, using the top cover reinforced the hot air recirculation around the rows which declined the performance of the cooling system. For the raised floor configurations, partially contained aisles were preferred compared with open aisles, and the side cover performed better than the top cover, especially in the case of decreased flow rate.
In addition to hot spots at the end of the rows, potential hot spots were observed especially in the vicinity of the areas of high server load, hence individual supply flow rates between computer room air handling (CRAH) units should be further investigated.