Direct-Chip Cooling project team consisting of staff from EETD, HPC Services, Asetek, Cisco, and Intel
More than ever before, today's supercomputers require an increasing amount of power and cooling to operate. Faced with the prospect of having to limit its high performance computing capability, the IT Division, in conjunction with the Lab's Energy Environment Technologies Division and industry partners, embarked on a program to increase the capacity of it's 40-year old 5600 sq ft data center while implementing energy efficient strategies to make the most of the existing facility.
Wireless Sensor Monitoring
In 2007, the IT Division engaged Synapsense, which at the time was just beginning development of its real-time, wireless, monitoring application, to deploy a system which would permit detailed analysis of the environmental conditions (humidity and temperature) along with air pressure and power monitoring at hundreds of points within the datacenter. Once this system was deployed, the team used CFD modeling and this data to begin to change the airflow and make other operational adjustments in the datacenter. The team undertook a variety of fixes, some small, and some large:
These measures taken together allowed the IT Division to increase it's scientific computing capability by over 50% since 2007 when the room was assumed to be at capacity. The culmination of this initial work occurred in November 2010, when LBL became one of the first organizations in the federal space, and among a handful of smaller data centers in the world, to be able to calculate and view the data center’s Power Utilization Effectiveness (PUE) in real-time. This critical metric, which indicates the power used by the infrastructure in the data center in comparison to the power used by the computers themselves, helps staff manage the data center on a dynamic basis to best achieve environmental goals.
Variable Speed Drive (VSD) Cooling
In 2013, IT and EETD worked with several vendors to evaluate the use of direct-to-chip water cooling for our compute clusters. Used extensively by PC gamers wanting to overclock their home PC processors for maximum performance, this technology is now being adapted for large scale use in the datacenter environment. By using non-chilled water in our datacenter, this should provide free cooling - up to the limits of the building plumbing - for our systems. As part of this project, we ran our regular mix of scientific jobs on the system and measuring the heat load rejected into the water system. The findings from this evaluation were presented by EETD research Henry Coles at at the sixth annual Data Center Efficiency Summit hosted by the Silicon Valley Leadership Group (SVLG).