Lawrencium Configuration

Contents of this Section

Overview
Compute nodes
Interconnect
Storage
Technical summary

Overview

Lawrencium LR1 is a 212-node, 1696 processor Linux cluster equipped with a high performance, low latency Infiniband interconnect that is suitable for providing an stable, high performing resource for running wide diversity of scientific applications. Currently, it is coupled with a BlueArc high performance NFS storage server that provides a total of 48TB home and a Data Direct Networks 300TB Lustre parallel filesystem that provides high performance scratch space to users. Lawrencium was ranked at number 500 on the November 2008 Top500 list of the world's fastest supercomputers.

Compute Nodes (LR1)

Each node is a Dell poweredge 1950 equipped with two Xeon Intel quad-core 64-bit Harpertown processors (8 cores in all) on a single board configured as an SMP unit. The core frequency is 2.66GHz and supports 4 floating-point operations per clock period with a peak performance of 10.64 GFLOPS/core or 85 GFLOPS/node. Each node contains 16GB of memory. The memory subsystem has dual channel 1333 MHz Front Side Bus connecting to 667 MHz Fully Buffered DIMMS. Both processors share access to the memory controllers in the memory controller hub (HCM or North Bridge).

Compute Nodes (LR2)

LR2 is an additional pool of 158 compute nodes intended to augment LR1 to meet the increased computational demand from LBNL researchers. Nodes are a combination of IBM iDataPlex dx360M2, HP SL390, and Dell C6100 servers each equipped with two Xeon Intel hex-core 64-bit Westmere processors (12 cores per node) on a single board configured as an SMP unit. The core frequency is 2.66GHz and supports 4 floating-point operations per clock period with a peak performance of 10.64 GFLOPS/core or 85 GFLOPS/node. Each node contains 24GB of NUMA memory connected via a triple QuickPath Interconnect (QPI) channels. LR2 has a peak performance rating of 20TF, but provides better computational efficiency making it much faster than LR1 even though the peak rating is similar.

Interconnect

LR1 uses a high performance, low latency interconnect utilizing DDR Infiniband technology provides a bandwidth of 20Gb/s. The topology is a fat tree configured with a 3:1 blocking factor. Specifically, the thirty-six nodes of each frame are connected to 2 Cisco 7000D 24-port DDR Infiniband switches (leafs). (eighteen nodes per leaf switch). From each leaf switch, the remaining 6 ports are used to uplink to the core fabric consisting of six Cisco 7000D 24-port DDR Infiniband switches. A total of 6 frames are connected in this fashion to the core fabric to support the 212 LR1 compute nodes. The core fabric has capacity to support up to 432 nodes.

Storage

The cluster is equipped with high performance NFS storage from BlueArc with 48TB of 7200RPM SATA disk storage for home directory use and a Data Direct Networks S2A9900 providing 300TB of Lustre parallel filesystem for high performance scratch space.

Technical summary

Component	Description
System Name	Lawrencium
Login Host	lrc-login.lbl.gov
Operating System	CentOS 5 x86_64 (based on RedHat Linux)
Number of LR1 nodes (cores)	212 (1696)
Total LR1 Aggregate Memory	3.4 TB
LR1 Performance	18 TFlops (Peak)
LR1 Processor	Intel 2.66Ghz Xeon Quad core 64-bit Harpertown
LR1 Network interconnect	Double Data Rate Infiniband (20 gb/s)
Resource Manager	PBS Torque
Job Scheduler	Moab
Disk (Storage)	Bluearc NFS (9TB Scratch & 48TB Home) Data Direct Networks Lustre (300TB)
User Environment Management	Modules
Serial Compilers	Intel ver 10.1: Fortran 90/95, C, C++ GNU ver 4.1.2: Fortran 90/95, C, C++
Parallel Compilers	OpenMPI ver 1.2.7 (built with above Intel compilers)