The present compute system at NHR@ZIB is system Lise which is named after the physicist Lise Meitner. It consists of CPU and GPU partitions, holding more than 120,000 CPU cores, more than 150 GPUs, and leading to a peak performance of almost 9 Pflop/s. All compute nodes are attached to global file systems, Home, Work, and type archive.

 

CPU partition

Since the installation of the Atos/Bull vendor in the end of 2019, the NHR@ZIB operates a CPU system with 1270 CPU nodes and 121,920 CPU cores.

• 1270 compute nodes
  • 1236 nodes with 384 GB main memory (standard node)
  • 32 nodes with 768 GB main memory (large node)
  • 2 nodes with 1,5 TB main memory (huge node)
  • per node two CPUs and one Intel Omni-Path host fabric adapter
  • per CPU one Intel Cascade Lake Platinum 9242 (CLX-AP), 48 cores
• Omni-Path network configuration
  • 2 x 1162 port OPA100 director switches
  • 56 x 48 port edge switches
  • Fat Tree topology
  • 14 TB/s bisection bandwidth
  • 1,65 μs maximum latency
• 8 login nodes
  • per node two CPUs with 384 GB main memory
  • per CPU one Intel Cascade Lake Silver 4210 (CLX), 20 cores

 

GPU A100 partition

• 42 compute nodes
  • per node two Intel CPUs, 1 TB main memory
  • per node four Nvidia A100 80GB HBM GPUs with NVLink 3.0
  • per CPU one Intel Xeon 8360Y, 2,4 GHz 36 cores

 

GPU PVC partition

• 8 compute nodes
  • per node two Intel CPUs und 4 Intel Xe GPUs
  • CPU: Intel Xeon
  • GPU: Intel Xe Ponte Vecchio HPC, 128 GB HBM

Storage systems

System Lise is attached to three file systems.

 

Work file system

The Work file system is dedicated to large amounts of data.

• 10 PB netto storage capacity
• 80 GB/s aggregated write and read bandwidth

The Work file system is based on a DDN EXAScaler with a parallel DDN-Lustre file system. It contains 2 x DDN ES18K embedded storage systems with integrated OSS servers and a scalable MDS unit. Four of 32 object storage targets (OST) are equipped with non-volatile memory express (NVMe) on SSDs which allows FastIO for high-performance IO applications with random access patterns. This area of scratch data is not included into a backup mechanism.

 

Home file system

The Home file system is dedicated to small scale data like source code and executable files.

• 340 TB netto storage capacity
  • 280 TB user data
  • 60 TB for snapshots, system data, and user software

The Home file system is based on a DDN GRIDScaler (Spectrum Scale) with a DDN SFA7700X block storage. It is exported via NFS to all login and compute nodes, it offers disk quotas with respect to groups and users for capacity and inodes. A backup mechanism saves the data frequently and daily snapshots help the users to recover removed data by accident.

 

Archive system

For medium and long term data management the NHR@ZIB offers an archive system based on a tape archive. This archive consists of a hierarchical storage management system (HSM) with a control software based on Sun StorEdge SAM-FS.