The present compute system at NHR@ZIB is system Lise which is named after the physicist Lise Meitner. It consists of CPU and GPU partitions, holding more than 120,000 CPU cores, more than 150 GPUs, and leading to a peak performance of almost 9 Pflop/s. All compute nodes are attached to global file systems, Home, Work, and type archive. The system consists of the following partition for CPUs and GPUs.

• CPU partitions
  • Intel CLX
  • AMD Genoa
• GPU partitions
  • Nvidia
  • Intel PVC

CPU partition Intel CLX

Since the installation of the Atos/Bull vendor in the end of 2019, the NHR@ZIB had operated a CPU system with 1270 CPU nodes. At the time of the installation of the AMD Genoa partition in 2024, the system size of the Intel CLX partition has been reduced to 982 nodes.

• 982 compute nodes
  • 948 nodes with 384 GB main memory (standard node)
  • 32 nodes with 768 GB main memory (large node)
  • 2 nodes with 1.5 TB main memory (huge node)
  • per node two CPUs and one Intel Omni-Path host fabric adapter
  • per CPU one Intel Cascade Lake Platinum 9242 (CLX-AP), 48 cores
• Omni-Path network configuration
  • 2 x 1162 port OPA100 director switches
  • 56 x 48 port edge switches
  • Fat Tree topology
  • 14 TB/s bisection bandwidth
  • 1,65 μs maximum latency
• 8 login nodes
  • per node two CPUs with 384 GB main memory
  • per CPU one Intel Cascade Lake Silver 4210 (CLX), 20 cores

CPU partition AMD Genoa

• 168 compute nodes
  • per node 2 CPUs and 1.5 TB main memory
  • per node 3.5 TB NVMe local SSD disk space
  • per node 400 Gbit/s Infiniband NDR interconnect
  • per CPU 1 AMD EPYC 9654, 96 cores

GPU partition Nvidia

• 42 compute nodes
  • per node two Intel CPUs, 1 TB main memory
  • per node four Nvidia A100 80GB HBM GPUs with NVLink 3.0
  • per CPU one Intel Xeon 8360Y, 2,4 GHz 36 cores

 GPU partition Intel PVC

• 8 compute nodes
  • per node two Intel CPUs und 4 Intel Xe GPUs
  • CPU: Intel Xeon
  • GPU: Intel Xe Ponte Vecchio HPC, 128 GB HBM

Storage systems

System Lise is attached to three file systems.

Home and Work file system

The Home file system is backuped continuously. Here we feature daily snapshots for user files.

The Work file system is a fast storage for large data holding a temporary scratch character. Because no backup is available, data backup is under responsibility of the projects.

• 20 PB storage capacity
• 1 PB SSD cache
• 185 GB/s aggregated performance for write and read

Both file systems Home and Work are configured on the same storage system, from the technical point of view. It is based on an IBM Storage Scale System. The storage system is based on rotating disks complemented by an automatic SSD cache for acceleration. The system supports quotas for volume and inodes, for groups and users respectively. The quotas can be configured per directory and include soft and hard limits as well as grace periods.

Archive system

For medium and long term data management the NHR@ZIB offers an archive system based on a tape archive. This archive consists of a hierarchical storage management system (HSM) with a control software based on Sun StorEdge SAM-FS.