HPE SGI ICE XA - Cheyenne

The Cheyenne supercomputer was installed at the NCAR-Wyoming Supercomputing Center (NWSC) during the second half of 2016, run through a rigorous acceptance testing period, and began full-scale production in January 2017. Retired at the end of 2023 after seven years of service, Cheyenne was one of the longest-serving supercomputers in NCAR history.

A 5.34-petaflops system, Cheyenne tripled its predecessor’s power. At launch, it ranked as the 20th fastest supercomputer on the Top500.org list. The system was originally slated to be replaced after five years, but then the COVID-19 pandemic struck, severely disrupting supply chains. Cheyenne was called upon to serve an additional two years—maintained and supported by a dedicated staff of about 20 NWSC engineers—until the necessary parts were available to build its successor. 

Cheyenne was procured as an SGI ICE XA cluster—HPE purchased SGI prior shortly before the system was installed—with 4,032 nodes, each with two 18-core, 2.3-GHz Intel Xeon E5-2697v4 "Broadwell" processors, for a total of 145,152 cores. Cheyenne nodes had two memory sizes: 3,164 nodes had 64 GB and 864 had 128 GB. The nodes were connected by a Mellanox EDR InfiniBand interconnect in a 9-D enhanced hypercube topology. The system used the SuSE Linux operating system and the Altair PBS Pro scheduler. Accompanying the compute hardware, Cheyenne also included a 40-PB storage resource based on DDN hardware and IBM SpectrumScale file system.

Cheyenne continued to improve on the power efficiency of NCAR's large-scale systems. With a peak power consumption of 1.75 MW, Cheyenne consumed only 25% more power than its predecessor, Yellowstone, while delivering twice the sustained computing performance on the typical NCAR workload.

In its lifetime, Cheyenne delivered over 7 billion core-hours, served over 4,400 users, and supported nearly 1,300 NSF awards. It played a key role in education, supporting more than 80 university courses and training events. Nearly 1,000 projects were awarded for early-career graduate students and postdocs. Perhaps most tellingly, Cheyenne-powered research generated over 4,500 peer-review publications, dissertations and theses, and other works.

Researchers used Cheyenne to run increasingly detailed models simulating complex processes and how they might unfold in the future. Scientists also harnessed its computing power to run multiple simulations, or ensembles, to quantify a given event’s probability—providing needed intelligence for policy and resource planning.

Examples of transformative research with Cheyenne include:

• Extremely high-resolution global simulations of the atmosphere. Researchers highlighted an underappreciated type of rapid intensification of hurricanes to help forecasters better anticipate such weather events.
• Insight into the impacts of wildfire emissions on air quality and climate. One study shows that smoke from massive Australian bushfires in 2019-20 helped generate a rare multi-year La Niña event. A separate study shows that increasingly large and intense wildfires in the Pacific Northwest have altered the seasonal pattern of air pollution across North America.
• New modeling approaches with large datasets. The Decadal Prediction Large Ensemble (DPLE), with a staggering 24,800 simulated years of climate information, helps identify the event types most predictable as far as a decade in advance. 

“Cheyenne was a special machine that has certainly earned its retirement,” said NSF NCAR Director Everette Joseph.