Powering Discoveries
Jorge Salazar
Texascale Days
A win-win for researchers and TACC
What would you do with the most powerful academic supercomputer in the U.S. at your fingertips?
Texascale Days at TACC allows awarded scientists exclusive access to the entire Frontera supercomputer. This quarterly event gives researchers an opportunity to run computing jobs that cannot be performed anywhere else in the world.
Deployed in 2019, Frontera is an HPC capability system intended for large applications requiring thousands of nodes and serving all domains of open science research.
“Any researcher who needs to experiment at a larger scale can run during Texascale Days,” said John Cazes, director of High Performance Computing at TACC. “They have to demonstrate that they can run their science at the large scale that we regularly make available to them, which is up to 2,000 nodes. Then we help them grow their application to run on 4,000 nodes, then to the full 8,192 nodes offered on Frontera.”
From Fall 2022 to Spring 2023, more than two dozen projects benchmarked, tested, and completed production runs on code that needed at least half of Frontera's compute nodes — several projects used the entire system.
What’s more, Texascale Days lays the groundwork for even bigger opportunities on future HPC leadership class systems.
“We're giving our researchers a head start toward expanding their codes to fit on the increased processing power we're going to have on Horizon,” Cazes added, referring to the supercomputer that will be developed by TACC for the NSF-funded Leadership Class Computing Facility.
Texascale Days also helps TACC ‘stress-test' Frontera to make improvements that add stability to the system and benefit all users.
“It’s thrilling to see these jobs run on the whole system in just a matter of a few minutes,” Cazes said.
The following are key projects that ran during Texascale Days. The scientific applications cover a wide range of disciplines including hypersonic flight, materials science, deep Earth imaging, quantum circuits, black holes, and stellar evolution.
Hypersonic Flight
Daniel Bodony
University of Illinois at Urbana-Champaign
Airflow moving over a plane’s wing at Mach 6 generates thermal and pressure loads that can change the shape of the airfoil through thermal expansion, thermal softening, and mechanical deformation.
Understanding the aerodynamic field and how it interacts and is affected by the airfoil is critical to enabling sustained hypersonic flight.
“As Frontera and our code have co-matured, running on 4,096 nodes has become routine,” said Daniel Bodony, professor of Aerospace Engineering at the University of Illinois at Urbana-Champaign.
Frontera’s scale enabled the Bodony Research Group to use high-fidelity models for the fluid and solid domains and to examine their time-dependent coupling.
“Reproducing the relevant flight conditions in ground-based experiments is not currently feasible, so supercomputers provide the scientific discovery possibilities.”
Deep Earth Imaging
Ebru Bozdag
Colorado School of Mines
Improving the resolution of seismic images is essential to understand the origin of our planet, mantle hotspots, plate tectonics, volcanoes, and earthquakes.
"For each of our simulations during Texascale Days, we had 8,000 nodes on Frontera available, and some were able to scale to the limit," said Ridvan Örsvuran, a postdoctoral researcher at the Colorado School of Mines. He ran 250 real earthquakes simultaneously, covering the entire planet.
"In order to do these studies, we take the full complexity of global seismic wave propagation in our simulations. That's why we need large computational resources like Frontera," added Ebru Bozdag, principal investigator of the project.
Black Hole Jets
Kenichi Nishikawa
Alabama A&M University
The accretion disk surrounding a black hole in space can supply it with plasma and magnetic fields. A small fraction of the plasma in the disk can be magnetically diverted into jets that, paradoxically, are ejected out at large distances from the host galaxy.
The jets can also be powered by the rotational energy of the black hole, referred to as relativistic jets.
“We performed very large simulations during Texascale Days using 4,000 Frontera nodes. We successfully ran the code to resolve a new type of particle distribution in the plasma that we hope to provide us with a more realistic description of plasma in relativistic jets from black holes,” said Kenichi Nishikawa, a research professor at Alabama A&M University.
“Supercomputers support our efforts in understanding these fascinating black holes and the fundamental forces that shape our universe.”
Making Diamonds
Ivan Oleynik
University of South Florida
Extreme conditions of high temperature and pressure are created by a shock wave — a wave that moves with supersonic speed. If strong enough, it can induce the transformation of graphite into a diamond.
”During Texascale Days, we ran 200 million atomic molecular dynamics simulations of shocked graphite using quantum accurate machine learning interatomic potential for carbon,” said Ivan Oleynik, a professor in the Department of Physics at the University of South Florida. “We observed how graphite transforms into diamond at the atomic scale.”
“Quantum accuracy comes at a significant computational cost. It was critical for us to have access to all of Frontera’s 8,192 nodes to obtain meaningful results at experimental time and length scales that can be directly compared with experiment.”
Silicon Nanocrystals
James Chelikowsky
UT Austin
Silicon is the material that enabled the current technological age with the invention of the transistor and integrated circuits. Because of this technological interest in silicon, it is also widely studied from a science perspective.
"Our approach enables us to study nanocrystals of silicon and access this size regime for the first time using quantum methods," said James Chelikowsky of the Oden Institute for Computational Engineering and Sciences. His team invented new algorithms that are designed for nanoscale systems.
The project ran a system with more than 200,000 atoms on Frontera using 8,192 nodes during Texascale Days.
“As computational scientists, we're able to guide and aid experimental scientists thanks to our access to cutting-edge supercomputers.”
Inside a Star
Paul Woodward
University of Minnesota
Falk Herwig
University of Victoria
For this project which ran on 3,510 nodes of Frontera, Paul Woodward’s team at the University of Minnesota collaborated with Falk Herwig at the University of Victoria to study the process of material mixing at the boundaries of convection zones in stellar interiors.
"During two Texascale Days events, we performed very large simulations of a 25 solar-mass stellar model rotating at different speeds,” Woodward said.
The researchers are seeking to discover how their 1D model for the equilibrium convection zone structure is affected by rotation of a star.
"The tremendous computing power of Frontera enables accurate investigation of subtle internal gravity wave effects, which over time can have very significant impacts on the structure and evolution of stars.”
Electron-Phonon Calculations
Feliciano Giustino
UT Austin
The materials we know today are just a tiny fraction of what is possible.
“Supercomputers are key to exploring this valuable and gigantic space,” said Feliciano Giustino of the Oden Institute for Computational Engineering and Sciences.
Giustino is co-developer of the Electron-phonon Wannier (EPW) open-source code which calculates properties related to the electron-phonon interactions of materials.
His team used the Frontera supercomputer at half and full scale to carry out electron-phonon calculations with the EPW code.
"We demonstrated in both cases that the speedup for the full-system run with respect to the half-system run reaches above 75 percent,” Giustino said. “We believe that this low-I/O feature is pushing the limits of EPW in massively parallel calculations.”
Earthquake Impact
Yifeng Cui
San Diego Supercomputer Center
Large-scale high-resolution earthquake simulations are helping to transform earthquake system research from computational seismology into predictive science.
“The simulation we ran during Texascale Days used 7,680 Frontera nodes and ran for 22.5 hours to simulate 83 seconds of shaking duration,” said Yifeng Cui, director of the High Performance GeoComputing Laboratory at the San Diego Supercomputer Center. “It confirmed the sensitivity of ground motions in the Los Angeles basin to the strength of the sedimentary deposits.”
These simulations are helping scientists predict shaking at the higher frequencies relevant to common, low-rise buildings — something that was not possible in previous low-resolution simulations.
“The simulation is based on the ShakeOut scenario, a potential magnitude 7.8 earthquake on the southernmost San Andreas fault, and part of the largest-ever earthquake preparedness exercise involving millions of people worldwide who participate in the drill annually.”