Peeling Back the Darkness

TACC's supercomputers play pivotal role in the Event Horizon Telescope's first-ever black hole image

On April 10, a team of researchers from around the world revealed an image that many believed impossible to produce: a portrait of the shadow cast by the black hole that sits at the center of the galaxy Messier 87 (M87) — 53 million light years away.

A luminous orange circle with a dark center and a bright lip, the image was a product of the Event Horizon Telescope (EHT), a distributed collection of eight, high-altitude radio telescopes scattered around the globe that, when combined, form an Earth-sized observatory capable of capturing distant radio waves with a clarity not possible before.

"We are delighted to report to you today that we have seen what we thought was unseeable," said Shep Doeleman, project director of the Event Horizon Telescope, at the announcement event in Washington D.C.

“It’s the most important image in the history of astronomy,” UT Austin astronomer Karl Gebhardt said. “As important as the sketches of the moons that Galileo made of Jupiter.”

“It is very hard to claim what EHT observed is really a black hole without theoretical explanation,” Shiokawa said. “It is impossible to predict the image by incorporating all the complicated physics that includes many non-linear features, such as magnetized turbulence, without numerical simulations. The simulations aid scientists to interpret what is observed based on reliable physical theories.”

After modeling the physics of M87, further simulations converted the dynamics of the black hole into a picture of what such a system would look like from the perspective of Earth using ray tracing methods.

"You pretend you're an observer looking at the system from some viewing angle," said Narayan. "You ask: ‘What would I observe if this were the real system? What kind of image would I see at this spectrum?'"

The simulations and visualizations helped scientists confirm that the image obtained through observation matched up with reliable physical theories. The first-of-its-kind image also helped fine-tune aspects of the theoretical astrophysical models, including properties of accretion disks — the matter swirling around the edge of the black hole — and jets — energy spinning out of a black hole and shooting through space.

University of Illinois at Urbana-Champaign professor of Physics and Astronomy Charles Gammie, a member of the EHT Science Council Board who co-led the theory working group and served as co-coordinator of Paper V, was another researcher who used TACC resources to interpret the EHT data.

"The computing resources at TACC were crucial in enabling our survey of black hole spin, and in the development of our GRMHD (General Relativity Magnetohydrodynamic) code, our radiative transfer code, and other analysis codes, over the last decade," Gammie said.

Using Stampede2, Gammie produced simulations of accretion disks and jets related to the EHT targets.

"M87 is the nearest galaxy with a supermassive black hole that’s generating a powerful jet,” Gammie noted. “One of the great mysteries in astronomy has been how such jets are launched. Our simulations, which are based on the motion of magnetic fields and hot gas near the black hole, showed that the jets are powered by the black hole itself."

BRINGING THE BLACK HOLE TO THE CLOUD

Alongside the simulation and modeling effort, another group of researchers from the University of Arizona used Jetstream — a large-scale cloud environment for research located both at TACC and Indiana University — to develop cloud-based data analysis pipelines. These proved crucial for combining huge amounts of data taken from the geographically-distributed observatories, and sharing the data with researchers around the world.

"These groundbreaking results show how discovery and imagination are fueled by advanced computing systems combined with the talented researchers and developers that use them."

The data from the telescopes was collected during a 2017 global campaign after decades of scientific, engineering, and computational research and preparation. Results of the work describing various aspects of the research were published in six papers in the Astrophysical Journal Letters in April 2019.

TACC was cited four times in these papers for providing critical resources that helped lay the groundwork for the black hole imaging, and powering the theoretical calculations that enabled researchers to interpret the mass, underlying structure, and orientations of the black hole and its environment.

"For decades, we have studied how black holes swallow material and power the hearts of galaxies," said Harvard University professor and EHT researcher Ramesh Narayan, who used TACC resources in support of the project. "To finally see a black hole in action, bending its nearby light into a bright ring, is a breathtaking confirmation that supermassive black holes exist and match the appearance expected from our simulations."

Several teams of researchers, including Narayan, used TACC’s Stampede1 and Stampede2 supercomputers to model the physical attributes of M87 and predict observable features of the black hole.

"A good part of the work for these papers was doing the simulations, observing them on the computer, seeing whether we would get a ring or not, and trying to decide how close are the simulations to what is actually observed,” Narayan said. “If the simulation had come up with something completely different than what the image observed was, that would put the process a little bit in question. But if they are very similar, as they were, then you feel confident that the image is an accurate one and a good representation.”

The predicted images were also used to validate fundamental theories such as general relativity, according to Hotaka Shiokawa, a postdoctoral fellow at the Harvard-Smithsonian Center for Astrophysics and another member of the EHT team. Shiokawa used TACC systems to run relativistic magnetohydrodynamics simulations — simulations that include the magnetic properties of electrically conducting fluids which are believed to be key determinants of black hole features.

"New technologies such as cloud computing are essential to support international collaborations like this," said Chi-kwan Chan, leader of the EHT Computations and Software Working Group and an assistant astronomer at the University of Arizona. "The production run was actually carried out on Google Cloud, but much of the early development was on Jetstream. Without Jetstream, it is unclear that we would have a cloud-based pipeline at all."

Using Jetstream, Chan and his colleagues removed noise from the data and reduced the data from five petabytes to tens of terabytes so it could be more easily analyzed by research groups around the globe. They also prototyped a pipeline that removed systematic errors from the data.

The team received a grant from the National Science Foundation (NSF) to deploy the same pipeline for data analysis on Google Cloud, but "the only way we were able to use this cloud was because Jetstream was available to us to do all this development," Chan said.

KNOWING WHERE TO LOOK

If not for TACC supercomputers, scientists may not have tried to image M87 at all. Back in 2004, when Gebhardt studied the very galaxy where M87 resided, the black hole mass was thought to be relatively low.

“Using the McDonald Observatory and TACC’s computer at the time, I measured the mass of M87,” Gebhardt recalled. “We had to use a relatively complicated model — it took months of time on TACC computers — and the mass I came up with was a factor of two to three times higher than what was observed in the past. The astronomers running the Event Horizon Telescope project would not have looked at this galaxy, so they tell me, if I hadn’t published a study of how high that mass was. So as we always say, even black holes are bigger in Texas.”

With powerful TACC supercomputers at their disposal, researchers turned theoretical models of black holes into specific predictions about M87, which led the astronomers to launch the project and ultimately gave them confidence that the image they produced was accurate.

The first image of a black hole is both the end of one long road, and the beginning of another. Work to interpret the results of this major discovery will continue for many years with help from TACC supercomputers.

"These groundbreaking results show how discovery and imagination are fueled by advanced computing systems combined with the talented researchers and developers that use them," said Niall Gaffney, TACC's director of Data Intensive Computing.

"This is just the beginning of our understanding of the extreme gravitational environments of black holes. We at TACC will continue to provide the world-class support and services that lead to these and other fundamental discoveries about the nature of the Universe in which we live."