The World’s Largest Telescope Will Generate More Data Than the Entire Internet

Astronomers and engineers across the globe are about to build the world’s largest radio telescope. Well, actually it’s more like many telescopes. The Square Kilometre Array (as it’s called) will be made up of thousands of dishes and up to a million antennas linked together across South Africa and western Australia. The telescope will be able to look deep into space in search of things like gamma-ray bursts, exoplanets, and dark matter. A telescope this big will produce a massive amount of data, about 10 times the current annual global internet traffic, which will require a huge amount of computing clusters with the processing power of about one hundred million PCs. Next comes the challenge of making such a huge amount of data usable for astronomers and researchers. That’s where software engineers like Zheng Meyer-Zhao come in. Zheng is the Science Data Centre Development Lead at ASTRON, the Netherlands Institute for Radio Astronomy. ASTRON is one of around 100 organizations in 20 countries working on the Square Kilometre Array.

“It’s not very intuitive to process your data on a computing cluster or supercomputer instead of on your laptop,” explains Zheng. “When your data set gets too big you go to a computing cluster which is different because it has more cores and more memory. You need to know things like how to submit a job to the queue instead of doing it however you want because the computing resources are shared.” Just getting the data to the point where researchers and astronomers can use it requires people with knowledge of data processing. They have to figure out how to make the data easily searchable and accessible. After the data is found, how will researchers process it? If a researcher wants to publish or share their data, how will they save it? A laptop simply doesn’t have enough storage to do all of this and, since the Square Kilometre Array team is so international, researchers need to be able to do all of this remotely.

ASTRON’s Science Data Centre will bridge this gap and get researchers and astronomers onto the system to process their data. Zheng is leading the development of the web-based science analysis platform that scientists will use to access the computing clusters and data storage. The project is in its early stages, but the team would like to have a minimum viable product with selected data sets ready by the end of the year. From there, they will add more components and eventually connect the platform to other computing clusters outside the Netherlands. “All the instruments that ASTRON is building and all the data that has to come in and be processed, it’s basically a challenge. You don’t see this anywhere else and that makes it really special,” says Zheng.  

Before joining ASTRON, Zheng was a cluster computing and high-performance computing advisor at the Dutch national supercomputer provider where she refined the user support skills she now uses to develop ASTRON’s science analysis platform. She started her job at ASTRON about a year ago, but her connection with ASTRON goes back 10 years to when she worked as a grid expert at the Centre for IT of the University of Groningen. At the time ASTRON was in the process of constructing LOFAR, its low frequency radio telescope network, and devising a way to process the incoming data. ASTRON and the University of Groningen worked together to test how to process the LOFAR data on the grid system, a high-throughput computing system, and the high-performance computing system. Zheng was involved in the testing and enjoyed it so much that when she saw a vacancy at ASTRON years later she says she just couldn’t resist applying for it.

Throughout her career in high-performance computing, Zheng has always designed and developed software that helps scientists conduct their research. She was also briefly a researcher herself after completing her master’s in computer science but ultimately returned to software engineering. “I think it fits me better,” she says. “I like programming more than writing papers and publications. By working on ASTRON’s science analysis platform, I can focus on developing software and still contribute to scientific research.”

ASTRON’s goal is to make discoveries in radio astronomy happen by pushing technology to its limits, which wouldn’t be possible without people like Zheng with strong technical expertise. The Square Kilometre Array will be able to observe the sky much faster and in greater detail than any previous telescope. Using the data collected by the telescope, scientists will be able to challenge Einstein’s theory of relativity, discover what happened immediately after the Big Bang, and understand the nature of dark energy. And what they find could fundamentally change our understanding of the universe as we know it.