UC San Diego: Voyager Supercomputer Enters Testbed Phase

Voyager, the experimental compute resource newly installed at the San Diego Supercomputer Center (SDSC), is ready for use. Sanctioned for production by the National Science Foundation (NSF), the high-performance/high-efficiency supercomputer located at UC San Diego is moving into its operational testbed phase.

Envisioned as a system to facilitate exploration of new architectures in support of artificial intelligence (AI) in research and engineering, Voyager is a major departure from former NSF systems that have been focused on delivering computing resource in support of traditional applications and programming models. Instead, Voyager emphasizes deep engagement with the AI research community and features specialized hardware and software, close collaboration with applications teams and the opportunity to share these experiences within the community.

Voyager Principal Investigator Amit Majumdar explained that the NSF-supported Voyager project is structured in two phases: 1) a three-year testbed phase and 2) a two-year allocations phase.

“The testbed phase is centered around deep user engagement, whereby select research groups will provide information to help evaluate Voyager’s innovative deep learning (DL) hardware, software, libraries and machine learning (ML) application porting and performance,” said Majumdar.

The testbed phase will be guided by an External Advisory Board that will help recruit research groups. During the first years, the project will offer semiannual workshops and user forums to share lessons learned and to bring researchers together. These approaches will help to develop a knowledge base, best-use cases for future users and allocation policies. The allocations phase will follow via an NSF-approved process, which will be informed by the lessons learned from the testbed phase, regular and advanced user support, semiannual workshops and industry engagement for similar technology evaluation.

“The National Science Foundation is delighted to see the Voyager system move into its operational testbed phase,” said Manish Parashar, NSF director, Office of Advanced Cyberinfrastructure (OAC). “Artificial Intelligence research is playing an increasingly important role across all areas of science, engineering research and education. Voyager, with its specialized hardware and software capabilities and deep engagements, can be a tremendous resource for the community, providing new research opportunities and driving innovation.”

According to Majumdar, Voyager’s architecture features hardware and software innovations that will lead to performance gains and ease porting and model development in AI. Voyager comprises 42 Supermicro X12 Gaudi® AI Training Systems with 336 Habana Gaudi processors—designed for scaling large supercomputer training applications—and 16 Habana Goya processors to power AI inference models. Voyager’s networking is designed to support very large AI models. Each Gaudi processor has 10 integrated ports of RoCE (RDMA over converged Ethernet), with the 42 training systems connected by six 400Gpbs connections into a large Arista non-blocking switch.

“Porting of user applications has been relatively straightforward,” said Majumdar, adding that several applications are now running on Gaudi and Goya. “So far our experience is that codes need minimal changes.”

With support from Habana developers, in collaboration with SDSC and researchers, the porting of user applications is providing a basis for training materials, including a three-hour session that has been recorded for other users.

“One team has been running applications via Jupyter Notebook on Gaudi, and users can run with familiar TensorFlow and PyTorch frameworks,” said Majumdar, noting that the system was designed to support exploration in multiple dimensions (Gaudi and Goya processors, RoCE interconnect, 400 GbE switch, Kubernetes, Ceph, cnvrg.io, Slurm and more).

According to SDSC’s Deputy Director Shawn Strande, strong collaboration with technology partners at Supermicro and Habana allowed SDSC to bring this innovative architecture to the community.

“Considering the amount of innovation, the acquisition, deployment and installation have been remarkably smooth (even in the midst of Covid-19 conditions), with issues resolved jointly with Supermicro and Habana experts,” said Strande, who also serves as the project manager for Voyager.

SDSC experts reported that in most cases, measured performance has been better than projected, due in large part to software improvements by Habana and the collaboration.

“Supermicro is excited to continue supporting SDSC’s multi-year Voyager AI project as it enters the critical testbed phase of production operations,” said Ray Pang, vice president of technology enablement, Supermicro. “Supermicro’s ability to create complex AI solutions encompassing networking, compute, storage and AI, demonstrates how Supermicro’s AI and HPC solutions are ideal for science, medical and academic research.”