UC San Diego: Mentorship in the Sonic Arts Strikes a Chord at the Qualcomm Institute
For the past six years, Shahrokh Yadegari has led students with the UC San Diego Qualcomm Institute’s (QI) Sonic Arts Research & Development Group in developing novel audio technologies for academia and industry. Working with collaborators from the federal government and companies such as Disney and Google, students have gained an array of experiences that have fueled their passion for the field and propelled them into jobs in the tech industry.
“The interdisciplinary and hands-on context of the Sonic Arts Research and Development group allows students to learn and follow their passion in ways that are often hard to replicate within a single department,” said Yadegari, who is a QI associate director and faculty member in the UC San Diego Department of Music.
Coming Together
With Yadegari as their mentor, students with a passion for music and the technical sciences work in the Sonic Arts lab to design tools for applications in research, art and virtual reality (VR).
Three Sonic Arts alumni, Grady Kestler, Louis Pisha and John Burnett, joined the lab as Yadegari assumed the directorship in 2017. Kestler, who loved both music and problem-solving, had taken Yadegari’s sound design class and learned how to apply what he was learning as an Interdisciplinary Computing and the Arts major to new technology interfaces for artists.
Yadegari remained a mentor as Kestler went on to pursue a master’s in theater and sound design, advising him to take courses in electrical engineering and signal processing to supplement his studies. It was in an engineering class that Kestler met Pisha, a classical oboist and electrical engineering major. Upon discovering they shared similar interests, Kestler recommended Pisha meet Yadegari.
Meanwhile, Burnett had also been drawn in by one of Yadegari’s classes, which had “the craziest title [they’d] ever seen.” This and Yadegari’s interdisciplinary “Crossing Boundaries” course represented a unique blend of technology, art and philosophy that Burnett had never seen before. With Burnett showing interest in pursuing and experimenting with the electronics behind music, Yadegari extended an offer to join his growing team.
As new members of the Sonic Arts group, Kestler, Burnett and Pisha worked on a software program designed to create realistic acoustics in virtual worlds. Called “Space3D,” the program manipulates frequency, amplitude and sound delay through headphones and speakers to trick our brain into believing that sound is traveling from far across a room, through a closed door or from a moving object.
“There’s been a huge amount of effort put into developing virtual graphics, but audio is a lot harder,” said Pisha. “And we get a lot of information about our surroundings from sound.”
As Pisha learned graphics processing unit (GPU) programming to help design a new, dynamic algorithm for Space3D, Kestler focused on mapping a person’s head movement to object movement in VR. In October 2020, Yadegari, Burnett and Pisha, along with fellow Sonic Arts student Siddarth Atre, published the Space3D’s diffraction algorithm in the Journal of the Acoustical Society of America (JASA). Their paper made the front cover.
A Journey in VR
That same fall, the Sonic Arts group gained a new undergraduate member with interests in music, technology and game design. Like Burnett, Pisha and Kestler, Eito Murakami found a unique niche for his passions in Sonic Arts.
Murakami grew up training in piano. He moved to the San Francisco Bay Area from Japan when he was 14. In school, he befriended people who produced computer music and subsequently built his own DIY speakers for a school project. From then until his enrollment at UC San Diego as a political science and international relations student, he thought of music as a hobby.
It wasn’t until he began taking music as an elective and enrolled in a virtual class with Yadegari that Murakami decided to pursue his first interest as a major. Once he joined Sonic Arts, Murakami learned Unreal Engine, which creates photorealistic landscapes for gaming, VR and research, under Burnett.
The effort was building up to “Becoming,” a musical experience in VR directed by Yadegari and based on the work of the 13th-century Persian poet Mowlana Rumi. Arising from a partnership at the 2019 Opera Hack, a 48-hour music theater hackathon in San Diego, “Becoming” was meant to adapt Rumi’s work on the evolution of the human spirit into an all-encompassing experience engaging the senses of sight, hearing and touch with Space3D and haptic (touch-based) technology.
The Opera Hack team, consisting of Yadegari, Burnett and Pisha, with University of North Carolina at Chapel Hill Professor Emerita Francesca Talenti and McGill University Associate Professor of Electrical and Computer Engineering Jeremy Cooperstock, won $10,000 from the hackathon. As a new part of the “Becoming” team, Murakami worked on sound design while Burnett focused on the development of the play’s environment and fluid transitions between the shifts in the viewer’s surroundings.
The team members traveled to the SIGGRAPH 2022 Conference in Vancouver, Canada, in August 2022 to formally debut the complete piece. Features like haptic technology allowed visitors to interact with the piece, and have it respond in turn, making the human sense of touch seem cutting-edge. Burnett remarked that the team’s booth had attendees lining up to experience their work.
Taking it to the Next Step
Since graduating from Yadegari’s lab, Kestler, Pisha, Murakami, Burnett and other Sonic Arts alumni have leveraged the experience in their career paths.
Burnett spent the latter part of their time with Sonic Arts working on 3D videos and immersive music for performances. Simultaneously, they led the technical coordination of performances with QI’s Initiative for Digital Exploration of Arts and Sciences (IDEAS) series. Taken together, these experiences and the exposure to new and emerging technologies in music prepared them for a two-year contract as a software engineer with Meta Reality Labs Audio Research. In the future, they would like to continue creating immersive spaces at large scales.
“I’ve always been very inspired by the path [Yadegari] took, because I’ve always felt like a bit of an outlier in whatever program I’ve been in,” Burnett said. “[Yadegari] has shown me a way to [balance art and science] while being very encouraging of my work.”
Kestler, inspired by building the base technology and algorithms for Sonic Arts initiatives, is now pursuing a Ph.D. in physics with a speciality in quantum computing at UC San Diego. To him, the work is a welcome continuation of the tool-building he did for Sonic Arts, but for quantum simulators and technologies.
“[Yadegari] had the biggest effect on what I want to do,” said Kestler. “I can’t emphasize how crucial he has been in my career.”
Murakami, likewise, enrolled in a master’s program at the Center for Computer Research in Music and Acoustics at Stanford University. In the years since, he has kept his ties with QI, most recently through a collaboration on the IDEAS VR performance “Pigments of Imagination.”
Pisha, building on his experience learning and applying GPU programming to Space3D’s algorithm, soon secured two internships at artificial intelligence hardware and software provider NVIDIA. He now works as a senior GPU software engineer at the company Cruise Self-Driving Cars.
For Yadegari, seeing the lab’s alumni grow has been “absolutely fantastic and inspiring.”
“[Sonic Arts] creates space to innovate and solve problems with a more open mind,” he said. “We need innovative thinking, and the arts provide the context to practice and flex that muscle. QI offers the space for students to learn in that world while thinking and practicing outside of the box.”
The success of the Sonic Arts lab bodes well for a new master’s program Yadegari is developing at QI. If the program receives the green light, prospective students can expect to apply for a formal degree in Master of Science in Audio Technologies in fall 2025.