Cornell University: Grant from Meta to spark Cornell research into AR, VR

When most people think of augmented reality (AR) or virtual reality (VR), they likely think of gaming headsets and Meta Quest 2, or a similar diversion from the real world.

But what if AR/VR could be used to improve the telehealth experience, help detect cracks in roads and bridges, or calibrate mobile-phone cameras to better recognize faces of color?

These and many other applications are exactly what Abe Davis, assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science, and Harald Haraldsson, director of the XR Collaboratory at Cornell Tech, will explore with a two-year, $1.8 million grant from Meta, formerly known as Facebook, and Spark AR, which powers AR experiences for Meta’s apps and devices.

Cornell Bowers CIS will receive $675,000 in funding; the XR Collaboratory will receive $1.125 million. The grant will support research, teaching and diverse undergraduate opportunities in the field of augmented reality at both Cornell Bowers CIS and Cornell Tech.

“As a new faculty, you’re still figuring out how to how to best enable the research that you want to do,” said Davis, who arrived at Cornell in 2020 after a one-year postdoc assignment at Cornell Tech.

“This gift helps tremendously with that,” Davis said. “And it creates a clear opportunity to pursue this line of research, which is something that I’m really excited about but requires equipment and more people to be involved.”

“This will unlock new activities for us at the New York City campus,” Haraldsson said, “and also allows us to continue the work in XR (augmented, virtual and mixed reality) that we’ve been doing for the last three years.”

“At Meta, we’re working to usher in a truly dynamic, useful augmented reality future that will change the way people interact with their surroundings, and each other,” said Sue Young, head of AR Glasses at Reality Labs at Meta. “Support for pioneering efforts like Cornell’s is key to helping realize this vision fully and empowering the next-generation of XR builders, creators and researchers.”

Part of Davis’ goal with the grant is to expand AR/VR offerings on the Ithaca campus, similar to what Haraldsson is doing at Cornell Tech.

Projects at XRC have included remote collaboration for health care applications; locomotion techniques for VR; surface reflectance capture using AR headsets; and novel input techniques for VR.

“There’s a lot of interest in AR and VR among students [in Ithaca],” Davis said. “And Harald and the folks at Cornell Tech have built a nice infrastructure at Cornell Tech, which we’d really like to replicate in Ithaca, so that we can get more students involved.”

Haraldsson hopes the grant will enable expansion of the XR Collaboratory’s work in three key areas – instructional, project and community activities.

“We have already connected a community of XR researchers, faculty and students up in Ithaca, here at Cornell Tech and also at Weill Cornell Medicine,” Haraldsson said. “And this gift will help us accelerate activities in each of our categories.”

Diversity and inclusion are a major point of emphasis, both investigators said. Part of that, they said, is making sure everyone is starting from the same general knowledge base, and giving a boost to those who need it.

“Sometimes in computer science, you assume that the students know how to use all the development tools, such as how to do complex debugging, but that is not always the case, even at the master’s level,” Haraldsson said. “And this grant will help us to provide an additional layer of instruction, to be more inclusive of students with different backgrounds in terms of computer science, to get everybody on the same page.”

One line of research that Davis hopes to explore has to do with the mobile phone cameras that produce the pictures that fuel social media. Devices are calibrated to recognize a limited range of colors, Davis said, and white, Anglo-Saxon faces have long served as the benchmark for camera design.

As a result, he said, most cameras fail to faithfully capture nonwhite faces. The ability for people to calibrate their devices in a more personal way is one research avenue Davis hopes to pursue.

“I believe part of this shared interest with Meta,” he said, “is being able to build systems that are more customizable to the needs of individual users.”