Stanford University: New App VideoSticker Uses AI To Help Students Take Rich Notes from Video Lessons

Video has become an increasingly popular educational tool, and this trend has been greatly accelerated by the pandemic. As a storytelling medium, video is in many ways superior to text or lecture-based instruction, especially for learning linear processes and physical tasks, but it is not without drawbacks. It can be difficult to search or find already seen content for review. And note-taking is especially complicated and laborious.

“The bottom line is viewers find it difficult to extract and summarize important information from video,” says Hari Subramonyam, a research professor in the Stanford Graduate School of Education and a junior faculty fellow of the Stanford Institute for Human-Centered Artificial Intelligence.

Subramonyam and two colleagues, Yining Cao from the University of California, San Diego, and Eytan Adar at the University of Michigan, have harnessed computer vision and natural language processing tools to develop VideoSticker, a note-taking application specifically for video-based learning.

At present, note-taking of video means manually taking screenshots, clipping and cropping visual objects, reviewing transcripts and manually taking notes, all the while bouncing back and forth between video player and a note-taking app, Subramonyam says.

But VideoSticker combines video, audio, transcripts, and a powerful visual and a textual note-taking application. Better yet, using AI, VideoSticker can automatically identify and trim objects out of the video and place them in the note-taking area. VideoSticker then combs through the transcripts using tools like those behind popular voice recognition and chatbots to pull key text into the note-taking area, aligning it with the relevant imagery.

It all adds up to a big head start on manual note-taking and allows students to focus on the important content to improve comprehension and recall.

“Video is most often associated with entertainment, and that’s a very different experience from an educational context for knowledge gathering,” Subramonyam says. “More and more, we’re asking students in the TikTok era to consume hour-long videos and learn key concepts. Right now, they just don’t have the adequate software tools for that. VideoSticker fills that void.”

Theory to Practice
Research shows that students learn best when they are actively building mental models of a concept. Those models can take on differing forms—a concept map, a diagram, a timeline visualization, Subramonyam notes.

VideoSticker, in essence, turns the passive experience of watching video into an active one where the students learn by manipulating images and text and supplementing those elements with their own explanations and comments to cement comprehension and recall.

Yining Cao says, “It is essential that learners have flexible and interactive representations to support note-taking and comprehension.”

Students Say: “Cool”
To evaluate the effectiveness of the VideoSticker approach, the researchers conducted a preliminary user testing with a cohort of 10 graduate and undergraduate students. Each completed a 75- to 90-minute note-taking session that involved two passes at complex scientific content on cell mitosis—one initial viewing and a second for annotation. They then reported their experiences.

All participants noted VideoSticker’s flexibility in navigating between their notes and video content. “Detaching the objects and controlling them in the video viewing panel is cool. … So I can see how it transforms in the context of the video,” notes one participant. Users also said they appreciated the way VideoSticker extracted visual objects and aligned them with the text transcript.

“Overall, I think VideoSticker is a great example of how AI complements the learning context to balance manual note-taking with the experience of learning that is so important to real comprehension,” Subramonyam says. “As video becomes more commonplace in the classroom, as it is sure to in coming years, a tool like this will be very much needed.”

Going forward, the team plans to partner with educators to evaluate VideoStickers in real-world learning contexts and make the tool commercially available.