Hearst: From Footage to Knowledge, News Story Understanding From Raw Video with AI

Media companies are experimenting with software and systems that reduce the cost and improve the workflow of high quality content generation. From news to entertainment, the promise of automation and providing journalists and content producers with automatically generated building blocks to synthesize higher value content while driving efficiencies, is an area ripe for experiment. Open-source tools, ranging from natural language processing to deep learning, computer vision, and automatic video annotation, make the generation, optimization and recombination of content more compelling than ever.

Hearst is developing partnerships with universities and startups that will expose its data science team to new ideas emerging in data science disciplines, including machine learning and artificial intelligence. NYC Media Lab worked with Hearst and a team of researchers from Columbia University to develop an AI tool to process raw video footage of news stories. The resulting data and structured database contain comprehensive information around a story, including events, entities, and their relationships. Potential applications and use cases include video editing, search, summarization and more.

The team was led by Shih-Fu Chang, Senior Executive Vice Dean and the Richard Dicker Professor of The Fu Foundation School of Engineering and Applied Science at Columbia University. Chang also directs the Digital Video and Multimedia Lab, with research focused on multimedia information retrieval, computer vision, and machine learning. Alireza Zareian, PhD candidate and graduate research assistant participated in the program alongside Chang.


Shih-Fu Chang Presents research findings at NYC Media Lab’s 2019 Machines + Media Conference