Automated Video Annotation

A challenge from MLBAM

From radar technology to geometric data, MLB Advanced Media is at the forefront of experimenting with new technologies to extract insights and information from game play.

NYC Media Lab structured a competition that challenged faculty and students nationwide to use computer vision, video processing and machine learning research to automatically annotate video. The intent of the contest was to expose MLBAM’s compelling technological questions to potential new talent.


Grand prize: A suite of technologies for automatic on-screen analysis of player actions, completed by a team of PhD students from NYU Polytechnic School of Engineering’s Video Lab.

First prize: Deep learning models for image recognition, used to attempt broader tasks of scene understanding, player recognition, and object tracking. This was completed by Rodrigo Nogueira, a PhD student in NYU’s Department of Computer Science.

Runner-Up: Tracking model to automatically analyze objects on the field, submitted by Shaunak Ahuja, an engineer and entrepreneur from University of Illinois Urbana-Champaign.

Runner-Up: Machine learning algorithm trained to extract and provide on-screen labels of action highlights, completed by a team of professors and students from the Department of Electrical Engineering at the University of North Texas.


Results of this competition were used to test a wide range of technologies, ultimately improving MLBAM’s ability to capture and annotate video metadata.