The Fifth Elephant 2024 Annual Conference (12th &13th July)

Multimodal AI is revolutionizing video analysis, but practical insights on pipeline design are scarce. Traditional computer vision pipelines often involve a complex web of specialized models. This leads to high costs, maintenance burdens, and difficulty in adapting to new tasks. This talk will dissect a real-world case study where multimodal models dramatically simplified a large-scale video analysis pipeline, leading to significantly reduced costs and improved agility.

Outline

I want to illustrate in this talk the breakthroughs we were able to unlock with the help of Multimodal models. I will demonstrate how we use simple, often older models like CLIP to create a simplified pipeline that replaces complex, interdependent vision pipelines. We also show the metrics that we use to decide when to use more expensive models like GPT-4-Vision and LLaVA to ensure cost-efficient processing of each video file. Attendees will gain real-world knowledge based on our experiences refactoring a large-scale system, avoiding potential pitfalls.

Impact

I work for a large media organization where we need to perform multiple simple tasks with our content library. We operate a generic vision pipeline to which several specific components can be added/ subtracted to perform various downstream tasks - as diverse as trailer generation, thumbnail selection to content moderation. The use of Multimodal models significantly simplified our pipelines and also made them easy to extend to to additional use cases.