Interactive System for Video Summarization Based on Multimodal Fusion

Biography videos based on life performances of prominent figures in history aim to describe great men's life. In this paper, a novel interactive video summarization for biography video based on multimodal fusion is proposed, which is a novel approach of visualizing the specific features for biography video and interacting with video content by taking advantage of the ability of multimodality. In general, a story of movie progresses by dialogues of characters and the subtitles are produced with the basis on the dialogues which contains all the information related to the movie. In this paper, JGibbsLDA is applied to extract key words from subtitles because the biography video consists of different aspects to depict the character's whole life. In terms of fusing keywords and key-frames, affinity propagation is adopted to calculate the similarity between each key-frame cluster and key-words. Through the method mentioned above, a video summarization is presented based on multimodal fusion which describes video content more completely. In order to reduce the time spent on searching the interest video content and get the relationship between main characters, a kind of map is adopted to visualize video content and interact with video summarization. An experiment is conducted to evaluate video summarization and the results demonstrate that this system can formally facilitate the exploration of video content while improving interaction and finding events of interest efficiently.