IIT Jodhpur Designs Software to Convert Digital Comics to Video
Digital Humanities (DH) at IIT Jodhpur focuses on and contributes to a composite of approaches (ideas and methods), rather than different approaches, which emphasize preservation, reconstruction, transmission and interpretation human record history.
C2VNet evolves panel by panel in a comic strip and eventually produces a full video (with audio) of a digitized or born digital storybook
The Indian Institute of Technology (IIT) in Jodhpur has created a framework to design software that converts digital comics into videos. The software is called Comic-to-Video Network (C2VNet). With digitization, there is an increased need to digitize the cultural heritage of our country. This framework revolves around the creation of an audio-video storybook. The software is developed by researchers Vaibhavi Gupta, Vinay Detani and Vivek Khokhar under the supervision of Chiranjoy Chattopadhyay, professor at IIT Jodhpur.
Digital Humanities (DH) at IIT Jodhpur focuses and contributes to a set of approaches (ideas and methods), rather than different approaches, which focus on the preservation, reconstruction, transmission and interpretation of human archives in a historical and contemporary way. This mainly concerns epistemological questions on the production of knowledge on the generation of digital data from material objects and the overhaul of existing processes of knowledge production. The methods and methodologies for creating the desired multimedia content have developed thanks to technological advancements. One such example is “automatic image synthesis”, which has received a lot of research attention. On the other hand, the synthesis of audio-video scenes, such as that based on document images, remains difficult and under-researched. This area of HR lacks a sustained analysis of multimodality in automatic content synthesis and its growing impact on digital scholarship in the humanities. C2VNet is a step towards bridging this gap.
Chiranjoy Chattopadhyay, Assistant Professor, Computer Science and Engineering Department, IIT Jodhpur, said, “The C2VNet panel pull model has two internal networks to support video creation. “CPENet developed by the team offers an accuracy of over 97%, and the SBSNet speech bubble segmentation model gives an accuracy of 98% with fewer parameters. Both have outperformed the top models. C2VNet is the first step towards the great future of automatic comic book multimedia creation to bring new comic book reading experiences.
C2VNet evolves panel by panel into a comic book and ultimately produces a full video (with audio) of a digitized or born digital storybook. The objective was to design and develop software which takes as input a comic strip of digital or digitized origin and produces an audiovisual animated film.
In addition to the software, researchers at IIT Jodhpur have proposed a data set titled “IMCDB: Indian Mythological Comic Dataset of Digitized Indian Comic Storybook” in English. This contains full annotations for the panels, bitmask of the text bubble and text files for each speech bubble and narration box in a panel and plans to make the dataset accessible to the public.
Going forward, the team is working on improving the software so that these multimedia books become more immersive and engaging for the target audience. Usually this kind of work takes more time and effort, but with this software it can be done quickly and in a more interactive way.
Last updated 05 Jan 2022