Tan, H. L., Gu, Y., Li, L., Leong, M. C., & Chen, N. F. (2025). Contextualized Visual Storytelling for Conversational Chatbot in Education. Companion Proceedings of the 27th International Conference on Multimodal Interaction, 185–189. https://doi.org/10.1145/3747327.3764895
Abstract:
Interactive visual storytelling through conversational agents offers a means to enhance early childhood language learning. We developed a picture-guided conversational chatbot, driven by dense image captioning, for early childhood mother tongue language learning. However, state-of-the-art image captioning systems fall short in meeting the educational needs of young learners. They often lack cultural contextualization, use vocabulary that exceeds children’s developmental level, and fail to align with curriculum-relevant learning goals. We investigated a contextualized dense image captioning framework, which augments dense image captioning with cultural and curriculum-aligned keyword retrieval through a Retrieval-Augmented Generation (RAG) module. This enables the generation of culturally appropriate, age-level suitable, and educationally anchored captions that enhance learner engagement and pedagogical relevance. We demonstrate that our approach outperforms existing captioning models in terms of linguistic appropriateness, and curriculum and cultural alignment. The contextualized dense image captioning framework supports the development of culturally grounded, education-oriented conversational agents for young learners.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the National Research Foundation, Singapore - AI Singapore Programme
Grant Reference no. : AISG2-GC-2022-005
This research / project is supported by the A*STAR - Japan-Singapore Joint Call: Japan Science and Technology Agency (JST) and A*STAR 2024
Grant Reference no. : R24I6IR136
This research / project is supported by the National Research Foundation, Singapore - Campus for Research Excellence and Technological Enterprise (CREATE) programme (DesCartes)
Grant Reference no. : NA