A Study on Differentiable Logic and LLMs for EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2023

Page view(s)
9
Checked on May 18, 2024
A Study on Differentiable Logic and LLMs for EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2023
Title:
A Study on Differentiable Logic and LLMs for EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2023
Journal Title:
CVPRW
DOI:
Publication URL:
Publication Date:
31 July 2023
Citation:
Cheng, Y., Xu, Z., Fang, F., Lin, D., Fan, H., Wong, Y., ... & Kankanhalli, M. (2023). A Study on Differentiable Logic and LLMs for EPIC-KITCHENS-100 Unsupervised Domain Adaptation Challenge for Action Recognition 2023. arXiv preprint arXiv:2307.06569.
Abstract:
In this technical report, we present our findings from a study conducted on the EPIC-KITCHENS-100 Unsupervised Domain Adaptation task for Action Recognition. Our research focuses on the innovative application of a differentiable logic loss in the training to leverage the co-occurrence relations between verb and noun, as well as the pre-trained Large Language Models (LLMs) to generate the logic rules for the adaptation to unseen action labels. Specifically, the model's predictions are treated as the truth assignment of a co-occurrence logic formula to compute the logic loss, which measures the consistency between the predictions and the logic constraints. By using the verb-noun co-occurrence matrix generated from the dataset, we observe a moderate improvement in model performance compared to our baseline framework. To further enhance the model's adaptability to novel action labels, we experiment with rules generated using GPT-3.5, which leads to a slight decrease in performance. These findings shed light on the potential and challenges of incorporating differentiable logic and LLMs for knowledge extraction in unsupervised domain adaptation for action recognition. Our final submission (entitled `NS-LLM') achieved the first place in terms of top-1 action recognition accuracy.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the Agency for Science, Technology and Re- search (A*STAR) - AME Programmatic Funding Scheme
Grant Reference no. : A18A2b0046
Description:
ISSN:
1063-6919
Files uploaded:

File Size Format Action
epic-kitchens-2023-report-camready.pdf 536.69 KB PDF Request a copy