Automatic Labelling of Touristic Pictures Using CNNs and Metadata Information

Page view(s)
37
Checked on Nov 23, 2024
Automatic Labelling of Touristic Pictures Using CNNs and Metadata Information
Title:
Automatic Labelling of Touristic Pictures Using CNNs and Metadata Information
Journal Title:
2017 IEEE 2nd International Conference on Signal and Image Processing
DOI:
Publication URL:
Keywords:
Publication Date:
04 August 2017
Citation:
Abstract:
In this paper, we present a system that automatically recognizes pictures of landmarks in Singapore and combine the output with metadata information extracted from the picture if available in order to provide tourists with information about the recognized places. In detail, the system combines GPS information from the image metadata with a Convolutional Neural Network (CNN) based image recognition system to caption the image, where the GPS information is used to verify the caption given by the CNN, making the system more robust. For the training of the CNN, a total of ~67000 images were automatically downloaded from different search engines including Google, Flickr and Bing which were then filtered down to ~330 images per landmark to preserve the quality of the training dataset (e.g. removing repeated or confusing images), and then artificially augmented. After optimization, our CNN achieves a F1 score of 81% over a set of 6 different but very challenging locations, with each being a popular landmark in Singapore. With this combined system, the system can accurately tell the user where the image was taken and what the landmark in the image is. In addition, we created a database that provides background information for many landmarks (including, but not limited to the 6 landmarks used for training), such as their historical or cultural significance. In order to make the system accessible and intuitive, it was integrated into a website where a Google Map is generated showing the results of the recognition and metadata information.
License type:
PublisherCopyrights
Funding Info:
Description:
(c) 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.
ISSN:
978-1-5386-0968-2
ISBN:
978-1-5386-0969-9
978-1-5386-0967-5
Files uploaded:

File Size Format Action
papericsip17vs13.pdf 779.94 KB PDF Open