Dataset Information

ResMem-Net: memory based deep CNN for image memorability estimation.

ABSTRACT: Image memorability is a very hard problem in image processing due to its subjective nature. But due to the introduction of Deep Learning and the large availability of data and GPUs, great strides have been made in predicting the memorability of an image. In this paper, we propose a novel deep learning architecture called ResMem-Net that is a hybrid of LSTM and CNN that uses information from the hidden layers of the CNN to compute the memorability score of an image. The intermediate layers are important for predicting the output because they contain information about the intrinsic properties of the image. The proposed architecture automatically learns visual emotions and saliency, shown by the heatmaps generated using the GradRAM technique. We have also used the heatmaps and results to analyze and answer one of the most important questions in image memorability: "What makes an image memorable?". The model is trained and evaluated using the publicly available Large-scale Image Memorability dataset (LaMem) from MIT. The results show that the model achieves a rank correlation of 0.679 and a mean squared error of 0.011, which is better than the current state-of-the-art models and is close to human consistency (p = 0.68). The proposed architecture also has a significantly low number of parameters compared to the state-of-the-art architecture, making it memory efficient and suitable for production.

SUBMITTER: Praveen A

PROVIDER: S-EPMC8594589 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

ResMem-Net: memory based deep CNN for image memorability estimation.

Praveen Arockia A Noorwali Abdulfattah A Samiayya Duraimurugan D Zubair Khan Mohammad M Vincent P M Durai Raj DR Bashir Ali Kashif AK Alagupandi Vinoth V

PeerJ. Computer science 20211105

Image memorability is a very hard problem in image processing due to its subjective nature. But due to the introduction of Deep Learning and the large availability of data and GPUs, great strides have been made in predicting the memorability of an image. In this paper, we propose a novel deep learning architecture called ResMem-Net that is a hybrid of LSTM and CNN that uses information from the hidden layers of the CNN to compute the memorability score of an image. The intermediate layers are im ...[more]

PMID: 34825056

Similar Datasets

Project description:Images have been shown to consistently differ in terms of their memorability in healthy adults: some images stick in one's mind while others are forgotten quickly. Studies have suggested that memorability is an intrinsic, continuous property of a visual stimulus that can be both measured and manipulated. Memory literature suggests that important developmental changes occur throughout adolescence that have an impact on recognition memory, yet the effect that these changes have on image memorability has not yet been investigated. In the current study, we recruited adolescents ages 11-18 (n = 273, mean = 16) to an online visual memory experiment to explore the effects of developmental changes throughout adolescence on image memorability, and determine if memorability findings in adults can be generalized to the adolescent age group. We used the online experiment to calculate adolescent memorability scores for 1,000 natural images, and compared the results to the MemCat dataset-a memorability dataset that is annotated with adult memorability scores (ages 19-27). Our study finds that memorability scores in adolescents and adults are strongly and significantly correlated (Spearman's rank correlation, r = 0.76, p < 0.001). This correlation persists even when comparing adults with developmentally different sub-groups of adolescents (ages 11-14: r = 0.67, p < 0.001; ages 15-18: r = 0.60, p < 0.001). Moreover, the rankings of image categories by mean memorability scores were identical in both adolescents and adults (including the adolescent sub-groups), indicating that broadly, certain image categories are more memorable for both adolescents and adults. Interestingly, however, adolescents experienced significantly higher false alarm rates than adults, supporting studies that show increased impulsivity and reward-seeking behaviour in adolescents. Our results reveal that the memorability of images remains consistent across individuals at different stages of development. This consistency aligns with and strengthens prior research, indicating that memorability is an intrinsic property of images. Our findings open new pathways for applying memorability studies in adolescent populations, with profound implications in fields such as education, marketing, and psychology. Our work paves the way for innovative approaches in these domains, leveraging the consistent nature of image memorability across age groups.

Project description:Hybrid quantum systems have shown promise in image classification by combining the strengths of both classical and quantum algorithms. These systems leverage the parallel processing power of quantum computers to perform complex computations while utilizing classical algorithms to handle the vast amounts of data involved in imaging. The hybrid approach is intended to improve accuracy and speed compared to traditional classical methods. Further research and development in this area can revolutionize the way medical images are classified and help improve patient diagnosis and treatment. The use of Conventional Neural Networks (CNN) for the classification and diagnosis of medical images using big datasets requires, in most cases, the use of special high-performance computing machines, which are very expensive and hard to access by most researchers. A new form of Machine Learning (ML), Quantum machine learning (QML), is being introduced as an emerging strategy to overcome this problem. A hybrid quantum-classical CNN uses both quantum and classical convolution layers designed to use a parameterized quantum circuit. This means that the computing model utilizes a quantum circuits approach to construct QML algorithms, which are then used to transform the quantum state to extract image hidden features. This computational acceleration is expected to achieve better algorithm performance than classical CNNs. This study intends to evaluate the performance of a Hybrid Quantum CNN (HQCNN) against a conventional CNN. This is followed by some optimizer modifications for both proposed and classical CNN methods to investigate the possible further improvement of their performance. The optimizer modification is based on forcing the optimizer to be directly adaptive to model accuracy. The optimizer adaptiveness is based on the development of an optimizer with a loss base adaptive momentum. Several algorithms are developed to achieve the above-mentioned goals, including CNN, QCNN, CNN with the adaptive optimizer, and QCNN with the Adaptive optimizer. The four algorithms are tested against a Kaggle brain dataset containing over 7000 samples. The test results show the hybrid quantum circuit algorithm outperformed the conventional CNN algorithm. The performance of both algorithms was further improved by using a fully adaptive SGD optimizer.

Dataset Information

ResMem-Net: memory based deep CNN for image memorability estimation.

Publications

ResMem-Net: memory based deep CNN for image memorability estimation.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets