The Spectral Underpinning of word2vec.
Ontology highlight
ABSTRACT:
SUBMITTER: Jaffe A
PROVIDER: S-EPMC8425479 | biostudies-literature | 2020 Dec
REPOSITORIES: biostudies-literature
Publications
The Spectral Underpinning of word2vec.
Jaffe Ariel A Kluger Yuval Y Lindenbaum Ofir O Patsenker Jonathan J Peterfreund Erez E Steinerberger Stefan S
Frontiers in applied mathematics and statistics 20201203
Word2vec introduced by Mikolov et al. is a word embedding method that is widely used in natural language processing. Despite its success and frequent use, a strong theoretical justification is still lacking. The main contribution of our paper is to propose a rigorous analysis of the highly nonlinear functional of word2vec. Our results suggest that word2vec may be primarily driven by an underlying spectral method. This insight may open the door to obtaining provable guarantees for word2vec. We su ...[more]
Similar Datasets
Project description:The brain dynamics underlying working memory (WM) unroll via transient frequency-specific large-scale brain networks. This multidimensionality (time, space, and frequency) challenges traditional analyses. Through an unsupervised technique, the time delay embedded-hidden Markov model (TDE-HMM), we pursue a functional network analysis of magnetoencephalographic data from 38 healthy subjects acquired during an n-back task. Here we show that this model inferred task-specific networks with unique temporal (activation), spectral (phase-coupling connections), and spatial (power spectral density distribution) profiles. A theta frontoparietal network exerts attentional control and encodes the stimulus, an alpha temporo-occipital network rehearses the verbal information, and a broad-band frontoparietal network with a P300-like temporal profile leads the retrieval process and motor response. Therefore, this work provides a unified and integrated description of the multidimensional working memory dynamics that can be interpreted within the neuropsychological multi-component model of WM, improving the overall neurophysiological and neuropsychological comprehension of WM functioning.
| S-EPMC10593846 | biostudies-literature
Project description:BACKGROUND:Recent initiatives in psychiatry emphasize the utility of characterizing psychiatric symptoms in a multidimensional manner. However, strategies for applying standard self-report scales for multiaxial assessment have not been well-studied, particularly where the aim is to support both categorical and dimensional phenotypes. METHODS:We propose a method for applying natural language processing to derive dimensional measures of psychiatric symptoms from questionnaire data. We utilized nine self-report symptom measures drawn from a large cellular biobanking study that enrolled individuals with mood and psychotic disorders, as well as healthy controls. To summarize questionnaire results we used word embeddings, a technique to represent words as numeric vectors preserving semantic and syntactic meaning. A low-dimensional approximation to the embedding space was used to derive the proposed succinct summary of symptom profiles. To validate our embedding-based disease profiles, these were compared to presence or absence of axis I diagnoses derived from structured clinical interview, and to objective neurocognitive testing. RESULTS:Unsupervised and supervised classification to distinguish presence/absence of axis I disorders using survey-level embeddings remained discriminative, with area under the receiver operating characteristic curve up to 0.85, 95% confidence interval (CI) (0.74,0.91) using Gaussian mixture modeling, and cross-validated area under the receiver operating characteristic curve 0.91, 95% CI (0.88,0.94) using logistic regression. Derived symptom measures and estimated Research Domain Criteria scores also associated significantly with performance on neurocognitive tests. CONCLUSIONS:Our results support the potential utility of deriving dimensional phenotypic measures in psychiatric illness through the use of word embeddings, while illustrating the challenges in identifying truly orthogonal dimensions.
| S-EPMC7122719 | biostudies-literature
Project description:This paper presents an off-topic detection algorithm combining LDA and word2vec aiming at the problem of the lack of accurate and efficient off-topic detection algorithms in the English composition-assisted review system. The algorithm uses the LDA model to model the document and train the document through the word2vec, and uses the semantic relationship between the document's topics and words to calculate the probability weighted sum for each topic and its feature words in the document, and finally selects the off-topic composition by setting a reasonable threshold. Different F values are obtained by changing the number of topics in the document, and the best number of topics is determined. Experimental results show that the proposed method is more effective than vector space model, can detect more off-topic compositions, and the accuracy is higher, the F value is more than 88%, which realizes the intelligent processing of off-topic detection of composition, and can be effectively applied in English composition teaching.
| S-EPMC8880936 | biostudies-literature
Project description:The CDC reported that the overdose of prescription or illicit opioids was responsible for the deaths of over 80,000 Americans in 2021. Social media is a valuable source of insight into problematic patterns of substance misuse. The way people converse with illicit drugs in online forums is highly variable, and slang terms are frequently used. Manually identifying names of specific drugs can be difficult in both time and labor.Subjects and methodsThe study utilized the Gensim Python library and its Word2Vec neural network model to develop an auto-encoding neural network, enabling the innovative analysis of drug-related discourse downloaded from the Reddit website. The slang terms were then used to qualitatively analyze the topics and categories of drugs discussed on the forum.ResultsThe inclusion of slang terms facilitated the introduction of 200,000 specific mentions of opioid drugs and that stimulant drugs share a substantial semantic similarity with opioids, a 200 % increase in the number of drug-related terms as compared to using existing datasets.ConclusionsThis study advances the academic field with an extended collection of drug-related terms, offering a useful methodology and resource for tackling the opioid crisis with innovative, reduced-time detection and surveillance methods.
| S-EPMC11626628 | biostudies-literature
Project description:Drug discovery is an academical and commercial process of global importance. Accurate identification of drug-target interactions (DTIs) can significantly facilitate the drug discovery process. Compared to the costly, labor-intensive and time-consuming experimental methods, machine learning (ML) plays an ever-increasingly important role in effective, efficient and high-throughput identification of DTIs. However, upstream feature extraction methods require tremendous human resources and expert insights, which limits the application of ML approaches. Inspired by the unsupervised representation learning methods like Word2vec, we here proposed SPVec, a novel way to automatically represent raw data such as SMILES strings and protein sequences into continuous, information-rich and lower-dimensional vectors, so as to avoid the sparseness and bit collisions from the cumbersomely manually extracted features. Visualization of SPVec nicely illustrated that the similar compounds or proteins occupy similar vector space, which indicated that SPVec not only encodes compound substructures or protein sequences efficiently, but also implicitly reveals some important biophysical and biochemical patterns. Compared with manually-designed features like MACCS fingerprints and amino acid composition (AAC), SPVec showed better performance with several state-of-art machine learning classifiers such as Gradient Boosting Decision Tree, Random Forest and Deep Neural Network on BindingDB. The performance and robustness of SPVec were also confirmed on independent test sets obtained from DrugBank database. Also, based on the whole DrugBank dataset, we predicted the possibilities of all unlabeled DTIs, where two of the top five predicted novel DTIs were supported by external evidences. These results indicated that SPVec can provide an effective and efficient way to discover reliable DTIs, which would be beneficial for drug reprofiling.
| S-EPMC6967417 | biostudies-literature
Project description:Text classification, as an important research area of text mining, can quickly and effectively extract valuable information to address the challenges of organizing and managing large-scale text data in the era of big data. Currently, the related research on text classification tends to focus on the application in fields such as information filtering, information retrieval, public opinion monitoring, and library and information, with few studies applying text classification methods to the field of tourist attractions. In light of this, a corpus of tourist attraction description texts is constructed using web crawler technology in this paper. We propose a novel text representation method that combines Word2Vec word embeddings with TF-IDF-CRF-POS weighting, optimizing traditional TF-IDF by incorporating total relative term frequency, category discriminability, and part-of-speech information. Subsequently, the proposed algorithm respectively combines seven commonly used classifiers (DT, SVM, LR, NB, MLP, RF, and KNN), known for their good performance, to achieve multi-class text classification for six subcategories of national A-level tourist attractions. The effectiveness and superiority of this algorithm are validated by comparing the overall performance, specific category performance, and model stability against several commonly used text representation methods. The results demonstrate that the newly proposed algorithm achieves higher accuracy and F1-measure on this type of professional dataset, and even outperforms the high-performance BERT classification model currently favored by the industry. Acc, marco-F1, and mirco-F1 values are respectively 2.29%, 5.55%, and 2.90% higher. Moreover, the algorithm can identify rare categories in the imbalanced dataset and exhibit better stability across datasets of different sizes. Overall, the algorithm presented in this paper exhibits superior classification performance and robustness. In addition, the conclusions obtained by the predicted value and the true value are consistent, indicating that this algorithm is practical. The professional domain text dataset used in this paper poses higher challenges due to its complexity (uneven text length, relatively imbalanced categories), and a high degree of similarity between categories. However, this proposed algorithm can efficiently implement the classification of multiple subcategories of this type of text set, which is a beneficial exploration of the application research of complex Chinese text datasets in specific fields, and provides a useful reference for the vector expression and classification of text datasets with similar content.
| S-EPMC11488746 | biostudies-literature
Project description:Cavity analysis in molecular dynamics is important for understanding molecular function. However, analyzing the dynamic pattern of molecular cavities remains a difficult task. In this paper, we propose a novel method to topologically represent molecular cavities by vectorization. First, a characterization of cavities is established through Word2Vec model, based on an analogy between the cavities and natural language processing (NLP) terms. Then, we use some techniques such as dimension reduction and clustering to conduct an exploratory analysis of the vectorized molecular cavity. On a real data set, we demonstrate that our approach is applicable to maintain the topological characteristics of the cavity and can find the change patterns from a large number of cavities.
| S-EPMC6928730 | biostudies-literature
Project description:Viral infection involves a large number of protein-protein interactions (PPIs) between human and virus. The PPIs range from the initial binding of viral coat proteins to host membrane receptors to the hijacking of host transcription machinery. However, few interspecies PPIs have been identified, because experimental methods including mass spectrometry are time-consuming and expensive, and molecular dynamic simulation is limited only to the proteins whose 3D structures are solved. Sequence-based machine learning methods are expected to overcome these problems. We have first developed the LSTM model with word2vec to predict PPIs between human and virus, named LSTM-PHV, by using amino acid sequences alone. The LSTM-PHV effectively learnt the training data with a highly imbalanced ratio of positive to negative samples and achieved AUCs of 0.976 and 0.973 and accuracies of 0.984 and 0.985 on the training and independent datasets, respectively. In predicting PPIs between human and unknown or new virus, the LSTM-PHV learned greatly outperformed the existing state-of-the-art PPI predictors. Interestingly, learning of only sequence contexts as words is sufficient for PPI prediction. Use of uniform manifold approximation and projection demonstrated that the LSTM-PHV clearly distinguished the positive PPI samples from the negative ones. We presented the LSTM-PHV online web server and support data that are freely available at http://kurata35.bio.kyutech.ac.jp/LSTM-PHV.
| S-EPMC8574953 | biostudies-literature
Project description:Feelings of inferiority are complex emotions that usually indicate perceived weakness and helplessness. A lack of timely and effective interventions may bring serious consequences to individuals with inferiority feelings. Due to privacy concerns, those people often hesitate to seek face-to-face help, but they usually spontaneously share their feelings on social media, which makes social media a good resource for ample inferiority-related data. We randomly selected a sample of posts indicating inferiority feelings to explore the causes of inferiority. Through language analysis and natural language processing, we constructed a Word2Vec model of inferiority based on social media data and applied it to the cause analysis of inferiority feelings. The main causes of inferiority feelings are personal experience, social interaction, love relationship, etc. People feeling inferior about their personal experiences usually are largely influenced by their ways of thinking and life attitudes. Social and emotional factors overlap somewhat in the development of inferiority. In love relationships, males are more prone to inferiority feeling than females. These findings will help relevant institutions and organizations better understand people with inferiority feelings and facilitate the development of targeted treatment for those with potential self-esteem problems.
| S-EPMC8956725 | biostudies-literature
Project description:The assessment of examination questions is crucial in educational institutes since examination is one of the most common methods to evaluate students' achievement in specific course. Therefore, there is a crucial need to construct a balanced and high-quality exam, which satisfies different cognitive levels. Thus, many lecturers rely on Bloom's taxonomy cognitive domain, which is a popular framework developed for the purpose of assessing students' intellectual abilities and skills. Several works have been proposed to automatically handle the classification of questions in accordance with Bloom's taxonomy. Most of these works classify questions according to specific domain. As a result, there is a lack of technique of classifying questions that belong to the multi-domain areas. The aim of this paper is to present a classification model to classify exam questions based on Bloom's taxonomy that belong to several areas. This study proposes a method for classifying questions automatically, by extracting two features, TFPOS-IDF and word2vec. The purpose of the first feature was to calculate the term frequency-inverse document frequency based on part of speech, in order to assign a suitable weight for essential words in the question. The second feature, pre-trained word2vec, was used to boost the classification process. Then, the combination of these features was fed into three different classifiers; K-Nearest Neighbour, Logistic Regression, and Support Vector Machine, in order to classify the questions. The experiments used two datasets. The first dataset contained 141 questions, while the other dataset contained 600 questions. The classification result for the first dataset achieved an average of 71.1%, 82.3% and 83.7% weighted F1-measure respectively. The classification result for the second dataset achieved an average of 85.4%, 89.4% and 89.7% weighted F1-measure respectively. The finding from this study showed that the proposed method is significant in classifying questions from multiple domains based on Bloom's taxonomy.
| S-EPMC7081997 | biostudies-literature