Nystromformer: A Nystom-based Algorithm for Approximating Self-Attention
Ontology highlight
ABSTRACT:
SUBMITTER: Xiong Y
PROVIDER: S-EPMC8570649 | biostudies-literature | 2021 Jan
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Automobile intelligence is the trend for modern automobiles, of which environment perception is the key technology of intelligent automobile research. For autonomous vehicles, the detection of object information, such as vehicles and pedestrians in traffic scenes is crucial to improving driving safety. However, in the actual traffic scene, there are many special conditions such as object occlusion, small objects, and bad weather, which will affect the accuracy of object detection. In this research, the SwinT-YOLOv4 algorithm is proposed for detecting objects in traffic scenes, which is based on the YOLOv4 algorithm. Compared with a Convolutional neural network (CNN), the vision transformer is more powerful at extracting vision features of objects in the image. The CNN-based backbone in YOLOv4 is replaced by the Swin Transformer in the proposed algorithm. The feature-fusing neck and predicting head of YOLOv4 is remained. The proposed model was trained and evaluated in the COCO dataset. Experiments show that our method can significantly improve the accuracy of object detection under special conditions. Equipped with our method, the object detection precision for cars and person is improved by 1.75%, and the detection precision for car and person reach 89.04% and 94.16%, respectively.
| S-EPMC10194927 | biostudies-literature
Project description:Differential equations are commonly used to model various types of real life applications. The complexity of these models may often hinder the ability to acquire an analytical solution. To overcome this drawback, numerical methods were introduced to approximate the solutions. Initially when developing a numerical algorithm, researchers focused on the key aspect which is accuracy of the method. As numerical methods becomes more and more robust, accuracy alone is not sufficient hence begins the pursuit of efficiency which warrants the need for reducing computational cost. The current research proposes a numerical algorithm for solving initial value higher order ordinary differential equations (ODEs). The proposed algorithm is derived as a three point block multistep method, developed in an Adams type formulae (3PBCS) and will be used to solve various types of ODEs and systems of ODEs. Type of ODEs that are selected varies from linear to nonlinear, artificial and real life problems. Results will illustrate the accuracy and efficiency of the proposed three point block method. Order, stability and convergence of the method are also presented in the study.
| S-EPMC7880453 | biostudies-literature
Project description:Although the era of big data has brought convenience to daily life, it has also caused many problems. In the field of scenic tourism, it is increasingly difficult for people to choose the scenic spot that meets their needs from mass information. To provide high-quality services to users, a recommended tourism model is introduced in this paper. On the one hand, the tourism system utilises the users' historical interactions with different scenic spots to infer their short- and long-term favorites. Among them, the users' short-term demands are modelled through self-attention mechanism, and the proportion of short- and long-term favorites is calculated using the Euclidean distance. On the other hand, the system models the relationship between multiple scenic spots to strengthen the item relationship and further form the most relevant tourist recommendations.
| S-EPMC9409543 | biostudies-literature
Project description:BackgroundFast and accurate automatic segmentation of polyps in colonoscopy plays a crucial role in the early diagnosis and treatment of colon cancer. However, the current polyp segmentation algorithms based on deep neural networks suffer from the problems of larger models and lower segmentation accuracy. Meanwhile, achieving accurate segmentation of polyps is to improve the diagnostic efficiency of doctors, and this need motivates us to develop a set of lightweight models so that it can be easily embedded in clinical devices to meet the requirements of practical applications. This study aims to provide effective technical support for the rapid and precise segmentation of polyps in clinical applications.MethodsDeepNeXt, an innovative polyp segmentation model grounded in multi-scale attention mechanisms. DeepNeXt incorporates a multi-stage, lightweight convolutional encoder module, leveraging several lightweight convolutional layers for efficient and accurate feature extraction. Furthermore, it features a novel multi-stage feature fusion structure designed to circumvent the potential loss of feature information during the encoding phase. Additionally, the model employs a multi-scale attentional feature encoding module that harnesses multi-branch deep strip convolution techniques to extract multi-dimensional information from the feature maps post-encoding, thereby enhancing the neural network's capability to extract diverse feature information.ResultsExperimental validation on the Kvasir Segmentation Dataset (Kvasir-SEG dataset) and Colorectal Cancer-Clinic Datasetbase (CVC-ClinicDB datasets) demonstrates that DeepNeXt outperforms mainstream networks such as U-net, U-net++, TransUnet, SwinUnet, and TGANet in terms of parameters and floating-point operations (FLOPs). DeepNeXt achieved a FLOPs metric of only 3.04 G and a parameters (Params) metric of just 1.51 M, while delivering exceptional performance in segmentation. On the Kvasir-SEG dataset, it reached a mean intersection over union (mIOU) of 83.91, and on the CVC-ClinicDB dataset, 87.37. Additionally, the Dice and Recall metrics also showed superior results, highlighting that DeepNeXt strikes an optimal balance between computational efficiency, model compactness, and segmentation accuracy.ConclusionsIn conclusion, we have proposed the DeepNeXt network, a novel lightweight multi-scale attention segmentation network tailored for computationally limited medical devices, which provides strong support for accurate and efficient polyp segmentation in clinical applications.
| S-EPMC11652027 | biostudies-literature
Project description:In the research of face recognition technology, the traditional methods usually show poor recognition accuracy and insufficient generalization ability when faced with complex scenes such as lighting changes, posture changes and skin color diversity. To solve these problems, based on the improvement of adaptive boosting to improve the accuracy of face detection, the study proposes a residual network 18-layer face feature extraction algorithm based on hybrid domain attention mechanism algorithm. The study introduces channel-domain and spatial-domain attention mechanism to enhance the extraction of face image features. The outcomes indicated that the recognition accuracy of the proposed method on multiple face image datasets, labeled field face datasets, and celebrity facial attribute datasets exceeded 98.34% and reached up to 99.64%, which was better than the current state-of-the-art methods. After combining channel and spatial attention mechanism, the false detection rate was as low as 2.50%, which was lower than the false detection rate of other methods. In addition to enhancing face recognition's robustness and accuracy, the work offers fresh concepts and resources for face recognition's potential uses in intricate scenarios in the future.
| S-EPMC11922290 | biostudies-literature
Project description:The purpose of infrared and visible image fusion is to obtain an image that includes both infrared target and visible information. However, among the existing infrared and visible image fusion methods, some of them give priority to the fusion effect, often with complex design, ignoring the influence of attention mechanisms on deep features, resulting in the lack of visible light texture information in the fusion image. To solve these problems, an infrared and visible image fusion method based on dense gradient attention residuals is proposed in this article. Firstly, squeeze-and-excitation networks are integrated into the gradient convolutional dense block, and a new gradient attention residual dense block is designed to enhance the ability of the network to extract important information. In order to retain more original image information, the feature gradient attention module is introduced to enhance the ability of detail information retention. In the fusion layer, an adaptive weighted energy attention network based on an energy fusion strategy is used to further preserve the infrared and visible details. Through the experimental comparison on the TNO dataset, our method has excellent performance on several evaluation indicators. Specifically, in the indexes of average gradient (AG), information entropy (EN), spatial frequency (SF), mutual information (MI) and standard deviation (SD), our method reached 6.90, 7.46, 17.30, 2.62 and 54.99, respectively, which increased by 37.31%, 6.55%, 32.01%, 8.16%, and 10.01% compared with the other five commonly used methods. These results demonstrate the effectiveness and superiority of our method.
| S-EPMC11622899 | biostudies-literature
Project description:In the dynamic field of deep reinforcement learning, the self-attention mechanism has been increasingly recognized. Nevertheless, its application in discrete problem domains has been relatively limited, presenting complex optimization challenges. This article introduces a pioneering deep reinforcement learning algorithm, termed Attention-based Actor-Critic with Priority Experience Replay (A2CPER). A2CPER combines the strengths of self-attention mechanisms with the Actor-Critic framework and prioritized experience replay to enhance policy formulation for discrete problems. The algorithm's architecture features dual networks within the Actor-Critic model-the Actor formulates action policies and the Critic evaluates state values to judge the quality of policies. The incorporation of target networks aids in stabilizing network optimization. Moreover, the addition of self-attention mechanisms bolsters the policy network's capability to focus on critical information, while priority experience replay promotes training stability and reduces correlation among training samples. Empirical experiments on discrete action problems validate A2CPER's adeptness at policy optimization, marking significant performance improvements across tasks. In summary, A2CPER highlights the viability of self-attention mechanisms in reinforcement learning, presenting a robust framework for discrete problem-solving and potential applicability in complex decision-making scenarios.
| S-EPMC11232580 | biostudies-literature
Project description:The Transformer has achieved great success in many computer vision tasks. With the in-depth exploration of it, researchers have found that Transformers can better obtain long-range features than convolutional neural networks (CNN). However, there will be a deterioration of local feature details when the Transformer extracts local features. Although CNN is adept at capturing the local feature details, it cannot easily obtain the global representation of features. In order to solve the above problems effectively, this paper proposes a hybrid model consisting of CNN and Transformer inspired by Visual Attention Net (VAN) and CoAtNet. This model optimizes its shortcomings in the difficulty of capturing the global representation of features by introducing Large Kernel Attention (LKA) in CNN while using the Transformer blocks with relative position self-attention variant to alleviate the problem of detail deterioration in local features of the Transformer. Our model effectively combines the advantages of the above two structures to obtain the details of local features more accurately and capture the relationship between features far apart more efficiently on a large receptive field. Our experiments show that in the image classification task without additional training data, the proposed model in this paper can achieve excellent results on the cifar10 dataset, the cifar100 dataset, and the birds400 dataset (a public dataset on the Kaggle platform) with fewer model parameters. Among them, SE_LKACAT achieved a Top-1 accuracy of 98.01% on the cifar10 dataset with only 7.5M parameters.
| S-EPMC10280586 | biostudies-literature
Project description:A current challenge in RNA structure prediction is the description of global helical arrangements compatible with a given secondary structure. Here we address this problem by developing a hierarchical graph sampling/data mining approach to reduce conformational space and accelerate global sampling of candidate topologies. Starting from a 2D structure, we construct an initial graph from size measures deduced from solved RNAs and junction topologies predicted by our data-mining algorithm RNAJAG trained on known RNAs. We sample these graphs in 3D space guided by knowledge-based statistical potentials derived from bending and torsion measures of internal loops as well as radii of gyration for known RNAs. Graph sampling results for 30 representative RNAs are analyzed and compared with reference graphs from both solved structures and predicted structures by available programs. This comparison indicates promise for our graph-based sampling approach for characterizing global helical arrangements in large RNAs: graph rmsds range from 2.52 to 28.24 Å for RNAs of size 25-158 nucleotides, and more than half of our graph predictions improve upon other programs. The efficiency in graph sampling, however, implies an additional step of translating candidate graphs into atomic models. Such models can be built with the same idea of graph partitioning and build-up procedures we used for RNA design.
| S-EPMC3964109 | biostudies-literature
Project description:A method is proposed for choosing unit cells for a group of crystals so that they all appear as nearly similar as possible to a selected cell. Related unit cells with varying cell parameters or indexed with different lattice centering can be accommodated.
| S-EPMC10483589 | biostudies-literature