A fast fiducial marker tracking model for fully automatic alignment in electron tomography.
ABSTRACT: Automatic alignment, especially fiducial marker-based alignment, has become increasingly important due to the high demand of subtomogram averaging and the rapid development of large-field electron microscopy. Among the alignment steps, fiducial marker tracking is a crucial one that determines the quality of the final alignment. Yet, it is still a challenging problem to track the fiducial markers accurately and effectively in a fully automatic manner.In this paper, we propose a robust and efficient scheme for fiducial marker tracking. Firstly, we theoretically prove the upper bound of the transformation deviation of aligning the positions of fiducial markers on two micrographs by affine transformation. Secondly, we design an automatic algorithm based on the Gaussian mixture model to accelerate the procedure of fiducial marker tracking. Thirdly, we propose a divide-and-conquer strategy against lens distortions to ensure the reliability of our scheme. To our knowledge, this is the first attempt that theoretically relates the projection model with the tracking model. The real-world experimental results further support our theoretical bound and demonstrate the effectiveness of our algorithm. This work facilitates the fully automatic tracking for datasets with a massive number of fiducial markers.The C/C?++ source code that implements the fast fiducial marker tracking is available at https://github.com/icthrm/gmm-marker-tracking. Markerauto 1.6 version or later (also integrated in the AuTom platform at http://ear.ict.ac.cn/) offers a complete implementation for fast alignment, in which fast fiducial marker tracking is available by the '-t' email@example.com.Supplementary data are available at Bioinformatics online.
Project description:Motivation:Dual-axis electron tomography is an important 3?D macro-molecular structure reconstruction technology, which can reduce artifacts and suppress the effect of missing wedge. However, the fully automatic data process for dual-axis electron tomography still remains a challenge due to three difficulties: (i) how to track the mass of fiducial markers automatically; (ii) how to integrate the information from the two different tilt series; and (iii) how to cope with the inconsistency between the two different tilt series. Results:Here we develop a toolkit for fully automatic alignment of dual-axis electron tomography, with a simultaneous reconstruction procedure. The proposed toolkit and its workflow carries out the following solutions: (i) fully automatic detection and tracking of fiducial markers under large-field datasets; (ii) automatic combination of two different tilt series and global calibration of projection parameters; and (iii) inconsistency correction based on distortion correction parameters and the consequently simultaneous reconstruction. With all of these features, the presented toolkit can achieve accurate alignment and reconstruction simultaneously and conveniently under a single global coordinate system. Availability and implementation:The toolkit AuTom-dualx (alignment module dualxmauto and reconstruction module volrec_mltm) are accessible for general application at http://ear.ict.ac.cn, and the key source code is freely available under request. Supplementary information:Supplementary data are available at Bioinformatics online.
Project description:An image-based re-registration scheme has been developed and evaluated that uses fiducial registration as a starting point to maximize the normalized mutual information (nMI) between intraoperative ultrasound (iUS) and preoperative magnetic resonance images (pMR). We show that this scheme significantly (p<0.001) reduces tumor boundary misalignment between iUS pre-durotomy and pMR from an average of 2.5 mm to 1.0 mm in six resection surgeries. The corrected tumor alignment before dural opening provides a more accurate reference for assessing subsequent intraoperative tumor displacement, which is important for brain shift compensation as surgery progresses. In addition, we report the translational and rotational capture ranges necessary for successful convergence of the nMI registration technique (5.9 mm and 5.2 deg, respectively). The proposed scheme is automatic, sufficiently robust, and computationally efficient (<2 min), and holds promise for routine clinical use in the operating room during image-guided neurosurgical procedures.
Project description:<h4>Motivation</h4>Electron tomography (ET) has become an indispensable tool for structural biology studies. In ET, the tilt series alignment and the projection parameter calibration are the key steps towards high-resolution ultrastructure analysis. Usually, fiducial markers are embedded in the sample to aid the alignment. Despite the advances in developing algorithms to find correspondence of fiducial markers from different tilted micrographs, the error rate of the existing methods is still high such that manual correction has to be conducted. In addition, existing algorithms do not work well when the number of fiducial markers is high.<h4>Results</h4>In this paper, we try to completely solve the fiducial marker correspondence problem. We propose to divide the workflow of fiducial marker correspondence into two stages: (i) initial transformation determination, and (ii) local correspondence refinement. In the first stage, we model the transform estimation as a correspondence pair inquiry and verification problem. The local geometric constraints and invariant features are used to reduce the complexity of the problem. In the second stage, we encode the geometric distribution of the fiducial markers by a weighted Gaussian mixture model and introduce drift parameters to correct the effects of beam-induced motion and sample deformation. Comprehensive experiments on real-world datasets demonstrate the robustness, efficiency and effectiveness of the proposed algorithm. Especially, the proposed two-stage algorithm is able to produce an accurate tracking within an average of ≤ ms per image, even for micrographs with hundreds of fiducial markers, which makes the real-time ET data processing possible.<h4>Availability</h4>The code is available at https://github.com/icthrm/auto-tilt-pair . Additionally, the detailed original figures demonstrated in the experiments can be accessed at https://rb.gy/6adtk4.
Project description:Accurate lineage reconstruction of mammalian pre-implantation development is essential for inferring the earliest cell fate decisions. Lineage tracing using global fluorescence labeling techniques is complicated by increasing cell density and rapid embryo rotation, which hampers automatic alignment and accurate cell tracking of obtained four-dimensional imaging data sets. Here, we exploit the advantageous properties of primed convertible fluorescent proteins (pr-pcFPs) to simultaneously visualize the global green and the photoconverted red population in order to minimize tracking uncertainties over prolonged time windows. Confined primed conversion of H2B-pr-mEosFP-labeled nuclei combined with light-sheet imaging greatly facilitates segmentation, classification, and tracking of individual nuclei from the 4-cell stage up to the blastocyst. Using green and red labels as fiducial markers, we computationally correct for rotational and translational drift, reduce overall data size, and accomplish high-fidelity lineage tracing even for increased imaging time intervals - addressing major concerns in the field of volumetric embryo imaging.
Project description:MOTIVATION:Electron tomography (ET) is a widely used technology for 3D macro-molecular structure reconstruction. To obtain a satisfiable tomogram reconstruction, several key processes are involved, one of which is the calibration of projection parameters of the tilt series. Although fiducial marker-based alignment for tilt series has been well studied, marker-free alignment remains a challenge, which requires identifying and tracking the identical objects (landmarks) through different projections. However, the tracking of these landmarks is usually affected by the pixel density (intensity) change caused by the geometry difference in different views. The tracked landmarks will be used to determine the projection parameters. Meanwhile, different projection parameters will also affect the localization of landmarks. Currently, there is no alignment method that takes interrelationship between the projection parameters and the landmarks. RESULTS:Here, we propose a novel, joint method for marker-free alignment of tilt series in ET, by utilizing the information underlying the interrelationship between the projection model and the landmarks. The proposed method is the first joint solution that combines the extrinsic (track-based) alignment and the intrinsic (intensity-based) alignment, in which the localization of landmarks and projection parameters keep refining each other until convergence. This iterative approach makes our solution robust to different initial parameters and extreme geometric changes, which ensures a better reconstruction for marker-free ET. Comprehensive experimental results on three real datasets show that our new method achieved a significant improvement in alignment accuracy and reconstruction quality, compared to the state-of-the-art methods. AVAILABILITY AND IMPLEMENTATION:The main program is available at https://github.com/icthrm/joint-marker-free-alignment. SUPPLEMENTARY INFORMATION:Supplementary data are available at Bioinformatics online.
Project description:Real-time tracking of implanted fiducials in cine megavoltage (MV) imaging during volumetric modulated arc therapy (VMAT) delivery is complicated due to the inherent low contrast of MV images and potential blockage of dynamic leaves configurations. The purpose of this work is to develop a clinically practical autodetection algorithm for motion management during VMAT.The expected field-specific segments and the planned fiducial position from the Eclipse (Varian Medical Systems, Palo Alto, CA) treatment planning system were projected onto the MV images. The fiducials were enhanced by applying a Laplacian of Gaussian filter in the spatial domain for each image, with a blob-shaped object as the impulse response. The search of implanted fiducials was then performed on a region of interest centered on the projection of the fiducial when it was within an open field including the case when it was close to the field edge or partially occluded by the leaves. A universal template formula was proposed for template matching and normalized cross correlation was employed for its simplicity and computational efficiency. The search region for every image was adaptively updated through a prediction model that employed the 3D position of the fiducial estimated from the localized positions in previous images. This prediction model allowed the actual fiducial position to be tracked dynamically and was used to initialize the search region. The artifacts caused by electronic interference during the acquisition were effectively removed. A score map was computed by combining both morphological information and image intensity. The pixel location with the highest score was selected as the detected fiducial position. The sets of cine MV images taken during treatment were analyzed with in-house developed software written in MATLAB (The Mathworks, Inc., Natick, MA). Five prostate patients were analyzed to assess the algorithm performance by measuring their positioning accuracy during treatment.The algorithm was able to accurately localize the fiducial position on MV images with success rates of more than 90% per case. The percentage of images in which each fiducial was localized in the studied cases varied between 23% and 65%, with at least one fiducial having been localized between 40% and 95% of the images. This depended mainly on the modulation of the plan and fiducial blockage. The prostate movement in the presented cases varied between 0.8 and 3.5 mm (mean values). The maximum displacement detected among all patients was of 5.7 mm.An algorithm for automatic detection of fiducial markers in cine MV images has been developed and tested with five clinical cases. Despite the challenges posed by complex beam aperture shapes, fiducial localization close to the field edge, partial occlusion of fiducials, fast leaf and gantry movement, and inherently low MV image quality, good localization results were achieved in patient images. This work provides a technique for enabling real-time accurate fiducial detection and tumor tracking during VMAT treatments without the use of extra imaging dose.
Project description:This paper describes a non-invasive, automatic, and robust method for calibrating a scalable RGB-D sensor network based on retroreflective ArUco markers and the iterative closest point (ICP) scheme. We demonstrate the system by calibrating a sensor network comprised of six sensor nodes positioned in a relatively large industrial robot cell with an approximate size of 10 x 10 x 4 m . Here, the automatic calibration achieved an average Euclidean error of 3 c m at distances up to 9 . 45 m . To achieve robustness, we apply several innovative techniques: Firstly, we mitigate the ambiguity problem that occurs when detecting a marker at long range or low resolution by comparing the camera projection with depth data. Secondly, we use retroreflective fiducial markers in the RGB-D calibration for improved accuracy and detectability. Finally, the repeating ICP refinement uses an exact region of interest such that we employ the precise depth measurements of the retroreflective surfaces only. The complete calibration software and a recorded dataset are publically available and open source.
Project description:Augmented reality (AR) surgical navigation systems have attracted considerable attention as they assist medical professionals in visualizing the location of ailments within the human body that are not readily seen with the naked eye. Taking medical imaging with a parallel C-shaped arm (C-arm) as an example, surgical sites are typically targeted using an optical tracking device and a fiducial marker in real-time. These markers then guide operators who are using a multifunctional endoscope apparatus by signaling the direction or distance needed to reach the affected parts of the body. In this way, fiducial markers are used to accurately protect the vessels and nerves exposed during the surgical process. Although these systems have already shown potential for precision implantation, delamination of the fiducial marker, which is a critical component of the system, from human skin remains a challenge due to a mechanical mismatch between the marker and skin, causing registration problems that lead to poor position alignments and surgical degradation. To overcome this challenge, the mechanical modulus and stiffness of the marker patch should be lowered to approximately 150 kPa, which is comparable to that of the epidermis, while improving functionality. Herein, we present a skin-conformal, stretchable yet breathable fiducial marker for the application in AR-based surgical navigation systems. By adopting pore patterns, we were able to create a fiducial marker with a skin-like low modulus and breathability. When attached to the skin, the fiducial marker was easily identified using optical recognition equipment and showed skin-conformal adhesion when stretched and shrunk repeatedly. As such, we believe the marker would be a good fiducial marker candidate for patients under surgical navigation systems.
Project description:Cryo-electron tomography allows the visualization of macromolecular complexes in their cellular environments in close-to-live conditions. The nominal resolution of subtomograms can be significantly increased when individual subtomograms of the same kind are aligned and averaged. A vital step for such a procedure are algorithms that speedup subtomogram alignment and improve its accuracy to allow reference-free subtomogram classifications. Such methods will facilitate automation of tomography analysis and overall high throughput in the data processing. Building on previous work, here we propose a fast rotational alignment method that uses the Fourier equivalent form of a popular constrained correlation measure that considers missing wedge corrections and density variances in the subtomograms. The fast rotational search is based on 3D volumetric matching, which improves the rotational alignment accuracy in particular for highly distorted subtomograms with low SNR and tilt angle ranges in comparison to fast rotational matching of projected 2D spherical images. We further integrate our fast rotational alignment method in a reference-free iterative subtomogram classification scheme, and propose a local feature enhancement strategy in the classification process. As a proof of principle, we can demonstrate that the automatic method can successfully classify a large number of experimental subtomograms without the need of a reference structure.
Project description:A three-dimensional reconstruction of a nano-scale aqueous object can be achieved by taking a series of transmission electron micrographs tilted at different angles in vitreous ice: cryo-Transmission Electron Tomography. Presented here is a novel method of fine alignment for the tilt series. Size-selected gold clusters of ~2.7?nm (Au???±?? ), ~3.2?nm (Au???± ?? ), and ~4.3?nm (Au????±??) in diameter were deposited onto separate graphene oxide films overlaying holes on amorphous carbon grids. After plunge freezing and subsequent transfer to cryo-Transmission Electron Tomography, the resulting tomograms have excellent (de-)focus and alignment properties during automatic acquisition. Fine alignment is accurate when the evenly distributed 3.2?nm gold particles are used as fiducial markers, demonstrated with a reconstruction of a tobacco mosaic virus. Using a graphene oxide film means the fiducial markers are not interfering with the ice bound sample and that automated collection is consistent. The use of pre-deposited size-selected clusters means there is no aggregation and a user defined concentration. The size-selected clusters are mono-dispersed and can be produced in a wide size range including 2-5?nm in diameter. The use of size-selected clusters on a graphene oxide films represents a significant technical advance for 3D cryo-electron microscopy.