Dataset Information

Measuring Agreement Using Guessing Models and Knowledge Coefficients.

ABSTRACT: Several measures of agreement, such as the Perreault-Leigh coefficient, the [Formula: see text], and the recent coefficient of van Oest, are based on explicit models of how judges make their ratings. To handle such measures of agreement under a common umbrella, we propose a class of models called guessing models, which contains most models of how judges make their ratings. Every guessing model have an associated measure of agreement we call the knowledge coefficient. Under certain assumptions on the guessing models, the knowledge coefficient will be equal to the multi-rater Cohen's kappa, Fleiss' kappa, the Brennan-Prediger coefficient, or other less-established measures of agreement. We provide several sample estimators of the knowledge coefficient, valid under varying assumptions, and their asymptotic distributions. After a sensitivity analysis and a simulation study of confidence intervals, we find that the Brennan-Prediger coefficient typically outperforms the others, with much better coverage under unfavorable circumstances.

SUBMITTER: Moss J

PROVIDER: S-EPMC10444669 | biostudies-literature | 2023 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Measuring Agreement Using Guessing Models and Knowledge Coefficients.

Moss Jonas J

Psychometrika 20230608 3

Several measures of agreement, such as the Perreault-Leigh coefficient, the [Formula: see text], and the recent coefficient of van Oest, are based on explicit models of how judges make their ratings. To handle such measures of agreement under a common umbrella, we propose a class of models called guessing models, which contains most models of how judges make their ratings. Every guessing model have an associated measure of agreement we call the knowledge coefficient. Under certain assumptions on ...[more]

PMID: 37291419

Similar Datasets

Project description:BackgroundWith the availability of the Affymetrix exon arrays a number of tools have been developed to enable the analysis. These however can be expensive or have several pre-installation requirements. This led us to develop an analysis workflow for analysing differential splicing using freely available software packages that are already being widely used for gene expression analysis. The workflow uses the packages in the standard installation of R and Bioconductor (BiocLite) to identify differential splicing. We use the splice index method with the LIMMA framework. The main drawback with this approach is that it relies on accurate estimates of gene expression from the probe-level data. Methods such as RMA and PLIER may misestimate when a large proportion of exons are spliced. We therefore present the novel concept of a gene correlation coefficient calculated using only the probeset expression pattern within a gene. We show that genes with lower correlation coefficients are likely to be differentially spliced.ResultsThe LIMMA approach was used to identify several tissue-specific transcripts and splicing events that are supported by previous experimental studies. Filtering the data is necessary, particularly removing exons and genes that are not expressed in all samples and cross-hybridising probesets, in order to reduce the false positive rate. The LIMMA approach ranked genes containing single or few differentially spliced exons much higher than genes containing several differentially spliced exons. On the other hand we found the gene correlation coefficient approach better for identifying genes with a large number of differentially spliced exons.ConclusionWe show that LIMMA can be used to identify differential exon splicing from Affymetrix exon array data. Though further work would be necessary to develop the use of correlation coefficients into a complete analysis approach, the preliminary results demonstrate their usefulness for identifying differentially spliced genes. The two approaches work complementary as they can potentially identify different subsets of genes (single/few spliced exons vs. large transcript structure differences).

Project description:Small molecule distribution coefficients between immiscible nonaqueuous and aqueous phases-such as cyclohexane and water-measure the degree to which small molecules prefer one phase over another at a given pH. As distribution coefficients capture both thermodynamic effects (the free energy of transfer between phases) and chemical effects (protonation state and tautomer effects in aqueous solution), they provide an exacting test of the thermodynamic and chemical accuracy of physical models without the long correlation times inherent to the prediction of more complex properties of relevance to drug discovery, such as protein-ligand binding affinities. For the SAMPL5 challenge, we carried out a blind prediction exercise in which participants were tasked with the prediction of distribution coefficients to assess its potential as a new route for the evaluation and systematic improvement of predictive physical models. These measurements are typically performed for octanol-water, but we opted to utilize cyclohexane for the nonpolar phase. Cyclohexane was suggested to avoid issues with the high water content and persistent heterogeneous structure of water-saturated octanol phases, since it has greatly reduced water content and a homogeneous liquid structure. Using a modified shake-flask LC-MS/MS protocol, we collected cyclohexane/water distribution coefficients for a set of 53 druglike compounds at pH 7.4. These measurements were used as the basis for the SAMPL5 Distribution Coefficient Challenge, where 18 research groups predicted these measurements before the experimental values reported here were released. In this work, we describe the experimental protocol we utilized for measurement of cyclohexane-water distribution coefficients, report the measured data, propose a new bootstrap-based data analysis procedure to incorporate multiple sources of experimental error, and provide insights to help guide future iterations of this valuable exercise in predictive modeling.

Project description:Crystallization plays a crucial role as a separation and purification technique, particularly in the chemical and pharmaceutical industries. By adjusting process parameters, the productivity, product quality, and efficiency of downstream processes can be improved. In complex processes, model-based design becomes invaluable. Population Balance Models (PBMs) have successfully aided the chemical industry in achieving more effective production processes for decades. These models can utilize various input data sources to identify the dominant mechanisms and calibrate model parameters. While inline particle monitoring tools serve as excellent qualitative descriptors, challenges arise from experimentation and data interpretation, hindering their direct application in the kinetic parameter estimation of PBMs. In this study, we present a novel approach that utilizes information from inline particle monitoring tools for the kinetic parameter estimation of PBMs, bypassing the associated obstacles. Our pioneering approach relies on offline product size data and the correlation-based utilization of inline particle monitoring information. The paper compares this novel strategy with two parameter estimation techniques: the classical method using solute concentration and product size data and a somewhat naïve approach, which assumes that the inline particle monitoring data can directly be compared with the simulations. The prediction capabilities are evaluated through two in-silico case studies. The results indicate that the precision and predictive capability of the correlation-based technique are comparable to the classical approach, both for noisy data and for a system undergoing significant agglomeration and deagglomeration. The use of Pearson's correlation coefficient yields the best results in the novel cases. These findings in in-silico datasets provide a foundation and motivation for the practical application of this idea, unleashing the so-far hidden model development potential of such measurements.

Project description:Objective.Contrast recovery coefficient (CRC) is essential for image quality (IQ) assessment in positron emission tomography (PET), typically measured according to the National Electrical Manufacturers Association (NEMA) NU 2 standard. This study quantifies systematic uncertainties of the CRC measurement by a numerical investigation of the effects from scanner-independent parameters like voxel size, region-of-interest (ROI) misplacement, and sphere position on the underlying image grid.Approach.CRC measurements with 2D and 3D ROIs were performed on computer-generated images of a NEMA IQ-like phantom, using voxel sizes of 1-4 mm for sphere diameters of 5-40 mm-first in absence of noise and blurring, then with simulated spatial resolution and image noise with varying noise levels. The systematic uncertainties of the CRC measurement were quantified from above variations of scanner-independent parameters. Subsampled experimental images of a NEMA IQ phantom were additionally used to investigate the impact of ROI misplacement at different noise levels.Main results.In absence of noise and blurring, systematic uncertainties were up to 28.8% and 31.0% with 2D and 3D ROIs, respectively, for the 10 mm sphere, with the highest impact from ROI misplacement. In all cases, smaller spheres showed higher uncertainties with larger voxels. Contrary to prior assumptions, the use of 3D ROIs did not exhibit less susceptibility for parameter changes. Experimental and computer-generated images both demonstrated considerable variations on individual CRC measurements when background coefficient-of-variation exceeded 20%, despite negligible effects on mean CRC.Significance.This study underscores the effect of scanner-independent parameters on reliability, reproducibility, and comparability of CRC measurements. Our findings highlight the trade-off between the benefits of smaller voxel sizes and noise-induced CRC fluctuations, which is not considered in the current version of the NEMA IQ standards. The results furthermore warrant adjustments to the standard to accommodate the advances in sensitivity and spatial resolution of current-generation PET scanners.

Dataset Information

Measuring Agreement Using Guessing Models and Knowledge Coefficients.

Publications

Measuring Agreement Using Guessing Models and Knowledge Coefficients.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets