Dataset Information

Improving Model Transferability for Clinical Note Section Classification Models Using Continued Pretraining.

ABSTRACT:

Objective

The classification of clinical note sections is a critical step before doing more fine-grained natural language processing tasks such as social determinants of health extraction and temporal information extraction. Often, clinical note section classification models that achieve high accuracy for one institution experience a large drop of accuracy when transferred to another institution. The objective of this study is to develop methods that classify clinical note sections under the SOAP ("Subjective", "Object", "Assessment" and "Plan") framework with improved transferability.

Materials and methods

We trained the baseline models by fine-tuning BERT-based models, and enhanced their transferability with continued pretraining, including domain adaptive pretraining (DAPT) and task adaptive pretraining (TAPT). We added out-of-domain annotated samples during fine-tuning and observed model performance over a varying number of annotated sample size. Finally, we quantified the impact of continued pretraining in equivalence of the number of in-domain annotated samples added.

Results

We found continued pretraining improved models only when combined with in-domain annotated samples, improving the F1 score from 0.756 to 0.808, averaged across three datasets. This improvement was equivalent to adding 50.2 in-domain annotated samples.

Discussion

Although considered a straightforward task when performing in-domain, section classification is still a considerably difficult task when performing cross-domain, even using highly sophisticated neural network-based methods.

Conclusion

Continued pretraining improved model transferability for cross-domain clinical note section classification in the presence of a small amount of in-domain labeled samples.

SUBMITTER: Zhou W

PROVIDER: S-EPMC10168403 | biostudies-literature | 2023 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Improving Model Transferability for Clinical Note Section Classification Models Using Continued Pretraining.

Zhou Weipeng W Yetisgen Meliha M Afshar Majid M Gao Yanjun Y Savova Guergana G Miller Timothy A TA

medRxiv : the preprint server for health sciences 20230424

<h4>Objective</h4>The classification of clinical note sections is a critical step before doing more fine-grained natural language processing tasks such as social determinants of health extraction and temporal information extraction. Often, clinical note section classification models that achieve high accuracy for one institution experience a large drop of accuracy when transferred to another institution. The objective of this study is to develop methods that classify clinical note sections under ...[more]

PMID: 37162963

Similar Datasets

Project description:PurposeTo evaluate and compare the theoretically achievable accuracy of two families of two-parameter photon cross-section models: basis vector model (BVM) and modified parametric fit model (mPFM).MethodThe modified PFM assumes that photoelectric absorption and scattering cross-sections can be accurately represented by power functions in effective atomic number and/or energy plus the Klein-Nishina cross-section, along with empirical corrections that enforce exact prediction of elemental cross-sections. Two mPFM variants were investigated: the widely used Torikoshi model (tPFM) and a more complex "VCU" variant (vPFM). For 43 standard soft and bony tissues and phantom materials, all consisting of elements with atomic number less than 20 (except iodine), we evaluated the theoretically achievable accuracy of tPFM and vPFM for predicting linear attenuation, photoelectric absorption, and energy-absorption coefficients, and we compared it to a previously investigated separable, linear two-parameter model, BVM.ResultsFor an idealized dual-energy computed tomography (DECT) imaging scenario, the cross-section mapping process demonstrates that BVM more accurately predicts photon cross-sections of biological mixtures than either tPFM or vPFM. Maximum linear attenuation coefficient prediction errors were 15% and 5% for tPFM and BVM, respectively. The root-mean-square (RMS) prediction errors of total linear attenuation over the 20 keV to 1000 keV energy range of tPFM and BVM were 0.93% (tPFM) and 0.1% (BVM) for adipose tissue, 0.8% (tPFM) and 0.2% (BVM) for muscle tissue, and 1.6% (tPFM) and 0.2% (BVM) for cortical bone tissue. With exception of the thyroid and Teflon, the RMS error for photoelectric absorption and scattering coefficient was within 4% for the tPFM and 2% for the BVM. Neither model predicts the photon cross-sections of thyroid tissue accurately, exhibiting relative errors as large as 20%. For the energy-absorption coefficients prediction error, RMS errors for the BVM were less than 1.5%, while for the tPFM, the RMS errors were as large as 16%.ConclusionCompared to modified PFMs, BVM shows superior potential to support dual-energy CT cross-section mapping. In addition, the linear, separable BVM can be more efficiently deployed by iterative model-based DECT image-reconstruction algorithms.

Project description:An important problem with many current visio-linguistic models is that they often depend on spurious correlations. A typical example of a spurious correlation between two variables is one that is due to a third variable causing both (a "confounder"). Recent work has addressed this by adjusting for spurious correlations using a technique of deconfounding with automatically found confounders. We will refer to this technique as AutoDeconfounding. This article dives more deeply into AutoDeconfounding, and surfaces a number of issues of the original technique. First, we evaluate whether its implementation is actually equivalent to deconfounding. We provide an explicit explanation of the relation between AutoDeconfounding and the underlying causal model on which it implicitly operates, and show that additional assumptions are needed before the implementation of AutoDeconfounding can be equated to correct deconfounding. Inspired by this result, we perform ablation studies to verify to what extent the improvement on downstream visio-linguistic tasks reported by the works that implement AutoDeconfounding is due to AutoDeconfounding, and to what extent it is specifically due to the deconfounding aspect of AutoDeconfounding. We evaluate AutoDeconfounding in a way that isolates its effect, and no longer see the same improvement. We also show that tweaking AutoDeconfounding to be less related to deconfounding does not negatively affect performance on downstream visio-linguistic tasks. Furthermore, we create a human-labeled ground truth causality dataset for objects in a scene to empirically verify whether and how well confounders are found. We show that some models do indeed find more confounders than a random baseline, but also that finding more confounders is not correlated with performing better on downstream visio-linguistic tasks. Finally, we summarize the current limitations of AutoDeconfounding to solve the issue of spurious correlations and provide directions for the design of novel AutoDeconfounding methods that are aimed at overcoming these limitations.

Dataset Information

Improving Model Transferability for Clinical Note Section Classification Models Using Continued Pretraining.

Objective

Materials and methods

Results

Discussion

Conclusion

Publications

Improving Model Transferability for Clinical Note Section Classification Models Using Continued Pretraining.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets