Evaluating computerised health information systems: hard lessons still to be learnt.
Ontology highlight
ABSTRACT:
SUBMITTER: Littlejohns P
PROVIDER: S-EPMC153476 | biostudies-literature | 2003 Apr
REPOSITORIES: biostudies-literature
Publications
Evaluating computerised health information systems: hard lessons still to be learnt.
Littlejohns Peter P Wyatt Jeremy C JC Garvican Linda L
BMJ (Clinical research ed.) 20030401 7394
Similar Datasets
Project description:BackgroundThere are several frameworks that attempt to address the challenges of evaluation of health information systems by offering models, methods, and guidelines about what to evaluate, how to evaluate, and how to report the evaluation results. Model-based evaluation frameworks usually suggest universally applicable evaluation aspects but do not consider case-specific aspects. On the other hand, evaluation frameworks that are case specific, by eliciting user requirements, limit their output to the evaluation aspects suggested by the users in the early phases of system development. In addition, these case-specific approaches extract different sets of evaluation aspects from each case, making it challenging to collectively compare, unify, or aggregate the evaluation of a set of heterogeneous health information systems.ObjectivesThe aim of this paper is to find a method capable of suggesting evaluation aspects for a set of one or more health information systems-whether similar or heterogeneous-by organizing, unifying, and aggregating the quality attributes extracted from those systems and from an external evaluation framework.MethodsOn the basis of the available literature in semantic networks and ontologies, a method (called Unified eValuation using Ontology; UVON) was developed that can organize, unify, and aggregate the quality attributes of several health information systems into a tree-style ontology structure. The method was extended to integrate its generated ontology with the evaluation aspects suggested by model-based evaluation frameworks. An approach was developed to extract evaluation aspects from the ontology that also considers evaluation case practicalities such as the maximum number of evaluation aspects to be measured or their required degree of specificity. The method was applied and tested in Future Internet Social and Technological Alignment Research (FI-STAR), a project of 7 cloud-based eHealth applications that were developed and deployed across European Union countries.ResultsThe relevance of the evaluation aspects created by the UVON method for the FI-STAR project was validated by the corresponding stakeholders of each case. These evaluation aspects were extracted from a UVON-generated ontology structure that reflects both the internally declared required quality attributes in the 7 eHealth applications of the FI-STAR project and the evaluation aspects recommended by the Model for ASsessment of Telemedicine applications (MAST) evaluation framework. The extracted evaluation aspects were used to create questionnaires (for the corresponding patients and health professionals) to evaluate each individual case and the whole of the FI-STAR project.ConclusionsThe UVON method can provide a relevant set of evaluation aspects for a heterogeneous set of health information systems by organizing, unifying, and aggregating the quality attributes through ontological structures. Those quality attributes can be either suggested by evaluation models or elicited from the stakeholders of those systems in the form of system requirements. The method continues to be systematic, context sensitive, and relevant across a heterogeneous set of health information systems.
| S-EPMC4929349 | biostudies-literature
Project description:ObjectiveTo identify recurrent themes, insights, and process recommendations from stakeholders in US organizations during the health information technology (HIT) modernization of an existing electronic health record (EHR) to a commercial-off-the-shelf product in both resource-plentiful settings and in a resource-constrained environment, the US Indian Health Service.Materials and methodsThirteen qualitative interviews with stakeholders in various organizations were conducted about HIT modernization efforts. Using a Theory of Change framework, recurring themes were identified and analyzed.ResultsThe interviewees emphasized the importance of organizational and process revision during modernization, converting historical data, and clinical and leadership involvement. HIT implementation required technological and infrastructure redesign, additional training, and workflow reconfiguration. Motivations for modernization included EHR usability dissatisfaction, revenue enhancements, and improved clinical operations. Decision-making strategies, primarily during HIT selection, included meetings with stakeholders. Successful modernization resulted in improvements in clinical operations, patient experience, and financial outlay.DiscussionExisting implementation frameworks fail to provide experiential feedback, such as implementation challenges, like data conversion, regulatory, functionality, and interoperability requirements. Regardless of the healthcare environment, HIT modernization requires the engagement of leadership and end-users during HIT selection and through all stages of the implementation to prepare people, processes, and technology. Organizations must iteratively define the technological, infrastructure, organizational, and workflow changes required for a successful HIT modernization effort.ConclusionsHIT modernization is an opportunity for organizational and technological change. Successful modernization requires a comprehensive, intentional, well-communicated, and multidisciplinary approach. Resource-constrained environments have the additional challenges of financial burdens, limited staffing, and unstable infrastructure.
| S-EPMC7660948 | biostudies-literature
Project description:Bike-sharing programs, with initiatives to increase bike use and improve accessibility of urban transit, have received increasing attention in growing number of cities across the world. The latest generation of bike-sharing systems has employed smart card technology that produces station-based data or trip-level data. This facilitates the studies of the practical use of these systems. However, few studies have paid attention to the changes in users and system usage over the years, as well as the impact of system expansion on its usage. Monitoring the changes of system usage over years enables the identification of system performance and can serve as an input for improving the location-allocation of stations. The objective of this study is to explore the impact of the expansion of a bicycle-sharing system on the usage of the system. This was conducted for a bicycle-sharing system in Zhongshan (China), using operational usage data of different years following system expansion. To this end, we performed statistical and spatial analyses to examine the changes in both users and system usage between before and after the system expansion. The findings show that there is a big variation in users and aggregate usage following the system expansion. However, the trend in spatial distribution of demand shows no substantial difference over the years, i.e. the same high-demand and low-demand areas appear. There are decreases in demand for some old stations over the years, which can be attributed to either the negative performance of the system or the competition of nearby new stations. Expanding the system not only extends the original users' ability to reach new areas but also attracts new users to use bike-sharing systems. In the conclusions, we present and discuss the findings, and offer recommendations for the further expansion of system.
| S-EPMC5158202 | biostudies-literature
Project description:BackgroundSimulation based medical education (SBME) allows learners to acquire clinical skills without exposing patients to unnecessary risk. This is especially applicable to Emergency Medicine training programs where residents are expected to demonstrate proficiency in the management of time critical, low frequency, and highly-morbidity conditions. This study aims to describe the process through which a SBME curriculum was created, in a limited simulation resource setting at a 4-year Emergency Medicine (EM) residency program at the American University of Beirut Medical Center.MethodsA case-based pilot simulation curriculum was developed following Kern's 6 step approach to curriculum design. The curricular objectives were identified through an anonymous survey of the program's residents and faculty. Curriculum outcomes were assessed, and the curriculum was revised to address curricular barriers. Evaluations of the revised curriculum were collected during the simulation sessions and through a whole revised curriculum evaluation at the end of the first year of its implementation.Results14/20 residents (70%) and 8/8 faculty (100%) completed the needs assessment from which objectives for the pilot curriculum were developed and implemented through 6 2-h sessions over a 1-year period. Objectives were not met and identified barriers included cost, scheduling, resources, and limited faculty time. The revised curriculum addressed these barriers and 24 40-min sessions were successfully conducted during the following year. The sessions took place 3 at a time, in 2-h slots, using the same scenario to meet the objectives of the different learners' levels. 91/91 evaluations were collected from participants with overall positive results. The main differences between the pilot and the revised curricula included: a better understanding of the simulation center resources and faculty's capabilities.ConclusionSimulation-based education is feasible even with limited-resources. However, understanding the resources available, and advocating for protected educator time are essential to implementing a successful EM simulation curriculum.
| S-EPMC7792073 | biostudies-literature
Project description:Introduction: Firefighters were working in exceptionally difficult conditions during the Fort McMurray/Horse River fire in May 2016. Methods: From mid-May, we recruited firefighters from 13 fire services as they returned from the fire. In October 2016 we extended recruitment to all Alberta-based firefighters deployed to the fire. In December 2017-May 2018 we sent a first online follow-up: this concentrated on mental health supports. The second follow-up, in October 2018-January 2019, included screening scales for respiratory ill-health and PTSD. All three contacts included visual analogue scales for chest symptoms and the Hospital Anxiety and Depression Scale. We estimated exposure to PM2.5, and calculated an exposure mitigation index from reports of respiratory protective equipment (RPE). Results: We recruited 1,234 firefighters and examined the relation of respiratory symptoms to estimated particulate exposure. The relation was strong immediately post fire but weakened over time. We found less chest tightness and cough in those whose RPE in the first week mitigated exposure by at least 10%. We examined the relation between particulate exposure and mental ill-health from screening questionnaires and found those with high exposure (reflecting the ferocity of the fire) had poorer mental health scores. Firefighters reporting their "worst moment during the fire" was life threatening were more anxious at second follow-up. Overall both anxiety and depression scores increased at successive contacts, more so in those with mental ill-health recorded in physician billing records before the fire. Discussion: The results from this study overall suggest on-going fire-related health effects in a substantial minority of firefighters, similar to those reported in the longitudinal follow-up of firefighters after the collapse of the World Trade Centre. Self-reports of both respiratory symptoms and mental ill-health were strongly related, soon after the fire, to estimated particulate exposures. Anxiety increased over time since the fire in those who felt their life or safety had been threatened, underlining the need for ongoing support. Our conclusions about the benefits of rapid research relate particularly to the collection of biomarkers of exposure as quickly and widely as possible, and the establishment of a nominal list of participants before they are too widely dispersed.
| S-EPMC8632044 | biostudies-literature
Project description:BackgroundPrenatal exome sequencing (ES) for monogenic disorders in fetuses with structural anomalies increases diagnostic yield. In England there is a national trio ES service delivered from two laboratories. To minimise incidental findings and reduce the number of variants investigated, analysis uses a panel of 1205 genes where pathogenic variants may cause abnormalities presenting prenatally. Here we review our laboratory's early experience developing and delivering ES to identify challenges in interpretation and reporting and inform service development.MethodsA retrospective laboratory records review from 01.04.2020 to 31.05.2021.ResultsTwenty-four of 116 completed cases were identified as challenging including 13 resulting in difficulties in analysis and reporting, nine where trio inheritance filtering would have missed the diagnosis, and two with no prenatal diagnosis; one due to inadequate pipeline sensitivity, the other because the gene was not on the panel. Two cases with copy number variants identified were not detectable by microarray.ConclusionsVariant interpretation requires close communication between referring clinicians, with occasional additional examination of the fetus or parents and communication of evolving phenotypes. Inheritance filtering misses ∼5% of diagnoses. Panel analysis reduces but does not exclude incidental findings. Regular review of published literature is required to identify new reports that may aid classification.
| S-EPMC9325487 | biostudies-literature
Project description:The COVID-19 pandemic, caused by the novel coronavirus SARS-CoV-2, has been characterized by unprecedented rates of spatio-temporal spread. Here, we summarize the main events in the pandemic's timeline and evaluate what has been learnt by the public health community. We also discuss the implications for future public health policy and, specifically, the practice of epidemic control. We critically analyze this ongoing pandemic's timeline and contrast it with the 2002-2003 SARS outbreak. We identify specific areas (e.g., pathogen identification and initial reporting) wherein the international community learnt valuable lessons from the SARS outbreak. However, we also identify the key areas where international public health policy failed leading to the exponential spread of the pandemic. We outline a clear agenda for improved pandemic control in the future.
| S-EPMC8365337 | biostudies-literature
Project description:Adolescents 360 (A360) is a four-year initiative (2016-2020) to increase 15-19-year-old girls' use of modern contraception in Nigeria, Ethiopia and Tanzania. The innovative A360 approach is led by human-centred design (HCD), combined with social marketing, developmental neuroscience, public health, sociocultural anthropology and youth engagement 'lenses', and aims to create context-specific, youth-driven solutions that respond to the needs of adolescent girls. The A360 external evaluation includes a process evaluation, quasi-experimental outcome evaluation, and a cost-effectiveness study. We reflect on evaluation opportunities and challenges associated with measuring the application and impact of this novel HCD-led design approach. For the process evaluation, participant observations were key to capturing the depth of the fast-paced, highly-iterative HCD process, and to understand decision-making within the design process. The evaluation team had to be flexible and align closely with the work plan of the implementers. The HCD process meant that key information such as intervention components, settings, and eligible populations were unclear and changed over outcome evaluation and cost-effectiveness protocol development. This resulted in a more time-consuming and resource-intensive study design process. As much time and resources went into the creation of a new design approach, separating one-off "creation" costs versus those costs associated with actually implementing the programme was challenging. Opportunities included the potential to inform programmatic decision-making in real-time to ensure that interventions adequately met the contextualized needs in targeted areas. Robust evaluation of interventions designed using HCD, a promising and increasingly popular approach, is warranted yet challenging. Future HCD-based initiatives should consider a phased evaluation, focusing initially on programme theory refinement and process evaluation, and then, when the intervention program details are clearer, following with outcome evaluation and cost-effectiveness analysis. A phased approach would delay the availability of evaluation findings but would allow for a more appropriate and tailored evaluation design.
| S-EPMC6635668 | biostudies-literature
Project description:BackgroundEffective public health surveillance systems are crucial for early detection and response to outbreaks. In 2016, Kenya transitioned its surveillance system from a standalone web-based surveillance system to the more sustainable and integrated District Health Information System 2 (DHIS2). As part of Global Health Security Agenda (GHSA) initiatives in Kenya, training on use of the new system was conducted among surveillance officers. We evaluated the surveillance indicators during the transition period in order to assess the impact of this training on surveillance metrics and identify challenges affecting reporting rates.MethodsFrom February to May 2017, we analysed surveillance data for 13 intervention and 13 comparison counties. An intervention county was defined as one that had received refresher training on DHIS2 while a comparison county was one that had not received training. We evaluated the impact of the training by analysing completeness and timeliness of reporting 15 weeks before and 12 weeks after the training. A chi-square test of independence was used to compare the reporting rates between the two groups. A structured questionnaire was administered to the training participants to assess the challenges affecting surveillance reporting.ResultsThe average completeness of reporting for the intervention counties increased from 45 to 62%, i.e. by 17 percentage points (95% CI 16.14-17.86) compared to an increase from 49 to 52% for the comparison group, i.e. by 3 percentage points (95% CI 2.23-3.77). The timeliness of reporting increased from 30 to 51%, i.e. by 21 percentage points (95% CI 20.16-21.84) for the intervention group, compared to an increase from 31 to 38% for the comparison group, i.e.by 7 percentage points (95% CI 6.27-7.73). Major challenges for the low reporting rates included lack of budget support from government, lack of airtime for reporting, health workers strike, health facilities not sending surveillance data, use of wrong denominator to calculate reporting rates and surveillance officers having other competing tasks.ConclusionsTraining plays an important role in improving public health surveillance reporting. However, to improve surveillance reporting rates to the desired national targets, other challenges affecting reporting must be identified and addressed accordingly.
| S-EPMC7359619 | biostudies-literature
Project description:Public health and social measures (PHSM) have been central to the COVID-19 response. Consequently, there has been much pressure on decision-makers to make evidence-informed decisions and on researchers to synthesize the evidence regarding these measures. This article describes our experiences, responses and lessons learnt regarding key challenges when planning and conducting rapid reviews of PHSM during the COVID-19 pandemic. Stakeholder consultations and scoping reviews to obtain an overview of the evidence inform the scope of reviews that are policy-relevant and feasible. Multiple complementary reviews serve to examine the benefits and harms of PHSM across different populations and contexts. Conceiving reviews of effectiveness as adaptable living reviews helps to respond to evolving evidence needs and an expanding evidence base. An appropriately skilled review team and good planning, coordination and communication ensures smooth and rigorous processes and efficient use of resources. Scientific rigor, the practical implications of PHSM-related complexity and likely time savings should be carefully weighed in deciding on methodological shortcuts. Making the best possible use of modeling studies represents a particular challenge, and methods should be carefully chosen, piloted and implemented. Our experience raises questions regarding the nature of rapid reviews and regarding how different types of evidence should be considered in making decisions about PHSM during a global pandemic. We highlight the need for readily available protocols for conducting studies on the effectiveness, unintended consequences and implementation of PHSM in a timely manner, as well as the need for rapid review standards tailored to "rapid" versus "emergency" mode reviewing.
| S-EPMC9349463 | biostudies-literature