Title Picture Reference - AI
Generated Image at website - https://designer.microsoft.com/image-creator
Summary of the Blogpost – This blogpost is an attempt to illustrate an idea for doing performance assessment of healthcare facilities based on accuracy of provisional diagnosis for a given disease. To do this, the blogpost relies on the HIMS (Hospital Information Management System) data captured across the facilities (which are taken into consideration for comparison of performance). It is suggested that data for symptoms, provisional diagnosis and final diagnosis for facilities for a given time period can be subjected to Confusion Matrix for calculation of respective Performance of Facilities (represented through Sensitivity and Specificity for Provisional Diagnosis for the disease). It is further suggested that relative performance of facilities for accuracy of Provisional Diagnosis can be plotted on a S – S (Sensitivity and Specificity) Cartesian for classification of facilities. However, the analysis presents just a possible idea based on imaginary data as real data is not available to test the presented idea. The blogpost is written in the form of questions and answers.
- - - - - - - - - - - -
What is Health Management?
In brief, health management provides guidance and leadership
to administer health at the individual, organizational and systemic level.
Health management embraces a holistic vision of health, in which health is
impacted by behavioral, social, and environmental determinants. (Further
details are available under APPENDIX – C of this blogpost)
What is the role of Information Technology in Health
Management?
HMS (Hospital Management System) or HIMS (Hospital
Information Management System) provides the underpinnings for decision-making
and has four key functions: data generation, compilation, analysis and
synthesis, and communication and use. (Further details are available under
APPENDIX – C of this blogpost)
Why should data be analyzed for Provisional and Final
Diagnosis?
Provisional diagnosis (PD) is the first considered diagnosis
which initiates the first phase of management, whereas confirmatory or final
diagnosis (FD) is the chronological organization and critical evaluation of
information obtained from the history, physical examination, and
investigations. (NCBI).
Research shows the existence of gaps between provisional and
final diagnosis in both private and government institutions. Both knowledge and
skill gaps were evident in caregivers and gap in documentation was existent in
medical records. (Chatterjee S, Ray
K, Das AK. Gap analysis between provisional diagnosis and final diagnosis in
government and private teaching hospitals: A record-linked comparative study. J
Family Med Prim Care. 2016 Jul-Sep;5(3):637-640. doi: 10.4103/2249-4863.197318.
PMID: 28217597; PMCID: PMC5290774.). Matching of provisional diagnosis with
discharge diagnosis (final diagnosis) with greater accuracy and lesser number
of investigations can lead to greater patient satisfaction along with lesser
burden on health resources of the state. Accurate provisional diagnosis in the
Emergency Department (ED) is considered to be more important as it has a
significant impact on safety (Adnan
& Baharuddin "A Study on the Diagnostic Discrepancy between Admission
and Discharge in Hospital Universiti Sains Malaysia" January 2021
Malaysian Journal of Medicine and Health Sciences 17(1):105-110 17(1):105-110).
Historically, diagnosis relied more on symptoms. With the
development of systematic methods of clinical examination more reliance was
placed on signs. However major breakthroughs in diagnosis were achieved when
medical technology provided a wide array of accurate and reliable laboratory
and radiological investigations. (Chattopadhyay,
Amitabha, et al. "Gap analysis between provisional diagnosis on admission
and final diagnosis during discharge–A comparative study." J Dent Med Sci
8 (2013): 28-31.). A brief on Symptoms and Signs can be accessed under APPENDIX
- A for ready reference.
Analysis of HIMS/HMS generated (non-repudiable) data (mined
from large sample of anonymized patient records) for (disease wise) matching of
provisional and final diagnosis done at facility level may be helpful in many
respects, like: -
1.
Improving Differential Diagnosis - Differential diagnosis is defined as the process of
differentiating between the probability of one disease versus that of other
diseases with similar symptoms that could possibly account for illness in a
patient. (Refer - Deepti
Lamba, William H. Hsu, Majed Alsadhan, Chapter 1 - Predictive analytics and
machine learning for medical informatics)
2.
Observing Changing Signs and Symptoms of Diseases – Manifestations of diseases may keep on changing with changing times (Bennett
Lorber, Changing patterns of infectious diseases, The American Journal of
Medicine, Volume 84, Issue 3, Part 2, 1988, ISSN 0002-9343). Such changes
may happen due to numerous factors like (say) changing lifestyles of a
population or (say) mutating pathogens. The results of such changes could be observed
as variations in symptoms and in signs associated with diseases.
3.
Bringing Consistency in Diagnosis - Detailed analysis of mismatch in provisional and final
diagnosis across a large set of facilities may (in some cases) help in bringing
consistency in diagnosis of some of the diseases. In general, conducting such
an analysis periodically could even lead to improvement in quality of diagnosis
in a given set of facilities.
4.
Developing a better understanding on the way Patients express Symptoms of
a Disease to Care-Givers – Usually, patients come to a facility from extremely diverse backgrounds
(say in terms of languages spoken, education levels, religious and social beliefs)
and therefore may have different ways to articulate medical complains. These complaints
become the basis for recording of the symptoms. Quality of (interactive)
communication between a patient and clinical caregiver (or say the doctor) may
be critical for improving accuracy of symptoms narrated. These symptoms (and
observed signs) in-turn become the basis for provisional diagnosis by the
doctor (or an authorized clinical professional).
A possible cause of poor matching of
provisional diagnosis with the final diagnosis could be poor quality of
interactive communication between patient and doctor. This aspect may get
attention and most likely get improved with frequent assessment of gap in
provisional and final diagnosis.
Can you suggest a way to do an analysis for Provisional and
Final Diagnosis?
Let us try to develop an idea to do an analysis of
provisional and final diagnosis from data captured through HIMS. In this
analysis, we will try to apply Confusion Matrix to assess healthcare facilities
for correct provisional diagnosis of a given disease. This assessment will be
done for data collated during a fixed period of time across a given set of
facilities with an assumption that all other related parameters remain the same
for these facilities.
It is assumed that the following data is available for a
given facility, which has implemented HIMS - a) symptoms reported by patients for a disease
at triaging, b) the provisional diagnosis (likely-positive (likely to have a
disease) or likely-negative (not likely to have a disease)) and c) the final
diagnosis (positive or negative) for the disease. In this analysis we will
analyze capability of the facility to be correct in doing provisional diagnosis
for the disease. This capability of the facility will be expressed in terms of
two parameters Sensitivity and Specificity (described later in this blogpost). Once
capability of a facility is determined, the same will be repeated for a given
set of facilities. Thus, relative capabilities of facilities chosen for
analysis can be determined (in terms of respective Sensitivity and Specificity
values).
A Confusion Matrix is commonly used in assessment of accuracy
and reliability of outputs of Machine Learning models. Essentially, a confusion
matrix is a matrix that summarizes the performance of a machine learning model
on a set of test data. It is a means of displaying the number of accurate and
inaccurate instances based on the model’s predictions. It is often used to
measure the performance of classification models, which aim to predict a
categorical label for each input instance. (geeksforgeeks.org).
In table – 1, a confusion matrix is prepared by matching data of Provisional
Diagnosis with corresponding data for the Final Diagnosis.
The matrix displays the number of instances produced by the
model on the test data. These instances in reference to table – 1 (representing
our idea of analysis) are as under (geeksforgeeks.org):
-
1.
True Positive
(TP): The model correctly predicted a positive outcome (the actual outcome was
positive). This is represented as ‘a’ in Table – 1 in reference to this
analysis.
2.
True Negative
(TN): The model correctly predicted a negative outcome (the actual outcome was
negative). This is represented as ‘d’ in Table – 1 in reference to this
analysis.
3.
False Positive
(FP): The model incorrectly predicted a positive outcome (the actual outcome
was negative). Also known as a Type I error. This is represented as ‘b’ in
Table – 1 in reference to this analysis.
4. False Negative (FN): The model incorrectly predicted a negative outcome (the actual outcome was positive). Also known as a Type II error. This is represented as ‘c’ in Table – 1 in reference to this analysis.
Table – 1: Confusion
Matrix for Provisional Diagnosis and Final Diagnosis in Healthcare
We will apply the confusion matrix (as above in table-1) to
the sample of data available for a critical disease from a given set of
facilities (let us take 15 imaginary facilities and therefore 15 corresponding
Matrix with corresponding values of TP, FP, FN and TN). In context of India, the critical disease
could be any of the diseases with high prevalence (Communicable or
Non-Communicable). However, as an illustrative example, we will take
Tuberculosis (because, India
accounts for about 25% of global TB burden, with an estimated TB incidence of
2.77 million in 2022). In application of confusion matrix, the sample size
of data is recommended to be statistically significant (say typically around
500 cases from every facility for applying to the matrix), the period of
capture of data is to be considered reasonably recent and quality of data be
considered good in terms of completeness, cleanliness, accuracy and
reliability.
In relation to our analysis, the provisional diagnosis will
be assessed on Sensitivity and Specificity. In reference to the Matrix shown in
the table – 1, the formula and explanation for these two parameters are given as
under (a,b,c and d used in the following formula are taken from the Table – 1):
-
1.
Sensitivity
= [a/(a+c)]×100
2.
Specificity
= [d/(b+d)]×100
Sensitivity - The sensitivity of a test is also called the true positive
rate (TPR) and is the proportion of samples that are genuinely positive that
give a positive result using the test in question. For example, a test that
correctly identifies all positive samples in a panel is very sensitive. Another
test that only detects 60 % of the positive samples in the panel would be
deemed to have lower sensitivity as it is missing positives and giving higher a
false negative rate (FNR). Also referred to as type II errors, false negatives
are the failure to reject a false null hypothesis (the null hypothesis being
that the sample is negative).
Specificity - The specificity of a test, also referred to as the true
negative rate (TNR), is the proportion of samples that are genuinely negative
that give a negative result using the test in question. For example, a test
that identifies all healthy people as being negative for a particular illness
is very specific. Another test that incorrectly identifies 30 % of healthy
people as having the condition would be deemed to be less specific, having a
higher false positive rate (FPR). Also referred to as type I errors, false
positives are the rejection of a true null hypothesis (the null hypothesis
being that the sample is negative).
A possible process for carrying out collection of data for
the analysis is illustrated as under: -
A patient with symptoms of a disease is triaged and sent to a
doctor (in a of the given facility being considered for this analysis). Such
patients (with due consent) will be considered as subjects for this analysis. A
subject can be Provisionally Diagnosed as “Likely Positive” or “Likely Negative”
for Tuberculosis (TB) by the attending Doctor. Based on provisional diagnosis
the doctor will start suitable medication and at the same time recommend
confirmatory tests (for possible evidence(s) of TB). Once the results of tests
are available, the doctor will write the Final Diagnosis as a “Positive” or a
“Negative” case of TB. The data for Symptoms, for Provisional Diagnosis and for
Final Diagnosis will be used to be fed to analytical framework used (Confusion
Matrix).
In a real scenario, for every facility chosen for analysis
(like we have chosen 15 facilities for our imaginary case), one Confusion
Matrix will be prepared based on data captured at that facility (say clean dataset
in the range of 500 cases). Sensitivity and Specificity will be calculated for
every Confusion Matrix. These parameters (Sensitivity and Specificity) will
represent performance of diagnosis (on correctness of the provisional
diagnosis) for the given disease across the chosen facilities.
Since we do not have real data, let us try to work-out on the above analytical idea through some imaginary random data. So, with an imaginary data for Sensitivity and Specificity of 15 Imaginary Facilities (assumed to have been derived from huge data collected during 01-JAN-2023 to 31-DEC-2023 in respective facilities – typically valid dataset of around 500 for every facility), an S-S (Sensitivity and Specificity) Cartesian Plot has been prepared. This is a deviation form ROC curve (between values of ‘Sensitivity’ and ‘1-Specificity’) plotted by data analysts to test their models. The deviation is taken considering that we are not testing different provisional diagnosis AI model, but we are exploring an indicator for performance of facilities on Provisional and Final diagnosis. The imaginary data used is given under APPENDIX – B. The S-S Plot is as under: -
Figure – 2: Suggestive Sensitivity and Specificity Plot (S – S Plot) for
a selected set of Facilities based on Provisional Diagnosis and Final Diagnosis
data collected over a given period of time
Suggestive interpretations based on the above S – S curve is given in the following table. In case, it turns out that the S – S curve is found relevant on application of this idea for analysis (analytical framework) on real data, the suggestive interpretations may be used by policy makers and administrators for improving provisioning of healthcare.
Table – 2: Suggestive Interpretation of the S – S Plot with respect to the performance of Facilities on accuracy of Provisional Diagnosis
The assumptions made for the above S – S plot include: -
1.
Reliability
of Confirmatory Test is assured
2.
Clarity
in communication between Patient and Service Providers is observed across all
facilities
3.
Availability
of required Clinical and Non-Clinical Infrastructure is good in all facilities
4.
Patient
Load per Doctor do not vary significantly across facilities
5.
Qualification
of Clinical and Non-Clinical Staff do not vary significantly across facilities
6.
Patient
Load with multiple diseases (which may include or exclude the disease in
question) is largely uniform across the facilities
7.
Asymptomatic
patients are rare (statistically negligible) and will remain excluded from this
analysis
8. Prevalence of the disease does not vary in high magnitude across the facilities chosen and does not impact diagnostic accuracy in any significant way across the facilities
Should the suggested Analytical Framework (if found relevant
on real data) be used by Healthcare Administrators or Policy Makers?
Measurement of performance in healthcare is a critical issue
and needs adequate care in selection of indicators for administrative purposes.
Complexities in health administration include: -
a.
Freedom,
Flexibility, Discretion and Authority of Healthcare Service Providers
b.
Mutual
trust between Patient and Treating Doctor
c.
Right
to Information, Freedom of Choice and Right to Privacy of Patients
d.
Community
and Societal Values and Government Regulations
e.
Tradition
and Beliefs of stakeholders
f.
Diversity
in adherence to divergent, contradictory or competing School of Thoughts in
Medical Fraternity
g.
Resource
optimization challenges with prevailing Patient Load, Services Rendered,
Funding, Locality and Infrastructure availability at Facilities
h.
Cost
of Medical Expense to the Patient
i.
Temptation
of a facility to improve on measured indicators (at times, which may
unknowingly turn out to introduce biases in treatment)
I believe that the matter of healthcare administration be
left to the administrative experts. Ideas suggested by data analysts may be
scrutinized in the larger context of healthcare administration. Only those
ideas, which could stand scrutiny of healthcare experts be put to use for
administering healthcare facilities (and that too after incorporating suggested
modifications).
In this case, the idea expressed in this blogpost is at this stage, just a possibility. The idea is not even put to test through real data (which I do not have access to). The idea is not even debated and improved with domain experts. Thus, at the best this blogpost can be considered as an idea open to debate and corrections to get refined for a possible consideration to be tested with real data by real data analysts or researchers.
/***********************************/
APPENDIX – A: Brief on Symptoms and Signs
Symptoms and Signs are briefed as under: -
§ Symptoms - Something that a person feels or
experiences that may indicate that they have a disease or condition. Symptoms
can only be reported by the person experiencing them. They cannot be observed
by a health care provider or other person and do not show up on medical tests.
Some examples of symptoms are pain, nausea, fatigue, and anxiety. (NCI)
§ Signs - In medicine, a sign is something
found during a physical exam or as a result of a laboratory or imaging test
that shows that a person may have a condition or disease. Signs can be observed
by a health care provider or other person. Some examples of signs are fever,
swelling, skin rash, high blood pressure, and high blood glucose. (NCI)
APPENDIX – B: Imaginary Data to Plot the Illustrative S - S CURVE
APPENDIX
– C: Information
Technology in Health Management
What is Health Management?
Health management is complex, and its meaning can vary
depending on local contexts. EHMA (European Health Management Association) with
our members have developed a definition that captures the needs of health
systems all across Europe. In brief, health management provides guidance and
leadership to administer health at the individual, organisational and systemic
level. Health management embraces a holistic vision of health, in which health
is impacted by behavioral, social, and environmental determinants. Health
management includes and goes beyond healthcare management, which comprises
community, primary, secondary, and tertiary care provision. It also happens
outside of care settings and builds synergy with other related policy and
societal areas in line with the ‘One health’ concept. Health management
encompasses the entire health ecosystem in which health managers collaborate
with patients, informal and formal caregivers, patient organisations,
legislators, educators, policy makers and regulators, public health experts,
researchers, health insurance experts, and pharmaceutical industries. Together,
they aim to create a clear health vision and alignment strategy, as well as lay
down the organisational, societal and technological conditions to achieve
optimal health outcomes for individual patients and the entire community.
Health managers are jointly responsible for establishing effective and holistic
governance structures, built on a co-design and co-production model. (EHMA)
What is the role of Information Technology in Health
Management?
The evolving paradigm shift resulting from IT, social and
technological changes has created a need for developing an innovative
knowledge-based healthcare system, which can effectively meet global healthcare
system demands and also cater to future trends. The Hospital Information Management System
(HIMS) is developed with this sole aim in mind, which helps in processing and
management of hospital information not only inside the boundary, but also
beyond the hospital boundary, e.g., telemedicine or e-healthcare. (Wadhwa S, Saxena A, Wadhwa B.
Hospital information management system: an evolutionary knowledge management
perspective. Int J Electron Healthc. 2007;3(2):232-60. doi:
10.1504/IJEH.2007.013103. PMID: 18048272.)
