Healthcare technology is often driven by the pursuit of so-called “holy grail” concepts—advancements with the potential to transform the field. Among these, one of the most highly anticipated and impactful concepts has been the development of longitudinal healthcare records. This concept envisions a complete, continuous record of a patient’s health data, encompassing everything from clinical history and lab tests to radiology exams and prescriptions. Despite significant technological advancements, the reality of achieving true longitudinal records remains elusive for various reasons. The so called womb to tomb healthcare history of a person accessible by the clinicians across the healthcare continuum.

Imagine a routine visit to your doctor’s office. Your doctor’s insights rely on a range of documents, including clinical notes, lab results, radiology reports, and prescription histories. Most often every visit or discharge summary predominantly has information about these four Health Record Formats or Health Record Categories (https://sandbox.abdm.gov.in/sandbox/v3/new-documentation?doc=HealthRecordFormats).

These four categories of Health Records, at the minimum, can therefore be considered to represent the core types of data that are essential for a meaningful and holistic view of a patient’s health. By reviewing various datasets in an Indian context, we could consider most patients having information about these four health record types available with them.

In a way, these records can be thought of as forming a foundational basis for establishing longitudinal healthcare records or by extention multi-modal biomedical AI. These four types of health records are essential in the journey toward availability of longitudinal data and more importantly towards a “MultiModal BioMedical Ai” capabilities. (https://www.nature.com/articles/s41591-022-01981-2). The first step of leveraging the capabilities of a multi-modal data source for BioMedical AI can begin by adding these four Health Record types to enable them to play an important role in improving clinical precision and recall in the near term.

“MultiModal BioMedical Ai” capabilities. (https://www.nature.com/articles/s41591-022-01981-2)

The Essentials of the Health Records

Achieving a comprehensive longitudinal healthcare record involves unifying a patient’s medical history into a single, cohesive timeline. Currently, most patient data is fragmented across different providers and systems, leading to gaps in information and potential miscommunication. Additionally, if one were to do a clinical records inventory with the patient and patient family, one would more often than not, find the availability of these reports with the patients. A quick survey in any outpatient clinic of any healthcare establishment as an observation, you will find the patient’s in a majority will have one or more of these health records as part of their paper based (or in some cases digital) medical records. Some context about the health records we are focusing on:

a. Clinical Records: These include doctor visit notes, diagnoses, physical examinations, and clinical observations. Clinical records capture the nuances of a patient’s symptoms and the context of their health conditions over time.

b. Lab Tests: Laboratory results provide crucial biomarkers that help in understanding a patient’s physical and biochemical state. Blood tests, for example, can reveal changes in health long before symptoms arise.

c. Radiology Exams: Imaging records such as X-rays, MRIs, Ultrasounds and CT scans provide a visual history of structural changes in a patient’s body, offering insight into both acute and chronic conditions.

d. Pharmacy Prescriptions: Medication records not only show current treatments but also reveal the history of interventions tried, helping providers assess treatment efficacy and potential drug interactions.

As per Praxis Global Alliance’s 2021 India OPD market report, on an average, Indians do 3 doctor consultations in a year; each consultation leads to one pharma prescription, and a diagnostics test is prescribed in every 3.5 consultations.

• Some additional statistics about patient visit to the doctor from CDC https://www.cdc.gov/nchs/fastats/physician-visits.htm
• Life Satisfaction and Frequency of Doctor Visits, Eric S Kim et al | https://pmc.ncbi.nlm.nih.gov/articles/PMC4608020

These four types of records are essential because they collectively by itself can offer a multi-faceted view of a patient’s health, capturing everything from symptoms and lab results to treatment plans. Together, they establish a strong foundation for creating longitudinal records, and by extention multi-modal biomedical Ai, offering healthcare providers a clear, integrated and personalised snapshot of a patient’s health trajectory. In an India context one could consider starting the longitudnal healthcare records journey by considering populating the list of Health Records from the ABHA Sandbox, based on FHIR’s specifications, in Reference: https://sandbox.abdm.gov.in/sandbox/v3/new-documentation?doc=HealthRecordFormats

Name	Definition
Diagnostic Report Record	The Clinical Artifact represents diagnostic reports including Radiology and Laboratory reports that can be shared across the health ecosystem.
Discharge Summary Record	Clinical document used to represent the discharge summary record for ABDM HDE data set.
Health Document Record	The Clinical Artifact represents the unstructured historical health records as a single of multiple Health Record Documents generally uploaded by the patients through the Health Locker and can be shared across the health ecosystem.
Immunization Record	The Clinical Artifact represents the Immunization records with any additional documents such as vaccine certificate, the next immunization recommendations, etc. This can be further shared across the health ecosystem.
OP Consult Record	The Clinical Artifact represents the outpatient visit consultation note which may include clinical information on any OP examinations, procedures along with medication administered, and advice that can be shared across the health ecosystem.
Prescription Record	The Clinical Artifact represents the medication advice to the patient in compliance with the Pharmacy Council of India (PCI) guidelines, which can be shared across the health ecosystem.
Wellness Record	The Clinical Artifact represents regular wellness information of patients typically through the Patient Health Record (PHR) application covering clinical information such as vitals, physical examination, general wellness, women wellness, etc., that can be shared across the health ecosystem.
Invoice Record	This billing HI type comprises of invoice details such as pharmacy invoice, consultation invoice etc.

Why These Four Health Records Matter

Imagine a doctor diagnosing a complex condition without a complete picture of a patient’s health. Having a holistic view—one that encompasses clinical notes, lab results, imaging, and prescriptions—enables a more accurate and timely diagnosis. These four records represent the minimum essential data or for that matter minimal dataset that clinicians need to make informed decisions.

In a way these four Health Records help any clinician for context setting about the patients’ recent and relevant condition or diagnosis. Imagine if a patient were to be seen by a clinician in an ER, the clinician will be able to understand the contraindications of the medication being administered in the ER to the current list of medications from the patients’ drug profile or for that matter the pharmacy health record. The clinician will be able to get to know specific biomarkers from the patients clinical and laboratory health records. And be able to view radiology health records that might be recent and relevant for the current clinical encounter in the ER. Prior to ordering additional investigative tests and examinations the clinician will be able to alter the trajectory of treatment of the patient. Aditionally, the context these records provide can be transformative in other instances. For instance: A lab result might indicate elevated cholesterol, but combined with prescription data, a doctor can see if cholesterol-lowering medications are helping. A radiology exam could reveal early-stage bone degeneration, but clinical notes provide insight into lifestyle factors that could mitigate or accelerate this process.

When this recent and relevant information is readily available, healthcare providers can deliver precision in diagnosis and treatment recommendations. This approach not only saves time but also supports better clinical outcomes by minimizing the risk of missed or misinterpreted information. While working towards creating a complete patient picture, these four types of records enable healthcare providers to offer both precision and recall, helping ensure each patient receives accurate and personalised care.

Leveraging Longitudinal Records for AI in Healthcare

In recent years, healthcare technology companies have increasingly looked to artificial intelligence (AI) to improve diagnostics, personalize treatments, and streamline care delivery. For these AI models to be effective, they need comprehensive and contextually rich data—a need that longitudinal health records are uniquely positioned to meet. When AI models are trained on longitudinal data, they gain insights into how a patient’s health evolves over time. This enables AI to recognize patterns that might otherwise be missed in isolated snapshots of a patient’s health. Data Centric AI additionally is needed for enabling fine tuning in the context of LLMs. For instance:

• Predictive Analytics: Longitudinal records allow AI models to forecast health trajectories, such as predicting the likelihood of a chronic condition worsening or responding to treatment.
• Risk Stratification: By analyzing long-term data trends, AI can identify high-risk patients who may benefit from preventive interventions.
• Personalized Treatment Recommendations: AI can use past lab results, imaging, and treatment history to recommend more tailored treatment options based on what has proven effective for similar patients.

Image Ref: Foundation models for generalist medical artificial intelligence Michael Moor, Oishi Banerjee, Pranav Rajpurkar et al https://www.nature.com/articles/s41586-023-05881-4

The paper talks about the benefits of “generalist medical AI (GMAI): GMAI models will be capable of carrying out a diverse set of tasks using very little or no task-specific labelled data. Built through self-supervision on large, diverse datasets, GMAI will flexibly interpret different combinations of medical modalities, including data from imaging, electronic health records, laboratory results, genomics, graphs or medical text. Models will in turn produce expressive outputs such as free-text explanations, spoken recommendations or image annotations that demonstrate advanced medical reasoning abilities”

For healthcare technology companies aiming to create impactful AI applications, integrating these four core health record types—clinical documentation, laboratory results, radiology images and reports, and pharmacy prescription data—into data pipelines is a vital first step. Not only are these records highly relevant to a patient’s clinical picture, but two (lab and radiology, in most cases) of these four types are also already digitized in most healthcare facilities, making integration a realistic goal. By prioritizing these data sources, healthcare organisations and HealthTech companies can significantly enhance their AI models’ accuracy and relevance, building a foundation that can later be expanded to include additional data types. The importance of this information and the insights that can be provided as a dashboard to the clinician can for instance help answering the following questions (to list a few):

1. What were the various visit diagnosis for which the patient had an IP or OP encounter
2. What was the most common diagnosis for which the patient visited the doctor, can help guide the clinician about the patient’s prevalent diagnosis that might be important in the clinicians decisions.
3. The clinicians will be able to identify the drug patterns and any drug switching and the reasons for the same.
4. Understand medication adherence patterns based on the refill schedules and refill quantities.
5. Understand any contraindications for the medications being prescribed in the current context with the medications the patient has been prescribed previously.

These are just a few insights that a patient’s dashboard can provide to the clinician based on just these 4 health records and the information present in them. From presenting the list of discrete data elements the current capabilities of LLMs allow for the creation of very insightful summaries that can be augmented and validated by the consultants in the context of a surgical oncology ward during the ward rounds or for that matter in the surgical cardiology ward. Insightful dashboards for patients can now be created for enabling elderly care by keeping track of the early onset dementia.

Precision and Recall: The Diagnostic Impact of Longitudinal Records

In enabling multi-modal Biomedical Ai, precision and recall should be considered a critical metric that determines the effectiveness of diagnostic Ai tools towards influencing improved patient outcomes. Longitudinal records, by providing a comprehensive view of a patient’s health, can have a significant impact on both precision and recall in medical diagnosis. Precision refers to the accuracy of positive predictions—in this context, it measures the proportion of true positive diagnoses among all positive diagnoses made. High precision in healthcare means that when a diagnostic model predicts a disease, it is likely correct, reducing unnecessary treatments and anxiety for patients. Recall, on the other hand, measures a model’s ability to identify all relevant and recent cases within a dataset. High recall is crucial in healthcare because it ensures that patients with the disease are not overlooked. For instance, when screening for a serious disease like cancer, recall is paramount; even a single missed diagnosis could have life-threatening consequences. For chronic conditions like diabetes, precision is equally vital to avoid over-diagnosis, which could lead to unnecessary interventions. Longitudinal data can support precision by showing previous lab results and treatment responses, helping providers gauge whether a high blood sugar reading is part of a pattern or an isolated event. Balancing precision and recall is a complex challenge in healthcare, as the stakes are high. However, the availability of longitudinal health records and by extention multi-modal biomedical Ai offers a promising solution by enabling models for healthcare providers to access richer, more reliable data, ultimately supporting more accurate diagnostics and improved patient care outcomes.

Practical Considerations and Challenges

While the potential of multi-modal healthcare records is clear, the journey toward achieving them comes with significant practical and technical challenges. Integrating the four essential record types—clinical notes, lab results, radiology exams, and pharmacy prescriptions—into a unified system involves overcoming both logistical and technological hurdles.

1. Interoperability: Healthcare systems often use different platforms and standards for storing and sharing data, making it challenging to connect records across facilities. Without a common framework, data silos form, obstructing the continuity needed for longitudinal records.
2. Data Standardization: Clinical data varies in terminology, structure, and formatting. Harmonizing lab results, for instance, may require converting disparate units and standardizing test names, while clinical notes often need additional processing to structure free-text entries into usable formats.
3. Data Privacy and Security: Longitudinal health records are an attractive target for cybersecurity threats, as they consolidate sensitive data. Strict adherence to data protection laws, such as HIPAA in the U.S. and GDPR in Europe, is essential to safeguard patient privacy. Technologies like encryption and secure access protocols can help protect this data but may add complexity.
4. Adoption and Change Management: For healthcare providers, adopting systems that support longitudinal records can involve significant time and training investments. Many clinicians are already dealing with heavy digital workloads, so ensuring ease of use and demonstrating clear benefits are key to encouraging adoption.

Conclusion

In India, we have the ABHA Health Records framework that will allow each of the Healthcare Organisations to have the power of Precise Insights driven by a powerful Multi-modal Biomedical Ai paradigm and this can effort can begin by creating across the country, these 4 types of Health Records. Even today, there are many healthcare establishments that are generating these 4 types of health records on paper. One could consider this as an opportunity lost in moving from a illness to a wellness paradigm in healthcare and adhering and aligning with the sustainability goals for a country perspective. In the need to be frugal in delivery of healthcare services we are letting go of a major advantage towards enabling a data centric Ai ecosystem that helps us leapfrog major advancements that are happening across the world.

Our population advantage and the corresponding healthcare data digitisation via ABHA framework can allow the creation of LLMs that are fine-tuned for the indian context. The ability of large patient cohorts have provided some very interesting insights for Genomics in the UK and we have started to see a similar effort by efforts in IISc, Bangalore, IIT Kanpur and the Atlas project in the india context and many others that I hope to document in the context of this article in the future.

Multi-Modal Biomedical health records therefore represent one of the most powerful tools in healthcare technology’s arsenal, promising to transform patient care by creating a comprehensive, lifelong view of each patient’s health. Hopefully, these small steps can enable the baseline for the holy grail (this time in context of LLMs)

References

Longitudinal Healthcare Records & Digital Health

1. National Digital Health Mission. Health Record Formats Documentation [Internet]. ABDM Sandbox; 2023. https://sandbox.abdm.gov.in/sandbox/v3/new-documentation?doc=HealthRecordFormats
2. Jain D. Regulation of Digital Healthcare in India: Ethical and Legal Challenges. Healthcare (Basel). 2023 Mar 21;11(6):911. doi: 10.3390/healthcare11060911. PMID: 36981568; PMCID: PMC10048681. https://pmc.ncbi.nlm.nih.gov/articles/PMC10048681/

Multimodal AI in Healthcare

1. Acosta, J.N., Falcone, G.J., Rajpurkar, P. et al. Multimodal biomedical AI. Nat Med 28, 1773–1784 (2022). https://doi.org/10.1038/s41591-022-01981-2
2. Sheller MJ, Edwards B, Reina GA, et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci Rep. 2020;10(1):12598. https://doi.org/10.1038/s41598-020-69250-1
3. Huang SC, Pareek A, Seyyedi S, et al. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digit Med. 2020;3(1):136. https://doi.org/10.1038/s41746-020-00341-z
4. Liu, F., Zhu, T., Wu, X. et al. A medical multimodal large language model for future pandemics. npj Digit. Med. 6, 226 (2023). https://doi.org/10.1038/s41746-023-00952-2
5. Multimodal medical AI, Posted by Greg Corrado, Head of Health AI, Google Research, and Yossi Matias, VP, Engineering and Research, Google Research https://research.google/blog/multimodal-medical-ai/

Clinical Decision Support & Data Integration

1. Rajkomar A, Oren E, Chen K, et al. Scalable and accurate deep learning with electronic health records. NPJ Digit Med. 2018;1(1):18. https://doi.org/10.1038/s41746-018-0029-1
2. Esteva A, Robicquet A, Ramsundar B, et al. A guide to deep learning in healthcare. Nat Med. 2019;25(1):24-29. https://doi.org/10.1038/s41591-018-0316-z
3. Yu KH, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng. 2018;2(10):719-731. https://doi.org/10.1038/s41551-018-0305-z

Precision and Recall in Medical Diagnostics

1. Liu X, Faes L, Kale AU, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health. 2019;1(6):e271-e297. https://doi.org/10.1016/S2589-7500(19)30123-2
2. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44-56. https://doi.org/10.1038/s41591-018-0300-7

Healthcare Data Privacy and Security

1. Price WN, Cohen IG. Privacy in the age of medical big data. Nat Med. 2019;25(1):37-43. https://doi.org/10.1038/s41591-018-0272-7
2. Bates DW, Levine D, Syrowatka A, et al. The potential of artificial intelligence to improve patient safety: a scoping review. NPJ Digit Med. 2021;4(1):54. https://doi.org/10.1038/s41746-021-00423-6

Indian Healthcare Context

1. The future of health in India Tech-enabled care delivery for empowered consumers, Delloitte April 2024 https://www2.deloitte.com/content/dam/Deloitte/in/Documents/public-sector/in-ps-the-future-of-health-noexp.pdf
2. Maroju RG, Choudhari SG, Shaikh MK, Borkar SK, Mendhe H. Role of Telemedicine and Digital Technology in Public Health in India: A Narrative Review. Cureus. 2023 Mar 10;15(3):e35986. doi: 10.7759/cureus.35986. PMID: 37050980; PMCID: PMC10085457. https://pmc.ncbi.nlm.nih.gov/articles/PMC10085457/#:~:text=Even in the post-COVID,cost of in-person care.

Disclosure

Various Ai tools were used to proofread, research, generate some of the images in the article. The central idea about moving towards a data Centric Ai in an Indian context remains the main focus of the article