MedAI: Vision Language Models & Fine-Tuning (KnowAda) and Impact to Chinese TCM

Introduction

This article examines the challenges of fine-tuning smaller vision language models (VLMs) for medical applications, focusing on models with up to 8 billion parameters. It presents recent research on their effectiveness and limitations, including their impact on Chinese Medicine (TCM).

Smaller VLMs tend to hallucinate. Knowledge-Adapted Fine-Tuning (KnowAda) is a new approach to reduce hallucinations in VLMs when generating dense image captions. Traditional fine-tuning methods often struggle with balancing descriptiveness and factual accuracy in smaller-scale VLMs, especially for complex visual datasets.

KnowAda probes the VLM’s knowledge using generated visual questions to identify uncertain areas. It then adapts captions to exclude unreliable details while maintaining rich descriptions. Decomposed NLI (DNLI) evaluates caption claims against ground truth descriptions. These methods, tested across various datasets and models, significantly reduce hallucinations without compromising descriptiveness.

Additionally, the article discusses the impact of VLMs on Chinese Medicine (TCM).

All About VLM Fine-tuning for Medical AI Applications

Vision-Language Models (VLMs) have emerged as powerful tools in the realm of artificial intelligence, particularly in the medical field. By combining the capabilities of computer vision and natural language processing, VLMs can analyze and interpret complex medical data, leading to significant advancements in diagnosis, treatment, and research.

Understanding VLM Fine-tuning

Fine-tuning is a crucial technique in adapting pre-trained VLMs to specific medical tasks. It involves taking a model that has been trained on a massive dataset of images and text and further training it on a smaller, more specialized dataset relevant to the medical domain.

Key Steps in VLM Fine-tuning for Medical AI

1. Data Preparation:
   1. Data Collection: Gather a diverse and representative dataset of medical images (e.g., X-rays, CT scans, MRIs) paired with accurate textual descriptions.
   2. Data Cleaning and Preprocessing: Ensure data quality by removing noise, inconsistencies, and irrelevant information.
   3. Data Annotation: Annotate images with detailed labels and descriptions to provide the model with accurate ground truth.
2. Model Selection:
   1. Choose a Suitable VLM: Select a pre-trained VLM architecture that aligns with the complexity of medical data and the desired task.
   2. Consider Transfer Learning: Leverage the knowledge gained from pre-training on large-scale datasets to accelerate learning on medical data.
3. Fine-tuning Process:
   1. Parameter Tuning: Adjust hyperparameters like learning rate, batch size, and optimizer to optimize the training process.
   2. Loss Function: Employ a suitable loss function that balances image and text understanding, such as a combination of cross-entropy loss and contrastive loss.
   3. Training: Train the model on the medical dataset, iteratively updating its parameters to improve performance on the specific task.
4. Evaluation:
   1. Performance Metrics: Utilize appropriate metrics to assess the model’s accuracy, precision, recall, and F1-score.
   2. Validation: Evaluate the model on a validation set to monitor its performance during training and prevent overfitting.

Medical AI Applications of Fine-tuned VLMs

1. Medical Image Analysis:
   1. Disease Detection: Accurate identification of diseases like cancer, pneumonia, and Alzheimer’s disease.
   2. Image Segmentation: Precise delineation of anatomical structures for surgical planning and treatment.
   3. Image Captioning: Automatic generation of descriptive captions for medical images.
2. Medical Report Generation:
   1. Automated Report Writing: Generation of comprehensive and accurate medical reports from images and other clinical data.
   2. Natural Language Processing: Understanding and extracting information from clinical notes and literature.
3. Drug Discovery:
   1. Virtual Screening: Identification of potential drug candidates by analyzing molecular structures and properties.
   2. Target Identification: Discovery of new drug targets by analyzing protein-protein interactions and other biological processes.

Video about MedAI with VLM & Fine-Tuning:

Key Sections of the video

Current State of Medical AI Models

1. Discussion of pre-trained models (both large language models and vision language models)
2. Challenge of fine-tuning these models for specific medical tasks
3. Goal of combining visual information with textual diagnoses

Research Findings

1. November 6th publication from Kunmalan University and Johns Hopkins University
2. Analysis of 7 medical AI language models and 2 medical vision language models
3. Comparative results:
   1. LLM Zero-shot: 9% medical model wins, 56% tie, 34% medical model loses
   2. VLM Zero-shot: 6% medical model wins, 75% tie, 20% medical model loses
   3. Three-shot: 12% medical model wins, 50% tie, 38% medical model loses

Technical Limitations

1. Pre-training Knowledge Constraints
   1. Fine-tuning primarily adapts pre-existing knowledge
   2. Unable to acquire new information not present in pre-training
   3. Increased hallucination when dealing with unfamiliar content
2. Resolution Issues
   1. Most models trained on low-resolution images (1K-2K)
   2. Limited ability to process 4K or 8K images
   3. Struggle with fine details in high-resolution scenarios

The Impact of VLM Fine-tuning on Chinese Medicine (TCM):

VLM fine-tuning has the potential to significantly impact Chinese Medicine (CM) in several ways:

1. Enhanced Diagnosis and Treatment:

1. Image Analysis: VLMs can analyze medical images like tongue diagnoses, pulse readings, and herbal prescriptions. This can aid practitioners in making more accurate diagnoses and prescribing tailored treatments.
2. Pattern Recognition: VLMs can identify complex patterns in patient symptoms and medical histories, helping to uncover underlying imbalances and guide treatment strategies.
3. Personalized Medicine: By analyzing individual patient data, VLMs can help tailor treatments to specific needs, increasing efficacy and reducing side effects.

2. Advancements in Herbal Medicine:

1. Quality Control: VLMs can analyze the quality and authenticity of herbal medicines, ensuring consistency and safety.
2. Drug Discovery: By analyzing vast amounts of data on herbs and their interactions, VLMs can help identify new potential drug candidates.
3. Formulation Optimization: VLMs can optimize herbal formulas for specific conditions, improving their efficacy and reducing adverse effects.

3. Improved Patient Care:

1. Remote Consultations: VLMs can facilitate remote consultations, making CM more accessible to a wider population.
2. Patient Education: VLMs can provide patients with personalized health information and advice, empowering them to take an active role in their healthcare.
3. Early Disease Detection: By analyzing patient data, VLMs can help identify early signs of disease, enabling timely intervention.

4. Research and Development:

1. Literature Review: VLMs can efficiently analyze and synthesize vast amounts of traditional medical literature, accelerating research.
2. Clinical Trial Analysis: VLMs can analyze clinical trial data to identify trends and potential new treatments.
3. Knowledge Discovery: By analyzing large datasets, VLMs can uncover new insights into the mechanisms of action of CM therapies.

Challenges and Considerations:

1. Data Quality and Quantity: High-quality, annotated data is essential for effective VLM training.
2. Cultural and Contextual Understanding: VLMs must be trained on data that reflects the nuances and complexities of CM.
3. Ethical Considerations: Ensuring the ethical use of AI in healthcare, including data privacy and algorithmic bias.
4. Integration with Traditional Practice: Balancing the use of technology with the human touch of traditional CM practitioners.

Challenges and Future Directions:

1. Data Quality and Quantity: High-quality, annotated medical data is essential for effective fine-tuning.
2. Model Interpretability: Understanding the decision-making process of VLMs is crucial for clinical adoption.
3. Ethical Considerations: Addressing privacy, bias, and fairness issues in medical AI applications.

Conclusion

The article highlights significant challenges in current medical AI implementations, particularly in fine-tuning smaller vision language models. The current approach of fine-tuning pre-trained models for medical applications shows limited success, with many systems performing no better than their base models.  By addressing these challenges and leveraging the potential of VLM fine-tuning, it can usher in a new era of innovation and advancement in Chinese Medicine.

Key Takeaways

1. Fine-tuning alone is insufficient for specialized medical AI applications
2. Need for task-specific pre-training rather than relying on fine-tuning
3. Importance of high-resolution training data for medical applications
4. Current dependency on larger models to verify smaller model outputs
5. Questions about the sustainability of synthetic data generation for training

Related References

1. Google’s studies on hallucination in vision language models
2. LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
3. TCM-FTP: Fine-Tuning Large Language Models for Herbal Prescription Prediction
4. “Bridging Visual Gap” methodology
5. Use of larger models to verify and correct smaller model outputs