Research 03 · Under review · SN Computer Science
Medical Image Retrieval
A medical image search system that knows when to say “I don’t know” and hands the case to a clinician.
The problem
Content-based image retrieval can support diagnosis by surfacing similar past cases. But a single network architecture captures only part of what medical images look like, and standard networks return confidently wrong answers on ambiguous inputs.
The design principle: in a clinic, withholding a recommendation is far safer than giving a wrong one.
What we built
- 01Three backbones
EfficientNet-B0, ViT-Tiny and SE-ResNet34 encode complementary features.
- 02Cross-modal attention fusion
Multi-head self-attention makes the backbones cross-reference each other.
- 03PPO distance weighting
A reinforcement-learning agent tunes fusion weights per query.
- 04Monte Carlo dropout + BALD
Estimates epistemic uncertainty for every query.
- 05Defer to a clinician
Uncertain queries go to a human instead of getting a forced answer.
Results
Values are close together, so the axis starts at 0.95. Deferral trades coverage for precision.
What we found
Naive fusion is unstable
Concatenating or averaging backbone features is highly sensitive to the random seed. Attention fusion brings cross-seed variance down to 0.0014.
The RL layer is scoped honestly
The PPO agent is a dataset-specific optimizer that must be retrained per corpus, not a zero-shot generaliser, and the paper says so.
Uncertainty doubles as an intruder alert
Epistemic uncertainty rises by about 60% on out-of-distribution inputs, so unfamiliar or random images are caught and deferred.
The most useful thing a clinical model can do is know its limits. Precision climbs steadily as the system is allowed to defer.
Finding the right sensors and network settings for jet-engine failure prediction, with far fewer training runs.