Lightweight CNNs for Musculoskeletal X-ray Classification
The problem
Radiologists spend a few minutes, sometimes seconds, per image on musculoskeletal X-rays, and fatigue-driven misreads are a real contributor to diagnostic error. AI models can help, but most of the strong ones (DenseNet-169 and similar) are large: tens of millions of parameters, GPU-dependent, impractical for the clinics that would benefit most, the low-resource ones with the fewest radiologists per patient.
Our team set out to see how much of that performance we could keep while shrinking the model down to something that could plausibly run on constrained hardware.
Approach
We worked from the MURA dataset (Stanford ML Group), roughly 40,000 studies across seven upper-extremity body parts, labeled normal or abnormal. As a four-person team, we benchmarked established architectures, ResNet, VGG, DenseNet, in both pretrained and from-scratch configurations, then built a family of custom, lightweight CNNs designed specifically for the tradeoff between accuracy and model size.
We evaluated everything on Cohen’s Kappa rather than raw accuracy, since inter-rater agreement is the standard radiologists and regulators actually care about, and used Grad-CAM to check that the models were attending to clinically relevant regions rather than learning shortcuts.
What I built
My focus was the custom CNN track and the infrastructure the whole team relied on to evaluate it:
Data and metrics pipeline. Built the preprocessing utilities (data loading, transforms for training and validation) and the metrics module the team used throughout, confusion matrices, ROC curves, class weights, and Cohen’s Kappa with confidence intervals.
Interpretability tooling. Built the Grad-CAM visualization pipeline, including the logic to locate the last convolutional layer of an arbitrary architecture and run it against specific batches, body parts, or individual images. This is what let us confirm a model was learning fracture-relevant features rather than incidental image artifacts.
Three custom architectures.
- A baseline custom CNN (stem, middle, and final convolutional blocks feeding into global average pooling and a linear classifier), sized at roughly 1.3M parameters against DenseNet-169’s 12M.
- An attention-enhanced variant using depthwise separable convolutions and squeeze-and-excitation blocks to cut compute further while reweighting feature maps toward clinically relevant regions, down to about 55,000 parameters.
- A lightweight body-part-specific classifier used as the basis for the ensemble approach below.
Ensemble strategy. Built the notebook that combined body-part-specific models via weight transfer into the main classifier, which became our best-performing configuration.
Dynamic class weighting. Wrote the function that computed class weights on the fly to handle the dataset’s imbalance across body parts, which ranged from roughly 1:1 to nearly 3:1 depending on the joint.
Training and experimentation. Ran the training and hyperparameter sweeps across the custom DenseNet, custom CNN variants, the lightweight classifier, and the ensemble, optimizing jointly for Cohen’s Kappa and parameter count.
Key tradeoffs
The central tension in this project was accuracy versus deployability, and the results made that tradeoff concrete rather than theoretical:
- Pretrained DenseNet-169 hit the highest Cohen’s Kappa in the study (0.659) but at 12M parameters, effectively ruling it out for edge deployment.
- Our baseline custom CNN reached 0.525 at 1.3M parameters, roughly a tenth of the size for a meaningful fraction of the agreement.
- The ensemble of body-part-specific models, built through weight transfer rather than a heavier architecture, pushed that up to 0.589, within striking distance of DenseNet-169’s ceiling, without the deployment cost.
- The attention-enhanced variant traded some raw performance (0.468) for a roughly 24x reduction in parameters versus the baseline, which is the right tradeoff if the constraint is storage or edge inference rather than raw accuracy.
None of the custom models beat DenseNet-169 outright. That wasn’t the goal. The goal was closing most of the gap at a fraction of the size, and the ensemble result is where that showed up most clearly.
Lessons learned
Working from scratch versus fine-tuning pretrained weights mattered more than architecture choice in a lot of our experiments, pretrained models consistently outperformed their from-scratch counterparts, which says as much about ImageNet transfer learning’s reach into medical imaging as it does about any specific architecture.
The bigger lesson was about what “lightweight” actually buys you. A smaller model isn’t just cheaper to run, it’s also easier to interpret and iterate on. The Grad-CAM tooling I built made it fast to sanity-check whether a much smaller model was learning the right thing, not just producing a plausible number, and that mattered more to the team’s confidence in the ensemble result than the Kappa score alone would have.
Team and code
Built with three teammates as coursework for CS 7643 (Deep Learning) at Georgia Tech, Fall 2024. This write-up covers my individual contribution; the full team’s work, including the DenseNet and VGG benchmarking, dataset engineering, and repository infrastructure, is documented in the linked repository and paper.