My Experience Fine-Tuning a Vision-Language Model on Low Compute
Fine-tuning a Vision-Language Model (VLM) sounds exciting — until you try doing it with limited data and limited compute.
This is the story of how I fine-tuned a VLM on ~500 usable images, failed multiple times, wasted compute on Colab, misunderstood LoRA, and finally improved performance by 15.6% — after going back to fundamentals. GIT Finetunedmodel paper
The Setup
My task was to fine-tune a VLM for a Societal problem.
I scraped ~1500 images using the Google Street View API. After building a preprocessing pipeline to filter noisy and unusable samples, I was left with roughly 500 clean images.
That's when the real challenge started.
500 images is small for fine-tuning a large multimodal model.
And I made my first mistake.
Mistake #1: Full Fine-Tuning on 500 Images
I attempted full model training.
Yes — I tried updating the entire model with just ~500 images.
What happened?
- Training instability
- Poor generalization
- Overfitting
- Wasted compute on Colab
The model simply did not learn meaningful representations.
This was predictable in hindsight — but I didn't think deeply before running the experiment.
Lesson: Large models don't magically adapt to small datasets.
Discovering LoRA
On my next attempt, I shifted to Low-Rank Adaptation (LoRA) — a parameter-efficient fine-tuning method.
LoRA works by freezing most of the model weights and training only low-rank adapter layers. This reduces memory usage and compute cost significantly.
For low-data and low-compute setups, this is usually the correct approach.
But I made another mistake.
Mistake #2: Treating a VLM Like an LLM
I fine-tuned the VLM as if it were just a Large Language Model.
That was naïve.
A Vision-Language Model has:
- A vision encoder
- A language encoder
- A cross-modal alignment mechanism
I didn't properly configure which parts should be adapted.
As a result:
- Either the visual representations were poorly aligned
- Or the language decoder failed to condition properly on image embeddings
The outputs were inconsistent and often garbage.
At this point, frustration kicked in.
The Turning Point: Breaking It Into Phases
Instead of blindly tuning everything, I stepped back and analyzed how the model generates captions.
Then I tried a structured approach:
- Tune the vision encoder
- Tune the language encoder
- Fine-tune both together
This modular approach gave me better control over what was actually learning.
It improved stability.
But results were still not satisfactory.
That's when I realized something critical.
The Real Problem Was the Data
I had not segmented the images.
The model was trying to caption cluttered, complex street scenes without clear object focus.
So I introduced segmentation using Meta's Segment Anything Model (SAM-2).
Using SAM-2, I:
- Segmented key regions
- Cleaned background noise
- Improved object-level focus
- Refined the preprocessing pipeline
This dramatically improved input quality.
Then I re-ran LoRA-based fine-tuning.
This time:
- Training was stable
- Captions were more aligned with scene content
- Performance improved by 15.6% on my task-specific evaluation
That was the breakthrough.
What I Learned
Fine-tuning is not magic.
It is not:
- "Run training and hope."
- "Add LoRA and expect miracles."
- "Blame the model."
It is:
- Data quality first
- Architecture awareness second
- Training strategy third
If your data is weak, no tuning method will save you.
If you don't understand how the model is structured, you'll tune the wrong parts.
If you don't measure properly, you won't know what improved.
Final Reflection
This journey taught me something simple but powerful: Fine-tuning is controlled trial and error — not blind experimentation.
You fail.
You diagnose.
You adjust.
You repeat.
Low compute forces you to think.
Small data forces you to engineer carefully.
And honestly — that constraint made me better.