Guides
How Much VRAM Do I Need for LLM Work?
Every time I spin up a cloud GPU for a new job, the first question isn't which model to use — it's how much vram do i need. Get that wrong and you're either paying for idle capacity or watching OOM crashes at 2 AM. I've rented A100s, H100s, and a few L40S boxes across Lambda, RunPod, and Vast. The math is predictable, but only if you break down what's actually consuming memory.
Start With the Model Size, Then Add Overhead
VRAM requirements for LLM training aren't just the raw weight size. For a 7B model in FP16, the weights take about 14 GB. But once you add gradients, optimizer states (Adam is two states per parameter), and activations, you're looking at roughly 3–4x the weight size for full fine-tuning. That puts you around 50–60 GB minimum for a 7B model with a modest batch size.
For inference, it's simpler. You need the weights plus KV cache and some activation overhead. A 7B model in FP16 needs about 14 GB for weights, but you'll want at least 16–20 GB total to handle sequence length and batching. That's why a 24 GB RTX 4090 or an L4 works fine for small-scale inference but feels tight the moment you add longer contexts.
What About Quantization?
Quantization changes the math. Running a 7B model in 8-bit drops weights to about 7 GB, and 4-bit gets you under 4 GB. But quantized training is a different beast — you still need full precision gradients or you'll lose quality. For fine-tuning, I usually budget for FP16 or BF16 even if I plan to deploy quantized.
How Much VRAM for Fine Tuning? Real Numbers
The question how much vram for fine tuning depends on your method. Full fine-tuning of a 7B model needs around 60 GB with a batch size of 1 and sequence length of 2048. That's a single A100 80GB or a H100 80GB. If you use LoRA or QLoRA, you can get away with much less — often a single 24 GB GPU for 7B, because you're only training a small set of adapters.
Here's a quick breakdown I use when planning:
| Model Size | Full Fine-Tune (FP16) | LoRA / QLoRA | Inference (FP16) |
|---|---|---|---|
| 7B | ~60 GB | ~16–24 GB | ~16 GB |
| 13B | ~100 GB | ~24–32 GB | ~26 GB |
| 70B | ~350 GB | ~48–80 GB | ~140 GB |
These are ballpark figures, not gospel. Activation memory grows with sequence length and batch size. But they're close enough to decide which GPU class to rent.
VRAM Needed for 7B Model: The Sweet Spot
For most small teams, the vram needed for 7b model is the most common question. If you're doing inference, a 24 GB GPU is comfortable. You can batch requests, run a 4K context, and still have headroom. For fine-tuning, a 24 GB card works with LoRA, but full fine-tuning will make you sweat — you'll need gradient checkpointing and a micro batch size of 1, and even then it might not fit.
I've personally fine-tuned a 7B model with LoRA on a single A10G (24 GB) and it worked. Training took about 3 hours on 10K examples. Full fine-tuning on the same data would have needed an A100 80GB and probably run 5–6 hours. The tradeoff is quality vs. speed, but for most tasks LoRA is fine.
How Much VRAM for 70B Model? Don't Guess
For a 70B model, the question how much vram for 70b model has a different answer depending on task. Inference in FP16 needs about 140 GB for weights alone. That's two A100 80GB or one H100 80GB with heavy offloading — which is slow. In practice, I rent a node with 2–4 GPUs for inference, or use quantization to fit on a single 80GB card.
For fine-tuning a 70B, full fine-tuning is a multi-GPU affair. You're looking at 4x A100 80GB minimum, and even that's tight. With LoRA, you can get away with a single 80GB GPU if you use 4-bit base weights, but the adapter training still needs memory for activations. I've done QLoRA on a 70B with a single A100 80GB and a sequence length of 1024 — it fit, but barely.
VRAM for Inference vs Training: Know the Difference
The biggest mistake I see is confusing vram for inference vs training requirements. Inference is mostly weights plus a small KV cache. Training includes weights, gradients, optimizer states, and activations. That's why a 7B model that runs on a 16 GB GPU for inference might need 60 GB for training.
When you're choosing a cloud GPU, always ask: what's the peak memory during the forward/backward pass? For training, that's often the activation memory, which grows with batch size and sequence length. For inference, it's the KV cache, which grows with batch size and context length. If you're serving a model, you can also trade off batch size for latency to fit in less VRAM.
How to Choose VRAM for Cloud GPU Rentals
When I choose vram for cloud gpu, I don't just look at the model size. I look at the whole pipeline. Do I need to train, fine-tune, or just serve? What's my expected batch size? What sequence length? Do I want to run multiple replicas on one GPU?
Here's a practical checklist I use:
- List the model size and precision (FP16, BF16, INT8, etc.).
- Estimate weights + optimizer states for training (roughly 3–4x weights).
- Add activation memory — usually 20–40% of weights for short sequences, more for long ones.
- For inference, add KV cache: roughly 1–2 GB per 1K tokens for a 7B model, more for larger models.
- Leave 10–20% headroom for framework overhead and fragmentation.
If you're renting, look for GPUs with 24 GB, 48 GB, or 80 GB VRAM. The 24 GB class (RTX 4090, A10G, L4) handles most 7B inference and LoRA fine-tuning. The 48 GB class (A6000, L40S) is great for 13B LoRA or small full fine-tunes. The 80 GB class (A100, H100) is where you live for serious training or 70B inference with quantization.
Compare GPU Classes for Deep Learning
When you're deciding on gpu vram size for deep learning, think in terms of tiers:
- 16–24 GB: Good for 7B inference, LoRA fine-tuning, and small batches. Cost-efficient for prototyping.
- 48 GB: Comfortable for 13B fine-tuning with LoRA, or 7B full fine-tune with gradient checkpointing. Good middle ground.
- 80 GB+: Required for full fine-tuning of 13B+ models, or 70B inference with quantization. This is where multi-GPU setups start to matter.
Don't over-buy. I've seen people rent an H100 to run a 7B inference endpoint that would've worked fine on a 24 GB card. That's just burning money. But under-buying is worse — you'll waste hours debugging OOMs and re-running jobs.
Final Thoughts: Match VRAM to Your Workload
The short answer to how much vram do i need is: enough to hold your model plus the overhead of your method. For 7B inference, 16–24 GB. For 7B fine-tuning, 60 GB full or 24 GB with LoRA. For 70B, plan on multiple 80 GB GPUs or accept heavy quantization. Always estimate before you rent, and check the memory usage of your framework — PyTorch's torch.cuda.memory_summary() is your friend.
I've stopped guessing and started budgeting. Now I know that a 7B LoRA run costs me about $1.50/hour on a 24 GB GPU, and a 70B QLoRA run on an 80 GB card runs me $4–5/hour. That's the kind of clarity that keeps me from overpaying. You don't need the biggest GPU on the market — you need the right one for your workload.