The sticker shock that made me rethink GPU renting

I’ve been fine-tuning 7B and 13B parameter models for client work for about two years now. My first mistake was renting an 8x H100 node from a big cloud provider and letting it run idle for three days while I debugged a data pipeline. That bill came to $1,872. For nothing. The node sat at 3% utilization for 72 hours. I could have bought a used 4090 for that money. After that, I started treating GPU rental like buying airplane tickets — same destination, wildly different prices depending on where and when you look.

The raw numbers matter more than most tutorials admit. A single H100 on demand runs $2.50 to $4.50 per hour at hyperscalers, but I’ve seen the same card for $1.85 on secondary markets. An A100 80GB is typically $1.20 to $2.00 per hour. L40S, which is my go-to for LoRA and QLoRA on 7B models, sits around $0.60 to $1.10. H200s are newer and pricier — usually $3.50 to $5.50 per hour, and honestly, for most fine-tuning jobs, you don’t need that extra memory bandwidth unless you’re doing full-parameter training on 70B+ models)Skip the H200 unless you’re doing full-parameter 70B work. For 95% of my jobs, A100s are the sweet spot.

What I actually pay for different fine-tuning jobs

Let me give you real numbers from my last three projects. A Mistral 7B QLoRA with a 50k row dataset, 3 epochs, sequence length 2048, took 2 hours and 40 minutes on a single L40S. At $0.85/hour, that’s $2.27. The same job on an A100 80GB took 1 hour and 15 minutes at $1.50/hour — that’s $1.88. So the A100 was actually cheaper per run despite the higher hourly rate, because it finished faster. That’s the math most people skip.

For a Llama 3 8B full fine-tune (not LoRA), I needed 4x A100s for gradient checkpointing to fit a batch size of 16. That ran 6 hours at $1.40/hour per card = $33.60 total. On the same cluster, an 8x H100 node would have done it in 2.5 hours but at $3.20/hour per card = $64. So I saved 47% just by choosing A100s and waiting a bit longer.

Here’s a quick breakdown of what I’ve seen work well:

• 7B QLoRA → 1x L40S or 1x A100 80GB. Budget $2–$6 per run.
• 13B LoRA → 2x A100 80GB. Budget $8–$15 per run.
• 70B QLoRA → 8x A100 80GB. Budget $60–$120 per run.
• Full fine-tune 7B → 4x A100. Budget $30–$70 per run.

The key is to benchmark one epoch on a small sample first. I always run a 200-row test batch to estimate throughput, then multiply by the full dataset size. This has saved me from renting a 8x H100 node for a job that would have finished on a single L40S.

The hidden costs that eat your budget

Hourly rates are only half the story. The real killers are idle time, storage egress, and setup overhead. I once paid $0.15/GB to download a 40GB model checkpoint from a provider’s storage bucket. That’s $6 just to move files out. Another time, I rented a node, spent 20 minutes installing CUDA and dependencies, and the clock was running the whole time. On a $2/hour card, that’s $0.67 wasted before training even started.

To avoid this, I now do three things religiously:

1. Pre-build my Docker image with all dependencies and upload it to the provider’s registry. This cuts setup time from 20 minutes to 3.

2. Use spot or interruptible instances for anything that can tolerate a restart. I’ve gotten L40S cards for $0.38/hour on spot — that’s 60% off the on-demand price. The tradeoff is you might lose your progress, so I checkpoint every 10 minutes.

3. Check egress fees before renting. Some providers charge $0.10–$0.20/GB for downloads. If I’m pulling a 70B model, that’s 140GB of weights — suddenly $14–$28 in invisible costs.

I also stopped renting multi-GPU nodes by default. For a 7B LoRA, one L40S is enough. For a 13B LoRA, two A100s are fine. Most people overprovision because they think they need “bigger” — but memory-efficient attention and 4-bit quantization have made single-GPU fine-tuning totally viable for most open models.

Where I actually rent and how I compare prices

I use three main sources: the big three clouds (AWS, GCP, Azure), dedicated GPU rental startups, and community marketplaces. The hyperscalers are reliable but overpriced — you’re paying for their support and uptime SLAs. The startups (think RunPod, Lambda, Vast.ai) undercut them significantly because they aggregate idle GPUs from smaller data centers and individual owners.

My current workflow is embarrassingly simple: I check a price comparison site that pulls live rates from multiple providers. I filter by GPU model, RAM, and region. I sort by price per hour, then look at the top 5 results. I check the provider’s reputation score and whether they have a “guaranteed” vs “spot” option. Then I rent for 1 hour first, run a quick bandwidth test (I download a 1GB file and time it), and if it’s under 30 seconds, I commit to the longer job.

For example, last week I needed a single A100 80GB for a 3-hour job. The comparison site showed prices ranging from $1.10/hour (a small provider in the Netherlands) to $2.30/hour (a big cloud in Virginia). I picked the Dutch one, paid $3.30 total, and the job ran flawlessly. The same job on the big cloud would have been $6.90. That’s a 52% saving for exactly the same hardware.

I’ve also started using Vast.ai more often because their marketplace has the widest selection of older cards like A100s and L40S at rock-bottom prices. I found a 4x A100 80GB node for $1.05/hour per card — that’s $4.20/hour total for four cards. I ran a 13B fine-tune on it for 5 hours and paid $21. On Lambda, the same node would have been $1.70/hour per card = $34. I’m not saying Vast.ai is perfect — you have to check the host’s uptime stats and read reviews — but for my money, it’s been the most consistent way to cut fine-tuning costs by 40–60%.

My final rule of thumb

Stop renting by the hour. Rent by the job. Before you spin up anything, estimate your total compute time using a 200-row benchmark. Multiply that by the cheapest per-hour rate you can find for the GPU you need. Add 10% for setup overhead and 5% for egress. If that number is more than $50, you should consider whether you can use QLoRA instead of full fine-tuning — the quality difference is often negligible for instruction tuning.

The one thing I never do anymore is rent an H100 for a 7B model. It’s like renting a Ferrari to drive to the corner store. An L40S or A100 will do the job for a third of the price. And if you’re fine-tuning models regularly, the savings compound fast. My last month of fine-tuning work cost $214 total across 14 jobs. The same jobs on AWS would have been $580. That’s $366 saved — enough to buy a nice SSD or just pocket it.

So here’s my honest advice: check a price comparison tool before every rental. Set a max hourly rate for your target GPU. And if you see an A100 80GB for under $1.20/hour, grab it — that’s a good deal. For the widest selection of cheap cards, I keep coming back to Vast.ai because their marketplace is where the idle GPUs end up. Just read the host reviews, check the bandwidth, and you’ll be fine-tuning LLMs without overpaying, just like I do.