Guide
How to size a GPU cluster
A method to estimate the number of GPUs and nodes for training, fine-tuning or inference before you ask for an offer.
By Tera Compute · GPU rental and colocation
The correct cluster size is the smallest one that does your work in the time that you have. Start from the workload, turn it into GPU memory and GPU-hours, then turn those into nodes. An estimate is enough to start. A short test confirms it.
How much GPU memory does the model need?
The weights of a model need memory in proportion to the number of parameters. At 16-bit precision each parameter takes 2 bytes, so a model with 70 billion parameters needs about 140 GB for its weights alone. Inference needs more for the context that it processes. Training needs several times more for gradients, optimizer state and activations.
Divide the memory that you need by the memory of one GPU. The result is the minimum number of GPUs for one copy of the model. B300, B200, H200 and H100 compared has the memory of each model.
How many GPUs for inference?
Measure how many requests or tokens per second one copy of the model serves at a response time that you accept. Divide your peak load by that figure, then add spare capacity for failures and growth. Size for the peak, not for the average.
How many GPUs for training or fine-tuning?
Estimate the total GPU-hours: take the time of a short test run and scale it to the full dataset and the number of passes. Divide by the time that you can wait. More GPUs shorten the run, but not in exact proportion, because the nodes spend time on communication. A job that runs across nodes needs a fast connection between them. See InfiniBand vs. RoCE for a GPU cluster.
How do GPUs become nodes?
Dedicated capacity is rented in whole nodes. The nodes that Tera rents have 8 GPUs each, so round your GPU count up to a multiple of 8. Four nodes are 32 GPUs.
What else should you size?
- Storage: datasets, checkpoints and model files, and how fast they must load.
- CPU and system memory: data loading and preparation can hold back the GPUs.
- Network: the connection between nodes, and the connection to your users and your data.
- Spare capacity: what happens to your work when one node is out for repair.
The GPU rental cost calculator turns a number of nodes into a contract total. To talk through the sizing, schedule a call.
About Tera Compute. Tera Compute provides GPU rentals to startups: dedicated bare-metal nodes for AI teams. Tera also finds colocation for the GPUs that a startup owns. The company is based in the United States and is backed by Alliance. It is built by an operating team with deep experience in industrial-scale compute and data centers. Learn more at tera.gw.