While analysing the GPU memory footprint of a few models ‘prepared’ via accelerator.prepare(), I found the models occupying more than twice the GPU memory compared to normally loaded models on device. What can be the reason for this inflated memory footprint? I am using a single V100 32 GB GPU and …

Same problem. Did you make any progress in resolving it?

@tongyx361 I stopped passing my model through model.prepare(). I could afford to do this as I was only doing inference but I didn’t find a solution.

Thanks! But how to do distributed inference without prepare()?

Inflated GPU memory footprint of model prepared via accelerate

varadhbhatnagar September 15, 2023, 4:40am 5

Topic		Replies	Views
Data Parallel Multi GPU Inference 🤗Accelerate	9	4892	September 15, 2023
Accelerator OOM 🤗Accelerate	2	1346	July 5, 2023
`Accelerator.prepare` utilize only one GPU instead of all the 8 available GPUs and raises "CUDA out of memory" 🤗Accelerate	3	2951	July 19, 2024
Loading a HF Model in Multiple GPUs and Run Inferences in those GPUs 🤗Accelerate	10	9970	October 16, 2024
Docs Clarification: Is prepare() inefficient for models that are frozen? 🤗Accelerate	0	224	January 22, 2024