I was wondering if this parameter is only set for inference as the document states ( Handling big models for inference ) or does it actually have an effect during training? Thanks!

It loads a model onto multiple GPUs. Once loaded, the model can be run forward or backward. I have only used ”auto” for training as of yet, and it works. If you refer to this section: This only supports the inference of your model, not training. Most of the computation happens behind torch.no_gra…

Thanks @hansekbrand that is helpful. With more search, I think it became clear to me that device_map=“auto” is doing naive MP for training (ref: Make all Transformer models compatible with model parallelism · Issue #22561 · huggingface/transformers · GitHub ).

[image] Kenkentron: is doing naive MP for training Correct, any form of distributed training aside MP is not supported, and as of the next version will raise a proper error if you try to do so: Expand device-map warning hu…

training mode does not support device_map=“auto”, it will throw an error suggesting you load model on single device.

Device_map="auto"

Beginners

Kenkentron August 9, 2023, 5:04pm 5

Thanks @muellerzr, could you also take a look at a related problem I have ZeRO uses more RAM than DDP??

Topic		Replies	Views
Using device_map='auto' for training 🤗Accelerate	5	39689	January 24, 2025
Infer_auto_device_map returns empty 🤗Accelerate	2	3420	March 15, 2023
Trainer API for Model Parallelism using AutoModelForQuestionAnswering 🤗Transformers	1	202	June 5, 2024
Would PyTorch's FSDP work with a model loaded using device_map='auto'? 🤗Transformers	0	299	April 17, 2024
How to load model on multiple GPUs for inference? Beginners	0	817	September 28, 2023

Device_map="auto"

Related topics