"Expected all tensors to be on the same device" with SFTTrainer

[John6666]

It seems that this error may occur depending on the version of Transoformers. Of course, there are other possibilities…

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cuda:0!

RuntimeError Traceback (most recent call last) Cell In[29], line 2 1 # Train model ----> 2 trainer.train() 4 # # Start training from the last checkpoint 5 # trainer.train(resume_from_checkpoint=checkpoint) File ~/anaconda3/envs/python3/lib/python3.10/site-packages/transformers/trainer.py:2245, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs) 2243 hf_hub_utils.enable_progress_bars() 2244 else: → 2245 return i…

BitsandBytes conflict with Accelerate

I’m running inference on a custom VLM derived model. Inference works fine when using the weights in their bfloat16 precision. However, when I try defining a BitsandBytes config, I receive errors that I suspect is due to conflicts between BitsandBytes and Accelerate, where Accelerate and BitsandBytes are both trying to set the compute device and hence generating the following stack trace. Traceback (most recent call last): File "/home/tyr/RobotAI/openvla/scripts/extern/verify_prismatic.py", l…