How do I load a trained checkpoint model?

[John6666]

I think that method is correct, but there seem to be reports of tensor mismatch issues when training with FSDP.

Key errors when trying to load an accelerate-FSDP model checkpoint

Hi, I have been training a pythia-2.8b model, using the DPOTrainer in Huggingface TRL. I am training a LORA adaptor on top of the reference SFT model. In order to speed it up and address memory bottlenecks, I used fsdp and launched my code using accelerate launch --config_file=fsdp.yaml --num_processes=4 run_dpo.py After training, I observed my checkpoints to be of the following form optimizer_0/ pytorch_model_fsdp_0/ rng_state_0.pth rng_state_1.pth scheduler.pt trainer_…

How to load a checkpoint model with SHARDED_STATE_DICT?

How to load a checkpoint model with SHARDED_STATE_DICT? I have a checkpoint which is place in a folder pytorch_model_0, which contains multiple distcp files.

Serialization