So far, I’ve been using pre-trained models. For my task, it seems that I am required to perform pre-training on GLUE task just to see how it performs. I wanted to confirm what modifications need to be done to do this ? I’m not sure about using the same tokenizer. I want to randomly initialize it an…

You can initialize a model without pre-trained weights using from transformers import BertConfig, BertForSequenceClassification # either load pre-trained config config = BertConfig.from_pretrained("bert-base-cased") # or instantiate yourself config = BertConfig( vocab_size=2048, max_posit…

Thanks for replying. Okay, so the process is mostly same. Do we need to make any changes to tokenizer ? Saw few posts where people had encountered issues during pre-training, so thought of confirming this.

Also any training tips when not using pre-trained weights ?

If you train the tokenizer from scratch as well, then make sure to change the vocab size in config accordingly

I haven’t done so myself for this task so can’t say much, but probably start with higher LR than the default one in the Trainer (which is 5e-5) as we are training from scratch, experiment with LR schedule, haparms search will definitely help choose the right params. @sgugger might have better tips …

Is there any problem if we use AutoTokenizer.from_pretrained() ? That’s where I am unsure. Or do we use custom tokenizer ? What’s the recommended way of proceeding.

You can use pre-trained tokenizer, it shouldn’t cause any issues. And IMO using pre trained tokenizer makes sense than training from scratch on limited data

With the standard way, BERT-base is training, BERT-large doesn’t seem to respond. I seem to be missing something in regards to training dynamics.

is there any exception , error ?

Just to add to this, training your own tokenizer is mainly useful if you are working with a specific genre, domain, and/or language.

Tips for PreTraining BERT from scratch

🤗Transformers

prajjwal1 September 24, 2020, 2:31pm 17

I’ve opened a new issue which is about pre-training. Training on GLUE part is resolved. Thanks for asking.

Topic		Replies	Views
Pre-training BERT Models	1	415	May 21, 2024
Pre-Train BERT from scratch 🤗Transformers	5	16707	May 30, 2023
How can I pretrain a new model re-initializing with my own vocab? 🤗Transformers	0	314	May 25, 2021
How to "further pretrain" a tokenizer (do I need to do so?) 🤗Tokenizers	5	4539	February 20, 2022
Pre-training a BERT model from scratch with custom tokenizer Intermediate	5	3214	January 11, 2022

Tips for PreTraining BERT from scratch

Related topics