Background I have followed this amazing blog Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers on fine tuning whisper on my dataset and the performance is decent! However, as my dataset is in Bahasa Indonesia and my use case would be to use to as helpline phone chatbot where the users would…

Hey @notha99y ! Cool to see that you’ve been using the blog post! The Whisper model is pre-trained on 96 languages, Indonesian being one of them. This means that the pre-trained tokenizer already has all of the Indonesian words you need! I would recommend that you leverage this pre-trained tokenizer…

Hi @sanchit-gandhi thank you for writing back! And once again thank you for the amazing blog post. I did try to instantiate the processor with the correct language as stated in your blog, but there are occasions where Whisper transcribed my audio into Japanese. Haha. Indeed, changing the tokenizer…

Perfect! All the best with your training runs :hugs:

If i Have my own dataset in that dataset contains medical terms and its audio. In that case i want to create my own model for Speech to text conversion. so how can i approach this problem. or is there any specific dataset is already in hugging face hub.

Hey @Ajayagnes ! Welcome to the HF community and thanks for posting this awesome question :hugs: It should be possible to fine-tune the Whisper model on your own dataset for medical audio/text. You can follow the steps outlined in this blog post: Fine-Tune Whisper For Multilingual ASR with 🤗 Transfor…

Thank you for the reply sir. I’ll do it with the above mentioned tips

@sanchit-gandhi can you please explain why do we need to add audio dataset in HF? Does hugging face preprocess audio and text in any format to fine tune the model or we can maintain the audio and text in local and fine tune? Is there any standard format such as pitch, speed for audio and format suc…

@sanchit-gandhi I am using my custom dataset . Now , while running [trainer.train()] , i am getting error as [ RuntimeError: The size of tensor a (517) must match the size of tensor b (448) at non-singleton dimension 1] Please note I am using whisper medium model as seen below : from transformers…

Thank you this was helpful . @sanchit-gandhi : One more question - how do i save the model as pickel file “pytorch_model.bin”. When i push the model to the repo using “trainer.push_to_hub(**kwargs)” [ex:CKSINGH/whisper-small-hi-firefox] , i dont see the pickel file pushed along side . How , i ca…

Fine Tuning Whisper on my own Dataset with a customized Tokenizer

Beginners

sanchit-gandhi November 17, 2023, 6:38pm 10

Looks to be the same issue as: [Open-to-the-community] Whisper fine-tuning event - #21 by sanchit-gandhi

Topic		Replies	Views
Whisper is not learning a new tokenizer, even when i make test and train dataset the same Beginners	0	472	November 20, 2023
Whisper model fine tuning Models	7	2476	June 8, 2024
How to finetune whisper model 🤗Transformers	0	630	May 7, 2023
[Open-to-the-community] Whisper fine-tuning event Community Calls	31	12426	December 10, 2023
How to finetune Whisper with language which is not supported in WhisperTokenizer Beginners	4	908	May 18, 2024

Fine Tuning Whisper on my own Dataset with a customized Tokenizer

Related topics