Im thinking of using Transformer models to classify other sequential data, namely time series data. My idea is to feed fixed-sized sequences of time series value as input into a BERT-like model with a classification head. Since using pre-trained models probably makes no sense, I would train it from…

you might find some pointers here [image] Using transformers (BERT, RoBERTa) without embedding layer Research I’m looking to train a RoBERTa model on protein sequences, which is in many ways similar to normal nlp training, but in others quite different. In the language of…

@MJimitater I am currently trying to perform a similar experiment. Here were my thoughts: Given the data already comes in a fixed size, I did not perform any tokenization. This is mainly due to the fact that the data I have comes with labels, so I opted not to perform the self-supervised learning…

Using time series for SequenceClassification models

🤗Transformers

valhalla January 20, 2021, 8:02am 2

you might find some pointers here

Topic		Replies	Views
Train BERT on time-series data Models	1	3100	September 7, 2022
Can BERT take numerical values as input for masked time-series modeling? Beginners	1	1019	December 18, 2022
Temporal Information 🤗Transformers	0	326	October 8, 2020
Train model from scratch on own dataset Beginners	0	631	February 26, 2024
Sequence numerical clasification 🤗Transformers	1	998	October 3, 2023

Using time series for SequenceClassification models

Related topics