{"id":31893,"date":"2021-10-12T09:11:21","date_gmt":"2021-10-12T08:11:21","guid":{"rendered":"https:\/\/www.inovex.de\/?p=31893"},"modified":"2023-06-06T08:05:00","modified_gmt":"2023-06-06T06:05:00","slug":"pretraining-language-models","status":"publish","type":"post","link":"https:\/\/www.inovex.de\/de\/blog\/pretraining-language-models\/","title":{"rendered":"Pretraining Language Models: Quality Over Quantity?"},"content":{"rendered":"

Transformer-based language models trained on massive datasets, such as Google\u2019s BERT, have undeniably pushed the frontier of natural language processing (NLP) in recent years. Due to the heterogeneous nature of the training data, the models improve when shown supplementary knowledge during pretraining.<\/p>\n

Pretraining algorithms often call for large datasets \u2013 the bigger the better. But what if only a limited amount of data is available for pretraining? Is it beneficial to focus on quality instead of quantity? In this article, we investigate whether it is possible to invest in small, high-quality datasets for pretraining language models instead of relying on large, more general corpora.<\/p>\n

 <\/p>\n

\n

<\/p>\n<\/div>