Changes between Initial Version and Version 1 of en/AdvancedNlpCourse2020/LanguageModelling

Aug 31, 2021, 2:11:49 PM (10 months ago)
Ales Horak

copied from private/AdvancedNlpCourse/LanguageModelling


  • en/AdvancedNlpCourse2020/LanguageModelling

    v1 v1  
     1= Language modelling =
     3[[|IA161]] [[en/AdvancedNlpCourse|Advanced NLP Course]], Course Guarantee: Aleš Horák
     5Prepared by: Pavel Rychlý
     7== State of the Art ==
     9The goal of a language model is to assign a score to any possible input sentence. In the past, this was achieved mainly by n-gram models known since WWII. But recently, the buzzword deep learning penetrated also into language modelling and it turned out to be substantially better than Markov's n-gram models.
     11The current state of the art models are build on neural networks using transformers.
     14=== References ===
     15 1. Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". [[|arXiv:1810.04805v2]]
     16 1. Polosukhin, Illia, et al. "Attention Is All You Need". [[|arXiv:1706.03762]]
     17 1. Alammar, Jay (2018). The Illustrated Transformer [Blog post]. Retrieved from
     18 1. Alammar, Jay (2018). The Illustrated BERT, ELMo, and co. [Blog post]. Retrieved from
     24== Practical Session ==
     26=== Technical Requirements ===
     28The task will proceed using Python notebook run in web browser in the Google [[|Colaboratory]] environment.
     30In case of running the codes in a local environment, the requirements are Python 3.6+, jupyter notebook,
     31the main module `huggingface/transformers` is installed at the beginning of the notebook.
     37=== BERT-like language model from scratch ===
     39In this workshop, we create a BERT-like language model for Czech from own texts.
     40We investigate tokenization of such models and experiment with ''fill mask'' task
     41for learning and evaluating neural language models.
     43Access the [[|Python notebook in the Google Colab environment]]. Do not forget to save your work if you want to see your changes later, leaving the browser will throw away all changes!
     47download the notebook or plain python file from the shared notebook (File > Download) and run in your local environment.
     51=== Training data ===
     531. Small text for fast setup: RUR from Project Gutenberg
     551. Sample from Czech part of the Europarl corpus, (1 MB, 10 MB, 150 MB)
     59=== Task ===
     61Change the training data, tune parameters (vocab size, training args, ...) to get
     62reasonable answer to simple ''fill mask'' questions, for example:
     64fill_mask("směrnice je určena členským <mask>")
     67=== Upload ===
     68Upload your modified notebook or python script with results to the [[|homework vault (odevzdávárna)]].