Version 24 (modified by pary, 2 months ago) (diff)


Language modelling

IA161 Advanced NLP Course, Course Guarantee: Aleš Horák

Prepared by: Pavel Rychlý

State of the Art

The goal of a language model is to assign a score to any possible input sentence. In the past, this was achieved mainly by n-gram models known since WWII. But recently, the buzzword deep learning penetrated also into language modelling and it turned out to be substantially better than Markov's n-gram models.

The current state of the art models are build on neural networks using transformers.


  1. Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". arXiv:1810.04805v2
  2. Polosukhin, Illia, et al. "Attention Is All You Need". arXiv:1706.03762
  3. Alammar, Jay (2018). The Illustrated Transformer [Blog post]. Retrieved from
  4. Alammar, Jay (2018). The Illustrated BERT, ELMo, and co. [Blog post]. Retrieved from

Practical Session

Technical Requirements

The task will proceed using Python notebook run in web browser in the Google Colaboratory environment.

In case of running the codes in a local environment, the requirements are Python 3.6+, jupyter notebook.

Language models from scratch

In this workshop, we create two language models for English and/or Czech from own texts. The models do not use any framework or complex library, only NumPy? to work with vectors and matrices.

We generate random text using these models and/or use the model in a real application of diacritics restoration.

Access the Python notebook in the Google Colab environment. Do not forget to save your work if you want to see your changes later, leaving the browser will throw away all changes!


download the notebook or plain python file from the shared notebook (File > Download) and run in your local environment.

Training data

  1. Small text for fast setup: *1984 book* from Project Gutenberg

  1. Sample from Czech part of the Europarl corpus, (1 MB, 10 MB, 150 MB),,


Choose one of the following tasks.

Task 1

Use a LM for a diacritics restoration function.

Write a function with text without diacritics as input and same text with added diacritics as a return value. For example:

>>> add_dia('cerveny kriz')
'červený kříž'

You can compare your results with the czaccent service.

Task 2

Generate text using neural LM.

Write a function to generate random text using neural LM. Optional parameter is the start of the text. The text could be printed or returned as the result of the function.

The function could work in the similar way as generate_text from the notebook, but it has to use the *neural* language model.


Upload your modified notebook or python script with results to the homework vault (odevzdávárna).