Changes between Version 3 and Version 4 of private/AdvancedNlpCourse/LanguageModelling


Ignore:
Timestamp:
Aug 24, 2015 8:55:54 AM (4 years ago)
Author:
Vít Baisa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • private/AdvancedNlpCourse/LanguageModelling

    v3 v4  
    55Prepared by: Vít Baisa
    66
    7 == State of the Art =
     7== State of the Art ==
    88
    9 The goal of language model is to a) predict a following word or phrase based on a given text history and b) assign a probability (=score) to any possible input sentence. This was done mainly by n-gram models known since WWII. But recently, the buzzword deep learning penetrated also into language modelling and it turned out neural networks beat classic n-gram models.
     9The goal of a language model is a) to predict a following word or phrase based on a given history and b) to assign a probability (= score) to any possible input sentence. In the past, this was achieved mainly by n-gram models known since WWII. But recently, the buzzword deep learning penetrated also into language modelling and it turned out to be substantially better than Markov's n-gram models.
    1010
    1111=== References ===
    12 
    13 Approx 3 current papers (preferably from best NLP conferences/journals, eg. [[https://www.aclweb.org/anthology/|ACL Anthology]]) that will be used as a source for the one-hour lecture:
    1412
    1513 1. Bengio, Yoshua, et al. "A neural probabilistic language model." The Journal of Machine Learning Research 3 (2003): 1137-1155.
     
    2119
    2220We will build a simple language model (skip-gram) which has very interesting properties. When trained properly, the vectors of words obey simple space arithmetics, e.g.
    23 vector "king" − vector "man" + vector "woman" ~= vector of "queen".
    24 We will train this model on a large Czech and English corpora and evaluate the result.
     21vector "king" − vector "man" + vector "woman" ~= vector "queen".
     22We will train this model on a large Czech and English corpora and evaluate the results.