Changes between Version 3 and Version 4 of private/NlpInPracticeCourse/LanguageModelling
- Timestamp:
- Aug 24, 2015, 8:55:54 AM (8 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
private/NlpInPracticeCourse/LanguageModelling
v3 v4 5 5 Prepared by: Vít Baisa 6 6 7 == State of the Art = 7 == State of the Art == 8 8 9 The goal of language model is to a) predict a following word or phrase based on a given text history and b) assign a probability (=score) to any possible input sentence. This was done mainly by n-gram models known since WWII. But recently, the buzzword deep learning penetrated also into language modelling and it turned out neural networks beat classicn-gram models.9 The goal of a language model is a) to predict a following word or phrase based on a given history and b) to assign a probability (= score) to any possible input sentence. In the past, this was achieved mainly by n-gram models known since WWII. But recently, the buzzword deep learning penetrated also into language modelling and it turned out to be substantially better than Markov's n-gram models. 10 10 11 11 === References === 12 13 Approx 3 current papers (preferably from best NLP conferences/journals, eg. [[https://www.aclweb.org/anthology/|ACL Anthology]]) that will be used as a source for the one-hour lecture:14 12 15 13 1. Bengio, Yoshua, et al. "A neural probabilistic language model." The Journal of Machine Learning Research 3 (2003): 1137-1155. … … 21 19 22 20 We will build a simple language model (skip-gram) which has very interesting properties. When trained properly, the vectors of words obey simple space arithmetics, e.g. 23 vector "king" − vector "man" + vector "woman" ~= vector of"queen".24 We will train this model on a large Czech and English corpora and evaluate the result .21 vector "king" − vector "man" + vector "woman" ~= vector "queen". 22 We will train this model on a large Czech and English corpora and evaluate the results.