Changes between Version 1 and Version 2 of private/NlpInPracticeCourse/LanguageModelling

Timestamp:

Jun 1, 2015, 9:36:52 AM (9 years ago)

Author:

Vít Baisa

Comment:

--

private/NlpInPracticeCourse/LanguageModelling

@@ -5 +5 @@
 Prepared by: Vít Baisa
 
-== TODO til 31.5.2015 ==
+== State of the Art =
 
- 1. choose particular papers for [[#References|References]] below (that will serve as input for the lecture later on)
- 1. prepare the [[#PracticalSession|Practical Session]]
-
-== State of the Art ==
+The goal of language model is to a) predict a following word or phrase based on a given text history and b) assign a probability (=score) to any possible input sentence. This was done mainly by n-gram models known from WWII. But recently, buzzword deep learning penetrated also into language modelling and it turned out neural networks beat classic n-gram models substantially.
 
 === References ===
@@ -16 +13 @@
 Approx 3 current papers (preferably from best NLP conferences/journals, eg. [[https://www.aclweb.org/anthology/|ACL Anthology]]) that will be used as a source for the one-hour lecture:
 
- 1. paper1
- 1. paper2
- 1. paper3
+ 1. Bengio, Yoshua, et al. "A neural probabilistic language model." The Journal of Machine Learning Research 3 (2003): 1137-1155.
+ 1. Mikolov, Tomas, et al. "Efficient estimation of word representations in vector space." arXiv preprint arXiv:1301.3781 (2013).
+ 1. Mikolov, Tomas, et al. "Distributed representations of words and phrases and their compositionality." Advances in Neural Information Processing Systems. 2013.
+ 1. Chelba, Ciprian, et al. "One billion word benchmark for measuring progress in statistical language modeling." arXiv preprint arXiv:1312.3005 (2013).
 
 == Practical Session ==
 
-Concrete description of work assignment for students for the second one-hour part of the lecture. The work will consist of tasks connected with practical implementations of algorithms connected with the current topic (probably not the state-of-the-art algorithms mentioned in the first part) and with real data. Students can test the algorithms, evaluate them and possibly try some short adaptations for various subtasks.
-
-Students can also be required to generate some results of their work and hand them in to prove completing the tasks.
+We will build a simple language model (skip-gram) which has very interesting properties. When trained properly, the vectors of words obey simple space arithmetics, e.g.
+vector "king" − vector "man" + vector "woman" ~= vector of "queen".
+We will train this model on a large Czech and English corpora and evaluate the result.