Word Level Analysis
Motivation
Many applications need a tool for “clustering” of word forms appearing in texts:
- chladniček
- chladničky
- chladničkách <=> chladnička
- chladničce
- ...
Usage:
- Indexing, searching, keyword extraction, ...
- And almost all NLP tools
Word Level Processing Data for Czech
For almost 12 M word forms (incl. colloquial forms):
- lemma (canonical form, dictionary form)
- grammatical information: part of speech, number, case etc.
Word form stroj has 3 interpretations:
- lemma stroj, nominative
- lemma stroj, accusative
- noun, masculine animated, singular
- lemma strojit
- verb, 2nd person, singular, imperative mood
Possible Applications
Various types of analyses:
- word form => lemma (many types of searching/indexation)
- nebral => brát/nebrat (úplatky)
- nejstaršího => nejstarší/starý (člověk)
- chladnička => chladničky (as a class)
- bavlna => bavlněný (word derivation)
- word form/lemma + gram. info. => word form
- e.g. salutation generation: pane Procházko
- word form/lemma => all word forms
- word form => lemma + full/partial grammatical information
The analysis is very fast - approx. 1 million word forms per second
Processing Unknown Words
Some word forms in processed texts are unknown:
- terms polydaktylie, neologisms klausoviny, typos bizardního, colloquial words plaťáky, etc.
An ending of the word form is able to determine e.g.
- lemma: klausoviny => klausovina
- grammatical information: bizardního => genitive, etc.
- derivational relations: plaťáky => plaťákový
Texts from a particular domain allows grouping of unknown word forms:
- polydaktylie, polydaktiliích, polydaktylií, ... <=> polydaktylie
- => extension of data or more precise “guessing”
Resolving Ambiguities Using Context
An extreme case Stroj ženu holí.
- Já stroj ženu holí, ty stroj ženu holí, ten stroj ženu holí.
Usual case is e.g. stát
- noun: Stát jsem já.
- verb: Celá továrna musela hodinu stát.
- at the part of speech level, it is a bigger problem for English
The context of the word determines its interpretation
- rules and/or statistical data describe typical contexts of nouns, verbs, etc.
- using such information one can tell that stát is noun/verb
Example of Contexts — Word Sketches
Spellchecking and Diacritics Restoration
Data also allow spellchecking and diacritics restoration:
Universality
All the mentioned processes can be
- tuned for a specific domain
- using texts from this domain
- applied to a language other than Czech
- (Slovak, Polish, German, English, ...)
Latest Applications
Seznam.cz, Yandex.ru, Aukro.cz, Václav Havel Library
- indexing and searching
Information System of Masaryk University
- other universities and schools (FHS UK, JAMU, VŠFS, ...)
- affiliate projects (theses.cz, odevzdej.cz, repozitar.cz)
- indexing, searching and plagiarism detection
“Internetová jazyková příručka”
- online source on Czech orthography and grammar
- NLP Centre data were a starting point for word form tables
Conclusions
Word level processing of texts allows:
- various types of base word determining which forms are to be grouped together
- ambiguity resolution according to the context
- word form generation
- spellchecking, diacritics restoration
The tools/data can be domain specific and for various languages