Indexing and Searching Very Large Texts

​IA161 NLP in Practice Course, Course Guarantee: Aleš Horák

Prepared by: Miloš Jakubíček

State of the Art

References

1. RYCHLÝ, Pavel, et al. Korpusové manažery a~ jejich efektivní implementace. 2000.
2. JAKUBÍCEK, Miloš; KILGARRIFF, Adam; RYCHLÝ, Pavel. Effective Corpus Virtualization. In: Challenges in the Management of Large Corpora (CMLC-2) Workshop Programme. p. 7.
3. JAKUBICEK, Milos, et al. Fast Syntactic Searching in Very Large Corpora for Many Languages. In: PACLIC. 2010. p. 741-747.

Practical Session

Compare search through (A) plain text using grep, (B) an indexed corpus using Manatee, (C) a corpus indexed in an arbitrary SQL database Use vertical text for BNC available at aurora:/corpora/vert/bnc/bnc.vert.xz or download the compressed file via this URL (224 MB) or uncompressed here (2 GB).

Search for the phrase "test case", display context of 10 words before and after each occurrence of the search phrase.

(A) plain

Hint: use grep -C to display context.

(B)

Corpus is already indexed on Manatee, try:

time corpquery bnc '[word="test"] [word="case"]'

See ​https://www.sketchengine.eu/documentation/corpus-querying/

(C)

Use your favourite SQL database, you can use e.g. sqlite3. Hint how to import vertical text:

​https://stackoverflow.com/questions/26065872/how-to-import-a-tsv-file-with-sqlite3

(D) bonus task: use SQLite3 with FTS extension and a custom FTS tokenizer to index all attributes (word, lemma, tag).

For (A), (B) and (C), submit the commands you used and how long the search took to evaluate.