| 4 | | ONION (ONe Instance ONly) is a tool for removing duplicate parts from large collections of texts. The tool has been implemented in Python, licensed under New BSD License and made an open source software (available for download including the source code at http://code.google.com/p/onion/). It is being successfuly used for cleaning large textual corpora at Natural language processing centre at Faculty of informatics, Masaryk university Brno and it's industry partners. The research leading to this piece of software was published in author's Ph.D. thesis "Removing Boilerplate and Duplicate Content from Web Corpora". The deduplication algorithm is based on comparing n-grams of words of text. The author's algorithm has been shown to be more suitable for textual corpora deduplication than competing algorithms (Broder, Charikar): in addition to detection of identical or very similar (95 %) duplicates, it is able to detect even partially similar duplicates (50 %) still achieving great performace (further described in author's Ph.D. thesis). The unique deduplication capabilities and scalability of the algorithm were been demonstrated while building corpora of American Spanish, Arabic, Czech, French, Japanese, Russian, Tajik, and six Turkic languages consisting --- several TB of text documents were deduplicated resulting in corpora of 70 billions tokens altogether. |
| | 4 | JusText is a heuristic based boilerplate removal tool useful for cleaning documents in large textual corpora. The tool has been implemented in Python, licensed under New BSD License and made an open source software (available for download including the source code at http://code.google.com/p/justext/). It is successfully used for cleaning large textual corpora at Natural language processing centre at Faculty of informatics, Masaryk university Brno and it's industry partners. The research leading to this piece of software was published in author's Ph.D. thesis "Removing Boilerplate and Duplicate Content from Web Corpora". The boilerplate removal algorithm is able to remove most of non-grammatical sentences from a web page like navigation, advertisements, tables, short notes and so on. It has been shown it overperforms or at least keeps up with it's competitors (according to comparison with participants of Cleaneval competition in author's Ph.D. thesis). The precise removal of unwanted content and scalability of the algorithm has been demonstrated while building corpora of American Spanish, Arabic, Czech, French, Japanese, Russian, Tajik, and six Turkic languages consisting --- over 20 TB of HTML pages were processed resulting in corpora of 70 billions tokens altogether. and PRESEMT, Lexical Computing Ltd |
| 6 | | == How to use the tool |
| 7 | | {{{onion [OPTIONS] [FILE]}}} |
| 8 | | |
| 9 | | Mark duplicate text parts in the input vertical file. |
| 10 | | {{{ |
| 11 | | -f FILE hashes of duplicate n-grams |
| 12 | | -n NUM n-gram length (default: 5) |
| 13 | | -t NUM duplicate content threshold (default: 0.5) |
| 14 | | -d STR document tag (default: doc) |
| 15 | | -p STR paragraph tag (default: p) |
| 16 | | -s strip duplicate parts (rather than mark) |
| 17 | | -m no smoothing |
| 18 | | -T NUM trim n-gram hashes to NUM bits (default: 64) |
| 19 | | -l NUM max stub length (default: 20) |
| 20 | | -b NUM buffer size, in bytes (default: 16777216) |
| 21 | | -q quiet; suppress all output except for errors |
| 22 | | -V print version information and exit |
| 23 | | -h display this help and exit |
| 24 | | }}} |
| 25 | | With no FILE, or when FILE is -, read standard input. Output is written to standard output |
| | 6 | == Example |
| | 7 | See what is kept and what is discarded from a [attachment:https://corpus.tools/raw-attachment/wiki/Justext/nlp_jusText_fi.jpg typical web page]. |
| | 22 | } |
| | 23 | |
| | 24 | @misc{Zamazal2024thesis, |
| | 25 | author = {Zamazal, Kryštof}, |
| | 26 | title = {Evaluation of web page cleaning tool Justext}, |
| | 27 | year = {2024}, |
| | 28 | type = {Bachelor's thesis}, |
| | 29 | school = {Masaryk university, Faculty of informatics, Brno, Czech Republic}, |
| | 30 | supervisor = {Vít Suchomel} |
| | 31 | } |
| | 32 | |
