JusText

Description

JusText is a heuristic based boilerplate removal tool useful for cleaning documents in large textual corpora. The tool has been implemented in Python, licensed under New BSD License and made an open source software (available for download including the source code at ​http://code.google.com/p/justext/). It is successfully used for cleaning large textual corpora at Natural language processing centre at Faculty of informatics, Masaryk university Brno and it's industry partners. The research leading to this piece of software was published in author's Ph.D. thesis "Removing Boilerplate and Duplicate Content from Web Corpora". The boilerplate removal algorithm is able to remove most of non-grammatical sentences from a web page like navigation, advertisements, tables, short notes and so on. It has been shown it overperforms or at least keeps up with it's competitors (according to comparison with participants of Cleaneval competition in author's Ph.D. thesis). The precise removal of unwanted content and scalability of the algorithm has been demonstrated while building corpora of American Spanish, Arabic, Czech, French, Japanese, Russian, Tajik, and six Turkic languages consisting --- over 20 TB of HTML pages were processed resulting in corpora of 70 billions tokens altogether. and PRESEMT, Lexical Computing Ltd

Example

See what is kept and what is discarded from a typical web page.

Source

​https://corpus.tools/wiki/Justext

Acknowledgements

This software has been developed at the ​Natural Language Processing Centre of ​Masaryk University in Brno with financial support from ​PRESEMT and ​Lexical Computing Ltd. It also relates to Jan Pomikálek's ​PhD research.

If you use the system, please cite the related publication as well as the LINDAT/CLARIAH infrastructure: ​http://hdl.handle.net/11858/00-097C-0000-000D-F696-9

@phdthesis{pomikalek2011removing,
  title={Removing boilerplate and duplicate content from web corpora},
  author={Pomik{\'a}lek, Jan},
  school={Masaryk university, Faculty of informatics, Brno, Czech Republic},
  year={2011}
}

@misc{Zamazal2024thesis,
  author = {Zamazal, Kryštof},
  title = {Evaluation of web page cleaning tool Justext},
  year = {2024},
  type = {Bachelor's thesis},
  school = {Masaryk university, Faculty of informatics, Brno, Czech Republic},
  supervisor = {Vít Suchomel}
}

License

Justext is licensed under the ​BSD 3-Clause License