Chared

Description

Chared is a software tool which can detect character encoding of a text document provided the language of the document is known. The language of the text has to be specified as an input parameter so that the corresponding language model can be used. The package contains models for a wide range of languages (currently 57 --- covering all major languages). Furthermore, it provides a training script to learn models for additional languages using a set of user supplied sample html pages in the given language. The detection algorithm is based on determining similarity of byte trigrams vectors. In general, chared should be more accurate than other character encoding detection tools with no language constraints. This is an important advantage allowing precise character decoding needed for building large textual corpora. The tool has been used for building corpora in American Spanish, Arabic, Czech, French, Japanese, Russian, Tajik, and six Turkic languages consisting of 70 billions tokens altogether.

Source

​https://corpus.tools/wiki/Chared

Acknowledgements

This software has been developed at the ​Natural Language Processing Centre of ​Masaryk University in Brno with financial support from ​PRESEMT and ​Lexical Computing Ltd.

If you use the system, please cite the related publication as well as the LINDAT/CLARIAH infrastructure: ​http://hdl.handle.net/11858/00-097C-0000-000D-F67A-9

@inproceedings{pomikalek2011character,
   title = {chared: Character Encoding Detection with a Known Language},
   author = {Pomikálek, Jan and Suchomel, Vít},
   booktitle = {RASLAN 2011},
   pages = {125--129},
   year = {2011},
   address = {Brno, Czech Republic},
   editor = {Horák, Aleš and Rychlý, Pavel},
   location = {Brno, Czech Republic},
   publisher = {Tribun EU},
   isbn = {978-80-263-0077-9}

Other

​Unicode over 60 percent of the web at Google blog