Lancaster EPrints

From legacy encodings to Unicode: the graphical and logical principles in the scripts of South Asia.

Hardie, Andrew (2007) From legacy encodings to Unicode: the graphical and logical principles in the scripts of South Asia. Language Resources and Evaluation, 41 (1). pp. 1-25. ISSN 1574-020X

Full text not available from this repository.

Abstract

Much electronic text in the languages of South Asia has been published on the Internet. However, while Unicode has emerged as the favoured encoding system of corpus and computational linguists, most South Asian language data on the web uses one of a wide range of non-standard legacy encodings. This paper describes the difficulties inherent in converting text in these encodings to Unicode. Among the various legacy encodings for South Asian scripts, the most problematic are 8-bit fonts based on graphical principles (as opposed to the logical principles of Unicode). Graphical fonts typically encode several features in ways highly incompatible with Unicode. For instance, half-form glyphs used to construct conjunct consonants are typically separate code points in 8-bit fonts; in Unicode they are represented by the full consonant followed by virama. There are many more such cases. The solution described here is an approach to text conversion based on mapping rules. A small number of generalised rules (plus the capacity for more specialised rules) captures the behaviour of each character in a font, building up a conversion algorithm for that encoding. This system is embedded in a font-mapping program, outputting CES-compliant SGML Unicode. This program, a generalised text-conversion tool, has been employed extensively in corpus-building for South Asian languages.

Item Type: Article
Journal or Publication Title: Language Resources and Evaluation
Uncontrolled Keywords: Unicode - Font - Devanagari - South Asian languages/scripts - Legacy text - Encoding - Conversion - Virama - Conjunct consonant - Vowel diacritic
Subjects: P Language and Literature > P Philology. Linguistics
Departments: Faculty of Arts & Social Sciences > Linguistics & English Language
ID Code: 1075
Deposited By: Dr Andrew Hardie
Deposited On: 30 Jan 2008 13:50
Refereed?: Yes
Published?: Published
Last Modified: 18 Sep 2013 15:57
Identification Number:
URI: http://eprints.lancs.ac.uk/id/eprint/1075

Actions (login required)

View Item