Lancaster EPrints

Classification procedures for software evaluation

Amar, Muriel and David, Sophie and Panckhurst, Rachel and Whistlecroft, Lisa (2008) Classification procedures for software evaluation. In: Language Resources and Evaluation Conference (LREC) 2008, 2008-05-282008-05-30, Marrakech, Morocco.

[img]
Preview
PDF (160_paper.pdf)
Download (438Kb) | Preview

    Abstract

    We outline a methodological classification for evaluation approaches of software in general. This classification was initiated partly owing to involvement in a biennial European competition (the European Academic Software Award, EASA) which was held for over a decade. The evaluation grid used in EASA gradually became obsolete and inappropriate in recent years, and therefore needed to be revised. In order to do this, it was important to situate the competition in relation to other software evaluation procedures. A methodological perspective for the classification is adopted rather than a conceptual one, since a number of difficulties arise with the latter. We focus on three main questions: What to evaluate? How to evaluate? and Who does evaluate? The classification is therefore hybrid: it allows one to account for the most common evaluation approaches and is also an observatory. Two main approaches are differentiated: system and usage. We conclude that any evaluation always constructs its own object, and the objects to be evaluated only partially determine the evaluation which can be applied to them. Generally speaking, this allows one to begin apprehending what type of knowledge is objectified when one or another approach is chosen.

    Item Type: Conference or Workshop Item (Paper)
    Journal or Publication Title: Language Resources and Evaluation Conference (LREC) 2008
    Uncontrolled Keywords: Evaluation ; Methodological classification ; Software ; Competitions ; TREC ; MUC ; EASA ; Epistemology
    Subjects: Z Bibliography. Library Science. Information Resources > Z665 Library Science. Information Science
    P Language and Literature > P Philology. Linguistics
    Departments: Faculty of Arts & Social Sciences > Lancaster Institute for the Contemporary Arts
    ID Code: 23619
    Deposited By: Lisa Whistlecroft
    Deposited On: 13 Feb 2009 11:47
    Refereed?: Yes
    Published?: Published
    Last Modified: 27 Jul 2012 02:01
    Identification Number:
    URI: http://eprints.lancs.ac.uk/id/eprint/23619

    Actions (login required)

    View Item