Keynote 3: John de Jong - Innovative Items: Measuring Something More or Different

John de Jong starts with a short introduction to language testing. Paper-based tests lack the immediate response/award/feedback factor that we know from for example gaming machines or tablet technology and which can be achieved using machine testing.

Automated essay scoring tends to be mistrusted, the general view being that machines cannot be taught to interpret meaning. However, de Jong takes us through recent developments that seem to show that IEA (intelligent essay assessor) can do just that: a machine learning to score like human markers by measuring different aspects of the responses collected from a large body of text input. Such scoring uses latent semantic analysis to score content Latent semantic analysis (LSA) is based on the machine reading vast amounts of texts, learning what words mean and how they relate to each other, ie it learns the concepts, rather than just the vocabulary resulting in the creation of a semantic space. The following slide summarises the concept:

In terms of reliability and validity LSA has been tested on millions of essays and its scoring vis-a vis a human rater compares well to human raters vis-a-vis human raters. De Jong goes on to say that such automatic processes are already being used frequently, for example in Interwiki bots, an automated system that immediately takes off text written into Wikipedia that is not relevant to the area in question. Associated Press agency has also started using software that will automatically generate thousands of financial reports without the need for reporters. The presentation then turned to the challenges in assessing 21. century skills, and the following slides summarise the most salient points:

De Jong gives further application examples of the machine, such as oral exams. Here, the machine acts as interlocutor, always adapting to the level of the student. For De Jong, standardisation of oral exams is not really possible when using human interlocutor because of different affective factors. Using machine interlocuteurs would ensure that each students is tested against the same interlocuteur/rater. A further area for machine application for de Jong are tandem programmes, i.e. the machine can becomes a partner in collaborative projects. In closing, de Jong urges us to think about further possibilities of integrating technologies into our work, as well as exploring the idea of how the role of the teacher might be changing in the future.