News & Events: Newsletter

Feature: The Future of Computerized Language Assessments

By Chris Dalessandri

When it comes to computerized language assessments, one of the questions people most commonly ask is, "Does your software automatically rate speech samples?" Even among the most knowledgeable language instruction professionals, the misconception lingers that computers can evaluate and rate communicative speech and writing samples. Why so much confusion about what computers can and cannot do?

As ACTFL President Ray Clifford noted in his President's Message in the April 2008 edition of The Language Educator, the Wall Street Journal predicted in 1998 that within 10 years, computer-powered translation would eliminate the need for the study of foreign language. Movies and television may raise our expectations as well. Remember the heroes of Independence Day programming their computer to "talk" to an alien computer? Or the ship’s computer in Star Trek translating alien speech as the Enterprise faced an unknown race? Between respected but misinformed media and Hollywood's pervasive imagination, it is no wonder people are confused!

So what is the current capability of computers in regard to rating speech samples? While a great deal of technology can be applied to speaking assessments, human speech is very complex and computers are not yet capable of understanding it. As the President of OWL Testing Software Company, I have the opportunity and the responsibility to stay abreast of the advances in technology as related to language assessment and instruction. OWL staff members—myself included—attend and present sessions at leading national and international conferences, keep abreast of major technical innovations through journal publications, and engage in face-to-face discussions with leaders in many fields related to computer-assisted language learning. Recently, I attended a presentation by researchers focused on the current "translation" technology. Some military personnel who were present discussed the handheld "translator" devices currently being used in Iraq. The user types a message in English and the device will speak the phrase in Arabic. The device has no ability to comprehend speech, and there is no data yet on how successfully the user has conveyed the desired message.

We've also checked in with researchers conducting speech recognition projects at leading technical institutions^[1][2], a company CEO whose entire business is analyzing a specialized subset of speech used within a specific industry, and attended a demonstration of advanced speech recognition technologies designed to assist paraplegics. Some of the best minds in the world are working to improve language comprehension technology.

Yet based on everything that we have seen and heard, computer analysis of speaking is still many years away. Yes, there is software that can transcribe a small set of words. However, even without considering the issues of dialects, regional accents, variations in tonality of male, female and children's speech, one can say the current, limited state of word recognition software programs is not equivalent to actual language comprehension, let alone evaluating and rating speech samples.

Can a computer recognize that a test taker used 50 words in the target language? Maybe. Some very expensive systems can recognize a limited number of specific vocabulary items, as long as the test taker speaks with the correct tempo, pitch, and accent. But can a computer determine whether or not the test taker actually understands those words or if he/she can put them together in a sensible order? Not yet. This is a much more complex problem than playing chess, for instance. When a computer plays chess, it can decisively choose among the limited number of moves that are possible from any given position. Unlike chess, human speech is extremely complex with an almost infinite number of possible responses to any given situation. Speech is filled with meaning and nuance, and many words have multiple meanings depending on context, intonation, etc. Computers have not yet achieved the sophistication of artificial intelligence necessary to stand in for the human brain.

Despite such limitations, however, computers offer a variety of benefits for language assessments. Computers are especially effective at removing the constraints of time and space from assessments; test-takers, administrators and raters don’t have to be in the same place or working at the same time to achieve their objectives. Computers make it much easier to create and edit tests, administer them, and assist in rating and scoring in prescribed ways. They provide a way to deliver many assessments simultaneously. Computers are also extremely good at storing vast amounts of data and they facilitate easy data retrieval. And computers are good at delivering a wide range of audio and video. Computers simply cannot be used for automated evaluation and rating of open-ended speech and writing samples. Yet.

In the meantime, what work can be done to improve the field? To quote ACTFL President Ray Clifford^[3] once again, "one of the greatest challenges facing world language educators is conveying clear expectations of what a language learner will be able to do upon completion of a course of study." Better assessments combined with easily understood and agreed upon standards will make this possible. One way to achieve this is to implement a computerized assessment program that can make assessment and the reporting and analysis of results as easy as possible. OWL Testing Software is committed to providing state-of-the-art assessment software and will introduce new, useful features as advancements are made in technology. FOCUS One of the exciting new features coming next quarter is the new Portfolio Export Feature. How does it work? Imagine a student has been studying Spanish for four years. Every task completed and assessment taken with OWL software, including the student’s speech samples, can be output to a CD, flash drive or other portable media. The student’s portfolio can then be shared with their prospective college or employer.

Chris Dalessandri is the President & CEO of OWL Testing Software. Questions or comments? Write him at chris@owlts.com.

¹http://www.nist.gov/speech/
²http://www.speech.cs.cmu.edu/
³President’s Message, Foreign Language Annals, Vol 41, No. 2, Summer

Focus on Features: Portfolio Export

One of the exciting new features coming next quarter is the new Portfolio Export Feature. How does it work?

Imagine a student has been studying Spanish for four years. Every task completed and assessment taken with OWL software, including the student'’s speech samples, can be output to a CD, flash drive or other portable media. The student's portfolio can then be shared with their prospective college or employer.

OWL Partner: Babowal & Associates

With the prevalence of global markets, and multilingual and multicultural workforces, employers need to accurately determine how well their present and future workers communicate in English. A new partnership between Babowal & Associates, Inc., a leading workforce development firm, and OWL Testing Software will simplify the process of assessing employees’ ability to effectively communicate in English with customers and clients.

Babowal & Associates has been developing standardized English assessment tools since 1991 for use in a variety of business, industry and academic settings. Founder Chris Babowal, a member of Teachers of English to Speakers of Other Languages (TESOL)’s Standards Committee, is enthusiastic about partnering with OWL Testing Software to expand the capabilities of their proven testing and certificate programs. "Our partnership with OWL Testing Software will permit us to more efficiently deliver and score our English exams."