Not a Luddite

“Because I’m not a Luddite.” This was Les Perelman’s response when asked, in a Boston Globe interview, why he joined a web-based writing tutorial company after years of railing against computerized writing evaluations. Having recently retired from the directorship of MIT’s Writing Across the Curriculum program, Perelman is now the chief research scientist for WriteLab, a startup company that has partnered with dozens of college writing centers. WriteLab uses computerized algorithms to offer students feedback on their writing and guide them toward revisions.

Writelab Logo

As Perelman noted in the Globe interview, the software is not a replacement for a human teacher, but rather a supplement. By providing suggestions and questions, it not only facilitates improvement but helps students become more aware of their writing, whether they defend or reject their original choices. Perelman explained in the interview that automated writing instruction can be valuable despite computers’ shortcomings, and that he got involved because, “…if we don’t do it well, other people are going to do it badly.”

Doing it badly is what Perelman became concerned about a few years ago when the Educational Testing Service, which develops and administers the SAT, developed an e-Rater to automatically grade students’ essays. The New York Times noted that Perelman exposed significant flaws in the system by showing that he was able to earn high scores by submitting to the e-Rater prose that was essentially gibberish. Included in his findings was that the e-Rater values number and size of words over truth and logical coherence.

For example, the e-Rater generated positive feedback in response to this:
Competition which mesmerizes the reprover, especially of administrations, may be multitude. As a result of abandoning the utterance to the people involved, a plethora of cooperation can be more tensely enjoined. Additionally, a humane competition changes assemblage by cooperation. In my semiotics class, all of the agriculturalists for our personal interloper with the probe we decry contend.. . .

Clearly, Perelman had good reason to be cynical about algorithms’ evaluative and informative capabilities, although ETS disputed his findings and conclusions. However, he also has good reason to have some faith in them; and his move to WriteLab may signify that understanding. As anyone who uses a GPS knows, technology certainly can be harnessed for helpful guidance. What one recent study found, though, is that people actually tend to underestimate how much algorithms should be trusted. When researchers at the University of Pennsylvania had subjects observe and then choose between a human or statistical model to make predictions, the subjects were more likely to pick the human model. These results followed a number of other studies’ findings regarding the tendency to dismiss algorithms. In various domains such as stock forecasts or medical decisions, people tended to favor human judgement. However, research suggests that mechanical predictions often beat personal judgement, contrary to what we might be inclined to think is the case.

An openness to the power of technology along with a healthy skepticism and understanding of its limitations seems to be the best approach, in education and in general. Because, after all, we are not Luddites.