Cory Doctorow #BLM @doctorow · Sep 03

Here's your heartbreaking algorithmic cruelty story of the week: UC Riverside history prof @DanaJSimmons has a son who's just started junior high and loves his history teacher:

https://t.co/KZcqy13OMK

1/ https://t.co/k2BnHjuu6a

Cory Doctorow #BLM @doctorow · Sep 03

But when he submitted his first assignment, he was aghast to receive a 50% grade on it.

The assignment was graded by @EdgenuityInc, a machine learning grift that purports to generate grades for overworked, beleagured teachers.

2/

Cory Doctorow #BLM @doctorow · Sep 03

Simmons calls it an "automatic grading algorithm that values only rote repetition," and to prove the point, she told her son how to please the machine: long sentences with a lot of proper names.

3/

Cory Doctorow #BLM @doctorow · Sep 03

It worked: now her son submits "word salads" consisting of two sentences (long ones, presumably) and a bunch of keywords from the lesson, and is consistently earning 100% grades. As Simmons says, "He went from an F to an A+ without learning a thing."

4/

Cory Doctorow #BLM @doctorow · Sep 03

In @mathbabedotorg's seminal "Weapons of Math Destruction," she doesn't just provide a devastating critique of the underlying statistical basis for machine learning, but also a set of VERY useful rules of thumb for spotting AI grifters.

https://t.co/YGw1rnY7OR

5/

Cory Doctorow #BLM @doctorow · Sep 03

First: if a company provides an AI but doesn't check its predictions, they don't give a shit about its accuracy.

Think about it: if Amazon uses ML to predict whether moving the BUY button will generate more sales, the DEFINITELY measure whether sales after moving it.

6/

Cory Doctorow #BLM @doctorow · Sep 03

The fact that this automated grading system produces estimates that are so easily gamed tells you it is a straight-up grift. I mean, the Bayesian spam filters of the mid-2000s were able to detect the "word salad" attack.

7/

Cory Doctorow #BLM @doctorow · Sep 03

This isn't one of those subtle, amazing "adversarial example" attacks on a ML model - like that weird thing where a vision system is tricked into thinking that a rifle is a helicopter:

https://t.co/efhpoA4Mo8

8/

Cory Doctorow #BLM @doctorow · Sep 03

It's more like a password prompt that you can bypass just by hitting the spacebar a bunch of times.

That is: negligent garbage serving no pedagogical purpose, an embodiment of the enterprise software pathology where the person who buys the product doesn't have to use it.

9/

Cory Doctorow #BLM @doctorow · Sep 03

Simmons is admirably compassionate about the whole affair: "teaching online is overwhelming and you can't do it all. _Please_, use the algorithm to track their learning. But don't post to them as if it's a measure of their performance. It's more destructive than you know."

10/

Cory Doctorow #BLM @doctorow · Sep 03

Amen.

School boards shouldn't be buying this tool for teachers.

Teachers shouldn't be using it to assess students.

Students should not be made to see those assessments.

Image:
Cryteria (modified)
https://t.co/ICebVcdH1f

CC BY:
https://t.co/5YJhpDj3vT

eof/