We are automating judgement before we understand it

What worries me is not that models are too powerful. It is that we keep handing them decisions whose reasoning we could never reconstruct, in domains where being wrong ruins a life. Bail, credit, hiring, visas. A system that is right ninety four percent of the time is a catastrophe when the remaining six percent is always the same kind of person. I am not a luddite, I build these things. But 'the model beats the human baseline' has become a way to end conversations that should be starting.