Spoiler: it's not more automation.
Last week Meta laid off some more people I used to work with, specifically people with 10+ years of experience in T&S. It reminded me of a warning the Electronic Frontier Foundation (EFF) made at the start of the COVID-19 pandemic: "protocols adopted in times of crisis often persist when the crisis is over.” They were referring to the increase in automated moderation in lieu of human oversight. For Meta’s most recent round of layoffs, there wasn’t even a crisis. This month, the EFF came back to that warning in a two-part series, "Automated Moderation Is Here to Stay.”
I was there during the exact window mentioned in the EFF article, starting from the early introduction of automated moderation tools around 2017, through to the 2020 COVID era when so much of the moderation work moved toward heavy automation. From 2020 to 2022 I was on the Adult Nudity team at Meta, which had the highest volume of incoming reports - over a million pieces of content a day. Of course it had to be automated. And stay automated.
Reading their 2020 piece back now, the one ask that hasn't aged well is "temporary." Their asks around more transparency and appeals hold up, mainly due to regulations. But asking platforms to make automation temporary was like asking someone who'd just switched to GPS to go back to unfolding a paper map. No T&S team who'd just watched automation clear a massive backlog was going to volunteer to unwind it once the pressure eased, and even if they did, no leadership was going to sign off on it.
So when I read that automated moderation is here to stay, I first thought "obviously". But as I read on, some familiar themes came up. Censorship, bias, transparency, how to audit. And the human moderators who need to train the automation and whose work gets most impacted by automation. Last week I saw so many highly experienced, tenured T&S people get affected. So I have to ask: how do the layoffs keep falling on the very people meant to check these systems, in favor of the systems themselves? We keep adding automation and cutting the people who know when it has gone wrong. Just look at the recent Discord news for example.
Earlier this month, Discord announced that their automated CSAM detection permanently banned around 8,200 accounts over 2 months. The images that set off the bans were grid-like images like chessboards and Minecraft inventories. Maybe it was a bad hash, or their model mistaking the images for an obfuscation pattern. It's easy to say that the "AI" got it wrong. Yes, a CSAM hash system producing false positives on low-detail images is not surprising. The real failure came after the flag. Some of these banned users were reviewed by a human, cleared, and stayed banned regardless. The "don't worry, there's a human in the loop" safeguard was there, but the human's decision to clear them never reached the enforcement system and no one noticed for two months.
That's why I don't think it's a matter of whether we should automate or not. The EFF is right that automation isn't going anywhere, and it shouldn't. Unless you have Meta/Google/Amazon levels of resources to put behind your content moderation workflows, there is no way T&S teams will be able to handle that without automation (and even the big tech giants can't). So the "automate or don't" argument is basically over. Discord raises a different question: when a human overrules the machine, does the overrule take effect? And, who monitors the system-level health? It's the 2,000-year-old question Juvenal asked: who watches the watchmen? Except in content moderation, the watchmen are now the models.
Agentic AI can be one part of the solution. A model that returns its reasoning rather than only a decision gives human moderators the opportunity to audit the reasoning rather than a binary decision. It surfaces policy gaps, and it can ease some of the burden on human moderators, reduce mental fatigue and make operations more efficient.
However, none of that helps if the automation is a black box the operational team doesn't understand or can't control. Because if the people accountable for the automation can't operate it themselves, that accountability isn't real. With agentic AI, the policy is written into the model as a prompt, so a T&S team can write and edit the rules in plain language, the same way they would write guidance for a human moderator. If the model starts over-flagging a category, they can change the wording or the threshold and see what it does to precision on a sample set, without waiting for an engineering sprint. In effect, they can speak to the automation the way they have always spoken to their moderators.
Here I'll add a disclaimer that I'm obviously biased, because this is what we've built at Checkstep. I'm not claiming this prevents all mistakes or solves all issues. It will still come with a rate of false positives, and sometimes a confidently written rationale can make a wrong decision seem reasonable to an auditor moving quickly. And transparency to your own moderators is not the same as transparency to the users, though agentic AI can at least help craft clearer explanations of a decision for the affected user, and generate the statements of reason the DSA requires.
But I have to recognize that there is tension in what I'm saying. Tooling like this can go two ways. It can move experienced moderators up into oversight, where they design and audit the systems, or it can become the reason to cut their jobs. When the EFF followed up with their recommendations, the very first one was that automation should assist human moderators rather than replace them. Terry Pratchett put it more bluntly. When asked who watches the watchmen, his character Sam Vimes just says: "Me". The people working in T&S were the "Me’s” watching the automated watchmen - my former colleagues at Meta who spent years learning to adjust and audit complex automated systems, the ones who understand the system well enough to catch it when it's wrong.
So, who watches the watchmen? For now, still a "Me". The layoffs are working on that.
If you're navigating this exact tension - scaling automated moderation while keeping real human accountability behind it - this is exactly the problem Checkstep helps solve. We help T&S teams build agentic AI that surfaces its reasoning rather than a black-box verdict, keeps a human overrule enforceable end-to-end, and gives your own moderators the ability to edit policy in plain language rather than waiting on an engineering sprint.
If you'd like to learn more, we're offering an exclusive 1-1 with Deniz for a limited time only. Click the link below to schedule a time to meet.
