Context is important, especially in Trust & Safety.
In this exclusive interview with JustGiving, an FCA-regulated online fundraising platform, we discuss what it takes to build moderation systems that are accurate, scalable, and human enough for the contexts in which they operate.
JustGiving's platform hosts sensitive content that could be considered upsetting, but that also plays a key role in raising awareness of critical issues. For them, every false positive is a learning curve.
Amy Somerford, Trust and Safety Lead, shares how moderation works in practice for an online platform that sits at a unique intersection of vulnerability and trust. She goes into how JustGiving uses AI to scale its Trust & Safety workflows - and how human oversight is critical to delivering moderation with empathy.
Watch the video for the full interview. Prefer to read? Check out our detailed summary below.
Trust and Safety has come a long way in the last 20 years. What now has dedicated teams and technology was once a small group of people working to solve an issue no one had been thinking about yet. With the introduction of AI content moderation into Trust & Safety, how do brands navigate teaching AI to understand context, and keep that all-important ‘human touch’?
Amy Somerford is the Trust & Safety lead at JustGiving, an online donation and fundraising platform. Right now, she sits at the intersection of both Trust and Safety and compliance, but her career actually started within the finance team, leaning more into the compliance side. Amy has worked at JustGiving for an impressive 11 and a half years, giving her a unique perspective on what works and what doesn't.
“It's not just about taking stuff down — it's also about helping us understand if we have vulnerable customers on the platform and how we can better support them, which has been really helpful”
Where does AI content moderation come into JustGiving's moderation stack, and where is it genuinely good at its job? We all know AI is good at tackling patterns at scale, but JustGiving also leverages AI for a separate reason. It uses the information flagged by Checkstep to identify potentially vulnerable customers. Then, its team takes action to try and safeguard and support them. Ultimately, it shows the ongoing need for a human in the process, someone who understands the story and what an individual may be going through. Fundraising campaigns should really be held in a category of their own; they often use highly emotional language that, in other spaces, you'd often train the AI to flag as a violation.
"The more we understand what's on the platform, not just what the AI model is learning - but what we're enforcing and not enforcing - the better, and that's true for our moderators too".
A battle JustGiving has faced has been fine-tuning AI, so that there is less misunderstanding of context and fewer false positives. The reality is that it's a battle that will always rage on. Situations change, another event happens, and language changes - it's about keeping up with those trends through constant iteration of the rules. The lesson Amy has learned is that sometimes, even when there are a lot of false positives, brands can use that for learning. Constant iteration, not a lot of content, or sometimes vagueness can actually be a flag in itself. That's where you need a human to notice it. Checkstep and JustGiving's perspective is that brands should never try to remove the human from the equation. There's so much pressure right now to show AI adoption, and often Trust & Safety teams lack a full understanding of what that means on their platform. If adopted well, AI should better equip your team to make the judgment calls they need to make.
"It's very easy to want to jump straight in and see quick results, but I don't think that's always the right answer. You can miss things that way".
Integrating moderation across a platform takes a lot of time and patience.
By using the Checkstep platform, JustGiving was able to take time to customise its policies, test out how they play out in real time and train both the AI and their moderators.
If you'd like to learn more, Checkstep is filled to the brim with Trust & Safety experts. Book a meeting with us today if you want to learn more about Trust & Safety or AI content moderation strategies.
