Automate the Obvious, Review the Rest: Designing AI-Powered Content Moderation

Description and use

An AI‑powered content moderation system that triages clear cases automatically and escalates ambiguous ones, scaling moderation while protecting the feed.

Problem

Moderating user‑generated content at scale without delaying uploads or over‑relying on human review.

Outcome

The system achieves scalable moderation by automating clear cases, escalating ambiguous ones, and maintaining an audit trail, reducing human workload while protecting the feed.

Workflow

• 1. User creates content (text or media).
• 2. Text: row saved, Hasura event triggers NestJS, creates content record, enqueues BullMQ job.
• 3. Media: uploaded to S3, ObjectCreated triggers SNS, fan‑out to ingest Lambda.
• 4. Text worker calls OpenAI Moderation API, converts max category score to risk score, applies thresholds (40/75).
• 5. Media scorer Lambda calls Amazon Rekognition, applies thresholds (75/90).
• 6. Results stored in content_moderation_status and moderation_results tables.
• 7. Content with status approved is shown in feeds; under_review or rejected stay hidden until moderator action or owner notification.

Evidence

We needed a moderation system that could inspect captions, comments, ads, stories, images, and videos as soon as they were created without delaying uploads or posts for human review or automating every borderline case.

Source

Text goes through NestJS, Hasura event triggers, BullMQ, and the OpenAI Moderation API. Images and videos go through S3, SNS, SQS, AWS Lambda, and Amazon Rekognition.

Source

Risk below 40 → approved. The content can enter the feed. Risk from 40 up to (but not including) 75 → under_review. Risk 75 or above → rejected.

Source

Images/video (Rekognition): start stricter on auto-approve than text (review at 75%, reject at 90%).

Source