Automate the Obvious, Review the Rest: Designing AI-Powered Content Moderation

说明与用途

AI驱动的内容审核系统，自动处理明确案例，模糊案例交给人工审核，实现可扩展的审核流程。

问题

Moderating user‑generated content at scale without delaying uploads or over‑relying on human review.

结果

The system achieves scalable moderation by automating clear cases, escalating ambiguous ones, and maintaining an audit trail, reducing human workload while protecting the feed.

工作流

• 1. User creates content (text or media).
• 2. Text: row saved, Hasura event triggers NestJS, creates content record, enqueues BullMQ job.
• 3. Media: uploaded to S3, ObjectCreated triggers SNS, fan‑out to ingest Lambda.
• 4. Text worker calls OpenAI Moderation API, converts max category score to risk score, applies thresholds (40/75).
• 5. Media scorer Lambda calls Amazon Rekognition, applies thresholds (75/90).
• 6. Results stored in content_moderation_status and moderation_results tables.
• 7. Content with status approved is shown in feeds; under_review or rejected stay hidden until moderator action or owner notification.

证据

We needed a moderation system that could inspect captions, comments, ads, stories, images, and videos as soon as they were created without delaying uploads or posts for human review or automating every borderline case.

来源

Text goes through NestJS, Hasura event triggers, BullMQ, and the OpenAI Moderation API. Images and videos go through S3, SNS, SQS, AWS Lambda, and Amazon Rekognition.

来源

Risk below 40 → approved. The content can enter the feed. Risk from 40 up to (but not including) 75 → under_review. Risk 75 or above → rejected.

来源

Images/video (Rekognition): start stricter on auto-approve than text (review at 75%, reject at 90%).

来源